Working with the MagIC database using PmagPy¶
- The Magnetics Information Consortium (MagIC) maintains a database of published rock and paleomagnetic data: https://www.earthref.org/MagIC
- Many PmagPy scripts are designed to work with data in the MagIC format. This notebook uses Data Model 3.0: https://www.earthref.org/MagIC/data-models/3.0 There are nine basic tables: contribution, locations, sites, samples, specimens, measurements, criteria, ages and images. These are tab delimited data tables with the first line consisting of a delimiter and the table name: (e.g., tab measurements). All of the examples here are tab delimited. The second line are the column names: (e.g., specimen experiment method_codes treat_temp.....). Each subsequent line is a single record.
See the first few lines of this sample file below:
with open('data_files/3_0/McMurdo/samples.txt') as f:
for line in f.readlines()[:3]:
print(line, end="")
Guide to PmagPy¶
The notebook is one of a series of notebooks that demonstrate the functionality of PmagPy. The other notebooks are:
- PmagPy_introduction.ipynb This notebook introduces PmagPy and lists the functions that are demonstrated in the other notebooks.
- PmagPy_calculations.ipynb This notebook demonstrates many of the PmagPy calculation functions such as those that rotate directions, return statistical parameters, and simulate data from specified distributions.
- PmagPy_plots_analysis.ipynb This notebook demonstrates PmagPy functions that can be used to visualize data as well as those that conduct statistical tests that have associated visualizations.
Running this notebook¶
You are currently looking at static html, but you may want to run this notebook interactively. To do so, you'll need to first install Python and PmagPy (see the Cookbook for install instructions). You can then launch the notebook from your command line (see more details here).
Get started¶
To use the functions in this notebook, we have to import the PmagPy modules pmagplotlib, pmag and ipmag and some other handy functions. This is done in the following code.
In order to access the example data, these examples are meant to be run in the PmagPy-data directory (PmagPy directory for developers).
import pmagpy.pmag as pmag
import pmagpy.pmagplotlib as pmagplotlib
import pmagpy.ipmag as ipmag
import pmagpy.contribution_builder as cb
from pmagpy import convert_2_magic as convert
import matplotlib.pyplot as plt # our plotting buddy
import numpy as np # the fabulous NumPy package
import pandas as pd # and of course Pandas
# test if Basemap and/or cartopy is installed
has_basemap, Basemap = pmag.import_basemap()
has_cartopy, Cartopy = pmag.import_cartopy()
# test if xlwt is installed (allows you to export to excel)
try:
import xlwt
has_xlwt = True
except ImportError:
has_xlwt = False
# This allows you to make matplotlib plots inside the notebook.
%matplotlib inline
from IPython.display import Image
import os
print('All modules imported!')
Table of contents¶
- Functions in PmagPy_MagIC.ipynb
- reading MagIC files : reading in MagIC formatted files
- writing MagIC files : outputing MagIC formatted files
- combine_magic : combines two MagIC formatted files of same type
- convert_ages : convert ages in downloaded MagIC file to Ma
- grab_magic_key : prints out a single column from a MagIC format file
- magic_select : selects data from MagIC format file given conditions (e.g., method_codes contain string)
- sites_extract : makes excel or latex files from sites.txt for publications
- criteria_extract : makes excel or latex files from criteria.txt for publications
- specimens_extract : makes excel or latex files from specimens.txt for publications
- contributions work with data model 3.0 MagIC contributions
- download_magic : unpacks a contribution text file downloaded from the MagIC website
- upload_magic : prepares a directory with a MagIC contribution for uploading to MagIC
- cb.add_sites_to_meas_table : completes a measurements data frame with the information required for plotting by site.
- cb.get_intensity_col : finds the first non-zero type of intensity data in a measurements dataframe.
- conversion scripts : convert many laboratory measurement formats to the MagIC data model 3 format
- _2g_asc_magic : converts 2G ascii files to MagIC
- _2g_bin_magic : converts 2G binary files to MagIC
- agm_magic : converts Princeton Measurements alternating gradient force magnetization (AGM) files to MagIC.
- bgc_magic : convert Berkeley Geochronology Center files to MagIC.
- cit_magic : convert Cal Tech format files to MagIC.
- generic_magic : converts generic files to MagIC.
- huji_magic : converts Hebrew University, Jerusalem, Israel files to MagIC.
- huji_sample_magic : converts HUJI files to a MagIC format.
- jr6_jr6_magic : converts the AGICO JR6 spinner .jr6 files to MagIC
- jr6_txt_magic : converts the AGICO JR6 .txt files to MagIC
- k15_magic : converts 15 measurement anisotropy of magnetic susceptibility files to MagIC.
- kly4s_magic : converts SIO KLY4S formatted files to MagIC.
- ldeo_magic : converts Lamont-Doherty files to MagIC.
- livdb_magic : converts Liverpool files to MagIC.
- mst_magic : converts Curie Temperature experimental data to MagIC
- sio_magic : converts Scripps Institution of Oceanography data files to MagIC
- sufar4_magic : converts AGICO SUFAR program (ver.1.2.) ascii files to MagIC
- tdt_magic : converts Thellier Tool files to MagIC
- utrecht_magic : converts Fort Hoofddijk, Utrecht University Robot files to MagIC
- orientation_magic : converts an "orient.txt" formatted file with field notebook information into MagIC formatted files
- azdip_magic : converts an "azdip" formatted file to a samples.txt file format
- other handy scripts
- chartmaker : script for making chart to guide IZZI lab experiment
I/O with MagIC data files¶
magic_read¶
MagIC formatted data files can be imported to a notebook in one of two ways: a
- importing to a Pandas DataFrame using the Pandas pd.read_csv() function
importing to a list of dictionaries using the pmag.magic_read() function.
In this notebook, we generally read MagIC tables into a Pandas Dataframe with a command like:
meas_df = pd.read_csv('MEASUREMENTS_FILE_PATH',sep='\t',header=1)
These data can then be manipulated with Pandas functions (https://pandas.pydata.org/)
meas_df=pd.read_csv('data_files/3_0/McMurdo/measurements.txt',sep='\t',header=1)
meas_df.head()
Alternatively, the user may wish to use a list of dictionaries compatible with many pmag functions. For that, use the pmag.magic_read() function:
help (pmag.magic_read)
meas_dict,file_type=pmag.magic_read('data_files/3_0/McMurdo/measurements.txt')
print (file_type)
print (meas_dict[0])
magic_write¶
To write out a MagIC table from a Pandas DataFrame, first convert it to a list of dictionaries using a command like:
dicts = df.to_dict('records')
then call pmag.magic_write().
From a list of dictionaries, you can just call pmag.magic_write() directly.
pmag.magic_write¶
help(pmag.magic_write)
meas_dicts = meas_df.to_dict('records')
pmag.magic_write('my_measurements.txt', meas_dicts, 'measurements')
combine_magic¶
[MagIC Database] [command line version]
MagIC tables have many columns only some of which are used in a particular instance. So combining files of the same type must be done carefully to ensure that the right data come under the right headings. The program combine_magic can be used to combine any number of MagIC files from a given type.
It reads in MagIC formatted files of a common type (e.g., sites.txt) and combines them into a single file, taking care that all the columns are preserved. For example, if there are both AF and thermal data from a study and we created a measurements.txt formatted file for each, we could use combine_magic.py on the command line to combine them together into a single measurements.txt file. In a notebook, we use ipmag.combine_magic().
help(ipmag.combine_magic)
Here we make a list of names of two MagIC formatted measurements.txt files and use ipmag.combine_magic() to put them together.
filenames=['data_files/combine_magic/af_measurements.txt','../combine_magic/therm_measurements.txt']
outfile='data_files/combine_magic/measurements.txt'
ipmag.combine_magic(filenames,outfile)
convert_ages¶
Files downloaded from the MagIC search interface have ages that are in the original units, but what is often desired is for them to be in a single unit. For example, if we searched the MagIC database for all absolute paleointensity data (records with method codes of 'LP-PI-TRM') from the last five million years, the data sets have a variety of age units. We can use pmag.convert_ages() to convert them all to millions of years.
First we follow the instructions for unpacking downloaded files in download_magic.
ipmag.download_magic('magic_downloaded_rows.txt',dir_path='data_files/convert_ages/',
input_dir_path='data_files/convert_ages/')
After some minimal filtering using Pandas, we can convert a DataFrame to a list of dictionaries required by most PmagPy functions and use pmag.convert_ages() to convert all the ages. The converted list of dictionaries can then be turned back into a Pandas DataFrame and either plotted or filtered further as desired.
In this example, we filter for data older than the Brunhes (0.78 Ma) and younger than 5 Ma, then plot them against latitude. We can also use vdm_b to plot the intensities expected from the present dipole moment (~80 ZAm$^2$).
help(pmag.convert_ages)
# read in the sites.txt file as a dataframe
site_df=pd.read_csv('data_files/convert_ages/sites.txt',sep='\t',header=1)
# get rid aof any records without intensity data or latitude
site_df=site_df.dropna(subset=['int_abs','lat'])
# Pick out the sites with 'age' filled in
site_df_age=site_df.dropna(subset=['age'])
# pick out those with age_low and age_high filled in
site_df_lowhigh=site_df.dropna(subset=['age_low','age_high'])
# concatenate the two
site_all_ages=pd.concat([site_df_age,site_df_lowhigh])
# get rid of duplicates (records with age, age_high AND age_low)
site_all_ages.drop_duplicates(inplace=True)
# Pandas reads in blanks as NaN, which pmag.convert_ages hates
# this replaces all the NaNs with blanks
site_all_ages.fillna('',inplace=True)
# converts to a list of dictionaries
sites=site_all_ages.to_dict('records')
# converts the ages to Ma
converted_df=pmag.convert_ages(sites)
# turn it back into a DataFrame
site_ages=pd.DataFrame(converted_df)
# filter away
site_ages=site_ages[site_ages.age.astype(float) <= 5]
site_ages=site_ages[site_ages.age.astype(float) >=0.05]
Let's plot them up and see what we get.
plt.plot(site_ages.lat,site_ages.int_abs*1e6,'bo')
# put on the expected values for the present dipole moment (~80 ZAm^2)
lats=np.arange(-80,70,1)
vdms=80e21*np.ones(len(lats))
bs=pmag.vdm_b(vdms,lats)*1e6
plt.plot(lats,bs,'r-')
plt.xlabel('Latitude')
plt.ylabel('Intensity ($\mu$T)')
plt.show()
That is pretty awful agreement. Someday we need to figure out what is wrong with the data or our GAD hypothesis.
grab_magic_key¶
[MagIC Database] [command line version]
Sometimes you want to read in a MagIC file and print out the desired key. Pandas makes this easy! In this example, we will print out latitudes for each site record.
sites=pd.read_csv('data_files/download_magic/sites.txt',sep='\t',header=1)
print (sites.lat)
magic_select¶
[MagIC Database] [command line version]
This example demonstrates how to select MagIC records that meet a certain criterion, like having a particular method code.
Note: to output into a MagIC formatted file, we can change the DataFrame to a list of dictionaries (with df.to_dict("records")) and use pmag.magic_write()..
help(pmag.magic_write)
# read in the data file
spec_df=pd.read_csv('data_files/magic_select/specimens.txt',sep='\t',header=1)
# pick out the desired data
method_key='method_codes' # change to method_codes for data model 3
spec_df=spec_df[spec_df.method_codes.str.contains('LP-DIR-AF')]
specs=spec_df.to_dict('records') # export to list of dictionaries
success,ofile=pmag.magic_write('data_files/magic_select/AF_specimens.txt',specs,'pmag_specimens') # specimens for data model 3.0
sites_extract¶
It is frequently desirable to format tables for publications from the MagIC formatted files. This example is for the sites.txt formatted file. It will create a site information table with the location and age information, and directions and/or intenisty summary tables. The function to call is ipmag.sites_extract().
help(ipmag.sites_extract)
Here is an example for how to create Latex files:
#latex way:
ipmag.sites_extract(directions_file='directions.tex',intensity_file='intensities.tex',
output_dir_path='data_files/3_0/McMurdo',info_file='site_info.tex',latex=True)
And here is how to create Excel files:
#xls way:
if has_xlwt:
print(ipmag.sites_extract(output_dir_path='data_files/3_0/McMurdo'))
criteria_extract¶
This example is for the criteria.txt formatted file. It will create a criteria table suitable for publication in either LaTex or .csv format. The function to call is ipmag.criteria_extract().
help(ipmag.criteria_extract)
# latex way:
ipmag.criteria_extract(output_dir_path='data_files/3_0/Megiddo',
latex=True,output_file='criteria.tex',)
#xls way:
if has_xlwt:
print(ipmag.criteria_extract(output_dir_path='data_files/3_0/Megiddo'))
specimens_extract¶
Similarly, it is useful to make tables for specimen (intensity) data to include in publications. Here are examples using a specimens.txt file.
help(ipmag.specimens_extract)
#latex way:
ipmag.specimens_extract(output_file='specimens.tex',landscape=True,
output_dir_path='data_files/3_0/Megiddo',latex=True,longtable=True)
#xls way:
if has_xlwt:
print(ipmag.specimens_extract(output_dir_path='data_files/3_0/Megiddo'))
Contributions¶
Here are some useful functions for working with MagIC data model 3.0 contributions.
download_magic¶
[MagIC Database] [command line version]
This program unpacks the .txt files downloaded from the MagIC database into individual text files. It has an option to also separate files for each location
As an example, go to the MagIC data base at http://earthref.org/MAGIC/doi/10.1029/2003gc000661 and dowload the contribution. Make a folder into which you should put the downloaded txt file called MagIC_download and move the file into it. Now use the program download_magic to unpack the .txt file (magic_contribution_16533.txt).
To do this within a notebook, use the function ipmag.download_magic().
help(ipmag.download_magic)
ipmag.download_magic(infile='magic_contribution_16533.txt',\
input_dir_path='data_files/download_magic',dir_path='data_files/download_magic')
You could look at these data with dmag_magic for example...
upload_magic¶
[MagIC Database] [command line version]
We can just turn around and try to upload the file downloaded in download_magic. For this we use ipmag.upload_magic() in the same directory as for the download. You can try to upload the file you create to the MagIC data base as a private contribution here: https://www2.earthref.org/MagIC/upload
help(ipmag.upload_magic)
ipmag.upload_magic(dir_path='data_files/download_magic',concat=True)
If this were your own study, you could now go to https://earthref.org/MagIC and upload your contribution to a Private Workspace, validate, assign a DOI and activate!
cb.add_sites_to_meas_table¶
MagIC data model 3 took out redundant columns in the MagIC tables so the hierarchy of specimens (in the measurements and specimens tables) up to samples, sites and locations is lost. To put these back into the measurement table, we have the function cb.add_sites_to_meas_table(), which is super handy when data analysis requires it.
help(cb.add_sites_to_meas_table)
status,meas_df=cb.add_sites_to_meas_table('data_files/3_0/McMurdo')
meas_df.columns
cb.get_intensity_col¶
The MagIC data model has several different forms of magnetization with different normalizations (moment, volume, or mass). So to find the one used in a particular measurements table we can use this handy function.
help(cb.get_intensity_col)
magn_col=cb.get_intensity_col(meas_df)
print (magn_col)
_2g_asc_magic¶
This conversion has not been written yet. If you have this file format and wish to convert it to the MagIC file format, please let us know.
_2g_bin_magic¶
[MagIC Database] [command line version]
To convert the binary formatted 2G Enterprises measurement files, we can use the function convert._2g_bin() in the convert_2_magic module (imported as convert).
help(convert._2g_bin)
# set the input directory
input_dir='data_files/convert_2_magic/2g_bin_magic/mn1/'
mag_file='mn001-1a.dat'
convert._2g_bin(mag_file=mag_file,input_dir=input_dir,dir_path=input_dir)
These are measurement data for a single specimen, so we can take a quickie look at the data in an equal area projection.
help(ipmag.plot_di)
meas_df=pd.read_csv(input_dir+'measurements.txt',sep='\t',header=1)
ipmag.plot_net(1)
ipmag.plot_di(dec=meas_df['dir_dec'],inc=meas_df['dir_inc'])
agm_magic¶
[MagIC Database] [command line version]
This program converts Micromag hysteresis files into MagIC formatted files. Because this program creates files for uploading to the MagIC database, specimens should also have sample/site/location information, which can be provided on the command line. If this information is not available, for example if this is a synthetic specimen, specify syn= True for synthetic name.
Someone named Lima Tango has measured a synthetic specimen named myspec for hysteresis and saved the data in a file named agm_magic_example.agm in the agm_magic/agm_directory folder. The backfield IRM curve for the same specimen was saved in same directory as agm_magic_example.irm. Use the function convert.agm() to convert the data into a measurements.txt output file. For the backfield IRM file, set the keyword "bak" to True. These were measured using cgs units, so be sure to set the units key word argument properly. Combine the two output files together using the instructions for combine_magic. The agm files can be plotted using hysteresis_magic but the back-field plots are broken.
help(convert.agm)
convert.agm('agm_magic_example.agm',dir_path='data_files/convert_2_magic/agm_magic/',
specimen='myspec',fmt='old',meas_outfile='agm.magic')
convert.agm('agm_magic_example.irm',dir_path='data_files/convert_2_magic/agm_magic/',
specimen='myspec',fmt='old',meas_outfile='irm.magic')
infiles=['data_files/convert_2_magic/agm_magic/agm.magic','data_files/convert_2_magic/agm_magic/irm.magic']
ipmag.combine_magic(infiles,'data_files/convert_2_magic/agm_magic/measurements.txt')
We can look at these data using hysteresis_magic:
# read in the measurements data
meas_data=pd.read_csv('data_files/convert_2_magic/agm_magic/agm.magic',sep='\t',header=1)
# pick out the hysteresis data using the method code for hysteresis lab protocol
hyst_data=meas_data[meas_data.method_codes.str.contains('LP-HYS')]
# make the dictionary for figures that pmagplotlib likes
# make a list of specimens
specimens=hyst_data.specimen.unique()
cnt=1
for specimen in specimens:
HDD={'hyst':cnt,'deltaM':cnt+1,'DdeltaM':cnt+2}
spec_data=hyst_data[hyst_data.specimen==specimen]
# make a list of the field data
B=spec_data.meas_field_dc.tolist()
# make a list o the magnetizaiton data
M=spec_data.magn_moment.tolist()
# call the plotting function
hpars=pmagplotlib.plot_hdd(HDD,B,M,specimen)
hpars['specimen']=specimen
# print out the hysteresis parameters
print (specimen,': \n',hpars)
cnt+=3
bgc_magic¶
[MagIC Database] [command line version]
Here we convert the Berkeley Geochronology Center's AutoCore format to MagIC use convert.bgc().
help(convert.bgc)
dir_path='data_files/convert_2_magic/bgc_magic/'
convert.bgc('15HHA1-2A',dir_path=dir_path)
And let's take a look
meas_df=pd.read_csv(dir_path+'measurements.txt',sep='\t',header=1)
ipmag.plot_net(1)
ipmag.plot_di(dec=meas_df['dir_dec'],inc=meas_df['dir_inc'])
cit_magic¶
[MagIC Database] [command line version]
To convert the CalTech format to MagIC, use convert.cit().
Craig Jones’ PaleoMag software package (http://cires.colorado.edu/people/jones.craig/PMag3.html) imports various file formats, including the ’CIT’ format developed for the Caltech lab and now used in magnetometer control software that ships with 2G magnetometers that utilized a vertical sample changer system. The documentation for the CIT sample format is here: http://cires.colorado.edu/people/jones.craig/PMag_Formats.html#SAM_format. Demagnetization data for each specimen are in their own file in a directory with all the data for a site or study. These files are strictly formatted with fields determined by the character number in the line. There must be a file with the suffix ‘.sam’ in the same directory as the specimen data files which gives details about the specimens and a list of the specimen measurementfiles in the directory.
The first line in the .sam file is a comment (in this case the site name), the second is the latitude and longitude followed by a declination correction. In these data, the declination correction was applied to the specimen orientations so the value of the declination correction is set to be 0.
For detailed description of the .sam and sample file formats, check the PaleoMag Formats website linked to above.
help(convert.cit)
Use the function convert.cit() to covert the CIT data files from Swanson-Hysell lab at Berkeley for the PI47 site in the data_files/convert_2_magic/cit_magic/PI47 directory. The site (PI47) was part of a data set published in Fairchild et al., (2016) (available in the MagIC database: (https://earthref.org/MagIC/11292/). The location name was “Slate Islands”, the naming convention was #2, the specimen name is specified with 1 character, we don’t wish to average replicate measurements and they were collected by drilling and with a magnetic compass (”FS-FD",and "SO-MAG”).
dir_path='data_files/convert_2_magic/cit_magic/PI47/'
convert.cit(dir_path=dir_path,
magfile='PI47-.sam',locname="Slate Islands",specnum=1,samp_con='2',
methods=['FS-FD','SO-MAG'],noave=True)
We can make some Zijderveld diagrams (see zeq_magic).
ipmag.zeq_magic(input_dir_path=dir_path, save_plots=False)
Use the function convert.cit() to covert the CIT data files from the USGS lab at Menlo Park. The data file is in the data_files/convert_2_magic/cit_magic/USGS/bl9-1 directory, the file name is bl9-1.sam, and the analyst was Hagstrum. The location name was “Boring volcanic field”, and this site name was set by Hagstrum to BL9001 because the site name cannot be determined from the sample name with the current available options. The samples were collected by drilling and with a magnetic compass and sun compass (”FS-FD",and "SO-MAG”), the measurement are in Oersted instead of the standard milliTesla, and we don’t wish to average replicate measurements.
dir_path='data_files/convert_2_magic/cit_magic/USGS/bl9-1'
convert.cit(dir_path=dir_path,
magfile='bl9-1.sam',user='Hagstrum',locname="Boring volcanic field",
sitename='BL9001',methods=['FS-FD','SO-SM','LT-AF-Z'], oersted=True,
noave=True)
We can look at the Zijderveld, etc. Diagrams with zeq_magic.
ipmag.zeq_magic(input_dir_path=dir_path, save_plots=False)
Use the function convert.cit() to convert the CIT data files from Ben Wiess's lab at MIT. This data was part of a set published in ESPL. "A nonmagnetic differentiated early planetary body", doi:10.1016/j.epsl.2017.03.026 The data can be found in MagIC at https://earthref.org/MagIC/11943
The data file is in the data_files/convert_2_magic/cit_magic/MIT/7325B directory, the file name is 7325B.sam, and the analyst was Wiess. The location name was “NWA 7325” with the site name coming from the sample name with the "1" convention. The samples are described with the method codes DE-VM, LP-DIR-T, LT-AF-Z, LT-NO, LT-T-Z, and SO-CMD-NORTH (see https://www2.earthref.org/MagIC/method-codes for full descriptions). We also don’t wish to average replicate measurements.
convert.cit(dir_path='data_files/convert_2_magic/cit_magic/MIT/7325B',
magfile='7325B.sam',user='Wiess',locname="NWA 7325",samp_con='1',
methods=['DE-VM', 'LP-DIR-T', 'LT-AF-Z', 'LT-NO', 'LT-T-Z', 'SO-CMD-NORTH'],
noave=True)
And take a look see:
ipmag.zeq_magic(input_dir_path=dir_path, save_plots=False)
generic_magic¶
[MagIC Database] [command line version]
If you have a data file format that is not supported, you can relabel column headers to fit the generic format as in the generic_magic example data file.
To import the generic file format, use convert.generic().
help(convert.generic)
convert.generic(magfile='data_files/convert_2_magic/generic_magic/generic_magic_example.txt',
experiment='PI',dir_path='data_files/convert_2_magic/generic_magic')
# let's take a look
dir_path='data_files/convert_2_magic/generic_magic/'
ipmag.zeq_magic(input_dir_path=dir_path, save_plots=False)
huji_magic¶
[MagIC Database] [command line version]
To import the Hebrew University, Jerusalem, Israel file format to MagIC, use convert.huji().
help(convert.huji)
dir_path='data_files/convert_2_magic/huji_magic/'
convert.huji(dir_path=dir_path,
magfile='Massada_AF_HUJI_new_format.txt',codelist='T')
ipmag.zeq_magic(input_dir_path=dir_path, save_plots=False, n_plots=10)
huji_sample_magic¶
[MagIC Database] [command line version]
To convert a Hebrew University Jersalem, Israel sample format to MagIC, use convert.huji_sample().
help(convert.huji_sample)
convert.huji_sample('magdelkrum_datafile.txt',
dir_path='data_files/convert_2_magic/huji_magic/')
help(ipmag.combine_magic)
jr6_jr6_magic¶
[MagIC Database] [command line version]
The AGICO JR6 spinner magnetometer has two output formats, the .jr6 and the .txt. Here we illustrate the conversion of the .jr6 format. There are data from two different studies in the example folder. One (from Anita di Chiara) has the suffix '.JR6' and the other (from Roi Granot) are lower case (.jr6'). Each file has the data from a single specimen's experiment. So, we can convert Anita's data to a series of MagIC formatted measurement files, combine them with ipmag.combine_magic and look at them with Demag GUI (on the command line) or zeq_magic within the notebook.
help(convert.jr6_jr6)
Let's start with Anita's files
dir_path='data_files/convert_2_magic/jr6_magic/'
files=os.listdir(dir_path)
meas_files,spec_files,samp_files,site_files=[],[],[],[]
for file in files:
if '.JR6' in file:
print (file)
stem=file.split('.')[0]
meas_file=stem+'_measurements.txt' # make a unique measurements file
spec_file=stem+'_specimens.txt'
samp_file=stem+'_samples.txt'
site_file=stem+'_sites.txt'
convert.jr6_jr6(file,dir_path=dir_path,
meas_file=meas_file,spec_file=spec_file,samp_file=samp_file,
site_file=site_file,user='Anita')
meas_files.append(dir_path+meas_file) # save the file name to a list
spec_files.append(dir_path+spec_file)
samp_files.append(dir_path+samp_file)
site_files.append(dir_path+site_file)
# combine the files
ipmag.combine_magic(meas_files,dir_path+'measurements.txt')
ipmag.combine_magic(spec_files,dir_path+'specimens.txt',magic_table='specimens')
ipmag.combine_magic(samp_files,dir_path+'samples.txt',magic_table='samples')
ipmag.combine_magic(site_files,dir_path+'sites.txt',magic_table='sites')
ipmag.zeq_magic(input_dir_path=dir_path, save_plots=False)
Now we can do Roi's files
dir_path='data_files/convert_2_magic/jr6_magic/'
files=os.listdir(dir_path)
meas_files,spec_files,samp_files,site_files=[],[],[],[]
for file in files:
if file.endswith('.jr6'):
stem=file.split('.')[0]
meas_file=stem+'_measurements.txt' # make a unique measurements file
spec_file=stem+'_specimens.txt'
samp_file=stem+'_samples.txt'
site_file=stem+'_sites.txt'
convert.jr6_jr6(file,dir_path=dir_path,
meas_file=meas_file,spec_file=spec_file,samp_file=samp_file,
site_file=site_file,user='Roi')
meas_files.append(dir_path+meas_file) # save the file name to a list
spec_files.append(dir_path+spec_file)
samp_files.append(dir_path+samp_file)
site_files.append(dir_path+site_file)
# combine the files
ipmag.combine_magic(meas_files,dir_path+'measurements.txt')
ipmag.combine_magic(spec_files,dir_path+'specimens.txt',magic_table='specimens')
ipmag.combine_magic(samp_files,dir_path+'samples.txt',magic_table='samples')
ipmag.combine_magic(site_files,dir_path+'sites.txt',magic_table='sites')
ipmag.zeq_magic(input_dir_path=dir_path, save_plots=False)
jr6_txt_magic¶
[MagIC Database] [command line version]
We can repeat the exercise for the JR6 .txt format using convert.jr6_txt().
help(convert.jr6_txt)
There are only data from Roi Granot in this format. The measurement values should be identical to the convert.jr6_jr6() function on .jr6 files with the same stem. Additional columns will be found when converting the .JR6 format as that format contains more information than the .txt files.
dir_path='data_files/convert_2_magic/jr6_magic/'
files=['AF.txt','TRM.txt','AP12.txt']
meas_files,spec_files,samp_files,site_files=[],[],[],[]
for file in files:
print (file)
stem=file.split('.')[0]
meas_file=stem+'_measurements.txt' # make a unique measurements file
spec_file=stem+'_specimens.txt'
samp_file=stem+'_samples.txt'
site_file=stem+'_sites.txt'
convert.jr6_txt(file,dir_path=dir_path,
meas_file=meas_file,spec_file=spec_file,samp_file=samp_file,
site_file=site_file,user='Roi')
meas_files.append(dir_path+meas_file) # save the file name to a list
spec_files.append(dir_path+spec_file)
samp_files.append(dir_path+samp_file)
site_files.append(dir_path+site_file)
# combine the files
ipmag.combine_magic(meas_files,dir_path+'measurements.txt')
ipmag.combine_magic(spec_files,dir_path+'specimens.txt',magic_table='specimens')
ipmag.combine_magic(samp_files,dir_path+'samples.txt',magic_table='samples')
ipmag.combine_magic(site_files,dir_path+'sites.txt',magic_table='sites')
ipmag.zeq_magic(meas_file='AP12_measurements.txt',input_dir_path=dir_path, save_plots=False)
k15_magic¶
[MagIC Database] [command line version]
Someone took a set of samples from a dike margin in the Troodos Ophiolite and measured their anisotropy of magnetic susceptibility on an a Kappabridge KLY 2.0 instrument in the SIO laboratory. An example of the data file format is in k15_magic.
The first line of each set of four has the specimen name, azimuth, plunge, and bedding strike and dip the next three lines are sets of five measurements in the 15 positions recommended by Jelinek (1977):
Image('data_files/Figures/meas15.png')
The 15 measurements for each specimen, along with orientation information and the specimen name were saved in the file data_files/k15_magic/k15_example.dat.
To convert 15 measurement anisotropy of magnetic susceptibility file format to MagIC, use convert.k15().
help(convert.k15)
convert.k15('k15_example.dat',dir_path='data_files/convert_2_magic/k15_magic/',
location='Troodos Ophiolite')
ipmag.aniso_magic_nb(infile='specimens.txt',dir_path='data_files/convert_2_magic/k15_magic/')
kly4s_magic¶
[MagIC Database] [command line version]
The program AMSSpin available for downloading from http://earthref.org/ERDA/940/ generates data for the Kappabridge KLY4S spinning magnetic susceptibility instrument as described by Gee et al. (2008).
Output files are in the format of the file KLY4S_magic_example.dat (found in the measurement_import/kly4s_magic folder).
The columns in the example file are:
Specimen S_1 S_2 S_3 S_4 S_5 S_6 χb(μSI) date time user
To convert the Agico Kappabridge KLY4S files generated by the SIO labview program (written by Jeff Gee), use convert.kly4s(). This function will create the files needed by the MagIC database and the data can be plotted using aniso_magic. If you were to import the sample files from the LIMS data base for these samples, you could plot them versus depth, or as equal area projections using ani_depthplot and aniso_magic respectively.
help(convert.kly4s)
convert.kly4s('KLY4S_magic_example.dat',
dir_path='data_files/convert_2_magic/kly4s_magic/')
ipmag.aniso_magic_nb(infile='specimens.txt',dir_path='data_files/convert_2_magic/kly4s_magic/')
ldeo_magic¶
[MagIC Database] [command line version]
To convert Lamont-Doherty Earth Observatory data files to MagIC, use convert.ldeo().
NB: this doesn't seem to work properly at all.
help(convert.ldeo)
convert.ldeo('ldeo_magic_example.dat',codelist='AF',
dir_path='data_files/convert_2_magic/ldeo_magic/')
ipmag.zeq_magic(input_dir_path='data_files/convert_2_magic/ldeo_magic/', save_plots=False)
livdb_magic¶
[MagIC Database] [command line version]
To convert the Liverpool university database format to MagIC use convert.livdb().
Here we have several experiment types as examples as examples.
help(convert.livdb)
Here's an example for an IZZI style, thermal experiment:
convert.livdb('data_files/convert_2_magic/livdb_magic/TH_IZZI+/',
output_dir_path='data_files/convert_2_magic/livdb_magic/TH_IZZI+',
site_name_con=2,site_num_chars=3)
ipmag.thellier_magic(input_dir_path='data_files/convert_2_magic/livdb_magic/TH_IZZI+',
save_plots=False, interactive=False, n_specs=5)
here's one for microwave "C+" experiment
convert.livdb('data_files/convert_2_magic/livdb_magic/MW_C+/',
output_dir_path='data_files/convert_2_magic/livdb_magic/MW_C+',
site_name_con=2,site_num_chars=3)
An example for both microwave IZZI and C++:
convert.livdb('data_files/convert_2_magic/livdb_magic/MW_IZZI+andC++/',
output_dir_path='data_files/convert_2_magic/livdb_magic/MW_IZZI+andC++',
samp_name_con='2', samp_num_chars=1,site_name_con=2,site_num_chars=1)
An example for both microwave OT+:
convert.livdb('data_files/convert_2_magic/livdb_magic/MW_OT+/',
output_dir_path='data_files/convert_2_magic/livdb_magic/MW_OT+',
site_name_con=2,site_num_chars=3)
And an example for MW_P experiments.
convert.livdb('data_files/convert_2_magic/livdb_magic/MW_P/',
output_dir_path='data_files/convert_2_magic/livdb_magic/MW_P',
site_name_con=2,site_num_chars=3)
Now you can look at these data (except for MW_P) with thellier_gui or thellier_magic.
mst_magic¶
[MagIC Database] [command line version]
To convert a Curie Temperature experiment to MagIC, use convert.mst(). The data file format should be a space delimited file with temperature and magnetization couplets.
help(convert.mst)
convert.mst('curie_example.dat',samp_con="5",
dir_path='data_files/convert_2_magic/mst_magic/')
We can now use curie to plot the data.
ipmag.curie(path_to_file='data_files/convert_2_magic/mst_magic/',file_name='measurements.txt',magic=True)
pmd_magic¶
[MagIC Database] [command line version]
This format is the one used to import .PMD formatted magnetometer files (used for example in the PaleoMac software of Cogné, 2003) into the MagIC format. (See http://www.ipgp.fr/~cogne/pub/paleomac/PMhome.html for the PaleoMac home page. The version of these files that pmd_magic expects (UCSC version) contains demagnetization data for a single specimen and has a format as in the example file in ../measurement_import/pmd_magic/PMD/ss0207a.pmd
The first line is a comment line. The second line has the specimen name, the core azimuth (a=) and plunge (b=) which are assumed to be the lab arrow azimuth and plunge (Orientation scheme #4)D. The third line is a header explaining the columns in the file.
Use convert.pmd() to convert the file ss0101a.pmd in the directory ’PMD’ in the ’pmd_magic’ folder of the measurement_import directory in the example data_files directory. These were taken at a location named ’Summit Springs’ and have a naming convention of the type XXXX[YYY], where YYY is sample designation with Z characters from site XXX, or naming convention # 4-2. A single character distinguishes the specimen from the sample (specnum=1). All samples were oriented with a magnetic compass.
help(convert.pmd)
convert.pmd('ss0207a.pmd',dir_path='data_files/convert_2_magic/pmd_magic/PMD/',
samp_con='4-2',location='Summit Springs',specnum=1)
sio_magic¶
[MagIC Database] [command line version]
This program allows conversion of the SIO format magnetometer files to the MagIC common measurements format. The columns in the example data file are:
Specimen treatment intensity declination inclination optional_string
The treatment field is the temperature (in centigrade), the AF field (in mT), the impulse field strength, etc. For special experiments like IRM acquisition, the coil number of the popular ASC impulse magnetizer can be specified if the treatment steps are in volts. The position for anisotropy experiments or whether the treatment is “in-field” or in zero field also require special formatting. The units of the intensity field are in cgs and the directions are relative to the ‘lab arrow’ on the specimen. Here are some examples of commonly used specimens and conversions from field arrow to lab arrow.
Image('data_files/Figures/samples.png')
As an example, we use data from Sbarbori et al. (2009) done on a set of samples from the location “Socorro”, including AF, thermal, and thellier experimental data. These were saved in sio_af_example.dat, sio_thermal_example.dat, and sio_thellier_example.dat respectively. The lab field for the thellier experiment was 25 μT and was applied along the specimen’s Z axis (phi=0,theta=90).]
We can convert the example files into measurement formatted files with names like af_measurements.txt, etc. using the function convert.sio(). Then combine them together following the instructions for combine_magic.
help(convert.sio)
convert.sio('sio_af_example.dat',dir_path='data_files/convert_2_magic/sio_magic/',
specnum=1,location='Isla Soccoro',codelist='AF',samp_con='1',
meas_file='af_measurements.txt',spec_file='af_specimens.txt',
samp_file='af_samples.txt',site_file='af_sites.txt')
convert.sio('sio_thermal_example.dat',dir_path='data_files/convert_2_magic/sio_magic/',
specnum=1,location='Isla Soccoro',codelist='T',
meas_file='thermal_measurements.txt',spec_file='thermal_specimens.txt',
samp_file='thermal_samples.txt',site_file='thermal_sites.txt')
And combine them together...
# combine the measurements files
measfiles=['data_files/convert_2_magic/sio_magic/af_measurements.txt',
'data_files/convert_2_magic/sio_magic/thermal_measurements.txt']
ipmag.combine_magic(measfiles,'data_files/convert_2_magic/sio_magic/measurements.txt')
specfiles=['data_files/convert_2_magic/sio_magic/af_specimens.txt',
'data_files/convert_2_magic/sio_magic/thermal_specimens.txt']
ipmag.combine_magic(specfiles,'data_files/convert_2_magic/sio_magic/specimens.txt', magic_table='specimens')
sitefiles=['data_files/convert_2_magic/sio_magic/af_sites.txt',
'data_files/convert_2_magic/sio_magic/thermal_sites.txt']
ipmag.combine_magic(sitefiles,'data_files/convert_2_magic/sio_magic/sites.txt',magic_table='sites')
sufar4_magic¶
[MagIC Database] [command line version]
The AGICO program SUFAR creates ascii txt files as output. convert.sufar4() will convert these to the MagIC format.
help(convert.sufar4)
convert.sufar4('sufar4-asc_magic_example.txt',dir_path='data_files/convert_2_magic/sufar_asc_magic/',
sample_naming_con='5',locname='U1356A')
Now we can test it out with, for example, ipmag.aniso_magic_nb()
ipmag.aniso_magic_nb(infile='data_files/convert_2_magic/sufar_asc_magic/specimens.txt')
tdt_magic¶
[MagIC Database] [command line version]
Convertions of the Thellier Tool format of Leonhardt et al., 2004 can be done with convert.tdt(). THERE IS A PROBLEM WITH THE XXX.4 TREATMENT STEP CONVERSION.
help(convert.tdt)
convert.tdt('data_files/convert_2_magic/tdt_magic/')
utrecht_magic¶
help(convert.utrecht)
convert.utrecht('Utrecht_Example.af',dir_path='data_files/convert_2_magic/utrecht_magic',
specnum=0,samp_con='3')
orientation_magic¶
[Preparing for MagIC] [command line version]
orientation_magic is meant to import the field book data as entered into the format like in orientation_example.txt into the MagIC format samples, sites and location tables.
Click here for details about the orient.txt file format. The example file used here has field information for a few sites. The samples were oriented with a Pomeroy orientation device (the default) and it is desirable to calculate the magnetic declination from the IGRF at the time of sampling (also the default). Sample names follow the rule that the sample is designated by a letter at the end of the site name (convention #1 - which is also the default). We can do this from within a notebook by calling ipmag.orientation_magic().
help(ipmag.orientation_magic)
We need to know which orientation convention was used to take the samples (it was with a Pomeroy, so, the default). We want to use the igrf calculated magnetic declination at each site (so dec_correction_con=1, the default). These samples were collected in Antarctica with a local time of GMT+13, so we need to subtract 13 hours so hours_from_gmt should be 13. we are using data model 3.0 for this notebook, so data_model=3. Also, input_dir_path and output_dir_path are both ../orientation_magic.
ipmag.orientation_magic(input_dir_path='data_files/orientation_magic',orient_file='orient_example.txt',
hours_from_gmt=13,data_model=3,output_dir_path='data_files/orientation_magic')
azdip_magic¶
[MagIC Database] [command line version]
Many paleomagnetists save orientation information in files in this format: Sample Azimuth Plunge Strike Dip (AZDIP format), where the Azimuth and Plunge are the declination and inclination of the drill direction and the strike and dip are the attitude of the sampled unit (with dip to the right of strike). Of course there are many ways to think about sample orientation and the MagIC database convention is to store the direction of the X coordinate of the specimen measurements. To convert an AzDip formatted file (example in data_files/azdip_magic/azdip_magic_example.dat), we can use ipmag.azdip_magic().
help(ipmag.azdip_magic)
The method_codes are important. If you don't specify any sample orientation method, for example, the program will assume that they are unoriented. Pick the appropriate method codes for field sampling (FS-) and sample orientation (SO-) from the lists here: https://www2.earthref.org/MagIC/method-codes
ipmag.azdip_magic(orient_file='azdip_magic_example.dat',input_dir='data_files/azdip_magic/',
output_dir='data_files/azdip_magic/', method_codes='FS-FD:SO-MAG')
Other handy scripts¶
Image('data_files/Figures/chartmaker.png')
You can print it out and tape it to the oven in the lab to help keep track of this annoyingly complicated experiment. :)
To make this from within a notebook, call pmag.chart_maker().
help(pmag.chart_maker)
To perform 50 degree intervals from 100 to 500, followed by 10 degree intervals from 500 to 600 set up the Int and Top lists like this:
Int=[50,10]
Top=[500,600]
pmag.chart_maker(Int,Top)
You can now print out chart.txt. Happy IZZI-ing.
Clean up¶
import glob
import os
# remove some individual files
filenames = ['chart.txt',
'data_files/azdip_magic/samples.txt', 'data_files/download_magic/criteria.txt',
'data_files/orientation_magic/samples.txt', 'data_files/orientation_magic/sites.txt',
'data_files/download_magic/ages.txt', 'data_files/download_magic/contribution.txt',
'data_files/download_magic/measurements.txt', 'data_files/download_magic/samples.txt',
'data_files/download_magic/specimens.txt', 'data_files/download_magic/locations.txt']
for fname in filenames:
try:
os.remove(fname)
except FileNotFoundError:
pass
# remove all MagIC-generated files from a given directory
def remove_magic_files(directory):
magic_files = ['specimens.txt', 'samples.txt', 'sites.txt', 'locations.txt', 'measurements.txt',
'contribution.txt', 'ages.txt']
dir_files = os.listdir(directory)
for dtype in magic_files:
try:
os.remove(dtype)
except FileNotFoundError:
pass
for fname in dir_files:
if fname.endswith(dtype):
try:
os.remove(os.path.join(directory, fname))
except FileNotFoundError:
pass
for full_fname in glob.glob(os.path.join(directory, '*.magic')):
os.remove(full_fname)
# not convert_2_magic/jr6_magic
for directory in ['.', 'data_files/convert_2_magic/2g_bin_magic/mn1', 'data_files/convert_2_magic/pmd_magic/PMD/',
'data_files', 'data_files/k15_s', 'data_files/convert_2_magic/agm_magic',
'data_files/convert_2_magic/huji_magic', 'data_files/convert_2_magic/bgc_magic',
'data_files/convert_2_magic/kly4s_magic', 'data_files/convert_2_magic/mst_magic',
'data_files/convert_ages', 'data_files/convert_2_magic/cit_magic/MIT/7325B',
'data_files/convert_2_magic/cit_magic/USGS/bl9-1', 'data_files/convert_2_magic/tdt_magic',
'data_files/convert_2_magic/ldeo_magic', 'data_files/convert_2_magic/k15_magic',
'data_files/convert_2_magic/generic_magic']:
remove_magic_files(directory)
lst = ['*.png', './data_files/convert_2_magic/jr6_magic/SML*.txt', './data_files/download_magic/Snake*',
'./data_files/convert_2_magic/jr6_magic/AP12_*.txt',
'./data_files/convert_2_magic/jr6_magic/*_measurements.txt', './data_files/convert_2_magic/jr6_magic/*.magic',
'./data_files/3_0/McMurdo/*.tex', './data_files/3_0/McMurdo/*.xls', './data_files/3_0/Megiddo/*.tex',
'data_files/3_0/Megiddo/*.xls']
for directory in lst:
for fname in glob.glob(directory):
os.remove(fname)