MagIC  |  Help Contents  |  Full Text    

MagIC Help Library
First  |  Previous  |  Next  |  Last    

3.1  Introduction to the Standardized MagIC Data Format

Establishing an integrated paleomagnetic and rock magnetic database is difficult as we need to move large quantities of data into the MagIC database from both legacy and new studies. To address this challenge we have chosen an approach that makes the archiving of these data Synchronous with the Scientific Publication process and is entirely based on User Contributions. Around the time of publication each scientist has all relevant data for a publication at his fingertips and knows best how to deal with it. In fact, he probably went through a sustained effort to collect all measurement data (and most of the relevant metadata) to perform his or her scientific research. The MagIC Console Software is developed to aid the scientist to collate all information at this opportune time and to do so by making use of a Standardized MagIC Data Format.

3.1.1  What is a SmartBook?

We wish to tap into this process by supplying scientists with a tool that they can use to collect all data that are relevant to one particular publication (henceforth referred to as a project). Key to this approach is a standard Data and Metadata Template in the form of an Microsoft Excel© SmartBook in which they can enter their data and in which they can further process their data to eventually upload into the online MagIC database.

Because we use this standardized SmartBook protocols can be established around which scientists can build (or adapt) their (current) laboratory protocols. For example, they can streamline the collection of measurement data by enabling the export of standard MagIC Format Text Files that can be readily imported into the MagIC SmartBooks. Similar export functions can be established for various data reduction software and geomagnetic modeling codes. This approach significantly increases the flow of magnetic data into the MagIC database and circumvents labor-intensive data entry for individual scientists.

The MagIC SmartBook has been defined in a way so that it can store all measurements and their derived properties for studies of paleomagnetic directions and intensities, and rock magnetic experiments, such as hysteresis, remanence, susceptibility and anisotropy. The basic design of this SmartBook focuses on the work-flow in typical paleomagnetic and rock magnetic studies. This ensures that individual data points can be traced between actual measurements and their related specimens, samples, sites, locations, and so forth.

To make working with the MagIC SmartBooks more straightforward, we have developed the MagIC Console Software that, in addition to many other functions, contains an Import function for standard text files. This software helps you to Enter Data, it helps you to check the Correctness and Coherence of the data entries, and it helps you Prepare for Uploading your SmartBook in the MagIC database. Since the MagIC SmartBooks have been developed as Microsoft Excel© Files, most users will feel comfortable with the setup, which makes use of standard toolbars and dialogboxes.

3.1.2  Structure of the MagIC SmartBooks

The MagIC SmartBooks are standard Microsoft Excel© Workbooks that contain predefined tables stored in separate worksheets. There is an obvious hierarchy in these tables, where the EarthRef (ER) tables are the most general and are applied to other databases hosted under this umbrella website as well.

Note   For a detailed overview please open the MagIC.vXX.metadata.definition.xls file in Microsoft Excel© and use the hyperlinks in the Frontpage (or the standard Microsoft Excel© tabs at the bottom of the page) to show the description for each MagIC table, including example input data. This information can also be found by following the http://earthref.org/MAGIC/metadata.htm web link or by selecting the Data and Metadata Definition option from the MagIC Help menu.

The four MagIC tables (measurements, methods, instruments and calibrations) are less general, but contain data and metadata that are common for typical paleomagnetic and rock magnetic projects. The PMAG and RMAG tables contain the most specialized, highly derived data.

3.1.3  MagIC Table Layout

Each table is divided into several pre-titled columns. The first Five rows of each column characterize the data to be entered below the headings. The first row gives the Label of the column and describes the data to be entered (explanation in plain English). The second row displays the Oracle Variable Name associated to these data in the relational database and used by the MagIC Console Software. The third row indicates the Data Type and maximum length of text strings. The fourth row shows the expected Unit of the data. The fifth row indicate the Status of the data to show whether the data is required, recommended or optional. All five heading rows are fixed in the MagIC Data and Metadata definition, which is strictly maintained and versioned.

Upon opening a SmartBook in the MagIC Console Software they will be used to check the validity of the tables and columns residing in the Microsoft Excel© File.

3.1.4  Standard MagIC Text File Format

The MagIC Console Software has the ability to export and import Flat Text Files. To be able to easily transfer data between outside software packages, data collection computers and the MagIC SmartBooks, we have developed the Standard MagIC Text File Format. Experience shows that this is the most efficient way when dealing with large data sets. It allows you to quickly import these data into the MagIC SmartBooks without making mistakes. In the example below, we show how this text file is based on the layout of the table displayed above.

Header Structure

The Standard MagIC Text File Format starts out with two header lines (red fonts) indicating the delimiter used, table name and column names. Below these header lines, the data appears in the same order as indicated by the column names (black fonts).

Multiple Data Blocks

Possible delimiters include tab and pipe symbols. Note that single or double quotes around text strings are not required. You can also store multiple data blocks (for one project) in one text file, where each data block is stored according to the above rules but is separated by the standard >>>>>>>>>> divider. Since each block has its own header lines, in principle, you can store the results from different experiments or tables in one and the same Standard MagIC Text File. This will result in a text file that may look as follows.

Note   Because in the above example both experiments KOPA-2004-01 and KOPA-2004-05 are properly divided and because each block has its own header lines, each block can store different parameters and in a random order (compare the placement of the treatment_temp field, for example). This will keep the file size minimal because only the data relevant for that one particular experiment needs to be stored. It is also flexible, because it allows you to store data measured using different equipment or measurement protocols in one-and-the-same data file.

Always store the text files in the same directory as the MagIC SmartBook file itself.

3.1.5  Software Installation

The MagIC Console Software does not work with a fixed directory structure. However, we recommend that you generate a main MagIC directory on your hard disk and that you store the populated SmartBook for each of your MagIC Projects in sub directories.

Required Files

In the table below we have listed Four files that are required for a proper functioning of the MagIC Console Software. Note that each file has been named according to its current version, that is v20 or v24 (instead of vXX) for example.

Note   Every time you download the software from the MagIC Website you will receive a zipped archive that contains all four files (see table) and some subdirectories with example SmartBook files. When unzipping this software archive, be sure to retain the relative paths and folder names.