gridfile

Catalogue and load gridded data sets

Description

The gridfile class creates and manages gridfile objects. These objects catalogue metadata and data source files for a gridded dataset. The catalogue is stored in a file with a “.grid” extension.

A key function of gridfile is to provide an interface for loading data from data source files with diverse formatting. This interface allows the user to efficiently:

Load subsets of the catalogued data
Order the dimensions of loaded data
Apply linear or log transforms to loaded data, and
Perform simple arithmetic on large gridded datasets

Another function of gridfile is to associate all gridded data with dimensional metadata. This allows data to be accessed using human-readable metadata, rather than array indices.

The following is an outline for using the gridfile class.

Use the “new” command to initialize a new .grid file
Use “add” to add data source file to the gridfile’s catalogue
Use “fillValue”, “validRange”, and “transform” to apply data transformations when data is loaded from the source files
Use the output of the “metadata” command to locate interesting subsets of the catalogued data.
Use “load” to load data subsets in customizable dimension orders.
Use “plus”, “minus”, “times”, or “divide” to implement arithmetic across multiple data catalogues.

The class also includes many methods for accessing and editing metadata, which are summarized in the methods list.

Key Methods

The following methods are the most essential methods for users.

new - Create a new, empty .grid file
gridfile - Return a gridfile object for a .grid file.
add - Catalogue a data source in a .grid file
metadata - Return the metadata for a gridfile or data sources
load - Load data from the sources catalogued in a .grid file

All User Methods

The complete list of gridfile methods for users.

Create

new - Create a new, empty .grid file

gridfile - Return a gridfile object for a .grid file.

Metadata

metadata - Return the metadata for a gridfile
edit - Rewrite gridfile metadata
expand - Increase the length of a dimension in a .grid file
addDimension - Add a new dimension to a .grid file

Metadata Attributes

attributes - Return the metadata attributes for a gridfile
addAttributes - Add attributes to gridfile metadata
removeAttributes - Remove attributes from the metadata of a gridded dataset
editAttributes - Change existing metadata attributes

Data Sources

add - Catalogue a data source in a .grid file
remove - Remove data sources from a .grid file’s catalogue
relocate - Update paths to data sources catalogued in a gridfile
absolutePaths - Save data source file names as absolute or relative paths

Data Adjustments

fillValue - Specify a fill value for data catalogued in a .grid file
validRange - Specify a valid range for data catalogued in a .grid file
transform - Transform data loaded from a .grid file

Load

load - Load data from the sources catalogued in a gridfile

Arithmetic

plus - Sum the data in two gridfiles
minus - Subtract the data in a gridfile from the current gridfile
times - Multiply the data in two gridfiles
divide - Divide the data in the current gridfile by a second gridfile

Return Information

info - Return information about a gridfile object and data source files
dimensions - Return the dimensions of a gridfile and their sizes
sources - Return an ordered list of data source files catalogued in a gridfile
name - Return the name of the .grid file (excluding path and extension)

Print Information

disp - Displays a gridfile object in the console

source - Displays a gridfile data source in the console

Utility Methods

Under-the-hood methods that help the class run. These are not intended for users, and do not implement error checking on their inputs.

File Interactions

update - Update a gridfile object to match the contents of its .grid file
save - Save a gridfile object to a .grid file
assertValid - Throw error if gridfile array has deleted elements

Console Display

dispDimensions - Print dimension sizes and metadata to the console
dispSources - List gridfile data sources in the console
dispAdjustments - Print fill value, valid range, and data transformation to console

Load

getLoadIndices - Organize the dimension indices required to implement a load operation
sourcesForLoad - Return the indices of data sources needed to load requested data
buildSources - Build dataSources for a gridfile load
loadedPrecision - Return the precision of data loaded from mixed data types
loadInternal - Load requested data from pre-built dataSource objects

Arithmetic

arithmetic - Arithmetic operations across two gridfiles

Unit Tests

tests - Unit tests for the gridfile class