PRV system

Universal file pointers (.prv files)

A .prv file is a very small text file that serves as a universal file pointer to another (possibly very large) file. The original file may be located on the same machine, a local server, or on a remote server somewhere in the cloud.

For example, here is an example file named raw.mda.prv:

{
  "original_checksum": "9a42d2fb7d0e2cfd8c206c2ddc30640472ffab3d",
  "original_fcs": "head1000-da39a3ee5e6b4b0d3255bfef95601890afd80709",
  "original_path": "20160426_kanye_02_r1.nt16.mda",
  "original_size": 666616580,
  "prv_version": 0.1
}

The fields in this JSON document represent information about the file raw.mda at the time the .prv file was created using the prv-create command-line utility as described below.

• original_checksum: The sha-1 hash of the entire raw file.
• original_fcs: (optional) a code that can be used to quickly determine whether a given file is a candidate for a match.
• original_path: Only used for information in case the user needs a reminder on the name and location of the raw file at the time the .prv file was created.
• original_size: The size in bytes of the raw file.
• prv_version: Always 0.1 for now.

This small text file can be used as a convienent substitute for the original file on a file system or in a document. If the original file is located somewhere on the local computer, the prv-locate utility may be used to efficiently find the path to that file.

The prv command-line utility

The prv command-line utility can be used to create .prv files, locate the original files at a later date, or download the original files from a remote server.

Use prv-create to create a new .prv file that serves as a pointer to the original file:

> prv-create [source_file_name] [destination_file_name]
> prv-create raw.mda raw.mda.prv

Later, use prv-locate to find the original file, even if it had moved:

> prv-locate [prv_file_name]
> prv-locate raw.mda.prv

The full path to the file is written to stdout.

Use the mlconfig utility to set up the search paths for prv-locate on your system:

> mlconfig

The prv utility can also be used to locate and/or download files stored on a remote computer that is running a processing node server:

> prv-locate [prv_file_name] --server=[name of processing server]

If found, this utility writes to stdout a url path to the remote file. The file can be downloaded using:

> prv-download raw.mda.prv /path/to/destination/raw.mda --server=[name of processing server]

Using prv files in processing

It is easy to use .prv files in the Processing System. Simply use the path of the .prv file intead of the orignial in the appropriate fields:

> mp-run-process pyms.extract_clips --timeseries=raw.mda.prv --firings=firings.mda.prv --clips_out=output.mda --clip_size=123

The system will automatically search the local computer for the corresponding file prior to running the processing.