User's Guide

NLDC321

Copy multiple NLDC 3480 formatted tape-sets to one 3490 archive tape.

Function:

Converts multiple NLDC 3480 formatted tape-sets to one 3490 archive tape. The archive tape contains all bands and associated files. As each scene is processed, meta data is extracted and used to update the NLDC database with the new archive location for the scene.

Parameters:

Subcommand -TTRANSFER:
Merge the ddr's from the TTRANSFER 3480 tapes. Then copy the single ddr and 7 band images to a 3490 archive tape.

TAPEID
3480 Tape Id's. The tape identification numbers for the input tapes which will be merged.

ROOTDIR(/SG1/CSB1/OPS/NLDC321/)
Root Directory. Location of the root directory for all subdirectories used for processing.

DUMPDIR(EDCSGS10:/SGS10/PROD/DSB/DATA/NLDC/)
Dump Directory. Location of the remote directory where header dumps will be copied to.

INGESTDIR(EDCSGS10:/SGS10/PROD/IMS/DB/NLDC/INDATA/COMPLETED_ARCDATA)
Remove Meta Directory. Location of the remote directory where new meta data files will be copied to.
Subcommand -TTRANSFR6:
Merge the ddr's from the TTRANSFER6 3480 tapes. Then copy the single ddr and 6 band images to a 3490 archive tape.

TAPEID
3480 Tape Id's. The tape identification numbers for the input tapes which will be merged.

ROOTDIR(/SG1/CSB1/OPS/NLDC321/)
Root Directory. Location of the root directory for all subdirectories used for processing.

DUMPDIR(EDCSGS10:/SGS10/PROD/DSB/DATA/NLDC/)
Dump Directory. Location of the remote directory where header dumps will be copied to.

INGESTDIR(EDCSGS10:/SGS10/PROD/IMS/DB/NLDC/INDATA/COMPLETED_ARCDATA)
Remove Meta Directory. Location of the remote directory where new meta data files will be copied to.
Subcommand -MTRANSFER:
Merge the ddr headers on the MTRANSFER 3480 tape, and copy the single ddr and 4 band images to a 3490 archive tape.

TAPEID
3480 Tape Id's. The tape identification numbers for the input tapes which will be merged.

ROOTDIR(/SG1/CSB1/OPS/NLDC321/)
Root Directory. Location of the root directory for all subdirectories used for processing.

DUMPDIR(EDCSGS10:/SGS10/PROD/DSB/DATA/NLDC/)
Dump Directory. Location of the remote directory where header dumps will be copied to.

INGESTDIR(EDCSGS10:/SGS10/PROD/IMS/DB/NLDC/INDATA/COMPLETED_ARCDATA)
Remove Meta Directory. Location of the remote directory where new meta data files will be copied to.
Subcommand -TFASTEOSAT:
Merge the headers of the TFASTEOSAT 3480 tapes. Then copy the new header and 7 band images to a 3490 archive tape.

TAPEID
3480 Tape Id's. The tape identification numbers for the input tapes which will be merged.

ROOTDIR(/SG1/CSB1/OPS/NLDC321/)
Root Directory. Location of the root directory for all subdirectories used for processing.

DUMPDIR(EDCSGS10:/SGS10/PROD/DSB/DATA/NLDC/)
Dump Directory. Location of the remote directory where header dumps will be copied to.

INGESTDIR(EDCSGS10:/SGS10/PROD/IMS/DB/NLDC/INDATA/COMPLETED_ARCDATA)
Remove Meta Directory. Location of the remote directory where new meta data files will be copied to.
Subcommand -TFASTINPE:
Merge the headers of the TFASTINPE 3480 tapes. Then copy the new header and 6 band images to a 3490 archive tape.

TAPEID
3480 Tape Id's. The tape identification numbers for the input tapes which will be merged.

ROOTDIR(/SG1/CSB1/OPS/NLDC321/)
Root Directory. Location of the root directory for all subdirectories used for processing.

DUMPDIR(EDCSGS10:/SGS10/PROD/DSB/DATA/NLDC/)
Dump Directory. Location of the remote directory where header dumps will be copied to.

INGESTDIR(EDCSGS10:/SGS10/PROD/IMS/DB/NLDC/INDATA/COMPLETED_ARCDATA)
Remove Meta Directory. Location of the remote directory where new meta data files will be copied to.
Subcommand -TFASTACRES:
Merge the headers of the TFASTACRES 3480 tapes. Then copy the new header and 7 band images to a 3490 archive tape.

TAPEID
3480 Tape Id's. The tape identification numbers for the input tapes which will be merged.

ROOTDIR(/SG1/CSB1/OPS/NLDC321/)
Root Directory. Location of the root directory for all subdirectories used for processing.

DUMPDIR(EDCSGS10:/SGS10/PROD/DSB/DATA/NLDC/)
Dump Directory. Location of the remote directory where header dumps will be copied to.

INGESTDIR(EDCSGS10:/SGS10/PROD/IMS/DB/NLDC/INDATA/COMPLETED_ARCDATA)
Remove Meta Directory. Location of the remote directory where new meta data files will be copied to.
Subcommand -TNDF:
Merge the headers of the TNDF 3480 tapes. Then copy the new header, 7 band images, work file, and history file to a 3490 archive tape.

TAPEID
3480 Tape Id's. The tape identification numbers for the input tapes which will be merged.

ROOTDIR(/SG1/CSB1/OPS/NLDC321/)
Root Directory. Location of the root directory for all subdirectories used for processing.

DUMPDIR(EDCSGS10:/SGS10/PROD/DSB/DATA/NLDC/)
Dump Directory. Location of the remote directory where header dumps will be copied to.

INGESTDIR(EDCSGS10:/SGS10/PROD/IMS/DB/NLDC/INDATA/COMPLETED_ARCDATA)
Remove Meta Directory. Location of the remote directory where new meta data files will be copied to.
Subcommand -TCEOSTRSC:
Merge the header and supplemental files of the TCEOSTRSC 3480 tapes. Then copy the new header, 7 leader-image-trailer file sets, supplemental file, and 360-byte null volume file to a 3490 archive tape.

TAPEID
3480 Tape Id's. The tape identification numbers for the input tapes which will be merged.

ROOTDIR(/SG1/CSB1/OPS/NLDC321/)
Root Directory. Location of the root directory for all subdirectories used for processing.

DUMPDIR(EDCSGS10:/SGS10/PROD/DSB/DATA/NLDC/)
Dump Directory. Location of the remote directory where header dumps will be copied to.

INGESTDIR(EDCSGS10:/SGS10/PROD/IMS/DB/NLDC/INDATA/COMPLETED_ARCDATA)
Remove Meta Directory. Location of the remote directory where new meta data files will be copied to.

Examples:

  1. LAS> nldc321-ttransfer tapeid=("132000","128022","132001")

    The tapes 132000, 128022, and 132001 will be processed for the Transer 7 Band format. The operator will be prompted for each input tape as well as for the output archive tape which will be a 3490. The ROOTDIR, DUMPDIR, and INGESTDIR parameters will be set to their defaults (see User Notes).

  2. LAS> nldc321-ttransfr6 tapeid=("112921","112923") rootdir="/sg1/csb/ops/work/nldc/nldc321" ingestdir="edcsgs10:/sgs10/prod/ims/nldc/ingest"

    The tapes 112921 and 112923 will be processed for the Transer 6 Band format. The operator will be prompted for each input tape as well as for the output archive tape which will be a 3490. The root directory will be /sg1/csb/ops/work/nldc/nldc321 instead of the default. The DUMPDIR parameter will be set to its default (see User Notes). The ingest directory will be set to edcsgs10:/sgs10/prod/ims/nldc/ingest instead of the default.

  3. LAS> nldc321-mtransfer tapeid="123025" rootdir="/sg1/csb/ops/work/nldc/nldc321" dumpdir="edcsgs10:/sgs10/dev/devsys/nldc"

    The tape 123025 will be processed for the Transer 4 Band format. The operator will be prompted for the input tape as well as for the output archive tape which will be a 3490. The root directory will be /sgs6/pds2/ops/work/nldc/nldc321 and the header dump directory will be edcsgs10:/sgs10/dev/devsys/nldc. The INGESTDIR parameter will be set to its default (see User Notes).

  4. LAS> nldc321-tfasteosat tapeid=("111001","111002","111003") rootdir="/sg1/csb/ops/work/nldc/nldc321" dumpdir="edcsgs10:/sgs10/dev/devsys/nldc" ingestdir="edcsgs10:/sgs10/prod/ims/nldc/ingest"

    The tapes 111001, 111002, and 111003 will be processed for the EOSAT Fast, 7 band format. The operator will be prompted for each input tape as well as for the output archive tape which will be a 3490. The root directory will be /sg1/csb/ops/work/nldc/nldc321 instead of the default. The dump directory will be set to edcsgs10:/sgs10/dev/devsys/nldc. The ingest directory will be set to edcsgs10:/sgs10/prod/ims/nldc/ingest instead of the default.

  5. LAS> nldc321-tfastinpe tapeid=("232434","232506") dumpdir="edcsgs10:/sgs10/prod/dsb/nldc/data"

    The tapes 232434 and 232506 will be processed for the INPE Fast, 6 band format. The operator will be prompted for each input tape as well as the output archive tape which will be a 3490. The ROOTDIR paramter will be set to its default (see User Notes). The dump directory will be set to edcsgs10:/sgs10/prod/dsb/nldc/data. The INGESTDIR parameter will be set to its default (see User Notes).

  6. LAS> nldc321-tfastacres tapeid=("100281","099123","100282") ingestdir="edcsgs10:/sgs10/prod/dsb/nldc/indata"

    The tapes 100281, 099123, and 100282 will be processed for the ACRES Fast, 7 band format. The operator will be prompted for each input tape as well as the output archive tape which will be a 3490. The ROOTDIR parameter will be set to its default (see User Notes). The DUMPDIR paramter will be set to its default (see User Notes). The ingest directory will be set to edcsgs10:/sgs10/prod/dsb/nldc/indata.

  7. LAS> nldc321-tndf tapeid=("108611","108501","108612","108502") dumpdir="edcsgs10:/sgs10/prod/dsb/nldc/header_dump" ingestdir="edcsgs10:/sgs10/prod/dsb/nldc/indata"

    The tapes 108611, 108501, 108612, and 108502 will be processed for the 7 band NDF-formatted scene. The operator will be prompted for each input tape as well as the output archive tape which will be a 3490. The ROOTDIR parameter will be set to its default (see User Notes). The dump directory will be set to edcsgs10:/sgs10/prod/dsb/nldc/header_dump. The ingest directory will be set to edcsgs10:/sgs10/prod/dsb/ nldc/indata.

  8. LAS> nldc321-tceostrsc tapeid=("131725","131719","131720") rootdir="/sg1/csb/ops/nldc321/"

    The tapes 131725, 131719, and 131720 will be processed for the 7 band CEOS-formatted scene. The operator will be prompted for each input tape as well as the output archive tape which will be a 3490. The root directory will be set to /sg1/csb/ops/nldc321/ instead of its default. The DUMPDIR and INGESTDIR parameters will be set to their defaults (see User Notes).

Description/Algorithm:

NLDC tapes which will be processed by the program are formatted in one of the following formats:

     TTRANSFER  - (TM, 7 band) TIC format processed for HTFP
     TTRANSFER6 - (TM, 6 band) TIC format processed for HTFP
     MTRANSFER  - (TM, 4 band) TIC format processed for HTFP
     TFASTEOSAT - (TM, 7 band) EOSAT Fast Format (Version B)
     TFASTINPE  - (TM, 6 band) INPE Fast Format (Version A)
     TFASTACRES - (TM, 7 band) EOSAT Fast Format (Version B)
                  Originating from Australia
     TCEOSTRSC  - (TM, 7 band) CEOS Format -- Thailand Data
     TNDF       - (TM, 7 band) NLAPS Data Format
The NLDC formats are currently stored on multiple 3480 archive tapes. The different bands, which make up the scene, are located on the different tapes. Normally, bands 1, 2, and 3 are located on the first tape, bands 4, 5, and 6 are located on the second tape, and band 7 is located on the third and final tape. This is not consistent, however, because of different format specific volume spanning requirements.

The basic algorithm for converting the multiple 3480 tapes to one 3490 tape is as follows. All files from the input tapes of a particular scene are copied to disk. The header files (depending on the format) from each tape are merged into one header file. This is accomplished by making the necessary changes and additions to the first header file so it will represent all of the bands for the scene (See User Notes). All other duplicate trailer and supplementary files, which describe the entire scene, are also merged. The header, all bands, and all other files are then copied from disk to a 3490 output tape. A meta data file is then created for updating the NLDC database. An operations working file is also changed to specify the scene's completion status.

Nonfatal Error Messages:

  1. [nldc321-alloc] Error allocating <xxx> <xxx> drive

    A non-fatal error occurred while attempting to allocate a tape drive for input or output. The tape should be remounted. After the tape is mounted, the program will request a tape mount. The program will allow three attempts to mount the tape correctly. If three attempts is exceeded, then a fatal error occurs.

  2. [nldc321-allout-d] Error allocating disk file for output

    A non-fatal error occurred while attempting to allocate disk file for output. Check to see if the disk is full. The program will try three times to allocate the disk. If the number of attempts exceeds three, then a fatal error will occur.

  3. [nldc321-badtape] Incorrect tape mounted

    The wrong input tape was mounted. The program will again request a tape mount for the tape. Please mount the correct tape. The program will try three times to mount the correct tape. If three attempts are exceeded, then a fatal error will occur.

  4. [nldc321-badtpid] Output tape id is not valid

    The output tape id, which the operator responded with after mounting the 3490 tape, is not valid. The tape id is alphanumeric and 6 digits long. The program will once again request the tape to be mounted. Remount the tape and enter the tape id. The program will try three times to get a correct tape id. If three attempts are exceeded, then a fatal error occurs.

  5. [nldc321-copy] Error copying from tape to disk

    A non-fatal error occurred while trying to copy from tape to disk. A system anomaly may have occurred. The input tape will be rewound and dismounted. Check the tape drive for any errors. Sometimes, if the drive is cross-mounted, tape I/O errors can occur frequently. The program will again request for a mount for the tapes. Remount the tapes. The program will try three times to copy from tape to disk. After three attempts, a fatal error occurs.

  6. [nldc321-copy] Error copying from disk to tape

    A non-fatal error occurred while trying to copy from disk to tape. A system anomaly may have occurred. The output tape will be rewound and dismounted. Check the 3490 tape for write-protection. Sometimes, if the drive is cross-mounted, tape I/O errors can occur frequently. The program will again request for a mount for the tape. Remount the tape. The program will try three times to copy from tape to disk. After three attempts, a fatal error occurs.

  7. [nldc321-info] Attempting to copy again

    After a non-fatal error has occurred during the output or input error-handling loops, this message will be displayed to the terminal. It indicates that another attempt to copy the files will be performed. After 3 attempts, this message will not be displayed, but a fatal error will occur.

  8. [nldc321-rcpagain] Error remote copying meta data...try again

    A non-fatal occurred while attempting to copy the meta data to a remote ingest directory. The program will perform three attempts to copy the new meta data file to the remote directory. This message will be displayed for each unsuccessful attempt. After three attempts, another non-fatal error message will be displayed.

Fatal Error Messages:

  1. [nldc321-allin-d] Error allocating disk file for input

    A fatal error occurred while attempting to allocate disk file for input. Check to see if the disk files exist. Re-run the program for the same tape ids.

  2. [nldc321-badformt] Format does not match nldc formats

    The format entered by the operator does not match the list of expected NLDC formats. Check the syntax of the format and re-run the program. Otherwise, contact the System Administrator.

  3. [nldc321-badscene] Scene id (<xxx>) does not match expected id

    The scene id which was extracted from the tape header does not match either the old_scene_id or new_scene_id from operations working meta data file for the format. Either the tapes do not match the database or the scene ids in the data base are incorrect. Contact Data Management with the scene id and the list of tape ids.

  4. [nldc321-fatal] Fatal error encountered

    A fatal error was encountered during processing. The error message that is displayed immediately preceding this message is the specific error that was encountered.

  5. [nldc321-getpar] Error retrieving input parameters

    An error occurred while the program attempted to retrieve input parameters from the variable block. Contact the System Administrator.

  6. [nldc321-metaout] New meta data file for <xxx> already exists

    An ingest meta data file for the specified output tape id already exists in the ingest directory. Contact Data Management.

  7. [nldc321-metadir] Metadata directory does not exist

    The directory where the operations working meta data files are located does not exist. The directory should be located within the root directory. Contact the System Administrator.

  8. [nldc321-newhdr] Error opening new header for <xxx> format

    A fatal error was encountered while trying to open a new header file for the specified format. Ensure the assemble directory exists and the permissions of the directory are correct. Re-run the program with the same tape ids.

  9. [nldc321-openhdr] Error opening header for <xxx> format

    A fatal error was encountered while trying to open the header file for the specified format. Check the permissions on the assemble directory. Re-run the program with the same tape ids.

  10. [nldc321-openlead] Error opening image leader for ceos format

    A fatal error was encountered while trying to open the image leader file. Check the permissions on the assemble directory. Re-run the program with the same tape ids.

  11. [nldc321-openmeta] Error opening new meta data file for <xxx>

    A fatal error was encountered while creating a new meta data file that will be ingested into the database. The meta data file should be created for the specified output tape id. Lookup the permissions on the ingest directory and check for the existence of the ingest directory. Contact the System Administrator.

  12. [nldc321-openwo] Error opening ndf work file

    An error occurred while trying to open the NDF workfile (*.wo). Check the permissions on the assemble directory. Re-run the program with the same tape ids.

  13. [nldc321-openwrk] Error opening operations work file for <xxx> format

    A fatal error was encountered while trying to open the operations working meta file for the specified format. Check the file in the meta directory for corruption or existence. Contact the System Administrator.

  14. [nldc321-q_out] Error writing <xxx> to variable block (#<xxx>)

    An error with the specified number occurred while attempting to write the specified variable to the variable parameter block. Contact the System Administrator.

  15. [nldc321-rcpmeta] Error remote copying <xxx> meta file to directory

    After three unsuccessful attempts to copy the specified meta data file to a remote directory have occurred, the fatal error message occurs. The program will leave the new meta data file in the data directory. The status for the scene in the operations working meta file will be updated to 'e' for error. Contact the System Administrator with the input tape ids and output 3490 tape id.

  16. [nldc321-readhdr] Error reading from header for <xxx> format

    A fatal error was encountered while attempting to read from the header file for the specified format. Check the file to see if it exists or is corrupted. Re-run the program with the same tape ids.

  17. [nldc321-readlead] Error reading from image leader for ceos format

    A fatal error was encountered while attempting to read from a image leader file. Check the permissions on the assemble directory. Re-run the program with the same tape ids.

  18. [nldc321-readmax] Maximum number of read attempts reached

    The maximum number of attempts to read the input tapes has been reached. The run will now clean up and abort. Check the drive and tape for errors. If the drive has errors, re-run the program with the same tape ids but on a different drive. If the tape has errors, contact Data Management.

  19. [nldc321-readwo] Error reading from ndf work file

    An error occurred while trying to read from the NDF workfile (*.wo). Check the permissions on the assemble directory. Re-run the program with the same tape ids.

  20. [nldc321-tpstatus] Tape ids with a status of <xxx> will not be run

    Tape ids which have the specified status in the operations working meta file will not be processed by the program. If the status is (i)nprocess, then the tape ids are currently being run. Do not run the tape ids. If you positively know, the tape ids are not currently being run, then contact the System Administrator. If the status is (y) completed, then the tape ids have been completed. Do not run the tape ids. If you suspect the tape ids have not been completed, then contact the System Administrator.

  21. [nldc321-tapecnt] Number of input tapes does not match number of expected

    The number of tapes entered by the operator does not match the number that is expected in the operations work file for the scene. Lookup the correct number of tape ids in the operations work file. Re-run the program and enter the correct tape ids. If the Number of Volumes field (4th field) in the working meta file does not match the number of tapes for the scene, then contact Data Management with the tape ids and a description of the error.

  22. [nldc321-tapelst] Tape series not found in operations working meta file

    The input tapes entered in the TAPEID parameter were not found in the operations working meta file. Lookup the tape ids in the operations working meta file. Re-run the program and enter the tape ids as parameters. If the same error occurs again, contact the System Administrator.

  23. [nldc321-updwork] Error updating operations work file for <xxx> format

    An error occurred while attempting to update the operations working meta file for the specified format. Lookup the permissions for the file and check to see if the file is corrupted or exists. Contact the System Administrator.

  24. [nldc321-writehdr] Error writing to new header for <xxx> format

    An error occurred while trying to write to the new header file for the specified format. Check assemble directory for write permissions and fix them. Re-run the program with the same tape ids.

  25. [nldc321-writemax] Maximum number of write attempts reached

    The maximum number of attempts to write to the output tape has been reached. The run will now clean up and abort. Check the drive and tape for errors. If the drive has errors, re-run the program with the same tape ids but on a different drive. If the tape has errors, contact Data Management.

  26. [nldc321-writmeta] Error writing to the new meta data file for <xxx>

    A fatal error was encountered while writing to the ingest meta data file for the specified output tape id. Check for the permissions of the ingest directory and determine if the ingest directory exists. Contact the System Administrator.

User Notes:

  1. The root, header dump, and ingest directories are determined by input parameters. The root directory defaults to /sg1/csb1/ops/nldc321. The dump directory defaults to edcsgs10:/sgs10/prod/dsb/data/nldc. The ingest directory defaults to edcsgs10:/sgs10/prod/ims/db/nldc/indata/completed_arcdata/new_data Operators can specify different values for the root and dump directories, but the defaults are preferred.

  2. All processing on disk takes place in a assemble directory. The assemble directory is found in the root directory. After NLDC321 runs, all data files from the run are removed from the assemble directory.

  3. The ingest directory contains all meta data files (name <output tape id>) created by NLDC321. These files are ingested by a CRON program which updates the NLDC_SCENE database with the new 3490 Archive Location. The ingest directory is located on a remote system.

  4. The meta directory contains a set of operations working meta files. These files contain a list of all scenes and tape ids that need to be converted to 3490. There is one file for each NLDC data format. Operations uses these working files to determine which tapes they will work on. The meta directory is located in the root directory.

  5. Operations will extract a list of tape ids to work on from the working meta files. The statuses of the tape ids are initially (n)o. When a group of tape ids for a scene is entered into the program, the program updates the statuses on the tape ids to (i)nprocess. After the tapes for a scene have been run through this program, the statuses of the tape ids in the files are updated to (y)es or completed. If an error occurred while creating the new output meta data files, then an (e)rror is placed in the status field for the tape ids.

  6. Operations should not run any tape ids that have statuses of (y) (i), or (e). Only tape ids with (n) will be allowed to run.

  7. The following lists the fields within the different format headers that are modified before the header file is written to tape:

      TTRANSFER, TTRANSFER6, and MTRANSFER:
         1) The Number of Bands field (4, 6 or 7)
         2) Append all band specific information of all 7
            bands to the end
         3) For TM 7 and 6 band scenes, set Minimum and Maximum
            to 0-Valid and 2.55-Valid
      TFASTEOSAT, TFASTACRES, TFASTINPE:
         1) The LANDSAT TM bands present on the archive tape
         2) The sequential tape number within the set
         3) The number of archive tapes within the set (1)
         4) For EOSAT and ACRES, the GAINS/BIAS fields
      TCEOSTRSC:
         1) The Tape Spanning fields in all pointer records (1)
      TNDF:
         1) The Tape Spanning Flag (1/1)
         2) The Number of Bands in Volume (7)
         3) Calculate the Lines Per Volume
         4) Append all band specific information to the end
    

  8. The output tape id and input tape id will be requested from the operator as a mount request is received. The tape id will be used by the program for verification and output.

  9. A log file can be generated each time NLDC321 is run. The operator does this by running NLDC321_LOG at the LAS prompt. The file name is <format><first tape id>.log_<timestamp>. The log file contains all messages that are displayed to the terminal. Once the program is done, the log file is copied to the summary directory, which is located in the root directory.