User's Guide

INGEST

Ingest raw AVHRR data into an AVHRR archive format image

Function:

Ingests raw AVHRR data into an AVHRR archive format image. "Raw" AVHRR data is data that is in the format of the receiving station that acquired it. Raw data must be ingested into the standard EDC AVHRR archive format before it can be processed by the AVHRR Data Acquisition and Processing System (ADAPS).

Parameters:

INFILE
Input file. The name of the input raw AVHRR data file. The input file may contain HRPT, LAC, or GAC AVHRR data. The default extension is ";ACQ".

YEAR
Acquire year. The last two digits of the year the data was acquired. The year will be extracted from the data if it is available. See User Note 3.

CEOSID
CEOS ID. The alphanumeric code that identifies the station that acquired the raw AVHRR data. Refer to the STATION;IDS table in ADAPSTABLES for a list of valid CEOS IDs.

ACQTYPE(--)
Acquire type. The type of AVHRR data to be ingested. If NULL, INFILE is assumed to be a scene ID and is used to set the data type (see User Note 6).

  = HRPT:  High Resolution
           Picture Transmission
  = LAC:   Local Area Coverage
  = GAC:   Global Area Coverage

OUTFILE(--)
Output file. The name of the output AVHRR archive image. By default, the output image is created in the current directory and named "<scene ID>;ARCH" (see User Note 6). When only a directory is specified, the output file is named "<scene ID>;ARCH" and is placed in the specified directory. The default extension for OUTFILE is ";ARCH".

TIPOPT(NO)
TIP option. Option to save the TIROS Information Processor (TIP) data to a file associated with the output image. The file will have the same root name as OUTFILE with a ";TIP" extension. For NOAA 15, AMSU data will also be saved (see User Note 7).

  = YES:  Save the TIP/AMSU data
  = NO:   Don't save the TIP/AMSU data

Examples:

  1. LAS> ingest infile=africa year=92 ceosid=nmy outfile=[ingest]

    The raw AVHRR data in AFRICA;ACQ will be ingested into AVHRR archive format. The input file contains HRPT data acquired in 1992 from the Niamey, Niger (NMY) receiving station. The archive image will be placed in the [INGEST] directory and will be named based on the scene ID.

  2. LAS> ingest infile=[acquire.hrpt]edc year=92 ceosid=sfl acqtype=lac outfile=sfl92 tipopt=yes

    LAC data acquired in 1992 at the Sioux Falls, USA (SFL) receiving station will be ingested from EDC;ACQ in the [ACQUIRE.HRPT] directory and converted to AVHRR archive format. The archive image will be SFL92;ARCH in the current directory. If the acquired data contains TIP/AMSU information, it will be saved in the SFL92;TIP file associated with the archive image.

Description/Algorithm:

The format of the input file (raw acquired AVHRR data) is defined by the station that received it. The specified CEOS ID is used to extract information about the station's format from the STATION;IDS table in the ADAPSTABLES directory. The information stored in this table includes the size and location of the HRPT minor frame, TIP/AMSU data, and image data, along with the record size and pixel format (BIP, BIL, or BSQ). This information is used to open and read the input file properly.

The satellite number, start date, and start time are retrieved from the first valid line of the input file. The lines from the input file are sequentially read, reformatted, and written to a temporary output archive image in the output directory. The date and time from each line is compared with the date and time from the previous line to determine if there are any data gaps or bit drop errors (see User Notes 4 and 5 for a description of data gaps and bit drops). Data gaps are filled with zeros, and bit drop errors in the date or time are corrected. If TIPOPT=YES, the TIP/AMSU data is stripped from each line and written to a file associated with the output image (see User Note 7).

A scene ID is generated based on the satellite number, start date, and start time of the input file. The temporary output archive image is renamed to the output file name, if specified, or <scene ID>;ARCH. The satellite number, start time, end time, and missing line information are used to retrieve the ephemeris data, calculate satellite-related information (pass direction, sun zenith, equatorial crossing, etc.), and generate image related information (number of lines, channels, latitude and longitude coordinates, etc.). This information is written to the archive image's associated header (AHDR) file.

Nonfatal Error Messages:

  1. [ingest-ignore] Ignoring incomplete line at EOF

    An incomplete line of AVHRR data was encountered at the end of the file. The incomplete line is ignored and processing completes successfully.

  2. [ingest-invyr] Invalid year specified. Year reset from <XX> to <YY>

    The year in the associated AVHRR archive header (AHDR) was reset. This situation may occur if the date specified by the time-stamp in the raw data along with the value of YEAR is later than the current date. The year specified by the user is reset to a reasonable value. This situation may also occur if valid ephemeris data doesn't exist within 30 days of the date specified by the time-stamp and YEAR. If, in this case, the year was erroneously reset, update the ephemeris data and rerun INGEST to produce a correct AHDR.

  3. [ingest-nlines] Error determining number of lines in acquired image

    INGEST was unable to determine the number of lines that are expected from the acquired image. Processing continues.

  4. [ingest-warn] EOF encountered before a bad time stamp could be resolved

    Data at the end of the input file had errors in time-stamp codes. All lines successfully ingested will be written to the output archive image.

Fatal Error Messages:

  1. [ingest-expected] Error placing the expected number of lines in the parameter block

    An error occurred returning the expected number of lines in the image to the TAE parameter block. Contact the system administrator.

  2. [ingest-fatal] Fatal error encountered

    A fatal error was encountered during processing. Processing is terminated. The error message that is displayed immediately preceding this message is the specific error that was encountered.

  3. [ingest-gaps] The number of dropped line gaps exceeds the limit of <XXX> at line <YYY>

    The number of data gaps allowed was exceeded. The data should be reviewed for usability.

  4. [ingest-invid] Invalid satellite id: <XXX>

    An invalid satellite ID was found in the first valid record of raw data. Only data from NOAA satellites 6 through 15 may be processed. Strip off the first record of the raw acquire file and rerun INGEST. If the error persists, contact the system administrator.

  5. [ingest-nmissed] Error placing the total number of missed lines in the parameter block

    An error occurred returning the total number of missed lines to the TAE parameter block. Contact the system administrator.

  6. [ingest-noval] No valid data found in acquired image

    The end of the input file was encountered before the first valid record was found. No output archive image is produced.

  7. [ingest-read] Error reading from the acquisition file

    An error occurred reading the input file. Verify that you have read access on the file and that the correct CEOS ID was specified.

  8. [ingest-sceneid] Error placing the scene ID in the parameter block

    An error occurred returning the scene ID to the TAE parameter block. Contact the system administrator.

  9. [ingest-tip] Error writing TIP data to an output file

    An error was encountered while trying to write the TIP information to a file. Disk space, directory protections, and the data usability should be checked.

  10. [ingest-tstamp] Error placing the time stamp in the parameter block

    An error occurred returning the current time stamp to the TAE parameter block. Contact the system administrator.

  11. [ingest-vblock] Error sending the parameter block

    An error occurred returning the parameter block to TAE. Contact the system administrator.

  12. [ingest-nleap] Specified year, XX, is not a leap year

    The julian day of the first data line is 366 and the year is not a leap year. Determine the correct year of the scene and try again.

User Notes:

  1. See the NOAA POD Guide for a description of the HRPT minor frame format and TIP data for NOAA 6 through NOAA14. See the NOAA KLM Users Guide for information on NOAA 15.

  2. Refer to the STATION;IDS table in ADAPSTABLES for a list of valid CEOS IDs.

  3. The acquire year is a required parameter because this information cannot be derived from the data from some receiving stations. If the data does contain the acquire year, it is extracted, and the user-specified value of YEAR is ignored.

  4. Data gaps (missing lines) are lines that are missing due to a transmission, frame synchronization, disk, or other problems during acquisition. Data gaps are detected by calculating the difference between the current and previous time-stamps and dividing the difference by a sixth of a second. (AVHRR data is scanned at six lines per second.) When the gap is greater than one, it is necessary to fill in the missing lines with zero lines to maintain the proper along-track perspective.

  5. Bit drops are caused by problems similar to those that cause data gaps but only affect bits within the data. Bit drops can corrupt the data and give unexpected date or time values. When an unexpected date or time value is encountered due to a bit drop, the date is set to the previous line's date, and the time is set to be one sixth of a second from the previous line.

  6. The scene ID is in the form Atssmmddyyhhmmss, where A stands for AVHRR data, t represents the data type (H for HRPT, L for LAC, or G for GAC), ss is the satellite number, mmddyy is the acquisition date, and hhmmss is the start time of acquisition. An example scene ID is AL11080191130734. This indicates AVHRR LAC data, satellite 11, acquisition date 08/01/91, and scene start time 13:07:34.
  7. Satellites in the KLM series, which includes NOAA 15, differ slightly in their minor frame format. One difference is in the TIP portion of the frame. Each minor frame in previous satellites contained the same information, however the NOAA 15 HRPT major frame consists of three minor frames. Minor frame one contains TIP data, frame two has spare data, and frame three has AMSU data. When the TIPOPT flag is set to YES when ingesting a NOAA 15 scene the spare and AMSU data are saved as well as the TIP data. See the NOAA KLM User's Guide for more detailed information.