The ADAPS Programmer's Guide and the LAS Programmer's Guide provides the programmer with the necessary reference material and knowledge for developing transportable LAS software in the ADAPS environment.
The ADAPS guide is organized in sections that allow for additions which may come at a later date. Topical areas discussed include standards involved, general conventions, descriptions of file formats, compiling and linking an application, and support routines.
In order to maintain cohesiveness in a software system, development procedures must adhere to the standards presented in this and the LAS Programmer's Guide document. Along with the standards for software development and documentation, the reference material includes various descriptions of file formats for the image and its associated files. In order for processes to flow from one module to the next, the formats must be consistent and contain all the necessary fields for automated processing. In the development of the image and associated files, development efforts included the creation of support routines. These routines are individually documented and should be reviewed closely to avoid redundant code development and to improve development time. This document also contains examples showing the programmer how to compile and link an ADAPS application module in the UNIX environment.
There are three types of environment variables that ADAPS software uses: archive, product, and LAS. These environment variables are set up by the system manager as part of the LAS/ADAPS installation. To maintain portability all environment variables are set to directories instead of specific files. Environment Variables Assignments section of the System Manager's Guide for more information on how these variables are setup.
The archive environment variables are used to stream-line the ingest, referance aid, and archive process. These processes frequently use environment variables to locate images or tables, and create output files in specific directories. The following is a list of archive environment variables:
ADAPSACQ Location of the AVHRR scenes that have been acquired ADAPSING Location of the AVHRR scenes that have been converted to the EDC AVHRR archive format ADAPSFAX Location of the facsimile (quick-look) images ADAPSBRW Location of the browse images ADAPSIMG Location of the facsimile header and temporary location of the microfiche images to be built into a chip ADAPSMIC Location of the microfiche chip image ADAPSDBTMP Temporary location of metadata until the archive tape id can be filled in ADAPSARC Location of the AVHRR scenes that need to be archived to tape ADAPSDB Location of the metadata information that is to be transferred to a database (contains the archive tape id) ADAPSERR Location were images that had problems during processing are moved so that processing can continue ADAPSLOG Location of processes log/error information ADAPSORB Location of scenes to be stitched into half orbits ADAPSPROD Location of archive products (archive copy or level-1b) ADAPSTABLES Location of ephemeris and satellite specific files
Refer the ADAPS Directory Structure of the Configuration Notes.
The product environment variables are used by ADAPS applications to locate tables and sources of vector data. The following is a list of product environment variables:
ADAPSTABLES Location of calibration tables WDBII Location of the World Data Bank II data DCW Location of the Digital Chart of the World data WVS Location of the World Vector Shoreline data
The LAS environment variables are global variables defined in TAE and are used for scheduling and acquisition of AVHRR images. The following is a list of LAS environment variables:
A_ACQSYS Acquisition host system A_HOSSYS Current host system A_SCHEDL Location of the "master" schedule
Every AVHRR archive image consists of two files: A Committee on Earth Observing Satellites (CEOS) Inventory Exchange Format (IEF) header file and an AVHRR image file. The EDC AVHRR archive images are stacked on ANSI labeled 3480 tapes. See AVHRRSTACK User's Guide for details on the AVHRR archive tape format.
The AVHRR archive header file used by EDC is a CEOS_IEF data exchange file. Refer to the CEOS Inventory Exchange Format Document. The CEOS_IEF file has the following format:
The following is an example of an EDC AVHRR archive header file in the CEOS_IEF format; a description of each line format follows the example.
/*CEOS_IEF */ /*SFL */ /*930723103219 */ /*NEWACQ */ /*00001 */ /* SFL_ARCH_HEAD_START_V2 */ /* 00129 11 LAC NGC 07/23/1993 204 12:05:31.855 12:16:31.193 24880 24880 */ /* ASC ASC ASC 5 12345 03956 2048 00012 DAY SunZenith +000047.82679 */ /* NWest +0059.8108509 +0007.0322189 NNadir +0068.1191888 +0032.1800354 */ /* NEast +0069.8961397 +0070.6172010 CWest +0044.8338675 +0027.4776481 */ /* CNadir +0049.7703229 +0046.1252497 CEast +0051.2038640 +0067.3432594 */ /* SWest +0027.3978317 +0038.0648744 SNadir +0030.7428887 +0052.9408622 */ /* SEast +0032.2884958 +0068.5860595 EqCrs +0060.5875648 SatVw 1 */ /* Dtime +000000.00000 */ /* Roll +0.2499999926 +0.0000000000 +0.0000000000 +0.0000000000 +0.0000000000*/ /* Pitch +0.0000000000 +0.0000000000 +0.0000000000 +0.0000000000 +0.0000000000*/ /* Yaw +0.0000000000 +0.0000000000 +0.0000000000 +0.0000000000 +0.0000000000*/ /* EPHEM 195312487202669232828990880000000117756944661586660170127536892930723*/ /* GAPS 00006: 00939-00001 01438-00003 02441-00004 03445-00002 03199-00001 */ /* 05101-00001 */ /* SFL_ARCH_HEAD_END */ N11AVHLAC 930723 120531 121631 NGCSFL24880A ??????+060.59 012345- 10D C 1 +32.29+068.59+27.40+038.06+59.81+007.03+69.90+070.62N? ?????? ???????? 09 /*END_IEF */
The first five lines of the archive header file are required by the CEOS_IEF format. Lines 6 through 21 are specific to EDC AVHRR archive and lines 21 through 23 are defined by the CEOS_IEF format and are dependent upon the image. A short description of each line follows:
Line Pos Type Description ---- --- ------------- ------------------------------------------------------ 1 3 7 characters Defines the metadata format (Always "CEOS_IEF") 2 3 3 characters A unique code for the agency/data center holding the data. The valid inventory codes are defined in the CEOS Guidelines Document. SFL is the EDC CEOS inventory exchange code. 3 3 12 characters Date and time the header information was created 4 3 6 characters Type of records in the inventory exchange. New acquisitions, updates, or deletions. 5 3 5 characters Total number of inventory exchange lines 6 4 22 characters Denotes the start of the EDC AVHRR archive header information and the header version number 7 4 5 digit int Satellite clock correction. This value is the offset in milliseconds that the satellite time stamp codes need to be adjusted due to satellite clock drift. 10 2 digit int Satellite number (09, 10, 11, 12, ...) 13 4 characters Data type (GAC, LAC, HRPT) 18 3 characters CEOS inventory exchange code for the receiving station that acquired the image. 22 10 characters Date at the start of acquisition 33 3 digit int Julian day at the start of acquisition 37 12 characters Time at the start of acquisition. This is the time stamp code of the first valid line in the image plus the satellite clock correction. 50 12 characters Time at the end of acquisition. This is the time stamp code of the last valid line in the image plus the satellite clock correction. 63 5 digit int Orbit number at the start of acquisition 69 5 digit int Orbit number at the end of acquisition 8 4 4 characters Direction of the pass at the start (ASC, DESC) 9 4 characters Direction of the pass at the center (ASC, DESC) 14 4 characters Direction of the pass at the end (ASC, DESC) 19 1 digit int Number of bands in the image 21 5 characters Indicates which bands are present 27 5 digit int Number of lines in the image 33 4 digit int Number of samples in the image 38 5 digit int Total number of dropped lines in the image 44 5 characters Indicates daytime or nighttime data (DAY, NIGHT) based on the solar zenith angle at the center. 60 13 digit real Solar zenith angle at the center of the image in degrees. 9 11 13 digit real Latitude of the Northwest point in degrees 24 13 digit real Longitude of the Northwest point in degrees 48 13 digit real Latitude of the North Nadir point in degrees 61 13 digit real Longitude of the North Nadir point in degrees 10 11 13 digit real Latitude of the Northeast point in degrees 24 13 digit real Longitude of the Northeast point in degrees 48 13 digit real Latitude of the Center West point in degrees 61 13 digit real Longitude of the Center West point in degrees 11 11 13 digit real Latitude of the Center Nadir point in degrees 24 13 digit real Longitude of the Center Nadir point in degrees 48 13 digit real Latitude of the Center East point in degrees 61 13 digit real Longitude of the Center East point in degrees 12 11 13 digit real Latitude of the Southwest point in degrees 24 13 digit real Longitude of the Southwest point in degrees 48 13 digit real Latitude of the South Nadir point in degrees 61 13 digit real Longitude of the South Nadir point in degrees 13 11 13 digit real Latitude of the Southeast point in degrees 24 13 digit real Longitude of the Southeast point in degrees 48 13 digit real Equatorial crossing in degrees 63 1 digit int Satellite view flag (1) Sat view (-1) Non-Sat view (North-up). 14 11 12 digit real Delta time correction in milliseconds 15 11 12 digit real Five roll correction coefficients in degrees 16 11 12 digit real Five pitch correction coefficients in degrees 17 11 12 digit real Five yaw correction coefficients in degrees 18 10 69 characters Ephemeris information. (Satellite id, orbit number, mean anomaly, mean motion, orbital decay, eccentricity, argument of perigee, longitude of ascending node, inclination angle, and date of epoch). 19 9 5 digit int Total number of gaps in the data. If there are no gaps in the data, this line will not be present. 16 5 digit ints If there are data gaps in the image then each data gap will have an entry of the form xxxxx-yyyyy. Where xxxxx denotes the first line number of the gap and yyyyy denotes the number of dropped lines (zeroed lines) in the gap. Each data gap entry will be separated by a space. When there are more than five data gaps in the image, additional data gap lines will be added. 20 4 5 digit ints An additional data gap entry. Each additional data line will contain from one to six data gap entries. 21 4 17 characters Denotes the end of an SFL archive header information 22 1 80 characters The first record of metadata as part of the inventory exchange line. Refer to the CEOS Inventory Exchange Format Document. 23 1 80 characters The second record of metadata as part of the inventory exchange line. Refer to the CEOS Inventory Exchange Format Document. Any additional metadata (inventory exchange lines) would be added after these lines. 3 7 characters Defines the end of the metadata (Always "END_IEF")
The archive image file is similar to the "raw" data stream (HRPT minor frame) acquired from the satellite. Refer to the NOAA Technical Memorandum NESS 107 for a description of the HRPT minor frame.
An archive record contains the first 103 words from the HRPT minor frame and the earth video data packed into 13,796 bytes. The first 103 (10 bit) words from the HRPT minor frame are packed into 140 bytes and the 10,240 (10 bit) words of earth video data are packed into 13,656 bytes. Every three words (30 bits) are packed into four bytes (32 bits), right justified. All unused bits are set to zero. The earth video data consists of five bands for each of the 2048 samples in a scan line. The 2048 samples are ordered in Band Interleaved by Pixel (BIP) format.
The TIP data from the HRPT minor frame is only extracted from newly acquired images on request and is not saved as part of the EDC AVHRR archive.
The format of LAS files on disk is similar for all currently supported computer systems. The image file is a raw binary file in band sequential format. This file can also be referred to as a flat, DAL, or transfer file. AVHRRIN will convert an EDC AVHRR archive image into a LAS AVHRR image. AVHRRIN will also flip ascending images and any associated data information to be north up.
In the course of processing AVHRR images, several types of associated data files are created. All files in a directory with the same filename but different extensions are considered to be associated. This allows the image and its associated files to be easily treated as a "data set". Standard extensions are used in naming types of files that are frequently used. Some of the ADAPS file types are:
addr AVHRR Data Descriptor Record file drpl Dropped line information file mnr HRPT Minor Frame file cal Calibration coefficients
Refer to the Associated Data section of the LAS Programmers Guide for further LAS associated files and extensions.
Nearly all associated binary data files have the same format provided by the label service utility routines. Refer to the Associated File Formats section of the LAS Programmer's Guide for more information on the LAS label services utility routines and associated file formats.
The ADDR file contains model information, nine coordinate locations, and ephemeris data for the image. This data is used to geo-reference the image data in satellite perspective. Each record contains 80 bytes of ASCII data in the label services file format. The ADDR file structure contains the following label service records for each AVHRR image in LAS:
First record long clock_err; /* Satellite clock error (milliseconds) */ long satnum; /* Satellite number */ long dtype; /* Data type (1)LAC, (2)GAC, (3)HRPT */ char station[4]; /* Station id */ long month; /* Month of acquisition */ long day; /* Day of acquisition */ long year; /* Year of acquisition */ long jday; /* Julian day of acquisition */ long shour; /* Starting hour of acquisition */ long smin; /* Starting minute of acquisition */ long ssec; /* Starting second of acquisition */ long sfrac; /* Starting fractional seconds of acq */ long ehour; /* Ending hour of acquisition */ long emin; /* Ending minute of acquisition */ long esec; /* Ending second of acquisition */ long efrac; /* Ending fractional seconds of acq */ long sorbit; /* Orbit number at start of acquisition */ long eorbit; /* Orbit number at end of acquisition */ Second record long spass; /* Direction of the pass at start */ /* (-1) Descending (1) Ascending */ long cpass; /* Direction of the pass at center */ /* (-1) Descending (1) Ascending */ long epass; /* Direction fo the pass at end */ /* (-1) Descending (1) Ascending */ long nchans; /* Number of channels in image */ long chans[5]; /* Channels numbers in the image */ long nlines; /* Number of lines in image */ long nsamples; /* Number of samples in image */ long totmiss; /* Total missed lines in image */ long day_night; /* Day (1) Night (-1) flag at center */ long sat_view; /* Satellite view (1) yes (0) data flipped */ double sunzen; /* Sun zenith angle at center */ Third record double nw[2]; /* Lat/Long at North West corner */ double nn[2]; /* Lat/Long at North Nadir point */ Fourth record double ne[2]; /* Lat/Long at North East corner */ double cw[2]; /* Lat/Long at Center West point */ Fifth record double cn[2]; /* Lat/Long at Center Nadir point */ double ce[2]; /* Lat/Long at Center East point */ Sixth record double sw[2]; /* Lat/Long at South West corner */ double sn[2]; /* Lat/Long at South Nadir point */ Seventh record double se[2]; /* Lat/Long at South East corner */ double eqcrs; /* Equatorial crossing */ long satvw;i /* Eighth record double delta_time; /* Delta time correction (milliseconds) */ double delta_alt; /* Delta error in altitude (meters) */ Ninth record double roll_coefs[5]; /* Roll correction coefficients (degrees) */ Tenth record double pitch_coefs[5]; /* Pitch correction coefficients (degrees) */ Eleventh record double yaw_coefs[5]; /* Yaw correction coefficients (degrees) */ Twelfth record char ephemeris[70]; /* Ephemeris for date of image */
The dropped line (zero filled lines) information is stored in and ASCII file. This information is used to correct/fill the the dropped lines within an image. This file contains the starting line number and the number of dropped lines. The following dropped line file indicates that there are four dropped lines in the image. Two starting at line one and two starting at line 511.
1 2 511 2
The minor frame file consists of the first 103 words of the HRPT minor frame data from each image line. The minor frame data is used to generate calibration coefficients, in AVHRRCAL, for the thermal bands. This file is in a label services file format.
All AVHRR images received are ingested and reformatted into a standard EDC AVHRR archive format. Deciphering the various formats is accomplished by setting up an entry in the "station.ids" table in the $ADAPSTABLES directory for each format that is to be ingested. The following is an example of the ingest table:
ID T SH RL Byt/L Mptr Msiz Tptr Tsiz Iptr Isize F S FF Station name --- - -- -- ----- ---- ---- ---- ---- ---- ----- - - -- ------------------------ BAA 1 0 1 14848 0 103 206 520 750 20480 3 1 0 Buenos_Aires_Argentina BJG 1 1 -2 7400 308 140 -1 0 448 13656 3 0 0 Beijing_China CAI 1 0 1 21980 0 206 206 1040 1500 20480 3 0 0 Cairo_Egypt DAR 1 0 1 14788 0 103 206 520 750 20480 3 0 0 Darwin_Australia DSA 1 0 1 22528 0 206 206 1040 1500 20480 3 1 0 Dhahran_Saudi_Arabia EAF 1 0 1 20992 0 206 -1 0 512 20480 1 1 0 EROS_Archive_Format GAC 2 0 1 5790 0 103 206 520 750 409 3 0 0 Global_Area_Coverage
INGEST uses this file to determine how to read and extract information to create the standard EDC AVHRR archive format. Each format is identified with a Committe on Earth Observing Satellite (CEOS) ID. Refer to the Station.ids section of the Configuration Notes for more information on the contents of the ingest file. Also refer to Appendix A: New Ingest Formats of the ADAPS Programmer's Guide to determine how to define a new ingest format.
A pre-processing step may be necessary to extract the image data and ancillary files from the various formats and media before ingesting. TPACQUIRE uses UNIX commands to copy the defined formats from tape to disk. If a new format is received the TPACQUIRE function must be modified to add the new CEOS ID and the steps necessary to extract the AVHRR image and any useful ancillary files from tape. Any information needed from the ancillary files, such as the year of acquisition, would require modifications to the INGEST function to extract this information. Refer to the Appendix A: New Ingest Formats of the ADAPS Programmer's Guide for information on how to modify the TPACQUIRE and INGEST functions.
Data gaps are lines that are missing due to a transmission, frame synchronization, disk, or other problems during acquisition. Data gaps are detected by calculating the difference between the current and previous time-stamps and dividing the difference by a sixth of a second (AVHRR data is scanned at six lines per second). When the gap is greater than one, it is necessary to fill in the missing line(s) with valid time stamps and zeroed lines to maintain the proper along-track perspective. These data gaps are called dropped lines.
Bit drops are caused by problems similar to those which cause data gaps but only affect bits within the data. Bit drops can corrupt the data and give unexpected date or time values. When an unexpected date or time value is encountered due to a bit drop, the date is corrected with the previous line's date, and the time is corrected to be one sixth of a second from the previous line.
When all gaps are filled and the time stamp codes have been checked and corrected the ephemeris information for the image is retrieved and the following information about the image is calculated:
This information is written to the archive header associated with the AVHRR archive image. An AVHRR archive image has an extension of ".arch" and an archive header file has the same root file name as the AVHRR archive image with an extension of ".ahdr".
If the TIP data is available as part of the input AVHRR data, it can be optionally stripped from each line and written to a file associated with the AVHRR archive image. It has the same root file name as the AVHRR archive image with an extension of ".tip". This data is not archived as part of the EDC AVHRR archive.
A browse image is a band from the AVHRR image that is scaled from ten bits to eight bits, enhanced and subsampled by four in the line direction and five in the sample direction. An average browse image is 1350 lines by 409 samples. Scene information similar to the quick-look product is written to the header record and the browse images are transferred to the Global Land Information System (GLIS). The following describes the browse format image and the header information:
BROWSE IMAGE FORMAT: _________________________ | 408 bytes Header info | |-----------------------| | 408 bytes Image data | |-----------------------| | . | | . | | . | |-----------------------| | 408 bytes Image data | -------------------------
The browse header contains the following ASCII information:
Browse version number 1 Character Scene ID 30 Characters Minimum pixel value 3 Characters (0 -> 255) Maximum pixel value 3 Characters (0 -> 255) Number of lines 4 Characters (400 max for LAC) Number of samples 4 Characters (408) Bands 9 Characters NW Latitude 6 Characters (total minutes) NW Longitude 6 Characters (total minutes) NE Longitude 6 Characters (total minutes) NE Latitude 6 Characters (total minutes) SW Longitude 6 Characters (total minutes) SW Latitude 6 Characters (total minutes) SE Longitude 6 Characters (total minutes) SE Latitude 6 Characters (total minutes) CN Longitude 6 Characters (total minutes) CN Latitude 6 Characters (total minutes)
As the AVHRR data is received and archived, a mechanism is needed to track and locate images that have been processed. Metadata for each image is generated and transferred to a database which can be used to select and locate images. The metadata contains the type of data (HRPT, LAC, or GAC), satellite number (11 for NOAA-11), date, start time, end time, day/night flag, direction of the pass, equatorial crossing, starting orbit number, pass duration, number of lines, number of bands, bands present, archive tape location, receiving station code, microfiche number, and nine geo-location points. A portion of the metadata is also transferred to GLIS and matched with the browse image to assist customers in making selections. The following is an example of a metadata transfer record:
Sceneid |End time|Enter date|Update date|Day/Night|Pass direction|Eq Cross|Mgr|Orbit|Pass dur|Master|# lines|Ordered AH11040192122401|123328 |09/30/92 |09/30/92 |D |A |006202 |32 |18130| 9 |M | 3400 | 0 Sceneid |Tape ID|File #|Format|Archive status|Channels|Enter date|Update date|???|Station|Time stamp AH11040192122401|022784 |01 |02 |G |12345 |09/30/92 |09/30/92 | 1|HBK |102321 Sceneid |Locator|Enter date|Update date|Latitude|Longitude| AH11040192122401|NW |09/30/92 |09/30/92 | -350 | 2339 | AH11040192122401|NN |09/30/92 |09/30/92 | -146 | 3658 | AH11040192122401|NE |09/30/92 |09/30/92 | 026 | 5034 | AH11040192122401|CW |09/30/92 |09/30/92 |-2002 | 2641 | AH11040192122401|CN |09/30/92 |09/30/92 |-1818 | 4052 | AH11040192122401|CE |09/30/92 |09/30/92 |-1530 | 5500 | AH11040192122401|SW |09/30/92 |09/30/92 |-3616 | 2846 | AH11040192122401|SN |09/30/92 |09/30/92 |-3446 | 4522 | AH11040192122401|SE |09/30/92 |09/30/92 |-3103 | 6117 |
The HRPT minor frame from the AVHRR satellite contains 11090 ten bit words of telemetry, TIPs, and earth video data. Each ten bit word is stored in a 16 bit word (two bytes) for a total of 22180 bytes per line. The maximum pass (15 minutes) that can be acquired from a receiving station has 5400 lines for a total file size of 119,772,000 bytes (approximately 120 megabytes).
An EDC AVHRR archive image contains no TIP data and is packed by placing 3 ten bit words (30 bits) into four bytes (32 bits). The first 103 (10 bit) words from the HRPT minor frame are packed into 140 bytes and the 10,240 (10 bit) words of earth video data are packed into 13,565 bytes for a total of 13,796 bytes per line. A 15 minute pass in this format contains a total of 74,498,400 bytes (approximately 75 megabytes).
A browse image is a one band (band two for day and band four for night) image that has been scaled from ten bit data to eight bit data. The image is also sub-sampled every fourth line by every fifth sample. A 5400 line by 2048 sample, one band image would be sub-sampled to 1350 lines by 409 samples for a grand total of 552,150 bytes (approximately 1/2 megabyte).
On UNIX the make utility is used to compile an application program. The commands are stored in the file named either makefile or Makefile. Whenever any of the independent modules is changed, executing the UNIX command:
% make
ensures the executable file of the application program is up to date.
Note that an application program depends on several external files and object libraries. For example, a typical C application program depends on:
Therefore, if any of the previously listed dependencies are modified, the program is rebuilt.
The ADAPS includes and libraries are located in the $ADAPSINC and $ADAPSLIB directories, respectively. Refer to the Environment Variables Assignments section of the System Manager's Guide for more information on how these variables are setup.
As examples, the Makefile for the C application AVHRRIN may be used with the EROS Data Center's release procedure or with a global makefile. To install the executable files in the directory indicated by the environment variable $DSTDIR, the following UNIX command should be run:
% make las
The user installing the executable must set up the environment variable $DSTDIR and have write permission into the directory.