Download and ingest NCEP water vapor data into a LAS image
Download the four daily GRIB, NCEP forecast files; extract the precipitable water (PWAT) record from each GRIB file downloaded; convert the extracted PWAT data records to water vapor images and store them on silo.
- SILOBASEDIR("/HSM/LTA1/AVHRR/NCEP_WATERVAPOR")
- Base SILO directory (See User Notes 2 and 3)
- FORECASTLIST("04","09","18","20")
- List of forecast availability times. These are the local hours that forecasts are available for download. (See User Note 4)
The NCEPIN_DAEMON is started in batch queue ageneral and its standard output is redirected to ncepin_12345.log. Output images will be placed on the silo in default directory: [avhrr.ncep_watervapor] in the current year subdirectory.
NCEPIN_DAEMON runs continuously and checks the NOAA National Center Environmental Prediction (NCEP) ftp site for the four daily forecast files. These forecasts are stored in GRIdded Binary (GRIB) format for times of 0000, 0600, 1200 and 1800 UTC. The daemon determines which forecast file is available for download and adds the file name to an internal transfer list. An attempt is made to ftp each file on the transfer list during four predetermined hour long periods. The PWAT record is extracted from each file that is succesfully transferred.For each extracted PWAT record: the name is removed from the transfer list and the transfer file is converted to a LAS image; the values are converted from kg/m^2 to centimeters; the left and right halves of the image are swapped so the center of the image is 0 degrees longitude; and the DDR is updated to indicate geographic space. The file is then stored on silo.
If a file has not been transferred and is older than two days, it is removed from the transfer list. The forecast files only reside on the ftp server for two days and are therefore not retrievable via ftp. Processing messages are logged in the ADAPSLOG directory in NCEPIN.LOG. If errors occur during conversion, the transfer file and images are moved to ADAPSERR.
The NCEPIN_DAEMON was terminated by the user with a stop file.
An error occurred transferring to image file <XXXXX> to the silo. The file must be moved manually. Processing continues. Notify system adminstrator if problem persists.
An error occurred deleting the tranfer file and the image file. These files must be manually deleted. Processing continues. Notify system adminstrator if problem persists.
The image and transfer file <XXXXX> were moved to ADAPSERR. The error message preceding this one is the specific error. Processing continues.
A fatal error occurred. The message preceding this one was the specific error. The process halts.
ADAPSTOP-FUNC FUNCNAM=NCEPIN_DAEMON
If it is signaled to stop, NCEPIN_DAEMON will terminate when it starts a new processing loop.
basedir -\--- 2000 \--- 2001 \--- 2002 \--- ...
FORECASTLIST(1) the 00Z forecast FORECASTLIST(2) the 06Z forecast FORECASTLIST(3) the 12Z forecast FORECASTLIST(4) the 18Z forecast
a. Execute the nceptrans.sch script with appropriate arguments nceptrans.csh HH YYMMDD where HH is the forecast hour, and YYMMDD is the 2-digit year, month and day of the forecast. Include leading zeros for all values (eg 00 000504 is May 4, 2000 00Z forecast). The file returned will be gdas1.THHZ.PGrbF00 with the HH replaced with the forecast hour. b. Extract the precipitable water (PWAT) record from the file obtained in a. with the following commands: set recnum=`$ADAPSCONTRIB/wgrib <INFILE> -v | grep PWAT | \ awk -F: '{print $1}'` $ADAPSCONTRIB/wgrib <INFILE> -d $recnum -ieee -o <TRANFILE> where <INFILE> is the file downloaded in step a. and <TRANFILE> is the desired output file name. The output file should be named fnl_gdas1_pwatHH_YYMMDD.trn where HH YYMMDD is as explained in a. Note: Steps c, d and e can be accomplished by placing the transfer image created in step b in the ADAPSACQ/ncepin directory. The next time the ncepin_daemon runs, during a local forecast availability hour, the transfer file will be processed. c. Convert the <TRANDFILE> created in b. by running the NCEPIN procedure in TAE. The output must be named as follows: fnl_gdas1_pwatHH_YYMMDD where HH YYMMDD is as explained in a. d. Store the fnl_gdas1_pwatHH_YYMMDD.img file on silo in the appropriate NCEP water vapor subdirectory. (See User Note 2) e. Once the image is safely stored on silo the downloaded file, transfer and image files may be removed.