User's Guide

NCEPIN_DAEMON

Download and ingest NCEP water vapor data into a LAS image

Function:

Download the four daily GRIB, NCEP forecast files; extract the precipitable water (PWAT) record from each GRIB file downloaded; convert the extracted PWAT data records to water vapor images and store them on silo.

Parameters:

SILOBASEDIR("/HSM/LTA1/AVHRR/NCEP_WATERVAPOR")
Base SILO directory (See User Notes 2 and 3)

FORECASTLIST("04","09","18","20")
List of forecast availability times. These are the local hours that forecasts are available for download. (See User Note 4)

1. LAS> NCEPIN_DAEMON|runtype=(batch,ageneral) stdout=ncepin_12345|

    The NCEPIN_DAEMON is started in batch queue ageneral and its standard output is redirected to ncepin_12345.log. Output images will be placed on the silo in default directory: [avhrr.ncep_watervapor] in the current year subdirectory.

Description/Algorithm:

NCEPIN_DAEMON runs continuously and checks the NOAA National Center Environmental Prediction (NCEP) ftp site for the four daily forecast files. These forecasts are stored in GRIdded Binary (GRIB) format for times of 0000, 0600, 1200 and 1800 UTC. The daemon determines which forecast file is available for download and adds the file name to an internal transfer list. An attempt is made to ftp each file on the transfer list during four predetermined hour long periods. The PWAT record is extracted from each file that is succesfully transferred.

For each extracted PWAT record: the name is removed from the transfer list and the transfer file is converted to a LAS image; the values are converted from kg/m^2 to centimeters; the left and right halves of the image are swapped so the center of the image is 0 degrees longitude; and the DDR is updated to indicate geographic space. The file is then stored on silo.

If a file has not been transferred and is older than two days, it is removed from the transfer list. The forecast files only reside on the ftp server for two days and are therefore not retrievable via ftp. Processing messages are logged in the ADAPSLOG directory in NCEPIN.LOG. If errors occur during conversion, the transfer file and images are moved to ADAPSERR.

Nonfatal Error Messages:

  1. [ncepin-ustop] NCEPIN_DAEMON terminated by user

    The NCEPIN_DAEMON was terminated by the user with a stop file.

  2. [ncepin-silo] Error occurred writing <XXXXX> to silo

    An error occurred transferring to image file <XXXXX> to the silo. The file must be moved manually. Processing continues. Notify system adminstrator if problem persists.

  3. [ncepin-delete] Error occurred deleting transfer and image

    An error occurred deleting the tranfer file and the image file. These files must be manually deleted. Processing continues. Notify system adminstrator if problem persists.

  4. [ncepin-error] <XXXXX> moved to ADAPSERR

    The image and transfer file <XXXXX> were moved to ADAPSERR. The error message preceding this one is the specific error. Processing continues.

Fatal Error Messages:

  1. [ncepin-fatal] Fatal error occurred

    A fatal error occurred. The message preceding this one was the specific error. The process halts.

User Notes:

  1. NCEPIN_DAEMON enters an infinite loop to check, download and convert NCEP data. To stop the procedure, run

        ADAPSTOP-FUNC FUNCNAM=NCEPIN_DAEMON
    

    If it is signaled to stop, NCEPIN_DAEMON will terminate when it starts a new processing loop.

  2. All NCEP water vapor images are stored in a directory structure with a base directory containing subdirectories by year. Within the year subdirectories the daily observation image files are stored.

    
        basedir -\--- 2000
                 \--- 2001
    	     \--- 2002
    	     \--- ...
    

  3. The host directories on the SILO begin with /dmf/edc, the uftp routines now require the prefix to work.

  4. This procedure is only active during the four forecast availability hours defined by the input parameter FORECASTLIST. Each entry in the list indicates when the respective forcast will be available for download. The entries are defined as follows:

    
        FORECASTLIST(1) the 00Z forecast
        FORECASTLIST(2) the 06Z forecast
        FORECASTLIST(3) the 12Z forecast
        FORECASTLIST(4) the 18Z forecast
    
    

  5. If it is necessary to manually download and process a NCEP file accomplish the following steps:

    
      a. Execute the nceptrans.sch script with appropriate arguments
         
    	 nceptrans.csh HH YYMMDD
    
         where HH is the forecast hour, and YYMMDD is the 2-digit year,
         month and day of the forecast.  Include leading zeros for all
         values (eg 00 000504 is May 4, 2000 00Z forecast).  The file
         returned will be gdas1.THHZ.PGrbF00 with the HH replaced with
         the forecast hour.
    
      b. Extract the precipitable water (PWAT) record from the file
         obtained in a. with the following commands:
    
    	 set recnum=`$ADAPSCONTRIB/wgrib <INFILE> -v | grep PWAT | \
    	     awk -F: '{print $1}'`
    	 $ADAPSCONTRIB/wgrib <INFILE> -d $recnum -ieee -o <TRANFILE>
    
         where <INFILE> is the file downloaded in step a. and <TRANFILE>
         is the desired output file name.  The output file should be
         named 
             
    	 fnl_gdas1_pwatHH_YYMMDD.trn
         
         where HH YYMMDD is as explained in a.
      
      Note: Steps c, d and e can be accomplished by placing the transfer
            image created in step b in the ADAPSACQ/ncepin directory.
    	The next time the ncepin_daemon runs, during a local forecast
    	availability hour, the transfer file will be processed.
    
      c. Convert the <TRANDFILE> created in b. by running the NCEPIN
         procedure in TAE.  The output must be named as follows:
    
    	 fnl_gdas1_pwatHH_YYMMDD
    
         where HH YYMMDD is as explained in a.
    
      d. Store the fnl_gdas1_pwatHH_YYMMDD.img file on silo in the
         appropriate NCEP water vapor subdirectory. (See User Note 2)
    
      e. Once the image is safely stored on silo the downloaded file,
         transfer and image files may be removed.