User's Guide

SILO

Provide TAE interface to the silo.

Function:

Provide a TAE interface to the silo. This function allows the user to read, write, and delete files on the silo, create symbolic links on the silo, and list the files that exist on the silo.

Parameters:

Subcommand -DELETE:
Delete file(s) from the silo.

INFILE
Input file(s). Specifies the file(s) to be deleted from the silo. If a specified file is a symbolic link, the file being pointed to will not be affected. UNIX syntax is expected. An absolute path or a relative path from the user's home directory on the silo may be specified. Wildcard characters are permitted. (See User Note 8)
Subcommand -DIR:
List files on the silo.

INFILE
Input file(s). Specifies the file(s) to be searched for on the silo. UNIX syntax is expected. An absolute path or a relative path from the user's home directory on the silo may be specified. Wildcard characters are permitted. See User Note 1 and 8.

OUTFILES
Output files. The names of the files that matched INFILE are returned in the specified TAE string variable. This variable must have a count of 0:N, where N should not be less than MAXFILES. The directory path is excluded in the file names that are returned. (See User Note 8)

LOCATION
File location. The location of the files that matched INFILE are returned in the specified TAE string variable. This variable must have a count of 0:N, where N should not be less than MAXFILES. The strings returned are either DISK or TAPE. See User Note 2 and 8.

NFILES
Number of files. The number of values returned in OUTFILES and LOCATION is returned in this TAE integer variable. The specified variable must be able to accept a value between zero and MAXFILES.

MAXFILES(1)
Maximum files. The maximum number of file names and locations to be returned in OUTFILES and LOCATION.

LOCKFLG(YES)
Lock flag. Flag indicating if locked files are to be ignored and not returned in OUTFILES.

  = YES: Check for locked files.  Return only
         unlocked files.
  = NO:  Do not check for locked files.  Return both
         locked and unlocked files.
Subcommand -READ:
Read a file or files from the silo to the current host system.

INFILE
Input file(s). Specifies which file(s) are to be read from the silo. UNIX syntax is expected. An absolute path or a relative path from the user's home directory on the silo may be specified. Wildcard characters are permitted. (See User Note 8)

OUTDIR(--)
Output directory. Specifies the destination directory on the host system. If NULL, the current directory is assumed. OUTDIR will be created if it does not exist. UNIX syntax is expected. OUTDIR is ignored if CACHFLG=YES.

CACHFLG(NO)
Cache flag. Specifies whether to transfer the file(s) from tape to cache disk only, or copy it to the host system as well. If CACHFLG=YES and the file already exists on the cache disk, the request will be ignored. See User Note 3.

  = YES:  Copy INFILE to cache disk only
  = NO:   Copy INFILE to cache disk and host system
Subcommand -WRITE:
Write a file or files from the current host system to the silo.

INFILE
Input file(s). Specifies which file(s) are to be written to the silo. UNIX syntax is expected. An absolute path or a relative path from the current directory may be specified. Wildcard characters are permitted. (See User Note 8)

OUTDIR(--)
Output directory. Specifies which directory the files are to be placed in on the silo. If NULL, the user's home directory on the silo is assumed. OUTDIR will be created on the silo if it does not exist. UNIX syntax is expected.

Examples:

  1. LAS> silo-delete infile=/csb/baltzer/filex

    The file "filex" will be deleted from the specified directory on the silo.

  2. LAS> silo-dir infile=a* outfiles=names location=loc nfiles=num maxfils=10 lockflg=no

    All files matching a* in the user's home directory on the silo are examined. The name of each file found is returned in the TAE string variable NAMES, the location of the files are returned in the TAE string variable LOC, and the number of files found is returned in the TAE integer variable NUM. At most 10 file names will be returned. Both locked and unlocked files are returned.

    NOTE: If a subdirectory matching a* exists in the user's home directory, all files matching a* in those subdirectories will also be returned. See User Note 1.

  3. LAS> silo-read infile=testdir/filex outdir=/usr/people/ops

    The file "filex" will be read from the "testdir" directory on the silo. This directory lies directly below the user's home directory. The file will be copied into the directory "/usr/people/ops" on the host system. If the environment variable DATAPATH is set, a symbolic link will be created in OUTDIR and the actual file will be placed in the appropriate directory specified by DATAPATH.

  4. LAS> silo-read infile=/csb/baltzer/file2

    The file "file2" will be read from the specified directory on the silo. Since OUTDIR has not been specified, the file will be copied into the user's current directory on the host system. If the environment variable DATAPATH is set, a symbolic link will be created in the current directory and the actual file will be placed in the appropriate directory specified by DATAPATH.

  5. LAS> silo-read infile=/csb/baltzer/file2 cachflg=yes

    The file "file2" will be transferred from tape to cache disk on the silo. The file will not be copied to the host system. The file may be copied later to the host system by executing a SILO-READ command with CACHFLG=NO.

  6. LAS> silo-write infile=*.arch outdir=testdir

    The files "*.arch" within the current directory on the host system are written to the silo. The files are all written to the directory "testdir" in the user's home directory on the silo.

  7. LAS> silo-write infile=../archdir/*.arch

    The path of the files "*.arch" has been specified relative to the current directory. This path name could have instead been specified relative to the root directory. Since OUTDIR has not been specified, the files will be written to the home directory of the user on the silo.

Description/Algorithm:

SILO makes use of the system commands "umput", "umget", "umdel", and "udir" to transfer, retrieve, delete, and determine the status of files on the silo. In the event of an error, three attempts to complete the process will be made before terminating with an error.

Nonfatal Error Messages:

    None.

Fatal Error Messages:

  1. [silo-abort] User aborted SILO-<xxxxx>

    The user aborted the specified subcommand.

  2. [silo-alloc] Error allocating memory

    The necessary memory could not be allocated. Contact the system administrator.

  3. [silo-chdir] Error moving to specified directory

    SILO was unable to change directory to OUTDIR. Ensure the directory was specified correctly and that the permissions are set appropriately.

  4. [silo-ddiag] Error deleting <xxxxx> on silo

    The specified file(s) could not be deleted from the silo. Ensure INFILE was specified correctly, the file(s) exist on the silo, and the file protections set appropriately.

  5. [silo-errno] System errno = <xxx>

    To help track certain fatal errors, the value of the C global variable errno is printed. This value may help the system administrator track the error if the user cannot resolve the problem.

  6. [silo-fatal] Fatal error encountered

    A fatal error was encountered. The error message preceding this message is the specific error encountered. Processing terminates.

  7. [silo-link] Error following symbolic link

    INFILE is a symbolic link on the silo. An error was encountered following the link to the target file. Contact the system administrator.

  8. [silo-mkdir] Error creating specified directory

    SILO was unable to create the directory. Ensure the directory was specified correctly and that the permissions are set appropriately.

  9. [silo-queue] Queue operation error

    An error was encountered operating the queue. Contact the system administrator.

  10. [silo-rdiag] Error reading <xxxxx> from silo

    An error was encountered reading the specified file(s) from the silo. Ensure INFILE was specified correctly and the permissions are set appropriately.

  11. [silo-sdiag] Error getting status for <xxxxx> on silo

    An error was encountered determining if INFILE is currently located on DISK or TAPE. Ensure the permissions are set appropriately.

  12. [silo-status] Error retrieving file status

    An error was encountered retrieving the necessary file status information from the silo. Ensure the file exists and the permissions are set appropriately.

  13. [silo-syscall] Error returned from system call

    An error was returned from the operating system. If the error persists, contact the system administrator.

  14. [silo-systatus] Return status from system call = <xxx>

    To help track certain fatal errors, the returned value from the system call is printed. This value may help the system administrator track the error if the user cannot resolve the problem.

  15. [silo-tae] Error sending variable to TAE

    An error was encountered sending a variable value to TAE. When running the -DIR subcommand, too many file names may have been found to pass back to the specified TAE variable. Reduce the value of MAXFILES and re-run. If the error persists, contact the system administrator.

  16. [silo-wdiag] Error writing <xxxxx> to silo

    An error was encountered writing the specified file(s) to the silo. Ensure INFILE was specified correctly and the permissions are set appropriately.

  17. [silo-wild] Wildcard characters not permitted in <xxxxx>

    Wildcard characters are not permitted when creating a symbolic link or when waiting for a file to be transferred to the cache disk. Do not use wildcard characters for the -LINK and -WAIT subcommands.

User Notes:

  1. Care must be taken when calling SILO-DIR and INFILE matches subdirectories within the target directory. In such cases, all of the files contained in the subdirectory will be returned as if they are located in the target directory.

  2. SILO will follow symbolic links to the file being pointed to. Thus, the file location returned by the -DIR subcommand is not the location of the link, but the file the link is pointing to. Likewise, the -WAIT subcommand will wait until the file the symbolic link is pointing to is located on the disk cache.

    The one exception is when running the -READ subcommand. If CACHFLG=YES and INFILE contains a wildcard character, the symbolic links are not followed by SILO but are passed directly to the silo. At the time SILO was developed, the silo software contained a bug which did not properly update the location of the file a symbolic link points to when the link is cached. To avoid this problem, it is recommended that all files be cached individually when symbolic links are present in a directory. If INFILE does not contain a wildcard character, the symbolic link is followed and the file that is pointed to will be cached.

  3. When running subcommand -READ and CACHFLG=YES a request is issued to the silo to transfer the file(s) from tape onto the cache disk. The user does not have to wait for this to occur. If the file already exists on the cache disk, the request will be ignored. Once the file has been cached, the user may transfer it to the host file system by specifying CACHFLG=NO. See User Note 2 about caching multiple files. An advantage of using this capability is that multiple files may be cached. This allows the silo to transfer files to the cache disk while the user is copying another file to the host system. In this scenario, the user may only have to wait for the first file to be transferred to the cache disk and subsequent files will be cached while the current file is being copied to the host system.

  4. The SILO transfer system is based on the File Transfer Protocol (FTP) and the EROS Data Center system commands "umput", "umget", "umdel", and "udir". Their syntax follows:

           umput [-v] [-m] [-d directory]
    
           umget [-v] [-d directory] [-s] file1 [file2 file3 ... ]
    
           umdel [-v] [-d directory] file1 [file2 file3 ... ]
    
           udir  [-d directory] file1 [file2 file3 ... ]
    

    A description of the optional flags follows:

    	v	Verbose mode.  Ftp messages are displayed.
    	d	Directory specification.
    	s	The file(s) are to be staged onto cache disk,
    		but not transferred to the host file system.
            m       Make the silo directory
    
    

    Other commands are available from UNIX by entering the FTP environment (enter "uftp"). Other silo interface functions may be performed via the Network File System (NFS), however, it is recommended that large files be transferred to and from the silo using FTP commands.

  5. silo interface functions are only supported on UNIX systems at the EROS Data Center.

  6. The file name displayed in the completion message does not include the directory specification.

  7. If the enviroment variable DATAPATH is set, SILO-READ will place the files read from the silo in in one of the directories specified by DATAPATH and a symbolic link to the file will be placed in the directory specified by OUTDIR.
  8. SILO has been modified to accept a LAS syntax file name specification with no window specifications. In addition, the unitree prefix may be specified. If the unitree prefix is specified it will be removed before being used. The following are examples of acceptable file specifications:

            /csb/dlloyd/test_us.img
    	/unitree/csb/dlloyd/test_us.img
    	[csb.dlloyd]test.us;img
    	[unitree.csb.dlloyd]test.us;img
    
  9. When using square brackets as a wildcard specification, make sure that a directory is also specified, otherwise the wildcard will be interpreted as a LAS directory specification.