DMGET(1) General Commands Manual DMGET(1) NAME dmget - Recalls previously migrated files SYNOPSIS dmget [-h | -a] [-c] [-n] [-q] [-v] [-0] [-B byterange-list] [-U priority] [-V mspvglist] [--list | --defer] [--recurse] [file-list] IMPLEMENTATION SGI systems licensed for the Data Migration Facility (DMF) and systems functioning as DMF clients. DESCRIPTION NB: Altered by CSIRO to reflect their use of a wrapper script around the SGI program. The dmget command recalls previously migrated files. You can use this command to retrieve migrated files for which you have read permission; the system administrator can use this command to retrieve any migrated files. Only root users can use this command to recall files for which they do not have read permission. (CSIRO only) CSIRO's wrapper around the SGI dmget program is intended to prevent one user who requires a large amount of data to be recalled from locking out a following user with more mod- est demands. It does this by determining which tapes the files reside on, and processes them in batches tape by tape. This minimises tape mounts and multiple passes over tapes, which is kinder to the system and would result in faster processing for the user. Batches by one user may be interleaved with batches from a different one. In addition, if the standard error file is attached to the screen, the wrapper provides the user with a rough indication of how much work has been requested and how long it might take. For example: ruby$ dmget * You are recalling 12 of the 19 files specified. The oldest currently queued recall request has been waiting for 0h 2m 5 tape mounts may be required. ruby$ A prompt is also issued if interactive entry of filenames is required. It also provides a solution to the problem of recalling a large amount of data, where the first files recalled revert to off-line status before they can be processed. (See Example 2 below.) A recursive option is provided to make recalling files in a multi-level directory structure easier. (See Example 1 below.) CSIRO's wrapper also allows you to specify files which you would like recalled for you overnight, for use the next day. There is no guarantee that these requests will be acted on, but if they are, then access (via a second normal dmget) ought to be much faster. The dmget command accepts the following options: Option Description -h Causes the file-list to be treated as a list of Data Management API (DMAPI) fhandles rather than file paths. The fhandle is the DMF representation of the DMAPI name for the file in the file system. This option requires root privileges. -a Update the access time of the file. This option cannot be specified when -h is used. -c (CSIRO only - not supported) Converts any octal escape sequences in the list of file paths (i.e. output of dmscanfs) to the original characters. -n Directs the dmget command to abort if the DMF daemon is not responding or if the daemon stops responding while the request is being processed. The default is to wait until the re- quest has finished processing. -q Directs the dmget command to exit as soon as all recall requests have been sent to and acknowledged as received by the DMF daemon. The command will not wait for verification that the recall has completed, successfully or otherwise. It is then up to the caller to determine when and if the recall succeeds. Note: If the DMF daemon is stopped or otherwise exits before the recall request is complete, the regions of the file being recalled will be returned to the offline state when the DMF daemon is restarted. In this case the recall request will never be observed as complete and the request must be re-issued. If the dmget command is executed without the -q op- tion, the command will persist across DMF daemon restarts to complete the recall. Also note that the -q option does not change the dmget command's behavior in regards to the presence or absence of the -n option. That is, if the -n option is not present and the DMF daemon is not running, the dmget command with the -q option will wait until the DMF daemon has been started and the command has received acknowledgment that the requests were received before exiting. -v Specifies verbose mode. This option is useful for administrators when trying to determine the cause of operational problems between DMF client commands and the server host machine. This option might cause the command to display more information regarding the cause of an error than the command does without the option, if such information is available. The op- tion is not useful in cases of normal client/server operation. -0 This option can be used when dmget is reading a list of items (i.e. file paths) from standard input. It specifies that the items are separated by a null character (instead of by a newline). dmfind with the -print0 expression produces input suitable for this mode. -B byterange-list Specifies that only a part of the file should be recalled to disk; the other parts may or may not be offline. A byterange-list is one or more byteranges separated by commas. A byterange is of the form "mmmm:nnnn" where "mmmm" specifies the first byte in the range and "nnnn" specifies the last byte in it. Each of "mmmm" and "nnnn" counts from the be- ginning of the file if unsigned, and backwards from the end if preceded by "-". The first byte in the file is byte "0", "BOF", or "bof", and the last one is byte "-0", "EOF", or "eof", so "bof:eof", "BOF:EOF", and "0:-0" all define a range covering the entire file. A byterange may also of the form "mmmm+llll" where "mmmm" is a byte position as above, and "llll" is the number of bytes in the range, starting at "mmmm". If "llll" starts with a minus sign, it specifies a range of "llll" bytes ending at "mmmm". A single number "llll" (without a colon or plus symbol) is also acceptable as a byterange, being interpreted as "EOF+llll" if "llll" starts with a "-", or "BOF+llll" if it doesn't. Each number can be either a hexadecimal number of the form "0xnnnn", a based number of the form "base#nnnn", or a decimal number with an optional trailing scaling character. The decimal number may include a "." and exponent. The trailing scaling character may be k, K, m, M, g, G, t, T, p, or P, all of which are powers of 1000, not 1024. Byteranges are rounded to disk block boundaries. DMF may decide to join nearby ranges together if necessary, or adjust them to match the size of the file. -U priority Specifies a priority for the recall (think urgency mnemonically). The valid range is 1-1024. The default value is 512. Higher values have higher priority. Only the root user on the DMF server or on machines defined in the VALID_ROOT_HOSTS configuration file parameter may specify this parameter. -V mspvglist Specifies a list of MSP, volume group, or migrate group names from which the file(s) should be recalled if possible. If the file has a copy in one of the elements in the list, the DMF daemon will attempt to recall the file first using that copy, overriding the default recall order as defined in the DMF configuration file. This option is only available to the root user on the DMF server or on machines defined in the VALID_ROOT_HOSTS configuration file parameter; otherwise the command behaves as if this option had not been entered. --defer (CSIRO only) Specify files which should be recalled overnight if system load permits. If the system is too busy then, these prefetch requests will be ignored. This means that use of this option is not a replacement for a dmget just prior to file use, but it may well accelerate it. This option cannot be specified when -h, --list or -B is used. It implies the -a option. --list (CSIRO only) No recalls are performed if this option is specified. Instead, the file paths specified by file-list (or read from standard input) are written to standard output in the order that they would probably have been recalled had this option been omitted. This list includes dual-state and unmigrated files as well as offline and partial-state ones. See below for an example of how this is intended to be used. This option cannot be specified when --defer is used. --recurse (CSIRO only) Specify that all files in the nominated directory(ies) and all their subdirectories should be recalled, thereby reducing the need to use the find command for this pur- pose. file-list Specifies a list of file paths or fhandles to recall. If you do not specify file-list, dmget reads the list of file paths or fhandles from standard input. The dmget command is synchronous. Therefore, you cannot initiate another process until all of the file paths or fhandles in file-list are recalled. If the offline medium is tape, this could result in a delay while waiting for tape resources. Running the command in the background will avoid this behaviour if you wish. Only files residing on file systems configured for use with DMF may be recalled. EXAMPLES Example 1: To efficiently recall all files in a directory named mydir and its subdirectories, use the following command: dmfind mydir -state MIG -o -state OFL -o -state PAR | dmget Note that piping the pathnames directly into dmget is much more efficient than the use of the -exec option or the xargs command. dmattr, dmput and dmtag also accept pathnames on their standard input file. The following CSIRO-only command is equivalent to the above: dmget --recurse mydir Example 2: (CSIRO only) When recalling a large number of files in one operation, there is the risk of the files recalled earlier in the recall reverting to offline before the user has had a chance to process them. Use of the -a option (equivalent to a prior touch -a of the files) may help, but this is not guaranteed. A different solution, if the order in which files are processed doesn't matter, is to use a feature of the wrapper where it will perform a dummy run, listing the files in the order in which it would have recalled them, but without actually doing so. This allows the user's processing to be arranged in the same order. A second (genuine) dmget of the same files is run in background. Assuming the processing is not significantly slower than the rate of recalls, the foreground and background process would stay in step with each other as they work their way through the files. It would look something like this: dmget --list file1 file2 file3 file4 > $TMPDIR/lof dmget < $TMPDIR/lof & for f in `cat $TMPDIR/lof`; do process_one_file $f dmput -r $f done The dmput is not strictly necessary, but is a politeness to DMF to say that the files have been finished with, if of course this is so. Example 3: The following command recalls the first 4000 bytes (rounded to a disk block boundary) of partfile back to disk: dmget -B 0:4000 partfile FILES /etc/dmf/dmf.conf DMF configuration file SEE ALSO dmfind(1) for information about finding files dmput(1) for information about migrating online files to offline media dmfsmon(8) for information about selecting files for migration DMGET(1)