All FTP extensions described in Chapter 2 are supported by PFTP. In addition, the following commands (abbreviations) are supported by PFTP:
Pipes are NOT supported for the pget and pput commands even if the HPSS.conf specifies a valid pipe directory.To use pipes it is mandatory that you specify the
autop
command to disable automatic substitution of parallel commands and then explicitely use the
get
or
put
commands.
Connected to water.clearlake.ibm.com.
220-# HPSS Parallel FTP Deamon on water
220 water FTP server (HPSS 4.3 PFTPD V1.1.4 Tue May 29 14:55:58 CDT 2001) ready.
331 Password required for /.../water_cell.clearlake.ibm.com/whrahe.
230 User /.../water_cell.clearlake.ibm.com/whrahe logged in.
Using binary mode to transfer files.
**** NOTE: Server supports Parallel Features ****
**** Auto-Parallel Substitution Enabled. ****
The pappend command transfers a file from the local machine to HPSS. The transfer starts at the end of the remote file and continues until the entire file is moved or until an error occurs.
local_file - Identification of the file to transfer on the local machine.
remote_file - Optional file name to the remote file. If not supplied then the remote (HPSS) file name defaults to be the same as the local file name.
The pput command transfers a file from the local machine to HPSS. If offsets and size of transfer are not specified, the transfer starts at the beginning of the local file and continues until the entire file is moved or until an error occurs. However, flexibility is provided to perform partial file transfers by specifying local file offset, remote file offset, and size of transfer. The local_offset , remote_offset , and size may be specified using a decimal and magnitude representation string.
The normal pput command functions just like the standard ftp put command and transfers an entire file.
-l local_offset - Optional byte offset into the local file where the transfer is to begin.
-r remote_offset - Optional byte offset into the remote file where the data is to be placed.
-s size - Optional byte size of the amount of data to transfer.
local_file - Identification of the file to transfer on the local machine.
remote_file - Optional file name to the remote (HPSS) file. If not supplied then the remote file name defaults to be the same as the local file name.
Connection Failures - data transfer connection malfunction.
Network Failures - data transfer malfunction.
Allocation Failures - no space on remote machine for file.
Error codes may also be returned from HPSS. The most common error codes are:
1. Transfer local file testfile to the user's HPSS home directory.
2. Transfer local file testfile to HPSS file prod1 in the current working directory.
3. Transfer 1MB from offset 1MB of local file testfile to offset 0 of HPSS file /home/bob/prod1.
ftp> pput -l 1048576 -r 0 -s 1048576 testfile /home/bob/prod1.
4. Transfer all local files which begin with "test" to the user's HPSS home directory using a pipe and tar (bundling).
ftp> pput " | tar cf - ./test*" my_test.tar
Parallel Pipes are NOT supported! Specify the "autop" command to disable automatic parallel command substitution nad use the "put" command!
The mpput command expands the files specified in the local_files parameter at the local host and copies the indicated files to HPSS. The mpput command functions just like the standard ftp mput command.
The pget command transfers a file to the local machine from HPSS. If offsets and size of transfer are not specified, the transfer starts at the beginning of the remote file and continues until the entire file is moved or until an error occurs. However, flexibility is provided to perform partial file transfers by specifying remote file offset, local file offset, and size of transfer. The local_offset , remote_offset , and size may be specified using a decimal and magnitude representation string. See Section 2.1.10 for use of this notation.
The standard pget command transfers entire files similar to the standard ftp get command.
-r remote_offset - Optional byte offset where transfer is to begin in the remote file.
-l local_offset - Optional parameter where the data is transferred in the local file.
-s size - Optional number of bytes to transfer.
remote_file - Identification of the file to transfer from the remote (HPSS) host.
local_file - Optional file name to the local file. If not supplied then the local file name defaults to be the same as the remote file name.
Connection Failures - data transfer connection malfunction.
Network Failures - data transfer malfunction.
Allocation Failures - no space on local machine for file.
Error codes may also be returned from HPSS. The most common error code is:
1. Transfer HPSS file /home/bob/prod1 to the user's local directory.
2. Transfer HPSS file prod1 in the current working directory to local file testfile1.
3. Transfer 1MB from offset 0 of HPSS file /home/bob/prod1 to offset 1048576 of local file testfile.
ftp> pget -r 0 -l 1048576 -s 1048576 /home/bob/testfile1 testfile
4. Transfer and untar a tar file into the user's current working directory using a pipe and tar (unbundling).
ftp> pget my_test.tar " | tar xf -"
Parallel Pipes are NOT supported! Specify the "autop" command to disable automatic parallel command substitution nad use the "get" command!
The mpget command expands the remote_files parameter at the remote (HPSS) host and copies the indicated HPSS files to the current directory on the local host. The mpget command functions just like the standard ftp mget command.
The lfappend is a performance optimized Parallel FTP Client command used to append a "globally available" file into HPSS using the "parallel" protocols. IF the input file is not available to the mover(s) machine (NOT "globally available") the transfer will fail because the mover(s) will NOT be able to locate the desired file. The difference between lfappend and pappend is that the mover(s) involved in the transfer, will not use the network to move the data providing improved performance. The mover(s) machine MUST be correctly configured. The file: /var/hpss/etc/ hpss_mvr_localfilepath.conf MUST exist on each "local file" aware mover machine and MUST contain entries specifying which directories are eligible for "local file" transport.
The lfappend command transfers a file from the local machine to HPSS. The transfer starts at the end of the remote file and continues until the entire file is moved or until an error occurs.
local_file - Identification of the "globally available" file to transfer.
remote_file - Optional file name to the remote file. If not supplied then the remote (HPSS) file name defaults to be the same as the "globally available" file name.
Connection Failures: data transfer connection malfunction.
Network Failures: data transfer malfunction.
Allocation Failures: no space on remote machine for file.
Error codes may also be returned from HPSS. The most common error codes are:
1. Append "globally available" file testfile to the same file name in the user's HPSS home directory.
. . . (Information returned by the PFTP Daemon)
2. Append "globally available" file testfile to HPSS file prod1 in the current working directory.
The lfput is a performance optimized Parallel FTP Client command used to transfer a "globally available" file into HPSS using the "parallel" protocols. IF the file is not available to the mover(s) machine (NOT "globally available") the transfer will fail because the mover(s) will NOT be able to locate the desired file. The difference between lfput and pput is that the mover(s) involved in the transfer, will not use the network to move the data providing improved performance. The mover(s) machine MUST be correctly configured. The file:
/var/hpss/etc/hpss_mvr_localfilepath.conf MUST exist on each "local file" aware mover machine and MUST contain entries specifying which directories are eligible for "local file" transport.
If offsets and size of transfer are not specified, the transfer starts at the beginning of the local file and continues until the entire file is moved or until an error occurs. However, flexibility is provided to perform partial file transfers by specifying local file offset, remote file offset, and size of transfer. The local_offset , remote_offset , and size may be specified using a decimal and magnitude representation string. See Section 2.1.10 for use of this notation. The normal lfput command functions just like the standard ftp put command and transfers an entire file.
-l local_offset - Optional byte offset into the "globally available" file where the transfer is to begin.
-r remote_offset - Optional byte offset into the remote file where the data is to be placed.
-s size - Optional byte size of the amount of data to transfer.
local_file - Identification of the "globally available" file to transfer. (MUST be available to mover(s) machines)
remote_file - Optional file name to the remote (HPSS) file. If not supplied then the remote file name defaults to be the same as the "globally available" file name.
Connection Failures - data transfer connection malfunction.
Network Failures - data transfer malfunction..
Allocation Failures - no space on remote machine for file.
Error codes may also be returned from HPSS. The most common error codes are:
1. Transfer local file testfile in the current working directory of the client to the user's HPSS home directory.
. . . (Information returned by the PFTP Daemon)
. . . (Information returned by the PFTP Daemon)
2. Transfer local file testfile in the current working directory of the client to HPSS file prod1 in the current working directory.
. . . (Information returned by the PFTP Daemon)
3. Transfer 1MB from offset 1MB of local file testfile in the current working directory of the client to offset 0 of HPSS file /home/bob/prod1 with a new name testfile2..
ftp> lfput -l 1048576 -r 0 -s 1048576 testfile /home/bob/prod1/testfile2
The lfget is a performance optimized Parallel FTP Client command used to transfer an HPSS file into a "globally available" file using the "parallel" protocols. IF the current working directory or the specified directory is not available to the mover(s) machine (NOT "globally available") the transfer will fail because the mover(s) will NOT be able to locate the desired file. The difference between lfget and pget is that the mover(s) involved in the transfer, will not use the network to move the data providing improved performance. The mover(s) machine MUST be correctly configured. The file: /var/hpss/etc/ hpss_mvr_localfilepath.conf MUST exist on each "local file" aware mover machine and MUST contain entries specifying which directories are eligible for "local file" transport.
The lfget command transfers a file from HPSS to a "globally available" directory on the mover machine(s). If offsets and size of transfer are not specified, the transfer starts at the beginning of the remote file and continues until the entire file is moved or until an error occurs. However, flexibility is provided to perform partial file transfers by specifying remote file offset, local file offset, and size of transfer. The local_offse t, remote_offse t, and size may be specified using a decimal and magnitude representation string. See Section 2.1.10 for use of this notation.
The standard lfget command transfers entire files similar to the standard ftp get command.
-r remote_offset - Optional byte offset where transfer is to begin in the remote file.
-l local_offset - Optional parameter where the data is transferred in the local file.
-s size - Optional number of bytes to transfer.
remote_file - Identification of the file to transfer from the remote (HPSS) host.
local_file - Optional file name to the local file. If not supplied then the local file name defaults to be the same as the remote file name.
Connection Failures - data transfer connection malfunction.
Network Failures - data transfer malfunction.
Allocation Failures - no space on local machine for file.
Error codes may also be returned from HPSS. The most common error code is:
1. Transfer HPSS file /home/bob/prod1 to the user's local directory.
. . . (Information returned by the PFTP Daemon)
2. Transfer the HPSS file prod1 in the current working directory to local file testfile1.
. . . (Information returned by the PFTP Daemon)
3. Transfer the HPSS file testfile into the "globally available" directory /home/bob renaming the file to testfile.
ftp> lfget prod1 /home/bob/testfile1 testfile
. . . (Information returned by the PFTP Daemon)
4. Transfer 1MB from offset 0 of HPSS file /home/bob/prod1 to offset 1048576 of local file testfile.
ftp> lfget -r 0 -l 1048576 -s 1048576 /home/bob/testfile1 testfile
The mlfput is a performance optimized Parallel FTP Client command used to transfer multiple "globally available" files into HPSS using the "parallel" protocols. IF the file(s) is not available to the mover(s) machine (NOT "globally available") the transfer will fail because the mover(s) will NOT be able to locate the desired file. The difference between mlfput and mpput is that the mover(s) involved in the transfer, will not use the network to move the data providing improved performance. The mover(s) machine MUST be correctly configured. The file: /var/hpss/etc/ hpss_mvr_localfilepath.conf MUST exist on each "local file" aware mover machine and MUST contain entries specifying which directories are eligible for "local file" transport.
The mlfput command expands the files specified in the local_files parameter at the local host and copies the indicated files to HPSS. The mlfput command functions just like the standard ftp mput command.
Connection Failures - data transfer connection malfunction.
Network Failures - data transfer malfunction.
Allocation Failures - no space on remote machine for file.
Error codes may also be returned from HPSS. The most common error codes are:
1. Transfer all "globally available" files in the current directory to the user's HPSS home directory.
. . . (Information returned by the PFTP Daemon)
ftp> prompt(<== Toggles File Prompting)
. . . (Information returned by the PFTP Daemon)
2. Transfer all "globally available" files which begin with test in directory /usr/bob to the user's PSS home directory.
. . . (Information returned by the PFTP Daemon)
The mlfget is a performance optimized Parallel FTP Client command used to transfer an HPSS file into a "globally available" file using the "parallel" protocols. IF the current working directory or the specified directory is not available to the mover(s) machine (NOT "globally available") the transfer will fail because the mover(s) will NOT be able to locate the desired file. The difference between mlfget and mpget is that the mover(s) involved in the transfer, will not use the network to move the data providing improved performance. The mover(s) machine MUST be correctly configured. The file: /var/hpss/etc/ hpss_mvr_localfilepath.conf MUST exist on each "local file" aware mover machine and MUST contain entries specifying which directories are eligible for "local file" transport.
The mlfget command expands the remote_files parameter at the remote (HPSS) host and copies the indicated HPSS files to the current directory on the local host. The mlfget command functions just like the standard ftp mget command.
Connection Failures - data transfer connection malfunction.
Network Failures - data transfer malfunction.
Allocation Failures - no space on remote machine for file.
Error codes may also be returned from HPSS. The most common error code is:
1. Transfer all files in HPSS directory /home/bob to the "globally available" current working directory
. . . (Information returned by the PFTP Daemon)
2. 2. Transfer all HPSS files which begin with test in directory /home/bob to the "globally available" current working directory.
The setpwidth command is used to specify the size of the client side stripe to the FTP client code.
stripe_width - The width of the PFTP client-side stripe. The width can have a value of 1 through 16. The default width is 1. The stripe width from the PFTP client perspective is the number of client processes spawned to handle the data transfers. Stripe width from the server perspective is the number of volumes the file is striped across.
A general guideline would be to set stripe_width to an even divisor of the number of volumes the file is striped across. For example, if the Class of Service for a file were set up for a 4-way stripe, suggested values for stripe_width might be 2 or 4.
If the stripe width of the file is unknown, consult your HPSS administrator or follow the following steps to determine the stripe width.
1. Enter the lshpss -cos command to list Class of Service information. From the entry with the CID value equal to your Class of Service ID, locate the HID (hierarchy ID) field.
2. Enter the lshpss -hier command to list hierarchy information. From the entry matching the HID value above, locate the StorageClassID field.
3. Enter the lshpss -sc command to list the storage class information. From the entry matching the StorageClassID value above, locate the W field. This is the stripe width for the storage class of the file.
The setpblocksize command is used to specify the block size to be used for parallel transfers. The block_size may be specified using a decimal and magnitude representation string. See Section 2.1.10 for use of this notation. The maximum blocksize is 16mb.
block_size - The number of bytes to be transferred to each element of the stripe before data is sent to the next element. The current allowable transfer sizes range from 1 through 16MB. The default block size is 256KB
A general guideline would be to set block_size to the virtual volume block size. Consult your HPSS administrator or follow the following steps to determine the virtual volume blocksize.
1. Enter the lshpss -cos command to list Class of Service information. From the entry with the CID value equal to your Class of Service ID, locate the HID (hierarchy ID) field.
2. Enter the lshpss -hier command to list hierarchy information. From the entry matching the HID value above, locate the StorageClassID field.
3. Enter the lshpss -sc command to list the storage class information. From the entry matching the StorageClassID value above, locate the VVBlk field. This is the virtual volume block size.
The multinode command is used to enable/disable the "desire" to perform a parallel file transfer using multiple nodes. When multinode is enabled, the pftp_client will process the multinode configuration file. If the process cannot obtain a single node to perform the parallel transfer, then the transfer will occur using non-multinode parallel method.
The autoparallel command is used to enable/disable the automatic mapping of non-parallel commands to parallel commands; e.g., get ==>pget. In autoparallel mode (enabled), transfers involving files smaller than the "Auto Parallel Size" specification in the HPSS.conf will NOT be auto-mapped.
Display the (transfer) parameters between the Client and other hosts (default is the PFTP Daemon host.)
Effective Tuning parameters from saux22 to sair031.sandia.gov
Using PDATA_AND_MOVER protocol
Parallel Transfer Size = 2147483647
Transfer Buffer Size = 16777216
No Interfaces Found (ret_code = -2)
Using Default Interface for 1 stripes(s)
Processing the multinode list, please wait.....
Using 1 remote node(s) from the following:
Control Interface ==> Data Interface:
Using "Default" destination characteristics
PdataSockBufSize = 1048576 based on user input
recv_socksize = send_socksize = 1048576 based on
PdataSockBufSize
NOTE: Parallel Pipes are NOT supported regardless of the following:
recv_socksize = 262144 based on Network Options
send_socksize = 262144 based on Network Options
PdataSockBufSize = 262144 based on send_socksize
PipeFileSize TOO Large reset to 2147483647
Pipe Files NOT supported on this machine - Open Failed 2
Pipe Files are supported on this machine
Explicitly specify parallel transfers to use the PDATA_AND_MOVER protocol (Original protocol) regardless of what is specified in the HPSS.conf file.