NcFTP: The flexible FTP client
Using NcFTP client utilities
NcFTP is a product that comes as either an FTP server or FTP client. However, both utilities are referenced as NcFTP. In this article, I only use the NcFTP client. Therefore, when I mention NcFTP I am referring to just the NcFTP client. Additionally, various command-line utilities come with the client, which makes the client very attractive for use on the command line or in batch mode.
NcFTP utilities let system administers provide FTP flexibility and performance. NcFTP can be used with the current AIX® default FTP service with no known issues. NcFTP offers the ability to resume failed transfers and retries, which is very useful when transferring large files across the network or across networks that suffer from bottlenecks. NcFTP can output a status bar detailing the progress of the transfer (much like SCP). However, NcFTP does not support any encryption, which might be an issue for some installations.
It's very common for installations to have a current AIX FTP server running while using NcFTP as an ad hoc client, and this type of installation is what this article focuses on.
The NcFTP client comes with the following utilities:
Table 1. NcFTP utilities
|ncftp||Main FTP interface|
|ncftpbatch||Batch command line|
|ncftpspooler||Spooler for FTP|
Note: In this article, I will not be discussing the ncftpspooler utility.
NcFTP can be downloaded from the NcFTP home page (see Related topics for a link).
After you have downloaded the client, extract the binaries, as shown in Listing 1:
Listing 1. Extracting the binaries
# gunzip /tmp/ncftp-3.2.3-aix5.3.0-export.tar.gz # cd /opt # tar –xvf /tmp/ncftp-3.2.3-aix5.3.0-export.tar
Check to ensure that NcFTP starts, as shown in Listing 2.
Listing 2. Starting NcFTP
$ ncftp -v $ Usage: ncftp [flags] [<host> | <directory URL to browse>] Flags: -u XX Use username XX instead of anonymous. -p XX Use password XX with the username. -P XX Use port number XX instead of the default FTP service port (21). -j XX Use account XX with the username (rarely needed). -F Dump a sample $HOME/.ncftp/firewall prefs file to stdout and exit. Program version: NcFTP 3.2.3/433 Jul 28 2009, 10:02 AM Library version: LibNcFTP 3.2.3 (July 23, 2009) Build system: AIX Voltigeur 3 5 000D3A9F4C00
Depending on the type of remote FTP host you are accessing, you may have to prefix the hostname with ftp://, such as:
Note: For the ncftpls utility, you must use the previous format when connecting to the remote host.
If you experience connection issues with FTP, use this checklist to try and troubleshoot your problem.
- Check that the remote FTP service is running.
- Check that the remote FTP port is not blocked.
- Check to ensure that the tcp wrappers are not denying the connection.
- Check that your user name is not in /etc/ftpusers.
- Check that the host or user is not being denied in /etc/ftpaccess.ctl.
Using NcFTP interactively
Confirm that you can connect to a remote server of your choice using NcFTP. In Listing 3, I connect to the private host tardis, using the user name of dxtans and master as the password.
Listing 3. Connecting to a remote server
$ ncftp -u dxtans -p master tardis NcFTP 3.2.3 (Jul 28, 2009) by Mike Gleason (http://www.NcFTP.com/contact/). Connecting to 192.168.4.10... Logging in... Password Ok, User logged in Logged in to tardis. Current remote directory is /. ncftp/>
If all looks good, let's explore NcFTP.
Listing 4 demonstrates how NcFTP is used in a common FTP session. A
cd command is used to change to /tmp on the local side, then to the /ftproot/stats directory on the remote side.
A couple of
ls commands are executed before a
get command is issued to retrieve the remote file stat_09.txt. It is very similar to a normal FTP session, but there are extra features, as you will see.
Listing 4. Ftp_connect
$ ncftp -u dxtans -p master tardis NcFTP 3.2.3 (Jul 28, 2009) by Mike Gleason (http://www.NcFTP.com/contact/). Connecting to 192.168.4.10... Logging in... Password Ok, User logged in Logged in to tardis. ncftp / > lcd /tmp ncftp / > cd /ftproot/stats Change directory ok ncftp /ftproot/stats > ls desktop.ini pebs/ spec_ja stat_09.txt stat_jan.rtf ncftp /ftproot/stats > ls -l -------rw- 48 Jan 18 13:58 desktop.ini d------rwx 0 Jan 23 02:36 pebs -------rw- 564 Jan 23 08:17 spec_ja -------rw- 61 Jan 18 13:53 stat_09.txt -------rw- 18927 Mar 23 2004 stat_jan.rtf ncftp /ftproot/stats > get stat_09.txt
When you initially invoke NcFTP, it creates a .ncftp directory in your HOME directory. From here you can personalize your NcFTP sessions, including any proxy settings. The most amended file will be the prefs_v3 file. The content of the prefs_v3 file is pretty self-explanatory. After you view it, you can edit the file to tailor it for your personal FTP sessions. Each entry has the format:
For example, to set the connection timeout I could use:
The preferences can also be set on the NcFTP command line as well, using the
ncftp/> set connect-time 20
Frequent FTP destinations can be saved as bookmarks, as demonstrated in Listing 5.
Looking more closely at Listing 5, you can see that a connection is made to the remote host tardis, specifying the destination directory as /ftproot/stats. The FTP session is then terminated. As this is a new remote connection to a remote directory, NcFTP asks if you want to save this connection and password. I typed yes, and called the saved session tardis_stats. The bookmark entry is saved to the bookmarks file in the .ncftp directory.
Listing 5. Bookmarks
$ ncftp -u dxtans -p master tardis /ftproot/stats NcFTP 3.2.3 (Jul 28, 2009) by Mike Gleason (http://www.NcFTP.com/contact/). Connecting to 192.168.4.10... Logging in... Password Ok, User logged in Logged in to tardis. Current remote directory is /ftproot/stats. ncftp /ftproot/stats > quit You have not saved a bookmark for this site. Would you like to save a bookmark to: ftp://dxtans:PASSWORD@tardis/ftproot/stats/ Save? (yes/no) yes Enter a name for this bookmark, or hit enter for "tardis": tardis_stats You logged into this site using a password. Would you like to save the password with this bookmark? Save? [no] yes Bookmark "tardis_stats" saved
The next time I want to use this remote connection I only need to supply the bookmark as part of the
ncftp parameter, as shown in Listing 6.
If you do not want to see a bookmark prompt every time you go to a new remote destination, set the
confirm-close value to no in the prefs_v3 file:
Or, from the NcFTP command line, enter the following command:
ncftp/>set confirm-close no
Listing 6. Load bookmark
$ ncftp tardis_stats NcFTP 3.2.3 (Jul 28, 2009) by Mike Gleason (http://www.NcFTP.com/contact/). Connecting to 192.168.4.10... Logging in... Password Ok, User logged in Logged in to tardis. Change directory ok Current remote directory is /ftproot/stats. ncftp /ftproot/stats >
Over time, you will undoubtedly get a fairly populated bookmarks file, which may require some housekeeping. To edit this file, you can use the NcFTP utility ncftpbookmarks, which is a menu-type interface, but is not very intuitive. I recommend simply editing the bookmarks file using
vi, and change as needed.
When transferring large files or transferring files over networks that suffer from outages, it is inevitable that at some point you will get a failed FTP, with the file partially transferred. Normally for an FTP, this would mean resubmitting the job and overwriting the original transfer. However, with NcFTP you can resume the session by either appending or overwriting. When uploading or downloading a file that already exists, but with different file sizes, NcFTP will try and resume the session where it stopped, without prompting. If you want to be prompted on what action NcFTP should take, change the auto-resume preference to no:
Listing 7 shows the action you can take after a failed partial FTP transfer when you have
auto-resume=no. Notice that NcFTP has detected that the file sizes are different and prompts for what action to take. In this example, I typed
Resume (R) to resume the transfer.
Listing 7. Resume transfer
ncftp /ftproot/stats > put stat_jan The remote file "stat_jan" already exists. Local: 102400000 bytes, dated Tue Jan 19 19:19:19 GMT 2010. Remote: 57347024 bytes, dated Tue Jan 19 19:24:28 GMT 2010. [O]verwrite? [R]esume? [A]ppend to? [S]kip? [N]ew Name? [O!]verwrite all? [R!]esume all? [S!]kip all? [C]ancel > R stat_jan: 97.66 MB 938.15 kB/s
If you try to FTP a file and it has the same size and file name on both the local and remote side, NcFTP will issue the following message:
ncftp /ftproot/stats > put stat_jan The remote file "stat_jan" already exists. Local: 102400000 bytes, dated Tue Jan 19 19:19:19 GMT 2010. (Files are identical, skipped)
Files that are uploaded or downloaded can be renamed after the transfer as long as the corresponding FTP server supports the rename function. To rename a file use the
-z option. The basic format is:
<get / put> -z <file to transfer> <new_filename>
To pull the file stat_jan from a remote server and rename it to stat_jan_rs6.txt, you could use:
ncftp /ftproot >get -z stat_jan stat_jan_rs6.txt stat_jan_rs6.txt: ETA: 2:11 19.77/ 97.66 MB 610.62 kB/s
Notice in this example that a status line of the transfer is produced. This lets you know approximately how long the transfer will take. A status line is produced for any transfer, whether it's a
get command. A similar status line is produced when using ncftpput and ncftpget.
While using NcFTP is convenient when you are at the keyboard, it may not be as useful when you want to issue noninteractive one-liners, especially when you need to script the FTP sessions. However, the utilities ncftpls, ncftpput, and ncftpget provide the necessary tools for this job.
Using command lines
The three clients ncftpls, ncftpget, and ncftpput will undoubtedly be used more than NcFTP. These utilities are very useful when you need to batch or script ad hoc transfers. You cannot use bookmarks when using these utilities.
The basic format of ncftpls is:
ncftpls <authentication details /options> <-x ls flags> <ftp://remote host>
Using ncftpls lets you connect to the remote server and extract a command line
ls listing. This utility has two advantages:
- You can execute this utility prior to an FTP session to ensure the remote host FTP service is running before performing an upload or download.
- You can also test whether the file or directory on the remote host is present.
In the following example, a connection is made to the remote host tardis and to the users HOME directory using /./. Then the command
ls pre_stop.sh is issued to see if the file pre_stop.sh is present.
$ ncftpls -u dxtans -p master "-x ls pre_stop.sh" ftp://tardis/./ -rwxrwxrwx 1 apps staff 14 Aug 25 08:51 pre_stop.sh
If you are using ncftpls in a script, you will probably need to capture the output. This is because as long as the connection is successful, the last exit status will be zero. It will not return an error to standard output if the file is not found. One way to test the return of the command is to use substitution, like shown in Listing 8:
Listing 8. Using substitution
myfile=`ncftpls -u dxtans -p master "-x ls pre_stop.sh" ftp://tardis/./ if [ "$myfile" != "" ] then echo "file present" # do processing here else echo "no file present" exit 1 fi
Full directory listings can also be executed. In the next example, a connection is made to the remote host tardis, and a directory listing is executed on the remote directory <HOME>/gems. Notice the use of the
-1 flag. This ensures that you get just the file names, with none of the file name's other attributes:
$ ncftpls -u dxtans -p master -1 ftp://tardis/./gems/ shrm.sh lgett.sh myfile.txt
When dealing with busy networks, it can take some time before a connection is made. To ensure your FTP session does not time out prior to this, make NcFTP retry the connection using the redial (r) and timeout (t) values, where:
r is the < number of tries>
t is the <timeout in seconds>
Listing 9 shows a partial script example in which a connection is made to the remote host tardis. The connection is tried (redial) twice, and within each redial it tries to connect for ten seconds before timing out. If it fails to connect, then the script will either exit or continue processing the rest of the script.
Listing 9. Connection retries
log=mylog ncftpls –r2 -t10 -u dxtans -p master ftp://tardis/./ if [ $? -ne 0 ] then echo "Could not connect to FTP server" >>$log exit 1 else echo "NCFTP connect test OK" >>$log fi # carry on ncftp commands here..
Ncftpput is used to put the files or directories onto the remote host. The basic format of ncftpput is:
ncftpput <authentication details/options> <remote host> <remote direc> <local files>
The following example puts the local file /home/dxtans/spec_ja onto the remote host tardis, in the remote directory /ftproot/stats.
$ ncftpput -u dxtans -p master tardis /ftproot/stats /home/dxtans/spec_ ja/home/dxtans/spec_ja: 564.00 B 3.26 kB/s
Ncftpget is used to get files or directories from the remote host. The basic format of ncftpget is:
ncftpget <authentication details/options> <remote host> <local files> <remote direc>
The following example gets the remote file peb_aud.rtf from the remote host tardis located in the remote directory /ftproot/stats/pebs/peb_aud, and places it in the local directory /tmp.
ncftpget -u dxtans -p master tardis /tmp /ftproot/stats/pebs/peb_aud* /home/dxtans/holding/peb_aud.rtf: 18.48 kB 102.92 kB/s
Notice in the previous example I used the * wildcard character to download any file matching peb_aud*.
With ncftpput and ncftpget you can also do recursive transfers. However, not all FTP servers support this, especially public FTP servers. To use recursive transfer, supply the
-R option. Listing 10 shows recursive transfers from the remote host tardis in the remote directory /ftproot/stats/pebs into the local directory /home/dxtans/hold. The local directory hold is created if it does not exist:
Listing 10. Recursive transfers
$ ncftpget -R -u dxtans -p master tardis /home/dxtans/hold/ /ftproot/stats/pebs /home/dxtans/hold/pebs/peb_post2.rtf: 18.48 kB 101.53 kB/s /home/dxtans/hold/pebs/peb_pre.txt: 0.00 B 0.00 B/s /home/dxtans/hold/pebs/peb_runin.rtf: 18.48 kB 101.09 kB/s /home/dxtans/hold/pebs/peb_runoff.rtf: 18.48 kB 103.80 kB/s /home/dxtans/hold/pebs/pre_post.txt: 0.00 B 0.00 B/s /home/dxtans/hold/pebs/mth/prepost.txt: 23.01 B 108.00 kB/s /home/dxtans/hold/pebs/mth/pre_run2.txt: 22.01 B 106.00 kB/s
You may want to turn the status bar off at times; to do this, use the
-V option and supply it along with the other options. For example:
$ ncftpget -V -R -u dxtans -p master tardis /home/dxtans/hold/ /ftproot/stats/pebs
You can use the ncftpbatch command to create many FTP jobs to run in a queue or to submit many jobs. In a nutshell, it works like this:
- Submit an FTP command using the ncftpput or ncftpget command.
- Specify the
-bboption to run the jobs in the background. These jobs are then sent to the spooler.
- Use Ncftpbatch to process the FTP jobs that are in the spooler.
To start the batch in the background, use:
To view jobs in the spooler, use:
All batch related jobs go to the spool directory:
Within that directory is a spool log file. Also contained in that directory are files for each spooled FTP job.
Listing 11 submits all files that end with a .rtf extension in the current directory. These files are sent to the remote server tardis in the remote directory /ftproot/process using the ncftpput command:
Listing 11. Submitting .rtf files
$ ncftpput -bb -u dxtans -p master tardis /ftproot/process *.rtf Spooled; sending remotely as /ftproot/process/peb_post2.rtf. Spooled; sending remotely as /ftproot/process/peb_pre.rtf. Spooled; sending remotely as /ftproot/process/stat_jan.rtf.
To start ncftpbatch in the background to process the jobs, use:
$ ncftpbatch –d
To view the spooler to see what jobs are being processed, use:
$ ncftpbatch -l ---Scheduled-For-----Host----------------------------Command-------------------- 2010-01-23 21:06 tardis PUT stat_jan.rtf
The jobs will be processed, and after job completion the spooler will be empty. To check on the activity of the spooler and jobs, check the log file .ncftp/spool/log.
The process of submitting many FTP jobs to a queue does save time because you do not have to wait for the previous job to complete before committing to the next job. However, please be aware that if you have any issues in your NcFTP batch-submitted commands, such as authentication or wrong
get/put commands, the jobs will just sit there. The path of least resistance is to just remove the jobs, which are located in the .nctp/spoolwith directory, using the
rm command, and then resubmit them.
In this article, I supplied the password on the command line; in normal operations you would not do this. There are a couple of ways you would normally enter the password. You can wait to be prompted for the password at the remote host, by specifying only a username on the command line on the local side. Alternatively, you can put the password in a file and source it into the variable. For the NcFTP utilities ncftpput and ncftpget, you can use a password authentication file. This is invoked by using the
-f <authentication file> option. The authentication file should be readable only by the user. The format for the file is:
host < hostname> user <user name> pass <user password>
A sample file could contain the following authentication details:
$ cat /home/dxtans/myncftp host tardis user dxtans pass master
To use the file to connect to a remote host, you could issue ncftpput with the following example. The code will put the local file /home/dxtans/chatter.log into the remote directory /ftproot/gen:
$ ncftpput -f /home/dxtans/myncftp /ftproot/gen /home/dxtans/chatter.log
On an earlier version and on a different operating system, I discovered that you may have to specify the remote host on the command line, like so:
$ ncftpput -f /home/dxtans/myncftp tardis /ftproot/gen /home/dxtans/chatter.log
If a remote host supports anonymous FTP logins, you can hold the password, which is usually your e-mail address, in the preferences file. To set up NcFTP to parse the password for an anonymous user, you can use the NcFTP command line by setting the
anon-password value, such as:
ncftp> set anon-password firstname.lastname@example.org
Alternatively, you can directly edit the prefs_v3 file:
To connect to the remote host tardis as an anonymous user, you can use:
$ ncftp -u anonymous tardis NcFTP 3.2.3 (Jul 28, 2009) by Mike Gleason (http://www.NcFTP.com/contact/). Connecting to 192.168.4.10... Logging in... Password Ok, User logged in Logged in to tardis
When dealing with authentication or file placement issues on a remote host, it is a good practice to debug the FTP session. Using the
-d <logfile> option, NcFTP will send debugging information to the file's log file. In the following example, the debug file is called output.log, which is created in the current working directory. The local file /home/dxtans/gen_rep is transferred onto the remote machine in the /ftproot directory. You can see the complete transaction process by viewing the file's log file if you need to diagnose any issues. Please note that information given in the debug file is very verbose and detailed, which is what you would want if you are trying to solve any problems.
$ ncftpput -d output.log -u dxtans -p master tardis /ftproot /home/dxtans/gen_prep /home/dxtans/gen_prep: 573.00 B 6.71 kB/s
The debug option can also be used with ncftpget and ncftpls.
The NcFTP client is an efficient method of FTPing. With NcFTP, you can run commands from the command line, which makes it ideal for scripting, and NcFTP can also resume a failed FTP transfer, as well as provide batch processing facilities. This article described how the NcFTP client is very useful in ad hoc FTP transfers and explained how to use the client utilities within scripts
- NcFTP software provides information on NcFTP as well as downloads.
- Vist the AIX Open Source page to get NcFTP rpm and tarball downloads.