NcFTP is a product that comes as either an FTP server or FTP client. However, both utilities are referenced as NcFTP. In this article, I only use the NcFTP client. Therefore, when I mention NcFTP I am referring to just the NcFTP client. Additionally, various command-line utilities come with the client, which makes the client very attractive for use on the command line or in batch mode.

NcFTP overview

NcFTP utilities let system administers provide FTP flexibility and performance. NcFTP can be used with the current AIX® default FTP service with no known issues. NcFTP offers the ability to resume failed transfers and retries, which is very useful when transferring large files across the network or across networks that suffer from bottlenecks. NcFTP can output a status bar detailing the progress of the transfer (much like SCP). However, NcFTP does not support any encryption, which might be an issue for some installations.

It's very common for installations to have a current AIX FTP server running while using NcFTP as an ad hoc client, and this type of installation is what this article focuses on.

The NcFTP client comes with the following utilities:

Note: In this article, I will not be discussing the ncftpspooler utility.

Back to top

Installing NcFTP

NcFTP can be downloaded from the NcFTP home page (see Resources for a link).

After you have downloaded the client, extract the binaries, as shown in Listing 1:

# gunzip /tmp/ncftp-3.2.3-aix5.3.0-export.tar.gz
# cd /opt
# tar –xvf  /tmp/ncftp-3.2.3-aix5.3.0-export.tar

Check to ensure that NcFTP starts, as shown in Listing 2.

$ ncftp -v
$ Usage:  ncftp [flags] [<host> | <directory URL to browse>]
Flags:
  -u XX  Use username XX instead of anonymous.
  -p XX  Use password XX with the username.
  -P XX  Use port number XX instead of the default FTP service port (21).
  -j XX  Use account XX with the username (rarely needed).
  -F     Dump a sample $HOME/.ncftp/firewall prefs file to stdout and exit.

Program version:  NcFTP 3.2.3/433 Jul 28 2009, 10:02 AM
Library version:  LibNcFTP 3.2.3 (July 23, 2009)
Build system:     AIX Voltigeur 3 5 000D3A9F4C00

Depending on the type of remote FTP host you are accessing, you may have to prefix the hostname with ftp://, such as:

ftp://<hostname>

Note: For the ncftpls utility, you must use the previous format when connecting to the remote host.

If you experience connection issues with FTP, use this checklist to try and troubleshoot your problem.

1. Check that the remote FTP service is running.
2. Check that the remote FTP port is not blocked.
3. Check to ensure that the tcp wrappers are not denying the connection.
4. Check that your user name is not in /etc/ftpusers.
5. Check that the host or user is not being denied in /etc/ftpaccess.ctl.

Back to top

Using NcFTP interactively

Confirm that you can connect to a remote server of your choice using NcFTP. In Listing 3, I connect to the private host tardis, using the user name of dxtans and master as the password.

$ ncftp -u dxtans -p master tardis
NcFTP 3.2.3 (Jul 28, 2009) by Mike Gleason (http://www.NcFTP.com/contact/).
Connecting to 192.168.4.10...
Logging in...
Password Ok, User logged in
Logged in to tardis.
Current remote directory is /.
ncftp/>

If all looks good, let's explore NcFTP.

Listing 4 demonstrates how NcFTP is used in a common FTP session. A cd command is used to change to /tmp on the local side, then to the /ftproot/stats directory on the remote side.

A couple of ls commands are executed before a get command is issued to retrieve the remote file stat_09.txt. It is very similar to a normal FTP session, but there are extra features, as you will see.

$ ncftp -u dxtans -p master tardis
NcFTP 3.2.3 (Jul 28, 2009) by Mike Gleason (http://www.NcFTP.com/contact/).
Connecting to 192.168.4.10...
Logging in...
Password Ok, User logged in
Logged in to tardis.
ncftp / > lcd /tmp
ncftp / > cd /ftproot/stats
Change directory ok
ncftp /ftproot/stats > ls
desktop.ini    pebs/          spec_ja        stat_09.txt    stat_jan.rtf
ncftp /ftproot/stats > ls -l
-------rw-           48   Jan 18 13:58   desktop.ini
d------rwx            0   Jan 23 02:36   pebs
-------rw-          564   Jan 23 08:17   spec_ja
-------rw-           61   Jan 18 13:53   stat_09.txt
-------rw-        18927   Mar 23  2004   stat_jan.rtf
ncftp /ftproot/stats > get stat_09.txt
        

When you initially invoke NcFTP, it creates a .ncftp directory in your HOME directory. From here you can personalize your NcFTP sessions, including any proxy settings. The most amended file will be the prefs_v3 file. The content of the prefs_v3 file is pretty self-explanatory. After you view it, you can edit the file to tailor it for your personal FTP sessions. Each entry has the format:

key=value

For example, to set the connection timeout I could use:

connect-timeout=20

The preferences can also be set on the NcFTP command line as well, using the set command:

ncftp/> set connect-time 20

Frequent FTP destinations can be saved as bookmarks, as demonstrated in Listing 5.

Looking more closely at Listing 5, you can see that a connection is made to the remote host tardis, specifying the destination directory as /ftproot/stats. The FTP session is then terminated. As this is a new remote connection to a remote directory, NcFTP asks if you want to save this connection and password. I typed yes, and called the saved session tardis_stats. The bookmark entry is saved to the bookmarks file in the .ncftp directory.

$ ncftp -u dxtans -p master tardis /ftproot/stats
NcFTP 3.2.3 (Jul 28, 2009) by Mike Gleason (http://www.NcFTP.com/contact/).
Connecting to 192.168.4.10...
Logging in...
Password Ok, User logged in
Logged in to tardis.
Current remote directory is /ftproot/stats.
ncftp /ftproot/stats > quit
You have not saved a bookmark for this site.
Would you like to save a bookmark to:
        ftp://dxtans:PASSWORD@tardis/ftproot/stats/
Save? (yes/no) yes
Enter a name for this bookmark, or hit enter for "tardis": tardis_stats
You logged into this site using a password.
Would you like to save the password with this bookmark?
Save? [no] yes
Bookmark "tardis_stats" saved
        

The next time I want to use this remote connection I only need to supply the bookmark as part of the ncftp parameter, as shown in Listing 6.

If you do not want to see a bookmark prompt every time you go to a new remote destination, set the confirm-close value to no in the prefs_v3 file:

confirm-close=no

Or, from the NcFTP command line, enter the following command:

ncftp/>set confirm-close no

                
$ ncftp tardis_stats
NcFTP 3.2.3 (Jul 28, 2009) by Mike Gleason (http://www.NcFTP.com/contact/).
Connecting to 192.168.4.10...
Logging in...
Password Ok, User logged in
Logged in to tardis.
Change directory ok
Current remote directory is /ftproot/stats.
ncftp /ftproot/stats >
        

Over time, you will undoubtedly get a fairly populated bookmarks file, which may require some housekeeping. To edit this file, you can use the NcFTP utility ncftpbookmarks, which is a menu-type interface, but is not very intuitive. I recommend simply editing the bookmarks file using vi, and change as needed.

When transferring large files or transferring files over networks that suffer from outages, it is inevitable that at some point you will get a failed FTP, with the file partially transferred. Normally for an FTP, this would mean resubmitting the job and overwriting the original transfer. However, with NcFTP you can resume the session by either appending or overwriting. When uploading or downloading a file that already exists, but with different file sizes, NcFTP will try and resume the session where it stopped, without prompting. If you want to be prompted on what action NcFTP should take, change the auto-resume preference to no:

auto-resume=no

Listing 7 shows the action you can take after a failed partial FTP transfer when you have auto-resume=no. Notice that NcFTP has detected that the file sizes are different and prompts for what action to take. In this example, I typed Resume (R) to resume the transfer.

ncftp /ftproot/stats > put stat_jan

The remote file "stat_jan" already exists.
        Local:     102400000 bytes, dated Tue Jan 19 19:19:19 GMT 2010.
        Remote:     57347024 bytes, dated Tue Jan 19 19:24:28 GMT 2010.

        [O]verwrite?  [R]esume?  [A]ppend to?  [S]kip?  [N]ew Name?
        [O!]verwrite all?  [R!]esume all?  [S!]kip all?  [C]ancel  > R
        stat_jan:                                               97.66 MB  938.15 kB/s
        

If you try to FTP a file and it has the same size and file name on both the local and remote side, NcFTP will issue the following message:

 ncftp /ftproot/stats > put stat_jan

The remote file "stat_jan" already exists.
        Local:     102400000 bytes, dated Tue Jan 19 19:19:19 GMT 2010.
        (Files are identical, skipped)

Files that are uploaded or downloaded can be renamed after the transfer as long as the corresponding FTP server supports the rename function. To rename a file use the -z option. The basic format is:

<get / put> -z <file to transfer>  <new_filename>

To pull the file stat_jan from a remote server and rename it to stat_jan_rs6.txt, you could use:

ncftp /ftproot >get -z stat_jan stat_jan_rs6.txt
stat_jan_rs6.txt:                 ETA:   2:11   19.77/ 97.66 MB  610.62 kB/s

Notice in this example that a status line of the transfer is produced. This lets you know approximately how long the transfer will take. A status line is produced for any transfer, whether it's a put or get command. A similar status line is produced when using ncftpput and ncftpget.

While using NcFTP is convenient when you are at the keyboard, it may not be as useful when you want to issue noninteractive one-liners, especially when you need to script the FTP sessions. However, the utilities ncftpls, ncftpput, and ncftpget provide the necessary tools for this job.

Back to top

Using command lines

The three clients ncftpls, ncftpget, and ncftpput will undoubtedly be used more than NcFTP. These utilities are very useful when you need to batch or script ad hoc transfers. You cannot use bookmarks when using these utilities.

The basic format of ncftpls is:

ncftpls  <authentication details  /options>  <-x ls flags> <ftp://remote host>

Using ncftpls lets you connect to the remote server and extract a command line ls listing. This utility has two advantages:

• You can execute this utility prior to an FTP session to ensure the remote host FTP service is running before performing an upload or download.
• You can also test whether the file or directory on the remote host is present.

In the following example, a connection is made to the remote host tardis and to the users HOME directory using /./. Then the command ls pre_stop.sh is issued to see if the file pre_stop.sh is present.

$ ncftpls -u dxtans -p master "-x ls pre_stop.sh" ftp://tardis/./
-rwxrwxrwx    1 apps      staff            14 Aug 25 08:51 pre_stop.sh

If you are using ncftpls in a script, you will probably need to capture the output. This is because as long as the connection is successful, the last exit status will be zero. It will not return an error to standard output if the file is not found. One way to test the return of the command is to use substitution, like shown in Listing 8:

myfile=`ncftpls -u dxtans -p master "-x ls pre_stop.sh" ftp://tardis/./
if [ "$myfile" != "" ]
 then
 echo "file present"
# do processing here
else
 echo "no file present"
  exit 1
fi

Full directory listings can also be executed. In the next example, a connection is made to the remote host tardis, and a directory listing is executed on the remote directory <HOME>/gems. Notice the use of the -1 flag. This ensures that you get just the file names, with none of the file name's other attributes:

$ ncftpls -u dxtans -p master -1 ftp://tardis/./gems/
shrm.sh
lgett.sh
myfile.txt

When dealing with busy networks, it can take some time before a connection is made. To ensure your FTP session does not time out prior to this, make NcFTP retry the connection using the redial (r) and timeout (t) values, where:

r is the < number of tries>

t is the <timeout in seconds>

Listing 9 shows a partial script example in which a connection is made to the remote host tardis. The connection is tried (redial) twice, and within each redial it tries to connect for ten seconds before timing out. If it fails to connect, then the script will either exit or continue processing the rest of the script.

log=mylog
ncftpls –r2 -t10 -u dxtans -p master ftp://tardis/./
if [ $? -ne 0 ]
 then
 echo "Could not connect to FTP server" >>$log
 exit 1
 else
 echo "NCFTP connect test OK" >>$log
fi
# carry on ncftp commands here..

Ncftpput is used to put the files or directories onto the remote host. The basic format of ncftpput is:

ncftpput <authentication details/options>  <remote host> <remote direc>  <local files>

The following example puts the local file /home/dxtans/spec_ja onto the remote host tardis, in the remote directory /ftproot/stats.

 $ ncftpput -u dxtans -p master tardis /ftproot/stats /home/dxtans/spec_
ja/home/dxtans/spec_ja:                                  564.00 B     3.26 kB/s

Ncftpget is used to get files or directories from the remote host. The basic format of ncftpget is:

ncftpget <authentication details/options> <remote host> <local files> <remote direc>

The following example gets the remote file peb_aud.rtf from the remote host tardis located in the remote directory /ftproot/stats/pebs/peb_aud, and places it in the local directory /tmp.

ncftpget -u dxtans -p master tardis /tmp /ftproot/stats/pebs/peb_aud*
/home/dxtans/holding/peb_aud.rtf:                       18.48 kB  102.92 kB/s

Notice in the previous example I used the * wildcard character to download any file matching peb_aud*.

With ncftpput and ncftpget you can also do recursive transfers. However, not all FTP servers support this, especially public FTP servers. To use recursive transfer, supply the -R option. Listing 10 shows recursive transfers from the remote host tardis in the remote directory /ftproot/stats/pebs into the local directory /home/dxtans/hold. The local directory hold is created if it does not exist:

$ ncftpget -R -u dxtans -p master tardis /home/dxtans/hold/ /ftproot/stats/pebs
/home/dxtans/hold/pebs/peb_post2.rtf:                   18.48 kB  101.53 kB/s
/home/dxtans/hold/pebs/peb_pre.txt:                      0.00 B     0.00 B/s
/home/dxtans/hold/pebs/peb_runin.rtf:                   18.48 kB  101.09 kB/s
/home/dxtans/hold/pebs/peb_runoff.rtf:                  18.48 kB  103.80 kB/s
/home/dxtans/hold/pebs/pre_post.txt:                     0.00 B     0.00 B/s
/home/dxtans/hold/pebs/mth/prepost.txt:                  23.01 B  108.00 kB/s
/home/dxtans/hold/pebs/mth/pre_run2.txt:                 22.01 B  106.00 kB/s

You may want to turn the status bar off at times; to do this, use the -V option and supply it along with the other options. For example:

$ ncftpget -V -R -u dxtans -p master tardis /home/dxtans/hold/ /ftproot/stats/pebs

Back to top

Batch processing

You can use the ncftpbatch command to create many FTP jobs to run in a queue or to submit many jobs. In a nutshell, it works like this:

1. Submit an FTP command using the ncftpput or ncftpget command.
2. Specify the -bb option to run the jobs in the background. These jobs are then sent to the spooler.
3. Use Ncftpbatch to process the FTP jobs that are in the spooler.

To start the batch in the background, use:

     
ncftpbatch -d 

To view jobs in the spooler, use:

ncftpbatch -l
        

All batch related jobs go to the spool directory:

  
.ncftp/spool
        

Within that directory is a spool log file. Also contained in that directory are files for each spooled FTP job.

Listing 11 submits all files that end with a .rtf extension in the current directory. These files are sent to the remote server tardis in the remote directory /ftproot/process using the ncftpput command:

$ ncftpput  -bb -u dxtans -p master tardis /ftproot/process *.rtf
Spooled; sending remotely as /ftproot/process/peb_post2.rtf.
Spooled; sending remotely as /ftproot/process/peb_pre.rtf.
Spooled; sending remotely as /ftproot/process/stat_jan.rtf.

To start ncftpbatch in the background to process the jobs, use:

$ ncftpbatch –d
        

To view the spooler to see what jobs are being processed, use:

$ ncftpbatch -l
---Scheduled-For-----Host----------------------------Command--------------------
   2010-01-23 21:06  tardis                          PUT stat_jan.rtf
        

The jobs will be processed, and after job completion the spooler will be empty. To check on the activity of the spooler and jobs, check the log file .ncftp/spool/log.

The process of submitting many FTP jobs to a queue does save time because you do not have to wait for the previous job to complete before committing to the next job. However, please be aware that if you have any issues in your NcFTP batch-submitted commands, such as authentication or wrong get/put commands, the jobs will just sit there. The path of least resistance is to just remove the jobs, which are located in the .nctp/spoolwith directory, using the rm command, and then resubmit them.

Back to top

Passwords

In this article, I supplied the password on the command line; in normal operations you would not do this. There are a couple of ways you would normally enter the password. You can wait to be prompted for the password at the remote host, by specifying only a username on the command line on the local side. Alternatively, you can put the password in a file and source it into the variable. For the NcFTP utilities ncftpput and ncftpget, you can use a password authentication file. This is invoked by using the -f <authentication file> option. The authentication file should be readable only by the user. The format for the file is:

host < hostname>
user <user name>
pass <user password>

A sample file could contain the following authentication details:

$ cat /home/dxtans/myncftp
host tardis
user dxtans
pass master

To use the file to connect to a remote host, you could issue ncftpput with the following example. The code will put the local file /home/dxtans/chatter.log into the remote directory /ftproot/gen:

$ ncftpput -f /home/dxtans/myncftp  /ftproot/gen  /home/dxtans/chatter.log

On an earlier version and on a different operating system, I discovered that you may have to specify the remote host on the command line, like so:

$ ncftpput -f /home/dxtans/myncftp  tardis /ftproot/gen  /home/dxtans/chatter.log

If a remote host supports anonymous FTP logins, you can hold the password, which is usually your e-mail address, in the preferences file. To set up NcFTP to parse the password for an anonymous user, you can use the NcFTP command line by setting the anon-password value, such as:

ncftp> set anon-password david.tansley@btinternet.com

Alternatively, you can directly edit the prefs_v3 file:

 anon-password=david.tansley@btinternet.com

To connect to the remote host tardis as an anonymous user, you can use:

$  ncftp  -u anonymous tardis
NcFTP 3.2.3 (Jul 28, 2009) by Mike Gleason (http://www.NcFTP.com/contact/).
Connecting to 192.168.4.10...
Logging in...
Password Ok, User logged in
Logged in to tardis

Back to top

Debugging

When dealing with authentication or file placement issues on a remote host, it is a good practice to debug the FTP session. Using the -d <logfile> option, NcFTP will send debugging information to the file's log file. In the following example, the debug file is called output.log, which is created in the current working directory. The local file /home/dxtans/gen_rep is transferred onto the remote machine in the /ftproot directory. You can see the complete transaction process by viewing the file's log file if you need to diagnose any issues. Please note that information given in the debug file is very verbose and detailed, which is what you would want if you are trying to solve any problems.

$ ncftpput -d output.log -u dxtans -p master tardis /ftproot /home/dxtans/gen_prep
/home/dxtans/gen_prep:                                 573.00 B     6.71 kB/s

The debug option can also be used with ncftpget and ncftpls.

Back to top

Conclusion

The NcFTP client is an efficient method of FTPing. With NcFTP, you can run commands from the command line, which makes it ideal for scripting, and NcFTP can also resume a failed FTP transfer, as well as provide batch processing facilities. This article described how the NcFTP client is very useful in ad hoc FTP transfers and explained how to use the client utilities within scripts

Resources

Get products and technologies

• NcFTP software provides information on NcFTP as well as downloads.
  
  
• Vist the AIX Open Source page to get NcFTP rpm and tarball downloads.
  
  

Discuss

• Follow developerWorks on Twitter.
  
  
• Get involved in the My developerWorks community.
  
  
• Participate in the AIX and UNIX forums:
  • AIX Forum
  • AIX Forum for developers
  • Cluster Systems Management
  • IBM Support Assistant Forum
  • Performance Tools Forum
  • Virtualization Forum
  • More AIX and UNIX Forums
  
  

About the author

David Tansley is a freelance writer. He has 15 years of experience as a UNIX administrator, using AIX the last eight years. He enjoys playing badminton, then relaxing watching Formula 1, but nothing beats riding and touring on his GSA motorbike with his wife.

Rate this article

Comments

Back to top

Table of contents

• NcFTP overview
• Installing NcFTP
• Using NcFTP interactively
• Using command lines
• Batch processing
• Passwords
• Debugging
• Conclusion
• Resources
• About the author
• Comments

Local resources

• IBM Innovation Center - Austin, TX
• IBM Innovation Center - Chicago, IL
• IBM Innovation Center - Dallas, TX
• IBM Innovation Center - San Mateo, CA
• IBM Innovation Center - Waltham, MA
• Technical Briefings in North America

Dig deeper into AIX and UNIX on developerWorks

Try IBM PureSystems. No charge.

Experience today!

---

Get started with the cloud-based trial and pattern development kit.

Special offers