FTPMIRROR

Introduction

Ftpmirror is an utility to copy directory hierarchy (this is called ``mirror'') with FTP. A similar perl script exists, whose name is `mirror', but ftpmirror requires less memory than mirror. You need perl-5.003 or later to use ftpmirror. Please ensure that perl-5.003 or later was installed in your environment, at first.

How to use ftpmirror

  1. Install ftpmirror first. The newest version of ftpmirror is available from ftp.intec.co.jp. Since ftpmirror is a set of perl scripts, you must only extract the packages. Do one of following: 
     [ if you have ftpmirror-x.y.tar.gz ]
 # cd /usr/local/lib
 # gzip -cd < /tmp/ftpmirror-x.y.tar.gz | tar xf -
    or 
     [ if you have ftpmirror-x.y.shar ]
 # cd /usr/local/lib
 # sh /tmp/ftpmirror-x.y.shar
  2. $prefix defined in ftpmirror must be configured as the install directory. The default is /usr/local/lib/ftpmirror. 
     # cd /usr/local/lib/ftpmirror
 # vi +/prefix/ ftpmirror
 ...
  3. Create default configuration file in the "config" directory (Please refer the "default-sample" file). This file contains default values described in Parameters for ftpmirror section. 
     # cd /usr/local/lib/ftpmirror/config
 # cp default-sample default
 # vi default
 ...
  4. Create package configuration files as you need. You may define multiple packages in the default configuration file. For more details, please refer the package section. The simplest way to configure a package is defining three entry, ftp-server, remote-directory and local-directory. 
     # cp GNU-sample GNU
 # vi GNU
 ...
  5. Invoke ftpmirror with the package name. Ftpmirror reads the configuration file for the package if there is a file, whose name is same as the package name, in the current directory or the "config" directory, 
     # /usr/local/lib/ftpmirror/ftpmirror FreeBSD
  6. Any information about transfer is written to STDOUT.

Parameters for ftpmirror

You can define parameters for ftpmirror with the default configuration file, package specific configuration file, and run-time options. Ftpmirror overwrites parameters in order above, e.g., run-time options are most preferred.

There are many acceptable parameters as follows:

Parameters from run-time options

You can specify parameters by command line options. For example, if you want to invoke ftpmirror with test mode (no actual transfer), you can type: 
 # ftpmirror --test-mode=1 FreeBSD
As in this example, any parameter must be prepended by "--" in command line options.

Parameters are configured as follows:

  1. Parameters defined in command line option are most preferred.
  2. If not defined in command line option, parameters in package specific configuration file is used.
  3. If defined in neither command line option nor package specific configuration file, ftpmirror activates parameters in the default configuration file.
  4. Otherwise, ftpmirror's default value (defined in param.pl) is used.

About directory information

Ftpmirror-1.2 or later can generate or use directory information. Directory information is stored in ".dirinfo" file in each directory, and used by ftpmirror in directory compare stage. Using directory information, you can mirror directory hierarchy more efficiently, and more completely.

For example, directory information contains following entries: 

 file = LICENSE
  md5checksum = 83cdabcad3c9c3480faa6a75c47c7207
  modified = 800142639
  size = 4631
  owner = root
  group = wheel
 directory = OLD
  begin-update = 842336105
  end-update = 842336105
  owner = root
  group = wheel
 symlink = Welcome.html
  linkto = ../index.html
  owner = root
  group = wheel
 directory = archive
  begin-update = 842336105
  end-update = 842336105
  owner = root
  group = wheel
 file = index
  md5checksum = b5aeac559ee486a0f12493ce52f216f9
  modified = 801170780
  size = 799
  owner = root
  group = wheel
 ....
In directory information, any file must be defined in a file-type entry. Currently, there are three file-types:
  1. file = file-name ... a normal file
  2. symlink = link-file ... a symbolic link
  3. directory = dir-name ... a directory
and, each file-type is followed by some parameters:
  1. file file-type
  2. symlink file-type
  3. directory file-type
Ftpmirror tries to compare files in local and remote with these entries. For example, ftpmirror compares as:
  1. file file-type
    Are sizes same? and are md5checksums same?
  2. symlink file-type
    Do they point same file?
  3. directory file-type
    Is the begin-time of last update of local directory is before the end-time of last update of remote directory?
Especially, comparing directory update times are effective, if no file is changed in the remote directory, because ftpmirror do nothing about the directory (even directory scan is skipped).

Ftpmirror generate ".dirinfo" file if store-local-dirinfo parameter is 1. You can also generate this file with mkdirinfo: 

 # mkdirinfo /ftp/pub/utils

mirror.sh - auto mirroring script

Ftpmirror package contains shell script mirror.sh. This is a script to mirror automatically(e.g., by using cron). The author mirrors by this script, daily, weekly and monthly. Here is a simple way to use mirror.sh
  1. Create list.xxx in the directory defined as $prefix. This file contains a list of package names you want to mirror. An empty line or a line begins with # is ignored.
  2. Invoke mirror.sh. If you create $prefix/list.daily in adobe stage, please type: 
      # /usr/local/lib/ftpmirror/mirror.sh daily
    and, you can mirror all packages in the list file.
  3. By default, any output is logged to /var/log/mirror.xxx if you use mirror.sh. Mirror.sh rotate these log files with rotate. So you should install this package, or comment out some potion calling rotate from mirror.sh.

Notice

Ftpmirror uses STAT or LIST, and SIZE/MDTM commands to compare files in local and remote directory. Since LIST command requires new TCP connection per command, it may take many time if network is so slow. Using ls-lR files (or directory information, if server supports it) may be efficient, if you must mirror on a slow network.

If clocks of FTP server and local machine are different too much, ftpmirror may fail to transfer files. Please keep your clock of your machine correctly. NTP is a good way to keep your clock right.

Mailing list for ftpmirror

Technical questions, suggestions, bug reports are welcome. If you are interesting to ftpmirror, there is a mailing-list to discuss about ftpmirror. To subscribe, send a message 
 subscribe ftpmirror
to majordomo@ftp.intec.co.jp.

Credits

I would like to thank to anyone who helped me by testing, bug-reports, suggestions, sending patches. Especially, members in RingProject helped me by heavy testing :-) and many suggestions.

Miscellaneous...

Please read COPYRIGHT for copyright notice. Any comments, bug fixes, suggestions are welcome. Please contact to Ikuo Nakagawa <ikuo@intec.co.jp>.
Last updated: 1997/01/27 by Ikuo Nakagawa