Using LFTP to Mirror a Web Repo

While in the process of researching the best way to create my own local repository of Solaris packages, I ran across LFTP which is a ftp/http client with I built-in mirror command.

For more information on LFTP, look here or  here

The repo that I wanted to Mirror was Blastwave.org, so the first thing that I did was mirror thir directory structure on my local server. I created a 100 gb mount on /v1, and then created the following directories (these are the only ones that I need). They pretty much match the structure that blastwave is using.

/v1/repo/stable/sparc/5.8
/v1/repo/stable/sparc/5.10
/v1/repo/stable/i386/5.10

Here is an example of what I ran to create the initial x86 Solaris 10 repo.

lftp -c ‘open -e "mirror /csw/stable/i386/5.10/ /v1/repo/stable/5.10/" http://blastwave.network.com’

Which is based on the basic structure outlined below

lftp -c ‘open -e "mirror <remote_dir_path> <local_dir>/" <remote_site>’

Below are a couple of the options that you can use with LFTP

In the example below, LFTP removes any local packages that no longer exist on the source, and only downloads newer package versions from the host mirror.

lftp -e "mirror –delete –only-newer

You can also use the -f option to read your settings from a file.

lftp -f /scriptfile

Or you can log to a file.
–log=FILE      write lftp commands being executed to FILE

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s