[nycbug-talk] Off topic: Best way to mirror large ftp

[Matt Juszczak]

I have about 3 TB of data I need to mirror off of an FTP box.  Using 
traditional methods, it would take me about 16+ days to get all of that 
information.

I've looked at things like lftp, and a few other "scripts" out there, but 
ideally I would love to find something that can:

1) Index the entire FTP
2) Split the downloads into multiple threads
3) Update the index at any time (the FTP server changes) and download the 
differences (yes, this may be an expensive operation I know)

Any suggestions?  Off topic I know, but I've been struggling for some time 
now on this issue and I'm hoping some of you fellow sysadmins have some 
suggestions.

Thanks!

-Matt