[nycbug-talk] Off topic: Best way to mirror large ftp

Matt Juszczak matt at atopia.net
Wed Nov 26 11:12:06 EST 2008

I have about 3 TB of data I need to mirror off of an FTP box.  Using 
traditional methods, it would take me about 16+ days to get all of that 

I've looked at things like lftp, and a few other "scripts" out there, but 
ideally I would love to find something that can:

1) Index the entire FTP
2) Split the downloads into multiple threads
3) Update the index at any time (the FTP server changes) and download the 
differences (yes, this may be an expensive operation I know)

Any suggestions?  Off topic I know, but I've been struggling for some time 
now on this issue and I'm hoping some of you fellow sysadmins have some 



More information about the talk mailing list