[nycbug-talk] Off topic: Best way to mirror large ftp

Okan Demirmen okan at demirmen.com
Wed Nov 26 16:47:17 EST 2008

On Wed 2008.11.26 at 11:12 -0500, Matt Juszczak wrote:
> I have about 3 TB of data I need to mirror off of an FTP box.  Using 
> traditional methods, it would take me about 16+ days to get all of that 
> information.
> I've looked at things like lftp, and a few other "scripts" out there, but 
> ideally I would love to find something that can:
> 1) Index the entire FTP
> 2) Split the downloads into multiple threads
> 3) Update the index at any time (the FTP server changes) and download the 
> differences (yes, this may be an expensive operation I know)
> Any suggestions?  Off topic I know, but I've been struggling for some time 
> now on this issue and I'm hoping some of you fellow sysadmins have some 
> suggestions.


More information about the talk mailing list