[nycbug-talk] Off topic: Best way to mirror large ftp

[Okan Demirmen]

On Wed 2008.11.26 at 11:12 -0500, Matt Juszczak wrote:
> I have about 3 TB of data I need to mirror off of an FTP box.  Using 
> traditional methods, it would take me about 16+ days to get all of that 
> information.
> 
> I've looked at things like lftp, and a few other "scripts" out there, but 
> ideally I would love to find something that can:
> 
> 1) Index the entire FTP
> 2) Split the downloads into multiple threads
> 3) Update the index at any time (the FTP server changes) and download the 
> differences (yes, this may be an expensive operation I know)
> 
> Any suggestions?  Off topic I know, but I've been struggling for some time 
> now on this issue and I'm hoping some of you fellow sysadmins have some 
> suggestions.

mirror