[nycbug-talk] Off topic: Best way to mirror large ftp
Matt Juszczak
matt at atopia.net
Wed Nov 26 11:12:06 EST 2008
I have about 3 TB of data I need to mirror off of an FTP box. Using
traditional methods, it would take me about 16+ days to get all of that
information.
I've looked at things like lftp, and a few other "scripts" out there, but
ideally I would love to find something that can:
1) Index the entire FTP
2) Split the downloads into multiple threads
3) Update the index at any time (the FTP server changes) and download the
differences (yes, this may be an expensive operation I know)
Any suggestions? Off topic I know, but I've been struggling for some time
now on this issue and I'm hoping some of you fellow sysadmins have some
suggestions.
Thanks!
-Matt
More information about the talk
mailing list