[talk] Climate Mirror

[Thomas Levine]

Any form of cold storage would be great; we don't need to read from them
quickly, so the choice is mostly about cost. I thought twelve redundant
cheap/free hard drives might be cheaper or easier than like three
redundant tapes.

Since there will be like one file per disk, it could be neat to not use
a filesystem and instead develop some custom compression and error
correction method for the specific file formats. But I don't trust
anyone to implement that perfectly or to remember ten years from now how
that works, so I don't recommend it.

Regarding my speculation about the utility of this project: I will be
very disturbed if people haven't already been mirroring the data, so I
am puzzled as to why a potentially enormous change in government policy
should make the mirroring into a pressing issue.

On Wed, Dec 14, 2016, at 07:31 PM, Pete Wright wrote:
> 
> 
> On 12/14/16 11:28 AM, Isaac (.ike) Levy wrote:
> >
> >> On Dec 14, 2016, at 2:23 PM, Pete Wright <pete at nomadlogic.org> wrote:
> >>
> >>
> >>
> >> On 12/14/16 11:15 AM, Thomas Levine wrote:
> >>> The data don't need to be online; save them to a redundant bunch of
> >>> cheap hard drives (or maybe tapes), and distribute them among lots of
> >>> bookshelves. They can even be slow and small hard drives pulled from old
> >>> computers; we need to write to each one only once, we might need to read
> >>> from each one once, and we otherwise only need to turn them on once
> >>> every couple years to make sure that they're still intact. Maintain a
> >>> website with a list of the datasets, the datasets' checksums, and the
> >>> contact information for the people with the hard drives on their
> >>> bookshelves.
> >>>
> >>> Note that this is my opinion only on how this project could be
> >>> implemented. I don't know enough about the datasets or the likely
> >>> effects of geopolitics on their implementation in order to comment as to
> >>> whether I think the project should be implemented.
> >>>
> >>
> >> not to nit-pick but i would strongly recommend *against* using HDD's in this manner (magnetic spinning ones, or SSD ones).  Drives are not designed to reliably store data cold like this mechanically or electrically.  This is why tapes are still in use to this day - they *are* designed for cold store.  And if you do hit a bad sector it is quite possible to skip that sector and continue reading data.
> >>
> >> This is coming from quite a bit first-hand experience where I've lost data-sets which were in cold-storage on HDD's for about a year that were totally lost, versus data on tapes which were in cold-store for around 5-7years where we had few problems recovering our assets.
> >
> > Hrm.  You may have put the silver bullet in my previous post.
> >
> > Pete: any thoughts on mitigating this effect by using ZFS mirrors?  Perhaps even increasing the block mirror count across disks, so even on one of the mirrored disks there are 2 mirrored blocks?
> >
> > I mean, one crappy way to test this is to just do it and wait a year :P
> 
> tbh - i'd be most concerned about mechanical issues on magnetic HDD's.
> 
> Unlike a tape where I can physically forward the tape to a new sector if 
> I run into a problem (something i've had to do!) I have seen my fair 
> share of drives sit idle for a period of time only to refuse to spin-up 
> when i tried to revive/recycle them.
> 
> -p
> 
> 
> -- 
> Pete Wright
> pete at nomadlogic.org
> nomadlogicLA