Data storage

I’ve mentioned before that at the rate our disks are growing we will soon have all the data we will ever need stored on our own hard drives. Although there are Petabytes of information flowing around every day most of them are just people like me downloading the same stuff for storage on our own systems. YouTubes, Games, PDFs, Sounds, etc. Just think of how many Petabyes are downloaded from Torrents and consider that some of them are downloaded hundreds of times and the files are then distributed in other ways as well. So the actually amount of real data out there must be significantly lower than the figures bandied about for the Internet where they are talking about some really large numbers and making up new names every six months.

Also consider that many of us already have multi Terabyte systems and they are becoming more common. 1TB USB drives are available for £60 and bare drives even less. The curse of having all this space available is finding your stuff. I try and catalogue the files as I download them but to be honest I fail miserably and put a lot of stuff in folders labelled ‘ToBeSorted’. Sometimes it is easier to search the web again for specific files as I have no idea where they are and can’t lay my cursor on them quickly. I really need a sort out and I’m sure I’m not the only one.

I’ve been intending to write my own indexing solution using a MySQL DB and get it to trawl all the files but so far I don’t have the time. In the meantime my 4TB SAN is creaking at the seams and I’m looking to upgrade it again before the end of the year.

There is also another consideration media type. Originally we had Hard Disks and then we had the option of Solid State Drives with a new media type on the way. Read here about Ferroelectric Data Storage. It sounds to me very much like how Hard Disks work but we will see. The smaller the size and the bigger the storage the better.

I also have differing requirements from most. I want my media to be destructable. I want it to fit in a single safe place and be destroyed at the touch of a key. HDAs are too tough to do easily, although they can be done, although I suspect illegally, and SSDs are not up to replacing HDAs yet with their rewrite 1000 times although they can be destroyed very easily, probably illegally also. Good enough for those with deep pockets and high security requirements but not for us.

So I’ve built a hybrid system. 4Tb of downloaded files on a HDA SAN with my private stuff on SSD. I have about 50Gb of private stuff. Only problem is that the way windows works I have to keep my private stuff off line on a different system so I’m actually keeping it in an awkward to get to place where my SSD data, easily destroyed, is actually all over the HDA, not easily destroyed, because of the way windows works and swap files. Bugger.

I wish someone would create a Storage Array that would have a mixture of HDAs and SSD and have a destructable component that I could just buy. The alternative is that I build one myself and we know what will happen there. Although I would have thought a SAN with destructable components would be very popular with the world the way it is today. Maybe I should invest some time and effort on it.