PDA

View Full Version : Future-Proof Digital Storage Solution



leelive
01-06-2010, 03:16 PM
Every year I have had to migrate to an ever larger HDD.

Now at 1.5TB with 2,000 CDs / 35,000 wav files and it is almost full.

I have Produb licences to cover the copies from my existing CDs and will buy another 10,000 track produb licence in July to cover the 500+ CDs that I will buy in the next year.

I aim to grow my collection by just 250CDs / 5,000 wav files (250GB) per year from July 2011 onwards.

So by July 2015 I will need more than 3TB.

I run two laptops side-by-side and would like a future-proof Storage Solution that can very quickly be connected to both laptops.

Ideas and suggestions welcome.

Lee

Excalibur
01-06-2010, 03:46 PM
Every year I have had to migrate to an ever larger HDD.

Now at 1.5TB with 2,000 CDs / 35,000 wav files and it is almost full.

I have Produb licences to cover the copies from my existing CDs and will buy another 10,000 track produb licence in July to cover the 500+ CDs that I will buy in the next year.

I aim to grow my collection by just 250CDs / 5,000 wav files (250GB) per year from July 2011 onwards.

So by July 2015 I will need more than 3TB.

I run two laptops side-by-side and would like a future-proof Storage Solution that can very quickly be connected to both laptops.

Ideas and suggestions welcome.

Lee
Which will cost about 50p, and be the size of a postage stamp? :confused: I can remembr spending over £200 upgrading an internal HD to the dizzy heights of: 250Mb. :eek: :eek: :eek:

DJ Jules
01-06-2010, 03:51 PM
In short, there isn't really one. You're about to hit a barrier because even the manufacturers are only just starting to play with the idea of making a HDD bigger than 2Tb. You'll probably have to buy something which ties multiple large hard disks together and presents them as one large one. But then you might have another problem in that Windows has issues recognising a single volume bigger than 2Tb too!

Better solution... compress your audio using a lossless format (AAC, MP4, etc?) Why do you need to hold your CD's as .wav's?

Julian

Vectis
01-06-2010, 04:02 PM
Who mentioned windows? :)

internal solution? RAID card or software (built in to mac osx) no idea about windows.

External solution any one of a number of RAID enabled NAS boxes such as Buffalo or QNap brands.

Excalibur
01-06-2010, 04:38 PM
Who mentioned windows? :)

internal solution? RAID card or software (built in to mac osx) no idea about windows.

External solution any one of a number of RAID enabled NAS boxes such as Buffalo or QNap brands.

I understood "Windows". Where the bovines and thievery come in, I haven't the faintest. :confused: :confused: :confused:

Daryll
01-06-2010, 04:50 PM
In principle the Linux EXT3 file system , can "see" a hard drive up to 64TiB , this can be one file :eek:

Daryll

Excalibur
01-06-2010, 04:55 PM
In principle the Linux EXT3 file system , can "see" a hard drive up to 64TiB , this can be one file :eek:

Daryll

Wasn't he in the Peanuts strip? :confused: :D :D :D :D

Marc J
01-06-2010, 05:02 PM
External solution any one of a number of RAID enabled NAS boxes such as Buffalo or QNap brands.

A vote for QNAP here, can't fault mine (TS-439 Pro II (http://www.qnap.com/pro_detail_feature.asp?p_id=148))

Tony Scott
01-06-2010, 05:34 PM
:confused: :confused: :confused: :confused: :confused: :confused: :confused:

I feel so stupid right now!!! :(

Vectis
01-06-2010, 05:38 PM
A vote for QNAP here, can't fault mine (TS-439 Pro II (http://www.qnap.com/pro_detail_feature.asp?p_id=148))

Yep. Posted QNAP because I have good experiences too :)

I know others using similarly specced Buffalo products also give them a thumbs up.

If you have a spare £30K+ lying around then the EMC (formerly Data General) Clariion range are also well recommended here at VV Towers :D

leelive
01-06-2010, 08:50 PM
A vote for QNAP here, can't fault mine (TS-439 Pro II (http://www.qnap.com/pro_detail_feature.asp?p_id=148))

Thanks. Just looking at the spec. What are those "Switches" that are shown for it to be connected to a Laptop. Would I need two of them to connect to two laptops?

Lee

Vectis
01-06-2010, 09:04 PM
NAS = Network Attached Storage.

You can link a whole bunch of computers via Ethernet. I suspect the 'switch' referred to in the spec. (not read it) is an ethernet switch or hub. There will be a limit as to the number of computers (back to the spec again) but it'll be way more than two ;)

Marc J
02-06-2010, 08:19 AM
As VV said, the NAS connects to your network (hub / router / switch) and is then available to any device attached to it (PC / Laptop / Mac / PS3 / XBox etc).

These connected devices will see it as network drive(s), or a media server, depending on the setup.

You don't need to fill all drive bays at first, but you do need at least two drives to benefit from a RAID setup. Let's say you put 2 x 1Tb drives in it to start with and use RAID1 (mirroring) - you'd only have 1Tb of useable space but your data will survive the failure of either drive as there are always two copies. You'd have to replace that failed drive ASAP though.

I've got 4 x 1Tb in mine, setup with RAID 5 + spare, which basically means 3 x 1Tb drives are in use on the RAID array giving 2Tb of useable storage. The fourth drive is set as a "hot spare" which will automatically kick in if any of the other 3 drives fails. It could then survive the failure of another drive before I would have to replace at least one to maintain redundancy.

You should never keep all your data in one place, though. RAID is not backup. I've another 2 x 2Tb external drives which I alternate backups to weekly, and one of these is always kept off-site. So in the worst case (my house burns down) I lose 1 week's data.

And it is of course expandable - you just add a drive if you have space, or replace smaller drives with larger ones, then rebuild the RAID array.

It's not cheap though - for your 3Tb of useable storage you'd need a minimum of 3 x 2Tb drives, so total cost would be ball park £800. Your problem then might be backing up to a single external drive if you were using all 3Tb!

Vectis
02-06-2010, 10:26 AM
:agree: wot Marc said ... just didn't have time yesterday to type all that :p

I have a similar RAID5 + hot spare configuration but was once bitten by a backplane issue which corrupted the whole RAID set which has made me extra-cautious about backup. Also, the way in which RAID is implemented is typically proprietary, so if you were to try to recover data from any individual disk in a RAID 1 set, then without an exact copy of the hardware which was used to create it in the first place you'd be into specialist recovery. With RAID 5 it's even more tricky to reconstruct as the data is 'striped' across the disks in the set, although some of the capacity of each disk is given over to holding parity data which can be used to reconstruct others (which is what happens when a hot spare kicks in). Just be aware that it's not a simple case of whipping out a disk and connecting it to a USB cable and off you go :)

Here's how my general backup works (simplistic description)....

The majority of my work is done on a mac, with working changes synced off to a Time Machine drive hourly.

Daily, the Time Machine volume is archived off to my NAS which is kept in a secure outhouse some 20m from the main building (where I keep my surplus disco gear :p ). The time machine volumes are rotated a week about, so I always have 6 good files to go back to. Being a TM volume means that if necessary I can rebuild an entire mac just from the volume image.

Also daily, I have a subset of data (just my most precious things) which I dump to one of three 1.8" 80GB USB-attached HDD's which I rotate. On a day where I haven't been productive - ie an all-day wedding disco type day - I've been known to skip this step. These drives I pop into the fire safe. This data is also encrypted.

Approximately monthly (usually the last Friday of the month, when I also do my invoicing) I snatch a full backup from the NAS onto a 2GB external USB HDD which lives round at the inlaws about 5 miles away. I have two of these drives on rotation so that one is always away from the house.



As far as music goes...

I have a full copy of my music database on a grandfather-father-son arrangement of 1TB HDDs. For those unfamiliar with the concept, the 'son' copy is the newest complete backup, 'father' being one step behind and 'grandfather' being two steps behind. Therefore if there's ever a problem with disk corruption you don't have three duff copies - you just go back a generation and abandon the changes made inbetween times.

The generation backup I tend to do twice a year which lets me fold in any new purchases and downloads. These are organised by artist name.

In the meantime I have three gigging drives (Freecom Toughdrive Pro's) of 160GB, two in HFS+ format which macs and d2 director/iDJ2's like, and one in NTFS format for emergency backup which can attach to one of the gigging laptops (lighting/video). These drives obviously only carry a subset of just under 5000 tracks (;) ) for gigging purposes and are organised how I like them for rapid access on the road.

New downloads go to memory sticks as do playlists for imminent gigs. Playlists are abandoned afterwards, and new purchases are folded into the generation backups in due course.

So a typical gig will see me rock up with a d2, plug in a 160GB HDD stuffed full of party tunes and a memory stick full of recent downloads/specifically requested songs for the gig at hand.



A bit long-winded but hope it helps someone sometime.

DJ Jules
02-06-2010, 01:02 PM
A bit long-winded but hope it helps someone sometime.

So that's the ideal solution. Now for the one for those of us who prefer to spend less than 50% of our waking lives backing up and managing our data... :D

Just kidding, I'm sure that doesn't take as long as it took to type :)

Julian

Vectis
02-06-2010, 01:06 PM
So that's the ideal solution. Now for the one for those of us who prefer to spend less than 50% of our waking lives backing up and managing our data... :D

Just kidding, I'm sure that doesn't take as long as it took to type :)

Julian

Dunno about ideal... but it's a strong solution.

Actual time investment is minimal to be honest; once you make it part of your daily routine you don't even notice it. Until something breaks that is.

1878blues
04-06-2010, 05:44 PM
ITB of Data is more than enough to handle an entire DJ collection

Excalibur
04-06-2010, 06:35 PM
ITB of Data is more than enough to handle an entire DJ collection

Not according to an OP using wav files. ;)

Vectis I salute you. Didn't understand a word of it, but I salute you.

I knew there was someone on here using freecom drives, I accused Steve of being that person, but he denied it. :D

leelive
04-06-2010, 06:55 PM
After realising that I am not playing 10% of the 35,000 tracks in my music collection I have decided to give up buying and copying CDs so will now just buy additional wav files from mobiledjdownloads and therefore will not be looking to upgrade my HDD afterall. Thanks to everyone for spending so much time explaining the options. Lee

Solitaire Events Ltd
05-06-2010, 12:16 PM
ITB of Data is more than enough to handle an entire DJ collection

Surely that depends on the amount of tracks and the bit rate they are ripped at.

What a strange comment!