60TB / 250 million file implementation?

[jhelgren]

As my first post, thank you to all who contribute to the AMANDA project and community! I have been searching for a good backup strategy for a couple of months now and have finally gotten serious about it. I am not looking on HOW to implement it but more of a quick fact check to see if there are any obvious snags that I would be up against. My implementation would be the following:
- Main Storage System: 60TB IBRIX Storage system - http://en.wikipedia.org/wiki/IBRIX_Fusion This system is at 57TB containing over 250 Million files with roughly 20,000-30,000 thousand modifications/additions per week.
- Clients / OSes: Various Windows and Linux Boxes (Both metal and virtual)
- Backup system: The backup system will be the recommended install of OS Flavor and AMANDA with a SAS attached TL2000 Dell Backup Library System with 2 magazines and a billion tapes
- Backup Data Type: Most of our data is multimedia content in form of wma and mp3’s. Little to no compression is expected.
- Network: Everything is connected through 1Gbps Ethernet to a Cisco 4006. There is approximately 6Gbps of network throughput from the IBRIX system to the system (Link Aggregation). The TL2000 is attached to a PowerEdge 860 with 2 x 1Gbps links. So theoretical throughput to the backup system is 2Gbps which equates to about 2.7 days of continual throughput on the backup system in a perfect world. I’m guessing I would be happy with 4 or 5 days.
The question I guess I need to ask is – Theoretically I should be able to pull this off?

[zmanda_jacob]

Hello,

Well, I know this is a little late, but what you're wanting to do is possible. In general, breaking down the data into smaller pieces will be the key. What I mean by this is it sounds like the build of your data will be coming from a single host/filer. Backing up 250 million files in one piece will be more challenging than 60Tb. Rather than add a single entry for "/", you would fare better if you added entries for /subdir1, /subdir2, etc. If you can find a logical way to break down the data that will help. We have customers who are backing up this much data with a large number of files successfully(maybe not 250 million, but certainly a lot of files).

-Jacob

[jhelgren]

Thanks for the post, - no reply is ever too late we will be implementing a proof of concept in the coming months and will post back feedback and suggestions of how to accomplish this.