PDA

View Full Version : backup performance - split_diskbuffer bottleneck



IMP
October 10th, 2007, 08:07 PM
Greetings!

I'm administering an environment that includes a server with a 6Tb RAID array and a HP Ultrium 960 (LTO3) tape drive, running plain ordinary Debian etch (32-bit). Amanda is 2.5.1p1-2.1

It's fallen to me to back up this 6Tb array, which is set up as RAID 5 under a single mount point, and amanda has some form here, so I'm pressing on with that. My current config is OK, the backup is happening, spans tapes even. Very happy so far! Kudos to the amanda team.

Unfortunately it's taking more than a day to fill a tape :-( Only get 1 tape change per day, ~3 weeks for a full backup. I have tape_splitsize 2G (appears to be a limit of the 32-bit environment), and split_diskbuffer is on a separate filesystem on the IDE boot drive. Not using a holding disk.

I see in the dump log many lines similar to:
CHUNK taper server3 /data2 20071005124252 788 0 [sec 281.269 kb 2097120 kps 7455.9 {wr: writers 6
5536 rdwait 232.279 wrwait 42.379 filemark 2.910}]

rdwait is 82% of the time taken to write each chunk. And I see on the server, top reports 70-90% wait, and iostat reports similar iowait. iostat also reports the partition where I've assigned split_buffer is doing up to 3000 tps, and reading AND writing up to 10Mb/sec. Sure enough, the IDE disk activity light on the server is hard on.

This looks like the bottleneck to me. No other users are generating this activity. lsof on this partition shows just a few of my shells, and taper having two fds open for 'splitdump_buffer' (one is 'mem', one is '3u')

Interestingly, the very first write to tape after a tape change is significantly faster than all the other writes to that tape, presumably because splitdump_buffer is only read, and not written.

Investigations with dd show that I can read 20-30Mb/sec off the RAID array; best case would be to fill a 400Gb tape in ~4 hours. I'm estimating it currently takes 12-15 hours, certainly longer than the 8 hours I'm in the office.

I've tried the 'blockdev' tip, and the 'hdparm' tip, with no noticeable improvement. I could try a faster disk for split_diskbuffer, but 3 times faster than a modern IDE drive? Maybe a USB memory stick?

Any other suggestions? All comments welcome!

IMP
October 11th, 2007, 04:46 PM
A brief test with a 1G memory stick shows 1Mb/sec reads :-(