PDA

View Full Version : GNU tar 1.16-2 Estimate and Dump Problem



netopsaj
April 23rd, 2007, 12:36 PM
Hi all...

I am using Amanda 2.5.1p1-2.1 and GNU tar 1.16-2 on Debian 2.6.18 and I have been running into some estimate and dump problems.

I've been noticing that the estimates are taking an exceptionally long time on a particular 100 GB file system which contains around 4 million files, many with rather complex file names. I switched the estimation over to calcsize instead and that seems to have improved the estimate performance, but then when amanda tries to tar the files over for dumping, I get an error stating:

"driver: (aborted:"[request failed: timeout waiting for REP]")(too many dumper retry)"

Then on another file system that is about 300GB I see an estimate that is about right but then I end up seeing a dump size of about 225GB, clearly well short of the goal. I know the real size is about 300GB.

Other file systems complete but some require retries. Check out this backup report below:

-------------------------------------------------------------
These dumps were to tapes Weekly05, Weekly06.
The next 4 tapes Amanda expects to use are: 4 new tapes.

FAILURE AND STRANGE DUMP SUMMARY:
amanda2 /backups/mnt/user lev 0 STRANGE
amanda2 /etc/amanda lev 0 STRANGE
ad /private-1/perforce lev 0 STRANGE
amanda2 /backups/mnt/user/assar lev 0 FAILED [dumper returned FAILED]
amanda2 /backups/mnt/user/assar lev 0 FAILED [data timeout]
amanda2 /backups/mnt/user/assar lev 0 FAILED [too many dumper retry: "[request failed: timeout waiting for REP]"]
amanda2 /backups/mnt/user/assar lev 0 FAILED [cannot read header: got 0 instead of 32768]
amanda2 /backups/mnt/user/mp lev 0 FAILED [missing size line from sendbackup]
amanda2 /backups/mnt/build lev 0 FAILED [missing size line from sendbackup]
amanda2 /backups/mnt/user/il lev 0 FAILED [missing size line from sendbackup]
amanda2 /backups/mnt/RT lev 0 FAILED [missing size line from sendbackup]
amanda2 /backups/mnt/user/qt lev 0 FAILED [missing size line from sendbackup]
amanda2 /backups/mnt/user/ad lev 0 FAILED [missing size line from sendbackup]
amanda2 /backups/mnt/release lev 0 FAILED [missing size line from sendbackup]
pd / lev 0 STRANGE
amanda2 /backups/mnt/build lev 0 was successfully retried
amanda2 /backups/mnt/user/il lev 0 was successfully retried
amanda2 /backups/mnt/user/qt lev 0 was successfully retried
amanda2 /backups/mnt/user/ad lev 0 was successfully retried
amanda2 /backups/mnt/user/mp lev 0 was successfully retried
amanda2 /backups/mnt/release lev 0 was successfully retried
amanda2 /backups/mnt/release lev 0 STRANGE
amanda2 /backups/mnt/RT lev 0 was successfully retried


STATISTICS:
Total Full Incr.
-------- -------- --------
Estimate Time (hrs:min) 2:36
Run Time (hrs:min) 32:06
Dump Time (hrs:min) 132:30 132:30 0:00
Output Size (meg) 634667.3 634667.3 0.0
Original Size (meg) 904641.3 904641.3 0.0
Avg Compressed Size (%) 70.2 70.2 --
Filesystems Dumped 22 22 0
Avg Dump Rate (k/s) 1362.5 1362.5 --

Tape Time (hrs:min) 3:57 3:57 0:00
Tape Size (meg) 572024.2 572024.2 0.0
Tape Used (%) 148.2 148.2 0.0
Filesystems Taped 22 22 0

Chunks Taped 34 34 0
Avg Tp Write Rate (k/s) 41197.2 41197.2 --

USAGE BY TAPE:
Label Time Size % Nb Nc
Weekly05 2:50 385757M 99.9 21 24
Weekly06 1:07 186266M 48.2 1 10

----------------------------------------

I am thinking that tar is the culprit here, particularly after reading this:
http://www.amanda.org/docs/install.html
But I'm not sure considering my version of tar is much newer. Is there anyone else out there who has run into this sort of snag and has found a way around?

Any advice or assistance would be greatly appreciated. Thanks in advance!

~Andy

netopsaj
May 3rd, 2007, 01:29 PM
So I just discovered something that should have been glaringly obvious... Amanda seems to use server-side compression by default unless you specify
"compress none" in your dumptypes. That would have been a nice little spec of information to have included in the default amanda.conf file!

Anyway... the compression is what's taking these backups so long and then to make matters worse I have hardware compression enabled on my tape drive. This is undoubtedly causing my backups to take up more tape and time than they would have if I had been writing uncompressed data to the tape. Not cool. I have a very large level 0 amdump running right now that I can't interrupt... but tonight I will try again with the "compress none" flag in the dumptypes and see how I fare.

paddy
June 5th, 2007, 03:48 PM
So I just discovered something that should have been glaringly obvious... Amanda seems to use server-side compression by default unless you specify
"compress none" in your dumptypes. That would have been a nice little spec of information to have included in the default amanda.conf file!


The default compression value is "client fast". Backup data is compressed on the client.

It is documented in amanda.conf man page. See http://wiki.zmanda.com/index.php/Amanda.conf#DUMPTYPE_SECTION (look for compress)

It is also documented in the example amanda.conf included in the distribution

# compress - specify compression of the backed up data. Valid values are:
# "none" - don't compress the dump output.
# "client best" - compress on the client using the best (and
# probably slowest) algorithm.
# "client fast" - compress on the client using fast algorithm.
# "client custom" - compress using your custom client compression program.
# use client_custom_compress "PROG" to specify# the custom compression program.
# PROG must not contain white space.
# "server best" - compress on the tape host using the best (and
# probably slowest) algorithm.
# "server fast" - compress on the tape host using a fast
# algorithm. This may be useful when a fast
# tape host is backing up slow clients.
# "server custom" - compress using your server custom compression program.
# use server_custom_compress "PROG" to specify# the custom compression program.
# PROG must not contain white space.
# Default: [compress client fast]