stevecs
December 14th, 2009, 03:30 AM
Ok, I am just starting w/ Amanda per my post here (http://forums.zmanda.com/showthread.php?t=2227) I have some large systems that I want to have a backup system locally attached to them (due to their size). Locally I can back up to tape (LTO4) with GNU Tar (v1.19 / ubuntu 8.04.3) at ~105-110MiB/s no problems. The drive array consists of 64 drives on hardware raid controllers using LVM to stripe the various raid-6's. RAW I/O to the volume is over 500MiB/s.
When using amanda with the configs below, I see that the estimate phase takes a very long time (fails to complete on /var/ftp) and from iostat seems to be doing only 1-2MiB/s? which could very well be a problem in that regard. When dump finally /does/ start on the disks the backup is about 1-2MiB/s (nearly 10 hours to back up less than 100GiB).
I did notice that amanda does NOT set any blocking factor to tar, so I wrapped tar (created a script that would add the blocking factor of 1024 (512KiB) which I use for direct tar backups and gave me the >100MiB/s numbers above.
Since this is so pathetic, I am hoping that there is something that someone can see here as I just can't believe that amanda or any backup utility could be this bad by design.
I also noticed that all the taper 'PART' sections of the log mention 10240 kbp which may be a hard coded limit somewhere but can't find it or even if that is important (does seem very low though).
EDIT: note, running 2.6.1p2 binary from zmanda.com for ubuntu 8.04
On a test machine I have this:
---(df -h)
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 58G 11G 44G 20% /
/dev/sda1 236M 58M 167M 26% /boot
/dev/mapper/vg_media-lv_ftpshare
40T 29T 12T 72% /var/ftp
/dev/mapper/vg_media-lv_usershare
2.0T 1.4T 685G 67% /home
---- (disklist)
xxxx / nocomp-tar
xxxx /boot nocomp-tar
xxxx /home nocomp-tar
xxxx /var/ftp nocomp-tar
----(amanda.conf)
org "BackupSetAA" # your organization name for reports
mailto "xxxxxx" # space separated list of operators at your site
dumpcycle 12weeks # the number of days in the normal dump cycle
runspercycle 12 # the number of amdump runs in dumpcycle days
tapecycle 40 # the number of tapes in rotation
runtapes 40 # number of tapes to be used in a single run of amdump
tpchanger "chg-manual" # the tape-changer glue script
tapedev "/dev/nst0" # the no-rewind tape device
changerfile "/etc/amanda/BackupSetAA/chg-manual.conf" # tape changer configuration parameter file
changerdev "/dev/null" # tape changer configuration parameter device
tapetype LTO4-HWC # what kind of tape it is
labelstr "^AA[0-9][0-9][0-9][0-9]*$" # label constraint regex: all tapes must match
dtimeout 1800 # number of idle seconds before a dump is aborted
ctimeout 30 # max number of secconds amcheck waits for each client
etimeout 3000 # number of seconds per filesystem for estimates
dumpuser "amandabackup" # the user to run dumps under
inparallel 4 # maximum dumpers that will run in parallel (max 63)
dumporder "sssS" # specify the priority order of each dumper
taperalgo first # The algorithm used to choose which dump image to send
displayunit "g" # Possible values: "k|m|g|t"
netusage 150000 Kbps # maximum net bandwidth for Amanda, in KB per sec
bumpsize 200 Gb # minimum savings (threshold) to bump level 1 -> 2
bumppercent 20 # minimum savings (threshold) to bump level 1 -> 2
bumpdays 1 # minimum days at each level
usetimestamps yes
device_output_buffer_size 128000k
label_new_tapes "AA%%%%"
maxdumpsize -1 # Maximum total size the planner will schedule
# for a run (default: runtapes * tape_length) (kbytes).
amrecover_do_fsf yes # amrecover will call amrestore with the
# -f flag for faster positioning of the tape.
amrecover_check_label yes # amrecover will call amrestore with the
# -l flag to check the label.
bumpmult 4 # threshold = bumpsize * bumpmult^(level-1)
amrecover_changer "changer" # amrecover will use the changer if you restore
# from this device. It could be a string like 'changer' and amrecover will use your
# changer if you set your tape to 'changer' with 'setdevice changer' or via
# 'tapedev "changer"' in amanda-client.conf
autoflush no
infofile "/etc/amanda/BackupSetAA/curinfo" # database DIRECTORY
logdir "/etc/amanda/BackupSetAA/logs" # log directory
indexdir "/etc/amanda/BackupSetAA/index" # index directory
tapelist "/etc/amanda/BackupSetAA/tapelist" # index directory
define interface local {
comment "a local disk"
use 200000 kbps
}
define dumptype global {
comment "Global definitions"
auth "bsdtcp"
}
define dumptype nocomp-tar {
comment "GNUTAR based dump"
global
program "GNUTAR"
exclude list "/etc/amanda/exclude.gtar"
compress none
priority high
tape_splitsize 25 Gb
index yes
record yes
}
define tapetype LTO4-HWC {
comment "just produced by tapetype prog (hardware compression on)"
length 788480 mbytes
filemark 0 kbytes
tape_splitsize 25 Gb
speed 94896 kps
}
------------- (log snippet)
INFO amdump amdump pid 30272
INFO planner planner pid 30318
DISK planner xxxx /
DISK planner xxxx /boot
DISK planner xxxx /home
DISK planner xxxx /var/ftp
START planner date 20091213212136
INFO planner Adding new disk xxxx:/.
INFO planner Adding new disk xxxx:/boot.
INFO planner Adding new disk xxxx:/home.
INFO planner Adding new disk xxxx:/var/ftp.
INFO driver driver pid 30319
START driver date 20091213212136
STATS driver hostname loki
STATS driver startup time 0.005
INFO dumper dumper pid 30323
INFO dumper dumper pid 30324
INFO dumper dumper pid 30321
INFO dumper dumper pid 30322
INFO taper taper pid 30320
ERROR planner Request to xxxx failed: timeout waiting for REP
WARNING planner disk xxxx:/var/ftp, estimate of level 0 timed out.
FAIL planner xxxx /var/ftp 20091213212136 0 "[disk /var/ftp, all estimate timed out]"
FINISH planner date 20091213212136 time 3601.427
INFO planner pid-done 30318
INFO dumper gzip pid 21929
START taper datestamp 20091213212136 label AA0001 tape 1
PART taper AA0001 1 xxxx /home 20091213212136 1/-1 0 [sec 18.061213 kb 10240 kps 566.960813]
PART taper AA0001 2 xxxx /home 20091213212136 2/-1 0 [sec 0.718334 kb 10240 kps 14255.207188]
PART taper AA0001 3 xxxx /home 20091213212136 3/-1 0 [sec 0.324078 kb 10240 kps 31597.331507
------
When using amanda with the configs below, I see that the estimate phase takes a very long time (fails to complete on /var/ftp) and from iostat seems to be doing only 1-2MiB/s? which could very well be a problem in that regard. When dump finally /does/ start on the disks the backup is about 1-2MiB/s (nearly 10 hours to back up less than 100GiB).
I did notice that amanda does NOT set any blocking factor to tar, so I wrapped tar (created a script that would add the blocking factor of 1024 (512KiB) which I use for direct tar backups and gave me the >100MiB/s numbers above.
Since this is so pathetic, I am hoping that there is something that someone can see here as I just can't believe that amanda or any backup utility could be this bad by design.
I also noticed that all the taper 'PART' sections of the log mention 10240 kbp which may be a hard coded limit somewhere but can't find it or even if that is important (does seem very low though).
EDIT: note, running 2.6.1p2 binary from zmanda.com for ubuntu 8.04
On a test machine I have this:
---(df -h)
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 58G 11G 44G 20% /
/dev/sda1 236M 58M 167M 26% /boot
/dev/mapper/vg_media-lv_ftpshare
40T 29T 12T 72% /var/ftp
/dev/mapper/vg_media-lv_usershare
2.0T 1.4T 685G 67% /home
---- (disklist)
xxxx / nocomp-tar
xxxx /boot nocomp-tar
xxxx /home nocomp-tar
xxxx /var/ftp nocomp-tar
----(amanda.conf)
org "BackupSetAA" # your organization name for reports
mailto "xxxxxx" # space separated list of operators at your site
dumpcycle 12weeks # the number of days in the normal dump cycle
runspercycle 12 # the number of amdump runs in dumpcycle days
tapecycle 40 # the number of tapes in rotation
runtapes 40 # number of tapes to be used in a single run of amdump
tpchanger "chg-manual" # the tape-changer glue script
tapedev "/dev/nst0" # the no-rewind tape device
changerfile "/etc/amanda/BackupSetAA/chg-manual.conf" # tape changer configuration parameter file
changerdev "/dev/null" # tape changer configuration parameter device
tapetype LTO4-HWC # what kind of tape it is
labelstr "^AA[0-9][0-9][0-9][0-9]*$" # label constraint regex: all tapes must match
dtimeout 1800 # number of idle seconds before a dump is aborted
ctimeout 30 # max number of secconds amcheck waits for each client
etimeout 3000 # number of seconds per filesystem for estimates
dumpuser "amandabackup" # the user to run dumps under
inparallel 4 # maximum dumpers that will run in parallel (max 63)
dumporder "sssS" # specify the priority order of each dumper
taperalgo first # The algorithm used to choose which dump image to send
displayunit "g" # Possible values: "k|m|g|t"
netusage 150000 Kbps # maximum net bandwidth for Amanda, in KB per sec
bumpsize 200 Gb # minimum savings (threshold) to bump level 1 -> 2
bumppercent 20 # minimum savings (threshold) to bump level 1 -> 2
bumpdays 1 # minimum days at each level
usetimestamps yes
device_output_buffer_size 128000k
label_new_tapes "AA%%%%"
maxdumpsize -1 # Maximum total size the planner will schedule
# for a run (default: runtapes * tape_length) (kbytes).
amrecover_do_fsf yes # amrecover will call amrestore with the
# -f flag for faster positioning of the tape.
amrecover_check_label yes # amrecover will call amrestore with the
# -l flag to check the label.
bumpmult 4 # threshold = bumpsize * bumpmult^(level-1)
amrecover_changer "changer" # amrecover will use the changer if you restore
# from this device. It could be a string like 'changer' and amrecover will use your
# changer if you set your tape to 'changer' with 'setdevice changer' or via
# 'tapedev "changer"' in amanda-client.conf
autoflush no
infofile "/etc/amanda/BackupSetAA/curinfo" # database DIRECTORY
logdir "/etc/amanda/BackupSetAA/logs" # log directory
indexdir "/etc/amanda/BackupSetAA/index" # index directory
tapelist "/etc/amanda/BackupSetAA/tapelist" # index directory
define interface local {
comment "a local disk"
use 200000 kbps
}
define dumptype global {
comment "Global definitions"
auth "bsdtcp"
}
define dumptype nocomp-tar {
comment "GNUTAR based dump"
global
program "GNUTAR"
exclude list "/etc/amanda/exclude.gtar"
compress none
priority high
tape_splitsize 25 Gb
index yes
record yes
}
define tapetype LTO4-HWC {
comment "just produced by tapetype prog (hardware compression on)"
length 788480 mbytes
filemark 0 kbytes
tape_splitsize 25 Gb
speed 94896 kps
}
------------- (log snippet)
INFO amdump amdump pid 30272
INFO planner planner pid 30318
DISK planner xxxx /
DISK planner xxxx /boot
DISK planner xxxx /home
DISK planner xxxx /var/ftp
START planner date 20091213212136
INFO planner Adding new disk xxxx:/.
INFO planner Adding new disk xxxx:/boot.
INFO planner Adding new disk xxxx:/home.
INFO planner Adding new disk xxxx:/var/ftp.
INFO driver driver pid 30319
START driver date 20091213212136
STATS driver hostname loki
STATS driver startup time 0.005
INFO dumper dumper pid 30323
INFO dumper dumper pid 30324
INFO dumper dumper pid 30321
INFO dumper dumper pid 30322
INFO taper taper pid 30320
ERROR planner Request to xxxx failed: timeout waiting for REP
WARNING planner disk xxxx:/var/ftp, estimate of level 0 timed out.
FAIL planner xxxx /var/ftp 20091213212136 0 "[disk /var/ftp, all estimate timed out]"
FINISH planner date 20091213212136 time 3601.427
INFO planner pid-done 30318
INFO dumper gzip pid 21929
START taper datestamp 20091213212136 label AA0001 tape 1
PART taper AA0001 1 xxxx /home 20091213212136 1/-1 0 [sec 18.061213 kb 10240 kps 566.960813]
PART taper AA0001 2 xxxx /home 20091213212136 2/-1 0 [sec 0.718334 kb 10240 kps 14255.207188]
PART taper AA0001 3 xxxx /home 20091213212136 3/-1 0 [sec 0.324078 kb 10240 kps 31597.331507
------