Slow streaming throughput for a very large filesystems

[puthick]

Hi,

I'm trying to use Amanda to backup 1.4 Terabytes of data with an HP 1/8 Ultrium 960 Autoloader over the network. The network maximum throughput is 100Mb/s. It is measured by copying a 20GB tar file from the host client to the backup server. The host client has a SATA raid 5 array holding the 1.4TB data and a single SATA as its system drive. The backup server has a holding disk of 230GB on a SATA drive. I don't use any software nor hardware compression.

I want to backup the data of 1.4TB in 48 hours which is not unreasonable. However, so far it takes more than 4 days to backup 1.2TB of data. The log statistics is at the bottom of this message. It backups at around 3MB/s.

I do a backup test of 86GB of data. It can backup at round 8MB/s.

I'm using SUSE Linux 9.3 for the host client and the backup server.

Could you please help me explaining the cause of this problem? What can I do to solve this problem? Am I doomed?

I understand that backing up more than 1.4TB is a challenge, but it should be possible to do it in less than 60 hours. Please see my amanda statistics log below.

-- Log for 1.2TB backup.

These dumps were to tapes WeeklySet-4-01, WeeklySet-4-02, WeeklySet-4-03, WeeklySet-4-04.
The next 4 tapes Amanda expects to use are: WeeklySet-1-02, WeeklySet-1-03, WeeklySet-2-01, WeeklySet-2-02.
The next new tape already labelled is: WeeklySet-2-03.

STATISTICS:
Total Full Incr.
-------- -------- --------
Estimate Time (hrs:min) 0:00
Run Time (hrs:min) 110:12
Dump Time (hrs:min) 110:11 110:11 0:00
Output Size (meg) 1128387.4 1128387.4 0.0
Original Size (meg) 1124861.2 1124861.2 0.0
Avg Compressed Size (%) -- -- --
Filesystems Dumped 1 1 0
Avg Dump Rate (k/s) 2912.9 2912.9 --

Tape Time (hrs:min) 110:01 110:01 0:00
Tape Size (meg) 1128387.4 1128387.4 0.0
Tape Used (%) 292.3 292.3 0.0
Filesystems Taped 1 1 0

Chunks Taped 112839 112839 0
Avg Tp Write Rate (k/s) 2917.4 2917.4 --

USAGE BY TAPE:
Label Time Size % Nb Nc
WeeklySet-4-01 35:04 367222592k 92.9 0 35974
WeeklySet-4-02 35:03 367222592k 92.9 0 35974
WeeklySet-4-03 35:05 367222592k 92.9 0 35974
WeeklySet-4-04 4:47 50190144k 12.7 1 4917

NOTES:
planner: Adding new disk host_client:/data1/tiff.
driver: WARNING: This is not the first amdump run today. Enable the usetimestamps option in the configuration file if you want to run amdump more than once per calendar day.
taper: no split_diskbuffer specified: using fallback split size of 10240kb to buffer host_client:/data1/tiff.0 in-memory
taper: tape WeeklySet-4-01 kb 368373792 fm 35975 writing file: No space left on device
taper: continuing host_client:/data1/tiff.0 on new tape from 368373760kb mark: [writing file: No space left on device]
taper: tape WeeklySet-4-02 kb 368373792 fm 35975 writing file: No space left on device
taper: continuing host_client:/data1/tiff.0 on new tape from 736747520kb mark: [writing file: No space left on device]
taper: tape WeeklySet-4-03 kb 368373792 fm 35975 writing file: No space left on device
taper: continuing host_client:/data1/tiff.0 on new tape from 1105121280kb mark: [writing file: No space left on device]
taper: tape WeeklySet-4-04 kb 50347488 fm 4917 [OK]

DUMP SUMMARY:
DUMPER STATS TAPER STATS
HOSTNAME DISK L ORIG-kB OUT-kB COMP% MMM:SS KB/s MMM:SS KB/s
-------------------------- ------------------------------------- -------------
host_client -data1/tiff 0 1151857910 1155468736 -- 6611:13 2903.8 6600:5 2917.4

(brought to you by Amanda version 2.5.1p3)

-- Log for 86GB backup.

These dumps were to tape WeeklySet-3-04.
The next 4 tapes Amanda expects to use are: WeeklySet-4-01, WeeklySet-4-02, WeeklySet-4-03, WeeklySet-4-04.
The next 2 new tapes already labelled are: WeeklySet-2-04, WeeklySet-2-03.

STATISTICS:
Total Full Incr.
-------- -------- --------
Estimate Time (hrs:min) 0:00
Run Time (hrs:min) 3:16
Dump Time (hrs:min) 2:35 2:35 0:00
Output Size (meg) 87281.8 87281.8 0.0
Original Size (meg) 87281.8 87281.8 0.0
Avg Compressed Size (%) -- -- --
Filesystems Dumped 1 1 0
Avg Dump Rate (k/s) 9582.1 9582.1 --

Tape Time (hrs:min) 0:36 0:36 0:00
Tape Size (meg) 87284.4 87284.4 0.0
Tape Used (%) 22.6 22.6 0.0
Filesystems Taped 1 1 0

Chunks Taped 86 86 0
Avg Tp Write Rate (k/s) 41463.8 41463.8 --

USAGE BY TAPE:
Label Time Size % Nb Nc
WeeklySet-3-04 0:36 89376544k 22.6 1 86

NOTES:
planner: Adding new disk host_client:/data1/dat.
driver: WARNING: This is not the first amdump run today. Enable the usetimestamps option in the configuration file if you want to run amdump more than once per calendar day.
taper: tape WeeklySet-3-04 kb 89379296 fm 86 [OK]

DUMP SUMMARY:
DUMPER STATS TAPER STATS
HOSTNAME DISK L ORIG-kB OUT-kB COMP% MMM:SS KB/s MMM:SS KB/s
-------------------------- ------------------------------------- -------------
host_client -/dat 0 89376520 89379264 -- 155:27 9582.1 35:56 41463.8

(brought to you by Amanda version 2.5.1p3)

Puthick

[martineau]

Because your smaller dle do through the holding disk while the biggest go
directly to tape without using the holding disk.

You should split the big dle in many smaller dle with size smaller that half your holding disk size.

[puthick]

Thanks. It works.

[nickhere]

Thanks. It works.
Hello,

I am using amandav2.5.p2 and I have 30 production servers each having around 30 gB of data to be backed up. My backup server is having 800 GB of hardisk space.

My backup schedule is to take full backups every week clearing the older full backup and do incrementals in between . My amanda conf is pated below.

dumpuser "amandabackup"
inparallel 25
dumporder "tttttttttttttttttttttTTTT"
taperalgo largestfit
netusage 8000 Kbps
dumpcycle 1 week
runspercycle 5
tapecycle 30 tapes
bumpsize 500 Mb
bumppercent 0
bumpdays 1
bumpmult 4
etimeout 7200
dtimeout 1800
ctimeout 30
tapebufs 40
runtapes 4
tpchanger "chg-disk"
maxdumpsize -1
tapetype HARDDISK
chunksize 3Gb

define tapetype HARDDISK {
length 400000 mbytes
}

define dumptype global {
auth "bsdtcp"
index yes
}

define dumptype mydump {
global
strategy standard
program "GNUTAR"
exclude list "/etc/exclude_list"
compress server fast
record no
estimate server
maxdumps 1
comprate 0.25, 0.25
priority medium
}

define interface local {
comment "a local disk"
use 8000 kbps
}

define interface le0 {
comment "10 Mbps ethernet"
use 400 kbps


Currently a full backup is taking 35+ hours to complete. And it is taking everyday full backup instead of incremental s.
Is there anything wrong in my configuration??

Any help appreciated.

[martineau]

set record to yes