Results 1 to 5 of 5

Thread: Slow streaming throughput for a very large filesystems

  1. #1

    Default Slow streaming throughput for a very large filesystem

    Hi,

    I'm trying to use Amanda to backup 1.4 Terabytes of data with an HP 1/8 Ultrium 960 Autoloader over the network. The network maximum throughput is 100Mb/s. It is measured by copying a 20GB tar file from the host client to the backup server. The host client has a SATA raid 5 array holding the 1.4TB data and a single SATA as its system drive. The backup server has a holding disk of 230GB on a SATA drive. I don't use any software nor hardware compression.

    I want to backup the data of 1.4TB in 48 hours which is not unreasonable. However, so far it takes more than 4 days to backup 1.2TB of data. The log statistics is at the bottom of this message. It backups at around 3MB/s.

    I do a backup test of 86GB of data. It can backup at round 8MB/s.

    I'm using SUSE Linux 9.3 for the host client and the backup server.

    Could you please help me explaining the cause of this problem? What can I do to solve this problem? Am I doomed?

    I understand that backing up more than 1.4TB is a challenge, but it should be possible to do it in less than 60 hours. Please see my amanda statistics log below.

    -- Log for 1.2TB backup.

    These dumps were to tapes WeeklySet-4-01, WeeklySet-4-02, WeeklySet-4-03, WeeklySet-4-04.
    The next 4 tapes Amanda expects to use are: WeeklySet-1-02, WeeklySet-1-03, WeeklySet-2-01, WeeklySet-2-02.
    The next new tape already labelled is: WeeklySet-2-03.

    STATISTICS:
    Total Full Incr.
    -------- -------- --------
    Estimate Time (hrs:min) 0:00
    Run Time (hrs:min) 110:12
    Dump Time (hrs:min) 110:11 110:11 0:00
    Output Size (meg) 1128387.4 1128387.4 0.0
    Original Size (meg) 1124861.2 1124861.2 0.0
    Avg Compressed Size (%) -- -- --
    Filesystems Dumped 1 1 0
    Avg Dump Rate (k/s) 2912.9 2912.9 --

    Tape Time (hrs:min) 110:01 110:01 0:00
    Tape Size (meg) 1128387.4 1128387.4 0.0
    Tape Used (%) 292.3 292.3 0.0
    Filesystems Taped 1 1 0

    Chunks Taped 112839 112839 0
    Avg Tp Write Rate (k/s) 2917.4 2917.4 --

    USAGE BY TAPE:
    Label Time Size % Nb Nc
    WeeklySet-4-01 35:04 367222592k 92.9 0 35974
    WeeklySet-4-02 35:03 367222592k 92.9 0 35974
    WeeklySet-4-03 35:05 367222592k 92.9 0 35974
    WeeklySet-4-04 4:47 50190144k 12.7 1 4917

    NOTES:
    planner: Adding new disk host_client:/data1/tiff.
    driver: WARNING: This is not the first amdump run today. Enable the usetimestamps option in the configuration file if you want to run amdump more than once per calendar day.
    taper: no split_diskbuffer specified: using fallback split size of 10240kb to buffer host_client:/data1/tiff.0 in-memory
    taper: tape WeeklySet-4-01 kb 368373792 fm 35975 writing file: No space left on device
    taper: continuing host_client:/data1/tiff.0 on new tape from 368373760kb mark: [writing file: No space left on device]
    taper: tape WeeklySet-4-02 kb 368373792 fm 35975 writing file: No space left on device
    taper: continuing host_client:/data1/tiff.0 on new tape from 736747520kb mark: [writing file: No space left on device]
    taper: tape WeeklySet-4-03 kb 368373792 fm 35975 writing file: No space left on device
    taper: continuing host_client:/data1/tiff.0 on new tape from 1105121280kb mark: [writing file: No space left on device]
    taper: tape WeeklySet-4-04 kb 50347488 fm 4917 [OK]

    DUMP SUMMARY:
    DUMPER STATS TAPER STATS
    HOSTNAME DISK L ORIG-kB OUT-kB COMP% MMM:SS KB/s MMM:SS KB/s
    -------------------------- ------------------------------------- -------------
    host_client -data1/tiff 0 1151857910 1155468736 -- 6611:13 2903.8 6600:5 2917.4

    (brought to you by Amanda version 2.5.1p3)

    -- Log for 86GB backup.

    These dumps were to tape WeeklySet-3-04.
    The next 4 tapes Amanda expects to use are: WeeklySet-4-01, WeeklySet-4-02, WeeklySet-4-03, WeeklySet-4-04.
    The next 2 new tapes already labelled are: WeeklySet-2-04, WeeklySet-2-03.

    STATISTICS:
    Total Full Incr.
    -------- -------- --------
    Estimate Time (hrs:min) 0:00
    Run Time (hrs:min) 3:16
    Dump Time (hrs:min) 2:35 2:35 0:00
    Output Size (meg) 87281.8 87281.8 0.0
    Original Size (meg) 87281.8 87281.8 0.0
    Avg Compressed Size (%) -- -- --
    Filesystems Dumped 1 1 0
    Avg Dump Rate (k/s) 9582.1 9582.1 --

    Tape Time (hrs:min) 0:36 0:36 0:00
    Tape Size (meg) 87284.4 87284.4 0.0
    Tape Used (%) 22.6 22.6 0.0
    Filesystems Taped 1 1 0

    Chunks Taped 86 86 0
    Avg Tp Write Rate (k/s) 41463.8 41463.8 --

    USAGE BY TAPE:
    Label Time Size % Nb Nc
    WeeklySet-3-04 0:36 89376544k 22.6 1 86

    NOTES:
    planner: Adding new disk host_client:/data1/dat.
    driver: WARNING: This is not the first amdump run today. Enable the usetimestamps option in the configuration file if you want to run amdump more than once per calendar day.
    taper: tape WeeklySet-3-04 kb 89379296 fm 86 [OK]

    DUMP SUMMARY:
    DUMPER STATS TAPER STATS
    HOSTNAME DISK L ORIG-kB OUT-kB COMP% MMM:SS KB/s MMM:SS KB/s
    -------------------------- ------------------------------------- -------------
    host_client -/dat 0 89376520 89379264 -- 155:27 9582.1 35:56 41463.8

    (brought to you by Amanda version 2.5.1p3)

    Puthick
    Last edited by puthick; April 19th, 2007 at 08:01 PM.

  2. #2
    Join Date
    Nov 2005
    Location
    Canada
    Posts
    1,049

    Default

    Because your smaller dle do through the holding disk while the biggest go
    directly to tape without using the holding disk.

    You should split the big dle in many smaller dle with size smaller that half your holding disk size.

  3. #3

    Default

    Thanks. It works.

  4. #4

    Default very slow backup process : taking 35 hours for 900GB

    Quote Originally Posted by puthick View Post
    Thanks. It works.
    Hello,

    I am using amandav2.5.p2 and I have 30 production servers each having around 30 gB of data to be backed up. My backup server is having 800 GB of hardisk space.

    My backup schedule is to take full backups every week clearing the older full backup and do incrementals in between . My amanda conf is pated below.

    dumpuser "amandabackup"
    inparallel 25
    dumporder "tttttttttttttttttttttTTTT"
    taperalgo largestfit
    netusage 8000 Kbps
    dumpcycle 1 week
    runspercycle 5
    tapecycle 30 tapes
    bumpsize 500 Mb
    bumppercent 0
    bumpdays 1
    bumpmult 4
    etimeout 7200
    dtimeout 1800
    ctimeout 30
    tapebufs 40
    runtapes 4
    tpchanger "chg-disk"
    maxdumpsize -1
    tapetype HARDDISK
    chunksize 3Gb

    define tapetype HARDDISK {
    length 400000 mbytes
    }

    define dumptype global {
    auth "bsdtcp"
    index yes
    }

    define dumptype mydump {
    global
    strategy standard
    program "GNUTAR"
    exclude list "/etc/exclude_list"
    compress server fast
    record no
    estimate server
    maxdumps 1
    comprate 0.25, 0.25
    priority medium
    }

    define interface local {
    comment "a local disk"
    use 8000 kbps
    }

    define interface le0 {
    comment "10 Mbps ethernet"
    use 400 kbps


    Currently a full backup is taking 35+ hours to complete. And it is taking everyday full backup instead of incremental s.
    Is there anything wrong in my configuration??

    Any help appreciated.

  5. #5
    Join Date
    Nov 2005
    Location
    Canada
    Posts
    1,049

    Default

    set record to yes

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •