Results 1 to 9 of 9

Thread: Amanda Client hangs server for about 20 mins during Index creation

  1. #1
    Join Date
    Nov 2013
    Posts
    9

    Default Amanda Client hangs server for about 20 mins during Index creation

    Hello,

    I'm having an issue with one of our 9 servers that we backup with a central Amanda server. For some reason, this particular mail server becomes unresponsive (all mail services timeout, console unresponsive etc) for about 20 minutes during the backup process. It looks like during that time, the client is doing the indexing step. I have servers with larger partitions and less RAM and they backup without a hitch. The backup does finish successfully from what i can tell, but I can't have this production mail server down for 20 mins each weeknight. Below are the particulars. Please let me know any further info that would help out in troubleshooting this issue.

    Amanda Server
    OS: FreeBSD 7.2-STABLE
    Amanda Version: 3.3.2

    Problem Server
    OS: FreeBSD 9.1
    Amanda Version: 3.3.2
    RAM: 4 GB
    NIC: Gigabit Ethernet

    sendbackup.20131107012147.debug log from the Problem Server:

    Code:
    Thu Nov  7 01:21:47 2013: thd-0x28837800: sendbackup: pid 95559 ruid 140 euid 140 version 3.3.2: start at Thu Nov  7 01:21:47 2013
    Thu Nov  7 01:21:47 2013: thd-0x28837800: sendbackup: Version 3.3.2
    Thu Nov  7 01:21:47 2013: thd-0x28837800: sendbackup: pid 95559 ruid 140 euid 140 version 3.3.2: rename at Thu Nov  7 01:21:47 2013
    Thu Nov  7 01:21:47 2013: thd-0x28837800: sendbackup:   Parsed request as: program `DUMP'
    Thu Nov  7 01:21:47 2013: thd-0x28837800: sendbackup:                      disk `/dev/da0p2'
    Thu Nov  7 01:21:47 2013: thd-0x28837800: sendbackup:                      device `/dev/da0p2'
    Thu Nov  7 01:21:47 2013: thd-0x28837800: sendbackup:                      level 1
    Thu Nov  7 01:21:47 2013: thd-0x28837800: sendbackup:                      since NODATE
    Thu Nov  7 01:21:47 2013: thd-0x28837800: sendbackup:                      options `'
    Thu Nov  7 01:21:47 2013: thd-0x28837800: sendbackup:                      datapath `AMANDA'
    Thu Nov  7 01:21:47 2013: thd-0x28837800: sendbackup: start: imap:/dev/da0p2 lev 1
    Thu Nov  7 01:21:47 2013: thd-0x28837800: sendbackup: Spawning "/usr/bin/gzip /usr/bin/gzip --fast" in pipeline
    Thu Nov  7 01:21:47 2013: thd-0x28837800: sendbackup: dump: pid 95561: /usr/bin/gzipThu Nov  7 01:21:47 2013: thd-0x28837800: sendbackup:  --fastThu Nov  7 01:21:47 2013: thd-0x28837800: sendbackup: 
    Thu Nov  7 01:21:47 2013: thd-0x28837800: sendbackup: dumping device '/dev/da0p2' with 'ufs'
    Thu Nov  7 01:21:47 2013: thd-0x28837800: sendbackup: Spawning "/sbin/dump dump 1ubLshf 64 1048576 0 - /dev/da0p2" in pipeline
    Thu Nov  7 01:21:47 2013: thd-0x28837800: sendbackup: Started backup
    Thu Nov  7 01:21:47 2013: thd-0x28837800: sendbackup: Started index creator: "/sbin/restore -tvf - 2>&1 | sed -e '
    s/^leaf[        ]*[0-9]*[       ]*\.//
    t
    /^dir[  ]/ {
    s/^dir[         ]*[0-9]*[       ]*\.//
    s%$%/%
    t
    }
    d
    '"
    Thu Nov  7 02:12:41 2013: thd-0x28837800: sendbackup:  90:  normal(|):   DUMP: Date of this level 1 dump: Thu Nov  7 02:12:41 2013
    Thu Nov  7 02:12:41 2013: thd-0x28837800: sendbackup:  90:  normal(|):   DUMP: Date of last level 0 dump: Wed Nov  6 02:09:01 2013
    Thu Nov  7 02:12:41 2013: thd-0x28837800: sendbackup:  90:  normal(|):   DUMP: Dumping snapshot of /dev/da0p2 (/) to standard output
    Thu Nov  7 02:12:52 2013: thd-0x28837800: sendbackup:  90:  normal(|):   DUMP: mapping (Pass I) [regular files]
    Thu Nov  7 02:18:30 2013: thd-0x28837800: sendbackup:  90:  normal(|):   DUMP: mapping (Pass II) [directories]
    Thu Nov  7 02:20:54 2013: thd-0x28837800: sendbackup:  90:  normal(|):   DUMP: estimated 3580006 tape blocks.
    Thu Nov  7 02:20:54 2013: thd-0x28837800: sendbackup:  90:  normal(|):   DUMP: dumping (Pass III) [directories]
    Thu Nov  7 02:21:28 2013: thd-0x28837800: sendbackup:  90:  normal(|):   DUMP: dumping (Pass IV) [regular files]
    Thu Nov  7 02:25:54 2013: thd-0x28837800: sendbackup:  90:  normal(|):   DUMP: 38.08% done, finished in 0:08 at Thu Nov  7 02:34:01 2013
    Thu Nov  7 02:30:54 2013: thd-0x28837800: sendbackup:  90:  normal(|):   DUMP: 98.03% done, finished in 0:00 at Thu Nov  7 02:31:06 2013
    Thu Nov  7 02:30:55 2013: thd-0x28837800: sendbackup:  43:    size(|):   DUMP: DUMP: 3520526 tape blocks
    Thu Nov  7 02:30:55 2013: thd-0x28837800: sendbackup:  90:  normal(|):   DUMP: finished in 601 seconds, throughput 5857 KBytes/sec
    Thu Nov  7 02:30:55 2013: thd-0x28837800: sendbackup:  90:  normal(|):   DUMP: level 1 dump on Thu Nov  7 02:12:41 2013
    Thu Nov  7 02:30:55 2013: thd-0x28837800: sendbackup:  90:  normal(|):   DUMP: DUMP IS DONE
    Thu Nov  7 02:30:56 2013: thd-0x28837800: sendbackup: Index created successfully
    Thu Nov  7 02:31:01 2013: thd-0x28837800: sendbackup: Sending SIGHUP to dump process 95563
    Thu Nov  7 02:31:06 2013: thd-0x28837800: sendbackup: Sending SIGKILL to dump process 95563
    Thu Nov  7 02:31:11 2013: thd-0x28837800: sendbackup: Parsed backup messages
    Thu Nov  7 02:31:11 2013: thd-0x28837800: sendbackup: pid 95559 finish time Thu Nov  7 02:31:11 2013
    Any help would be greatly appreciated.

    Matt Rauch

  2. #2
    Join Date
    Nov 2005
    Location
    Canada
    Posts
    1,019

    Default

    If it's a disk I/O problem, try the solution from [URL="https://forums.zmanda.com/showthread.php?2543-Using-ionice-to-reduce-disk-I-O-impact"]https://forums.zmanda.com/showthread.php?2543-Using-ionice-to-reduce-disk-I-O-impact[/URL]

  3. #3
    Join Date
    Nov 2013
    Posts
    9

    Default

    Quote Originally Posted by martineau View Post
    If it's a disk I/O problem, try the solution from [URL="https://forums.zmanda.com/showthread.php?2543-Using-ionice-to-reduce-disk-I-O-impact"]https://forums.zmanda.com/showthread.php?2543-Using-ionice-to-reduce-disk-I-O-impact[/URL]
    I could be wrong but FreeBSD is not using xinetd on the server or the client and I don't think FreeBSD has ionice. If I am incorrect, if someone could point me in the right direction I'd gladly have a look. This idea certainly seems like the path to take.

    Matt Rauch

  4. #4
    Join Date
    Nov 2005
    Location
    Canada
    Posts
    1,019

    Default

    Another possibility is to slow down the transfer to the server:

    In the dumptype:
    encrypt client
    client-encrypt "/usr/bin/pv-300k"

    $cat /usr/bin/pv-300k
    #!/bin/bash
    exec /usr/bin/pv -L300k -q

    You need the pv utility.

  5. #5
    Join Date
    Nov 2013
    Posts
    9

    Default

    Quote Originally Posted by martineau View Post
    Another possibility is to slow down the transfer to the server:

    In the dumptype:
    encrypt client
    client-encrypt "/usr/bin/pv-300k"

    $cat /usr/bin/pv-300k
    #!/bin/bash
    exec /usr/bin/pv -L300k -q

    You need the pv utility.
    Ok, so I have this as my current dumptype:

    Code:
    define dumptype hard-disk-dump {
        global
        auth "bsd"
        comment "Back up to hard disk instead of tape - using dump"
        holdingdisk auto
        index yes
        priority high
        record yes
    }
    So I'm thinking I can just copy that, give it a different name and add in the lines you mentioned? so something like this:

    Code:
    define dumptype hard-disk-dump-test {
        global
        auth "bsd"
        comment "Back up to hard disk instead of tape - using dump"
        encrypt client
        client-encrypt "/usr/bin/pv-300k"
        holdingdisk auto
        index yes
        priority high
        record yes
    }
    Then I install pv from ports and confirm that /usr/bin/pv-300k exists?

    Matt Rauch

  6. #6
    Join Date
    Nov 2013
    Posts
    9

    Default

    Quote Originally Posted by martineau View Post
    Another possibility is to slow down the transfer to the server:

    In the dumptype:
    encrypt client
    client-encrypt "/usr/bin/pv-300k"

    $cat /usr/bin/pv-300k
    #!/bin/bash
    exec /usr/bin/pv -L300k -q

    You need the pv utility.
    Ok sorry I didn't fully understand what was happening in your example. So I would create the pv-300k script myself. Now this example would slow the dump to 300kbps to the amanda server correct? The issue I think here is that the dump itself doesn't hang the box, it is while it is creating the index that the box becomes unresponsive. Would your suggestion have any effect on the index step or only on the dump itself?

    Matt Rauch

  7. #7
    Join Date
    Nov 2005
    Location
    Canada
    Posts
    1,019

    Default

    If the dump process is doing a lot of I/O without sending data, then the pv solution will not work.

    You can try to set the nice command (like ionice) in the xinet or inetd configuration file.

    You never tell why the box become unresponsive, it is because of disk I/O, cpu, memory, ...

  8. #8
    Join Date
    Nov 2013
    Posts
    9

    Default

    I don't know the cause as to why the box becomes unresponsive, that is what I am trying to determine. If you have any suggestions on how I can do that, I'm definitely ready to try anything. I'm assuming since it appears to happen during the index creation that is it either memory or I/O. Seems like a long time for the indexing as well, as you can see in my log it takes about 50 minutes for the indexing to complete. During that 50 minutes, its the last 20 minutes when the box becomes unresponsive. I'll do some digging into the nice command. Am I right to assume it is to be used on the Client box?

    Thanks for your assistance so far!

    Matt Rauch

  9. #9
    Join Date
    Nov 2013
    Posts
    9

    Default

    So it appears to be the snapshot creation step during the DUMP process that is causing the box to hang for about 15 minutes. I'd like to be able to have Amanda use the -C flag for the dump command if possible as I have read that can improve snapshot time. Or alternatively, for the time being I'd like to remove the -L flag so that snapshots are not created. Since it's a mail server I can't have the uptime interrupted nightly. Does anyone know a way of doing this?

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •