Page 1 of 3 123 LastLast
Results 1 to 10 of 21

Thread: amdump segfault in 2.5.1b

  1. #1
    Join Date
    Aug 2006
    Posts
    34

    Default amdump segfault in 2.5.1b

    This week I installed Amanda 2.5.1b from RPM on a RedHat EL4 x86_64 machine. When I run amdump, I get the following:

    Code:
     ~]$ sudo su amandabackup -c "amdump  ISDaily2.5"
    *** glibc detected *** double free or corruption (fasttop): 0x0805d658 ***
    This shows up in dmesg:
    Code:
    driver[11483]: segfault at 0000000000000011 rip 0000000000707f5b rsp 00000000ffff1bec error 4
    excerpt from /var/lib/amanda/ISDaily/amdump.1:
    Code:
    taper: DONE [idle wait: 2.637 secs]
    /usr/sbin/amdump: line 109: 11847 Done                    $libexecdir/planner$SUF $conf "$@"
         11848 Aborted                 | $libexecdir/driver$SUF $conf "$@"
    taper: writing end marker. [ISDaily04 OK kb 0 fm 0]
    The only other record I see of the glibc double free error on this forum is a user who didn't have the right version of Tar. I'm running tar-1.14-9.RHEL4. Could this be the issue?

    What other info can I provide to help debug? Output of amadmin version?

    thanks!
    B.

  2. #2
    Join Date
    Oct 2005
    Posts
    1,559

    Default

    Which version of 2.5.1? Is it 2.5.1b1 or 2.5.1b2? 2.5.1b2 addresses significant
    memory allocation issues.

    Please use tar 1.15.1 version. Some tar 1.14 versions have known to problems
    that will cause restoration issues.

    Are you using rpms from [url]http://www.zmanda.com/downloads.html[/url] or
    are you building on your own? If you have compiled on your own,
    please provide output of "amadmin xx version"

    Thanks,
    Paddy

  3. #3
    Join Date
    Aug 2006
    Posts
    34

    Default

    Hi Paddy - it's 2.5.1b2 from the zmanda downloads page:

    output of 'rpm -qa | grep amanda':
    amanda-backup_client-2.5.1b2-1.rhel4
    amanda-backup_server-2.5.1b2-1.rhel4

    I'll update Tar to 1.15.1 and see how it goes.

  4. #4
    Join Date
    Aug 2006
    Posts
    34

    Default

    Hm... same result with 1.15.1:

    Code:
    # su amandabackup -c "amdump  ISDaily2.5"
    *** glibc detected *** double free or corruption (fasttop): 0x0805d658 ***
    # tar --version
    tar (GNU tar) 1.15.1

  5. #5
    Join Date
    Oct 2005
    Posts
    1,559

    Default

    Can you please run amdump with -x option?

    Change the first line in amdump script to
    #!/bin/sh -x

    Depending on which command in the script causes this problem, you will
    have to run the command under "valgrind". Please note that you will
    have to run as a root.

    Thanks for your help in debugging the problem. Unfortunately, we are not
    seeing this problem.

    Paddy

  6. #6
    Join Date
    Aug 2006
    Posts
    34

    Default

    # su amandabackup -c "amdump ISDaily2.5"
    Code:
    + prefix=/usr
    + exec_prefix=/usr
    + sbindir=/usr/sbin
    + libexecdir=/usr/lib/amanda
    + confdir=/etc/amanda
    + PATH=/usr/sbin:/usr/lib/amanda:/usr/bin:/bin:/usr/sbin:/sbin:/usr/ucb
    + export PATH
    + USE_VERSION_SUFFIXES=no
    + test no = yes
    + SUF=
    + '[' 1 -lt 1 ']'
    + conf=ISDaily2.5
    + '[' '!' -d /etc/amanda/ISDaily2.5 ']'
    + shift
    + cd /etc/amanda/ISDaily2.5
    ++ amgetconf ISDaily2.5 logdir
    + logdir=/var/lib/amanda/ISDaily
    + '[' 0 -ne 0 ']'
    + errfile=/var/lib/amanda/ISDaily/amdump
    ++ amgetconf ISDaily2.5 tapecycle
    + tapecycle=15
    + '[' 0 -ne 0 ']'
    ++ amgetconf ISDaily2.5 dumpuser
    + dumpuser=amandabackup
    + '[' 0 -ne 0 ']'
    + runuser=amandabackup
    + '[' 0 -ne 0 ']'
    + test -f hold
    + test -f /var/lib/amanda/ISDaily/amdump
    + test -f /var/lib/amanda/ISDaily/log
    + umask 077
    + touch /var/lib/amanda/ISDaily/amdump
    + exec
    *** glibc detected *** double free or corruption (fasttop): 0x0805d658 ***
    This seems to imply that either planner or dumper are at fault since the last line in amdump to run is exec:
    Code:
    touch $errfile
    exec </dev/null 2>>$errfile 1>&2
    echo "amdump: start at `date`"
    echo "amdump: datestamp `date +%Y%m%d`"
    $libexecdir/planner$SUF $conf "$@" | $libexecdir/driver$SUF $conf "$@"
    echo "amdump: end at `date`"
    I removed the suid bit for dumper and ran Valgrind but am not sure what I'm looking for:
    Code:
    #  valgrind  --tool=memcheck /usr/lib/amanda/dumper ISDaily2.5
    ==15828== Memcheck, a memory error detector for x86-linux.
    ==15828== Copyright (C) 2002-2004, and GNU GPL'd, by Julian Seward et al.
    ==15828== Using valgrind-2.2.0, a program supervision framework for x86-linux.
    ==15828== Copyright (C) 2000-2004, and GNU GPL'd, by Julian Seward et al.
    ==15828== For more details, rerun with: -v
    ==15828== 
    ==15828== Warning: invalid file descriptor 820 in syscall close()
    ==15828== Warning: invalid file descriptor 821 in syscall close()
    ==15828== Warning: invalid file descriptor 822 in syscall close()
    ==15828==    Use --log-fd=<number> to select an alternative log fd.
    ==15828== Warning: invalid file descriptor 823 in syscall close()
    ==15828== Warning: invalid file descriptor 824 in syscall close()
    ==15828== Warning: invalid file descriptor 825 in syscall close()
    ==15828== Warning: invalid file descriptor 826 in syscall close()
    ==15828== Warning: invalid file descriptor 827 in syscall close()
    ==15828== Warning: invalid file descriptor 828 in syscall close()
    < snip >
    15828== Warning: invalid file descriptor 1020 in syscall close()
    ==15828== Warning: invalid file descriptor 1021 in syscall close()
    ==15828== Warning: invalid file descriptor 1022 in syscall close()
    ==15828== Warning: invalid file descriptor 1023 in syscall close()
    dumper: pid 15828 executable /usr/lib/amanda/dumper version 2.5.1b2
    dumper> (null)
    ==15828== Invalid read of size 1
    ==15828==    at 0x1B9651E0: quote (token.c:178)
    ==15828==    by 0x1B9653A6: squote (token.c:161)
    ==15828==    by 0x804C79A: main (dumper.c:514)
    ==15828==  Address 0x1B9ADAD0 is 0 bytes inside a block of size 128 free'd
    ==15828==    at 0x1B904EA5: free (vg_replace_malloc.c:153)
    ==15828==    by 0x1B92D56B: getcmd (server_util.c:74)
    ==15828==    by 0x804C834: main (dumper.c:335)
    ==15828== 
    ==15828== Invalid read of size 1
    ==15828==    at 0x1B96524F: quote (token.c:205)
    ==15828==    by 0x1B9653A6: squote (token.c:161)
    ==15828==    by 0x804C79A: main (dumper.c:514)
    ==15828==  Address 0x1B9ADAD0 is 0 bytes inside a block of size 128 free'd
    ==15828==    at 0x1B904EA5: free (vg_replace_malloc.c:153)
    ==15828==    by 0x1B92D56B: getcmd (server_util.c:74)
    ==15828==    by 0x804C834: main (dumper.c:335)
    BAD-COMMAND 
    dumper> (null)==15828== 
    ==15828== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 24 from 2)
    ==15828== malloc/free: in use at exit: 29448 bytes in 496 blocks.
    ==15828== malloc/free: 656 allocs, 160 frees, 42370 bytes allocated.
    ==15828== For a detailed leak analysis,  rerun with: --leak-check=yes
    ==15828== For counts of detected errors, rerun with: -v
    Last edited by bethany; August 10th, 2006 at 05:02 AM.

  7. #7
    Join Date
    Aug 2006
    Posts
    34

    Default

    I think I'm going to rebuild this machine in 32-bit mode and see if it makes a difference...

  8. #8
    Join Date
    Aug 2006
    Posts
    34

    Default still segfaulting in 32bit mode

    Interesting - I rebuilt the machine with the 32bit version of RHEL4 and it is still dumping:

    $ sudo su amandabackup -c "amdump ISDaily2.5"
    *** glibc detected *** double free or corruption (fasttop): 0x09f6a658 ***
    $ uname -a
    Linux 2.6.9-34.0.2.ELsmp #1 SMP Fri Jun 30 10:33:58 EDT 2006 i686 athlon i386 GNU/Linux

  9. #9
    Join Date
    Oct 2005
    Location
    Bay Area, CA
    Posts
    124

    Default

    Hi,
    I just downloaded amanda-2.5.1b2 and compiled. I don't see the problem on Red Hat:
    Linux boston.zmanda.com 2.6.9-11.ELsmp #1 SMP Fri May 20 18:26:27 EDT 2005 i686 i686 i386 GNU/Linux
    [amandabackup@boston DailySet1]$ amcheck DailySet1
    Amanda Tape Server Host Check
    -----------------------------
    Holding disk /backups/holding: 5707536 KB disk space available, that's plenty
    slot 8: read label `DailySet18', date `20060628101832'
    NOTE: skipping tape-writable test
    Tape DailySet18 label ok
    Server check took 0.112 seconds

    Amanda Backup Client Hosts Check
    --------------------------------
    Client check: 1 host checked in 0.070 seconds, 0 problems found

    (brought to you by Amanda 2.5.1b2)
    [amandabackup@boston DailySet1]$ amdump DailySet1
    [amandabackup@boston DailySet1]$




    Can you please do "amcheck ISDaily2.5" to see if there is any error?
    Next, please list the following files:
    1) /tmp/amanda/amandad/amandad.*.debug.
    2) /etc/amanda/ISDaily2.5/amdump.1
    3) /etc/amanda/ISDaily2.5/log.xxx <-- the latest one.

    Thanks!

    Kevin Till
    Zmanda

  10. #10
    Join Date
    Aug 2006
    Posts
    34

    Default

    OK, I've applied the latest round of Redhat errata that hit our satellite server last night and and booted into the new kernel.

    amcheck:
    $ sudo su amandabackup -c "amcheck ISDaily2.5"
    Code:
    Amanda Tape Server Host Check
    -----------------------------
    Holding disk /amandahold: 395262320 kB disk space available, using 393165168 kB
    slot 8: read label `ISDaily02', date `20060812004501'
    cannot overwrite active tape ISDaily02
    slot 9: Found a non-amanda tape, will label it `ISDaily03'.
    NOTE: skipping tape-writable test
    Found a brand new tape, will label it ISDaily03.
    NOTE: conf info dir /var/lib/amanda/ISDaily/curinfo does not exist
    NOTE: it will be created on the next run.
    NOTE: index dir /var/lib/amanda/ISDaily/index does not exist
    NOTE: it will be created on the next run.
    Server check took 0.423 seconds
    
    Amanda Backup Client Hosts Check
    --------------------------------
    WARNING: Usage of fully qualified hostname recommended for Client localhost.
    WARNING: Usage of fully qualified hostname recommended for Client localhost.
    WARNING: Usage of fully qualified hostname recommended for Client localhost.
    Client check: 2 hosts checked in 0.182 seconds, 0 problems found
    
    (brought to you by Amanda 2.5.1b2)
    [bj56@middenheap-dev ~]$ sudo su amandabackup -c "amdump ISDaily2.5"
    *** glibc detected *** double free or corruption (fasttop): 0x09dba658 ***

    1) /tmp/amanda/amandad/amandad.*.debug.

    $ sudo cat /tmp/amanda/amandad/amandad.20060812104701.debug
    Code:
    amandad: debug 1 pid 4104 ruid 502 euid 502: start at Sat Aug 12 10:47:01 2006
    security_getdriver(name=BSD) returns 0x6fb0e0
    amandad: version 2.5.1b2
    amandad: build: VERSION="Amanda-2.5.1b2"
    amandad:        BUILT_DATE="Fri Jul 28 18:18:32 PDT 2006"
    amandad:        BUILT_MACH="Linux rocky.zmanda.com 2.6.9-22.0.2.ELsmp #1 SMP Thu Jan 5 17:13:01 EST 2006 i686 i686 i386 GNU/Linux"
    amandad:        CC="gcc"
    amandad:        CONFIGURE_COMMAND="'./configure' '--build=i386-redhat-linux' '--prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--libexecdir=/usr/lib/amanda' '--datadir=/usr/share' '--sysconfdir=/etc' '--sharedstatedir=/var/lib/amanda' '--localstatedir=/var/lib/amanda' '--libdir=/usr/lib' '--includedir=/usr/include' '--infodir=/usr/info' '--mandir=/usr/share/man' '--with-gnutar=/bin/tar' '--with-gnutar-listdir=/var/lib/amanda/gnutar-lists' '--with-dumperdir=/usr/lib/amanda' '--with-index-server=localhost' '--with-tape-server=localhost' '--with-user=amandabackup' '--with-group=disk' '--with-owner=paddy' '--with-fqdn' '--with-bsd-security' '--with-bsdtcp-security' '--with-bsdudp-security' '--with-ssh-security' '--with-assertions'"
    amandad: paths: bindir="/usr/bin" sbindir="/usr/sbin"
    amandad:        libexecdir="/usr/lib/amanda" mandir="/usr/share/man"
    amandad:        AMANDA_TMPDIR="/tmp/amanda" AMANDA_DBGDIR="/tmp/amanda"
    amandad:        CONFIG_DIR="/etc/amanda" DEV_PREFIX="/dev/"
    amandad:        RDEV_PREFIX="/dev/" DUMP="/sbin/dump"
    amandad:        RESTORE="/sbin/restore" VDUMP=UNDEF VRESTORE=UNDEF
    amandad:        XFSDUMP=UNDEF XFSRESTORE=UNDEF VXDUMP=UNDEF VXRESTORE=UNDEF
    amandad:        SAMBA_CLIENT="/usr/bin/smbclient" GNUTAR="/bin/tar"
    amandad:        COMPRESS_PATH="/bin/gzip" UNCOMPRESS_PATH="/bin/gzip"
    amandad:        LPRCMD="/usr/bin/lpr" MAILER="/usr/bin/Mail"
    amandad:        listed_incr_dir="/var/lib/amanda/gnutar-lists"
    amandad: defs:  DEFAULT_SERVER="localhost" DEFAULT_CONFIG="DailySet1"
    amandad:        DEFAULT_TAPE_SERVER="localhost" HAVE_MMAP HAVE_SYSVSHM
    amandad:        LOCKING=POSIX_FCNTL SETPGRP_VOID ASSERTIONS DEBUG_CODE
    amandad:        AMANDA_DEBUG_DAYS=4 BSD_SECURITY RSH_SECURITY USE_AMANDAHOSTS
    amandad:        CLIENT_LOGIN="amandabackup" FORCE_USERID HAVE_GZIP
    amandad:        COMPRESS_SUFFIX=".gz" COMPRESS_FAST_OPT="--fast"
    amandad:        COMPRESS_BEST_OPT="--best" UNCOMPRESS_OPT="-dc"
    amandad: time 0.000: dgram_recv(dgram=(nil), timeout=7389296, fromaddr=0x1)
    amandad: (sockaddr_in *)0x70c070 = { 0, 0, 0.0.0.0 }
    security_handleinit(handle=0x99cbe88, driver=0x6fb0e0 (BSD))
    amandad: time 0.011: accept recv REQ pkt:
    <<<<<
    SERVICE noop
    OPTIONS features=fffffeff9ffeffffff7f;
    >>>>>
    amandad: time 0.012: creating new service: noop
    OPTIONS features=fffffeff9ffeffffff7f;
    
    amandad: time 0.013: sending ACK pkt:
    <<<<<
    >>>>>
    amandad: dgram_send_addr(addr=0xbfe629a0, dgram=0x6fc084)
    amandad: (sockaddr_in *)0xbfe629a0 = { 2, -5117, 127.0.0.1 }
    amandad: dgram_send_addr: 0x6fc084->socket = 0
    amandad: time 0.031: sending REP pkt:
    <<<<<
    OPTIONS features=fffffeff9ffeffffff7f;
    >>>>>
    amandad: dgram_send_addr(addr=0xbfe62ae0, dgram=0x6fc084)
    amandad: (sockaddr_in *)0xbfe62ae0 = { 2, -5117, 127.0.0.1 }
    amandad: dgram_send_addr: 0x6fc084->socket = 0
    amandad: time 0.031: dgram_recv(dgram=(nil), timeout=7389296, fromaddr=0xb26401)
    amandad: (sockaddr_in *)0x70c070 = { 2, -5117, 127.0.0.1 }
    amandad: time 0.031: received ACK pkt:
    <<<<<
    >>>>>
    security_close(handle=0x99cbe88, driver=0x6fb0e0 (BSD))
    amandad: time 0.032: dgram_recv(dgram=(nil), timeout=7389296, fromaddr=0xb2433a)
    amandad: (sockaddr_in *)0x70c070 = { 2, -5117, 127.0.0.1 }
    security_handleinit(handle=0x99cbe88, driver=0x6fb0e0 (BSD))
    amandad: time 0.032: accept recv REQ pkt:
    <<<<<
    SERVICE selfcheck
    OPTIONS features=fffffeff9ffeffffff7f;maxdumps=1;hostname=localhost;config=ISDaily2.5;
    GNUTAR /var 0 OPTIONS |;auth=BSD;compress-fast;index;
    GNUTAR / 0 OPTIONS |;auth=BSD;compress-fast;index;
    >>>>>
    amandad: time 0.032: creating new service: selfcheck
    OPTIONS features=fffffeff9ffeffffff7f;maxdumps=1;hostname=localhost;config=ISDaily2.5;
    GNUTAR /var 0 OPTIONS |;auth=BSD;compress-fast;index;
    GNUTAR / 0 OPTIONS |;auth=BSD;compress-fast;index;
    
    amandad: time 0.033: sending ACK pkt:
    <<<<<
    >>>>>
    amandad: dgram_send_addr(addr=0xbfe629a0, dgram=0x6fc084)
    amandad: (sockaddr_in *)0xbfe629a0 = { 2, -5117, 127.0.0.1 }
    amandad: dgram_send_addr: 0x6fc084->socket = 0
    amandad: time 0.133: sending REP pkt:
    <<<<<
    OPTIONS features=fffffeff9ffeffffff7f;
    OK /var
    OK /var
    OK /var
    OK /
    OK /
    OK /
    OK /usr/lib/amanda/runtar executable
    OK /bin/tar executable
    OK /var/lib/amanda/gnutar-lists/. read/writable
    OK /etc/amandates read/writable
    OK /bin/gzip executable
    OK /dev/null read/writable
    OK /tmp/amanda has more than 64 KB available.
    OK /tmp/amanda has more than 64 KB available.
    OK /etc has more than 64 KB available.
    >>>>>
    amandad: dgram_send_addr(addr=0xbfe62ae0, dgram=0x6fc084)
    amandad: (sockaddr_in *)0xbfe62ae0 = { 2, -5117, 127.0.0.1 }
    amandad: dgram_send_addr: 0x6fc084->socket = 0
    amandad: time 0.134: dgram_recv(dgram=(nil), timeout=7389296, fromaddr=0xbfe62aac)
    amandad: (sockaddr_in *)0x70c070 = { 2, -5117, 127.0.0.1 }
    amandad: time 0.134: received ACK pkt:
    <<<<<
    >>>>>
    security_close(handle=0x99cbe88, driver=0x6fb0e0 (BSD))
    amandad: time 30.127: pid 4104 finish time Sat Aug 12 10:47:31 2006

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •