Results 1 to 9 of 9

Thread: How to recover from Amazon S3 bucket?

  1. #1
    Join Date
    Jul 2013
    Posts
    5

    Default How to recover from Amazon S3 bucket?

    [SOLVED]

    Hello,
    I'm trying to finish my first configuration of Amanda 3.3.0 (Ubuntu 12.04) but while I can dump/backup without problems on my Amazon S3 bucket, I cannot restore/recover from it and this is a BIG problem. ;)

    Here is what I do:

    Code:
    root@backup:# amrecover DailySet
    amrecover> sethost localhost
    200 Dump host set to localhost.
    amrecover> setdisk /etc
    200 Disk set to /etc.
    amrecover> add group
    Added file /group
    amrecover> extract
    
    Extracting files using tape drive changer on host localhost.
    The following tapes are needed: Daily-2
    
    Extracting files using tape drive changer on host localhost.
    Load tape Daily-2 now
    Continue [?/Y/n/s/d]? 
    amrecover - can't talk to tape server: service amidxtaped:
    I think the problem is on the host it's trying to use "Extracting files using tape drive changer on host localhost" while it should use the S3 bucket, unfortunately I cannot understand how to configure this thing.

    These are some of my configs, please ask if you need more.

    Can you help me?
    Thanks


    /etc/amanda/DailySet/amanda.conf

    Code:
    [...]
    # amazonaws S3
    device_property "S3_ACCESS_KEY" "MYKEY"  
    device_property "S3_SECRET_KEY" "MYSECRET" 
    device_property "S3_SSL" "YES"                                       
    
    
    define changer s3 {
        tpchanger "chg-multi:s3:MYBUCKET/DailySet/slot-{01,02,03,04,05,06,07,08,09,10}" # Number of tapes in your "tapecycle"
    }
    
    tpchanger "s3"
    changerfile  "s3-statefile"					     
    tapetype S3
    
    define tapetype S3 {
        comment "S3 Bucket"
        length 10240 gigabytes # Bucket size 10TB
    }
    
    define dumptype global {
        comment "Global definitions"
        auth "bsdtcp"
    }
     
    define application-tool app_amgtar {
        comment "amgtar"
        plugin  "amgtar"
    }
     
    define dumptype gui-base {
            global
            program "APPLICATION"
            application "app_amgtar"
            comment "gui base dumptype dumped with tar"
            compress none
            index yes
    }
    [...]
    amandahosts

    Code:
    localhost backup
    localhost root amindexd amdump amidxtaped
    Last edited by michelem; July 17th, 2013 at 06:24 AM. Reason: Solved

  2. #2
    Join Date
    Jul 2013
    Posts
    5

    Default

    Here is a debug log from amandad:

    Code:
    Wed Jul 17 11:14:24 2013: amandad: pid 966 ruid 34 euid 34 version 3.3.0: start at Wed Jul 17 11:14:24 2013
    Wed Jul 17 11:14:24 2013: amandad: security_getdriver(name=local) returns 0x7f181da53b40
    Wed Jul 17 11:14:24 2013: amandad: version 3.3.0
    Wed Jul 17 11:14:24 2013: amandad:     build: VERSION="Amanda-3.3.0"
    Wed Jul 17 11:14:24 2013: amandad:            BUILT_DATE="Mon Jul 16 21:42:01 UTC 2012" BUILT_MACH=""
    Wed Jul 17 11:14:24 2013: amandad:            BUILT_REV="4084" BUILT_BRANCH="3_3" CC="gcc"
    Wed Jul 17 11:14:24 2013: amandad:     paths: bindir="/usr/sbin" sbindir="/usr/sbin"
    Wed Jul 17 11:14:24 2013: amandad:            libexecdir="/usr/lib/amanda"
    Wed Jul 17 11:14:24 2013: amandad:            amlibexecdir="/usr/lib/amanda" mandir="/usr/share/man"
    Wed Jul 17 11:14:24 2013: amandad:            AMANDA_TMPDIR="/tmp/amanda"
    Wed Jul 17 11:14:24 2013: amandad:            AMANDA_DBGDIR="/var/log/amanda" CONFIG_DIR="/etc/amanda"
    Wed Jul 17 11:14:24 2013: amandad:            DEV_PREFIX="/dev/" RDEV_PREFIX="/dev/r"
    Wed Jul 17 11:14:24 2013: amandad:            DUMP="/sbin/dump" RESTORE="/sbin/restore" VDUMP=UNDEF
    Wed Jul 17 11:14:24 2013: amandad:            VRESTORE=UNDEF XFSDUMP="/sbin/xfsdump"
    Wed Jul 17 11:14:24 2013: amandad:            XFSRESTORE="/sbin/xfsrestore" VXDUMP=UNDEF VXRESTORE=UNDEF
    Wed Jul 17 11:14:24 2013: amandad:            SAMBA_CLIENT="/usr/bin/smbclient" GNUTAR="/bin/tar"
    Wed Jul 17 11:14:24 2013: amandad:            COMPRESS_PATH="/bin/gzip" UNCOMPRESS_PATH="/bin/gzip"
    Wed Jul 17 11:14:24 2013: amandad:             LPRCMD=UNDEF  MAILER=UNDEF
    Wed Jul 17 11:14:24 2013: amandad:            listed_incr_dir="/var/lib/amanda/gnutar-lists"
    Wed Jul 17 11:14:24 2013: amandad:     defs:  DEFAULT_SERVER="localhost" DEFAULT_CONFIG="DailySet1"
    Wed Jul 17 11:14:24 2013: amandad:            DEFAULT_TAPE_SERVER="localhost" DEFAULT_TAPE_DEVICE=""
    Wed Jul 17 11:14:24 2013: amandad:            NEED_STRSTR AMFLOCK_POSIX AMFLOCK_FLOCK AMFLOCK_LOCKF
    Wed Jul 17 11:14:24 2013: amandad:            AMFLOCK_LNLOCK SETPGRP_VOID AMANDA_DEBUG_DAYS=4 BSD_SECURITY
    Wed Jul 17 11:14:24 2013: amandad:            USE_AMANDAHOSTS CLIENT_LOGIN="backup" CHECK_USERID HAVE_GZIP
    Wed Jul 17 11:14:24 2013: amandad:            COMPRESS_SUFFIX=".gz" COMPRESS_FAST_OPT="--fast"
    Wed Jul 17 11:14:24 2013: amandad:            COMPRESS_BEST_OPT="--best" UNCOMPRESS_OPT="-dc"
    Wed Jul 17 11:14:24 2013: amandad: security_handleinit(handle=0x1b966f0, driver=0x7f181da53b40 (LOCAL))
    Wed Jul 17 11:14:24 2013: amandad: security_streaminit(stream=0x1b968c0, driver=0x7f181da53b40 (LOCAL))
    Wed Jul 17 11:14:24 2013: amandad: authenticated peer name is 'backup'
    Wed Jul 17 11:14:24 2013: amandad: accept recv REQ pkt:
    <<<<<
    SERVICE amidxtaped
    OPTIONS features=ffffffff9efefbffffffffff0f;auth=local;
    >>>>>
    Wed Jul 17 11:14:24 2013: amandad: creating new service: amidxtaped
    OPTIONS features=ffffffff9efefbffffffffff0f;auth=local;
    
    Wed Jul 17 11:14:24 2013: amandad: sending ACK pkt:
    <<<<<
    >>>>>
    Wed Jul 17 11:14:24 2013: amandad: tcpm_send_token: data is still flowing
    Wed Jul 17 11:14:24 2013: amandad: ERROR service amidxtaped: 
    Wed Jul 17 11:14:24 2013: amandad: ERROR service amidxtaped: ** (process:967): WARNING **: Use of qw(...) as parentheses is deprecated at /usr/lib/amanda/perl/Amanda/Recovery/Clerk.pm line 231.
    Wed Jul 17 11:14:24 2013: amandad: ERROR service amidxtaped: 
    Wed Jul 17 11:14:24 2013: amandad: ERROR service amidxtaped: 
    Wed Jul 17 11:14:24 2013: amandad: ERROR service amidxtaped: ** (process:967): WARNING **: Use of qw(...) as parentheses is deprecated at /usr/lib/amanda/perl/Amanda/Recovery/Clerk.pm line 265.
    Wed Jul 17 11:14:24 2013: amandad: ERROR service amidxtaped: 
    Wed Jul 17 11:14:24 2013: amandad: ERROR service amidxtaped: 
    Wed Jul 17 11:14:24 2013: amandad: ERROR service amidxtaped: ** (process:967): WARNING **: Use of qw(...) as parentheses is deprecated at /usr/lib/amanda/perl/Amanda/Recovery/Planner.pm line 231.
    Wed Jul 17 11:14:24 2013: amandad: ERROR service amidxtaped: 
    Wed Jul 17 11:14:24 2013: amandad: ERROR service amidxtaped: 
    Wed Jul 17 11:14:24 2013: amandad: ERROR service amidxtaped: ** (process:967): WARNING **: Use of qw(...) as parentheses is deprecated at /usr/lib/amanda/perl/Amanda/Recovery/Planner.pm line 342.
    Wed Jul 17 11:14:24 2013: amandad: ERROR service amidxtaped: 
    Wed Jul 17 11:14:24 2013: amandad: ERROR service amidxtaped: 
    Wed Jul 17 11:14:24 2013: amandad: ERROR service amidxtaped: ** (process:967): WARNING **: Use of qw(...) as parentheses is deprecated at /usr/lib/amanda/perl/Amanda/Recovery/Planner.pm line 393.
    Wed Jul 17 11:14:24 2013: amandad: ERROR service amidxtaped: 
    Wed Jul 17 11:14:24 2013: amandad: sending REP pkt:
    <<<<<
    ERROR service amidxtaped: 
    ERROR service amidxtaped: ** (process:967): WARNING **: Use of qw(...) as parentheses is deprecated at /usr/lib/amanda/perl/Amanda/Recovery/Clerk.pm line 231.
    ERROR service amidxtaped: 
    ERROR service amidxtaped: 
    ERROR service amidxtaped: ** (process:967): WARNING **: Use of qw(...) as parentheses is deprecated at /usr/lib/amanda/perl/Amanda/Recovery/Clerk.pm line 265.
    ERROR service amidxtaped: 
    ERROR service amidxtaped: 
    ERROR service amidxtaped: ** (process:967): WARNING **: Use of qw(...) as parentheses is deprecated at /usr/lib/amanda/perl/Amanda/Recovery/Planner.pm line 231.
    ERROR service amidxtaped: 
    ERROR service amidxtaped: 
    ERROR service amidxtaped: ** (process:967): WARNING **: Use of qw(...) as parentheses is deprecated at /usr/lib/amanda/perl/Amanda/Recovery/Planner.pm line 342.
    ERROR service amidxtaped: 
    ERROR service amidxtaped: 
    ERROR service amidxtaped: ** (process:967): WARNING **: Use of qw(...) as parentheses is deprecated at /usr/lib/amanda/perl/Amanda/Recovery/Planner.pm line 393.
    ERROR service amidxtaped: 
    CONNECT CTL 50 DATA 51
    
    >>>>>
    Wed Jul 17 11:14:24 2013: amandad: received ACK pkt:
    <<<<<
    >>>>>
    Wed Jul 17 11:14:24 2013: amandad: security_close(handle=0x1b966f0, driver=0x7f181da53b40 (LOCAL))
    Wed Jul 17 11:14:24 2013: amandad: security_stream_close(0x1b968c0)
    Wed Jul 17 11:14:24 2013: amandad: security_stream_seterr(0x1b968c0, write error to: Broken pipe)
    Wed Jul 17 11:14:25 2013: amandad: pid 966 finish time Wed Jul 17 11:14:25 2013

  3. #3
    Join Date
    Nov 2005
    Location
    Canada
    Posts
    1,049

    Default

    The amanda version you are using is not compatible with the perl you are using.
    You must fix the perl warning in the script or upgrade to a newer amanda.

  4. #4
    Join Date
    Jul 2013
    Posts
    5

    Default

    Yes! You are right! Upgrading amanda to latest version (3.3.4) solved the problem.
    Thanks to point me there.

    Now I think I should open a bug on Ubuntu because Amanda version on 12.04 is not compatible with the perl version on it.

  5. #5

    Default Not able to connect to S3

    Hi,
    I am trying to connect to AWS S3; man configuration file is very much same as michelem's file except the credentials. But I am consistently getting an error..

    amandabackup@pc025:~$ amdevcheck DailySet1 s3:backup
    MESSAGE While trying to read tapestart header: Access Denied (AccessDenied) (HTTP 403)
    DEVICE_ERROR
    VOLUME_ERROR

    Any pointer on what could be the problem. My credentials are correct.

    Best Regards.

  6. #6
    Join Date
    Jul 2013
    Posts
    5

    Default

    I think your problem it's related with AWS credentials, please check them or check the credential has right permissions to access/manage your bucket.

  7. #7
    Join Date
    Nov 2005
    Location
    Canada
    Posts
    1,049

    Default

    It look like you are not the owner of the 'backup' bucket.

    Bucket name must be unique, I'm sure someone else already create the 'backup' bucket.
    You must use a bucket name that nobody else use.

  8. #8

    Default

    Michele,
    Thanks for the reply.

    Best Regards,

  9. #9

    Default

    Martine,
    Yes, "backup" bucket name is already taken. It worked when I used different bucket name. But now I see another issue; suddenly "amlabel" command has become extremely slow. For first few iteration it was fast but later it would get stuck after first label...something like this..

    amandabackup@pc025:/etc/amanda/DailySet1$ for i in 1 2 3 4 5 6 7 8 9 10; do amlabel DailySet1 DailySet1-$i slot $i; done;
    Reading label...
    Found an empty tape.
    Writing label 'DailySet1-1'...
    Checking label...
    Success!
    Reading label...
    Found an empty tape.
    Writing label 'DailySet1-2'...

    What could be the issue here?

    Best Regards,

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •