How to recover from Amazon S3 bucket?

[michelem]

[SOLVED]

Hello,
I'm trying to finish my first configuration of Amanda 3.3.0 (Ubuntu 12.04) but while I can dump/backup without problems on my Amazon S3 bucket, I cannot restore/recover from it and this is a BIG problem. ;)

Here is what I do:

[email protected]:# amrecover DailySet
amrecover> sethost localhost
200 Dump host set to localhost.
amrecover> setdisk /etc
200 Disk set to /etc.
amrecover> add group
Added file /group
amrecover> extract

Extracting files using tape drive changer on host localhost.
The following tapes are needed: Daily-2

Extracting files using tape drive changer on host localhost.
Load tape Daily-2 now
Continue [?/Y/n/s/d]? 
amrecover - can't talk to tape server: service amidxtaped:

I think the problem is on the host it's trying to use "Extracting files using tape drive changer on host localhost" while it should use the S3 bucket, unfortunately I cannot understand how to configure this thing.

These are some of my configs, please ask if you need more.

Can you help me?
Thanks


/etc/amanda/DailySet/amanda.conf

[...]
# amazonaws S3
device_property "S3_ACCESS_KEY" "MYKEY"  
device_property "S3_SECRET_KEY" "MYSECRET" 
device_property "S3_SSL" "YES"                                       


define changer s3 {
    tpchanger "chg-multi:s3:MYBUCKET/DailySet/slot-{01,02,03,04,05,06,07,08,09,10}" # Number of tapes in your "tapecycle"
}

tpchanger "s3"
changerfile  "s3-statefile"					     
tapetype S3

define tapetype S3 {
    comment "S3 Bucket"
    length 10240 gigabytes # Bucket size 10TB
}

define dumptype global {
    comment "Global definitions"
    auth "bsdtcp"
}
 
define application-tool app_amgtar {
    comment "amgtar"
    plugin  "amgtar"
}
 
define dumptype gui-base {
        global
        program "APPLICATION"
        application "app_amgtar"
        comment "gui base dumptype dumped with tar"
        compress none
        index yes
}
[...]

amandahosts

localhost backup
localhost root amindexd amdump amidxtaped

[michelem]

Here is a debug log from amandad:

Wed Jul 17 11:14:24 2013: amandad: pid 966 ruid 34 euid 34 version 3.3.0: start at Wed Jul 17 11:14:24 2013
Wed Jul 17 11:14:24 2013: amandad: security_getdriver(name=local) returns 0x7f181da53b40
Wed Jul 17 11:14:24 2013: amandad: version 3.3.0
Wed Jul 17 11:14:24 2013: amandad:     build: VERSION="Amanda-3.3.0"
Wed Jul 17 11:14:24 2013: amandad:            BUILT_DATE="Mon Jul 16 21:42:01 UTC 2012" BUILT_MACH=""
Wed Jul 17 11:14:24 2013: amandad:            BUILT_REV="4084" BUILT_BRANCH="3_3" CC="gcc"
Wed Jul 17 11:14:24 2013: amandad:     paths: bindir="/usr/sbin" sbindir="/usr/sbin"
Wed Jul 17 11:14:24 2013: amandad:            libexecdir="/usr/lib/amanda"
Wed Jul 17 11:14:24 2013: amandad:            amlibexecdir="/usr/lib/amanda" mandir="/usr/share/man"
Wed Jul 17 11:14:24 2013: amandad:            AMANDA_TMPDIR="/tmp/amanda"
Wed Jul 17 11:14:24 2013: amandad:            AMANDA_DBGDIR="/var/log/amanda" CONFIG_DIR="/etc/amanda"
Wed Jul 17 11:14:24 2013: amandad:            DEV_PREFIX="/dev/" RDEV_PREFIX="/dev/r"
Wed Jul 17 11:14:24 2013: amandad:            DUMP="/sbin/dump" RESTORE="/sbin/restore" VDUMP=UNDEF
Wed Jul 17 11:14:24 2013: amandad:            VRESTORE=UNDEF XFSDUMP="/sbin/xfsdump"
Wed Jul 17 11:14:24 2013: amandad:            XFSRESTORE="/sbin/xfsrestore" VXDUMP=UNDEF VXRESTORE=UNDEF
Wed Jul 17 11:14:24 2013: amandad:            SAMBA_CLIENT="/usr/bin/smbclient" GNUTAR="/bin/tar"
Wed Jul 17 11:14:24 2013: amandad:            COMPRESS_PATH="/bin/gzip" UNCOMPRESS_PATH="/bin/gzip"
Wed Jul 17 11:14:24 2013: amandad:             LPRCMD=UNDEF  MAILER=UNDEF
Wed Jul 17 11:14:24 2013: amandad:            listed_incr_dir="/var/lib/amanda/gnutar-lists"
Wed Jul 17 11:14:24 2013: amandad:     defs:  DEFAULT_SERVER="localhost" DEFAULT_CONFIG="DailySet1"
Wed Jul 17 11:14:24 2013: amandad:            DEFAULT_TAPE_SERVER="localhost" DEFAULT_TAPE_DEVICE=""
Wed Jul 17 11:14:24 2013: amandad:            NEED_STRSTR AMFLOCK_POSIX AMFLOCK_FLOCK AMFLOCK_LOCKF
Wed Jul 17 11:14:24 2013: amandad:            AMFLOCK_LNLOCK SETPGRP_VOID AMANDA_DEBUG_DAYS=4 BSD_SECURITY
Wed Jul 17 11:14:24 2013: amandad:            USE_AMANDAHOSTS CLIENT_LOGIN="backup" CHECK_USERID HAVE_GZIP
Wed Jul 17 11:14:24 2013: amandad:            COMPRESS_SUFFIX=".gz" COMPRESS_FAST_OPT="--fast"
Wed Jul 17 11:14:24 2013: amandad:            COMPRESS_BEST_OPT="--best" UNCOMPRESS_OPT="-dc"
Wed Jul 17 11:14:24 2013: amandad: security_handleinit(handle=0x1b966f0, driver=0x7f181da53b40 (LOCAL))
Wed Jul 17 11:14:24 2013: amandad: security_streaminit(stream=0x1b968c0, driver=0x7f181da53b40 (LOCAL))
Wed Jul 17 11:14:24 2013: amandad: authenticated peer name is 'backup'
Wed Jul 17 11:14:24 2013: amandad: accept recv REQ pkt:
<<<<<
SERVICE amidxtaped
OPTIONS features=ffffffff9efefbffffffffff0f;auth=local;
>>>>>
Wed Jul 17 11:14:24 2013: amandad: creating new service: amidxtaped
OPTIONS features=ffffffff9efefbffffffffff0f;auth=local;

Wed Jul 17 11:14:24 2013: amandad: sending ACK pkt:
<<<<<
>>>>>
Wed Jul 17 11:14:24 2013: amandad: tcpm_send_token: data is still flowing
Wed Jul 17 11:14:24 2013: amandad: ERROR service amidxtaped: 
Wed Jul 17 11:14:24 2013: amandad: ERROR service amidxtaped: ** (process:967): WARNING **: Use of qw(...) as parentheses is deprecated at /usr/lib/amanda/perl/Amanda/Recovery/Clerk.pm line 231.
Wed Jul 17 11:14:24 2013: amandad: ERROR service amidxtaped: 
Wed Jul 17 11:14:24 2013: amandad: ERROR service amidxtaped: 
Wed Jul 17 11:14:24 2013: amandad: ERROR service amidxtaped: ** (process:967): WARNING **: Use of qw(...) as parentheses is deprecated at /usr/lib/amanda/perl/Amanda/Recovery/Clerk.pm line 265.
Wed Jul 17 11:14:24 2013: amandad: ERROR service amidxtaped: 
Wed Jul 17 11:14:24 2013: amandad: ERROR service amidxtaped: 
Wed Jul 17 11:14:24 2013: amandad: ERROR service amidxtaped: ** (process:967): WARNING **: Use of qw(...) as parentheses is deprecated at /usr/lib/amanda/perl/Amanda/Recovery/Planner.pm line 231.
Wed Jul 17 11:14:24 2013: amandad: ERROR service amidxtaped: 
Wed Jul 17 11:14:24 2013: amandad: ERROR service amidxtaped: 
Wed Jul 17 11:14:24 2013: amandad: ERROR service amidxtaped: ** (process:967): WARNING **: Use of qw(...) as parentheses is deprecated at /usr/lib/amanda/perl/Amanda/Recovery/Planner.pm line 342.
Wed Jul 17 11:14:24 2013: amandad: ERROR service amidxtaped: 
Wed Jul 17 11:14:24 2013: amandad: ERROR service amidxtaped: 
Wed Jul 17 11:14:24 2013: amandad: ERROR service amidxtaped: ** (process:967): WARNING **: Use of qw(...) as parentheses is deprecated at /usr/lib/amanda/perl/Amanda/Recovery/Planner.pm line 393.
Wed Jul 17 11:14:24 2013: amandad: ERROR service amidxtaped: 
Wed Jul 17 11:14:24 2013: amandad: sending REP pkt:
<<<<<
ERROR service amidxtaped: 
ERROR service amidxtaped: ** (process:967): WARNING **: Use of qw(...) as parentheses is deprecated at /usr/lib/amanda/perl/Amanda/Recovery/Clerk.pm line 231.
ERROR service amidxtaped: 
ERROR service amidxtaped: 
ERROR service amidxtaped: ** (process:967): WARNING **: Use of qw(...) as parentheses is deprecated at /usr/lib/amanda/perl/Amanda/Recovery/Clerk.pm line 265.
ERROR service amidxtaped: 
ERROR service amidxtaped: 
ERROR service amidxtaped: ** (process:967): WARNING **: Use of qw(...) as parentheses is deprecated at /usr/lib/amanda/perl/Amanda/Recovery/Planner.pm line 231.
ERROR service amidxtaped: 
ERROR service amidxtaped: 
ERROR service amidxtaped: ** (process:967): WARNING **: Use of qw(...) as parentheses is deprecated at /usr/lib/amanda/perl/Amanda/Recovery/Planner.pm line 342.
ERROR service amidxtaped: 
ERROR service amidxtaped: 
ERROR service amidxtaped: ** (process:967): WARNING **: Use of qw(...) as parentheses is deprecated at /usr/lib/amanda/perl/Amanda/Recovery/Planner.pm line 393.
ERROR service amidxtaped: 
CONNECT CTL 50 DATA 51

>>>>>
Wed Jul 17 11:14:24 2013: amandad: received ACK pkt:
<<<<<
>>>>>
Wed Jul 17 11:14:24 2013: amandad: security_close(handle=0x1b966f0, driver=0x7f181da53b40 (LOCAL))
Wed Jul 17 11:14:24 2013: amandad: security_stream_close(0x1b968c0)
Wed Jul 17 11:14:24 2013: amandad: security_stream_seterr(0x1b968c0, write error to: Broken pipe)
Wed Jul 17 11:14:25 2013: amandad: pid 966 finish time Wed Jul 17 11:14:25 2013

[martineau]

The amanda version you are using is not compatible with the perl you are using.
You must fix the perl warning in the script or upgrade to a newer amanda.

[michelem]

Yes! You are right! Upgrading amanda to latest version (3.3.4) solved the problem.
Thanks to point me there.

Now I think I should open a bug on Ubuntu because Amanda version on 12.04 is not compatible with the perl version on it.

[pspuranik]

Hi,
I am trying to connect to AWS S3; man configuration file is very much same as michelem's file except the credentials. But I am consistently getting an error..

[email protected]:~$ amdevcheck DailySet1 s3:backup
MESSAGE While trying to read tapestart header: Access Denied (AccessDenied) (HTTP 403)
DEVICE_ERROR
VOLUME_ERROR

Any pointer on what could be the problem. My credentials are correct.

Best Regards.

[michelem]

I think your problem it's related with AWS credentials, please check them or check the credential has right permissions to access/manage your bucket.

[martineau]

It look like you are not the owner of the 'backup' bucket.

Bucket name must be unique, I'm sure someone else already create the 'backup' bucket.
You must use a bucket name that nobody else use.

[pspuranik]

Michele,
Thanks for the reply.

Best Regards,

[pspuranik]

Martine,
Yes, "backup" bucket name is already taken. It worked when I used different bucket name. But now I see another issue; suddenly "amlabel" command has become extremely slow. For first few iteration it was fast but later it would get stuck after first label...something like this..

[email protected]:/etc/amanda/DailySet1$ for i in 1 2 3 4 5 6 7 8 9 10; do amlabel DailySet1 DailySet1-$i slot $i; done;
Reading label...
Found an empty tape.
Writing label 'DailySet1-1'...
Checking label...
Success!
Reading label...
Found an empty tape.
Writing label 'DailySet1-2'...

What could be the issue here?

Best Regards,