Page 1 of 2 12 LastLast
Results 1 to 10 of 11

Thread: Tertiary back up question ...

  1. #1

    Default Tertiary back up question ...

    I've got zmanda working pretty nicely, I think, in terms of backing up data to RAID disk vtapes for 3 different data sets (nightly backups with 8 backups in rotation). I'm now interested in now backing up in a more permanent fashion using our StorEdge C2 tape changer. I'm confused as to whether I should:
    1. Create new datasets and back them up to tape, for example, as a weekly level 0 backup.
    2. Create a RAIT and try to backup simultaneously to a "real" tape and a vtape. (I think that a single tape will handle a week's worth of my vtapes for the three datasets).
    3. Simply dump the existing vtapes (and any appropriate indexes) to a real tape on a weekly basis.
    I thought that this might be the kind of question best asked on the forum so that others might also be able to comment and/or learn from this discussion.

    Thanks,

    John

    Here is the initial response that I received from Paddy at Zmanda in an initial support case question that I thought would be useful to post to the forum with his permission:.

    Option 1 would mean you will be performing the full backup of the clients twice (once to backup to disk and another to back them to tape).

    Option 2 would mean you will be backing up to vtape and "real" tape at the same time. You will get performance of slowest of the two.

    Option 3 - you can use amvault command (scheduled using crontab) to dump the vtapes to the physical tapes. This will not delay the actual backup as option 2 does and you will be performing backup once. amvault command is run only on the Amanda server and has no impact on the client. Please note that amvault cannot be done from Zmanda Management Console.

    We would recommend option 3. See [url]http://docs.zmanda.com/Project:Amanda_Enterprise_3.1/Using_Tertiary_Media[/url]

    thanks
    Paddy

  2. #2

    Default Followup question related to use of amvault ...

    Paddy et al:

    I've been looking into the use of amvault as a means of performing a tertiary backup to tape and have a couple of followup questions:

    If I want to use amvault to dump three different datasets on a weekly basis, it seems as if it only makes sense to dump level 0 backups to tape. It also seems as if it will be easiest to specify the 'latest' paramater with amvault for each dataset. That would seem to mean that I should specify a custom schedule for each dataset for the creation of my vtapes so that I can force a level 0 backup on a particular day, Saturday for example, so that when amvault runs later in the day, 'latest' will be guaranteed to find a full dump. Is that correct or is there a flag in amvault that is the equivalent of "latest full dataset"?

    Secondly, although 'man amvault' doesn't show it (for Zmanda Enterprise Server 3.1.5) is there a mechanism that will allow using amvault to dump the latest vtapes for more than one dataset onto a single tape?

    Thanks,

    John

  3. #3

    Default

    You will have to use custom schedule to force full backups on a particular day.

    Amanda Enterprise 3.1.5 requires one-one relationship between vtape and physical tape. It is not possible to dump vtapes from different data sets to a single tape.

    Paddy
    [URL=http://amanda.zmanda.com/]Amanda backup and recovery [/URL]
    [URL=http://www.zmanda.com/backup-mysql.html]MySQL backup and recovery[/URL]

  4. #4

    Default amvault and tertiary backup ...

    Paddy:

    Thank you for your response. I've now changed to a custom schedule to force full backups at known times and will plan to use a tape for each dataset.

    Now I need to go back to the documentation to make sure that I understand the relationship between what zmanda uses for tape labels and what the changer reads from it's barcode reader.

    Thanks again for your timely response to my questions,

    John

  5. #5
    Join Date
    Aug 2008
    Location
    Sunnyvale, CA
    Posts
    306

    Default

    Quote Originally Posted by shott View Post
    Now I need to go back to the documentation to make sure that I understand the relationship between what zmanda uses for tape labels and what the changer reads from it's barcode reader.
    The two are fairly independent. If you have indicated that your changer has a barcode reader, Amanda will associate barcodes with tape labels to locate tapes it is looking for faster. There is the option through The Zmanda Management Console to label tapes for the backup set with the barcode in the Amanda label which simply makes it easier for you to identify the tape. Amanda does not require the barcode to be a part of the Amanda label. For amvault, tapes are autolabeled per the label-template specified on the command line but will not include the barcode of the tape in the Amanda label.

    Paul

  6. #6

    Default Confused about use of amvault ...

    Paul, Paddy, and amvault experts:

    I seem to be unable to get amvault to run properly .... in part, I think, because I've likely not properly set up my dst-device.

    Here is an example of the amvault command that I have run and the result:

    The command that I ran:

    /usr/sbin/amvault --autolabel this_config Solaris_10_OS latest StorEdge_C2 SOS-%%

    And the result:

    Loading source volume Solaris_10_OS-003
    Loading next destination slot
    Could not read label from StorEdge_C2: Can't open tape device StorEdge_C2: No such file or directory

    Note: Solaris_10_OS is a working data set that I currently use to create the vtapes that I hope to be dumping to tape using amvault.

    I have done what I could to define the dst_changer as StorEdge_C2. Because that device isn't currently used as a part of a real dataset (as far as Zmanda knows), I believe that the only place the StorEdge_C2 is configured is in the file:

    /etc/zmanda/zmc_aee/device_profiles/StorEdge_C2.yml

    The content of that file includes:

    StorEdge_C2:
    zmc_device_type: changer
    changerdev: /dev/changer
    tpchanger: chg-zd-mtx
    autolabel: off
    dumptype_list:
    zmc_default_dev:
    name: zmc_default_dev
    comment: '*splitsize* can only be defined in the scope of a dumptype definition, not a device'
    fallback_splitsize: 100m
    tape_splitsize_auto: on
    tape_splitsize_percent: 5
    device_property_list:
    comment: 'http://wiki.zmanda.com/man/amanda-devices.7.html'
    COMPRESSION_comment: on
    FSF_comment: off
    BSF_comment: on
    fallback_splitsize_display: m
    tapetype:
    length: 450560m
    length_display: g
    filemark: 1m
    filemark_display: k
    changer:
    havereader: on
    driveslot: 0
    offline_before_unload: off
    poll_drive_ready: 3
    max_drive_wait: 300
    initial_poll_delay: 30
    autoclean: on
    cleanslot: 16
    autocleancount: 99
    autoflush: on
    holdingdisk_list:
    zmc_default_holding:
    use: 89
    use_display: '%'
    directory: /var/lib/amanda/staging/@@ZMC_AMANDA_CONF@@
    chunksize: 1024m
    chunksize_display: m
    labelstr: '^@@ZMC_AMANDA_CONF@@-[0-9][0-9]*(-.*)?$'
    schedule:
    dumpcycle: 7
    full_hours_same: 1
    hours:
    - 2
    minute: 15
    retention_policy: 7
    runtapes: 1
    schedule_type: 'Every Weekday'
    private:
    occ: '88622576915'
    zmc_show_advanced: '0'
    last_modified_time: '2011-06-23 20:50:57'
    last_modified_by: shott
    zmc_version: 3
    comment: 'StorEdge C2 Tape Changer\n/dev/changer is link to /dev/scsi/changer/c4t5d1 which is a link to the actual device'
    has_barcode_reader: true

    I'm confused as to whether this StorEdge_C2.yml is mis-configured in some way or whether amvault can't find it unless it is associated with some sort of "real" dataset.

    Does anyone have a suggestion of either what I need to do to properly configure the StorEdge_C2 dst_changer so that it knows how to open the device and eliminate the errors that amvault reports?:

    Could not read label from StorEdge_C2: Can't open tape device StorEdge_C2: No such file or directory

    Alternatively, is there a way to turn on more logging or debugging in the amvault command that will help me to resolve my problems?

    Thanks for you insights.

    John

  7. #7
    Join Date
    Aug 2008
    Location
    Sunnyvale, CA
    Posts
    306

    Default

    Quote Originally Posted by shott View Post
    I have done what I could to define the dst_changer as StorEdge_C2. Because that device isn't currently used as a part of a real dataset (as far as Zmanda knows), I believe that the only place the StorEdge_C2 is configured is in the file:

    /etc/zmanda/zmc_aee/device_profiles/StorEdge_C2.yml
    Hi John,

    The key is to put a changer definition directly into the amanda.conf file for the backup set. If you see the man page for amvault, it gives you a sample:

    define changer vaulting_tape {
    tapedev "/dev/rmt/1n"
    tpchanger "chg-zd-mtx"
    changerdev "/dev/sg0"
    changerfile "vaulting-changer.conf"
    }

    Creating a definition like this in the backup set's amanda.conf where "vaulting_tape" = "StorEdge_C2" in your case and tapedev and changerdev are set to the correct values for your server should do it.

  8. #8

    Default

    pyeatman:

    Thank you for the hints. Although I'm not certain that my configuration is exactly correct, I do now seem to be able to run amvault and see that it is writing information to a tape. Now I will have to wait until my weekly full dumps occur to see it that the full dump is written and then to begin to check to make sure that I can actually recover from what is written to tape.

    Note: the actual definition that I included at the end of each of the applicable amanda.conf file for all Backup Sets was:

    define changer StorEdge_C2 {
    tapedev "/dev/rmt/0n"
    tpchanger "chg-zd-mtx"
    changerdev "/dev/changer"
    changerfile "/etc/amanda/vault/StorEdge_C2.conf"
    }

    My creation and use of the changerfile in /etc/amanda/vault/StorEdge_C2.conf may not have been the best selection because it appears as if ZMC looks at the /etc/amanda/vault directory as another backup set that really isn't a backup set. My guess is that there is a better place to have located my vault directory that contains the StorEdge_C2.conf file.

    But, thanks again for your help ... aside from the important step of confirming that my tapes are rotating correctly and that the dumped data is correct, I am now able to get amvault to dump vtapes to "real tapes" for more archival storage.

    John

  9. #9

    Default

    amvault experts:

    I am now getting closer to getting amvault to run. Curiously, however, I can run it from the command line successfully, but, when the same command runs from a cron job, it doesn't seem to do anything. That seems particularly strange because all of my ZMC controlled jobs are running successfully (as amandabackup) when triggered by cron jobs.

    Yesterday morning I ran the following command from the command line:

    /usr/sbin/amvault SNF_User_Files latest StorEdge_C2 SNF_User_Files-%%%

    and got a full log file in:

    /var/log/amanda/server/SNF_User_Files
    /amvault.20110703091347.debug

    But, when the same thing was run last night from the cron job at 18:00, the log file

    /var/log/amanda/server/SNF_User_Files/amvault.20110703180000.debug

    only contains:

    Sun Jul 3 18:00:00 2011: amvault: pid 7106 ruid 41001 euid 41001 version 3.1.5: start at Sun Jul 3 18:00:00 2011
    Sun Jul 3 18:00:01 2011: amvault: pid 7106 ruid 41001 euid 41001 version 3.1.5: rename at Sun Jul 3 18:00:01 2011

    Normally I would expect an environment variable problem for amandabackup, but all of the ZMC jobs that dump datasets are working properly.

    Does anyone have a suggestion as to where to look or what may be causing these problems?

    Thanks,

    John

    p.s. In case it is relevant, here is the content log file associated with the command-line run of amvault.

    Thanks,

    John

    Sun Jul 3 09:13:47 2011: amvault: pid 18513 ruid 41001 euid 41001 version 3.1.5: start at Sun Jul 3 09:13:47 2011
    Sun Jul 3 09:13:47 2011: amvault: pid 18513 ruid 41001 euid 41001 version 3.1.5: rename at Sun Jul 3 09:13:47 2011
    Sun Jul 3 09:13:47 2011: amvault: Amanda::Changer::compat initialized with script /usr/lib/amanda/chg-disk, temporary directory /etc/amanda/SNF_User_Files
    Sun Jul 3 09:13:47 2011: amvault: Amanda::Changer::compat initialized with script /usr/lib/amanda/chg-zd-mtx, temporary directory /tmp/amanda/Amanda::Changer::compat/SNF_User_Files-StorEdge_C2
    Sun Jul 3 09:13:47 2011: amvault: Amanda::Changer::compat: invoking /usr/lib/amanda/chg-disk with -info
    Sun Jul 3 09:13:47 2011: amvault: Amanda::Changer::compat: invoking /usr/lib/amanda/chg-zd-mtx with -info
    Sun Jul 3 09:13:48 2011: amvault: Amanda::Changer::compat: Got response '3 8 1' with exit status 0
    Sun Jul 3 09:13:48 2011: amvault: info: chg-compat: beginning manual scan of 8 slots
    Sun Jul 3 09:13:48 2011: amvault: Amanda::Changer::compat: invoking /usr/lib/amanda/chg-disk with -slot current
    Sun Jul 3 09:13:48 2011: amvault: Amanda::Changer::compat: Got response '3 file:/var/lib/amanda/vtapes/SNF_User_Files' with exit status 0
    Sun Jul 3 09:13:48 2011: amvault: info: chg-compat: updated slot 3
    Sun Jul 3 09:13:48 2011: amvault: Opened source device
    Sun Jul 3 09:13:50 2011: amvault: Amanda::Changer::compat: Got response '2 16 1 1' with exit status 0
    Sun Jul 3 09:13:50 2011: amvault: Amanda::Changer::compat: invoking /usr/lib/amanda/chg-zd-mtx with -slot current
    Sun Jul 3 09:13:53 2011: amvault: Amanda::Changer::compat: Got response '2 /dev/rmt/0n' with exit status 0
    Sun Jul 3 09:13:53 2011: amvault: warning: "/dev/rmt/0n" uses deprecated device naming convention;
    using "tape:/dev/rmt/0n" instead.

    Sun Jul 3 09:13:55 2011: amvault: Opened destination device
    Sun Jul 3 09:13:55 2011: amvault: $VAR1 = bless( {}, 'Amanda::Header::Header' );

    Sun Jul 3 09:13:55 2011: amvault: Amanda::Changer::compat: invoking /usr/lib/amanda/chg-zd-mtx with -slot next
    Sun Jul 3 09:15:57 2011: amvault: Amanda::Changer::compat: Got response '3 /dev/rmt/0n' with exit status 0
    Sun Jul 3 09:15:57 2011: amvault: warning: "/dev/rmt/0n" uses deprecated device naming convention;
    using "tape:/dev/rmt/0n" instead.

    Sun Jul 3 09:15:57 2011: amvault: Device /dev/rmt/0n error = 'Error reading Amanda header: Error reading 32768 bytes from /dev/rmt/0n: I/O error'
    Sun Jul 3 09:15:57 2011: amvault: Device /dev/rmt/0n setting status flag(s): DEVICE_STATUS_DEVICE_ERROR, DEVICE_STATUS_VOLUME_UNLABELED, and DEVICE_STATUS_VOLUME_ERROR
    Sun Jul 3 09:15:57 2011: amvault: Opened destination device
    Sun Jul 3 09:15:57 2011: amvault: $VAR1 = bless( {}, 'Amanda::Header::Header' );

    Sun Jul 3 09:15:57 2011: amvault: Building type TAPESTART header of 32768-32768 bytes with name='SNF_User_Files-009' disk='' dumplevel=0 and blocksize=32768
    Sun Jul 3 09:16:02 2011: amvault: Amanda::Changer::compat: invoking /usr/lib/amanda/chg-zd-mtx with -label SNF_User_Files-009
    Sun Jul 3 09:16:05 2011: amvault: Amanda::Changer::compat: Got response '3 /dev/rmt/0n' with exit status 0
    Sun Jul 3 09:16:05 2011: amvault: Building type SPLIT_FILE header of 32768-32768 bytes with name='shine' disk='/home/User' dumplevel=0 and blocksize=32768
    Sun Jul 3 09:16:05 2011: amvault: starting transfer
    Sun Jul 3 09:16:05 2011: amvault: Starting <Xfer@89e3b50 (<XferSourceDevice@89e2100> -> <XferDestDevice@89e2328>)>
    Sun Jul 3 09:16:05 2011: amvault: Final linkage: <XferSourceDevice@89e2100> -(PULL_BUFFER)-> <XferDestDevice@89e2328>
    Sun Jul 3 09:20:17 2011: amvault: Device file:/var/lib/amanda/vtapes/SNF_User_Files error = 'EOF'
    Sun Jul 3 09:20:18 2011: amvault: transfer completed
    Sun Jul 3 09:20:18 2011: amvault: Building type SPLIT_FILE header of 32768-32768 bytes with name='shine' disk='/home/Devel' dumplevel=0 and blocksize=32768
    Sun Jul 3 09:20:18 2011: amvault: starting transfer
    Sun Jul 3 09:20:18 2011: amvault: Starting <Xfer@89e2380 (<XferSourceDevice@89e8278> -> <XferDestDevice@89e82d0>)>
    Sun Jul 3 09:20:18 2011: amvault: Final linkage: <XferSourceDevice@89e8278> -(PULL_BUFFER)-> <XferDestDevice@89e82d0>
    Sun Jul 3 09:24:39 2011: amvault: transfer completed
    Sun Jul 3 09:25:03 2011: amvault: pid 18513 finish time Sun Jul 3 09:25:03 2011

  10. #10
    Join Date
    Nov 2005
    Location
    Canada
    Posts
    1,049

    Default

    Do you get an email message from cron?

    The '%' must be escaped when run from cron:
    /usr/sbin/amvault SNF_User_Files latest StorEdge_C2 SNF_User_Files-\%\%\%

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •