Results 1 to 5 of 5

Thread: unable to flush - "process terminated while waiting to flush"

  1. #1

    Angry unable to flush - "process terminated while waiting to flush"

    hi all!

    since couple of days i began to get following message: process terminated while waiting to flush

    when trying to run amflush manually.

    amdump runs through by cron, but then no taping happens. could anyone help me? below is amcheck and amflush/amstatus results:

    Code:
    [1700][amandabackup@fs01:/var/log/amanda/server/cpst]$ amcheck cpst
    Amanda Tape Server Host Check
    -----------------------------
    Holding disk /data/bkcpsteam/amandahd/: 13030988 MB disk space available, using 10933836 MB
    Searching for label 'cpst-03':found in slot 3: volume 'cpst-03'
    Will write to volume 'cpst-03' in slot 3.
    NOTE: skipping tape-writable test
    Server check took 0.207 seconds
    
    Amanda Backup Client Hosts Check
    --------------------------------
    Client check: 1 host checked in 1.099 seconds.  0 problems found.
    
    (brought to you by Amanda 3.3.3)
    [1700][amandabackup@fs01:/var/log/amanda/server/cpst]$ amflush cpst
    
    Multiple Amanda runs in holding disks; please pick one by letter:
      A. 20130614010001
      B. 20130615010001
      C. 20130616010001
      D. 20130617010001
      E. 20130618010001
      F. 20130619010001
      G. 20130620010001
    Select datestamps to flush [A..G or <enter> for all]: 
    
    Today is: 20130620
    Flushing dumps from 20130614010001, 20130615010001, 20130616010001, 20130617010001, 20130618010001, 20130619010001, 20130620010001 using tape changer "chg-disk:/data/bkcpsteam/amandatapes".
    Expecting tape cpst-03 or a new tape.  (The last dumps were to tape cpst-02)
    Are you sure you want to do this [yN]? y
    Running in background, you can log off now.
    You'll get mail when amflush is finished.
    [1701][amandabackup@fs01:/var/log/amanda/server/cpst]$ amstatus cpst
    Using /etc/amanda/cpst/log/amflush.1
    From Thu Jun 20 17:01:06 CEST 2013
    
    nfs02.cluster1:/data/cockpit/cockpit 1        12m process terminated while waiting to flush
    nfs02.cluster1:/data/cockpit/cockpit 1        12m process terminated while waiting to flush
    nfs02.cluster1:/data/cockpit/cockpit 1        12m process terminated while waiting to flush
    nfs02.cluster1:/data/cockpit/cockpit 1        12m process terminated while waiting to flush
    nfs02.cluster1:/data/cockpit/cockpit 1        12m process terminated while waiting to flush
    nfs02.cluster1:/data/cockpit/cockpit 1        13m process terminated while waiting to flush
    nfs02.cluster1:/data/cockpit/cockpit 1        13m process terminated while waiting to flush
    nfs02.cluster1:/data/cpsteam/cpsdata 1       336m process terminated while waiting to flush
    nfs02.cluster1:/data/cpsteam/cpsdata 1       339m process terminated while waiting to flush
    nfs02.cluster1:/data/cpsteam/cpsdata 1       339m process terminated while waiting to flush
    nfs02.cluster1:/data/cpsteam/cpsdata 1       339m process terminated while waiting to flush
    nfs02.cluster1:/data/cpsteam/cpsdata 1      2242m process terminated while waiting to flush
    nfs02.cluster1:/data/cpsteam/cpsdata 1      2307m process terminated while waiting to flush
    nfs02.cluster1:/data/cpsteam/cpsdata 1      2346m process terminated while waiting to flush
    
    SUMMARY          part      real  estimated
                               size       size
    partition       :  14
    estimated       :   0                    0m
    flush           :  14      8341m
    failed          :   0                    0m           (  0.00%)
    wait for dumping:   0                    0m           (  0.00%)
    dumping to tape :   0                    0m           (  0.00%)
    dumping         :   0         0m         0m (  0.00%) (  0.00%)
    dumped          :   0         0m         0m (  0.00%) (  0.00%)
    wait for writing:   0         0m         0m (  0.00%) (  0.00%)
    wait to flush   :  14      8341m      8341m (100.00%) (  0.00%)
    writing to tape :   0         0m         0m (  0.00%) (  0.00%)
    failed to tape  :   0         0m         0m (  0.00%) (  0.00%)
    taped           :   0         0m         0m (  0.00%) (  0.00%)
    4 dumpers idle  : no-dumpers
    taper status: Idle
    taper qlen: 14
    network free kps:    180000
    holding space   :  10933836m (100.00%)
     0 dumpers busy :  0:00:00  (100.00%)
    [1701][amandabackup@fs01:/var/log/amanda/server/cpst]$
    attached are amflush, taper and amcheck-device logs where i could not find anything to explain what happens...

    could the problem be in the virtual tape?
    Attached Files Attached Files

  2. #2
    Join Date
    Nov 2005
    Location
    Canada
    Posts
    1,049

    Default

    What do you get in the email report?
    Post the /etc/amanda/cpst/log/amflush.1 file
    What are the flush-threshold-* setting?

  3. #3

    Default

    Quote Originally Posted by martineau View Post
    What do you get in the email report?
    Post the /etc/amanda/cpst/log/amflush.1 file
    What are the flush-threshold-* setting?
    email states following:
    Code:
    Hostname: fs01
    Org     : cpst
    Config  : cpst
    Date    : June 20, 2013
    
    There are 8341M of dumps left in the holding disk.
    They will be flushed on the next run.
    
    The next 4 tapes Amanda expects to use are: cpst-03, cpst-4, cpst-5, cpst-6.
    
    
    STATISTICS:
                              Total       Full      Incr.   Level:#
                            --------   --------   --------  --------
    Estimate Time (hrs:min)     0:00
    Run Time (hrs:min)          0:00
    Dump Time (hrs:min)         0:00       0:00       0:00
    Output Size (meg)            0.0        0.0        0.0
    Original Size (meg)          0.0        0.0        0.0
    Avg Compressed Size (%)      --         --         --
    DLEs Dumped                    0          0          0
    Avg Dump Rate (k/s)          --         --         --
    
    Tape Time (hrs:min)         0:00       0:00       0:00
    Tape Size (meg)              0.0        0.0        0.0
    Tape Used (%)                0.0        0.0        0.0
    DLEs Taped                     0          0          0
    Parts Taped                    0          0          0
    Avg Tp Write Rate (k/s)      --         --         --
    
    
    DUMP SUMMARY:
                                                       DUMPER STATS               TAPER STATS
    HOSTNAME       DISK                  L ORIG-MB  OUT-MB  COMP%  MMM:SS   KB/s MMM:SS   KB/s
    -------------------------------------- ------------------------------------- -------------
    nfs02.cluster1 /data/cockpit/cockpit                           NOT FLUSHED                
    nfs02.cluster1 /data/cpsteam/cpsdata                           NOT FLUSHED                
    
    (brought to you by Amanda version 3.3.3)

    amflush.1:
    Code:
    amflush: start at Thu Jun 20 17:01:06 CEST 2013
    amflush: datestamp 20130620170102
    amflush: starttime 20130620170102
    amflush: starttime-locale-independent 2013-06-20 17:01:06 CEST
    FLUSH nfs02.cluster1 /data/cockpit/cockpit 20130614010001 1 /data/bkcpsteam/amandahd//20130614010001/nfs02.cluster1._data_cockpit_cockpit.1
    FLUSH nfs02.cluster1 /data/cpsteam/cpsdata 20130614010001 1 /data/bkcpsteam/amandahd//20130614010001/nfs02.cluster1._data_cpsteam_cpsdata.1
    FLUSH nfs02.cluster1 /data/cockpit/cockpit 20130615010001 1 /data/bkcpsteam/amandahd//20130615010001/nfs02.cluster1._data_cockpit_cockpit.1
    FLUSH nfs02.cluster1 /data/cpsteam/cpsdata 20130615010001 1 /data/bkcpsteam/amandahd//20130615010001/nfs02.cluster1._data_cpsteam_cpsdata.1
    FLUSH nfs02.cluster1 /data/cockpit/cockpit 20130616010001 1 /data/bkcpsteam/amandahd//20130616010001/nfs02.cluster1._data_cockpit_cockpit.1
    FLUSH nfs02.cluster1 /data/cpsteam/cpsdata 20130616010001 1 /data/bkcpsteam/amandahd//20130616010001/nfs02.cluster1._data_cpsteam_cpsdata.1
    FLUSH nfs02.cluster1 /data/cockpit/cockpit 20130617010001 1 /data/bkcpsteam/amandahd//20130617010001/nfs02.cluster1._data_cockpit_cockpit.1
    FLUSH nfs02.cluster1 /data/cpsteam/cpsdata 20130617010001 1 /data/bkcpsteam/amandahd//20130617010001/nfs02.cluster1._data_cpsteam_cpsdata.1
    FLUSH nfs02.cluster1 /data/cockpit/cockpit 20130618010001 1 /data/bkcpsteam/amandahd//20130618010001/nfs02.cluster1._data_cockpit_cockpit.1
    FLUSH nfs02.cluster1 /data/cpsteam/cpsdata 20130618010001 1 /data/bkcpsteam/amandahd//20130618010001/nfs02.cluster1._data_cpsteam_cpsdata.1
    FLUSH nfs02.cluster1 /data/cockpit/cockpit 20130619010001 1 /data/bkcpsteam/amandahd//20130619010001/nfs02.cluster1._data_cockpit_cockpit.1
    FLUSH nfs02.cluster1 /data/cpsteam/cpsdata 20130619010001 1 /data/bkcpsteam/amandahd//20130619010001/nfs02.cluster1._data_cpsteam_cpsdata.1
    FLUSH nfs02.cluster1 /data/cockpit/cockpit 20130620010001 1 /data/bkcpsteam/amandahd//20130620010001/nfs02.cluster1._data_cockpit_cockpit.1
    FLUSH nfs02.cluster1 /data/cpsteam/cpsdata 20130620010001 1 /data/bkcpsteam/amandahd//20130620010001/nfs02.cluster1._data_cpsteam_cpsdata.1
    ENDFLUSH
    driver: pid 5112 executable driver version 3.3.3
    driver: tape size 524288000
    driver: adding holding disk 0 dir /data/bkcpsteam/amandahd/ size 11196248064 chunksize 5242880
    reserving 11196248064 out of 11196248064 for degraded-mode dumps
    driver: send-cmd time 0.032 to taper: START-TAPER worker0 20130620170102
    driver: start time 0.032 inparallel 4 bandwidth 180000 diskspace 11196248064  dir OBSOLETE datestamp 20130620170102 driver: drain-ends tapeq LARGESTFIT big-dumpers sssS
    driver: state time 0.032 free kps: 180000 space: 11196248064 taper: idle idle-dumpers: 4 qlen tapeq: 0 runq: 0 roomq: 0 wakeup: 0 driver-idle: not-idle
    driver: interface-state time 0.032 if default: free 180000
    driver: hdisk-state time 0.032 hdisk 0: free 11196248064 dumpers 0
    driver: state time 0.296 free kps: 180000 space: 11196248064 taper: idle idle-dumpers: 4 qlen tapeq: 14 runq: 0 roomq: 0 wakeup: 0 driver-idle: not-idle
    driver: interface-state time 0.296 if default: free 180000
    driver: hdisk-state time 0.296 hdisk 0: free 11196248064 dumpers 0
    driver: result time 0.297 from taper: TAPER-OK worker0
    driver: state time 0.297 free kps: 180000 space: 11196248064 taper: idle idle-dumpers: 4 qlen tapeq: 14 runq: 0 roomq: 0 wakeup: 0 driver-idle: no-dumpers
    driver: interface-state time 0.297 if default: free 180000
    driver: hdisk-state time 0.297 hdisk 0: free 11196248064 dumpers 0
    driver: QUITTING time 0.297 telling children to quit
    driver: send-cmd time 0.297 to taper: QUIT
    driver: FINISHED time 1.300

    and the config thresholds are:
    Code:
    flush-threshold-dumped	100 
    flush-threshold-scheduled	100 
    taperflush			100
    autoflush			yes

  4. #4
    Join Date
    Nov 2005
    Location
    Canada
    Posts
    1,049

    Default

    Your tape is 512GB
    You ask amanda to not write to tape until it fill a tape (all *flush* parameters set to 100).

    Why are you surprised it doesn't flush when you have less than 9GB of data to flush?

  5. #5

    Default

    Quote Originally Posted by martineau View Post
    Your tape is 512GB
    You ask amanda to not write to tape until it fill a tape (all *flush* parameters set to 100).

    Why are you surprised it doesn't flush when you have less than 9GB of data to flush?
    possible because i'm being genuinely stupid misreading Gb for Tb...

    the error message threw me off though. in previous version (i had 2.6 before) normally there was a message "flush-dump criteria is not met" or something like that.

    thanks for the answer, sorry for wasting your time.

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •