PDA

View Full Version : Backup failure - logs emailed



wonderwebs
April 3rd, 2010, 03:35 PM
Hi there,

I've emailed a log zip through to the to zcb@ and the name on it is wonderwebs.

I've created 5 backup set, one for a few Mb on MySQL, one for a couple hundred Mb of MSSQL, and 4 for website folders belonging to the databases.

Initial manual runs would fail for some of the file system sets, so I looked in the forum, saw another post for the same error "An existing connection was forcibly closed by the remote host." and opened 10081 inbound tcp, and 10080 outbound tcp - is that correct? I may need more accurate instructions on opening those.

Regardless, the next morning, two of the file system backs report the same failure, although one that previously failed appears to have worked. When I run it manually, it also fails.

Hoping you can tell me how to make them all work reliably.
Thanks heaps
Rob



P.s. Is it really necessary to make customers sign up with yousendit just to email you a 10mb file?

wonderwebs
April 3rd, 2010, 04:13 PM
I've just now put both port numbers in both the in and outbound rules in case I had ti wrong, and this time I was able to run one of the failed backups, however the next one promptly failed with the same message, so it's not a port issue as far as I can tell.

I'm also interested in knowing where the files are going on the s3 system. I had assumed that it would be using my own s3 account, but it appears not to?

Regards
Rob

paddy
April 3rd, 2010, 11:14 PM
Can you please open a support case? It will be easier to send support log files.
Go to network.zmanda.com and click on Support tab.

Are you running ZCB 2.0.3?

The files will go to your S3 account. These buckets will be specific for ZCB application and can be seen by ZCB application. If you want to see the buckets without ZCB, download modified jets3t tool available in Zmanda network downloads page.

thanks
Paddy

harsha
April 4th, 2010, 09:00 PM
Hi Rob,

We have received the logs and investigating the issue.

thanks
-Harsha

harsha
April 4th, 2010, 09:46 PM
Hi Rob,

The error message ""An existing connection was forcibly closed by the remote host." means that our backend service (viz ZWCService) has crashed.

To find out the location and cause of the crash, we need its memory dump file.

GOTO

"Control Panel\System and Security\Action Center\Check For Solutions"

Look for ZWCServie.exe. Click "View Technical Details"

Click "View a temporary copy of these files". Zip all files in this location and send us.

thanks
-Harsha

wonderwebs
April 5th, 2010, 03:57 PM
Thanks for checking this out Harsha.

There were 11 crashes listed for the service and so I zipped up the memory dump files for the most recent one, as I think they're all likely to be the same.

I've used the Yousendit service to email it to zcb@ with the subject: wonderwebs-WERF397.tmp.zip

Hopefully it will make sense to you.
Regards,
Rob

wonderwebs
April 5th, 2010, 04:04 PM
p.s. I'm running the following:


ZCB 2.0.3 Stable (64-bit)
Windows Server 2008 Web Edition 64bit SP1
Java(TM) 6 Upate 17 (64 bit)
Java(TM) 6 Upate 19 (this was an auto-update after installing the above)

harsha
April 5th, 2010, 11:28 PM
Thanks for checking this out Harsha.

There were 11 crashes listed for the service and so I zipped up the memory dump files for the most recent one, as I think they're all likely to be the same.

I've used the Yousendit service to email it to zcb@ with the subject: wonderwebs-WERF397.tmp.zip

Hopefully it will make sense to you.
Regards,
Rob

Thanks for sending these dump files Rob.

I will need some more help from you to debug this issue.

1. Raise the log level to debug from ZCB UI "Tools->Log Level->Debug"
2. Run the setup-debug-env.bat script located in "bin" folder of ZCB install directory.
3. Then rerun the backup which is causing the ZWCService to crash.
4. From ZCB UI "Tools->Collect logs" and send them.

thanks
-Harsha

wonderwebs
April 6th, 2010, 02:31 AM
Hi Harsha,

Ok yep, I've done that and have triggered the backups manually one at a time, but this time they all worked without failing.

I'll trigger it again, but if it won't fail, I'll leave it to run on schedule and see if that makes it fail.

Then I'll send the log in.
Rob

wonderwebs
April 6th, 2010, 08:13 PM
Ok, just reporting back:

The scheduled backup overnight worked fine with no errors reported in ZCB.

I've no idea why it has started working now. The server did download a small update for the Visual C++ Redistributable within a day of installing ZCB, and there was a Java update, and I did set the ZCB service to restart itself if it fails, but nothing else had been changed. Perhaps one of those items fixed it.

One question: Do I need to do anything to change out of ZCB debug mode after running setup-debug-env.bat script yesterday, or can I just ignore that?


And the last question:
I am now seeing another error in the server log, that I did not notice before. It looks like it happens each time the full backup is run. Can you let me know if this matters:

Volume Shadow Copy Service error: Unexpected error calling routine IVssBackupComponents::SetContextInternal. hr = 0x80042301.
Operation: Add a Volume to a Shadow Copy Set
Context: Execution Context: Requestor

Thanks heaps
Rob

harsha
April 6th, 2010, 10:11 PM
Hi Rob,

Nice to hear that your backups are running fine.

I also noticed VSS error event 8193 in the logs we received from you. Also some VSS writers are in inconsistent state ("vssadmin list writers" from command line gives VSS writers status).

Writer name: 'System Writer'
Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
Writer Instance Id: {c71d1e9a-d21c-4e45-8525-99003172d161}
State: [8] Failed
Last error: Inconsistent shadow copy

Pl. reboot your system so that all writers come into a consistent state. Also I guess it will help to get rid of event id 8193.

About setup-debug_env.bat - it just asks "Windows Error Reporting" to save ZWCService crashdumps in ZCB folders so that "Collect logs" tools can collect it along with all other logs. "Collect logs" tool reset the settings made by setup-debug-env.bat.

-Harsha

wonderwebs
April 11th, 2010, 06:50 PM
Thanks for the replies Harsha.

I've run it for a few days now, and it has been rebooted three times since. Once to update NIC drivers and twic

The backups appear to be working, however I'm still seeing the error below in the Windows Event Logs.

Further down is also the result of the "vssadmin list writers" command. Hopefully it will tell you what is needed.

Regards,
Rob


=====================================
Volume Shadow Copy Service error: Unexpected error calling routine IVssBackupComponents::SetContextInternal. hr = 0x80042301.
Operation:
Add a Volume to a Shadow Copy Set
Context:
Execution Context: Requestor
====================================

I typed the command provided into the CMD prompt and it returned the following:

Microsoft Windows [Version 6.0.6001]
Copyright (c) 2006 Microsoft Corporation. All rights reserved.

C:\Users\ServerAdmin>vssadmin list writers
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2005 Microsoft Corp.

Writer name: 'System Writer'
Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
Writer Instance Id: {f38355e8-c33b-42fa-be03-b6605277cb16}
State: [8] Failed
Last error: Inconsistent shadow copy

Writer name: 'SqlServerWriter'
Writer Id: {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}
Writer Instance Id: {89eaf26e-bd84-4c95-a28e-a8715833918c}
State: [1] Stable
Last error: No error

Writer name: 'IIS Config Writer'
Writer Id: {2a40fd15-dfca-4aa8-a654-1f8c654603f6}
Writer Instance Id: {a5985910-b37d-4862-ace6-6d64b9b50aad}
State: [8] Failed
Last error: Inconsistent shadow copy

Writer name: 'ASR Writer'
Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}
Writer Instance Id: {7199a7a2-e834-4d81-956f-5baaf120efbc}
State: [1] Stable
Last error: No error

Writer name: 'Registry Writer'
Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
Writer Instance Id: {8bfc043c-b103-4016-a962-b88ad73fc3b1}
State: [1] Stable
Last error: No error

Writer name: 'Shadow Copy Optimization Writer'
Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
Writer Instance Id: {4ce52f17-2a3f-423a-b87d-47a9360a4804}
State: [1] Stable
Last error: No error

Writer name: 'COM+ REGDB Writer'
Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
Writer Instance Id: {48135cc9-19c8-4843-a6ed-0574a05e137f}
State: [1] Stable
Last error: No error

Writer name: 'IIS Metabase Writer'
Writer Id: {59b1f0cf-90ef-465f-9609-6ca8b2938366}
Writer Instance Id: {12bd4c43-9dbe-4891-841c-2e9834d34153}
State: [8] Failed
Last error: Inconsistent shadow copy

Writer name: 'BITS Writer'
Writer Id: {4969d978-be47-48b0-b100-f328f07ac1e0}
Writer Instance Id: {9eefcfec-e64d-425a-8c77-b31eb45b3efd}
State: [1] Stable
Last error: No error

Writer name: 'WMI Writer'
Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
Writer Instance Id: {0ad442b6-4246-49ec-976b-61b0d602e2b1}
State: [8] Failed
Last error: Inconsistent shadow copy

harsha
April 11th, 2010, 10:33 PM
Hi Rob,

To nail down the problem. Pl. follow the steps.

1. Reboot the machine to bring all the writers to a consistent state.
2. Take a backup of backupsets one by one manually.
3. After each backup check the writers status (using "vaaadmin list writers") and also check if there is a VSS error in application event log.
4. If writers went into an inconsistent state or there is a VSS error, collect the logs immediately and send us.

Actully I want to know exactly which backup set is cauing the problem.

Also pl. let me know if this machine is on hyper-V.

-Harsha

wonderwebs
April 12th, 2010, 05:50 PM
That's going to be a bit of a job so I'll have to wait until the weekend to do it.

In the meantime I've disabled all the backups and the ZCB services so that it's not active.

I did this because the server has another backup system from r1soft

That system does backups of the entire server for the purposes of a bare-metal restore should the server fail.

I wanted the ZCB Backups to be an additional and very specific off-site backup of just my web folders plus the related MSSQL and MySQL databases. The features fit the bill perfectly.

Now I believe that these two systems are conflicting. The other backup reports this in the event log:

An error occured while executing VSS operation 'm_pVssObject->StartSnapshotSet(&m_latestSnapshotSetID)'. VSS_E_SNAPSHOT_SET_IN_PROGRESS - The creation of a shadow copy is in progress, and only one shadow copy creation operation can be in progress at a time. This can happen if multiple VSS-aware applications are running on the same machine.


Do you think this is likely to be the source of the problems?
Regards
Rob

p.s. the machine is not on HyperV. It is a standalone server running Server2008 SP1 natively. I had 5 backup sets defined. Three were of files and folders. One was a set of MySQL databases, adn the last one was the MSSQL databases.

harsha
April 12th, 2010, 10:29 PM
Yes Rob,

Multiple backups (of different vendors or same vendor) should not be scheduled at the same time especially if they are using the VSS based backups. Otherwise conflicts like you mentioned in the earlier message happens.

-Harsha

wonderwebs
April 23rd, 2010, 01:46 AM
Hi there,

I'm afraid this has all turned into too much of a drama for me to deal with. I just can't find the time spare to troubleshoot this application and so I've cancelled the app on amazon and have removed everything from my web server.

Apart from the problems getting ZCB working satisfactorily, I was also not very happy to discover I had to install JRE on the server. It just wasn't worth the extra footprint and hassle. In any case, it's all gone now and I'm fine with that.

My next step was to try and get the files deleted from my S3 account. I was aware of the "jets3t' application made available for download and so I've been trying to set that up. I don't run Java on my local machine either, but I went ahead and installed it for this purpose.

And nwo the problem: When I run the cockpit.bat file from Windows, a dommand window opens and closes immediately. So then I moved the cockpit-bundle file to the root of the C drive, opened a command prompt and ran the bat from there, but it says this:

C:\cockpit-bundle\bin>cockpit.bat
'"java"' is not recognized as an internal or external command,
operable program or batch file.

So then I went off to Sun to check the Java install and it tells me it is the latest and verifies that it is fine.

All I want to do is get rid of the files stored on S3 so that I'm not paying for them for the rest of eternity.

Is there any way I can do that without spending days on it?

Thanks for any advice on this.
Rob

p.s. The ZCB specification is exactly what I wanted, but the entire process really needs to be more bulletproof, and preferably a native app.

bberenberg
April 23rd, 2010, 10:47 AM
Hi Rob,

I have opened a support case on your behalf to handle closing your subscription and deleting the files on your S3 account. Could you please log into network.zmanda.com and click on the support link in the bottom right hand corner -> click open cases at the top -> and then select the single open case you have and just verify in there that this is what you want. This step is just to ensure that the real account owner is requesting this.

Regards,
Boris

wonderwebs
April 23rd, 2010, 05:18 PM
Thanks Boris, that's very much appreciated.
Rob

Saluki
August 24th, 2010, 09:47 AM
I know the original poster of this topic has given up on this issue, but we face the same root issue.

ZCB 3.x is installed on a Win 2008 server without Hyper-V. We get the same error regarding remote host closing connection. We get the same error from VSS: "Volume Shadow Copy Service error: Unexpected error calling routine IVssBackupComponents::SetContextInternal. hr = 0x80042301"

There is no other backup software running on this server.

Microsoft Windows [Version 6.0.6002] Copyright (c) 2006 Microsoft Corporation. All rights reserved. C:\Users\admin>vssadmin list writers vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool (C) Copyright 2001-2005 Microsoft Corp. Writer name: 'System Writer' Writer Id: {e8132975-6f93-4464-a53e-1050253ae220} Writer Instance Id: {64a21691-a215-4cfb-8071-b380713de3a4} State: [1] Stable Last error: No error Writer name: 'ASR Writer' Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4} Writer Instance Id: {8bd003de-de0e-454d-9177-6acce890528f} State: [1] Stable Last error: No error Writer name: 'Shadow Copy Optimization Writer' Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f} Writer Instance Id: {e556637a-edff-40d1-951b-b2615b51b8cf} State: [1] Stable Last error: No error Writer name: 'BITS Writer' Writer Id: {4969d978-be47-48b0-b100-f328f07ac1e0} Writer Instance Id: {92f4a70c-ebab-4693-8ae7-4215e4380ece} State: [1] Stable Last error: No error Writer name: 'Registry Writer' Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485} Writer Instance Id: {0bee9147-d37d-4e32-8766-6b20090e43b1} State: [1] Stable Last error: No error Writer name: 'WMI Writer' Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0} Writer Instance Id: {5d3e5c22-a709-4870-8d00-eb20fcfc8f8b} State: [1] Stable Last error: No error Writer name: 'COM+ REGDB Writer' Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f} Writer Instance Id: {575e8ad7-a122-4ee8-94f2-fc54cac662fb} State: [1] Stable Last error: No error

paddy
August 24th, 2010, 10:59 AM
Hi Saluki,

can you open a customer support case at Zmanda Network? We will need logs to look at the problem. To understand this error message we need to see the system and application event information.

Click on Support tab after logging to Zmanda Network.
Paddy

cphastings
August 25th, 2010, 06:39 AM
To get around the 8193 error you need to do two things:

1. Run ZCB 3.0.0 - I have problems with 3.0.1

2. Write a batch file and make it a scheduled task to restart the VSS services between each backup program. So my Backup Exec jobs run at 10:00 pm, then my ZCB jobs run at 3:00 am. My batch file restarts the VSS services before the Backup Exec jobs run so that they will complete correctly. There is definitely a problem with the way ZCB uses VSS, it doesn't clear out the snapshot correctly after the backup job finishes.

paddy
August 25th, 2010, 01:49 PM
Saluki,

Did you open a support case? Support team can help you and provide you a fix. These fixes will be part of next ZCB release.

Paddy

cphastings
August 26th, 2010, 03:49 AM
Paddy - when will 3.0.2 be released? I was told by someone at Zmanda that it would be out over a week ago. Still nothing.

paddy
August 26th, 2010, 10:01 AM
QA has not been completed. We are expecting to release it in 10 days.

Paddy