I was just speaking with one of my Senior Oracle Database Administrators at Ntirety. I was asking her what some of the most common problems we
encounter when we first take over a database under management with our remote DBA service. I was surprised by the answer. She told me that we typically learn they (The Client) have never tested (Fire Drilled) the database backup.
Very Important: A database backup is no good until you have tested it. Let me say that again. A database backup is not good until you have actually physically restored something from the backup.
One of the things we typically do for prospects considering our service is a quick check-up on one of their databases. When we do this database check-up we have historically found that 1 in 5 database backup do not work correctly. What a frightening thought, 20% of the databases we check fail the valid backup portion of the database check-up. Yes even today that is still the case. I guess as a provider of remote DBA services I should not complain, because it very quickly sells the value of the Ntirety remote DBA service.
It is very important that you do a fire drill of your backups at least once a year. I would argue once a quarter. If you have not actually tested the backup, then don’t assume it will work. Experience has taught me it wont work when you need it most.
it reminds me of a something that happened in my carreer at a very major Bank in New England. They had put a new tape management system in place. With it came a new backup scheme. I was a consultant at the bank. I kept insisting they do a fire drill of the back up. My client told me I was just trying to generate additional consulting work for myself. I told the client I felt so strongly about this I would do it in my own time.
This reminds of a story from my past that illustrates this point very well. I went away on vacation with my family. We where camping in the mountains of New Hampshire. I camped with my family a lot since it was hard to get a hold of me, I tended to get quality time with my family. The 3rd night of this trip. A park ranger came up to me and asked me if I was Michael Corey that worked for the ________ Bank. There was an emergency. I hiked down to a pay phone so I could called in. My client asked me if I had developed that backup I had talked about. When I had proposed to the client that I fire drill the new database backup. I was informed I was just trying to get more revenue from the client I was consulting with. That the systems administration group had assured them all was okay and a fire drill was unnecessary. I told the client I felt so strongly about this, I would create an alternate backup method just in case on my own time.
So the client asked my while I was on the pay phone, had I created that alternate backup I had said I would. I told him I had. Well they lost the database and they learned they could not read past the second reel of the new tape management system. The only official backup the Bank had in place was useless. They were at risk of loosing 3 months of data. I asked the client if they would pay me for the time it took for me to develop the alternate backup system. He laughed; I knew he would have paid me what ever I asked.
The Moral of the story. If you don’t fire drill (test) your backup then in my opinion you don’t have a valid Oracle or Microsoft SQL Server backup. You don’t have a valid anything backup.
For everyone that reads this article please comment with one of the following
1. I have always worked for companies that Fire drill the backup
2. I have worked for a company in the Past that did not Fire Drill the database backup
I think the responses might be quit interesting to see.
Posted By Michael Corey, Ntirety