NYSE Outage Caused by Software Update – Really?
I just got off the phone with Joyce Wells the managing editor of Database Trends and Applications discussing the recent outage of the New York Stock Exchange (NYSE) which was caused by a software upgrade. This was confirmed in The Wall Street Journal article NYSE Says Wednesday Outage Caused by Software Update.
The first thought that came to my mind was a quote from Allan Hirt (Twitter: @SQLHA) in an online discussion we had yesterday on United Airlines. To quote Allan “You will die by incompetence well before hackers get in”.
In the technology field we have all heard the expression “People, Process and Technology“. If the New York Stock Exchange is telling the truth then the NYSE Outage was not a technology issue. Even though technology was involved, to me this was a people and process issue.
The technology exists today to make sure we don’t have outages like this. In fact we have so many different ways to implement high availability that at Ntirety we have a whole team of high-availability experts. To me the NYSE outage was caused by a lack of proper process during the upgrade. What kind of testing did NYSE do before the upgrade? What kind of redundancy and business continuity plans were in place at the New York Stock Exchange? The only silver lining here was that the New York Stock Exchange had backups that worked.
Was NYSE Another Firm Sticking Their Head Under the Sand?
At Ntirety if we performed a database upgrade and took the client out of business for close to four hours we would be out of business. Too many companies hide their heads under the sands. We have a free industry report at Ntirety titled The Disruption Epidemic Report focused on the results of the many audits of Microsoft SQL Server databases we have performed. Here are a few of the key findings from The Disruption Epidemic Report:
- 90% of SQL Server instances failed the Disaster Recovery Review
- 88% of SQL Server instances failed the SQL Server Configuration Review
- 40% of SQL Server instances failed the Security Review
- 39% of SQL Server instances failed the Database Backup Review
The bigger question to be asked is, are companies just putting their heads in the sand?. What is clear to me there is a ticking time bomb inside a lot of companies. 40% of the companies are at risk for a database security breach. It’s not if you will have a database security breach, its just a matter of when the database security breach will happen.
3rd Party Audit of Internal Processes a Must
A number of years ago at Ntirety we took a hard look at ourselves. We asked are we doing everything possible to make sure we are providing the highest possible quality service to our clients. We set a high standard for ourselves and then more importantly we are living up to that standard.
Just as auditors review a company’s financials to determine if they are a going concern, I feel all companies should have outside technology audits on an annual basis to look at process, security, internal controls and a number of other objectives.
To ensure we live up to the standards we have at Ntirety a number of 3rd party audits of our People, Process and Technology happen every year. A good example of this is the MSPAlliance MSP/Cloud Verify Program. We were one of the first companies to adopt the MSPAlliance standards.
MSPAlliance MSP/Cloud Verify Program
The MSPAlliance MSP/Cloud Verify Program is a globally recognized certification program. The MSPAlliance Unified Certification Standard, is the world’s oldest and most respected certification standard for the cloud computing and managed services industry.
To quote the MSPAlliance site:
The Unified Certification Standard (UCS) for Cloud & Managed Service Providers is based on 10 core principles of how a IT service organization should operate. These principles, or control objectives as they are called in the auditing world, provide the basis of the UCS and make it the best model for evaluating cloud and MSP companies. Each of the 10 control objectives in the UCS is supported by individual controls that are used when the company goes through the certification process. Furthermore, when the company successfully completes the certification process, they will receive a UCS audit report, that precisely details how the company implements those controls and control objectives.
Quality Services Require Quality Process, People & Technology
An independent 3rd party audit is a lot of work but it worth it if you are serious about quality. Get your head out of the sand and begin to make quality part of your company DNA. By preparing for and letting a 3rd party take a look under the covers you will be a better company for it. Anyone running a SQL Server Database should get a copy of The Disruption Epidemic Report. The report is free, full of useful data and will help you figure out if your organization has a ticking time bomb you don’t know about. If it can happen to the New York Stock Exchange, Amazon Web Service (See Article AWS Outage Takes Out Netflix, Experian, Yelp..) or United Airlines, it can happen to anyone.
My Personal Twitter Account: Michael_Corey
Ntirety Corporate Twitter Account: Ntirety
Database Administration As A Service® is a registered trademark of Ntirety, Inc