When AT&T and other carriers to some extent suffered a massive outage affecting major metropolitan areas, people couldn’t make phone calls, including 911 emergency calls. In one city, a fire department official suggested people use a landline to call 911.
Continue readingInternet Outages Show No Favorites
The 10 largest internet outages of 2022 managed to hit a wide swath of the user community, ranging from music and messaging sites to gaming sites. While more business-related sites didn’t make the top 10, one of our client’s experiences showed the benefits of planning ahead and staying nimble.
Having a plan you can implement immediately is critical because you can get knocked off the internet in a nanosecond. It’s especially problematic for businesses such as our clients who can have up to 30 users. We’re not big enough to have multiple redundancies for our systems, nor are we big enough to deal directly with big providers. Our businesses likely pass through several hands before we reach the internet, and any of them can go down in a heartbeat.
Because of this uncertainty, one of our clients asked us to help them put together a process. They learned the consequences of being planless while working with another IT company. They knew they were going to have a planned outage because they were moving their physical location. They sell a product and take orders online.
Our solution was to have them take “paper” orders over the telephone as soon as the internet service was shut down to move their equipment. We had a process in place to get their tech system operational before lunchtime on moving day, and everything worked smoothly. As the system started to come online, they used a combination of paper and online orders and then switched to online only. The online system worked while the employees physically moved to their new location.
Their solution succeeded because our plan also included having the right infrastructure in place. We knew the internet connections and Wi-Fi network could handle the load once we put everything online.
Your solution will depend on your business’s needs. Everyone should prioritize telephone and email service because you can’t afford to go incommunicado. Who provides those services for you – and who provides them for your provider? You might not be able to get that latter provider, but it would be helpful. Is the phone service in the cloud? How is it set up? What electricity service options are available in your building?
We can help you develop or improve an outage emergency plan by looking at your equipment and configurations. We can recommend new equipment or configurations as needed and help you install better backup systems to help prevent data loss. Call us – 973-433-6676 – or email us to talk about it.
Oh, and just what were the 10 biggest internet outages in 2022 based on outage reports? According to Downdetector, an internet tracking service of Ookla, which is best known for speed testing, they were:
- Spotify – 2.9 million
- WhatsApp – 2.9 million
- Discord – 1.1 million
- Roblox – 700,000
- Instagram – 600,00
- Twitter – 500,000
- Call of Duty – 350,000
- Reddit – 300,000
- Snapchat – 300,000
- TikTok – 300,000
The Night the ‘Office’ Went Dark
Two weeks ago, Microsoft’s Office 365/Microsoft 365 cloud-based services, which include the online Office apps, went down for as long as five hours in some parts of the United States, Western Europe and India. As of this writing last week, users were still reporting sporadic outages of some services. So far, our clients have not been affected, but It’s a fluid situation.
No matter what happened and what might happen, you can protect your data by backing up your files in more than one place. The outage hit Azure, a Microsoft platform, that we use for our backup for Microsoft 365, which was known as Office or Office 365. Microsoft 365 is built on Azure. It wasn’t the first time it was hit by an outage, and it won’t be the last. In its most recent outage, Azure and Microsoft 365 were affected by what was reported as a major Azure Active Directory authentication issue. In practical terms, users got kicked out of cloud-based applications, such as Office, Outlook, Exchange, Teams and SharePoint. It lasted from roughly 5 to 10 p.m. on Sept. 28.
Microsoft said very little but referred to an update in their network structure – and then said they rolled back the changes to an older version. It was all automated, but because of the authentication issues, some administrators couldn’t see the changes. Over the course of two weeks, problems cascaded worldwide – and randomly. Despite what Microsoft reports, we haven’t seen any satisfactory explanations of what happened and how it was fixed.
In this case, we don’t know of any data losses, but if you can’t access your files, they’re as good as lost until the service comes back. That’s where multiple backups are valuable. As a small business or home user, you may still have a computer with a version of Office installed. If you can pull a file from another storage site, such as Mozy (one of our partners) or Dropbox, or from an external hard drive, you may be able to work with your file. You also can store files on your computer’s hard drive.
One of the problems with an outage such as the one that hit Azure, you never know when it will hit. We reported on an outage two years ago, and we did have at least one client who was affected.
At the time we were affected, we were doing a setup at a client and needed to get a big file from Azure. When I logged in to get it, I got no access; I just got a message they would send a text. I had an external hard drive with an old version of the file, and that was not suitable. Transferring the file remotely from my office computer would have taken too long. We solved the immediate problem by transferring the file from my computer to my Dropbox account and then downloading it from there. We and our client were fortunate that I had the capability – files stored on a computer I could access and Dropbox – to initiate a solution.
Azure solves recovery issues for us because it works seamlessly in the background with Office 365, including Outlook and its PST files for your email. For some, backing up email may be more important than backing up files.
As an IT professional, I like Azure’s ability to generate reports – with more capabilities coming online all the time. Microsoft constantly uses customer feedback to add more power to the platform. That gives us the ability to go back into our clients’ backup records to trace incidents and to restore files after a catastrophic event. That’s critical because it can be 90 days – sometimes longer – before a hack or data loss is discovered by a client. When that happens, we can go back in time through the power of Azure to find data files that help us help you recover.
We can help you with continuity during various service outages, but it all starts with accessing your data. Call us – 973-433-6676 – or email us to discuss your backup and file storage options. The cost of a workable redundancy system may pale in comparison to the loss you could suffer from the loss of data or the loss of access to your data.
Pulling the Plug at BA
An IT contractor for British Airways accidentally pulled the wrong plug as travelers queued up for a holiday weekend. That pulled the plug on travel plans for some 75,000 passengers and cost the airline a reported $128 million. It makes you wonder: Who else is vulnerable to an “oops”? Probably everybody, but we can all reduce our risk exposure with good backup systems.
The contractor’s mistake occurred at BA’s data center, and it caused the airline to cancel flights at London’s two airports, Heathrow and Gatwick. Besides the millions BA will pay for their customers’ inconvenience, there will be an investigation that will draw on company resources. It affected operations throughout the BA empire.
The incident raises two questions?
- Why wasn’t that cord clearly marked in some way, shape or form to give anyone a clue that it absolutely had to stay plugged in?
- Why wasn’t some sort of backup system available?
To me, the second question gets to some very fundamental issues about how major companies operate in today’s world. One of them is cost-cutting. News reports indicate that BA’s management was under pressure to cut costs and boost profit margins in a highly competitive industry. Well, we’re all in highly competitive situations, and we all want to raise our profit margins because we can’t raise prices – at least not without significant pushback.
But at some point, the large corporations that provide so many services for small businesses and consumers, like us, need to step up their game. They should be taking the steps our clients and customers would demand of us to make sure we serve them as expected. If one of the package delivery services, such as UPS or FedEx – or even the Post Office – has an IT failure that causes one of our deliveries to miss a deadline, the consequences for us will be much greater proportionately than for the big corporation.
Mechanical problems at a specific location can happen, but a data center problem should never happen because there are so many ways to add backups. Here are a few examples of what they can do:
- Have a battery-powered back-up system in place so that everything in the system can be saved.
- Have a back-up location that can be immediately and automatically activated so that critical operations continue.
- Make time to make sure everyone is trained and retrained for all tasks they need to do on the system.
- Keep your hardware and software up-to-date to make sure you have all performance and security measures installed. One of the things we’ve seen in many IT-related catastrophes, such as WannaCry ransomware, is that large businesses simply don’t bother to invest in technology in order to cut costs. They wind up paying more when something happens.
Let’s take this one step farther. You can be exposed to many of the same risks and can benefit from the same preventive measures in your office and at home. You can buy battery back-up systems and plug in your servers, routers and computers to give you time to save your data. You can use remote storage – the cloud – to save data and apps. You can make sure everyone knows what to do and not do with your system. You can automatically update your systems – especially your operating system and app software – to keep them secure.
Call us – 973-433-6676 – or email us if you have any questions about keeping your home and office systems running in the face of any incidents – manmade or natural. We can also audit your system and give you a plan to stay plugged in.