Of all the categories I thought we might add to LowEndBox one day, “Datacenter Fires” was not on my short list. But maybe we need to create this section. On the heels of OVH’s huge SBG Datacenter fire in March, we’ve learned that WebNX has suffered a serious fire in their Ogden, Utah datacenter.
According to news reports, an emergency generator kicked in during a power outage, and then caught fire.
If you’re not familiar with datacenter power design, here’s a quick 101: most datacenters have a UPS system that is designed to provide short-term power should public power fail – long enough for an emergency generator to start up and begin providing long-term electricity. In most datacenters, generators can run for weeks or months and can be refilled with diesel while operating. Some datacenters even have multiple generators that can independently provide the full load of electricity, in theory providing infinite uninterrupted power as long as they are refueled.
In this case, public power failed and when the emergency generator started, it apparently caught fire for reasons unknown. For safety reasons, the fire department cut power to the entire building.
WebNX posted an update on Facebook and on their web site:
Now that we have a better understanding of what happened we would like to give everyone an update.
One of our old generators that have worked for years and was recently load tested had a mechanical failure and caught fire resulting in power being cut to our core routers and fire suppression system controlling the fire. Unfortunately, the fire department opted to cut power to the rest of the building as a precaution even though the power systems were independent. We are currently waiting for an emergency inspector to arrive to give the all-clear so we can bring most of the servers in Ogden back online. Some servers will have an extended outage as they may require rebuilds due to some water damage. Those builds have a high probability that data is intact.
We would like to thank you for your patience and know that we are doing everything we can to get everyone back online.
WebNX feels that many customer systems will be unaffected, but smoke damage is often very widespread. According to experts,
Smoke particles from fire travel fast, at around 10 metres per second, and can quickly fill a room or data centre hall. Under the microscope, these particles have jagged edges. As they spread through the air and enter servers and other equipment, these minute fragments cause scratching and damage.
Unfortunately, the deterioration does not stop there. Smoke settles on all surfaces of data centre equipment, even after the fire is extinguished, leaving a residue. Micro-pitting begins to occur on metal surfaces and although cleaning operations may initially look successful, the smoke residue that remains continues to react with moisture in the air, causing corrosion. Isolated faults typically begin to appear within a few days and can develop into much larger problems over time.
WebNX provides a 100% uptime SLA, so if you’re affected, please be sure to read the terms.
Let’s hope that articles in this category do not continue to grow!
Related Posts:
I also enjoy using LEBs! I have an empire of little guys for VPNs, larger guys for apps, and big guys for databases and storage. When I'm not in front a screen I'm into German Shepherd dogs, quality knives, target shooting, theology, tabletop roleplaying games, and forest hiking.
I enjoy writing technical articles here on LowEndBox to help people get more out of their systems. You can find me on LowEndTalk @raindog308
- One Week From Tomorrow…THE WORLD WILL LOSE THEIR MINDS!Lines Are Already Forming! - November 21, 2024
- Crunchbits Discontinuing Popular Annual Plans – The Community Mourns! - November 20, 2024
- RackNerd’s Black Friday 2024: Bigger, Better, and Now in Dublin! - November 19, 2024
Hoping that WebNX is able to recover quickly and efficiently, minimizing further negative impact for customers. Good luck to WebNX.
> WebNX provides a 100% uptime SLA, so if you’re affected, please be sure to read the terms.
From their SLA:
Outages due to Force Majeure, including FIRE, earthquakes, riots, explosion, Sharknado, declared or undeclared Wars, terrorist acts, insurrection, extra-terrestrial invasion, injunction or other event(s) outside the control of WebNX and its partners are not eligible for SLA credit.
Sharknado! LOL.
Other product is https://www.boxbilling.com/