MXroute has a rough go for a few days, as one of their servers failed spectacularly. Servers can fail multiple ways – blown CPUs, bad RAM, flakey networks – but the worst is a storage failure, and that’s what happened to the server named Lucy.
MXroute proprietor Jarland Donnell posted a lengthly root cause analysis which is quite interesting to read. Some things that are worth learning:
- Backing up huge servers via traditional backup tools is quite challenging, at least when it comes to restoring them.
- Software vendors don’t always set defaults to the most helpful settings and it’s dangerous to assume that what they ship is what you need.
- MXroute theorizes that extreme load on a RAID10 array can cause one drive to be so busy that it’s failed out of the RAID array, and when that problem cascades to a second disk, it’s game over for that storage.
There’s a wonderful line from the movie Repo Man in which Harry Dean Stanton tells his young protégé, “Most people spend their lives avoiding tense situations. Repo men spend their lives getting into tense situations.” And so it often is for sysadmins, and reading the story of the heroic efforts taken to recover these systems in the dark of the night gave me PTSD from previous incidents in my career. Man-vs-server combat can be intense and this is a harrowing tale.
One of the things we always preach is transparency and in this case, @jar is admirably telling the community exactly what’s happening. Kudos!
Would I sign up with MXroute today? You bet. A provider who works this hard to fix the inevitable problems is a provider worth using. I just checked and their Black Friday deals are still available:
Small
10GB Storage
Unlimited Domains
Unlimited Email Accounts
300 outbound per hour (per email acct)
$15 / 3 years
[ORDER]
Medium
25GB Storage
Unlimited Domains
Unlimited Email Accounts
300 outbound per hour (per email acct)
$30 / 3 years
[ORDER]
Large
50GB Storage
Unlimited Domains
Unlimited Email Accounts
300 outbound per hour (per email acct)
$34.50 / year
$69 / 2 years
$99 / 3 years
[ORDER]
Related Posts:
I also enjoy using LEBs! I have an empire of little guys for VPNs, larger guys for apps, and big guys for databases and storage. When I'm not in front a screen I'm into German Shepherd dogs, quality knives, target shooting, theology, tabletop roleplaying games, and forest hiking.
I enjoy writing technical articles here on LowEndBox to help people get more out of their systems. You can find me on LowEndTalk @raindog308
Bad service That have
All mails stop from last 3 days