Recently over on LowEndTalk, it’s been a huge controversy that HXServers decided they wanted to:
- Cancel a new location after recently releasing it (due to lack of demand)
- Cancel annual subscriptions
The controversial part, though, comes from the fact that they didn’t go about either of the above-mentioned tasks properly.
They didn’t notify customers they were going to delete their data first. They just deleted it, and then refunded payments.
And well…
Ever since, the HQ over at HXServers has represented something akin to this meme:
I decided to reach out to HXServers directly to get their response to the situation. Here’s what they had to say:
Many mistakes were made before, during, and after our plan to decommission our retired services and platform. Although refunds were processed correctly, we completely failed in notifying customers to prepare for migration. As a result, retired platform plans were terminated with seemingly no notice to our customers, causing data loss, mass confusion and frustration. As a company, HXServers needs to do better in the future, and we take this as a hard lesson to learn what not to do when handling customer data. Although this only affected a very small portion of our customer base, we’re handling this as a disaster, caused by gross negligence. Customers affected by our gross negligence will receive a new VPS, with the exact same pricing, and hardware specification free for the first month, and future months will match the previous price the customer paid. Although we’re taking ownership of the recent issues, we must express our worrisome of our operations. As a company, HXServers needs to reevaluate our workflows, our (now former) employee(s), and our policies on customer data and customer notification processes. HXServers is committed to constant improvement, and hopefully some day we can rebuild the trust we once had in this community. We’re extremely saddened by our recent actions and owe each and every one of our affected customers a huge apology. To those who are affected, we’re so sorry for how we handled this, and we will never make these mistakes again.
Thanks,
HXServers
They also released the following root cause analysis thread:
Hello all,
I’m sure by now you’ve heard about the recent issues with HXServers. I wanted to create a new discussion to better inform existing customers, and potential new customers with actual facts instead of conjecture and opinions of what happened.
What was the reason to “RETIRE” plans?
HXServers started on “Platform 1.0” – Using Virtualizor. We switched to VirtFusion, calling it “Platform 2.0”. All customers hosted on Virtualizor had their VPS plan named changed to “RETIRED”. Originally, we stated we had no intention on forcing customers to Platform 2.0 under VirtFusion. Unfortunately, we’ve had to change that decision, as part of Platform 1.0 was renting IPs at $2 / IP from ReliableSite, GTHost, etc. We don’t mind operating in the negative, but this was severely in the negative, and would’ve reduced our ability to improve our platform.What was the plan to retire Platform 1.0?
The plan was to notify customers on October 1st that migration needs to happen by October 21st. Our intention was to provide a new VPS to customers under Platform 2.0, with the same VPS specs and price. Any customer who paid for October would be refunded in exchange for the inconvenience of having to migrate.What actually happened?
We did not notify customers. We had thought we did. This was gross negligence on our part. We relied on automation to notify customers and never did double check that automation had run successfully. The only piece of automation that worked, was automated refunds and automatic VPS termination. What makes things worse, is we didn’t know automation failed for sending notifications until it was blatantly brought to our attention after VPS’ had already been terminated. Upon investigation, the Announcement in our billing system wasn’t even posted live, like we had initially claimed on LowEndTalk. The announcement, the email notification, every method we use to communicate to our customers, failed – without our knowledge. We could’ve checked this at any time, but we didn’t.What VPS’ were terminated??
LA – RETIRED – All VPS have been terminated, without notice, with refund.
NY – RETIRED – No VPS have been terminated.
Miami – RETIRED – All VPS have been terminated and refunded where renewal was recent. Any VPS with a renewal date after October 21st was not terminated but will not renew and will be terminated at the next billing cycle.My VPS is terminated, can I get my data??
This is the biggest problem in what we’ve done. Any VPS that has been terminated, has been fully removed from our servers. Data recovery is not possible. Unfortunately, when our system removes a VPS, it also removes any associated backup of that VPS at the same time.What happened with Dallas??
Dallas was a new location. We rented servers from GTHost. We rented an IP Block from IPXO. We launched Dallas on October 13th, and by October 16th, we decided Dallas wasn’t going to be a popular location. The 10 VPS that was hosted in Dallas, was refunded and terminated October 21st. Notifications were not sent for Dallas removal either.What are you going to do for RETIRED NY?
Retired plans in New York will be properly notified and terminated. We will be triple checking all customers on retired plans in NY are aware of the termination before any termination happens.Many mistakes were made before, during, and after our plan to decommission our retired services and platform. Although refunds were processed correctly, we completely failed in notifying customers to prepare for migration. As a result, retired platform plans were terminated with seemingly no notice to our customers, causing data loss, mass confusion and frustration. As a company, HXServers needs to do better in the future, and we take this as a hard lesson to learn what not to do when handling customer data. Although this only affected a very small portion of our customer base, we’re handling this as a disaster, caused by gross negligence. Customers affected by our gross negligence will receive a new VPS, with the exact same pricing (monthly only), and hardware specification free for the first month, and future months will match the previous price the customer paid. Although we’re taking ownership of the recent issues, we must express our worrisome of our operations. As a company, HXServers needs to reevaluate our workflows, our (now former) employee(s), and our policies on customer data and customer notification processes. HXServers is committed to constant improvement, and hopefully some day we can rebuild the trust we once had in this community. We’re extremely saddened by our recent actions and owe each and every one of our affected customers a huge apology. To those who are affected, we’re so sorry for how we handled this, and we will never make these mistakes again.
Number of VPS affected: 32
Number of existing VPS: 1,432
So, ultimately, everyone was refunded for the services rendered, and alternatives/credits were offered.
A relatively reasonable response to an extremely poorly thought out mistake.
Why Are Some Community Members Still Mad?
The biggest complaints are centered around:
- Data loss, which is understandable. Any provider should always do their best to protect their clients’ data and give them a heads up before cancelling any services. That’s a serious lapse of thought.
- How they went about communicating and explaining the situation.
For example, there does seem to be some questionable backtracking about when customers were informed about the location closure.
HXServers explained the situation as an employee forgetting to hit the publish button.
It’s a possibility. Unlikely, but a possibility…
But I think the important thing to keep in mind here is that all parties involved received their money back. Taking money and not rendering services (scamming) isn’t allowed by any provider on here or LET.
It was a terrible way to go about policy/location changes, but hopefully, we never see this from HXServers again.
Going forward, HXServers will have to repair their reputation to regain the trust of our audience.
We’ll see how they navigate these murky waters and their damaged reputation from now on.
You might also enjoy The Art of Writing a Good Root Cause Analysis/Reason for Outage.
Related Posts:
- Path.net Seeks Discord Subpoena for Ex-Employee and Competitor’s Data - December 12, 2023
- Interviewing Christopher Cantwell - December 6, 2023
- Uptime Kuma: Your New Favorite Self-Hosted Uptime Monitoring Tool - December 1, 2023
I really value the fact that you took the time to share the information with others. I had been doing research on a certain subject for quite some time when I came across this topic, and I realized that it was really relevant to that subject.