Kiwix is an information archival service and recently they were dropped by Hetzner.  It’s quite a tale.

This was posted on Mastodon:

Murphy’s law states that if things can go wrong, they will. Ideally in the worst possible way.

For us, that meant having our servers disconnected at 00:00 on a Sunday.

Our main storage backend became entirely unreachable. For the average user that meant not being able to access the library and download files, and for us that meant not being able to connect to it and see what was wrong.
Turns out that Hetzner has decided to cancel our account and terminate all servers. There was no warning (yes, we checked our spam folder), and nobody could be reached before Monday morning.

When reached, they could not explain the reason for the cancellation:
Them: – We sent you an email.
Us : -We did not receive it, can you please resend?
Them: – We don’t have it
Us: ಠ_ಠ
In the meantime, all servers had been wiped already so no way to retrieve our data.

If you are looking for a bad case of the Mondays, well, that was one.
Luckily we have mirrors and these were not affected. We grabbed a new machine somewhere else (Scaleway ; if we name-and-shame the one we might as well name-and-greet the other) and immediately started re-importing our data to our new Master server.

All in all, it still took about 48 hours to get these 8-ish TB back online.
If there is any silver lining to this, it is that we could see a few points of vulnerabilities as well as our ability to turn things around in a reasonably quick manner (here be kudos for the two heroes who manage our infra).

Learning were made, and we will see in the coming weeks/months how we can implement new safegards within our resource constraints.

/END

Hetzner is plugged into our community and they posted a followup on LowEndTalk:

Hi everyone, Our teams who review such cases do so on a case-by-case basis. We do not use AI or automated systems for these situations, but review them manually. As a general rule of thumb, we try to avoid commenting about these cases publicly. We do that so we can protect the affected customers’ personal data. But this doesn’t mean that we’re not in contact with the customer, which we are in this case.

And then Kiwix posted the letter they got from Hetzner:

Dear Kiwix

This is to inform you that your account K0799712724 and all services you have with us are going to be cancelled.

The cancellation is being done in accordance with our Terms and Conditions (https://www.hetzner.com/legal/terms-and-conditions/). The official cancellation date is the end of next month, 30.11.2024.

If you have servers that you would like to cancel to an earlier date, you can do so via the client web interface. Any servers that have not been marked for cancellation by the end of the month, will be automatically cancelled to the end of next month.

If you have domains registered via Hetzner, you will have to transfer them to a new registrar within the cancellation period.

This decision is final and cannot be appealed. Thank you for your understanding.

Kind regards

Your Hetzner Online Team

And then finally the Hetzner CEO chimed in:

We understand your frustration, and we’re sorry for the inconvenience and extra effort this issue has caused. In this case, a series of unfortunate coincidences led to this issue. The cancellation email we sent at the end of October was missed by the customer, which was beyond our control. Additionally, the termination date set for the end of the month, coincidentally fell on a Saturday, and the timeframe for deletion expired on a Sunday. We sincerely regret this sequence of events and the associated complications. However, we’ve been working on fixing the problem and have temporarily restored the servers that could be restored. And for our customers, we always assume that important data is backed up to prevent data loss in unforeseen scenarios.

While these circumstances were unusual, we recognize the need for improvement. We are committed to reviewing and optimizing our internal processes to prevent similar situations in the future. Account terminations from us are very rare, but not excluded in certain cases. We strive to be a reliable partner for our customers.

– Günther Müller, CEO Hetzner Online GmbH

Mixed in with all of this correspondence were a lot of comments by LET members, which highlighted a number of common complaints.

Hetzner has gotten big, and has become a sort of LowEndGoogle in my experience.  99.9% of the time, everything works great – provisioning, paying, etc.  But if something goes wrong, it’s hard to get someone to fix it.  Various nightmare stories were posted where people had problems, but customer service essentially ignored them or insisted it was the customer’s problem.

So what t

• Author
• Recent Posts

raindog308

Dread Lord of LowEnd Content at LowEndBox

I'm raindog308, techno polymath and long-time LowEndTalk community Administrator. My technical interests include all things Unix, perl, python, golang, shell scripting, vintage operating systems such as MVS, and relational database systems such as Oracle, PostgreSQL, and MySQL.

I also enjoy using LEBs! I have an empire of little guys for VPNs, larger guys for apps, and big guys for databases and storage. When I'm not in front a screen I'm into German Shepherd dogs, quality knives, target shooting, theology, tabletop roleplaying games, and forest hiking.

I enjoy writing technical articles here on LowEndBox to help people get more out of their systems. You can find me on LowEndTalk @raindog308

Latest posts by raindog308 (see all)

• Hetzner Terminates Kiwix With Extreme Prejudice – What Do You Think? - December 11, 2024
• Die Hard is the Greatest Christmas Movie Ever!Learn a Little Computer Trivia from the Film and Get Bonus Entries in RackNerd’s Holiday Giveaway! - December 10, 2024
• I Can’t Believe I Bought So Many VPSes on Black Friday (How to Dig Yourself Out) - December 9, 2024