Anubis is software that “weighs the soul of incoming HTTP requests”.  That is a clever, poetic description for this software, which has an interesting pedigree and a noble mission.

If you’re running a web site with any kind of substantial content, you’re being scraped by AI.  Hard.  Organizations training LLMs have an insatiable thirst for text and wantonly suck down data from blogs, forums, and any place else where humans congregate online.

Unfortunately, this puts a drain on hosters’ bandwidth and server resources.  This can be overcome by using something like Cloudflare.

But what if you just don’t want them sucking down your data?  Enter Anubis.

Weighing the Soul

Anubis uses a proof of work scheme to force clients to do computationally expensive work.  From the docs:

Anubis uses proof of work in order to validate that clients are genuine. The reason Anubis does this was inspired by Hashcash, a suggestion from the early 2000’s about extending the email protocol to avoid spam. The idea is that genuine people sending emails will have to do a small math problem that is expensive to compute, but easy to verify such as hashing a string with a given number of leading zeroes. This will have basically no impact on individuals sending a few emails a week, but the company churning out industrial quantities of advertising will be required to do prohibitively expensive computation. This is also how Bitcoin’s consensus algorithm works.

How It Works

• Some client makes an HTTP request
• The server (Anubis) intercepts the request and decides it needs to “test” the client.
• It sends a challenge based on computing a SHA256 hash.  The client brute-forces this hash puzzle, trying numbers until it finds one that meets the required condition.
• The client then resubmits with the solution.
• The server can verify the solution much faster than the client can compute it.  If the solution is valid, the request is allowed through.

Attackers trying to spam millions of requests have to burn CPU time on each request, making attacks expensive and slow.

It’s like a doorman saying: “Do 10 jumping jacks before you come in.”.  If you’re a normal visitor, that’s fine.  If you’re a bot trying to storm the door 1,000 times a second, it’s a showstopper.

The Admin Side

The nice thing is that Anubis doesn’t require you to rewrite your web site or insert Javascript.  It’s a proxy and integrates seamlessly with popular web servers like Nginx, Apache, Caddy, etc.  For example, with Nginx, you add a few lines to your site’s server block (some proxy_set_header lines and a proxy_pass) and Anubis does the rest.

Anubis is under active development (follow their blog) and is till early on in its lifetime.  Should be an interesting project to watch.

raindog308

Raindog308 is a longtime LowEndTalk community administrator, technical writer, and self-described techno polymath. With deep roots in the *nix world, he has a passion for systems both modern and vintage, ranging from Unix, Perl, Python, and Golang to shell scripting and mainframe-era operating systems like MVS. He’s equally comfortable with relational database systems, having spent years working with Oracle, PostgreSQL, and MySQL.

As an avid user of LowEndBox providers, Raindog runs an empire of LEBs, from tiny boxes for VPNs, to mid-sized instances for application hosting, and heavyweight servers for data storage and complex databases. He brings both technical rigor and real-world experience to every piece he writes.

Beyond the command line, Raindog is a lover of German Shepherds, high-quality knives, target shooting, theology, tabletop RPGs, and hiking in deep, quiet forests.

His goal with every article is to help users, from beginners to seasoned sysadmins, get more value, performance, and enjoyment out of their infrastructure.

You can find him daily in the forums at LowEndTalk under the handle @raindog308.