OpenBSD holds a special place in my heart.  I’ve spent a lot of time with it and getting to know its quirks.

OpenBSD was started as a fork of NetBSD because Theo de Raadt can’t play nicely with others. I kid, I kid, but that’s where it came from.  OpenBSD’s identity has always been “secure” and they do have a legendary focus and reputation.  Throughout the operating system, cryptography is thorough, randomization is everywhere, and safe defaults are chosen.  OpenBSD is battle-tested and continuously audits their code, attacking it with many eyes and many fuzzers.  OpenBSD is also the mother to many other important projects, such as OpenSSH (the SSH you are almost certainly using), OpenBGPD, LibreSSL, OpenSMPTD, OpenNTPD, the pf firewall, and others.

Some things I like about OpenBSD:

• You get a kick-ass song with each version!  Yes, I’m serious.  My favorite is Ponderosa Puff from 3.6.  Check out our article: Top Ten OpenBSD Release Songs.
• OpenBSD to me is a sysadmin’s operating system.  Essentials like stopping/starting/enabling/etc. services is as easy on OpenBSD as it is on Linux.  It’s also just as easy to install packages.  Configuring root filesystem backups is preconfigured.  A lot of the system scripts are in Perl.  Etc.
• The documentation is <chef’s kiss>
• I have not played with it, but it includes a OpenBSD-authored hypervisor
• I am not a security pro, but based on what I’ve read and the project’s reputation, I believe OpenBSD is highly secure in competent hands.  That doesn’t mean other operating systems can’t be highly secure in competent hands.
• I like its logo the best of all the BSDs but that’s just me.

Some things I don’t like:

• OpenBSD has a reputation for being slower than other operating systems.  I think early on this was due to OpenBSD having poorer SMP support.  On the other hand, it runs on a lot more hardware than most competitors and goes to great lengths to enhance security, so…
• The installer is fine but disk setup is a bit primitive.
• In my experience, the further you get from core project code, the more likely it is you’re going to have a problem.  See the Certbot example below.  This isn’t entirely OpenBSD’s fault, but it’s a small userbase that is never going to be the porting priority, plus it is quirky and persnickety about security and correctness, so packages primarily designed with other operating systems in mind are often problematic.  You’re always in the .01% of users that isn’t well-tested.
• The culture is a bit cringe.

The OpenBSD culture is not particularly newcomer-friendly, since the documentation is so good you should just read that, and if your question isn’t answered there, you should go look in the source code.

Nor are the developers particularly community-oriented.  The developers are making this OS.  If you like it, cool, support it, but regardless, the devs are going to do what they want.  There’s a tension between “what they want” and “continued community support” but I think the former wins out sometimes.

I’ve had great and awful experiences with OpenBSD.  I’ve run firewalls with it and they just go forever.  Also ran a RAID-1 fileserver for a long time with OpenBSD.  Early on, running OpenBSD on VMs seemed to always require a special parameter but these days it seems to work fine.  On the other hand, I got all kinds of weird crashes and dumps when playing with it on some boards (Beagle, etc.) over the years.

I’m installing OpenBSD 7.5 for amd64.

Booting off cd75.iso.  Either I decided to show how OpenBSD’s mirror system and so intentionally choose the .iso that doesn’t include file sets, or I picked the wrong .iso.

Ah, that famous white text on blue on black…why, I don’t know, but it’s a classic color scheme.  And then the famous prompt!

The OpenBSD installer is not GUI-based.  It’s not even TUI-based.  It’s a scrolling questionnaire.

I’m using DHCP again but regardless, it’s essentially filling out a short form.  Very efficient, if not the most approachable interface.

Still, if you can’t answer basic questions in a text interface, OpenBSD probably isn’t for you.

Interesting…also defaults to MBR instead of GPT.  So only NetBSD so far defaults to GPT!

To be honest, I find slicing things up like this tedious, not to mention error-prone.  You can’t just say “give me a 50GB root, 2GB for swap, and put the rest in /apps”.  I suppose if I did this all day long, it’d become second nature…maybe not.

I use Auto layout because OpenBSD lays out permissions and features properly for you, and this is recommended  When I run OpenBSD, I usually create a l (L) slice called /data where I put my apps and stuff.

Very quick to partition!  Note the different filesystem attributes.  Now let’s look at sets.

I entered ‘http’ and then was given the option to choose from mirrors.

These days, there’s no reason not to install everything.  Installing X is helpful because eventually you’ll come across some package that wants it.

‘bsd’ is the kernel.  ‘bsd.mp’ is the kernel for multiprocessing.  You would think that multiprocessing would just be the default in 2024, but remember that OpenBSD runs on a wide diversity of hardware.  bsd.rd is the rescue kernel.

Ripped through these very quickly.

CONGRATULATIONS!

Now here I made a mistake.  I knew I needed to remove the ISO in the Vultr panel (simulating a “CD eject”).  So I went and did that, and then remembered that if you remove an ISO on Vultr, the VM reboots.  I think that’s a silly policy but that’s how Vultr’s panel works.

If I had hit “(R)eboot” in OpenBSD first, it would have unmounted the new filesystems first.  Since it didn’t, on reboot…

Everything recovered fine.

And we’ve booted!

Note that on first boot, some special things are done.  I was curious what they were…but rc.firsttime doesn’t exist.  In fact, it’s kind of hard to see – the only time it exists is after the install is done but before first boot is complete.   I looked at it and it runs firmware updates, and also calls syspatch to tell you if there are any available patches for your system.

Of course, on every boot, the kernel is relinked to randomize it.

Secure Levels

OpenBSD has the notion of securelevel.  FreeBSD implements this as well.  NetBSD’s is a bit more primitive.  Here  I’ll explain OpenBSD’s model.

The kernel has a setting called secure mode.  It’s on a scale of -1 to 2 with rising security.  At -1 and 0, no extra security is enforced.  At security level 1 (the default), some new things take effect:

• /dev/mem and /dev/kmem cannot be opened

• raw disk devices of mounted file systems are read-only

• system immutable and append-only file flags may not be removed

• Some sysctl variables cannot be changed

• GPIO pins are locked down

At securelevel 2, you get all of the above plus time cannot be set back, and firewall rules cannot be changed.

Why would you want securelevel 2?  Perhaps you have a router where the rules are loaded at boot.  Logs are set to append-only and the firewall rules can’t be changed without a reboot.  If that router is compromised, you have a much better chance of seeing what happened in the logs and the attacker can’t change your rules.

Looking Around

openbsd# df -h
Filesystem     Size    Used   Avail Capacity  Mounted on
/dev/sd0a      986M    122M    815M    14%    /
/dev/sd0k     24.3G   18.0K   23.0G     1%    /home
/dev/sd0d      3.9G   14.7M    3.7G     1%    /tmp
/dev/sd0f      6.7G    1.4G    4.9G    23%    /usr
/dev/sd0g      986M    303M    634M    33%    /usr/X11R6
/dev/sd0h      8.8G    146K    8.4G     1%    /usr/local
/dev/sd0j      5.8G    2.0K    5.5G     1%    /usr/obj
/dev/sd0i      2.5G    2.0K    2.3G     1%    /usr/src
/dev/sd0e      6.9G   10.5M    6.5G     1%    /var

After all the recommended sizes, whatever’s left over was dumped into /home.  I could have changed this in the install but it’s fine for now.

Usually, my MO is to create /web and because I’m a one-big-partition kinda guy, it has plenty of space.  That won’t work here, because / is less than 1GB.  We’ll come back to this.

BTW, OpenBSD by default has a wheel group but it’s disabled (by not having any users in it).  “wheel” is a group (as in an /etc/group group) that lists users permitted to su to root.  If it’s empty, then there is no restriction – anyone who knows the root password can su.  OpenBSD ships with no one in the wheel group.

Installing Software

Like FreeBSD, OpenBSD has ports and packages.  If you want ports, you have to download and explode ports.tar.gz to create the ports hierarchy.  Then you can go into whatever application you want and configure/make/make install.

We’ll be using packages in this example.  Let’s try wget.

openbsd# pkg_add wget
quirks-7.14:updatedb-0p0: ok
quirks-7.14 signed on 2024-05-28T20:13:45Z
quirks-7.14: ok
wget-1.21.4p0:libiconv-1.17: ok
wget-1.21.4p0:gettext-runtime-0.22.5: ok
wget-1.21.4p0:libunistring-0.9.7: ok
wget-1.21.4p0:libidn2-2.3.0p0: ok
wget-1.21.4p0:libpsl-0.21.1: ok
wget-1.21.4p0:bzip2-1.0.8p0: ok
wget-1.21.4p0:pcre2-10.37p2: ok
wget-1.21.4p0: ok
openbsd# 

Way to go, OpenBSD.  It just works.  In the past, I think you had to configure some kind of PKG environment variables first, but I’m glad that’s gone and pkg_add (and its cousins, pkg_delete, pkg_search, etc.) work out of the box.

Let’s try MariaDB.

openbsd# pkg_info -Q mariadb
mariadb-client-10.9.8v1
mariadb-server-10.9.8p0v1
mariadb-tests-10.9.8v1
p5-DBD-MariaDB-1.23
openbsd# pkg_add mariadb-server mariadb-client
quirks-7.14 signed on 2024-05-28T20:13:45Z
mariadb-server-10.9.8p0v1:xz-5.4.5: ok
mariadb-server-10.9.8p0v1:lz4-1.9.4: ok
mariadb-server-10.9.8p0v1:zstd-1.5.5: ok
mariadb-server-10.9.8p0v1:snappy-1.1.10p1: ok
mariadb-server-10.9.8p0v1:lzo2-2.10p2: ok
mariadb-server-10.9.8p0v1:nghttp2-1.58.0: ok
mariadb-server-10.9.8p0v1:ngtcp2-1.3.0: ok
mariadb-server-10.9.8p0v1:nghttp3-1.2.0: ok
mariadb-server-10.9.8p0v1:curl-8.7.1: ok
mariadb-server-10.9.8p0v1:libxml-2.12.7: ok
mariadb-server-10.9.8p0v1:mariadb-client-10.9.8v1: ok
mariadb-server-10.9.8p0v1:p5-Params-Util-1.102: ok
mariadb-server-10.9.8p0v1:p5-Module-Runtime-0.016p0: ok
mariadb-server-10.9.8p0v1:p5-Math-Base-Convert-0.11p0: ok
mariadb-server-10.9.8p0v1:p5-Clone-0.46: ok
mariadb-server-10.9.8p0v1:p5-SQL-Statement-1.414: ok
mariadb-server-10.9.8p0v1:p5-FreezeThaw-0.5001p0: ok
mariadb-server-10.9.8p0v1:p5-MLDBM-2.05p0: ok
mariadb-server-10.9.8p0v1:p5-Net-Daemon-0.49: ok
mariadb-server-10.9.8p0v1:p5-PlRPC-0.2020p0: ok
mariadb-server-10.9.8p0v1:p5-DBI-1.643p0: ok
mariadb-server-10.9.8p0v1:p5-DBD-MariaDB-1.23: ok
mariadb-server-10.9.8p0v1: ok
Running tags: ok
The following new rcscripts were installed: /etc/rc.d/mysqld
See rcctl(8) for details.
New and changed readme(s):
	/usr/local/share/doc/pkg-readmes/mariadb-server

So I personally don’t think that MariaDB should require those Perl modules but apparently OpenBSD does.

Note the line about “rcscripts”.  Let’s start MariaDB.  I also think it should be /etc/rc.d/mariadb but I can understand the compatibility argument.

openbsd# rcctl start mysqld
mysqld(ok)
openbsd#

Take a bow, OpenBSD.  (But see below…no bow, actually).

So which do you prefer:

rcctl start mysqld
rcctl mysqld start

The argument for “rcctl start” is that it’s more “natural English like” and flows hierarchically: first which command, then what you want the command to do, and then the target.  On the other hand “rcctl mysql start” allows you to up-arrow and replace “start” with “stop” more easily.

Choices…

At this point I would install Nginx and PHP.  However, with OpenBSD we have a unique option.  This is the only BSD that has created its own web server.  It’s called, what else, OpenHTTPD.

I found this HOWTO on an interesting site called openbsdhandbook.com.  I have no idea why the site exists as I couldn’t find an “about” page but it uses OpenHTTPD for WordPress.

I’ll stick with Nginx.

openbsd# pkg_add certbot nginx php php-cgi fcgi php-curl php-mysqli php-zip unzip
quirks-7.14 signed on 2024-05-28T20:13:45Z
certbot-2.9.0:sqlite3-3.44.2: ok
certbot-2.9.0:libffi-3.4.4p1: ok
certbot-2.9.0:python-3.10.14: ok
certbot-2.9.0:py3-ConfigArgParse-1.5.3p1: ok
certbot-2.9.0:py3-zopeevent-4.5.0p0: ok
certbot-2.9.0:py3-zopeinterface-5.5.2: ok
certbot-2.9.0:py3-zopecomponent-4.2.2p8: ok
certbot-2.9.0:py3-six-1.16.0p3: ok
certbot-2.9.0:py3-configobj-5.0.8: ok
certbot-2.9.0:py3-parsedatetime-2.6p1: ok
certbot-2.9.0:py3-setuptools-68.0.0v0: ok
certbot-2.9.0:py3-cparser-2.21: ok
certbot-2.9.0:py3-cffi-1.16.0: ok
certbot-2.9.0:py3-cryptography-42.0.5: ok
certbot-2.9.0:py3-openssl-24.0.0: ok
certbot-2.9.0:py3-josepy-1.14.0: ok
certbot-2.9.0:py3-distro-1.8.0: ok
certbot-2.9.0:py3-tz-2024.1: ok
certbot-2.9.0:py3-pyRFC3339-1.1p2: ok
certbot-2.9.0:py3-brotli-1.0.9p3: ok
certbot-2.9.0:py3-certifi-2023.7.22: ok
certbot-2.9.0:py3-idna-3.6: ok
certbot-2.9.0:py3-urllib3-1.26.15: ok
certbot-2.9.0:py3-charset-normalizer-3.3.2: ok
certbot-2.9.0:py3-requests-2.31.0: ok
certbot-2.9.0:py3-requests-toolbelt-0.10.1: ok
certbot-2.9.0:py3-acme-2.9.0: ok
certbot-2.9.0: ok
nginx-1.24.0p0: ok
Ambiguous: choose package for php
a	0: 
	1: php-8.1.28
	2: php-8.2.19
	3: php-8.3.7
Your choice: 3
php-8.3.7:femail-1.0p1: ok
php-8.3.7:femail-chroot-1.0p3: ok
php-8.3.7:oniguruma-6.9.9: ok
php-8.3.7:capstone-5.0: ok
php-8.3.7:libsodium-1.0.19: ok
php-8.3.7:argon2-20190702p0: ok
php-8.3.7: ok
Ambiguous: choose package for php-cgi
a	0: 
	1: php-cgi-8.1.28
	2: php-cgi-8.2.19
	3: php-cgi-8.3.7
Your choice: 3
php-cgi-8.3.7: ok
fcgi-2.4.0p18:p5-FCGI-0.82: ok
fcgi-2.4.0p18: ok
Ambiguous: choose package for php-curl
a	0: 
	1: php-curl-8.1.28
	2: php-curl-8.2.19
	3: php-curl-8.3.7
Your choice: 3
php-curl-8.3.7: ok
Ambiguous: choose package for php-mysqli
a	0: 
	1: php-mysqli-8.1.28
	2: php-mysqli-8.2.19
	3: php-mysqli-8.3.7
Your choice: 3
php-mysqli-8.3.7: ok
Ambiguous: choose package for php-zip
a	0: 
	1: php-zip-8.1.28
	2: php-zip-8.2.19
	3: php-zip-8.3.7
Your choice: 3
php-zip-8.3.7:libzip-1.8.0p0: ok
php-zip-8.3.7: ok
Ambiguous: choose package for unzip
a	0: 
	1: unzip-6.0p17
	2: unzip-6.0p17-iconv
Your choice: 1
unzip-6.0p17: ok
The following new rcscripts were installed: /etc/rc.d/nginx /etc/rc.d/php83_fpm
See rcctl(8) for details.
New and changed readme(s):
	/usr/local/share/doc/pkg-readmes/femail-chroot
	/usr/local/share/doc/pkg-readmes/nginx
	/usr/local/share/doc/pkg-readmes/php-8.3

Cool.  Note that PHP is not configured yet.  That requires:

openbsd# ls /etc/php-8.3                                                                             
openbsd# cp /etc/php-8.3.sample/* /etc/php-8.3                                                       
openbsd# 

My preference would be for a sensible default.

php-fpm comes with a www pool already configured.  More on that in a moment.

Let’s start it:

openbsd# rcctl start php83_fpm 
php83_fpm(ok)
openbsd# rcctl enable php83_fpm
openbsd# ps ax |grep php
59013 ??  S        0:00.01 php-fpm-8.3: master process (/etc/php-fpm.conf) (php-fpm-8.3)
43846 ??  Ic       0:00.00 php-fpm-8.3: pool www (php-fpm-8.3)
12243 ??  Ic       0:00.00 php-fpm-8.3: pool www (php-fpm-8.3)

Nginx

The stock nginx.conf doesn’t have any kind of sites-available/sites-enabled, etc.  I created /etc/nginx/sites and put in a basic entry for blog.lowend.party.

I created /var/log/nginx, which didn’t exist even though nginx was installed, and then /var/log/nginx/blog.lowend.party/{access,error}.log

How does nginx rotate logs?  newsyslog.  Look at /etc/newsyslog.conf.  It’s a table and it seems I’d add a line for /var/log/nginx/blog.lowend.party/access.log, etc. just like the OpenHTTPD entries.  But that would get tedious fast for many sites.  No wildcards mention on the man page.

I didn’t want to disturb /var/www, which is for OpenHTTPD, so I created /var/nginx/blog.lowend.party.  I symlinked /web to /var/nginx.

And then nothing worked.

I got errors like these:

2024/05/29 16:25:24 [error] 44074#0: *1 open() "/web/blog.lowend.party/index.html" 
failed (2: No such file or directory), client: x.x.x.x, server: blog.lowend.party, 
request: "GET /index.html HTTP/1.1", host: "blog.lowend.party"

OK, maybe it doesn’t like my symlink.

2024/05/29 16:38:08 [error] 48621#0: *4 "/var/nginx/blog.lowend.party/index.html" 
is not found (2: No such file or directory), client: x.x.x.x, server: 
blog.lowend.party, request: "GET / HTTP/1.1", host: "149.28.120.232"

I banged my head against the all for a while.  Changed www’s shell to /bin/bash, switched to it, tried to access that file, no problems, fixed www’s shell back and scratched my head.

So in what kind of situation is a file perfectly accessible, but a running daemon can’t get to it?

chroot.

Here’s OpenBSD’s nginx man page:

 -u   By default nginx will chroot(2) to the home directory of the user running the daemon, typically “www”, or to the home directory of user in nginx.conf.  The -u option disables this behaviour, and returns nginx to the original “unsecure” behaviour.

Oh really…

That’s Nginx 1.24.  On my Debian 12.5 system with Nginx 1.22 installed, there is no such option on the man page.  And on that box:

# nginx -u
nginx: invalid option: “u”

So either this was added in 1.24 (possible) or OpenBSD has patched it to include this.

Anyway, www is chrooted to /var/www (that’s its home directory).  It doesn’t seem right to turn off Nginx chroot, so I moved my web root to

Anyway, if you look in php-fpm’s config, you find it’s running on a socket at /var/www/run/php-fpm.sock which just happens to be inside that.

Anyway, I changed things up in my Nginx site definition.  I created /var/www/nginx/blog.lowend.party and set the site’s web root as

root   /nginx/blog.lowend.party;

I didn’t want to mess with htdocs (since that’s OpenHTTPD territory).  There’s also /var/www/html but since I haven’t read up on OpenHTTPD I figured it’s best to have nginx in its own place.

After that…

OK, so that’s just getting static HTML going.  Now let’s work on php-fpm.

php-fpm’s definition says it has a socket at /var/www/run/php-fpm.sock, but I had to setup Nginx so that the site looked at /run/php-fpm.sock.  Everything that happens before the daemon drops to an unprivileged user (www) can be referred to by its normal, absolute paths.  But once it’s “www” you have to use the chroot references.

OK, we’re up and running with php-fpm.

MariaDB

I ran mysql_secure_installation and got this:

bash-5.2# mysql_secure_installation 

NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB
      SERVERS IN PRODUCTION USE!  PLEASE READ EACH STEP CAREFULLY!

In order to log into MariaDB to secure it, we'll need the current
password for the root user. If you've just installed MariaDB, and
haven't set the root password yet, you should just press enter here.

Enter current password for root (enter for none): 
ERROR 2002 (HY000): Can't connect to local server through socket '/var/run/mysql/mysql.sock' (2)
Enter current password for root (enter for none): 

Huh. So then…

# rcctl start mysqld
mysqld(ok)
# ps ax | grep my
# ps ax | grep mar
# ls -l /var/run/mysql/
total 0

So -1 points to OpenBSD for rcctl.  If the daemon doesn’t start, I expect the tool I use to start daemons to tell me it didn’t start.

openbsd# cd /var/log
openbsd# grep -i mysql messages 
May 29 15:15:04 openbsd groupadd[86986]: new group added: name=_mysql, gid=502
May 29 15:15:04 openbsd useradd[19098]: new user added: name=_mysql, uid=502, gid=502, home=/nonexistent, shell=/sbin/nologin
May 29 15:42:14 openbsd pkg_add: Added php-mysqli-8.3.7
openbsd# grep -i mysql daemon
openbsd# 

Ooooookkkkaaaayyy…

This is the stock /etc/my.cnf

[client-server]
#socket=/var/run/mysql/mysql.sock
#port=3306

# This will be passed to all MariaDB clients
[client]
#password=my_password

# The MariaDB server
[mysqld]
# To listen to all network addresses, use "bind-address = *"
bind-address=localhost
# Directory where you want to put your data
#data=/var/mysql
# This is the prefix name to be used for all log, error and replication files
#log-basename=mysqld
# Logging
#general-log
#slow_query_log

I uncommented general-log. No change.

I notice there is no /var/lib/mysql nor is there a /var/mysql.  However, there is a mysql_install_db command.  Let’s try running that.

openbsd# mysql_install_db 
WARNING: The host 'openbsd.my.domain' could not be looked up with /usr/local/bin/resolveip.
This probably means that your libc libraries are not 100 % compatible
with this binary MariaDB version. The MariaDB daemon, mysqld, should work
normally with the exception that host name resolving will not work.
This means that you should use IP addresses instead of hostnames
when specifying MariaDB privileges !
Installing MariaDB/MySQL system tables in '/var/mysql' ...
OK


Two all-privilege accounts were created.
One is root@localhost, it has no password, but you need to
be system 'root' user to connect. Use, for example, sudo mysql
The second is _mysql@localhost, it has no password either, but
you need to be the system '_mysql' user to connect.
After connecting you can set the password, if you would need to be
able to connect as any of these users with a password and without sudo

See the MariaDB Knowledgebase at https://mariadb.com/kb

You can start the MariaDB daemon with:
/etc/rc.d/mysqld start

Please report any problems at https://mariadb.org/jira

The latest information about MariaDB is available at https://mariadb.org/.

Consider joining MariaDB's strong and vibrant community:
Get Involved

And after that it was running:

83277 p0  Sp       0:00.02 /bin/sh /usr/local/bin/mariadbd-safe
32870 p0  S        0:00.18 /usr/local/libexec/mariadbd --basedir=/usr/local --datadir=/var/mysql --pl

Really, shouldn’t that be run as part of the package install?  I expect sensible defaults and working software.

Now I can run mysql_secure_installation and create a database and user.  After unpacking WordPress’s latest.zip:

Certbot

Of course, we’re on http, and we want https.  I did install the ‘certbot’ package.  Let’s see if it works:

openbsd# certbot --authenticator webroot --installer nginx --webroot-path /var/www/nginx/blog.lowend.party
Saving debug log to /var/log/letsencrypt/letsencrypt.log
The requested nginx plugin does not appear to be installed

Hmmm.  On Debian there is a python3-certbot-nginx package.  None such for OpenBSD.

openbsd# man certbot
man: No entry for certbot in the manual.

Disappointing.  Let me go over to my Linux box and look at the certbot manual there.  There is a ‘certbot plugins’ command and sure enough, there is no Nginx plugin installed.

I think I install it through pip…?  All of the googling I did refers a lot to apt commands, LOL.

BTW, OpenBSD 7.5 comes with Python 3.10.

openbsd# pip3 install certbot-nginx
error: externally-managed-environment

× This environment is externally managed
╰─> This Python installation is managed by pkg_add(1).
    
    To install Python packages system-wide, use the OS packages where
    possible, for example: "pkg_add py3-somepackage".
    
    Otherwise, for software which is not available in packages,
    it is recommended to create a "venv" (virtual environment, see
    https://docs.python.org/3/library/venv.html) and install it there.
    For standalone applications, pipx (in the py3-pipx package) can
    help manage this for you.

Oh for pity’s sake.

OK, so…

openbsd# mkdir -p /usr/local/python3-venv/certbot-venv
python3 -m venv /usr/local/python3-venv/certbot-venv
. /usr/local/python3-venv/certbot-venv/bin/activate
pip3 install certbot-nginx

…which failed on

Building wheels for collected packages: cryptography

…because I didn’t have a Rust compiler installed.  Yes, I need a Rust compiler in order to use Let’s Encrypt apparently.

pkg_add rust

And then

Building wheel for cryptography (pyproject.toml) 

took forever.  The tension!  Alas, it concluded with a wall of Rust errors that ended with this:

      Caused by:
        process didn't exit successfully: `/tmp/pip-install-g1i5s24a/cryptography_6751bdb65bd643328c0d0dd2cb9623e6/src/rust/target/release/build/cryptography-cffi-69dd56dd49fae026/build-script-build` (exit status: 101)
        --- stdout
        cargo:rustc-check-cfg=cfg(python_implementation, values("CPython", "PyPy"))
        cargo:rerun-if-env-changed=PYO3_PYTHON
        cargo:rerun-if-changed=../../_cffi_src/
        cargo:rerun-if-changed=../../cryptography/__about__.py
        cargo:rustc-cfg=python_implementation="CPython"
      
        --- stderr
        thread 'main' panicked at cryptography-cffi/build.rs:63:49:
        unable to find openssl include path
        note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
      error: `cargo rustc --lib --message-format=json-render-diagnostics --manifest-path src/rust/Cargo.toml --release -v --features pyo3/extension-module --crate-type cdylib --` failed with code 101
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for cryptography
Failed to build cryptography
ERROR: Could not build wheels for cryptography, which is required to install pyproject.toml-based projects

I’m out of my depth.

One CPU?

While poking around, I looked in top and only saw one CPU.  I double-checked my Vultr panel and the VM was indeed provisioned with two CPUs.

So how do you tell how many CPUs are on your OpenBSD system?  There’s no /proc.  dmesg shows:

cpu0: Intel Xeon Processor (Skylake, IBRS), 2594.04 MHz, 06-55-04
cpu1: Intel Xeon Processor (Skylake, IBRS), 2593.97 MHz, 06-55-04

Perhaps the correct answer here is sysctl(1).  Running that dumps all the sysctl values, including:

hw.ncpufound=2
hw.allowpowerdown=1
hw.smt=0
hw.ncpuonline=1

Whaaaaaa….?!?  Only one CPU online?  That would be consistent with seeing them in dmesg but not in top, which looks like this:

So what happened to cpu1?  Let’s look at dmesg:

OpenBSD 7.5 (GENERIC.MP) #82: Wed Mar 20 15:48:40 MDT 2024
    deraadt@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 2130554880 (2031MB)
avail mem = 2045083648 (1950MB)
random: good seed from bootblocks
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0
acpi0 at bios0: ACPI 1.0
acpi0: sleep states S3 S4 S5
acpi0: tables DSDT FACP APIC HPET WAET
acpi0: wakeup devices
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel Xeon Processor (Skylake, IBRS), 2594.04 MHz, 06-55-04
cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,AVX512F,AVX512DQ,CLWB,AVX512CD,AVX512BW,AVX512VL,PKU,IBRS,IBPB,SSBD,ARAT,XSAVEOPT,MELTDOWN
cpu0: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 4MB 64b/line 16-way L2 cache, 16MB 64b/line 16-way L3 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 1000MHz
cpu1 at mainbus0: apid 1 (application processor)
cpu1: Intel Xeon Processor (Skylake, IBRS), 2593.97 MHz, 06-55-04
cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,AVX512F,AVX512DQ,CLWB,AVX512CD,AVX512BW,AVX512VL,PKU,IBRS,IBPB,SSBD,ARAT,XSAVEOPT,MELTDOWN
cpu1: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 4MB 64b/line 16-way L2 cache, 16MB 64b/line 16-way L3 cache
cpu1: smt 1, core 0, package 0
ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 11, 24 pins
acpihpet0 at acpi0: 100000000 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
"ACPI0006" at acpi0 not configured
acpipci0 at acpi0 PCI0
"PNP0A06" at acpi0 not configured
"PNP0A06" at acpi0 not configured
"PNP0A06" at acpi0 not configured
"QEMU0002" at acpi0 not configured
acpicmos0 at acpi0
"ACPI0010" at acpi0 not configured
acpicpu0 at acpi0: C1(@1 halt!)
acpicpu1 at acpi0: C1(@1 halt!)
cpu0: using Skylake AVX MDS workaround
pvbus0 at mainbus0: KVM, Hyper-V 10.0
pvclock0 at pvbus0
hyperv0 at pvbus0hyperv0: posting vmbus message failed with 18
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 82441FX" rev 0x02
pcib0 at pci0 dev 1 function 0 "Intel 82371SB ISA" rev 0x00
pciide0 at pci0 dev 1 function 1 "Intel 82371SB IDE" rev 0x00: DMA, channel 0 wired to compatibility, channel 1 wired to compatibility
pciide0: channel 0 disabled (no drives)
atapiscsi0 at pciide0 channel 1 drive 0
scsibus1 at atapiscsi0: 2 targets
cd0 at scsibus1 targ 0 lun 0: <QEMU, QEMU DVD-ROM, 2.5+> removable
cd0(pciide0:1:0): using PIO mode 4, DMA mode 2
uhci0 at pci0 dev 1 function 2 "Intel 82371SB USB" rev 0x01: apic 0 int 11
piixpm0 at pci0 dev 1 function 3 "Intel 82371AB Power" rev 0x03: apic 0 int 9
iic0 at piixpm0
vga1 at pci0 dev 2 function 0 "Bochs VGA" rev 0x02
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
virtio0 at pci0 dev 3 function 0 "Qumranet Virtio Network" rev 0x00
vio0 at virtio0: address 56:00:04:f0:55:23
virtio0: msix per-VQ
azalia0 at pci0 dev 4 function 0 "Intel 82801I HD Audio" rev 0x03: apic 0 int 11
azalia0: No codecs found
virtio1 at pci0 dev 5 function 0 "Qumranet Virtio Storage" rev 0x00
vioblk0 at virtio1
scsibus2 at vioblk0: 1 targets
sd0 at scsibus2 targ 0 lun 0: <VirtIO, Block Device, >
sd0: 66560MB, 512 bytes/sector, 136314880 sectors
virtio1: msix per-VQ
virtio2 at pci0 dev 6 function 0 "Qumranet Virtio Memory Balloon" rev 0x00
viomb0 at virtio2
virtio2: apic 0 int 10
virtio3 at pci0 dev 7 function 0 "Qumranet Virtio RNG" rev 0x00
viornd0 at virtio3
virtio3: msix per-VQ
"Intel 6300ESB WDT" rev 0x00 at pci0 dev 8 function 0 not configured
isa0 at pcib0
isadma0 at isa0
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pms0 at pckbc0 (aux slot)
wsmouse0 at pms0 mux 0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
usb0 at uhci0: USB revision 1.0
uhub0 at usb0 configuration 1 interface 0 "Intel UHCI root hub" rev 1.00/1.00 addr 1
uhidev0 at uhub0 port 1 configuration 1 interface 0 "QEMU QEMU USB Tablet" rev 2.00/0.00 addr 2
uhidev0: iclass 3/0
ums0 at uhidev0: 3 buttons, Z dir
wsmouse1 at ums0 mux 0
vscsi0 at root
scsibus3 at vscsi0: 256 targets
softraid0 at root
scsibus4 at softraid0: 256 targets
root on sd0a (5f3c9329743bf957.a) swap on sd0b dump on sd0b
WARNING: /mnt was not properly unmounted
fd0 at fdc0 drive 1: density unknown

No real message why cpu1 is offline.

I fired up another Vultr VM with identical specs using their 7.5 template.  Top shows:

Remember when I said that OpenBSD on a VM always seems to require some special parameter…

No, actually what’s happening is that Vultr is presenting a 2 cores but OpenBSD detects that it’s hyperthreaded, and since OpenBSD believes hyperthreading presents unacceptable security risks, it’s disabled.  To enable both cpus:

openbsd# sysctl hw.smt=1
hw.smt: 0 -> 1

and then:

Sure enough, the Vultr template puts this in /etc/sysctl.conf

However, I think dmesg could be a little more explicit when it disables a CPU.

Ports

I was curious about that Nginx chroot thing, so I loaded ports.  The directions are copy-paste in the FAQ.  Once untarred, ports has tons of software you can install, and the binary packages are built from this.  I looked in www/nginx/patches and saw patch-man_nginx_8, which describes the -u flag.

I then did a “make” in /usr/ports/www/nginx and saw various “apply OpenBSD patch” messages go by.

Before We Go: pf

OpenBSD famously once used IPFilter but after licensing terms were changed, they created a replacement from scratch in one glorious hackathon.  pf is used not only by OpenBSD, but also by all the other BSDs (FreeBSD, DragonFlyBSD, NetBSD), macOS, iOS, and others.

If you’ve done one firewall, you’ve done them all in my opinion but I’m sure if I worked on networks and firewalls all day long I’d see the differences.  pf has a good reputation.

The default pf.conf and also /etc/examples/pf.conf both show ways how to setup your firewall rules.  Here’s is another:

# Set macros for port numbers.  pf uses the term 'macros'
SSH_PORT = 22
HTTP_PORT = 80

# Define interfaces
ext_if = "vio0"

# Set default policy
set block-policy drop

# Skip loopback interface
set skip on lo0

# Allow inbound SSH and HTTP traffic
pass in on $ext_if proto tcp from any to any port $SSH_PORT
pass in on $ext_if proto tcp from any to any port $HTTP_PORT

# Block all other inbound traffic
block in on $ext_if

Conclusion

I learned a lot in this, mainly because I’ve never used OpenBSD as a web server before.  I’m not sure it’s the platform’s strengths, though I’m sure with some research I could have got certbot working (manually if necessary).  BTW, there’s no cron job put in place for certbot renewals.

The experience with Certbot above is not atypical of my experiences with OpenBSD over the years.  You do learn a lot, but what is easy on Linux often requires a lot of manual work.  In this case, instead of a single pkg_add, you install pip, build a venv, install modules, install rust, and it still fails.  Now, true, maybe the real answer here is that I should build a certbot plugins package for OpenBSD on Nginx.  Or use OpenHTTPD.

OpenBSD’s assumption is that you are a very good sysadmin.  When I started back on SunOS, building your own kernel, downloading and compiling software from scratch, etc. was considered sysadmin work.  There was no package manager per se.  OpenBSD to me harkens back to that era, but in between there’s been an explosion of complexity in software.

In its core domain, OpenBSD works very well.  And of course there are developers who’ve got OpenBSD laptops with X-windows and people can do nearly everything with it.  But I think for me it’s going to remain as my firewall/edge/jump server/bastion host go-to, not as a daily app server.