Technology Stir Fry

Signing Mail, 2024 Edition

2024-05-28T14:19:55Z

I have been using S/MIME to sign e-mail for well over a decade now. Because certificates don’t last for ever, I need to renew mine every couple of years. This gives me an opportunity to experience the current state of that market and of client software.

This has seldom been a pleasant experience, and it has sometimes been downright awful (see 2019, 2022). This iteration wasn’t great, but there are some small signs of improvement.

I should make clear a couple of important qualifications which may mean that my experience isn’t applicable to your situation:

I’m looking at this from the point of view of an individual without access to some sort of corporate-level certificate provision.
I no longer even consider the PGP/GPG ecosystem for mail authentication, although I do still use it for artifact signing. There are enough systems in between myself and recipients these days that it is next to impossible to get even clear-signed text blocks through the system sufficiently unscathed for the signature to be verifiable. Broken signatures are not useful.

I mentioned last time that the process then depended on the long-standing capability of certain browsers to create an RSA keypair and use that in requesting an S/MIME certificate from a certification authority through the CA’s web site. This approach caused a large share of the problems in previous years because collecting the certificate, linking it to the private key in the browser and then exporting it to other clients was just full of flakiness, bugs and bad user interface design.

The browsers that had that ability are, I think, now fully extinct. The good news is that this means that all those problems went away. The bad news is that the alteratives aren’t great:

The option that I suspect most people will use is to allow the CA’s web site to create your RSA keypair and let you download it from them. I find it surprising that this is offered without any mention of the fact that the CA therefore knows your private key. In the world of TLS certificates, the key being at any time out of your sole control would be grounds to revoke your certificate immediately. Somehow, that rule apparently doesn’t apply here. Needless to say, I don’t feel that I can take this route with a straight face.
The second option, by analogy with the TLS certificate market, is for the customer to create their own keypair, construct an appropriate Certificate Signing Request (CSR) and submit that for signing. This obviously requires more technical knowledge and is probably something most individuals looking for S/MIME certificates won’t feel capable of doing. It is nevertheless the route I took.

Unfortunately, at least on the site of the CA I was working with, there seemed to be no documentation at all of what an appropriate CSR should look like. In the end, this is what I did:

touch smime.key
chmod 600 smime.key
openssl genrsa 3072 >>smime.key
openssl req -new -config smime.cnf -key smime.key -out smime.csr

The critical part of this is smime.cnf:

[req]
default_bits = 3072
distinguished_name = dn
prompt = no

[dn]
C=GB
ST=Scotland
L=Edinburgh
O=Ian A. Young
CN=ian@iay.org.uk
emailAddress=ian@iay.org.uk

As it turned out, the certificate product I was purchasing discarded all of the components of the distinguished name other than the emailAddress, so that could probably have been simplified.

Once I uploaded the CSR to the certification authority’s web site, I was able to download the new certificate in one of two ways:

I could get a PKCS#12 file (sometimes called a PFX file, particularly in Microsoft environments). This would be ideal for configuring client software, but there’s a catch: this requires you to upload your private key. Yes, again, there’s an option that asks you to lose control of your private key without mentioning that in many situations this would be an unacceptable security risk. As with the “easy” option for key generation, I didn’t feel that I could do this.
The second option is to have a PKCS#7 file generated, and that’s of course what I selected.

A PKCS#7 file is just a bag of certificates (i.e., public data). The end entity certificate (for ian@iay.org.uk in this case) is usually first, followed by the other certificates required to build a verification chain up to a trust root. This will be a familiar concept to anyone who has worked with X.509 certificates for transport layer security, as will the inevitable format conversion dance that followed on from downloading the file.

I had expected the download to be either:

A PEM-encoded PKCS#7 bundle with the usual textual BEGIN and END lines, or
A DER-encoded (binary) PKCS#7 bundle.

What I actually got was a text file containing a single very long line. You could think of this as the PEM variant with the header and trailer lines stripped off, or as a Base64-encoded form of the DER variant. I chose the latter viewpoint and converted it to something useful like this:

base64 --decode <smime.p7b >certs.der
openssl pkcs7 -in certs.der -inform der -print_certs -out certs.pem

Now I had a series of certificates in the textual PEM format in certs.pem, but to conifgure my mail clients I needed to further convert that to a PKCS#12 file so that I could incorporate the private (still private) key:

openssl pkcs12 -export -in certs.pem -inkey smime.key \
    -out smime.p12 -chain -untrusted certs.pem -legacy

The -legacy option changes the default encryption algorithm and was necessary to make the resulting PKCS#12 file acceptable to the Apple mail clients I use.

Once you have a PKCS#12 file, of course your troubles are over! I jest, of course: in practice, mail clients hide the configuration of signing certificates in the weirdest places. For Apple’s Mail under macOS, for example, you register the certificate in the separate Keychain Access utility and then hope that Mail notices when you enable signing in the main application.

One place things have improved a little since my previous iteration is in Apple’s mobile operating systems. Under both iOS and iPadOS, installing a new signing certificate was a bit of a pain: now, if you can locate your .p12 file in the Files application, you can just tap it and get an instruction to install the new “profile” in Settings.

In conclusion: on one hand, the bad news is that signing your e-mail is still quite problematic, sufficiently so that I wouldn’t expect most people to bother even if they knew it was possible. On the other hand, some of the old cruft has been stripped away in the last couple of years, and someone who is capable of understanding TLS certificates should be able to apply that knowledge to S/MIME certificates directly, and that’s good.

On the gripping hand, if you go through this process you get more than one opportunity to commit the cardinal sin of compromising your private key material by handing it over to a supplier. This is bad.

Docker and LXC

2024-02-09T20:51:49Z

My long-standing virtualisation platform of choice has been VMware’s; for many years I’ve had a couple of machines running ESXi hosting my virtual machines and I use their Fusion product on my Macs. With the acquisition of that business by Broadcom and the ongoing “business transformation”, though, I’ve moved the server side of things over to Proxmox Virtual Environment.

Proxmox is a more open virtualisation product built around Debian Linux. It is a little less featureful than the VMware suite in some ways, but it has what I need in my very not-Enterprise use case, and I’m less concerned that a new “business transformation” might suddenly compromise my access to it.

As well as running virtual machines, Proxmox has a very usable interface to Linux containers (LXC), a technology which I’d been aware of from my use of Docker but which I’ve not previously had the opportunity to look into in its own right. Having a new tool for the toolbox is always a joy, though, so I’ve been digging into this other form of virtualisation recently.

Linux containers are built on the same Linux kernel technologies that underlie Docker, Podman and Kubernetes: cgroups, namespaces and an isolated filesystem that (usually) looks a lot like a Linux installation. There are differences, though:

In an “application” container (Docker, etc.), setting up networking is the job of the container engine, not the container. In a “system” container (LXC) the container sees a raw (virtual) network interface and is responsible for its setup, in the same way as a virtual machine would be.
In an application container, things like environmental variables are passed in to the container’s entrypoint, which is often the only thing that runs. In a system container, there are fewer assumptions and you can do whatever you like but very little is done for you.

Simple container

As an example, let’s look inside a simple application container:

% docker run --rm -it alpine:latest
/ # ps gax
PID   USER     TIME  COMMAND
    1 root      0:00 /bin/sh
    7 root      0:00 ps gax
/ # ip a
...
21: eth0@if22: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 65535 ...
    link/ether 02:42:ac:11:00:03 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.3/16 brd 172.17.255.255 scope global eth0
/ # df -h /
Filesystem                Size      Used Available Use% Mounted on
overlay                  58.4G     22.4G     32.9G  41% /

Here, the process with PID 1 is the container’s entrypoint, the shell. An Ethernet interface is ready for use, and I have 33GB of space on my root drive. Those are all lies, of course, but they are the kind of lies a containerised application needs to hear.

What about LXC? A container is just a bunch of files invoked in some kind of context, so maybe we can just export that bunch of files and fire it up in Proxmox as a system container? Turns out that is exactly true:

% docker container create --name example1 alpine:latest
% docker container export --output example1.tar example1
% docker container rm example1
% gzip example1.tar
% scp example1.tar.gz root@pve02:/var/lib/vz/template/cache/

pve02 is one of my Proxmox nodes; the /var/lib/vz/template/cache/ directory is a location where Proxmox stores templates from which you can create containers using its graphical user interface. Creating such a container from the template that used to be a Docker image and firing it up gives us a login prompt, after which we can have a poke around:

example1:~# ps gax
PID   USER     TIME  COMMAND
    1 root      0:00 /sbin/init
   25 root      0:00 /bin/login -- root
   26 root      0:00 /sbin/getty 38400 tty2
   27 root      0:00 -ash
   28 root      0:00 ps gax
example1:~# ip a
...
2: eth0@if482: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 ...
    link/ether bc:24:11:fa:87:94 brd ff:ff:ff:ff:ff:ff
example1:~# df -h /
Filesystem                Size      Used Available Use% Mounted on
/dev/mapper/pve-vm--106--disk--0
                          7.8G      7.7M      7.4G   0% /

The lies are different this time. PID 1 is now an explicit init process created by the environment, and that has spun up the login process which leads to our shell. There’s a network interface, but it’s unconfigured. The root filesystem is a mounted virtual disk now, and a tiny 8MB of that is in use by our Alpine Linux userspace.

At this point, because this is Alpine Linux and even the tiniest container bundles BusyBox, you can create an appropriate /etc/network/interfaces, type ifup eth0 and be off to the races. Things will be a little trickier with images based on other distributions, but the principle still applies.

Complex container

In the above, I glossed over the fact that although I got a shell out of both container technologies, the mechanism was entirely different. That’s more obvious if the Docker container is supposed to be doing something other than running a shell, which will almost always be the case. The technique above will always give you a shell (if it works at all); the original image’s ENTRYPOINT, CMD, ENV and so on are all ignored.

To see what can be done in real use cases, let’s move on to my motivating example for looking into this at all. netboot.xyz is a neat thing to have hosted in your network if you’re trying different Linux distributions or just installing a lot of virtual machines. You can deploy it as a Docker container but it’s better for it to have its own independent IP address so that you can put that into your DHCP server’s configuration. Deploying as an LXC container is ideal, but that’s not something they provide.

Instead, we can take the Docker image, add some missing components to turn that into a fully functional LXC template using docker buildx, then deploy a container from that template. Here’s the Dockerfile:

FROM netbootxyz/netbootxyz:latest

RUN apk add --no-cache openrc ifupdown-ng

ADD netbootxyz /etc/init.d/
RUN rc-update add netbootxyz default

This Dockerfile installs two packages: openrc is Alpine’s init system; ifupdown-ng is a more advanced network interface package than that provided by BusyBox: it’s there to provide compatibility with Proxmox’s LXC network configuration UI.

We’re also adding a netbootxyz service to the system’s default run level. The configuration file (which needs to be executable) looks like this:

#!/sbin/openrc-run

command="/start.sh"
command_background="true"
pidfile="/run/netbootxyz.pid"

export TFTPD_OPTS=
export NGINX_PORT=80
export WEB_APP_PORT=3000

depend() {
        need net
}

The export statements provide the environmental variables required by the container image’s /start.sh entrypoint, which we’re defining here as a service dependent on the availability of networking. Because netbootxyz is part of the default run level, its dependency will be brought up before it whenever the system starts.

Building the new template is simple:

% docker buildx build --output type=tar,dest=example2.tar .
% gzip example2.tar
% scp example2.tar.gz root@pve02:/var/lib/vz/template/cache/

Deploying this template from the Proxmox UI and providing a static IP address results in the following:

example2:~# ps gax
PID   USER     TIME  COMMAND
    1 root      0:00 /sbin/init
  348 root      0:00 {start.sh} /bin/bash /start.sh
  357 root      0:00 /sbin/getty 38400 tty2
  384 root      0:00 {supervisord} /usr/bin/python3 /usr/bin/supervisord ...
  386 root      0:00 /usr/sbin/syslog-ng --foreground --no-caps
  387 root      0:00 nginx: master process /usr/sbin/nginx ...
  388 nbxyz     0:00 /usr/bin/node app.js
  389 root      0:00 /usr/sbin/in.tftpd -Lvvv --user nbxyz --secure /config/menus
  391 nbxyz     0:00 nginx: worker process
  392 nbxyz     0:00 nginx: worker process
  393 nbxyz     0:00 nginx: worker process
  394 nbxyz     0:00 nginx: worker process
  406 root      0:00 tail -f /var/log/messages
  412 root      0:00 /bin/login -- root
  413 root      0:00 -ash
  414 root      0:00 ps gax
example2:~# ip a
...
2: eth0@if486: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 ...
    link/ether bc:24:11:82:48:d1 brd ff:ff:ff:ff:ff:ff
    inet 192.168.117.1/24 scope global eth0
       valid_lft forever preferred_lft forever
...
example2:~# df -h /
Filesystem                Size      Used Available Use% Mounted on
/dev/mapper/pve-vm--106--disk--0
                          7.8G    182.3M      7.2G   2% /

You can see from this that the network interface has been configured with the supplied static address, and that the full netboot.xyz stack is running:

PID 348 is the service startup script, running as a daemon.
That has invoked a supervisord process (PID 384) to start the component services:
- syslog-ng
- tftpd
- a Node.js application for the user interface
- nginx and its worker processes to serve asset requests

Conclusions

Containers are variations on a bunch of files in an isolated runtime context. It’s possible — and even fairly straightforward — to convert a container image intended for one runtime for use in another by tweaking the set of files in use; they’re just files. docker buildx is, pretty much by definition, one way of performing manipulations of this kind.

Proxmox is a pretty nice platform to deploy both virtual machines and system containers (application containers too, with the proviso that the recommendation from Proxmox is to run your application container runtime inside a virtual machine). The resulting containers are tiny, start very quickly and are easy to move around a Proxmox cluster if needed, without dragging a virtual machine around with them.

Key Server Shuffle

2023-10-27T13:16:44Z

Every so often, I find myself interacting with the PGP/GPG ecosystem. This usually results in me bemoaning the latest step in its slow drift towards complete irrelevance.

Imagine my surprise when this week’s addition of a new signing subkey on my main key ended up finding that although the key server I had been using (pgp.re) had become defunct, quite a few new key servers seem to have sprung up, and there’s actually a reasonable number again.

I’ve pushed my most recent key update to the following:

keys.openpgp.org (a privacy-centric island)
pgp.mit.edu (“old faithful”)
keyserver.ubuntu.com
sks.pgpkeys.eu

These servers all seem to be fairly consistently available and moreover are again (mostly) participating in a functional distribution network. I’m not sure how the previous GDPR-related issues have been resolved, but it’s nice to have this back.

Unchromed

2023-09-21T11:54:12Z

It has been a long time since “Don’t be evil” was part of Google’s corporate philosophy. We could probably argue about whether they ever really believed in that idea, how long their belief lasted, and what about the environment they exist in was so corrosive to it. That discussion would probably include a lot of talking points along the lines of “well, actually, corporations are required by law to be evil in order to maximise shareholder value” and — at least in recent years — a lot of audible scoffing on my part.

I recently reached the point at which I wasn’t prepared to keep having that argument even with myself. I got pretty close a couple of years ago when FLoC came along (see No FLoC Here). The latest round in the exhausting saga of Google forcing surveillance advertising onto end users is the so-called Privacy Sandbox, in which Google’s preferred model for targeted web advertising is baked into the world’s dominant browser, which they not coincidentally control.

I recommend the Ars Technica article if you want depth, but here’s the lede:

Don’t let Chrome’s big redesign distract you from the fact that Chrome’s invasive new ad platform, ridiculously branded the “Privacy Sandbox,” is also getting a widespread rollout in Chrome today. If you haven’t been following this, this feature will track the web pages you visit and generate a list of advertising topics that it will share with web pages whenever they ask, and it’s built directly into the Chrome browser.

No end user asked for this. No end user benefits from this. The selected user interface (and messaging: “Privacy Sandbox”, my eye) mean that people who use Chrome will most often end up with this enabled by default and never find out how to turn it off. This benefits Google’s customers, who are not me and, probably, not you.

As a sophisticated technical user I could of course disable this and be on my way. And check it hasn’t been re-enabled every few days “by accident”. And make sure there’s no new wizard wheeze introduced that needs to be defended against. But, as I said above, this adversarial approach to a critical tool is just exhausting and at this point I’m done with it.

So, for what it’s worth, Chrome is now gone from all of my systems and won’t come back as long as there are less-abusive alternatives.

I’ve been primarily a Safari user for some years now, and although it has some flaws I’ve been pretty happy with it from a privacy perspective. Apple may not be as perfect as they would like everyone to think, but they’re not bad. Well, as long as you set DuckDuckGo as your default search engine anyway.

You can’t live with just Safari, though, which is of course the reason I was regularly using Chrome in the first place. There are still a wide variety of web sites which just don’t work perfectly on the world’s second most dominant browser. For that second, utility, browser I’ve returned after a long hiatus to good old Firefox. I don’t love it, but it works with most things that balk at Safari and it gets extra points for a nice logo.

Porcelain

2023-05-12T22:57:25Z

It was twenty years ago today, Sgt. Pepper told the band to play…

No. Wait. That’s not right; let me start again.

My very first post to this “weblog” (as the term still was then) was exactly twenty years ago. The colophon has some of the story of how the implementation has changed over the years, but today I thought it would be more interesting to look and see what if anything I have learned from two decades writing here. There are not many things in your life you end up devoting effort to over that span of time, after all.

Some statistics, just for fun: there are 260 posts here excluding this one but including the iay@there side-blog I ran for a few months in 2004¹. That’s an average of a bit over one post a month, but you can see from the Archives page that my output has been rather uneven: I’m not at all sure what happened to silence me in 2015–2016, as most people would claim it’s usually pretty difficult to get me to shut up.

Looking at the posts on the main blog by tag, I wasn’t surprised to find that the biggest cluster of subjects has been around digital identity, cryptography, and security more generally. That’s been my principal line of work for most of this time, after all.

If you include both blogs, though, “virtual reality” actually wins out. That might be a little surprising unless you’re aware of my brief career as a professional construction worker in the Second Life virtuality.

Over the years several communities have, I believe, found this blog useful. The most surprising to me, certainly at the time, was the result of the spectacular success of a pair of articles I wrote in 2003 about my experiments with 8-bit microcontrollers. They were so popular — presumably because they were just in the right place at the right time — that the advertisements I ran on PIC16C745/765: microcontrollers with USB and My Very First USB Peripheral were enough to pay my hosting fees for that year, which was nice. They are are the only pages I’ve ever run advertisements on, and I no longer serve up anything I’d need to host a cookie banner for, not even analytics.

In the early days, before the scourge of spam inevitably spread to the world of blogs, this one had both trackback and comment functionality. Equally inevitably, I gave up what would otherwise have become a full-time moderation job here so that I could devote time to my paid work. There are some crackers in there if you root around; they’re still visible because I imported everything from the database-backed versions of the blog when I Nanoced it into a static site. My favourite, I think, is the one where a commenter asks if he can buy my cuddly headcrab. I was not able to oblige him, alas.

So, twenty years. Has it been worth it? Yes, absolutely. I enjoy writing long form, and working with words in general. I do get to write from time to time for my paying work, and that’s enjoyable as an exercise of vital powers but it’s not really the same as self-directed writing.

So, writing has been worth it, but what about writing here, at iay.org.uk? Has the palaver of hosting my own site, learning Movable Type, Drupal, Nanoc, all that stuff… has that paid off? One of my answers should be obvious if you’ve glanced at the site title: as well as sometimes being a pain, digging into the technology has also been fun. So again, yes.

I’ll finish with a second and more serious answer to the same question, which I will shamelessly steal from John Scalzi (who has been blogging at Whatever for twenty-four years as I write this):

I’ve had Scalzi.com since March of 1998, which has been enough time for at least four generations of online social networking sites. They come and go; my site remains. And it will remain when the hip kids roll their eyes at whatever pathetic dinosaurs still remain on Facebook (hint: that’s already happening). Online, that’s as good as permanence gets.

Scalzi wrote this in 2011: long before the recent unpleasantness on Twitter, back in the early days when Twitter was becoming the New Hotness people were talking about. We may not be at the sad end of that particular mayfly’s lifecycle just yet, but who would start using it right now?

None of us are here for the long term, but I think it’s wise to treat all online services — other people’s computers — as entirely ephemeral. Whether a service survives for more than a handful of years is out of my control, and the last year has certainly demonstrated that what motivates the owners of such services does not necessarily align with what would be best for me.

Fortunately, many of the things I love about social media (the wellspring of ideas, the active conversation and, honestly, the cat pictures) don’t require persistence. They require active interest; they require being the thing that people think is the New Hotness. Until we all move on to the new place we’re all hearing about.

Kids today wouldn’t believe it, but back in my day, people in the metaverse had legs. ↩

infosec.exchange

2023-05-09T15:54:37Z

I started looking at Mastodon a bit more than a year ago, when the prospect of New Management at Twitter was becoming a looming disturbance in the timeline and although no-one really knew exactly what to expect, “something good” was clearly unlikely¹. As described in Tweets and Toots, I set up at mastodon.social because I really didn’t have a good answer to the question of which community I felt part of.

In December 2022, I posted Why I Unfollowed You to mark my progressive disengagement from Muskland, and really haven’t spent a lot of time there since. A couple of weeks later I pinned what I regard as my tombstone post and which still represents my position:

This place is best shunned and left uninhabited.

I’ve enjoyed the time I have spent on Mastodon a lot more than my latter-day experience with the Hellsite. It’s less exhausting and stressful, and even the cat pictures are better, to my mind. Or are those statements redundant?

I’m not finding myself posting that much more on Mastodon — most of my posts are essentially pointers to long-form posts here — but I do find myself boosting a lot more than I retweeted. Boosts don’t incorporate commentary in the way that quote-tweets can, so the semantics are simply and only “I think people who follow me might find this interesting” and I don’t find it particularly scary to hit a button with that label.

It turns out that the question of whether Mastodon should also provide an easy way to do the equivalent of a quote-tweet is a contentious one. Not having that in the experience was a conscious choice and while I’m not certain that the sky would fall if it was introduced, it’s a cultural thing now and I don’t think there’s a pressing need for in-line argumentation.

The one change I am making today is to move from my original instance, the flagship mastodon.social, to the smaller infosec.exchange. So, I am now @iay@infosec.exchange.

The new instance is quite a lot smaller: as I write this, mastodon.social has 878K users, of which around 220K are in some sense “active” (there are enough users on the instance that it’s no longer open access). infosec.exchange has closer to 30K users in total. This makes a big difference to the “local” timeline: it’s an unusable firehose on mastodon.social and while there’s probably a way to filter it down to a reasonable level the smaller instance just feels… friendlier, somehow. The denizens of infosec.exchange don’t feel like my family, necessarily, and they’re not even all in the same line of work as I am (my Mastodon bio describes me as “Infosec-adjacent”), but they’re definitely my tribe. The things they are talking about are the things I talk about.

As well as feeling more at home, and having a manageable local timeline, I’m also sympathetic to the idea that large instances (and implicitly a small number of large instances) is just a bad direction. Obviously that feeling is informed by the limiting case of conventional monolithic social media: the incentives for site owners are just too misaligned with those of the would-be socialites and enshittification is, if not actually inevitable, at least hard to avoid.

The move itself was very smooth, perhaps because I waited for a day when neither instance was being DDOSed by people who don’t want us to have nice things. If you’re thinking of doing this for any reason, my only word of advice would be to change your original profile to be exactly the way you want it before flipping the “send my followers over there” switch, as there are some things you can’t do with an active redirect in place.

“called it” ↩

New Rule

2022-12-29T16:09:10Z

Alongside “don’t drink coffee too late at night” I have a new rule to make sure I get a good night’s sleep: “never fire up Wireshark after 10pm”.

I’m moving some network functionality — specifically, local DNS servers, I run split-horizon DNS — from two old Raspberry Pi machines onto something more modern. The question arose of how I’d know when I was done: how would I know when there are no more requests being made to the old machines?

I had a “bright idea” and made the mistake of firing up a Wireshark network analyser to quickly answer this question. I got more than I bargained for, as there were a lot of DNS queries I couldn’t see a reason for. The most common one looked like this:

Some host on the network is performing a query for www.microsoft.com every 15 seconds, regular as clockwork.

Well, that’s on one server. I have two: this is what the other one was seeing:

That’s another two sources of these queries, each also ticking away every 15 seconds.

These three hosts turn out to be the nodes in my WiFi mesh network. The manufacturer, Tenda, have arranged for each node to acquire an IP address on the network and use that to query for the address of www.microsoft.com every 15 seconds. That’s 17,280 queries a day.

Why? I found this post suggesting that the idea here is to establish whether there’s internet connectivity. Why each and every node in a WiFi mesh needs to know more than which RJ45 to send the packets out of is a mystery to me. It’s also amusing to think that as I have local servers all these queries are being swallowed by them and aren’t establishing second-by-second connectivity at all; it’s all just wasted effort.

Never fire up Wireshark after 10pm.

Docker-in-Docker and host resolution errors

2022-12-07T10:37:32Z

This article documents a particularly niche issue I ran into which took a while to debug and resolve. I’ve posted it for the benefit of search engines, so that the next person to run into these exact conditions might save a few hours. You’re welcome.

This is probably interesting only to someone in my exact situation, so most people should find some cat pictures to look at instead. If you’ve been directed here by a search engine because you’re tearing your hair out, though, read on.

The very niche situation I alluded to breaks down to this:

A private GitLab instance, running a continuous integration job.
The job is docker building something based on a recent Linux base image (in my case, Rocky Linux 9).
The runner is executing on a slightly older Linux implementation (in my case, Ubuntu 20.04 LTS).
We’re using Docker-in-Docker mode to avoid giving the job access to the host’s Docker socket.

The failure looks like this:

---> Running in faab08d66352
+ dnf -y install httpd mod_ssl
Rocky Linux 9 - BaseOS                          0.0  B/s |   0  B     00:00    
Errors during downloading metadata for repository 'baseos':
  - Curl error (6): Couldn't resolve host name for https://mirrors.rockylinux.org/mirrorlist?arch=x86_64&repo=BaseOS-9 [getaddrinfo() thread failed to start]

You might think that the most likely cause of an issue like this would be DNS. After all, it’s always DNS, right? Not in this case (and it wasn’t IPv4 vs. IPv6 either, which was my other initial thought). Search engines brought up suggestions to restart my router. I rolled my eyes.

Eventually I happened across ubuntu:21.10 and fedora:35 do not work on the latest Docker (20.10.9) by Akihiro Suda, one of the maintainers of the various components used by Docker. He explains the root of the problem, which is a change in user-mode behaviour in the glibc shipped with newer Linux distributions (Fedora 35, Red Hat Enterprise Linux 9 and derivates like Rocky Linux 9, Ubuntu 22.04). The way older versions of Docker (20.10.9 and earlier) handle the new clone3 system call means that it’s treated as a permissions violation rather than a “not implemented” call which would have resulted in fallback behaviour appropriate to the host operating system.

As an aside, I think this is only the second time I’ve hit any kind of kernel mismatch problem while using Docker. That’s an impressive compatibility record given that I’ve been using it for eight years now.

Akihiro’s recommendation:

The right solution is to upgrade Docker to 20.10.10 or later.

This didn’t help me because all my Docker nodes were already running 20.10.21. However, I saw that the problem was only showing up on those nodes running Ubuntu 20.04 LTS: the node running Ubuntu 22.04 LTS wasn’t seeing the problem, so I felt I was probably on the right track.

The wrinkle in my case turned out to be that my GitLab continuous integration pipeline included the following:

image: docker:20.10.6
services:
  - docker:20.10.6-dind

This is configuring a “helper service” containing its own copy of the Docker daemon for use in the build. This “Docker in Docker” arrangement allows conventional docker build operations to be performed within the CI job without granting the build container access to the host’s Docker daemon. (The build does still need to be run “privileged”, which is still slightly scary, but those are the tools available.)

I deduce that the same interpretation of errant system calls is performed within the dind container as needs to be performed by “normal” Docker. This is a little surprising when you think about all the turtles being stacked on top of each other here, but also makes sense: how else could it work?

Updating the dind helper service to a more recent version of course addresses the original issue. One option is to use docker:20.10-dind to get maintenance updates without the exciting functional changes that might greet a user of the latest tag.

Why I Unfollowed You

2022-12-01T23:20:18Z

So, I used to follow you on Twitter, but now I don’t. What’s up with that?

The first thing to say is: it’s not personal. Unless it is, of course, personal: but in that case you already know what you did. You monster.

It’s obviously much more likely — given your upstanding character and sunny disposition — that you have fallen foul of my progressive campaign to free myself from dependence on The Bad Place. Did I say that out loud? I meant to say Twitter.

I like doomscrolling as much as the next fellow, of course. But since April when I last discussed matters of so-called “social” media, things on The Hellsite have gone about as badly as they could have, consequent to its purchase by the world’s richest tunneling equipment and flamethrower magnate. The ratio of outrage to useful information is going in the wrong direction fast, and the quality of doomscrolling available has plummeted as a result. It’s just exhausting watching all those strangers argue past each other.

My best guess at this stage is that the Bluebird of Unhappiness is doomed to circle the black hole of its new owner’s ego until it is either entirely crushed by the tidal forces, or he gets bored and wanders off to assist some other large corporation turn back into a struggling start-up. I think we have a responsibility to future civilisations to signpost the resulting floating rubble as a hazard to navigation; the message should be that this place is not a place of honor… this place is best shunned and left uninhabited. Social media implemented in that fashion, with those incentives, was not humanity’s best idea.

Sorry, where was I? Oh yes, why I unfollowed you. Well my hope is that the reason for that was that I decided to follow you somewhere else instead! For example, if you have a Mastodon or other Fediverse account, you’ll be picking up followers there as you lose them elsewhere. If you followed me on The Twitters, I invite you to do the same: unfollow me now, with gusto, and make my acquaintance anew at @iay@mastodon.social. I might actually post a bit more now that the environment I’m posting into is not actively horrible.

As well as relocating my primary doomscrolling location, I’ve started getting back into that retro technology we all used to use before microblogging was a thing. That means RSS and such, so if you have a real old-fashioned steam-powered blog-type blog, I’m probably subscribed to that already. Either that or my friend Huginn is watching over you for me, and I’ll wake up to your wisdom fresh every morning in my daily digest, to enjoy with my delicious morning cup of coffee.

Doesn’t that sound nicer?

Still Aggregating

2022-08-12T10:09:04Z

This morning, I was having a conversation with a colleague about the UK federation’s metadata publication system. We needed to reference the order of operations and I remembered that I had once published an article about this system, along with a diagram illustrating just the point under consideration.

Pulling up the article in question, I was slightly surprised to find that the original article was published exactly ten years ago today.

The original architecture of this system was deliberately flexible, because we didn’t really know for sure the environment we’d be operating in. It’s gratifying, therefore, to see that the basic design has held up well enough that we can still use the article as an informal reference.

In detail, of course, things are different now than we anticipated and the deployed system has aspects which are simpler than a decade ago, as well as others which have required some elaboration. At present, for example, eduGAIN has been successful enough that it is currently the only active source of metadata other than that registered by the UK federation itself. On the other side, we produce a number of additional outputs today that we hadn’t really considered at the time, the most important of those probably being that we now publish per-entity metadata as well as the aggregate metadata illustrated.

Another major change – one that happened in 2013, about a year after the article was published – was that we started publishing the tooling itself in a GitHub repository for the benefit of other organisations working in the same area. I remember the big challenge there being that the original repository (which dated back to 2004) contained customer data; the version we published had to be filtered using git filter-branch for privacy (see the ukf-meta-meta repository for the gory details). We’ve since rebased our development so that our internal repository matches the published one except that we don’t expose development work in progress. I’m really glad not to be reliant on filter-branch any more, and I’m sure any reader who has dipped a toe into those waters will sympathise.

Anyway: Happy birthday, helpful diagram!