ayekat.ch

karuiwm

2021-12-24T11:33:00+0100

Today, merge request 14 was accepted into karuiwm's master branch, and karuiwm can now finally be used without having to embed user preferences into the binary at compile-time via a configuration header.

But what even is karuiwm, you're asking?

Let me reach back a bit.

dwm

In the year 2013, Yes, I am aware that 2013 is 8 years ago and this seems excessive. Bear with me, this is relevant. ayekat was a user of suckless' dwm, a dynamically tiling window manager that is—among other things—known for not loading any configuration at runtime. Instead, things like key bindings, window border colours and other user preferences are all declared in a C source header file, config.h, that is then included in dwm's source code at compile-time.

This requires the user to have some basic understanding of the source code itself, but that sounds scarier than it really is: The configuration file is mostly just a bunch of variable declarations whose syntax is relatively easy to grasp, and an example configuration file is provided to showcase how you can invoke specific functions within dwm.

For more advanced tweaking, though, you have to patch parts of dwm's source code itself. This is even actively encouraged (and people's patches are also gathered on dwm's website itself). Overall, dwm users quickly get to know its codebase and how to extend it the way they like. There's probably a reason why many of today's existing tiling window managers can be traced back to dwm in some form or another.

One notable property of dwm is that it doesn't group windows together to workspaces (or "desktops"). Instead, it provides tags that you can assign to its windows. So as a user, rather than switching around between workspaces, you instead decide which subset of tags to select (= which windows to show). This allows selecting windows a lot more flexibly, and overall, this is quite powerful if you grasp the concept.

Of course, ayekat didn't grasp the concept.

Instead, ayekat spent considerable time and energy into patching dwm and defining key bindings such that only one tag at a time could be active, and a window could be tagged with only one tag at a time, and one could switch between "adjacent" tags (left/right) and optionally grab a window and pull it from one "tag" to another, and— … yeah, well, it was basically workspaces.

dwm statusbar showing "basically workspaces".

At some later point, I abused dwm's tags system further to get a special "scratchpad" tag, with one window assigned that I could easily toggle on and off with a key binding, overlaying the windows in the current workspace (note that a scratchpad patch did not exist back then). All in all, I lived a pretty happy life with "workspaces" in dwm.

That is, until one day I saw florv use an XMonad module called GridSelect, which looked something like this:

My brain somehow misinterpreted this as some sort of visualisation of a two-dimensional map of workspaces where one could move around and create workspaces dynamically, and I was intrigued (especially given my own limited ~~tags~~workspaces). And I wanted that as well.

The two-dimensionally arranged, dynamically created workspaces, I mean. Not GridSelect.

Now it probably doesn't come as a surprise to anyone that dwm's code isn't exactly designed with workspaces in mind. And it definitely isn't designed with dynamically allocated workspaces arranged in two dimensions in mind.

To patch this into dwm, I would have needed to change its codebase to a point where one could not really consider it "dwm" anymore. Maintaining patches for such a thing would've been a nightmare; at this point, it would be easier and less painful to just write my own window manager. But who in their right mind would do that?

stwm

In 2013—the same year, but a little later—ayekat decided to write the Simple Tiling Window Manager. It didn't do much, but it did tile. And it did manage windows. Not any windows, mind you, but at least its own, empty windows.

The goal was pretty straighforward: Write a window manager in the vein of dwm (and largely inspired by its codebase, because I didn't know C and Xlib very well back then), with the features I wanted directly baked into it:

Workspaces.
Workspaces arranged in two dimensions!
A pretty-looking map to switch around between those workspaces.
A Scratchpad.
Workspaces!!

And so I spent Winter 2013 on writing my second-ever major C program, a window manager. Around the end of the year, it was then feature-complete enough for me to switch from dwm to stwm. To be honest, I'm still quite impressed by my past self, because this took less than two months overall. There's probably a reason why I failed all my maths courses in that semester…

Here's a screenshot showcasing the scratchpad and the desktops arranged in two dimensions (inside a Xephyr window):

Yes, this is from today. While skimming through my old screenshots, I noticed that the looks of my setup haven't really changed at all since back then, so I might as well just take a screenshot right now. Please kindly ignore the 8 years gap.

At some point (actually barely a week after I had started using it), I decided that "simple tiling window manager" wasn't a particularly uniquely identifying name, so I went with "karuiwm" instead. The attentive reader may of course have noticed that "karui" (かるい, 軽い) means "lightweight" in Japanese, so this makes this the "lightweight window manager", which… meh

Nevermind.

karuiwm and The Weight of Legacy dwm

karuiwm is not a fork of dwm; it was—and the Git commit history will confirm that—implemented from scratch. But it certainly wasn't a clean-room implementation in any sense of that word.

Much of the structure and many of the function and variable names even to this day will remind readers of dwm. There's monitors and clients; there's mfact and nmaster; one can step() through clients and shift() and zoom() them in the layout; and one can quit() karuiwm.

More importantly, though, karuiwm adopted the approach of not reading any configuration files at runtime, but instead requiring the user to write a config.h file very much like in dwm (with a very similar structure, to some extent probably even compatible).

While that worked fine in the beginning, at some point I decided that I would want to package all software installed on my system for better system files trackingThat was an odyssey in its own right; see the Tale told here., and I realised that sometimes I want configuration for different machines to be different. But how do you do that if the configuration must be baked into the software when being built?

In karuiwm's case, I built one package for each system.

That wasn't terribly elegant, of course. Here's for instance what the pkgbuild file looked like when I added a third system to my fleet. And while the target userbase was originally just myself, vanity probably still got the better part of me, and I wanted it to be useful for others as well. Or maybe I simply got tired of having to reply to curious friends that, yes, this was a window manager, but, no, it currently wasn't meant for them to use. Or maybe my inner pedant woke up and stated loudly enough that this was, in fact, not terribly elegant.

Whatever the reason, I decided to work on adding "configuration file reading" as a feature. But given the lack of any abstraction layers and the abundance of spaghetti in my code at that point, this meant that it would first take a bit of untangling and refactoring.

Or maybe just rewrite the thing from scratch…?

life engineering lesson

That was around the end of 2014. You may have noticed that 2014 was seven years ago.

Here's the most important lesson I took from this project: If anyone ever suggests to "just rewrite it from scratch" because refactoring the code would be too tedious and unfunny, please

Slap them.2014 ayekat? *Slap!*
2017 ayekat? *Slap!*

I wasted almost exactly 4 years of my life on this before it finally dawned on me that this was doomed to fail everytime: With new code, it doesn't really matter which components you implement at what point. You can easily leave out (or even remove) some critical feature with the promise of adding it back sometime later when "it's all cleaned up a bit", because who cares? You aren't sitting on that branch that you just sawed off. Nobody is.

But with code that is actually in production, it counts. If you break some critical functionality, you better fix it quickly, because otherwise it'll hurt. It's ugly and painful, and it isn't as fun as spitting out hundreds and thousands of lines of new code, but it forces you to stick to the existing feature set, and it prevents you from "giving up". In return, because you usually only make small changes incrementally, it's easy to leave the codebase and then come back half a year later to continue working on it. There are no "fantasy features" where you don't remember anymore what exactly the goal was; every feature is actually in use.

And it worked! It took me another 3 years to refactor the code, but it worked. Yes, 3 years is still a long time, but in my defense, Life somehow continued to happen as well, and besides, you should have seenActually, just see for yourself: The codebase in 2018 before I started refactoring. the initial spaghetti.

Also, it wouldn't be a true product of mine without at least some amount of overengineering:

Everything a user can do is exposed as an action. An action has a uniquely identifying name and a list of arguments it accepts; for example:
- focus_step_window changes the window focus to the previous or next window in the layout, and it takes one argument (an integer);
- scratchpad_toggle toggles the visibility of the scratchpad, and takes zero arguments;
- key_bind binds a key to an action, and it takes three arguments (a keymap name, a key combination, and a string describing the action invocation).
To allow users to invoke actions, karuiwm exposes an RPC mechanism via Unix socket, and reads line-based "commands" that invoke an action.
When karuiwm starts up, it invokes a script that connects back to the RPC socket and invokes actions like key_bind and window_set_border_colour. That script is of course user-defined.

Ultimately, karuiwm doesn't actually do any specific "configuration" loading at all. It simply sets up the RPC interface, and then hands off the configuration to an external script.

outlook

With the runtime configuration in place, I now no longer need to feel (too) bad whenever someone requests to try out the window manager; it is now at an acceptable pain level, I thinkI mean, there's no documentation, there's glitchy bugs everywhere, and things still change all the time and probably break your configuration before you've finished writing it. But I feel comfortable enough with inflicting that amount of pain to you..

That being said, after 8 years of dog-feeding myself, I've started to notice some limitations. They aren't major, but… let's say that I shouldn't have to be annoyed even by only minor things in a tool that I've written myself, for myself.

So here's what's planned for the near and far future:

mouse follows focus

When using key bindings to switch focus between windows, I often end up with the mouse pointer not being in the focused window. When using multiple monitors, the distance between the mouse pointer and the focused window can become significant; sometimes I have to move it around by potentially several thousand pixels to get it into the window I'm currently working in, which—as much as I love the ThinkPad TrackPoint—is a bit painful with the ThinkPad TrackPointOr however you call that thing.

In those situations, I often wish I had a "mouse-follows-focus" feature, where the mouse would automatically be warped into the currently focused window upon focus change.

I still need to determine how exactly the coordinates should be set, of course. Remembering the last coordinates and restoring those might work in some cases, but what happens if the mouse was moved out of the window manually the last time the window had (lost) focus? Or what about windows that have changed their dimensions in the meantime (and the saved coordinates now lie outside the window)?

modules

Many of the features may not be needed by all users, so it would be nice to just load the parts one really needs, but also allow users to easily extend karuiwm's functionality however they like, without having to get their code into karuiwm directly.

Here, I'm thinking about a modules system, by dynamically loading shared objects and then establishing some API that those modules could use to announce themselves to karuiwm and expose additional functionality (actions, layouts, …); a bit similar to—but hopefully a lot cleaner than—what I'm doing in karuibar.

The challenge probably lies in keeping the API simple enough to avoid getting tangled up in compatibility issues, while still being versatile enough to allow users to easily write useful modules.

In an initial step (also as a sort of dog-feeding), I'm planning to split off much of the current functionality into modules, e.g. the scratchpad, the workspace map (including, ironically (given the history of karuiwm), the fact that workspaces are arranged in 2 dimensions), or even just the RPC socket handling itself.

proper statusbar support

Right now, karuiwm displays a dummy statusbar with override_redirect that only shows some minimal information about the currently viewed workspace (a logo of the applied layout and the workspace's name). The rest of the statusbar is empty and unused; I currently just overlay karuibar in that empty area, though karuibar doesn't interact with the window manager at all.

The part to the right is hidden underneath karuibar.

The goal here is to stop shipping a statusbar with karuiwm altogether, and instead pass all necessary WM-related information to karuibar. For that, karuibar needs to be extended to at least:

Be able to distinguish between a focused and an unfocused monitor; and
Allow an external component to independently update one slot in the bar.

On the karuiwm end, I'm probably going to create a module that prepares information to be forwarded and shown in karuibar. Potentially it could be abstracted away in a way that one could also use other statusbar implementations (most importantly, I should start handling the _NET_WM_STRUT property, to support all kinds of statusbar implementations).

dummy windows

karuiwm manages one single workspace set, no matter how many monitors are attached. So monitors are "sharing" a set of workspaces between them.

Consequently, it may happen that one monitor tries to view a workspace that is already visible on another monitor. But X cannot show the content of one window in multiple locations,At least not without a compositor involved, I think so I need to handle this "conflict" case.

The current approach is fairly straightforward: Simply swap the views between the monitors, and we're done. But that occasionally causes some confusion. Imagine the following:

On monitor A, show the dev workspace; on monitor B, show the chat workspace.
On monitor A, switch to the chat workspace, then notice the mistake, and switch to the mails workspace. Later, switch back to the dev workspace.
Now try to look at the chat (expected on monitor B), but instead see mails there.
Confused ayekat.

Here, I would prefer monitor B to simply stay at the chat workspace. But now I need a solution for what happens if multiple monitors try to view the same workspace (and thus the same windows).

The new idea here is to let the unfocused monitor stay on its workspace, but because the windows are also shown on the other (focused) monitor, have the unfocused monitor simply show some "dummy" windows in their stead, i.e. empty windows that only contain the name of each client window that would be shown there. As far as I know, there's a project called clfswm that implements something like this.

As soon as the other monitor moves away, "reclaim" the client window content again, and everything is fine. No confused ayekat.

Whether I'll implement this by switching between dummy windows and client windows, or whether I'll start reparenting windows (such that there's always a dummy window, but sometimes it simply doesn't have the client window mapped) remains to be defined.

Wayland

I guess one of the major issues of working on something for almost a decade is that the technology landscape around you changes so much that by the time you're finally done, it has already become obsolete.

X still isn't quite dead, fortunatelyI mean, mostly just fortunate for karuiwm., but I should still consider preparing for a transition to Wayland in some not-too-far future.

To do so, I will first need to split all the X-specific parts away from the generic window management part in karuiwm. Ideally, it will become its own module that provides some generic "compositor interface" to karuiwm, and then I could provide a second module exposing the same interface towards karuiwm, but implementing window management (or rather, compositing) for Wayland.

It probably isn't as simple as I make it sound here, but one can dream…

but, so, is it usable now?

For playing around? Sure!

For production use? Hell no.

As noted somewhere aboveAt this point, I wouldn't even be mad if you somehow missed it in this wall of text., documentation is still severly lacking, there are some awkward glitches and occasional crashes, and overall a lot of behavioural changes are still expected, especially in the context of the RPC.

I mainly wrote this article because I felt excited about the fact that I have finally completed something that had kept me busy for the past 7 years, and how a project that's been going for over 8 years at this point is finally starting to take some serious shape, and I somehow wanted to get my excitement out into the World.

That being said, a first release is now becoming more and more realistic, and I expect version 0.1 to drop sometime in 2022. Then again, there was a time in late 2019 where I thought I'd be done by Summer 2020, so…

Let's just not stress ourselves to much.

Freenode

2021-05-29T14:13:00+0200

Farewell!

13:58 ayekat│ info
13:58 -NickServ(NickServ@services.)- Information on ayekat (account ayekat):
13:58 -NickServ(NickServ@services.)- Registered : Aug 20 08:22:44 2010 (10y 40w 5d ago)
13:58 -NickServ(NickServ@services.)- Last addr  : ~ayekat@███████████
13:58 -NickServ(NickServ@services.)- Last seen  : now
13:58 -NickServ(NickServ@services.)- Logins from: ayekat
13:58 -NickServ(NickServ@services.)- Nicks      : ayekat
13:58 -NickServ(NickServ@services.)- Email      : ████████@█████████ (hidden)
13:58 -NickServ(NickServ@services.)- Flags      : HideMail
13:58 -NickServ(NickServ@services.)- *** End of Info ***
14:00 ayekat│ drop ayekat █████████████████
14:00 -NickServ(NickServ@services.)- This is a friendly reminder that you are about to destroy the account ayekat.
14:00 -NickServ(NickServ@services.)- To avoid accidental use of this command, this operation has to be confirmed. Please confirm by 
          replying with /msg NickServ DROP ayekat ██████████ ████████:████████
14:00 ayekat│ drop ayekat █████████████████ ████████:████████
14:00 -NickServ(NickServ@services.)- The account ayekat has been dropped.

You can now find me on Libera.Chat.

If you can't find me, occasionally I'll take on the name of this lovely fellow:

Artist: Alex Konstad
Image © 2018 Wizards of the Coast

In case you've been living under a rock, search the Web for "freenode libera.chat".

Not-a-bug fixing

2019-03-14T15:00:00+0100

As far as system maintenance is concerned, I guess I have some minor case of obsessive compulsive disorder. I very much dislike having random applications dropping files in my home directory, and I am equally displeased whenever I find software packages on my system that I don't need. While I consider it a waste of time and bandwidth to download and perform updates for such packages, what causes me more unease is the mere fact that the system does not appear clean to me. It is clutter that serves no purpose.

So whenever I see a package that I do not recognise, my very first reaction is typically

Why is this package installed?

In theory, this is very easily determined:

Was the package installed explicitly? Then I should know myself why I installed it.
Was the package installed as a dependency? Then I should look at its reverse dependencies, recursively, until I hit the "topmost" package (i.e. the one that was installed explicitly); then see above.

In practice, however, there are a couple of obstacles.

First, I don't always remember why I installed a given package. Sometimes I need it for some one-time task and then forget to remove it afterwards. And a few months later, I've forgotten about it entirely.

To avoid that sort of situations, I've started writing meta-packages for the various tasks. A meta-package has a name, a description of the task, and a list of package dependencies required for that task. Rather than directly installing the required tools explicitly, I install the meta-package and let that pull in all the wanted packages as dependencies.

After shifting to using meta-packages for almost everything on my system, I felt rather clever and happy, as I had found a way to exploit the package manager's dependency system to encode all the information I wanted.

I didn't feel clever for very long, though. Because…

Second,

Why are the dependencies broken?

My distribution of choice is currently Arch Linux.

Arch Linux uses the package manager pacman. Pacman defines so-called package groups, which on a technical level behave like tags or topics, i.e. they have no influence on how packages depend on, conflict with, or in any other way relate to each other. However, rather oddly, pacman allows installing and removing groups with the same syntax as used for packages. This explains why some pacman users (and even developers) occasionally mistake groups for meta-packages.

Arch Linux itself doesn't really help resolving this confusion either, as it doesn't use groups very consistently. In many cases, they are like tags; packages are assigned to groups like vim-plugins or xorg, which aren't intended to be installed all at once. But there are also groups like lxqt or mate, which may very well be installed in their entirety; however, simply replacing them by meta-packages would cause issues for users who wish to remove some of the packages.

Furthermore, there are groups like base-devel, a set of packages that are assumed to be present on a build system, and thus never explicitly listed among buildtime dependencies for Arch Linux packages; it makes little sense to install packages from this group only partially (at least if one installs packages within the context of base-devel), so replacing it by a meta-package would very much be feasible.

Given the confusing nature of groups, for a user it would be easiest to simply ignore their existence entirely (except for base-devel: turn that into a meta-package, and be done with it).

It would be easiest.

Because Arch Linux has managed to add to the confusion about groups by adding a group whose own purpose is entirely undefined: base.

Depending on whom you ask, the reply you get will be different. Some developers argue that all packages in base are supposed to be installed on all Arch Linux systems, and will intentionally not declare some dependencies when they are in base. There are other developers who take a more moderate stance and argue that only some packages should be expected on an Arch Linux installation (glibc or filesystem come to mind), while others should be uninstallable without issues; essentially they agree with the other ones, but argue that the current base is too large—unfortunately, nobody is clear where exactly they draw the line.

And last, there are the pedantic people like me who argue that base should be considered a set of "recommended packages" at most, and packages in there should be removable in any way the user wishes, i.e. none of the packages should be implicitly assumed to be installed.

But all those opinions don't matter, because this is a case of a metaphorical chain breaking at its weakest link: even just one maintainer intentionally not declaring dependencies on base packages is enough to "compromise" the entire system: users now have to globally assume that things may break if base is not fully installed.

Essentially, the user is forced to have installed a bunch of packages that they will never need.

Arch Linux is a general-purpose distribution. Upon installation, only a command-line environment is provided: rather than tearing out unneeded and unwanted packages, the user is offered the ability to build a custom system […]

Arch Linux: Versatility

The lack of policy also means that users cannot submit any bug reports for such packages: if there is no document that properly describes the base group, any interpretation of it is a valid one, no matter how absurd. In fact, if I were to only submit packages to the Arch User Repository where I state all the base packages as dependencies, no matter if required or not, my packages might get removed by an AUR overseer at some point, but the reasoning would probably be something in the vein of "absurd" or "silly", but nobody would be able to refer me to a document stating that my interpretation of the base group was technically wrong.

But there was a time when I didn't know that. In autumn 2017, when setting up a new system, I managed to crash the locale-gen command by not having sed installed. The glibc package (which provides the locale-gen command) doesn't mention sed at all, neither as a mandatory nor optional dependency. So I did what every optimistic, enthusiastic, naive Arch Linux user would do: open a bug report. This one. And this is what followed on the developers-only mailing list.

… and there went my enthusiasm.

persona non grata

Based on that reaction in particular and what I had otherwise seen of the Arch Linux bug tracker overseers' behaviour on the forums and mailing lists, I got the feeling that they would now be specifically careful not to resolve any of my present or future requests. As far as the bug tracker was concerned, I was now practically dead.

Not that this would change much; the majority of issues I had previously reported (or voted for) had also been either rejected or simply ignored, despite having patches or trivial solutions attached. It appeared to me that my perception of what is considered a "bug" differed quite a bit from the Arch Linux maintainers' one. I concluded that I couldn't really trust my own judgement on that matter anymore: if something appeared like a bug to me, is it really a bug? Should I bother reporting it?

I decided no.

It wasn't worth the hassle.

I can fix this shit myself

Building packages for pacman is not rocket science. The packaging tooling isn't exactly powerful, but it is also rather simple, and it gets the job done without getting in my way—so actually quite fitting for a distribution that Arch Linux claims to be. And having used this distribution for more than seven years and packaged quite a few things, I decided that I was capable enough of taking over some of their packages as well.

While I didn't feel comfortable tackling a package as fundamental as glibc right away, I felt confident enough to at least adopt debootstrap (FS#48908), lighttpd (FS#45902, FS#51931) and pass (FS#55059, FS#55504).

I started by setting up a personal package repository on a publicly accessible server, archlinux.zuepfe.net, and configured my systems to add that repository to their pacman package sources. Then I wrote a convenience script that would allow me to more easily push packages to that repository, zr (for Züpfe Repositories), and added my custom variants of abovementioned packages to my custom repository. Up to this point, things were rather smooth.

But then I started realising that this was going to be a lot of work—at least for a regular user. I had no way of knowing when a library soname bump related rebuild was due, or when some bug was reported for the "real" version of the packages that would force me to react someway. I would also need to subscribe to upstream channels to get all the relevant information required as a packager for software I was not necessarily very familiar with.

A lot of duplicated efforts, ultimately. And all that for merely changing some metadata…?

That sounded a bit backwards.

Let's just repackage

Pacman packages are pretty simple. They are Xzipped tarballs containing the following three files at the top level:

.BUILDINFO: a list of packages installed on the build system as this package was built;
.MTREE: a Gzipped list of files contained in the package, following the mtree(5) format;
.PKGINFO: all package metadata (except what is listed in .BUILDINFO and .MTREE), i.e. name, version, description, dependencies, etc.

The tarball may also optionally contain a .CHANGELOG file and a .INSTALL file that defines actions to be triggered upon installing, upgrading or removing the package. Everything else in the tarball is the files that are part of the package, with a directory structure reflecting how they will be installed to the system.

That's all, no magic. Theoretically, we can easily modify a package directly: just unpack the tarball, then modify the metadata or add/rename/patch/remove a file. Sounds crazy? Then let's do it. Rather than maintaining our own version of the package and building the software from scratch, we just use the work that's already been done by the package maintainer. This is about as low-effort as it can get.

And thus, repkg was born.

And this is the list of packages that I currently modify with it. I can now get sane packages, and the Arch Linux maintainers can keep their bugs without having a pedantic who complains. That's a win-win.

future

It's now been 1.5 years since that "bug tracker incidence", and the base group still hasn't changed. I've seen 3 discussions among the developers come and go again without any notable progress. In the most recent iteration, people still seemed to have different ideas of what the purpose of the base group should be, so the one concrete proposal of a new (slimmed down) list of packages in base seemed just halfhearted at best. I don't expect this issue to be resolved anytime soon.

Then again, even if the Arch developers suddenly changed their minds and decided to fix all the things I consider issues, I'd still keep using repkg to modify packages to my needs, because it's just so damn convenient. Not only can I now easily fix all the cases of "that's not a packaging bug", but also fix upstream issues, and otherwise adapt packages to my personal needs—and I wouldn't want to miss that.

As far as repkg is concerned, I've written a wrapper script, remakepkg, that handles the downloading of packages and makes the command line invocation more convenient. But I've noticed that it becomes a bit tedious after a while:

First, it is currently hardwired to the current checkout of the package database, so it can't repackage a package from a newer, remote database "ahead of time", i.e. I first have to attempt a pacman -Syu, have diffrepo detect that the repositories are out of sync, then repackage and push the offending package to my repository, and try again.

Second, I'd like it all to happen more automatically. The repackaging rules are not bound to a specific package version; they can apply to any version of the package (as long as the relevant parts don't change), so it could also just run fully automated, rather than having to invoke it manually each time.

Third, it doesn't scale particularly well: it works fine for one or two dozens of packages, but if I crank up things to the hundreds (e.g. because I want to fix something more fundamentally that affects a great number of packages), running remakepkg for each one separately isn't terribly convenient.

But this is all assuming that I'll keep using this distribution. After 9 years, I've started to see quite a few cracks. And for a distribution that I would have otherwise considered as "not getting in my way", I must admit I've now spent a considerable time working around its flaws.

It's still good enough, though, so…… meh

Hierarchaeology

2017-10-30T22:22:00+0100

(and other halfway forgotten things)

Back in the old days of Unix, things used to be a little different. While this may sound like I'm another one of those who like to dwell in the "good old times", that is not really the case. I have only encountered Unix (through Linux) in the late 00's of the ongoing century, so the best I can do is an archaeological approach—which incidentally makes it all the more interesting.

A few random things I've found (and a few myths and backronyms that I've found to be rather wrong):

/etc really just means et cetera. The notion of "configuration files" didn't exist back then (one would have to recompile a program to change an aspect of its behaviour); this directory simply contained files that didn't fit elsewhere. Hell, even init used to be in here. Configuration files were not specifically only in this location (there was no specification for them), and it was only when other things got gradually moved out and the only remaining thing seemed to be configuration files, that this directory began to be seen as the "configuration directory". Note, though, that even today, it is not exclusively configuration files in here.
/sbin originally just meant static binariesThis is no longer the case on Linux (probably never has been), and quite probably neither on BSD., as in: Binaries that do not rely on an external library and are thus ideal for being used in a limited environment (e.g. early boot). This made them useful for low-level system plumbing—this aspect somehow morphed into the perception that sbin contains "system administrator tools", and nowadays, it is "hidden" from normal users by not having it in $PATH (and that separation causes issues more often than it solves any).
/usr originally just meant user. It was the mountpoint of a separate disk, and user data was moved here to keep the root filesystem from overfilling. Likewise, other files like manuals (man), source files (src), and "non-essential" binaries (bin) and libraries (lib) were moved into their equivalent under /usr for the same reason. This left the "first disk" with the root filesystem only with the tools to bootstrap the system and mount /usr. Note that these days, this is handled by the initial RAM disk, so the separation is pretty pointless.
When /usr started filling up as well, user-personal data was moved to a third disk, mounted under /home. Ironically, this left the original "user directory" as a location for non-user, system-wide data.

When GNU/Linux entered the scene (roughly 20 years later), that layout had already become semi-formalised. As a result, we now find ourselves in this legacy-loaded mess.

But hey, at least it is somewhat standardised! And there is a slow-but-steady effort to improve the hierarchy: currently, the various bin and lib directories are being merged together (although, on some distributions the arbitrary split between sbin and bin is still retained).

Other old trivia that is not hierarchy- but still filesystem-related:

Prepending dots to a filename to make that file "hidden" was an accident. Back in the starting days of Unix, ls used to show all entries in a directory. All of them, even . and .. for the current and parent directory. This was of course a bit redundant (every directory contains them), so the authors decided to add code to ls to hide those entries by default. But instead of ignoring . and .., they took a shortcut: ls should simply ignore each entry that would start with a dot.
Elegant, heh?Actually, the originally intended behaviour eventually got implemented as ls -A.

The list is admittedly a bit short. I will extend this over time whenever I stumble over another trivia that I find interesting.

Sketches

2017-04-23T17:11:00+0200

After about 3 years of being offline, I have finally found some time to hack together a webcomic-panel like structure on this website to re-publish my stick figure drawings. I used to refer to them as sannies, but I find the name a bit silly, so… sketches it is!

For those of you who don't know what this is all about: I used to draw stick figures during the classes in high school (and early EPFL) as "memory aids"—hence the name sannies (Sketches As Notes). Over the years, I have started to scribble down all kinds of other things, though; I got some positive feedback, and I thought "why not?"

So besides the early "course topic" ones and snapshots of awkward and interesting classroom scenes (sometimes depictions of my frustration), the portfolio now contains a somewhat extensive collection of "capture the mood"-like drawings:

And there should be most of it—the rather embarassing ones (for historical accuracy) as well as the countless depictions of our main class teacher. She might have had a little rough job with the bunch of us—me included… but hey, despite my abyssmal french skills back then, I've made it to Lausanne! Have I redeemed myself?

She was awesome.

I haven't yet finished the series on poor old Dr. Goldstein, although there are two more drawings that I haven't scanned in yet—there's more to come! Unfortunately, academic institutions don't value free time as much as I'd love them to, so that's going to be another thing to do this summer.

As a final remark, if I may: my main motivation for this was actually just to properly explain the origin of my online avatar picture:

the camera man

Epoch

2017-03-29T16:50:00+0200

I always wondered why we would anchor the counting of our years to an event mostly relevant to only one World religion: the "international and secular" year 0 really just denotes the year in which the Christian religion's prophet is believed to have been born. This might explain why some parts of the globe have not adopted that calendar yetOr rather simply that Status Quo has repeatedly shown itself to be the strongest force opposing any kind of improvements.. And who would blame them.

But then the question is, what makes a good secular calendar?

The Unix-phile person in me would welcome a system that is not too complicated to use and implement—I'd say the current Gregorian calendar is not all too bad in that respect. So the following suggestion by a friend of mine (who incidentally happens to be a Finn) got my full approval:

Year 0 denotes the beginning of the Unix epoch, i.e. AD 1970.
T. Tuuva

Seems simple. Relocate year 0, and keep things the same otherwise.

Of course this would blatantly ignore the fact that not all cultures use 365-day or 12-month cycles to mark a "year", or 7-day cycles to mark a weekAlthough from a quick glance I could not find any other week lengths., but those are rooted in very ancient religions or astrology, and are thus not necessarily limited to just one geographic region.

Either way, we need to "rethink" a few things, of course:

In the Unix epoch, the Christian epoch starts in −1970. Converting a year number from that "Old Calendar" is thus of course simply done by subtracting 1970.
Since negatively numbered years are not very user-friendly, those years should be referred to as Before Unix, or BU for short. To explicitly refer to a positive year number, we might adapt something like Anno Unix (AU). It's perhaps silly, I don't know—we'll see.
A few randomly selected major events in history (to "reorient"): Cæsar was murdered on the ides of March, in 2014 BU. The Middle Ages started around the 14^th century BU and ended around the 5^th century BU. Sgt. Pepper was released in the year 3 BU, Star Wars screened in AU 7. The Berlin Wall fell in 19, and 9/11 happened in 31. And (hopefully not last or least) we'll run into the Year 68 problem in… well, 68.
Many of us might be young enough to witness the second century.

To force myself to get into the habit of using that new system, I decided to concretise this idea a bit more and configure my computer to display and use dates in this system; the obvious first place for that was the standard C library. Since we do not touch the internal time system (it already is in the Unix epoch), we only need to adapt the external representation of the date, namely what strftime() returns.

Unfortunately, strftime() acts on a struct tm, which starts counting the years in 70 BU (AD 1900), so strftime() does some transforming on its own already, which does not exactly simplify things.

Moreover, there are several format specifiers for the year (e.g. %Y and %y for the year itself, %C for the century, and %G and %g for the ISO-week-based yearE.g. if december 31 is a sunday in 48, it is part of week 1 in the next year) + a few "composite" specifiers like %D (%m/%d/%y) and %F (%Y-%m-%d). To make matters more complicated, there are also format modifiers like %E and %O (although I have not yet looked into those).

So…

quick'n'dirty

For a quick test-"baptising", rather than trying to modify the abovementioned format specifiers correctly, I decided to simply add a new specifier, %f (note that this diff applies to glibc 2.25—it may need to be adapted for other releases):

diff --git a/time/strftime_l.c b/time/strftime_l.c
index eb3efb8129..18b339d926 100644
--- a/time/strftime_l.c
+++ b/time/strftime_l.c
@@ -1013,6 +1013,9 @@ __strftime_internal (CHAR_T *s, size_t maxsize, const CHAR_T *format,
      cpy (buf + sizeof (buf) / sizeof (buf[0]) - bufp, bufp);
      break;

+   case L_('f'):
+     DO_NUMBER(1, tp->tm_year - 70);
+
    case L_('F'):
      if (modifier != 0)
        goto bad_format;

Then, to enjoy my transition to the new epoch, I modified my status bar to use %f-%m-%d for the date display:

Should've waited another 6 minutes… just for the sake of it.

Of course, applications do not use %f, so this approach is a dead end. But it's a start.

slow'n'properly

I initially feared that changing the code for %Y, %y and all the other specifiers would degrade to a gorefest, but the code turned out to be remarkably clean. In comparison, the code I then introduced was a little less clean. But for testing purposes, I considered it "well enough":

--- a/time/strftime_l.c
+++ b/time/strftime_l.c
@@ -880,7 +880,7 @@ __strftime_internal (CHAR_T *s, size_t maxsize, const CHAR_T *format,
            }

          {
-           int year = tp->tm_year + TM_YEAR_BASE;
+           int year = tp->tm_year - 70;
            DO_NUMBER (1, year / 100 - (year % 100 < 0));
          }

@@ -1212,10 +1212,10 @@ __strftime_internal (CHAR_T *s, size_t maxsize, const CHAR_T *format,
            switch (*f)
              {
              case L_('g'):
-               DO_NUMBER (2, (year % 100 + 100) % 100);
+               DO_NUMBER (2, ((year - 1970) % 100 + 100) % 100);

              case L_('G'):
-               DO_NUMBER (1, year);
+               DO_NUMBER (1, year - 1970);

              default:
                DO_NUMBER (2, days / 7 + 1);
@@ -1257,7 +1257,7 @@ __strftime_internal (CHAR_T *s, size_t maxsize, const CHAR_T *format,
          if (modifier == L_('O'))
            goto bad_format;
          else
-           DO_NUMBER (1, tp->tm_year + TM_YEAR_BASE);
+           DO_NUMBER (1, tp->tm_year - 70);

        case L_('y'):
          if (modifier == L_('E'))
@@ -1276,7 +1276,7 @@ __strftime_internal (CHAR_T *s, size_t maxsize, const CHAR_T *format,
 # endif
 #endif
            }
-         DO_NUMBER (2, (tp->tm_year % 100 + 100) % 100);
+         DO_NUMBER (2, ((tp->tm_year - 70) % 100 + 100) % 100);

        case L_('Z'):
          if (change_case)

And indeed, after installing the patched glibc and rebooting the machine, the system started displaying dates with the new epoch format. Here's an example output of systemd:

$ systemctl --user status
● testbox
    State: running
     Jobs: 0 queued
   Failed: 0 units
    Since: Tue 47-03-28 15:47:49 CEST; 1h 4min ago
   CGroup: /user.slice/user-1000.slice/user@1000.service
           └─init.scope
             ├─306 /usr/lib/systemd/systemd --user
             └─315 (sd-pam)

So everything seemed neat and rainbow-y…

… except for coreutils

For some reason, date, ls and all the other programs provided by coreutils were still stuck in the Gregorian calendar. And indeed, after taking a closer look, it turned out that coreutils maintains its own version of strftime() (and quite probably a few other (g)libc components).

Yay! More patching!

--- a/lib/strftime.c
+++ b/lib/strftime.c
@@ -906,9 +906,9 @@
             }

           {
-            int century = tp->tm_year / 100 + TM_YEAR_BASE / 100;
-            century -= tp->tm_year % 100 < 0 && 0 < century;
-            DO_SIGNED_NUMBER (2, tp->tm_year < - TM_YEAR_BASE, century);
+            int century = (tp->tm_year - 70) / 100;
+            century -= (tp->tm_year - 70) % 100 < 0 && 0 < century;
+            DO_SIGNED_NUMBER (2, tp->tm_year - 70 < 0, century);
           }

         case L_('x'):
@@ -1275,18 +1275,17 @@
               {
               case L_('g'):
                 {
-                  int yy = (tp->tm_year % 100 + year_adjust) % 100;
+                  int yy = ((tp->tm_year - 70) % 100 + year_adjust) % 100;
                   DO_NUMBER (2, (0 <= yy
                                  ? yy
-                                 : tp->tm_year < -TM_YEAR_BASE - year_adjust
+                                 : tp->tm_year - 70 < -year_adjust
                                  ? -yy
                                  : yy + 100));
                 }

               case L_('G'):
-                DO_SIGNED_NUMBER (4, tp->tm_year < -TM_YEAR_BASE - year_adjust,
-                                  (tp->tm_year + (unsigned int) TM_YEAR_BASE
-                                   + year_adjust));
+                DO_SIGNED_NUMBER (4, tp->tm_year - 70 < -year_adjust,
+                                  (tp->tm_year - 70 + year_adjust));

               default:
                 DO_NUMBER (2, days / 7 + 1);
@@ -1326,8 +1325,7 @@
           if (modifier == L_('O'))
             goto bad_format;

-          DO_SIGNED_NUMBER (4, tp->tm_year < -TM_YEAR_BASE,
-                            tp->tm_year + (unsigned int) TM_YEAR_BASE);
+          DO_SIGNED_NUMBER (4, tp->tm_year -70 < 0, tp->tm_year - 70);

         case L_('y'):
           if (modifier == L_('E'))
@@ -1346,9 +1344,9 @@
             }

           {
-            int yy = tp->tm_year % 100;
+            int yy = (tp->tm_year - 70) % 100;
             if (yy < 0)
-              yy = tp->tm_year < - TM_YEAR_BASE ? -yy : yy + 100;
+              yy = tp->tm_year - 70 < 0 ? -yy : yy + 100;
             DO_NUMBER (2, yy);
           }

Now that seems almost fine. Notice how for the %Y modifier, coreutils actually pads the year number with zeroes to have it use at least 4 digits, in any case? So we get 0047 instead of just 47, for instance.

However, I sense that fixing this would cause a great disturbance in the Force. As if millions of badly written scripts suddenly errored out in terror and were suddenly terminated.

Alright, we done here?

a rabbit hole

What about the input? For instance, how would coreutils' date interpret dates passed with the --date option?

$ date --date='2017-03-28'
Tue 28 Mar 00:00:00 CEST 0047

Well, that doesn't look terribly right—that'll require patching, too…

At this point, I stopped.

The problem is, patching strftime() is comparatively trivial (as we've seen above) and solves the output problem. Most applications—independently of the language they are written in—somehow make use of libc's strftime() at some point either directly or through bindings, so there isn't really much of duplication of code here (except for coreutils, because).

But reading date and time values and converting them to an internal representation is a different thing: there are quite possibly countless different implementations for that, in all sorts of programs and languages, and they most likely all assume that we're in the Gregorian calendar. The fact that struct tm puts its year 0 to 70 BU (AD 1900) doesn't exactly simplify things (although, if we think about it, that's a different problemBut still a problem.).

That's a rabbit hole that I don't really want to step into.

final thoughts

The idea still seems tempting: switch to a different time epoch and get rid of the religious baggage. But as long as our everyday live keeps using the Gregorian calendar (and for us geeks in particular, as long as the computer keeps using the Gregorian calendar), it's rather difficult to get into the "Unix mode".

And while my naive approach of simply adding a new format specifier (%f, remember?) to strftime() seems a little lazy, the alternative would be a matter of "all or nothing", with "all" being pretty much impossible for a single person, and "nothing" being downright frustrating. "Some" is a dangerous middle ground where things will break sooner or later anyway.

Another aspect we haven't looked into yet is the way we treat leap years. Because the years in the Unix calendar basically have the same length as the ones in the Gregorian calendar, the same 4/100/400 rule can be applied here, too. But the leap years have been established using the Gregorian calendar; in the Unix calendar, they would be in 2, 6, 10, 14… with the first "non-leap-year" in AU 130 (AD 2100). We have to convert our dates to the Gregorian calendar before determining whether they are leap years or not—and frankly that isn't very convenient. But we'd need more adoption for changing that.

We'd need more people.

erratum

Actually, the Gregorian calendar starts counting years at AD 1. The year preceding AD 1 is 1 BC. Consequently, we have Gaius Julius Caesar who died in 44 BC, which is actually 2013 and not 2014 BU.

The astronomical calendar (and ISO-8601) do have a year 0 that corresponds to 1 BC (so the year numbers for everything before AD 1 are off by 1), even though it's not used very often in historical contexts.

Speaking of standards—RFC 3339 also states that year numbers are expected to be 4-digit numbers. So coreutils padding the year number as seen is actually standard-conform behaviour. The standard also explicitly states that years are expected to lie between AD 0000 and AD 9999—it does not mention negative years or years above 10,000 at all, so at least I can rest assured that the RFC will be adapted someday (and perhaps the Unix epoch can establish itself until then and profit from the era of the date notation instability).

Shadows

2017-03-08T00:10:00+0100

I apologise for the 4.1 MiB and their potential effect on your quota. Although… something something modern websites deal with it

Cleaning up my photographs from the past year, I came across a few things I considered worthy to be shared. The first thing is probably this video showing the effects of sunlight shining on a bird window sticker through leaves moving in the wind:

trees + wind + window = moving shadows!

… and yes, this is the original speed at which it was recorded. If you don't believe me, feel free to play the video again a few times. After all, make use of the 4.1 MiB I just made you download for seeing this page.

And just for the pinch of technical'ism in this blog:

ffmpeg -i DSC_0203.MOV -c:v libvpx -crf 5 -b:v 2M -an bird.webm

VP8/VP9 experts, awake and cringe! Or contact me and tell me all the things that are wrong with that line.

日本語 in LaTeX

2017-03-03T01:00:00+0100

Seldom is it a pleasant experience to discover that you are wrong and need to adapt. But unpleasant experiences make life… well, Life. In a good way; let's say, spices that give some perspective on the sweet things, to make a somewhat pityful attempt at poetry. Or WebKit security issues that make us abandon old and unmaintained web browser—ah yes, there, I think I'm slowly getting the hang of this. What would Life be without stories to tell of past disasters, both little and greatLet us live through the little ones and tell stories of the great ones! Or so.
… OK, I'll stop..

The other day in particular, I found myself defending my use of pdflatex to a much wiser person, stating that \usepackage[utf8]{inputenc} would indeed allow UTF-8-encoded characters in in @LaTeX@ documents to be treated correctly. That was until I was gently told that inputenc is really only a hack: try to use something marginally exotic, and pdflatex will show you the metaphoric door.

I got shown that door without much seeking:

ん	わ	ら	や	ま	は	な	た	さ	か	あ
		り		み	ひ	に	ち	し	き	い
		る	ゆ	む	ふ	ぬ	つ	す	く	う
		れ		め	へ	ね	て	せ	け	え
	を	ろ	よ	も	ほ	の	と	そ	こ	お

Ever since I have managed to make Japanese input work again with IBus, I seem to constantly seek excuses to type it everywhere. And yes, pdflatex did not cope with it.

So I have tinkered around on my playground-@LaTeX@ file, hunting it through various incarnations (while letting it produce the same blank page over and over), incidentally switching to xelatex—until it finally decided to give a glimpse at what Hiragana, Katakana and Kanji might actually look like if typeset by Knuth: And behold!, the Source.

publishing

Of course, the vanity in person that I am, I wanted to share this, so I had to adapt a few things in ayekat.ch, too:

Specify the Minchō font typefaces "IPAMincho" and "Sazanami Mincho" in the site's CSS, to have the Japanese fonts be more in line with the rest of the site's serif font—not sure what this actually looks like on a system that does not have these fonts installed.
Add a "token substitution" system to my markup handler, so that I can simply type @LaTeX@ to conveniently get the stylised @LaTeX@ logo printed. To do so, I had to Spaghetti-code some workarounds in order to get rid of those pesky HTML tags in the site's title;… yes, I would like some Bolognese sauce and Parmigiano cheese with that Spaghetti code, thank you very much. precisely speaking, I use an additional, "stripped" variable + a little condition checking for when to print which title variant.
Put in a hack to have the text/x-tex MIME type be recognised as being encoded with UTF-8 (as python doesn't detect that properly, apparently)… yes, yes, I would really love some sauce and cheese!, so the web browsers would properly show the @LaTeX@ source file for abovementioned outcome.

Coding at its finest, eh?

/var/log/computers

2017-02-23T22:29:00+0100

With a bit of nostalgia, after going through the history of waltz for the new incarnation of the about page, I thought I could also go through my history of desktop hardware. However, this turned out to be rather difficult, as I did not really keep track of my computers as a child and teenager. Consequently, this undertaking turned mostly into me digging through pictures on the web and comparing them with my ever-dwindling memories of what I had previously owned and used.

Nevertheless, I decided to devote this (mini-)article to my first-ever personal computer:

Compaq Presario 7222

Around '99 (when Compaq still used to dominate the personal computer market), my father acquired a shiny new tower (Compaq, too), running Windows 98.

And I was given this gem:

Image taken from Recycled Goods.
Please imagine a bulky CRT screen placed on top of it.

According to the data I could find about itSysOpt, Kahlon, the CPU was an Intel Pentium 1 (that would be x586, I believe) with a whooping 100 MHz, 8 MB of main memory, 1.2 GB of storage (not sure if the hard drive was 5.25 or 3.5 inches) and support for various phone technologies that I have never got to know before they became extinct in the personal PC sector. According to the data, it also had a sound card, even though I tend to associate this PC with "no sound"—mostly because I never had any audible warnings when enemies attacked me in Age of Empires.

Attached to it was a bulky CRT screen (the same dirty white), and—drum roll, please—my favourite keyboard of all time (in the same dirty white):

Notice the split spacebar? The Split Spacebar?? SPACEPAR!?!

Yes, the split spacebar. I think my love story with this computer was also partly grounded in this keyboard. The left space key was actually a backspace, which meant that I could comfortably erase my typing errors without having to move my fingers around too much—which in turn drove me to learn touch-typing (at the age of 11, I think, without even knowing how a computer really works).

I have kept that keyboard for as long as my computers used to provide a PS/2 port—which was roughly until 2007.

The machine itself was running Windows 95, and during my childhood, it served two purposes:

Playing Age of Empires—without sound.
Writing documents and storiesBecause I wanted to become the next J. K. Rowling. Or Ephraim Kishon. I don't know. with Microsoft Office 97 Word—without sound.

And things had to be done in precisely that order. If any application was launched before running AoE, AoE would simply hang. But it was enough, and I had a happy childhood—and who needs Internet, anyway?

Did I mention I had no sound?

Tufte CSS

2017-01-31T16:56:00+0100

Vanity has caught me again. Inspired by Dave Liepmann's Tufte CSS, a hommage to Edward Tufte's work on data visualisation and his effort to translate it to the Web, I have mostly tried to integrate those nifty sidenotesWebsites usually have more freedom in the horizontal space, and the concept of a footnote or a bottom of page really has no meaning here. Also they look cool, don't they? into my blog, but also wanted to solve the issue of text lines generally being too long, while other content (code, graphics) was generally too narrow. And since this required some tinkering in the CSS, I considered my days off from work over this prolonged weekend as a perfect opportunity to spice up my blog's design with this small change.

And indeed, getting the sidenotes into the blog turned out to be rather easy. But to keep things consistent, I considered it a necessity to apply this main–side area separation to all the parts of this website. Which of course meant I had to tinker around a little more in the Python CGI codebase.

Oh boy.

In 2015, I might have been proud about what I had done. But looking at that code today meant an unsettling amount of head-shaking and fist-rising, occasionally accompanied by a whispered "What the hell". The code had been split up, yet I failed to see any kind of logic of how it was done so. It seemed to me like I had simply split code away from a file once it had crossed some magical too many lines limit. Or something, I don't really know—there was a single 116-lines long script called scaffold.py that handled the HTTP request parsing and the overall HTML document structure build-up + HTTP response generation (latter in a very ugly way). I could recognise a half-hearted attempt to split the view generation from the logic, which unfortunately got somehow swallowed up by later code extensions for previously unforeseen features. Starting to fix something at one place caused a string of bugs to pop up at a completely different place, and the small change subsequently ate up my entire weekend.

Although—honestly—I can't say that it was completely devoid of fun. I mean, it's vanity tinkeringThe kind of tinkering that only affects parts of your IT infrastructure that revolve around your appearance to the outside world—after all, we're talking about a personal website., after all. And the result is pretty neat, I think:

The main content now takes 60% of the entire <article> width to the left, while the sidenote area takes 36% of the width to the right. The sidenotes are simply <small> elements, styled with CSS to float to the right. To be honest, it isn't terribly comfortable writing the markup for that in a blog article, but then again, it's probably fine this way, as I would otherwise abuse them.
The main page directly leads to the blog view. There is no point in showing a visitor an almost empty page with three links—that's just pseudo-artistic.
The gallery's CSS has been fixed to scale the image such that it uses the entire available screen space (similar to CSS's background-size: cover property—except that it's not a background).
The principal font has been changed to the more elegant and open Linux Libertine, with Georgia as a fallback font.

Looking at the diff of the commit that introduced the change, it feels like git futilely tried to make sense of the mess, only to resort to follow my exampleYou know, the head-shaking, fist-rising and curse-whispering.: In the cgi directory, the diff (with roughly 360 lines removed and 450 lines added) shows 9 unchanged non-whitespace lines, of which all are keywords like try: or else:, or import os.

showing off

I consider text lines at around 35 ems to be the most comfortable to read, while code should at least have a width of 80 characters. By the way, this is a sidethought, i.e. a sidenote without a specific numbered anchor in the text. Cool, eh? As noted above, the goal of this layout is also to solve the old issue of text lines generally being too long, while other elements like code blocks tend to become pretty useless if jammed into too little space. This can now easily be achieved by assigning the fullwidth class to any element in an article, and it will take up the entire 100% of the width. This is often useful for showing a graphics that requires a high enough resolution, or just for showing an image with added emphasis, like the photo of this lovely cat:

Of course it is not mandatory to display a picture in full-width— displaying a regular picture accompanied with a caption text to the side is now possible, too:

Shot from one of our barbecue evenings at the EPFL (model: Nikon D5100, focal length: 35mm, exposure: 15s, aperture: f1.8, flash: no, ISO: 100)

Another element that is best viewed in full width is a <pre> code block (note that this is a diversion from Tufte CSS, where code is kept within the boundary of normal text—I just thought that this doesn't make any sense); as an example, the CSS part that allowed me to properly center and zoom the picture in this website's gallery view:

article {
    width: 100%;
    height: calc(100% - 2.9rem); /* gallery navigation takes 2.9 rems */
}
article a {
    cursor: -webkit-zoom-in;
    cursor: zoom-in;
    display: block;
    position: relative;
    height: 100%;
    width: 100%;
    overflow: hidden;
}
article a img {
    /* http://stackoverflow.com/a/19414020/3673974 */
    position: absolute;
    top: -100%;
    bottom: -100%;
    left: -100%;
    right: -100%;
    margin: auto;
    min-height: 100%;
    min-width: 100%;
}

Lastly (and obviously) I learned a few things about the spacing around em-dashes (there is none), the proper use of markup tags introduced with HTML 5 (<article>, <header>, etc.), or just how to properly organise my code such that it doesn't end up as this pile of mess I had encountered on Friday evening.

And to explain my absence over the past 14 months: I had initially intended to write an article about how to set up an ALIX 2d3 with Debian for using as a home network router (similar to my QEMU/network/nftables guides), but the thing grew and grew and was never really finished, yet it did a remarkably good job at keeping me from writing any other new articles"If you work on a blog article, you might as well work on me!", until I concluded that the information might be better kept in my private little wiki that I have started to build up recently.

But that's for a different story.

As for the future: I have noticed that I prefer writing code over writing about code, so perhaps the focus of my blog will lie a little more on non-technical stuff (like politics—the world currently is quite a mess, honestly), or the more high-level, organisational aspects of technology (like "When will applications stop littering my home directory with dotfiles and adhere to the XDG base directory specifications?")—I have no idea how it will work out, but we'll (hopefully) see in the next few weeks.

Waiter! There's a VLA in my C!

2015-11-15T16:50:00+0100

Real world ayekat has chosen to pass on some of his knowledge to the microtechnics and electricity people on his daily living site. Mostly this consists of annoying some poor students with elaborated in-depth facts about the beauty of Unix and the C programming language.

While it might sound like a rather doable job, one never ceases to learn new things. Or as for this matter: one never ceases to encounter the ugly sides in the things one so loves and cherishes.

In my case it was a simple sequence of code that I thought would never compile — and the moment it did, I feared I would not be on good terms with it:

        int n;
        scanf("%d", &n);
        int array[n];

-Wall!

-Wextra! -pedantic! man gcc! -WHowCanThisNotThrowAWarning

Variable Length Arrays

What is happening? Obviously the array is allocated at runtime, yet it doesn't seem to require a manual free afterwards. And indeed, a quick test program confirmed my assumption: it is allocated on the stack.

That doesn't seem to be that much of black magick. After all, alloca — while non-standard — is also used to allocate memory on the stack, and apparently it is just C99 notation for

        int n, *array;

        scanf("%d", &n);
        array = alloca(n * sizeof(int));

So why do I react so allergic to those VLAs?

First, I do program in C99, but I tend to put all variable declarations at the top of the functions. That may be a relic from the C89 times, but for me, this also makes it easier to comprehend what is happening on the stack; I don't like random variable declarations popping up in the middle of the code (and to all people arguing about keeping variable declarations as local as possible: they are local to the function — if you argue with "but for longer functions", modularise your bloody code).

VLAs mess around with my mental picture of what's happening, because a variable declaration depends on executed code (the alloca way makes it clearer, while keeping the same functionality).

As for a technical reason: again, it is allocated on the stack, and the stack has a limited size, and all you can do is sit there and watch and pray… and subsequently enter Damnation:

RETURN VALUE

The alloca() function returns a pointer to the beginning of the allocated space. If the allocation causes stack overflow, program behavior is undefined.
alloca(3)

Undefined.

Try entering 192837465647382910 to the above program and watch the World burn. This is no longer some non-standard, unsafe technique — it is now some standard, unsafe technique. And to make matters worse, the abovementioned students use a two-dimensional VLA to store image data on the stack.

What could possibly go wrong…

The Rabbit Hole

As stated above, the students use two-dimensional arrays. So while messing around with those VLAs, I stumbled over another peculiarity in the C language.

void do_something(int width, int height, int **array);

// ...

        int array[w][h];
        do_something(w, h, array);

Now let's see what interesting things gcc has to say about this:

warning: passing argument 3 of ‘do_something’ from incompatible pointer type [-Wincompatible-pointer-types]
  do_something(w, h, array);
                     ^
note: expected ‘int **’ but argument is of type ‘int (*)[(sizetype)(a)]’
 void do_something(int width, int height, int **array);
      ^

So apparently a 2D array is not the same thing as **, fine — but what in the name of the Seven Hells is an int (*)[(sizetype)(a)]?

Multi-Dimensional Arrays

This might seem pretty logic, but I've nevertheless managed to write programs in C for about 8 years without paying proper attention to the subtle differences between pointers and arrays; I knew that if they were statically allocated, the program would handle them differently for sizeof, yet I failed to grasp the exact reason for that — and subsequently also what so radically separates multi-dimensional arrays from pointers.

Yet, if we look at how they are stored in memory, it makes perfect sense: a 2D array is not simply a 1D array of 1D arrays, but rather like one big 1D array that has been cut into n pieces. We can't just put classical "double-pointers" there (think about pointer arithmetics violently blowing up and cities burning and people dying and the air stinking and everything being really, really bad).

So how can we reference a 2D array?

        int a[3][4];
        int **p;
        p = a;      // OK

                    // ... or no, wait - no, actually: BOOM!

Not like this.

Let us have another look at that bizarre int (*)[(sizetype)(a)]. It resembles a function pointer, yet that would make no sense. And it isn't, anyway.

It is simply a notation. The (*) indicates that it is a pointer to something, and the brackets to the right indicate that the "something" is an array of size a. Consequently, a 2D array is like a 1D array of some "units", where each unit has a size of a. And sizetype is simply gcc's way of telling us that it expects a type that can be used to indicate a size — might be an int, a size_t, or something alike.

So we would need to write this instead:

        int a[3][4];
        int (*p)[4];
        p = a;      // yay!

a[3][4] is a array of three 4-element units. (*p)[4] is a pointer to the first element in an array of 4-element units (not an array of arrays) and the notation suddenly starts to make "sense", or whatever you would call that for some weird C notation.

Equally, a function would need to be declared like this:

void do_something(int width, int height, int (*array)[height]);

// or with some "sugar":
void do_something(int width, int height, int array[][height]);

// or with even MORE "sugar":
void do_something(int width, int height, int array[width][height]);

Science! Source: Tech Talk About C99 — What Are Variable Length Arrays?

Anarcho DHCP

2015-11-09T14:37:00+0100

There is a non-negligible amount of things that you learn by mistake. And if you know me, I make a lot of mistakes. Ergo, I learn a lot of things — even if they mostly have no useful purpose.

Yet, this is useful: Even if you have assigned an IP address to your WiFi interface that is different from the subnet you've joined, if dnsmasq is running, you might still give a DHCP lease to any other device that enters the network.

I already knew that from wired networks, yet I hadn't thought that the same "accident" could happen in wireless networks. Not until I heard my lovely girlfriend's voice say something like:

I don't have internet!

Instead of the usual 192.168.2.0/24, she had been assigned an address in the 172.16.12.0/24 subnet — which happens to be my laptop's subnet for sharing internet over WiFi/hostapd.

However, hostapd hadn't been running on my laptop, and she was connected to the right AP anyway. Yet somehow my laptop's dnsmasq must have got the DHCP discover message anyway, and answered to it faster than the router.

From that, I assumed that no matter if you're on a wired or wireless connection, rogue DHCP servers may be a problem. When I stated my discovery among my collegues, the reaction I got was something along the way of "of course — and it's not the first time your computer does that — fix your fucking networking!", followed by:

flor> ayekat: si j'étais l'opérateur réseau je ban ton compte directement si tu
      fais ça

Interestingly, even if the AP runs a DHCP server on it's own, it broadcasts the DHCP discover packet. This might seem weird at the first glance, but then again hostapd and dnsmasq are two different processes (that may as well run on two different devices), and things still work as expected if all is set up correctly.

If things are set up correctly.

For my part, I have decided not to put any IP addresses on my ethernet and WiFi interfaces that may be subject to accidental DHCP lease offers by dnsmasq, put a static address on my ethernet interface (for emergencies), and assign IP addresses manually to an interface if necessary — after all, it doesn't happen each day that I share my internet connection.

QEMU/networkd/nftables

2015-11-07T11:35:00+0100

Remember this post when I was all enthusiastic about using a local server with QEMU and some apparently simple network setup using systemd-networkd? Remember when you were like Oh!, I've gotta try this out! But wait — what about forwarding ports?

There was supposed to be a fourth section going something like "nftables — Now, let's add some port-forwarding!". Later, it became something like "Oh, it's a little more complicated — let's just complete it once I've got all the informations!". It then became "God, this is awfully fucked up", before finally becoming "Oh, it's actually pretty logic - I'm just really dumb."

Well, welcome to that follow-up.

systemd-networkd

As stated, I decided to use nftables for port-forwarding. Yet packets didn't seem to even enter my prerouting chain, which was rather unfortunate, as it is there where things like "this packet will get redirected to this host" should happen.

After asking on #netfilter and getting friendly troubleshooting advice (and involuntary invoking a pro-systemd vs. anti-systemd debate when I stated that the IPMasquerade option in my systemd-networkd setup might perhaps have had some bad influence on the matter), I decided to tear my network and its configuration down and restart from scratch (while people on #netfilter were ripping each other's head off about the purpose of systemd-networkd and its apparent non-conformity to some RFC).

When everything was back up, my setup looked almost the same, except for the missing IPMasquerade:

# /etc/systemd/network/qemu0.netdev

[NetDev]
Name=qemu0
Kind=bridge

# /etc/systemd/network/qemu0.network

[Match]
Name=qemu0

[Network]
172.16.10.1/24
IPForward=yes

nftables: masquerade

nftables is an effort to replace the existing netfilter/iptables framework. Although it is not production-ready, I decided to bleed and replace my existing iptables setup with nftables, since in the long run, learning the Ins and Outs of a filtering system that has a hard-to-parse syntax and that will soon get replaced didn't seem very lucrative to me.

Hence, the masquerading takes place in /etc/nftables.conf:

#!/usr/sbin/nft -f

table ip nat {
  chain prerouting {
    type nat hook prerouting priority 0;
  }

  chain postrouting {
    type nat hook postrouting priority 0;
    ip saddr 172.16.10.1/24 oifname {"wlp3s0", "enp0s25"} masquerade
  }
}

Looks sane enough, and works.

nftables: port-forwarding

Packet filtering is a topic that could fill a new article on its own — I usually stick to this mental picture (I may stand corrected):

Packet enters pre-routing chain, where we may alter it before it enters routing
if Packet is for the system itself:
- Packet enters input chain, where we decide whether we let it in
- Packet ends/starts here
- Packet enters output chain, where we decide whether we let it out
else if Packet is for a different system:
- Packet enters forward chain, where we can influence the routing
Packet enters post-routing chain, where we may alter it before it leaves the system

Alright! So let's add some classic rule: one that forwards connections to the host's port 2222 to the guest's port 22:

#!/usr/sbin/nft -f

table ip nat {
  chain prerouting {
    type nat hook prerouting priority 0;
    tcp dport 2222 dnat 172.16.10.2:22
  }

  chain postrouting {
    type nat hook postrouting priority 0;
    ip saddr 172.16.10.1/24 oifname {"wlp3s0", "enp0s25"} masquerade
  }
}

The Troubleshooting

Although everything looks fine, we've got the above-stated problem: Packets don't enter nftables' prerouting chain. Packets not entering the prerouting chain apparently ate an entire Friday afternoon away from me.

Let's prevent that for you:

As already stated in the original article, systemd-networkd provides the options IPForward and IPMasquerade. The tricky thing about IPForward is that if you don't specify it, it will set the sysctl option net.ipv4.conf.<interface>.forwarding to 0 — regardless of whether you set net.ipv4.ip_forward to 1 in /etc/sysctl.d/99-ip_forward.conf. But we already knew that.

The other tricky thing is that this is actually wrong:

Also, IPMasquerade replaces iptables/nftables rules for masquerading in this simple case.

This is incorrect because systemd will not handle IP masquerading on its own, but use iptables. It's actually good, because it doesn't reinvent the wheel, but delegate the filtering to an appropriate tool: the firewall. It also makes sense that if we remove the option, no iptables rules for masquerading will be added.

Yet … in my nftables setup, while packets seemed to enter the postrouting chain, they refused to do the same for prerouting.

At this point, I reread all documentation I could find on creating a NAT with port forwarding for nftables, before stumbling over this:

You cannot use iptables and nft to perform NAT at the same time. So make sure that the iptable_nat module is unloaded:
rmmod iptable_nat

Friday evening. Weekend. It's all fine.

QEMU/networkd

2015-11-06T17:45:00+0100

As I start to get more and more into touch with "the industry", I realise that it's pretty much impossible to just walk in with your laptop and start hacking straight away. Mostly, you'll end up in an environment where they're working with some "solution" X, where X isn't just any ordinary application, but some intrusive monster that will change at least 50% of the configuration that handles the fundamental behaviour of your system.

That is rarely nice.

Luckily, things can be virtualised, and shit doesn't need to be accepted like that. But setting up a proper virtualisation solution (that is not server-grade, like Xen) is a little less trivial. There is VirtualBox, but it's almost become one of those intrusive techs on its own, and not very configuration friendly. It doesn't play nice with KVM either. And it's been oracled.

But fortunately, there is QEMU, which is purely CLI-based, and it integrates with KVM, which is nice. It needs some scripting, though, since no one in their right mind would type a QEMU command manually. But that is usually trivially done in an evening full of cursing and cups of tea.

But then, alas! Networking!

systemd-networkd

systemd is quite a controversial piece of software. Basically it was written to replace formerly existing init/rc solutions. Then, it also replaced login. And docker. And incorporated udev. And dbus. And an HTTP server. That also serves QR code. For login.

Then, it added network handling. Which is cool. Because network handling is usually a pain in the ass. ifupdown, netcfg/netctl, NetworkManager, Wicd and alike have not just emerged for no reason — but they may not be available on all platforms, or be rather GUI-centric. And mostly they feel more like an overlay to the underlying network commands (ip, dhcpcd), than actual network managers.

At this point, systemd-networkd does quite a good job at handling the network, has a text-based configuration similar to ifupdown, and integrates rather well with the system (I mean … it's basically part of the system). The documentation for systemd-networkd — and for systemd in general — is respectable, and it will work on most systems, given that all noteworthy distributions apart from Gentoo have switched to systemd.

And of course, it can set up bridges:

# /etc/systemd/network/qemu0.netdev

[NetDev]
Name=qemu0
Kind=bridge

# /etc/systemd/network/qemu0.network

[Match]
Name=qemu0

[Network]
Address=172.16.10.1/24
IPForward=yes
IPMasquerade=yes

In order to allow QEMU to use qemu0 as the bridge, we will need to add an entry into /etc/qemu/bridge.conf:

allow qemu0

The caveat — and also the cool thing — is on the last two lines in the .network file: sysctl's IP packet forwarding settings (net.ipv4.ip_forward) are ignored and instead handled by systemd-network via net.ipv4.conf.<interface>.forwarding. Took me some time figuring this out. This also allows you to remove that weird /etc/sysctl.d/99-ip_forward.conf.

Also, IPMasquerade replaces iptables/nftables rules for masquerading in this simple case.

# systemctl start systemd-networkd

dnsmasq

The VM will be hooked up to the bridge generated above, and you'll need to serve it an IP address:

# /etc/dnsmasq.conf

dhcp-range=172.12.10.1,172.16.10.5,12h
dhcp-host=52:54:00:12:34:56,172.16.10.2

We know the host's IP address on the qemu0 interface, so we choose an appropriate IP address range for DHCP; and we can anticipate the IP address it will get, since we know the guest's network interface MAC address (see below).

# systemctl start dnsmasq

QEMU

QEMU is really nice. If you're used to point-and-click and it-just-werks, you might have a tough time, but if you know what you want, it's your best friend on a desktop:

qemu-system-x86_64 -enable-kvm -cpu host -smp 4 \
                   -m 4096 \
                   -net nic,model=virtio,macaddr=52:54:00:12:34:56 \
                   -net bridge,br=qemu0 \
                   -drive file=disk.raw,if=virtio,format=raw

-enable-kvm -cpu host -smp 4 will use KVM (you know, to make the performance bearable), use the host's CPU model, and assign it 4 cores.

-m 4096 will assign the VM 4 GiB of memory.

-net nic,model=virtio,macaddr=52:54:00:12:34:56 will create a new Network Interface Card through the virtio interface and 52:54:00:12:34:56 as MAC address.

-net bridge,br=qemu0 will attach said network interface to the network bridge created above.

-drive file=disk.raw,if=virtio,format=raw will attach the raw disk image disk.raw as a virtual disk device through the virtio interface.

You may want to script this.

With the abovementioned configuration, you should now be able to request an IP address via DHCP from the host, which will act as a gateway with NAT to the guest inside the VM.

Passing -nographic -vnc :0 to QEMU will make the VM run without graphical output and a VNC attached to display :0. With an SSH server on the guest, you now basically have a server, in a local network, on your machine.

Yay!

Dynamic Waltz

2015-09-05T23:13:00+0200

After my PHP-driven, bloated-as-hell nightmare of an insecure website succumbed to the evolution of sanity and died a long-awaited death, the choices laid out before me were:

to use a web development framework for dynamically generated websites: generally I tend to suffer from the NIH syndrome and I tend to dislike frameworks for imposing some arbitrary software design on me (albeit sometimes well-done). I also started to dislike the idea of having to launch a server-side process just for the sake of serving a single page.
to use a web development framework for statically generated websites: comment sections would either still need to pass through some dynamic part (which renders the whole "let's go static!" approach pointless) or be handled by some third party service like disqus (which pretty much goes against my no-pointless-javascript-in-the-Web attitude — I guess iframes have somehow gone out of fashion these days).
Also, frameworks.

In the end I came to a "Fuck it All" conclusion, abandoned website building for good, and turned the ayekat.ch web experience into what resembles the current about section. The time for expressing my narcism in pretty CSS designs (and re-designs every 6 months) was spent on more useful things, and life has gone on surprisingly well for the past year.

However, after an attack of insanity, I decided to make a pretty website again. Also I stumbled over

python

… yes, I can already hear you whimper

oh no, here it goes again…

There truly is some sort of hype around this language, and I kept away from it for the mere reason that most people advocating it used arguments comparing it with C. I may also just have hanged around in the wrong part of the web, though.

Anyway, I started using it, and I concluded that

the language is very nice. It is definitively better than PHP (but that isn't a huge accomplishment), but also trumps shell scripts when the task becomes a little more complex than a naive automation. It allows you to do many things without much fighting with the syntax, and provides lots of functionality straight out of the box without any need for installing additional OS packages; however
its module system is pretty archaic. It looks like it might have worked well in the past, but now it seems more like some mechanisms that were hacked together to make the whole thing work. Also, all imports seem to be resolved at the working directory of the top level file, which pretty much throws modularity out of the window: using if __name__ == '__main__' checks (which I find pretty handy for developing a certain component) becomes utterly useless if used in a module; also, at any given time a module down the line may break if it tries to import a file that happens to have the same name as a file in your own project.
… although, I have to admit that C's #include statements are not the pinnacle of intelligence either (and " and <> may not always behave very intuitively).

But we are not forced to modularise our code (in the sense of python) for a simple website, are we? Yet the nagging question remained: static or dynamic? Rationality pulled me towards static website building (why would I need a comment section, anyway? the last comment I got was an XSS attempt anyway), so

$ sloccount cgi

SLOC    Directory       SLOC-by-Language (Sorted)
366     cgi             python=366

CGI.

Because. Fuck. It. I had fun. It's my personal emotional link to the good old days when the Web was not shit yet. It compiles markdown to HTML, handles requests for static files (images, CSS, PDFs) and defines the HTTP header on its own. If it wasn't a launched process for each request, I'd be almost proud.

Then again, it's a bloody website. Why are we even talking?

waltz

Named after both a dance and an actor coming from a particular country to the East of mine, I am on a new server. We all are. "We" is a small group of people who have all been very excited by the delightful, frequent and unannounced server downtimes and I/O issues at our former host filemedia.de, such that we collectively decided to try out something less exciting by default.

Age, I guess. It's not good for the heart.

Since I'm not known for blogging (or if, then not for blogging particularly well), the direction this blog will take is yet unclear. Perhaps I'll tell you a lot about exciting technical things I've recently discovered. Perhaps I will also just post failed attempts at poetry. Or jokes. I might tell you about the weather in Switzerland, and how the political and economical relationship to the European Union looks like to a totally uninformed cave dweller. I might just post photographs of cats because I happen to have taken a shitton of them. Or maybe just remain dead. Perhaps I'll discover that, statistically, this place is more dead than Sun's open source projects after Oracle happened, and just archive and remove the published nonsense. Some older posts may appear, perhaps the future event prediction timeline based on dystopical books (and how the World has changed ever since I posted that back in 2009), if I happen to find it somewhere. Or maybe I'll make a new one.

Until then, I wish you a very nice rest of the summer.

ん	わ	ら	や	ま	は	な	た	さ	か	あ
		り		み	ひ	に	ち	し	き	い
		る	ゆ	む	ふ	ぬ	つ	す	く	う
		れ		め	へ	ね	て	せ	け	え
	を	ろ	よ	も	ほ	の	と	そ	こ	お

ん	わ	ら	や	ま	は	な	た	さ	か	あ
		り		み	ひ	に	ち	し	き	い
		る	ゆ	む	ふ	ぬ	つ	す	く	う
		れ		め	へ	ね	て	せ	け	え
	を	ろ	よ	も	ほ	の	と	そ	こ	お

ん	わ	ら	や	ま	は	な	た	さ	か	あ
		り		み	ひ	に	ち	し	き	い
		る	ゆ	む	ふ	ぬ	つ	す	く	う
		れ		め	へ	ね	て	せ	け	え
	を	ろ	よ	も	ほ	の	と	そ	こ	お