Digressive Developer

I almost made a 10x optimization

Sun, 27 Oct 2024

Working on a high-performance Web accelerator

I have been working on Varnish Cache for a decade and from day one former colleagues of mine persuaded themselves that from then on I was dealing with arcane performance topics on a daily basis so explaining that it was not the case puzzled them for a while.

As it turns out, Varnish Cache is a very small project, and yet there is plenty of variety in the project. Working on Varnish means that you can work on:

networking sockets
networking protocols
multi-threading
IPC of various kinds, including shared memory
terminal-based interfaces (TUI applications)
a programming language compiler (VCL)
a programming language interpreter (VTC)
a homemade test framework
libraries
gzip

And that is only what comes off the top of my head. There are so many things going on for such a small project that amazingly, after a decade, there are still several parts of the code base I never worked on, some of them I never even read.

I eventually worked on a performance bottleneck that almost resulted in a 10x speed increase, significantly reducing the time it would take me to perform certain tasks. It also turned out to be very similar to the first optimization I remember making in a professional setting.

Working on a low-performance Web application

Almost two decades ago, I ported a PHP application to Java, because I had joined a company doing everything in Java, and nobody was competent to work on this small internal PHP application. I was tasked to rewrite it from scratch in Java, and that would serve as a training vector to familiarize myself with the stack I would use next, on a regular Java project.

The result was a slow, slow, slow mess of an application universally hated by everyone who had to use it in the company, which was pretty much everyone. There were several reasons why it was so slow, and this was the opportunity to discover the art of profiling.

For the first problem I solved, I decided to blame Java for being magic and not warning me about the dangers of my ways.

The exception to the simplicity rule

Back then, Java 5 was brand new, and it came with “amazing” features like for-each loops, generics, and other quality-of-life features. The language was verbose and complicated, but it was also mostly unambiguous. It was almost always possible to figure immediately what was what in a code snippet.

Despite being a complicated language, as far as I could observe, javac had always been a simple piece of the tool chain. It ingested Java code and turned it into Java bytecode in such a way that most of the bytecode I have seen had a 1-1 correspondence to its Java code.

The beauty of javac’s straightforward translation of the Java code, without extensive optimization passes, was that the runtime would instead be in charge of optimizations taking advantage of the host system capabilities. In other words, javac could remain simple because the complexity was pushed at the edge. My assumption (that I’m not attempting to verify because at this point I no longer care) is that aggressive optimizations at compile time could have prevented better optimizations at run time, but I digress.

Isolating the performance hot spot

For the first performance problem I solved, I wrote a reproducer for the naive code that ended up eating a significant amount of cycles:

public class Concat {
	public static void main(String... args) {
		String cat = "";
		String sep = "";
		for (String arg: args) {
			cat += sep + arg;
			sep = ", ";
		}
		System.out.println(cat);
	}
}

This program loops over its command line arguments to build a comma-separated representation of them. It’s also branch-less, so the separator is handled as a string instead of, for example, a boolean.

Let’s compile it to bytecode for Java 5:

$ javac -source 5 -target 5 Concat.java
warning: [options] bootstrap class path not set in conjunction with -source 5
error: Source option 5 is no longer supported. Use 8 or later.
error: Target option 5 is no longer supported. Use 8 or later.

Unfortunately this is no longer possible with OpenJDK 21, so let’s try again with Java 8 as suggested by the error message:

$ javac -source 8 -target 8 Concat.java
warning: [options] bootstrap class path not set in conjunction with -source 8
warning: [options] source value 8 is obsolete and will be removed in a future release
warning: [options] target value 8 is obsolete and will be removed in a future release
warning: [options] To suppress warnings about obsolete options, use -Xlint:-options.
4 warnings

And let’s just make sure that it works as advertised:

$ javac Concat.java
$ java Concat a b c
a, b, c

The curious case of the magic bytecode

Looking at the loop, the problem was not obvious. It looked straightforward and nothing hinted at ill-advised operations in a tight loop. I eventually decided to look at the bytecode because that made me look competent, and at least busy and actively working on the problem.

It’s still trivial today with OpenJDK:

$ javap -c Concat.class
Compiled from "Concat.java"
public class Concat {
  public Concat();
    Code:
       0: aload_0
       // Method Object."<init>":()V
       1: invokespecial #1
       4: return

  public static void main(String...);
    Code:
       0: ldc           #7                  // String
       2: astore_1
       3: ldc           #7                  // String
       5: astore_2
       6: aload_0
       7: astore_3
       8: aload_3
       9: arraylength
      10: istore        4
      12: iconst_0
      13: istore        5
      15: iload         5
      17: iload         4
      19: if_icmpge     61
      22: aload_3
      23: iload         5
      25: aaload
      26: astore        6
      // class StringBuilder
      28: new           #9
      31: dup
      // Method StringBuilder."<init>":()V
      32: invokespecial #11
      35: aload_1
      // Method StringBuilder.append:(LString;)LStringBuilder;
      40: invokevirtual #12
      39: aload_2
      // Method StringBuilder.append:(LString;)LStringBuilder;
      40: invokevirtual #12
      43: aload         6
      // Method StringBuilder.append:(LString;)LStringBuilder;
      45: invokevirtual #12
      // Method StringBuilder.toString:()LString;
      48: invokevirtual #16
      51: astore_1
      52: ldc           #20                 // String ,
      54: astore_2
      55: iinc          5, 1
      58: goto          15
      // Field System.out:Ljava/io/PrintStream;
      61: getstatic     #22
      64: aload_1
      // Method java/io/PrintStream.println:(LString;)V
      65: invokevirtual #28
      68: return
}

I was really taken by surprise when I realized that the bytecode was not at all looking like a 1-1 translation of the Java code. My loop (much larger back then) was full of objects and instructions that appeared out of nowhere. And I really hate magic. I can’t reason about magic, I need my code to do what it’s told and not do anything that wasn’t explicitly said, otherwise I can’t build a reliable mental model.

To help myself reason about this bytecode, I manually decompiled it:

public class Concat {
	public static void main(String... args) {
		String cat = "";
		String sep = "";
		for (String arg: args) {
			StringBuilder tmp = new StringBuilder();
			tmp.append(cat);
			tmp.append(sep);
			tmp.append(arg);
			cat = tmp.toString();
			sep = ", ";
		}
		System.out.println(cat);
	}
}

And from then on the problem was obvious. The ‘+’ operator for the String class was backed by a StringBuilder. I’m saying “was” here because OpenJDK uses a different backend for string concatenation in Java 21. Because javac makes a straightforward translation of the Java code, we end up with a lot of churn inside the tight loop, creating and discarding one StringBuilder per iteration.

In my performance problem, I remember seeing several string builders being created and deleted in the same loop, as its body was doing a lot more. This was the occasion to discover that Java strings are immutable and what it means in practice. One conclusion I drew was that the runtime was not able to optimize this case at all. Even with other mechanisms like generational garbage collectors, these short-lived string builders slowed the application down to a crawl. I have no idea what OpenJDK 21 is capable of these days, and I will probably not find out.

Time for performance wizardry

Seeing what the ‘+’ and ‘+=’ operators were doing to the generated bytecode the solution was rather simple. If string concatenation is backed by a string builder, we can avoid the needless churn by moving the string builder out of the loop:

public class Concat {
	public static void main(String... args) {
		StringBuilder cat = new StringBuilder();
		String sep = "";
		for (String arg: args) {
			cat.append(sep);
			cat.append(arg);
			sep = ", ";
		}
		System.out.println(cat);
	}
}

And with a similar change in the code base, this hot spot no longer registered in the profiler. The rest of the hot spots had similar churn problems that became obvious after learning to identify them. From then on that pattern was burned into my brain and I would quickly spot it, taking action immediately to prevent it from turning into a bottleneck.

Spotting this pattern in Varnish

Fast forward years later, as of today also a few years ago, I find myself in a delicate position trying to understand a dramatic performance problem observed by a customer. Varnish is performing exceptionally poorly and they can prove it reliably, and repeatedly. It’s so bad that I end up traveling to work on it in person and on site with them.

Every time they run their benchmark, it results in more than 20GB of logs, and it takes me forever to collect data. I was using my own personal project varnishplot to automate the creation of latency graphs. It would take me forever to conclude that Varnish was reporting very low processing times. Being very attentive to the pitfall I first ran into as a Java developer, I had already noticed the likelihood of a similar occurrences in Varnish logs two years prior, when I initiated my work on varnishplot, and on the spot I decided to have a shot at it.

The Varnish Shared Memory Log

To summarize briefly how logs work in Varnish, we have on one end varnishd creating a memory-mapped file per shared memory segment. On the other end we have programs that can map the same files in memory. Addressing them via the file system is how they become shared memory segments. This particular segment I am interested in is called VSL for Varnish Shared Log, also referred to as shared memory log, or shmlog.

The VSL segment is itself split into several parts inside which actual logs are written. Logs have a structured binary representation, and varnishd will cycle through segments as it writes to them. Inside varnishd, there can be a large amount of threads logging concurrently, so long-running tasks log in a local buffer and ideally flush everything at once at the end of the task.

The circular nature of the live VSL means that consumers like varnishlog or varnishncsa may be overrun by varnishd. Grouping transactions increases the risk of being overtaken. As a result VSL consumers may make copies of log records to keep them available until a complete transaction or transaction group is ready to be processed. In terms of coordination, varnishd threads use a mutex to publish log records, and consumers synchronize in wait-free fashion with memory barriers.

Log consumers can also feed from regular files, written by varnishlog -w. This offers the ability to perform queries and filtering on structured binary logs at rest. This is essentially how I was processing logs from benchmark runs to plot latency histograms.

That should be it for the short digression, back to the problem now.

Pushing the cursor

The logs consumption itself is abstracted behind a cursor interface (VSLC or Varnish Shared Log Cursor). There is a VSM (Varnish Shared Memory) cursor for live logs and a file cursor for VSL files written down to disk. The churn pattern described with the Java anecdote does not appear as clearly as in the Java bytecode, but after asking the author of the current VSL implementation a dozen times when I started having my doubts two years before my benchmark woes I was fairly certain that the same scenario was happening.

The sequence of events was different, but is actually quite similar:

the file cursor opens a file
it reads data in a local buffer
it parses log records
- copies are made to outlive the buffer
once a transaction group is processed, copies are freed

The thing that was bothering me was the need to allocate copies at rest, so I made the hypothesis that regular files could be mapped in memory to work with them directly in place.

In other words, this would take the best of both worlds, working “live” on a memory buffer like the VSM cursor, without the fear of being overtaken like the file cursor. As an added bonus, the code for the mmap cursor ended up being the simplest of the three. A simple design often is a good foundation for good performance.

Almost 10x

With the new mmap cursor I was able to iterate much faster on large dumps of logs from benchmark runs. On my workstation the speed to process logs jumped to 6x faster for transaction groups and 9x for raw log records.

Faster iterations meant that I processed benchmark results more efficiently from this point on, and soon after the answer became obvious: all client requests came from a very small set of file descriptors.

There was a proxy in front of Varnish that would serve several purposes, but it was configured to open two connections to Varnish per client IP address. In this very synthetic benchmark, that proxy was funneling thousands of HTTP requests per second over a handful of TCP connections. Lifting that limit solved the problem and that was the end of a wild goose chase.

Soon after I submitted my change and all was well that ended well. I was hoping that reflecting on this would help me understand another ongoing performance problem, but no luck so far. At this rate I might write something up about the current problem in 2028…

Initial verdict on Hare safety

Mon, 20 May 2024

Note: this post was written in early March and supposed to be published then. It was extended with further observations after the closing words to make up for this unfortunate oversight.

Managing resources is a solved problem

In my tsess introduction I rambled on Hare and how it fares against C, Rust and Go. The answer was pretty well and tsess-0.2 is good occasion to revisit this.

As the project probably already covers 120% of my needs, there will be little to add besides small features I have in mind that only add convenience to a utility I already find more than good enough on a daily basis. These features will be a good excuse to explore a little more of Hare. That excuse has been the main driver for the initial attempt to write tsess in Rust and the current success in Hare.

As stated previously Valgrind found leaks in tsess so safety in Hare is already a failure. After all, Rust solved this with its ownership model enforced by its borrow checker. But the same borrow checker reached its limit when I tried to fit a single allocation model for the configuration file with parsing in place. This was a different kind of failure, either on Rust for not being able to get a &&str from a Box<str> or on me for not figuring out how.

Needless to say, I wouldn’t have this problem in C (although I would have a different problem called null-terminated strings) and I didn’t have this problem with Hare. But nevertheless, I leaked resources.

Revisiting tsess leaks

Since I’m too lazy to either go back in Git history before the commits that plugged leaks, restoring the tool chain I was using then, I will simply revert the bug fixes to revisit the leaks. Fortunately, so far, I made leaks easy to find from commit messages alone.

$ git revert --no-commit --no-edit $(git log --grep Plug --format=%H)
Auto-merging bin/tsess/config/env.ha

Interesting fact, so far I only fixed leaks in env.ha, and what’s is not visible is that two leaks were plugged.

Now, let’s ask Valgrind about problems, but let’s do it as it used to, from a statically linked tsess.

After running the following command:

valgrind --tool=memcheck --leak-check=full --track-fds=all ./tsess list

I get the following report from Valgrind, after cleaning it up:

FILE DESCRIPTORS: 4 open (3 std) at exit.
Open file descriptor 3: /home/dridi/.config/tsess/example.tsess
   at 0x8006BB1: ??? (in /home/dridi/src/tsess/tsess)
   by 0x8003721: ??? (in /home/dridi/src/tsess/tsess)
   by 0x80166FD: ??? (in /home/dridi/src/tsess/tsess)
   by 0x801657C: ??? (in /home/dridi/src/tsess/tsess)
   by 0x8011C8E: ??? (in /home/dridi/src/tsess/tsess)
   by 0x8012ED9: ??? (in /home/dridi/src/tsess/tsess)
   by 0x805421A: ??? (in /home/dridi/src/tsess/tsess)
   by 0x8053928: ??? (in /home/dridi/src/tsess/tsess)
   by 0x80531E2: ??? (in /home/dridi/src/tsess/tsess)
   by 0x80507EA: ??? (in /home/dridi/src/tsess/tsess)
   by 0x806241E: ??? (in /home/dridi/src/tsess/tsess)

Open file descriptor 2: /dev/pts/6
   <inherited from parent>

Open file descriptor 1: /dev/pts/6
   <inherited from parent>

Open file descriptor 0: /dev/pts/6
   <inherited from parent>


HEAP SUMMARY:
    in use at exit: 0 bytes in 0 blocks
  total heap usage: 0 allocs, 0 frees, 0 bytes allocated

All heap blocks were freed -- no leaks are possible

For lists of detected and suppressed errors, rerun with: -s
ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

Despite the lack of a useful stack trace, it was easy to find the source of the leak since there is only one place in the code that opens files. The bug fix is a one-liner:

$ git show 8d525d563f13c3ce995471e23fa1078bb90fe8e9
commit 8d525d563f13c3ce995471e23fa1078bb90fe8e9

    config: Plug file leak

    Spotted by Valgrind.

diff --git a/bin/tsess/config/env.ha b/bin/tsess/config/env.ha
index cf21695..7449318 100644
--- a/bin/tsess/config/env.ha
+++ b/bin/tsess/config/env.ha
@@ -74,6 +74,7 @@ fn file_size(h: io::handle) (size | io::error) = {

 fn file_load(path: str) ([]u8 | error) = {
        let file = os::open(path)?;
+       defer io::close(file)!;
        if (file_size(file)? == 0) {
                return errors::unsupported: io::error;
        };

With the only file leak plugged, it’s time to look at memory leaks. I know for a fact that there are more than zero allocations, and I also know for a fact that Valgrind could find them if tsess was dynamically linked.

Memory leaks

Since version 0.2, tsess is dynamically linked by default. It is however still possible to statically link it at build time:

$ make tsess_LDFLAGS=
  CCLD     gen_cmd
  GEN      bin/tsess/cli/gen/cmd.ha
  CCLD     gen_env
  GEN      bin/tsess/config/gen/env_vars.ha
  CCLD     gen_prop
  GEN      bin/tsess/config/gen/prop_defs.ha
  CCLD     tsess
  GEN      man/tsess.1
  GEN      man/tsess.5
$ file * | grep ELF
gen_cmd:        ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, with debug_info, not stripped
gen_env:        ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, with debug_info, not stripped
gen_prop:       ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, with debug_info, not stripped
tsess:          ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, with debug_info, not stripped

Side note, the invocation eventually changed to make tsess_LIBS=.

My rationale to dynamically link by default is that libc may provide certain services based on the system configuration. For example, a system user or group lookup could be done from /etc/passwd or /etc/group, or from an NSS module.

This can only work if the Hare runtime is implemented with fall-backs to libc functions. There is at least one service from libc that will be used, and it’s the memory allocator. This probably means that we can also use a different malloc implementation, for example linking to jemalloc for its profiling capabilities among other things, but I digress.

So, once I dynamically link tsess, let’s look at memory leaks. After running the same Valgrind command I get the following suggestion:

Reachable blocks (those to which a pointer was found) are not shown.
To see them, rerun with: --leak-check=full --show-leak-kinds=all

Since I was able to confirm that the file leak was plugged, I ran this command instead:

valgrind --tool=memcheck --leak-check=full --show-leak-kinds=all ./tsess list

And I get the following report, once again cleaned up to keep the essentials:

HEAP SUMMARY:
    in use at exit: 16,472 bytes in 5 blocks
  total heap usage: 102 allocs, 97 frees, 62,379 bytes allocated

12 bytes in 1 blocks are indirectly lost in loss record 1 of 5
   at 0x484280F: malloc (vg_replace_malloc.c:442)
   by 0x401F22: ??? (in /home/dridi/src/tsess/tsess)
   by 0x41844D: ??? (in /home/dridi/src/tsess/tsess)
   by 0x42B9C6: ??? (in /home/dridi/src/tsess/tsess)
   by 0x42B3DA: ??? (in /home/dridi/src/tsess/tsess)
   by 0x430182: ??? (in /home/dridi/src/tsess/tsess)
   by 0x47F907: ??? (in /home/dridi/src/tsess/tsess)
   by 0x47D2D5: ??? (in /home/dridi/src/tsess/tsess)
   by 0x48F631: ??? (in /home/dridi/src/tsess/tsess)
   by 0x408738: ??? (in /home/dridi/src/tsess/tsess)
   by 0x4897149: (below main) (libc_start_call_main.h:58)

19 bytes in 1 blocks are still reachable in loss record 2 of 5
   at 0x484280F: malloc (vg_replace_malloc.c:442)
   by 0x401F22: ??? (in /home/dridi/src/tsess/tsess)
   by 0x418A85: ??? (in /home/dridi/src/tsess/tsess)
   by 0x4824E6: ??? (in /home/dridi/src/tsess/tsess)
   by 0x482667: ??? (in /home/dridi/src/tsess/tsess)
   by 0x482794: ??? (in /home/dridi/src/tsess/tsess)
   by 0x408D51: ??? (in /home/dridi/src/tsess/tsess)
   by 0x408733: ??? (in /home/dridi/src/tsess/tsess)
   by 0x4897149: (below main) (libc_start_call_main.h:58)

25 bytes in 1 blocks are still reachable in loss record 3 of 5
   at 0x484280F: malloc (vg_replace_malloc.c:442)
   by 0x401F22: ??? (in /home/dridi/src/tsess/tsess)
   by 0x418A85: ??? (in /home/dridi/src/tsess/tsess)
   by 0x482A67: ??? (in /home/dridi/src/tsess/tsess)
   by 0x408D51: ??? (in /home/dridi/src/tsess/tsess)
   by 0x408733: ??? (in /home/dridi/src/tsess/tsess)
   by 0x4897149: (below main) (libc_start_call_main.h:58)

44 (32 direct, 12 indirect) bytes in 1 blocks are definitely lost in loss record 4 of 5
   at 0x484A074: realloc (vg_replace_malloc.c:1690)
   by 0x401F0B: ??? (in /home/dridi/src/tsess/tsess)
   by 0x402075: ??? (in /home/dridi/src/tsess/tsess)
   by 0x42B3FC: ??? (in /home/dridi/src/tsess/tsess)
   by 0x430182: ??? (in /home/dridi/src/tsess/tsess)
   by 0x47F907: ??? (in /home/dridi/src/tsess/tsess)
   by 0x47D2D5: ??? (in /home/dridi/src/tsess/tsess)
   by 0x48F631: ??? (in /home/dridi/src/tsess/tsess)
   by 0x408738: ??? (in /home/dridi/src/tsess/tsess)
   by 0x4897149: (below main) (libc_start_call_main.h:58)

16,384 bytes in 1 blocks are definitely lost in loss record 5 of 5
   at 0x484280F: malloc (vg_replace_malloc.c:442)
   by 0x401F22: ??? (in /home/dridi/src/tsess/tsess)
   by 0x4602A1: ??? (in /home/dridi/src/tsess/tsess)
   by 0x408D51: ??? (in /home/dridi/src/tsess/tsess)
   by 0x408733: ??? (in /home/dridi/src/tsess/tsess)
   by 0x4897149: (below main) (libc_start_call_main.h:58)

LEAK SUMMARY:
   definitely lost: 16,416 bytes in 2 blocks
   indirectly lost: 12 bytes in 1 blocks
     possibly lost: 0 bytes in 0 blocks
   still reachable: 44 bytes in 2 blocks
        suppressed: 0 bytes in 0 blocks

For lists of detected and suppressed errors, rerun with: -s
ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 0 from 0)

The main reason why I didn’t dive straight away in memory leaks is the uselessness of this report. From the stack trace I can easily infer which lines are inside tsess’ main() function but after that, I wasn’t going to follow all the paths that could lead to an allocation after “that many” function calls for each leak. Especially since that would also mean following paths into the standard library. I’d rather have a computer do this repetitive work for me.

I’m also worried about this 16kB allocation. What did I do so wrong to leak 16kB of memory when this Valgrind experiments loads one configuration file that has a size below 1kB? Did I forget to free a buffer for buffered I/O? That’s highly unlikely since tsess is very conservative with allocations. The reason why the Rust experiment failed was caused by my insistence in having one allocation per configuration file, and working with in-place substrings.

There should be no such buffer.

I was also wondering where the two reachable memory outstanding allocations came from. I was well-aware of one source of allocations that could be held until completion, and it happens to be again in env.ha: the configuration directory is the expansion of ${XDG_CONFIG_HOME}/tsess and this is done during startup by an @init function:

let tsess_dir = "";

@init fn env_init() void = {
        env_args.dir = arg_resolve(env_vars.dir, env_defs.dir);
        env_args.tmux = arg_resolve(env_vars.tmux, env_defs.tmux);
        env_args.tmpdir = arg_resolve(env_vars.tmpdir, env_defs.tmpdir);
        tsess_dir = strings::concat(env_args.dir: str, "/tsess");
};

It could even fall back to an expansion of ${HOME}/.config with the result expanded to ${HOME}/.config/tsess, leading to two reachable outstanding allocations instead of one. But based on the environment in which Valgrind was running, I had an explanation for only one reachable remain in the report.

My initial initial verdict on Hare safety was that it really sucked to have bugs in your code, and I was looking forward to proper debug symbols. Except that at the time I first looked at memory leaks I had proper DWARF support in my Hare tool chain, and I found how to take advantage of it by accident.

Preparing the second release

As I was heading towards a tsess-0.2 release, planets aligned and I was able to upgrade to Hare 0.24.0 with as little effort as I was planning to invest. The qbe package was already a snapshot of “almost qbe 1.2” and the harec package got upgraded to 0.24.0 so with a minimal change in my local packaging I was able to rebuild an up-to-date hare package.

For the tsess-0.2 release I toyed with the -R option for hare build and eventually left a build tree configured in release mode. Once I finally got motivated to revisit the Valgrind report, but this time I got this:

HEAP SUMMARY:
    in use at exit: 88 bytes in 4 blocks
  total heap usage: 101 allocs, 97 frees, 45,995 bytes allocated

12 bytes in 1 blocks are indirectly lost in loss record 1 of 4
   at 0x484280F: malloc (vg_replace_malloc.c:442)
   by 0x401F22: rt.malloc (malloc+libc.ha:7)
   by 0x4146BD: strings.dup (dup.ha:15)
   by 0x42DC60: fs.dirent_dup (types.ha:168)
   by 0x42D674: fs.readdir (util.ha:93)
   by 0x430E8C: os.readdir (os.ha:32)
   by 0x4633E3: bin.tsess.config.env_load (env.ha:119)
   by 0x460DB1: bin.tsess.config.list (list.ha:16)
   by 0x47A413: .main (tsess.ha:42)
   by 0x408738: main (start+libc.ha:20)

19 bytes in 1 blocks are still reachable in loss record 2 of 4
   at 0x484280F: malloc (vg_replace_malloc.c:442)
   by 0x401F22: rt.malloc (malloc+libc.ha:7)
   by 0x414CF5: strings.concat (concat.ha:10)
   by 0x465FC2: bin.tsess.config.arg_expand (env.ha:39)
   by 0x466143: bin.tsess.config.arg_resolve (env.ha:29)
   by 0x466270: initfunc.0 (env.ha:18)
   by 0x408D51: rt.init (initfini.ha:9)
   by 0x408733: main (start+libc.ha:19)

25 bytes in 1 blocks are still reachable in loss record 3 of 4
   at 0x484280F: malloc (vg_replace_malloc.c:442)
   by 0x401F22: rt.malloc (malloc+libc.ha:7)
   by 0x414CF5: strings.concat (concat.ha:10)
   by 0x466543: initfunc.0 (env.ha:21)
   by 0x408D51: rt.init (initfini.ha:9)
   by 0x408733: main (start+libc.ha:19)

44 (32 direct, 12 indirect) bytes in 1 blocks are definitely lost in loss record 4 of 4
   at 0x484A074: realloc (vg_replace_malloc.c:1690)
   by 0x401F0B: rt.realloc (malloc+libc.ha:21)
   by 0x402075: rt.ensure (ensure.ha:24)
   by 0x42D696: fs.readdir (util.ha:93)
   by 0x430E8C: os.readdir (os.ha:32)
   by 0x4633E3: bin.tsess.config.env_load (env.ha:119)
   by 0x460DB1: bin.tsess.config.list (list.ha:16)
   by 0x47A413: .main (tsess.ha:42)
   by 0x408738: main (start+libc.ha:20)

LEAK SUMMARY:
   definitely lost: 32 bytes in 1 blocks
   indirectly lost: 12 bytes in 1 blocks
     possibly lost: 0 bytes in 0 blocks
   still reachable: 44 bytes in 2 blocks
        suppressed: 0 bytes in 0 blocks

One things stands out immediately: the report becomes actionable with proper stack traces. It becomes apparent that my initial assessment on the reachable bits was wrong: one was the expansion of ${HOME}/.config and the other one was the one I identified. That left only one direct memory leak, with no trace of the 16kB leak. Since the -R option omits the debug package, I can only come to the conclusion that the remaining leaks were in the standard library.

Plugging the final leak

The bug fix for this second leak looks a lot like the fix from the first one:

$ git show --format=short 74c8d4cf5c94be9c41ef2615e1421b7f1d27809c
commit 74c8d4cf5c94be9c41ef2615e1421b7f1d27809c

    config: Plug leak spotted by Valgrind

diff --git a/bin/tsess/config/env.ha b/bin/tsess/config/env.ha
index de1b1d2..aa05b8a 100644
--- a/bin/tsess/config/env.ha
+++ b/bin/tsess/config/env.ha
@@ -122,6 +122,7 @@ export fn env_load(sess: *[]sess) (void | src_error) = {
        case let d: []fs::dirent =>
                yield d;
        };
+       defer fs::dirents_free(list);
        for (let i = 0z; i < len(list); i += 1) {
                if (!fs::isfile(list[i].ftype))
                        continue;

The problem with this one-liner is that I distinctly remember adding this line when I first started listing files from the configuration directory. But prior to this commit there are no traces of it. My only explanation is a Vim or Git accident, most likely an undo operation (simply pressing ‘u’) that flew under my radar.

This is a shiny example of something that wouldn’t have happened in Rust. I would likely never had freed that resource and simply relied on the fact that list would have been dropped at the end of its scope.

A quick check of memory leaks, without worrying about reachable memory:

valgrind --tool=memcheck --leak-check=full ./tsess list

And as expected, I get a clean report:

HEAP SUMMARY:
    in use at exit: 44 bytes in 2 blocks
  total heap usage: 101 allocs, 99 frees, 45,995 bytes allocated

LEAK SUMMARY:
   definitely lost: 0 bytes in 0 blocks
   indirectly lost: 0 bytes in 0 blocks
     possibly lost: 0 bytes in 0 blocks
   still reachable: 44 bytes in 2 blocks
        suppressed: 0 bytes in 0 blocks

Closing words

The key takeaways in my opinion are first that the only mistakes I made were the same: I forgot to immediately free resources with the defer keyword just after allocating them. I need to grow this reflex of planning a deferred free on the very next statement following an allocation. Even though in one case I did, but did not notice its absence during a review. Second, Hare leaves me to my own devices when it comes to resource management, but unlike C it offers a precious defer keyword that deals with multipath pitfalls. Third, as proven with Valgrind, I have some tricks up my sleeves to mitigate the inevitable mistakes I make. Fourth, the Hare model appears to be a successful recipe, as I was able to limit actual leaks to two spots for approximately 3000 lines of Hare code. And my last key takeaway is that despite learning Hare, I don’t remember ever wondering what I was doing with memory, unlike my experience with Go.

I need to mention that static linking has a benefit that dynamic linking lacks out of the box. At some point I tried to both defer a free and manually do it. Hare’s allocator warned me about a potential double free, and was of course spot on.

My actual initial verdict on Hare safety is very positive.

Or is it?

Fast forwarding to version 0.4

A couple releases later, tsess is still lacking a test suite. Thus, there has been no attempt at systematizing safety checks. As a result, another leak was introduced, unsurprisingly.

This is not a problem for several reasons, ranked by importance:

tsess is designed for short-lived executions, no lingering leaks
once the testing strategy is in place, it will include safety checks
tsess has not reached 1.0, whether in scope or quality

This time the leak was not a missing defer statement, so I can maybe claim that this reflex is settling in. It is probably too soon to tell anyway.

I also extended my safety checks to more than listing configurations and found new worrying complaints from Valgrind.

Not having time to focus on all of them I narrowed the easiest one down:

$ cat unsafe.ha
use fmt;
use unix;

export fn main() void = {
	let unsafe: (rune | void) = '\n';
	if (unix::getuid() != 0)
		unsafe = void;
	if (unsafe == '\n')
		fmt::println("ERROR")!;
};
$ hare build -qR -lc -o unsafe unsafe.ha
$ valgrind --tool=memcheck --leak-check=full ./unsafe
==601071== Memcheck, a memory error detector
==601071== Copyright (C) 2002-2024, and GNU GPL'd, by Julian Seward et al.
==601071== Using Valgrind-3.23.0 and LibVEX; rerun with -h for copyright info
==601071== Command: ./unsafe
==601071==
==601071== Conditional jump or move depends on uninitialised value(s)
==601071==    at 0x44628B: .main (unsafe.ha:8)
==601071==    by 0x408738: main (start+libc.ha:20)
==601071==
==601071==
==601071== HEAP SUMMARY:
==601071==     in use at exit: 0 bytes in 0 blocks
==601071==   total heap usage: 0 allocs, 0 frees, 0 bytes allocated
==601071==
==601071== All heap blocks were freed -- no leaks are possible
==601071==
==601071== Use --track-origins=yes to see where uninitialised values come from
==601071== For lists of detected and suppressed errors, rerun with: -s
==601071== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)

One thing that surprised me when I looked at my code was that it made a direct comparison between a tagged union type (rune | void) with a rune instead of using a match expression to decompose it. I read the Hare specification but it took me months and I do not remember the rules allowing expressions involving a union with one of its types without a match, a cast or the as keyword. I don’t even remember from reading the specification that it was possible in the first place, so I probably forgot that I was dealing with a tagged union when I did that.

The good news is that this is probably a false positive, at least as far as Hare is concerned. (Future me says no.) My assumption is that the offending if expression reads more than the type tag for its comparison, but at least it does not print an error so we get the expected behavior. (Future me says no.) Valgrind will stop complaining if the second if expression is turned into a match so there is still something amiss. (Future me shouts YES!)

The first if expression checks the PID to make sure the second assignment is not optimized away.

How do tagged union types actually work?

I explored several aspects of Hare as I was learning it, like for example how the moving parts of the build system are articulated. From my recollection of the Hare specification, the layout of tagged union types is an implementation detail, but how does the reference implementation manage them?

I tried to look at the generated assembly code to figure this out:

$ rm -rf ~/.cache/hare/tmp/
$ hare build -vvR -lc -o unsafe unsafe.ha 2>&1 | grep 'unsafe\.ha/.*\.s$'
as -o /home/dridi/.cache/hare/tmp/tmp.qYgG01hZNn/unsafe.ha/896798450034f4518c1b81f1230065157390a134955ec9fc76f91d699574c45b.o.tmp /home/dridi/.cache/hare/tmp/tmp.qYgG01hZNn/unsafe.ha/c3dd2ba88647977896af5bed73e7cbfb2ab8560fc3652a21996b94efcd048d45.s

Since this is the assembly code generated by qbe, it uses the AT&T notation I’m not familiar with. This is hopefully trivial enough for me… I mean, it can hardly be more trivial, can it? I digress.

And with that, let’s look at the beginning of the main() function:

.main:
        pushq %rbp
        movq %rsp, %rbp
        subq $112, %rsp
        .loc 2 4 25
        .loc 2 5 12
        movl $1737287038, -56(%rbp)
        .loc 2 5 40
        movl $10, -52(%rbp)

The second movl is for the '\n' assignment, a rune is a 32bit integer. My logical conclusion is that 1737287038 is the tag for the rune type either in Hare in general, in this program, or for this specific tagged union type (rune | void). After all, they both refer to line 5 where the variable is declared and first assigned.

If there is indeed an access to uninitialized memory, does it mean that the “value” for void is stored separately? Let’s skip ahead to line 7:

        .loc 2 7 30
        movl $3650376889, -112(%rbp)
        .loc 2 7 30
        movq -112(%rbp), %rax
        movq %rax, -56(%rbp)

First, we write the new tag 3650376889 for the void type in some scratch space on the stack. Then we copy 64bits to RAX and finally we copy the tag and the void “value” where the unsafe variable is stored.

If we check the beginning of the main() function again, and the code I’m omitting, we can confirm that we copied 4 initialized bytes for the tag along with 4 uninitialized bytes in RAX, and then smuggled them in unsafe.

So I can make a few observations. As usual, Valgrind was correctly complaining about uninitialized memory and I didn’t expect to prove it wrong. The layout of a tagged union type appears to be a 32bit tag followed by what looks like a C union. This was at least what I remembered from the presentation linked in this blog post (but I don’t have time for a rewatch).

I think that what’s happening is that a tag is set in the scratch space and that the void type has a zero size. So we end up with an uninitialized read because the assignment will copy the total union size. And finally, because the scratch space appears to be reserved at the beginning of the stack for the main() function, one tagged assignment could initialize this space and turn this into a false negative from Valgrind.

Another obvious consequence is that the rune check could accidentally match a 32bit uninitialized value already present on the stack.

Checking the spurious match hypothesis

Let’s tweak the program to see whether we can display “ERROR” in the standard output:

$ cat spurious.ha
use fmt;
use unix;

export fn main() void = {
	let unsafe: (rune | void) = '\0';
	if (unix::getuid() != 0)
		unsafe = void;
	if (unsafe == '\0')
		fmt::println("ERROR")!;
};
$ hare build -qR -o spurious spurious.ha
$ ./spurious
ERROR

Hypothesis confirmed.

Checking the false negative hypothesis

Let’s tweak the program to see whether we can silence Valgrind without solving the actual problem:

$ cat smuggle.ha
use fmt;
use unix;

export fn main() void = {
	let unsafe: (rune | void) = '\0';
	let unrelated: (rune | void) = void;
	if (unix::getpid() != 0)
		unrelated = 'u';
	if (unix::getuid() != 0)
		unsafe = void;
	if (unsafe == 'u')
		fmt::println("ERROR")!;
};
$ hare build -qR -lc -o smuggle smuggle.ha
$ valgrind --tool=memcheck --leak-check=full ./smuggle
==606695== Memcheck, a memory error detector
==606695== Copyright (C) 2002-2024, and GNU GPL'd, by Julian Seward et al.
==606695== Using Valgrind-3.23.0 and LibVEX; rerun with -h for copyright info
==606695== Command: ./smuggle
==606695==
ERROR
==606695==
==606695== HEAP SUMMARY:
==606695==     in use at exit: 0 bytes in 0 blocks
==606695==   total heap usage: 0 allocs, 0 frees, 0 bytes allocated
==606695==
==606695== All heap blocks were freed -- no leaks are possible
==606695==
==606695== For lists of detected and suppressed errors, rerun with: -s
==606695== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

Hypothesis confirmed.

Not only did the unrelated variable assignment hide the uninitialized access, it also managed to smuggle the 'u' character in the unsafe variable, even tagged as void. It was smuggled in the sense that the offending if expression doesn’t seem to check the tag and take this rune at face value.

Initial verdict on Hare safety

My initial verdict on Hare safety was very positive, and I can’t retroactively change that.

Did my new findings change my opinion on Hare safety? Not really. This finding was really interesting to dive into, being both really small and really trivial (emphasis on really).

I think that the (void | rune) == rune comparison should be a compile error, and I should probably submit a bug report. Valgrind is complaining about other things that are a little more worrying, but I already burnt a lot of my spare time looking into this one and writing about it.

I’m not too fond of the shared scratch space for tagged (and possibly other) assignments, but there are probably constraints I’m not aware of that lead to this design. I’m also not really familiar to how other compilers deal with their stack.

It should be noted that I found this with Hare 0.24.0, and even though the project is already quite advanced, it’s probably fair to say that Hare is still somewhat in its infancy. And I don’t expect 1.0 to somehow turn the reference implementation into perfect software. Software implies bugs and that’s a fact of life. Hare still offers me an interesting alternative to C without the complications of Rust (that I see more as a C++ alternative).

While Rust’s compiler has to behave like a static analyzer out of the box, I also believe that Hare generally leaves much less opportunities to lose track of the flow of a program, so writing a static analyzer for Hare would likely allow more accurate results than what state-of-the-art static analyzers for C can manage based on my unscientific wet-finger estimate, but I digress…

My verdict is still very positive.

Cross compiling tsess on Fedora

Sun, 25 Feb 2024

Updating my Hare tool chain

Considering the amount of spare time I can allocate to various projects I decided early on to build my own hare package and only that to learn Hare by rewriting my long-gestated tsess project. Both qbe and harec were available on Fedora, the latter as a snapshot.

Only recently we finally had a 0.24.0 release of hare and harec, and the long-awaited 1.2 release of qbe. And qbe is the reason why I stuck for so long on an ancient commit for my hare package. With limited time to track local packages, I had to draw the line somewhere or I would otherwise end up eating too much of my spare time on a regular basis.

Before the formal releases for the Hare tool chain, a release candidate of harec landed in Fedora Rawhide (currently f40), but relying on a snapshot of qbe in the absence of a proper release for features needed by harec. Luckily both qbe and harec packages would install just fine on Fedora 39 and after updating my local hare packaging I was ready to update tsess to “modern” Hare.

Cross compiling Hare code

Cross compilation is really easy when it comes to the hare build command. When I started learning Hare it was only a matter of adding a -t option to define the target architecture. Early on, I made my automake configuration build upon autoconf’s --host option. The configure script for tsess would look for a GCC tool chain for that architecture and fall back to a regular C compiler.

It does not matter since the Hare tool chain produces static binaries, right?

I already covered this earlier…

Dynamic linking of Hare programs

Hare offers C interoperability, which sounds like a requirement in the systems programming landscape. In that case it becomes possible to link with libc and use a variety of services it provides, ranging from its memory allocator to the resolution of domain or service names. To my knowledge, tsess would benefit from none of such services as of today, but it still creates new opportunities.

With dynamic linking it becomes possible to find memory leaks with Valgrind for example. And unfortunately, I found some, but haven’t studied them yet.

It is rather easy to compile a dynamically linked tsess:

$ ./configure HAREFLAGS=-lc >/dev/null
$ make
  CCLD     gen_cmd
  GEN      bin/tsess/cli/gen/cmd.ha
  CCLD     gen_env
  GEN      bin/tsess/config/gen/env_vars.ha
  CCLD     gen_prop
  GEN      bin/tsess/config/gen/prop_defs.ha
  CCLD     tsess
7/7 tasks completed (100%)
  GEN      man/tsess.1
  GEN      man/tsess.5
$ file * | awk -F '[:,] *' '$2 ~ "ELF" {printf "%s: %s, %s, %s\n", $1, $3, $5, $6}'
gen_cmd: x86-64, dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2
gen_env: x86-64, dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2
gen_prop: x86-64, dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2
tsess: x86-64, dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2

Cross compilation of tsess with dynamic linking is a little more involved, but not difficult on Fedora.

Cross compilation on Fedora

It is amazing to see how today’s tooling is making cross compilation so much easier. From package managers to emulation, it does not take much effort to get a working cross compilation environment on Fedora.

The first thing needed for cross compilation is a tool chain. As I stated before, by default the configure script will fall back to a regular C compiler which would work for Clang, but not GCC:

$ ./configure --host=aarch64-unknown-linux-gnu >/dev/null
$ grep ^CC= config.log
CC='gcc'

After installing the gcc-aarch64-linux-gnu package the configure script will find the right compiler out of the box:

$ ./configure --host=aarch64-unknown-linux-gnu >/dev/null
$ grep ^CC= config.log
CC='aarch64-linux-gnu-gcc'

In order to link with glibc, I need it installed in the tool chain’s sysroot, and the cross compiler has one configured by default:

$ gcc --print-sysroot
$ aarch64-linux-gnu-gcc --print-sysroot
/usr/aarch64-linux-gnu/sys-root

If you are wondering what the package ships in its sysroot, the answer is nothing:

$ rpm -ql gcc-aarch64-linux-gnu | grep sys-root
/usr/aarch64-linux-gnu/sys-root

The sys-root/ directory is empty, I need to populate it.

Creating a Fedora sysroot on Fedora

At the heart of Fedora we find several critical components, several of them constitute its software distribution tool chain. Fedora is an RPM-based Linux distribution, and the actual software distribution is operated by DNF. DNF introduces the notion of remote package repositories. I would install the gcc-aarch64-linux-gnu package with DNF if it wasn’t a dependency of my homemade hare package in the first place.

On top of it, another package called mock can be used to “take source RPMs and build them in a chroot”. Mock does a lot more but that’s literally how it is described today:

$ rpm -q --qf '%{description}\n' mock
Mock takes an SRPM and builds it in a chroot.

Mock can help set up a sysroot for a limited number of architectures. As of today that would be aarch64, i686, ppc64le, s390x and of course x86-64. As the description suggests, mock is backed by the chroot facility (not sure which one between the chroot(2) system call and chroot(1) program). Or rather, it was backed by chroot also known as its “simple” isolation backend. It now defaults to systemd-nspawn and can perform the bootstrap from an OCI image with podman (instead of installing all the packages from scratch).

In addition to this, DNF can work with packages on different architectures with the help of QEMU.

In other words, I can create a Fedora sysroot with a single command line thanks to mock. In fact, I can even create sysroots for other RPM-based distributions:

$ ls /etc/mock/templates/ | awk -F- '{print $1}' | sort | uniq
almalinux
amazonlinux
anolis
centos
circlelinux
custom
epel
eurolinux
fedora
mageia
navy
openeuler
openmandriva
opensuse
oraclelinux
rhel
rocky

I can imagine the amount of work it used to require to create and maintain a sysroot when we didn’t have such well-integrated tools.

For the sake of the exercise, I wanted to link tsess to both libc and libcurl. There is nothing in tsess requiring anything from libcurl, but it shows how easy it is to bring up, and how fast it happens:

$ sudo rm -rf /var/*/mock/fedora-39-aarch64*/
$ time mock -r fedora-39-aarch64 --install glibc-devel curl-devel
INFO: Unable to build arch aarch64 natively on arch x86_64. Setting forcearch to use software emulation.
[...]
INFO:
Finish: run

real	4m59.646s
user	4m45.281s
sys	0m13.513s

I couldn’t find how to really remove the OCI image with podman, it appeared to still be present after removing and pruning it, but regardless, in a few minutes I have a working sysroot. Most of the time spent was on downloading metadata and packages, on a DSL connection.

I ended up with a sysroot after next to no manual operations.

The only problems I have with the package management tool chain on Fedora is its lack of good and meaningful names. RPM is fine, it used to carry meaning before being turned into a backronym, but by that time it was well… well known? Then Yum, meaningless, but established. Why did DNF (is meaninglesser a word?) not keep the Yum name? Why is it still called Yum on RHEL? I don’t need an answer, that was a rhetorical question. And mock? Who calls a piece of software that has no relation to software testing mock? Probably somebody who wants to make sure it does not show up in search engine results.

I get it though, naming things is pretty damn hard, but it’s really a shame because as demonstrated, mock is a powerful tool.

Then again, it seems that meaningless names are the norm when it comes to package managers. Either that, or I don’t have the reference. The outcome is the same. See for example apt/aptitude, sbuild, pacman and zypper…

Cross compiling tsess

Now if I want to compile tsess for aarch64 on my x86-64 laptop I need to shoehorn my sysroot into my build system somehow. I could do it at configure time, and I actually documented how to do this. My preferred solution is to populate /usr/aarch64-linux-gnu/sys-root directly.

However, I treat /usr as immutable, so I don’t want to add anything inside. The simplest solution I found requires privileges, but is otherwise free of friction:

$ ls -1 /usr/aarch64-linux-gnu/sys-root
$ sudo mount --bind /var/lib/mock/fedora-39-aarch64/root /usr/aarch64-linux-gnu/sys-root
$ ls -1 /usr/aarch64-linux-gnu/sys-root
afs
bin
boot
builddir
dev
etc
home
installation-homedir
lib
lib64
media
mnt
opt
proc
root
run
sbin
srv
sys
tmp
usr
var

I’m now ready to build a tsess dynamically linked to both libc and libcurl, even though only libc is relevant:

$ ./configure --host=aarch64-unknown-linux-gnu HAREFLAGS='-lc -lcurl' >/dev/null
$ make
  CCLD     gen_cmd
  GEN      bin/tsess/cli/gen/cmd.ha
  CCLD     gen_env
  GEN      bin/tsess/config/gen/env_vars.ha
  CCLD     gen_prop
  GEN      bin/tsess/config/gen/prop_defs.ha
  CCLD     tsess
13/13 tasks completed (100%)
  GEN      man/tsess.1
  GEN      man/tsess.5
$ file * | awk -F '[:,] *' '$2 ~ "ELF" {printf "%s: %s, %s, %s\n", $1, $3, $5, $6}'
gen_cmd: x86-64, dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2
gen_env: x86-64, dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2
gen_prop: x86-64, dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2
tsess: ARM aarch64, dynamically linked, interpreter /lib/ld-linux-aarch64.so.1

Only tsess is built for the aarch64 target because the other programs are executed during the build.

We can easily check that the Hare flags were honored:

$ readelf --dynamic tsess | grep NEEDED
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libcurl.so.4]

Wrapping up

In the end, I find amazing how easy it is to cross compile a dynamically linked tsess, so easy it feels like cheating. It really boils down to a handful of commands:

mock
mount
configure
make

And that’s really it! It could have been a very short post, but I couldn’t help sprinkle a little digression here and there.

This apparent simplicity builds upon a huge pile of complications:

the autotools (autoconf, automake and libtool in this case)
years of RPM history (the whole tool chain, including pieces I omitted)
years of packaging refinement for cross GCC tool chains
years of Linux kernel history (everything podman relies on)

The effective simplicity comes from some degree of cohesion between all the moving parts. Catch me on a bad day and I will describe it as a high degree of coupling between components poorly cobbled together, but the cohesion is still there regardless.

For example, on my machine RPM ships with a %configure macro that takes care of applying system-wide configure flags recognized by any package relying on the autotools. I greatly suspect that the packaging of cross compilers evolved following autoconf’s conventions, and probably influenced autoconf in return.

The bottom line is that for this use case I only need to configure the bare minimum at each step:

tell mock to install my build dependencies in an aarch64 root
bind the aarch64 root directory to the cross compiler sysroot
tell the configure script to target aarch64
tell hare to link with libc (and for fun, libcurl)

And once again, that’s really it!

It’s amazing to see how our tools evolved and to see the concrete effect of iterative refinements. The Hare tool chain is still young but it is already quite usable and it shows that lessons from the past were effectively learned. It also didn’t take too much effort from me to find a crowbar and retrofit it in my familiar ecosystem, and just like that I can rebuild tsess in mere minutes and deploy it to the ARM board I occasionally SSH into.

And that’s a wrap for a yak that won’t need a shaving for a while.

Revisiting hare_build.sh

Sun, 07 Jan 2024

Previously on Digressive Developer

In my previous post introducing my new tsess project I spent much more time exploring the Hare language and reference implementation tool chain than the actual introduction.

This was not an exploration in depth of Hare, but more like a wide surface look at everything that stuck in my mind in my tsess journey. One thing in particular failed, and if I had paid attention it shouldn’t have.

Sorting out dependencies

In order to find dependencies the script contained the following function:

find_mods() {
	echo rt # always built
	hare deps -T +linux+x86_64 -d "$1" |
	awk '$2 == "->" {print $3 " " $1}' | # "foo"; "bar"
	tr -d '";' | # foo bar
	grep -Fv .ha | # only modules
	grep -wv rt | # already built first
	tsort
}

It lists dependencies in the dot format and arrange the output to be processed by tsort. With this simple trick I can build dependencies in the right order.

Notice however how the dependency order is reversed with the awk script printing the dependency first and the dependent next. I reversed this order because initially I ran into a missing dependency doing it in the right order and I used tac to reverse the order:

find_mods() {
	echo rt # always built
	hare deps -T +linux+x86_64 -d "$1" |
	awk '$2 == "->" {print $1 " " $3}' | # "foo"; "bar"
	tr -d '";' | # foo bar
	grep -Fv .ha | # only modules
	grep -wv rt | # already built first
	tsort |
    tac
}

And this would result in this output:

$ ./hare_build.sh hello-print.ha && ./hello-print
Building mod rt
Building mod types
Building mod bytes
Building mod encoding::utf8
Building mod math
Building mod strings
Could not open module 'encoding::utf8': typedef variable $HARE_TD_encoding::utf8 not set

That’s when I noticed that I was exporting HARE_TD_encoding_utf8 and trying to export the expected variable would not work. I was also not too fond of using tac so I ended up reversing the tsort input with awk instead of its output with tac.

So when I got this result:

$ ./hare_build.sh hello-print.ha
Building mod rt
Building mod encoding::utf8
Building mod errors
Building mod types
Building mod math
Building mod bytes
Building mod time
Could not open module 'linux::vdso': typedef variable $HARE_TD_linux::vdso not set

My reaction should not have been to conclude that I had run into the wrong environment variable being exported (HARE_TD_linux_vdso) but really to ask myself where the devil this module comes from and why it wasn’t listed as a dependency.

I should have known that, because I tried to submit a tsort format for the hare deps subcommand but it got postponed until after a large update to the build driver. And I knew then that the dependencies listing is not recursive, I was planning to send a patch series eventually with it.

I tweaked the script to incrementally add missing dependencies as I was trying to sort them out and eventually managed to manually link my hello-print.ha program. One thing that bothered me though, was the somewhat specific order of .a files on the ld command line to get a result.

Exporting problems

One thing that is not possible in a POSIX shell but that can be done from a POSIX shell is exporting variables like HARE_TD_encoding::utf8 to the environment.

It can’t be done in the shell’s environment, but it can be done for harec’s environment. I attempted just that with the help of the env command, and it worked.

I ended up using a construct I’m not fond of:

env $HARE_TD harec -o "$td".ssa -N $(mod_ns "$1") -t $td.td $(mod_src "$1" ha)
[...]
HARE_TD="$HARE_TD HARE_TD_$1=$td.td"

But it had the merit to work.

Putting everything together

At this point I wasn’t sure whether compiling all modules would be enough to successfully link the program. The dependencies listing ended up looking like this:

find_mods() {
	for mod
	do
		hare deps -T +linux+x86_64 -d "$mod"
	done |
	awk '$2 == "->" {print $1 " " $3}' | # "foo" "bar";
	tr -d '";' | # foo bar
	tsort |
	grep -Fv "$1" # only dependencies
}

Instead of giving rt a special treatment here, the only thing to filter out is the module itself, the $1 parameter. The topological sort as populated by awk is back to the “natural” order.

The complete script looks like this:

#!/bin/sh

set -e
set -u

tag_filter() {
	grep -Pv '\+(?!linux|x86_64)'
}

mod_dir() {
	printf %s "/usr/src/hare/stdlib/$1" |
	sed 's|::|/|g'
}

mod_ns() {
	printf %s "$1" |
	sed 's|::|.|g'
}

mod_td() {
	printf %s "$1" |
	sed 's|::|_|g'
}

mod_src() {
	dir=$(mod_dir "$1")
	ext=$2
	find "$dir" -name "*.$ext" -type f |
	grep -v "$dir/[^+-].*/" | # do not enter submodules
	tag_filter
}

find_mods() {
	for mod
	do
		hare deps -T +linux+x86_64 -d "$mod"
	done |
	awk '$2 == "->" {print $1 " " $3}' | # "foo" "bar";
	tr -d '";' | # foo bar
	tsort |
	grep -Fv "$1" # only dependencies
}

build_mod() {
	td=$(mod_td "$1")

	test -f "$td.td" &&
	return

	for mod in $(find_mods "$1")
	do
		build_mod "$mod"
	done

	echo "Building mod $1"
	td=$(mod_td "$1")

	env $HARE_TD harec -o "$td".ssa -N $(mod_ns "$1") -t $td.td $(mod_src "$1" ha)
	qbe -o "$td".s "$td".ssa
	as -o "$td".o "$td".s

	for s in $(mod_src "$1" s)
	do
		as -o "$td-$(basename "$s" .s).o" "$s"
	done
	ar -r -c "$td".a "$td"*.o

	HARE_TD="$HARE_TD HARE_TD_$1=$td.td"
}

rm -f -- *.td *.ssa *.s *.o *.a

HARE_TD=
for mod in $(find_mods "$1") rt
do
	build_mod "$mod"
done

prog=$(basename "$1" .ha)

env $HARE_TD harec -o "$prog".ssa "$1"
qbe -o "$prog".s "$prog".ssa
as -o "$prog".o "$prog".s

ld.lld -T /usr/src/hare/stdlib/rt/hare.sc -o "$prog" "$prog".o *.a

To solve the *.a ordering problem I tried other linkers. The default was bfd, gold had the same problem, and lld appeared to do just fine so I picked the latter and never tried to find the right command line options to ask the other linkers to unconfuse themselves.

Lo and behold, it works:

$ ./hare_build.sh hello-print.ha
Building mod rt
Building mod errors
Building mod types
Building mod bytes
Building mod encoding::utf8
Building mod strings
Building mod io
Building mod memio
Building mod path
Building mod format::elf
Building mod linux
Building mod types::c
Building mod linux::vdso
Building mod math
Building mod time
Building mod fs
Building mod bufio
Building mod os
Building mod ascii
Building mod strconv
Building mod fmt
$ ./hello-print
hello

When I linked the program manually I was able to get away with less modules, but that made a difference of only 1kB on the final binary. If I add the --gc-sections option I get the exact same file size for both manual and scripted linking, but the programs are not identical.

I will not go down this rabbit hole, but I’m happy to report that it is possible to emulate the Hare tool chain with a shell script (and all the transparency that comes with it). I could probably also emulate hare deps instead of invoking it but I was merely interested in the build pipeline here.

Simple and accessible.

Introducing tsess

Sat, 06 Jan 2024

Best wishes for the new year(s)

2024 is finally here, and a new entry in this blog was long overdue. It’s not that I had nothing to write about, I always have something silly in store I could mention here. I simply had little time to spare, and chose to do other things on my spare time.

In these last three and a half years I have kept up to date with many things, which is an implicit requirement when choosing Fedora as an operating system but one thing I failed to uphold was the maintenance of this blog. It was a little bit painful to deal with updates to Hugo and the Jane theme, but I somehow succeeded.

Since January is the time for new year’s resolution, this years I made up two for myself:

migrate my development workflow to tsess
write at least one blog post per year

I’m so good at upholding resolutions that I already succeeded both. I can go back to extreme procrastination with a good conscience. See you next year.

Introducing tsess?

In 2023 I discovered the Hare programming language when a FOSDEM presentation was brought to my attention, although admittedly I don’t remember how. I suspect it was from LWN because I was lagging far behind on this feed, and nowadays I’m not even trying to follow it, but I digress.

My first reaction was of course to compare it to Rust, a language I have been following since version 0.4 as a serious contender for C. The philosophy behind Hare matches many opinions I forged over time, and as it turned out, more than Rust. So I started reading about Hare and playing with it until it was time, six months later, to have a “real” project to effectively learn it.

I started musing about this tsess project when the tmuxinator package was orphaned in Fedora and eventually removed from the repositories. In late 2022, two years after maintaining my own RPMs for tmuxinator and its dependencies outside of Fedora, I started tsess in Rust and failed in the context of self-imposed constraints. In August 2023 I decided to try again in Hare.

In 2023 I also authored my first Go project for work and among other things it involved crafting a simple parser, which is most of what I have done in Rust so far. The Hare tool chain shows similarities with Go’s and the Hare language itself looks like C enhanced by concepts found in both Rust and Go. Rewriting tsess in Hare became all the more interesting for me in that context.

And in the common Internet tradition, I have to explain why Vim is better than Emacs, or the other way around.

The Rust-Go-Hare showdown

Rust

I have been following Rust’s evolution for years now but never worked on a real project doing actual work in production somewhere. I really love Rust so all the negative things I’m going to say about it must be framed in this context: I really love Rust.

In a nutshell, Rust is to me a language that encodes resource management in its type system (and to some extent its standard library) and effectively turns the compiler into a static analyzer. For forcing me to think harder about ownership I am forever grateful.

Rust also has relatively straightforward C interoperability but despite starting as a C-like language like so many others it embraces functional programming constructs. The standard library is by design wide, providing a lot of variants for a lot of features. Likewise, the syntax sometimes offers several ways radically different to achieve the same thing and interestingly the Clippy linter will steer you towards certain constructs.

Rust forces you to handle return values, elegantly generalizes null handling with its Option type and offers a pleasant higher-level error management with the Result type. It is also expression-oriented, which is refreshing coming from C.

In comparison, C is a simple statement-oriented imperative language. The simplicity of C is downright impressive, but the language is sometimes too simplistic and overall plagued with undefined behavior. Rust on the other hand is too complicated. I call it an imperacryptive language and my inability to fit it in my head is a problem, even assisted by tools. Mind you, there are some bits of C equally cryptic like the rules for struct layouts or the mere ability to come up with Duff’s device but the things difficult to remember (for me) about C are things I rarely need or actively avoid (unfortunately you can’t just look away from undefined behavior).

I’m also not in line with the Rust community philosophy of having many crates “doing one thing and doing it well” à la UNIX™. I don’t like that cargo is both a dependency manager and a build system, this is the main ingredient in the recipe for supply chain attacks and generally a lack of separation of two critical concerns. Cargo also poorly composes with polyglot projects, like any language’s self-centered all-encompassing tool chain. This is true of Go and Hare as well, but to a lesser extent for the latter. I became familiar with this problem in a past life working on Java projects built with the Maven build system, but don’t get me started on this one or I will dial digressions up to eleven.

Go

Go in comparison is much simpler than Rust, and has an overall pleasant look and feel. Rust has a rather heavily ceremonial syntax and in comparison Go is very straightforward.

Like Rust, Go has very interesting design choices, but my main problem is the amount of magic that goes behind the scenes. Rust is certainly not devoid of magic constructs but in general I know what I’m doing or trying to do. With Go most of the time all bets are off, as far as I’m concerned. It should be noted that I never tried to formally learn either Rust or Go, unlike Hare. In Go’s defense, before this first work project I had spent much less time looking at it than say, Rust.

Error handling in Go is interesting too, but not to my liking. Still, I tried to adhere to the coding standard and do things the Go way because I was going to ultimately hand over this code to someone else for future maintenance, but then I ran into exotic error handling in the text/scanner package I relied on for my parser. Navigating the standard library was not always trivial. For example do I use net package to work with CIDRs? What about the net/netip package?

The Go tool chain is clearly the worst of the three in my ranking. The way Go manages dependencies as Git repositories is horrendous. At least Cargo works with actual “crates” repositories. I understand some of the constraints behind the source-only compilation of mostly-statically-linked programs for both Rust and Go (and Hare!) but it doesn’t mean that I approve. At least Rust solves source code incompatibilities with an interesting system of editions. In Go, the resulting dependency hell appears to be universally solved with vendoring, and well, to me this is only adding one circle to this inferno. This is giving a license to software maintainers to not carefully design interfaces. Can’t just upgrade to the latest version of X, or need a specific patch? Lock your dependency or vendor it.

Oh, and I don’t like ending up with a hello world weighing as much as a web browser once compiled. I may be exaggerating a little here, but man, Go programs are anything but lean.

Hare

Now Hare is a beast of its own, but one I found incredibly easy to tame. Hare looks like C, with Rust-like type decomposition, a Go-like tool chain, and everything pretty much toned down.

Hare is expression-oriented like Rust, and unlike Go. While the syntax looks ceremonial, with for example code blocks delimited by curly braces requiring a semi-colon after the closing brace, once you realize the block is just another expression it makes sense. So far Hare has the most consistent syntax of the three.

In fact, Hare comes with a formal specification that I spent a lot of time reading on an on and off basis. In the roughly six month between the time I discovered Hare and the time I started writing tsess I had probably read half of the document. I’m not sure whether this is me being more mature, the Hare specification being more accessible (than say, the ISO C standards) or a case of all of the above, but it was instrumental helping me fit Hare in my brain.

Between the specification and overall consistency of the language design, Hare managed to enter my slow brain faster than any other programming language I can remember learning. I found one minor inconsistency not worth mentioning, but I may revisit it in the future.

While consistency probably plays a role, my favorite aspect of Hare is the emphasis on simplicity. I feel that nowadays simplicity is underappreciated in the software landscape. Making and keeping a non-trivial system simple is no small fit. In that regard Rust failed utterly, keeping the core language small and simple, but not the language we see, the internal representation, the one the compiler “desugars” actual Rust code to. I think I eventually had a sugar overdose that distanced myself from Rust. On the other hand of the spectrum C is so simple it tips over to being too simplistic in some areas. Meanwhile, Go has too much magic and I wasn’t able to get a good grasp on all the code I wrote.

Hare appears to draw inspiration from Rust in the error handling department. But instead of forcing an arbitrary Result type down our throat, it enables any type to become an error by suffixing it with a ! character. Like Rust, Hare requires error handling. Unlike Hare, Rust is explicit about handling everything (I mentioned Option already). Suffixing an expression with ! tells the compiler that it should be fail-safe, or otherwise abort. The ? suffix tells the compiler to return from the function if the return value is an error.

One thing I would like to see added is a _ suffix to ignore the value of an expression, and by extension the error cases, effectively casting it to void:

if (something == wrong) {
	fmt::fprint(os::stderr, "Something went wrong!\n")_;
	os::exit(STATUS_SOMETHING_WRONG);
};

Ignoring a potential failure to write the error message would ensure that the programs exits with the desired status, instead of aborting.

This can be done like this:

let ign = fmt::fprint(os::stderr, "Something went wrong!\n");

But I’m not sure whether the specification will evolve in a stricter direction where for example you can’t have unused variables. The problem with the PDF specification draft is the lack of git commit to review changes since the revision I downloaded. I can probably infer it to roughly a year back.

It also becomes unwieldy to ignore multiple expressions in the same scope, something that a _ suffix could elegantly solve while still being explicit. Something you would catch in a code review to dispute or argue.

Now here is something to watch, I just had an idea and I need to follow my intuition right now. I will be back soon:

I’m back, thank you for your patience. Complaining about the unwieldiness of the construct above and my suggestion to make _ cast away to void gave me the idea to try just that:

fmt::fprint(os::stderr, "Something went wrong!\n"): void;

And well, it does what I need the way I suggested it without polluting the code, so I think I have no further complaints about error handling.

Damn.

Moving on, another “problem” with Hare is that it’s still under development, so things change. I have been there since Rust 0.4, but I’m also stuck on an ancient version of Hare because I decided to use the qbe package available on Fedora.

One concrete example is @noreturn functions, or in more recent versions of Hare, functions returning never:

fn some_function(arg: some_type) (void | error) = {
	let obj = foo::allocate(arg);
	defer foo::finish(&obj);

	bar::do_some(&obj)?;
	bar::do_more(&obj)?;
	bar::prepare(&obj)?;
	os::exec(&obj.cmd);
};

The defer keyword (inspired by Go?) ensures that obj is freed if one of the three functions from the bar module fails. The problem is that it also ensures that it is freed before calling os::exec() that is not supposed to return. I haven’t been able to ponder this much, but I can see cases where a defer expression should be evaluated even when the program will not return, but I don’t know what kind of syntax could express both cases in a meaningful way. Maybe an always keyword to match the never type?

defer foo::rm_tmp_dir(): always;

One thing I love in Hare is the parity between logical and bitwise boolean operators. There are gaps like this or the lack of explicit endianness support in the C standards that I find surprising to still have to this day. There is however one boolean operator that does not have a counterpart:

if (res is error) {
	// do something
};

There is a match keyword similar to Rust’s, not as ergonomic, but still very usable. Sometimes it is simpler to go with the is operator and match is arguably overkill. I understand that is composes just fine with ! but for this very case I would arbitrate in favor of a little duplication:

if (!(this is less_readable)) {
	// do something
};

// vs

if (expr not convoluted) {
	// do something
};

I also really like that break and continue keywords only operate on loops, and it’s probably for the best that falling through switch or match cases is not possible. The multi-modal for loop is bold, but so far there has always been a mode that fits my needs. There’s just one thing that irritates me about both Rust, Go and Hare: what is wrong about the do-while style of loops? I’d love to see a single-mode do expression for ( expression ) kind of loop.

While I could complain about the Hare tool chain for being of the kind I do not like, I would rather point out that the hare command, unlike go and cargo is a build driver without dependency management. It will only discover Hare modules and not endeavor to fetch them or introduce a new “standard” form of repository. And this is not by accident or lack of implementation so far, this is a stated design choice:

Dependencies

We have a module system!

But no package manager

Use your distro’s package manager

Choose your dependencies conservatively

stdlib gets you most of the way there

How refreshing!

If I had to pick one word to summarize Hare, it would be thoughtful. I could go on praising or complaining about Hare, but at this rate I would never say a thing about tsess.

After this small digression it shouldn’t come as a surprise that Hare wins this showdown by a wide margin. I still love Rust and never really liked Go, but Hare was a major discovery for me in 2023, as big as Git and Varnish a little more than a decade ago.

Introducing tmuxinator

Now let’s rewind back forever ago, after I started using Linux as my main operating system. It was Ubuntu, and it was a mixture of pleasantness and frustration because I never really like the apt ecosystem. I knew about Fedora but so many things were missing, a different kind of frustration, and yet I eventually bit the bullet. I became a Fedora contributor the day I realized I had all the skills needed to maintain some packages I was lacking, but I digress…

At some point in my Linux journey, I settled on a terminal emulator called terminator that I think was unmaintained at the time. It appears to exist as Gnome Terminator today. The whole deal is to have multiple terminals in the same window, and do interesting things with them.

At some point in my current day job, a former colleague tried to convince me to use tmux by repeatedly telling me to use it. I eventually got tired of it and started looking, and that’s when I ran into tmuxinator. I can only assume the name was a reference to both tmux and terminator but I can’t tell for sure. After setting up tmux key bindings to manage tmux panes almost like terminator terminals (I couldn’t use the Alt key for keyboard shortcuts) I had a close enough setup to replace my previous one. And on top of that my colleague would no longer bother me about tmux.

Once the muscle memory was successfully rewired, it was an overall win.

What does tmuxinator bring to the table? The management of tmux sessions from a declarative configuration read from YAML files that happen to also be ERB templates, via a command line interface.

I had been a happy tmuxinator user until it disappeared from the Fedora repositories. To bring it back on my machine I had to make an RPM and two more for Ruby gems that also went away. I have no Ruby programming experience and felt really unfit for maintenance on behalf of Fedora (even though it proved to be relatively low maintenance).

I’m not particularly interested in Ruby. Interpreted languages are generally not my vibe and I run away from systems prone to “magic” constructs and my understanding is that Ruby is a very dynamic language that encourages magic syntax tricks.

Meeting the hare tool chain

The hare command I already mentioned mainly coordinates operations performed by other utilities, including the harec compiler that turns Hare code into QBE’s intermediate language. The hare command maintains a cache of build artifacts, its way of dealing with incremental rebuilds.

The pipeline goes like this:

harec translates .ha files to a .ssa file
- it may also produce a “typedef” .td file
qbe translates the .ssa file to a .s file
as compiles the .s file to a .o object file
- the module may ship .s files too
ar may collect object files into a .a archive
rinse next module and repeat
ld links .o and .a archives

The .td file could be compared to a .h file in C, it’s just a Hare file containing the module exported types and symbols.

If the Hare program is linking to shared objects, cc is used in place of ld for linking.

And finally, harec may compile multiple .ha files into a single .ssa (and .td) file because a translation unit in Hare is a module, so all the module files must be compiled at once. The hare program does a little bit more than just the above, it also manages a system of tags similar to Rust conditional compilation with the #[cfg_attr] attribute or cfg! macro, except that it takes the form of directories named after tag names, or files (Hare or assembly) suffixed with tag names.

To illustrate the process, let’s write a hare_build.sh script with hard-coded tags:

#!/bin/sh

set -e
set -u

tag_filter() {
	grep -Pv '\+(?!linux|x86_64)'
}

mod_dir() {
	printf %s "/usr/src/hare/stdlib/$1" |
	sed 's|::|/|g'
}

mod_ns() {
	printf %s "$1" |
	sed 's|::|.|g'
}

mod_td() {
	printf %s "$1" |
	sed 's|::|_|g'
}

mod_src() {
	dir=$(mod_dir "$1")
	ext=$2
	find "$dir" -name "*.$ext" -type f |
	grep -v "$dir/[^+-].*/" | # do not enter sub-modules
	tag_filter
}

build_mod() {
	td=$(mod_td "$1")
	harec -o "$1".ssa -N $(mod_ns "$1") -t $td.td $(mod_src "$1" ha)
	qbe -o "$1".s "$1".ssa
	as -o "$1".o "$1".s

	for s in $(mod_src "$1" s)
	do
		as -o "$1-$(basename "$s" .s).o" "$s"
	done
	ar -r -c "$1".a "$1"*.o

	eval "export 'HARE_TD_$td=$td.td'"
}

find_mods() {
	echo rt # always built
	hare deps -T +linux+x86_64 -d "$1" |
	awk '$2 == "->" {print $3 " " $1}' | # "foo"; "bar"
	tr -d '";' | # foo bar
	grep -Fv .ha | # only modules
	grep -wv rt | # already built first
	tsort
}

rm -f -- *.td *.ssa *.s *.o *.a

for mod in $(find_mods "$1")
do
	echo "Building mod $mod"
	build_mod "$mod"
done

prog=$(basename "$1" .ha)

harec -o "$prog".ssa "$1"
qbe -o "$prog".s "$prog".ssa
as -o "$prog".o "$prog".s

ld -T /usr/src/hare/stdlib/rt/hare.sc -o "$prog" "$prog".o *.a

I still relied on hare deps to collect modules, but otherwise tried to decompose the logic to better illustrate how the build system is glued together, for the static linking case. It doesn’t honor the HAREPATH and probably a lot of other things expected from hare build.

Let’s see it in action with a basic program that does nothing, possibly the smallest legal Hare program:

$ cat hello-void.ha
export fn main() void = void;
$ ./hare_build.sh hello-void.ha
Building mod rt
$ ./hello-void && echo OK
OK

Next, let’s try a program that writes hello to its standard output:

$ cat hello-write.ha
use rt;

let msg: []u8 = ['h', 'e', 'l', 'l', 'o', '\n'];

export fn main() void = rt::write(1, &msg[0], len(msg))!: void;
$ ./hare_build.sh hello-write.ha && ./hello-write
Building mod rt
hello

And finally, let’s see it in action with the fmt module:

$ cat hello-print.ha
use fmt;

export fn main() void = fmt::print("hello\n")!: void;
$ ./hare_build.sh hello-print.ha
Building mod rt
Building mod encoding::utf8
Building mod errors
Building mod types
Building mod math
Building mod bytes
Building mod time
Could not open module 'linux::vdso': typedef variable $HARE_TD_linux::vdso not set

This shell experiment only goes so far. For modules below the top level it appears that harec expects environment variables with :: in their names, which is not legal for shell variables. At least not possible with a portable POSIX shell script. Expecting HARE_TD_linux_vdso would be cleaner, but also conflict with a top-level module called linux_vdso. I don’t see an obvious solution to this besides forbidding underscores in module names.

Overall the tool chain is very lean. I can build the QBE compiler backend in less than a minute on my machine, including the time needed to browse to its web site and download a release archive.

Most of the stack is standard off-the-shelf tools, and the hare command takes care of coordinating everything. The directory nature of translation units make this harder to integrate in a Makefile, compared to the POSIX way of building C programs and libraries for example.

I even contributed a couple minor patches as I was playing with it and building my own RPM for hare.

Still, I proudly managed to commit the unforgivable in that department.

The GNU build system

The Autotools are a widely-hated set of tools that make up the GNU build system, and operate on top of the venerable make command. This is a radical departure from the general simplicity of the Hare ecosystem.

I wouldn’t be surprised if the autotools weren’t welcome in the Hare community after witnessing hostility towards them left and right over the last decade.

While I share a lot of grievances regarding the autotools, I stand by my choice. Even good old make has its share of problems but to this day, make is what offers the best level of abstraction for me. And automake has the best integration with its underlying make backend.

On top of that, it offers many services ranging from the installation of files of different natures, a decent test driver, and a form of standardization over a set of make targets. This is also the only build system I know that erases itself for distribution, requiring only a shell and make on top of the package’s tool chain for downstream consumers. Such downstream consumers can be your operating system’s package manager, and for example it integrates out of the box in an RPM spec.

I just wish it wasn’t oh so complicated underneath.

When I wrote my first toy project in Rust I decided to use the autotools and I was happy about the result, but I was also constantly fighting the Rust tool chain. For tsess I decided to throw the towel and use cargo, I’d figure the non-Rust bits later, but I never reached that point.

When I rebooted tsess in Hare, I ended up using the unholy trinity:

Autoconf

There isn’t much to say here. The Hare tool chain only brings a handful of dependencies, and I bring automake, libtool, hare and scdoc.

I wrote my own Autoconf macros for Hare configuration, but nothing worth dwelling on, besides maybe the creation of a HARE_ENV variable for the hare and harec commands, and the injection of the VPATH into HAREPATH.

One thing that is not easy to do is deriving the default HAREPATH from the hare command. For the ancient Hare tool chain I’m using, hare version -v does not inspire confidence and I considered submitting a hare config subcommand inspired by pkg-config to probe the Hare tool chain:

DEFAULT_HAREPATH=$(hare config harepath)

But I will wait until QBE 1.2 is released and qbe/harec packages are updated on Fedora. I’m not interested in maintaining more packages locally.

Libtool

Likewise, there isn’t much to say about Libtool except that once included it changes how Automake rules are put together, and incorporating the Libtool behemoth is what made the rest simpler.

How ironic.

Automake

Normally one would declare a list of programs or binaries, then exhaustively list its sources, and in theory things just work. Things may only just work when languages are known to Autoconf, which is obviously not the case here.

I managed to keep the complications for tsess relatively under control:

HARE_BUILD = env $(HARE_ENV) $(HARE) build

LINK = $(HARE_BUILD) $(HAREFLAGS) -t $(host_cpu) $($(@)_HAREFLAGS) -o $@

bin_PROGRAMS = tsess

tsess_SOURCES = \
	bin/tsess/attach/attach.ha \
	bin/tsess/cli/cli.ha \
	bin/tsess/config/env.ha \
	bin/tsess/config/error.ha \
	bin/tsess/config/list.ha \
	bin/tsess/config/prop.ha \
	bin/tsess/config/sess.ha \
	bin/tsess/lexer/lex.ha \
	bin/tsess/lexer/tok.ha \
	bin/tsess/parser/parse.ha \
	bin/tsess/parser/rule.ha \
	bin/tsess/text/pos.ha \
	bin/tsess/text/scan.ha \
	bin/tsess/text/text.ha \
	bin/tsess/tmux/proc.ha \
	bin/tsess/main.ha

nodist_tsess_SOURCES = \
	bin/tsess/cli/gen/cmd.ha \
	bin/tsess/config/gen/env_vars.ha

tsess_DEPENDENCIES = \
	$(tsess_SOURCES) \
	$(nodist_tsess_SOURCES)

tsess_HAREFLAGS = \
	-D 'TSESS_STRING="$(PACKAGE_STRING)"' \
	-D 'TSESS_VERSION="$(PACKAGE_VERSION)"'

tsess_LDADD = $(srcdir)/bin/tsess/main.ha

With Libtool the build is normally driven by COMPILE and LINK variables, and in turn they rely on the configured CC and CCLD variables. But if tsess_SOURCES contained .c files instead it would implicitly populate tsess_OBJECTS with their .o counterparts.

As illustrated earlier, a Hare build is not as straightforward as turning individual C files into object files and linking them together, so nothing picks up the .ha files. This is why tsess_DEPENDENCIES has to reference the sources explicitly, to trigger $(LINK) commands.

And so only the LINK variable is overridden to invoke hare build, and it appears in the output as a CCLD operation, which is not inaccurate. This means that a Makefile.am file cannot contain rules for both Hare code and another language managed by the autotools like C, but it is still possible to have both in the same project in separate Makefiles.

It would be that simple if it weren’t for the nodist_tsess_SOURCES. Hare does not have a macro system like C or Rust and encourages code generation instead, so I decided to find a reason to justify having some.

VPATH hell

At this point it should be mentioned that many things that are not strictly necessary in tsess are there for learning purposes. For code generation, it’s not so clear cut as I wanted to avoid disconnecting the documentation from the code.

With Automake the make VPATH contains the build directory and the source directory. They can be the same directory, which is usually the case when ./configure is executed. When the configure script is invoked from a different directory like ../tsess/configure it results in $srcdir having a different value than $builddir and in this example ../tsess. This is why $(srcdir) is added escaped to the HAREPATH so it’s expanded by make.

I don’t know what $builddir can be besides . and I suspect it only exists for parity. There are others “abs” and “top” variants of the VPATH components.

This means that for a single source tree there can be multiple concurrent build trees. For example one building with GCC and another with Clang. It also means that a “dist” archive, once unpacked, can be mounted read-only and not dirtied in a continuous integration environment (or fail the build if that happens).

Unfortunately there are some gotchas in the VPATH department. For example if we consider the following suffix rule:

SUFFIXES = .1.scd .1
.1.scd.1:
	$(SCDOC) <$< >$@

If the source lives in the VPATH outside of the current directory, gmake will create the target in the current directory, but bmake will create it in the same directory as the source. If the source tree is read-only, or in general with Automake’s model, bmake’s behavior is a problem.

This is also a problem with tools that don’t support searching files from a PATH. In scdoc’s case there isn’t an include feature in the first place, so I used Autoconf instead to reconstruct the manuals from multiple fragments.

The downside of this approach is that as far as make is concerned, the .scd files are always seen as updated after the configure step so manual pages always need to be rebuilt. So even when building from a “dist” archive scdoc fails to erase itself like the rest of the autotools and there is no actual benefit of shipping the man pages.

It’s still possible for end users to erase scdoc at configure time:

$ tar xf tsess-0.1.tar.gz
$ cd tsess-0.1
$ ./configure SCDOC=/bin/true

Or at least it would be if scdoc didn’t strictly work with standard input and output. The result of $(SCDOC) <$< >$@ would result in an empty manual. If it was possible to run $(SCDOC) -o $@ $< instead, then that would have been possible. I think this is a concrete example of too much simplicity making a system simplistic.

To work around these limitations, I wrote a script called mkmissing that rebuilds target based on the checksum of sources. So unless some scd file is patched, scdoc should never be needed during a downstream build from a release archive.

And I didn’t just wing it, I carefully crafted mkmissing. For some reason I really enjoy shell scripting, I can’t help it…

Other tools like hare can work with VPATH builds, but the way HAREPATH works adds a limitation: a module cannot be split across multiple components of the HAREPATH. This is why generated files land in gen sub-modules:

nodist_tsess_SOURCES = \
        bin/tsess/cli/gen/cmd.ha \
        bin/tsess/config/gen/env_vars.ha

Thankfully hare doesn’t enforce that sub-modules be sub-directories of their “parent” modules. There doesn’t seem to be a relationship beyond a logical hierarchy.

The idea here is to generate the model for environment variables and the command line interface based on what the manual says. This way the manual is authoritative and changes to the documentation are immediately effective, or at least actionable. The getopt module (as far as my ancient Hare goes) has poor ergonomics but an excellent design, so I was able to implement a thin wrapper on top to get my desired result:

$ tsess -h
Usage: tsess [-h] [-q|-v]

List available subcommands.

Options:
  -h              Print usage and exit.
  -q              Quiet mode.
  -v              Verbose mode.

Subcommands:
  attach          Attach to a session.
  list            List available sessions.
  version         Display the version number.

$ tsess -h attach
Usage: tsess attach [-L|-S] <name>

Attach to a session.

Options:
  -L socket-name  Alternative socket name of the tmux server.
  -S socket-path  Alternative socket path to the tmux server.

Parameters:
  name            The name of the session.

Throwaway Hare programs generate the Hare sources from the manual fragments, and that became a problem from cross compilation.

Cross compiling tsess

Hare currently supports Linux and FreeBSD, and apparently OpenBSD is being worked on. It also targets the three architectures supported by QBE: x86_64, aarch64 and riscv64.

Cross compiling tsess is incredibly easy:

$ ./configure --host=riscv64-unknown-linux-gnu >/dev/null
$ make
  CCLD     gen_cmd
  GEN      bin/tsess/cli/gen/cmd.ha
  CCLD     gen_env
  GEN      bin/tsess/config/gen/env_vars.ha
  CCLD     tsess
  GEN      man/tsess.1
  GEN      man/tsess.5
$ file gen_cmd gen_env tsess
gen_cmd: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, with debug_info, not stripped
gen_env: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, with debug_info, not stripped
tsess:   ELF 64-bit LSB executable, UCB RISC-V, double-float ABI, version 1 (SYSV), statically linked, with debug_info, not stripped

That alone is quite the payoff.

I think that it even is easier than cross compiling a C project with a mix of programs for the build and host architectures.

Part of this is solved by my local RPM packaging of Hare. In order to build Hare a config.mk file is needed to specify the various commands for normal and cross builds. In the %prep step of the RPM build I generate the right configuration on the fly based on the target architecture with a Lua macro:

%prep
%autosetup -p1 -n %{name}-%{shortcommit}

tee config.mk <<'EOF'
PREFIX = %{_prefix}
BINDIR = %{_bindir}
MANDIR = %{_mandir}
SRCDIR = %{_usrsrc}
STDLIB = $(SRCDIR)/hare/stdlib
HAREPATH = $(SRCDIR)/hare/stdlib:$(SRCDIR)/hare/third-party
PLATFORM = %{_host_os}
ARCH = %{_arch}
HAREC = harec
HAREFLAGS =
QBE = qbe
AS = %{__as}
AR = %{__ar}
LD = %{__ld}
SCDOC = scdoc
HARECACHE = .cache
BINOUT = .bin
%{lua:
for arch in string.gmatch(macros.hare_arches, '%S+') do
    local host = arch.."-linux-gnu-"
    local host_as = "as"
    local host_ar = "ar"
    local host_cc = "cc"
    local host_ld = "ld"
    if arch == macros._arch then
        host = ""
        host_as = macros.__as
        host_ar = macros.__ar
        host_cc = macros.__cc
        host_ld = macros.__ld
    end
    print(string.upper(arch).."_AS = "..host..host_as.."\n")
    print(string.upper(arch).."_AR = "..host..host_ar.."\n")
    print(string.upper(arch).."_CC = "..host..host_cc.."\n")
    print(string.upper(arch).."_LD = "..host..host_ld.."\n")
end}
EOF

So all I need is to target the host architecture when “linking” Hare programs:

LINK = $(HARE_BUILD) $(HAREFLAGS) -t $(host_cpu) $($(@)_HAREFLAGS) -o $@

And code generators can override it with their hare flags:

gen_cmd_HAREFLAGS = -t $(build_cpu)

This way, there’s no need to worry about treating code generation like manual page generation. The Hare tool chain is a requirement anyway, unlike scdoc that I made optional with the help of mkmissing.

Testing

Hare, like Go and Rust, comes with native support for unit testing in the form of a @test attribute that is part of the language specification. I’m not planning to use the @test attribute in this project.

As a matter of fact, the 0.1 release of tsess ships with no automated tests. I could spend an entire post on why I don’t engage in unit testing and prefer to focus on functional testing and why the proverbial test pyramid should be considered harmful, but I’m already spending an entire post on introducing tsess and I believe that so far it still hasn’t been formally introduced.

The bottom line is that I actually have plans for testing and I figured about fifty percent of the testing architecture. What’s certain is that it will rely on Automake’s default test driver.

For all its flaws, I’m convinced that adopting the GNU build system for tsess brought more value than liabilities.

From zero to zero dot one

The development of tsess happened in small bursts. It took me much less than a week’s worth of full time work to get to the first release. Except of course that if I had tried to do everything in one go it would likely have taken more time. Between bursts I would read or reread parts of the spec, read more code from the standard library or contribute trivial patches to the project. So I had time to let whatever I learned sink in or perform some experiments before putting them in practice on the tsess code base:

August 23rd, one evening, built a working lexer
August 27th, one evening, built a working parser
Septenber 2nd, one evening, ended up shaving the mkmissing yak
October 5th, a train ride, started modeling configuration
November 23rd, one evening, more work on configuration, first codegen
December 4th, one evening, command line interface, more codegen
December 14th, one afternoon, added a tmux controller
December 15th, one morning, a minimum viable tmuxinator replacement
December 31th, tsess 0.1 released

Well, most of it happened in December, and of course this project had to start with a parser.

Parsing the configuration file

I enjoy writing parsers. For my day job I worked on several occasions on the VCL parser and my first Rust project was a VCL preprocessor to experiment with the VCL syntax.

Why make up my own configuration file format when I could reuse an established format like tmuxinator did? The answer is two-fold: I wanted a configuration format really tailored to the user experience for tsess and it was a very good opportunity to evaluate Hare since I worked on more parsers than I can remember.

Here is for example the tmuxinator configuration of my Fedora “workspace”:

name: fedora
root: ~/fedora

windows:
<% for pkg in [
        'haproxy',
        'libslz',
        'makeself',
        'numatop',
        'python-funcparserlib',
        'python-webcolors',
        'ShellCheck',
        'tiptop',
        'vcsh',
        'vmod-querystring',
] %>
  - <%= "#{pkg}" %>:
      layout: main-horizontal
      root: ~/fedora/<%= "#{pkg}" %>
      panes:
        - vim <%= "#{pkg}" %>.spec
        - git status
<% end %>

Reminder, it’s an ERB template producing YAML.

In tsess my configuration file looks like this instead:

name = fedora
root = "~/fedora"

[@pkg@]
layout = main-horizontal
root = "@pkg@"

pane.editor.send-keys = << EOF
vim @pkg@.spec
EOF

pane.prompt.send-keys = << EOF
git status
EOF

[pkg:haproxy]
[pkg:libslz]
[pkg:makeself]
[pkg:numatop]
[pkg:python-funcparserlib]
[pkg:python-webcolors]
[pkg:ShellCheck]
[pkg:tiptop]
[pkg:vcsh]
[pkg:vmod-querystring]

Looking at my needs I took the problem from the other end. Instead of enabling a programmatic definition of my repetitive windows and panes, I opted for a window template called pkg and applied the template to all my windows. It just so happens that I don’t need to further configure individual windows, but I could do something like this for example:

[pkg:special]
pane.prompt.split = above

In theory I should be able to write this and still send an implicit C-m key at the end:

pane.editor.send-keys = "vim @pkg@.spec"

But the distinction between single- and multi-line properties is missing from the 0.1 release. So instead I’m using the here-document syntax. It’s also possible to use single-quoted strings and here-documents to disable template expansion.

This is arguably over-engineered, but again learning purposes… Also, it’s overall much less engineering than ERB templating (that can embed Ruby code) and as long as it is not YAML, probably anything is better.

The configuration format is decently documented in the tsess(5) manual. The tsess(1) manual documents ~/.config/tsess/*.tsess as the default location for configuration files, but the tsess/ directory is currently omitted.

Working with tmux

Another thing that inspired tsess a long time ago happened by accident one day, innocently running ps. There, I found quite the unexpected line, with a single process from the ps output filling my entire terminal.

Its command line started like this:

sh -c #!/bin/bash  # Clear rbenv variables before starting tmux

After collecting the actual shell script with its new line characters present, I could witness the translation of the YAML configuration into tmux commands and shell if statements. I started looking at the os::exec module and it didn’t take long to find my marks. Like the getopt module it is very well designed, but unlike the getopt module it also has excellent ergonomics, so I was ready to move on in no time.

When it was finally time to start interacting with a tmux server, I read the manual and found the control mode, that uses a simple text protocol to issue commands. Emphasis on simple!

Hare is one of those languages like Rust that only works with UTF-8 strings, so using tmux in control mode would be really no fun if there wasn’t a -u option to guarantee UTF-8 encoding. Thanks to that it becomes really trivial to work with off-the-shelf goodies from the strings and bufio modules.

Working with strings

Another reason to start my tsess project was that I was increasingly itching to work on a parser that would not need to duplicate strings per token. I gave up quickly when I wrote my VCL preprocessor, as a Rust beginner. Writing my parser in Rust was incredibly pleasant, besides this failure.

To explain what I mean by not duplicating strings, consider the following contents to parse:

property = 'value'

The self-imposed constraint I insisted on keeping was to load the contents just once ("property = 'value'\n") and reference tokens in place. In C that’s impossible with null-terminated char * strings.

In Varnish Cache, we use the following data structure:

typedef struct {
	const char *b;
	const char *e;
} txt;

A token has some data, and a txt field to track the base and end of its text in the loaded source. In other words, a token would reference a substring of the source being parsed, in-place. The important thing is of course that a token must not outlive its source material.

If C leaves me to my own devices, Rust on the other hand has the ability to express lifetimes of types in relation to others. So I decided that I wanted to parse tsess’s configuration in Rust with a single heap allocation.

I hit my head pretty hard several times on the advanced lifetimes wall, but I made progress. Approximately a year ago, I read this documentation page on advanced lifetime and that was the breakthrough I needed.

Here is the commit message I wrote to myself back then:

lexer: Let lexer::Eval reference its text

The reason why the first attempt failed was the single lexer lifetime. Working with a &&str reference allows a distinction between the string lifetime and the shorter lexer lifetime.

In fact, an Eval<’s> should now be able to outlive the lexer that made it in the first place, since it’s only bound to the source’s lifetime.

The Eval implementation of the Display trait is replaced with Debug, and now that it carries its own text, it can escape it by itself.

As a result the main loop now looks like what it was meant to look like from the beginning, this time with great success.

And here is the diff of the main loop in this commit:

 fn main() {
-    for eval in lexer::Lexer::from(SRC) {
-        match eval.input {
-            Ok(_) => {
-                print!("{} '", eval);
-                for c in eval.text(SRC).chars() {
-                    print!("{}", c.escape_default())
-                }
-                print!("'\n");
-            }
-            Err(_) => println!("{}", eval),
-        };
+    for eval in lexer::Lexer::from(&SRC) {
+        println!("{eval:?}")
     }
 }

Thank you past me for holding the hand of your future self, me, I couldn’t remember this without you. And I agree with you, the main loop became slightly more readable.

So after becoming a master of 'a, 'b: 'a lifetimes I completed the lexical analyzer and the parser. Eventually, when it was time to load configuration files and stop working with a hard-coded str, I could not figure how to get a &&str from a Box<str> to apply my mighty advanced lifetime scheme.

I tried many things and they all failed, and that’s when I heard about Hare for the first time. Hare is not as nice as Rust when it comes to massaging strings to do things with them, but Hare also doesn’t have half a dozen types to represent strings under various circumstances like Rust. I’m exaggerating on purpose here, but this typically the kind of complications that make Rust hard to fit into one brain without daily practice.

Hare works with UTF-8 strings. The str type is just a []u8 (slice of 8 bit unsigned integers) in disguise with the promise that it is valid UTF-8. Hare insists for the most part to work with iterators when it comes to strings. I’m surprised because it was my understanding that UTF-8 ensures that you always know whether you are at the boundary of a code point, but using indices to get substrings is not common practice. At least the [] operator is not available for str but we can find byte-wise and rune-wise indexing in the strings module.

The preferred way to work with strings appear to be iterators. I wrote my own text module to implement functions like C’s strcspn() on top of iterators and in general whatever convenience would make sense for the parser. I used it for the code generators as well.

The first release

I struggled with a couple things, like Vim rendering mangled during startup and until a pane is resized and it looks like a SIGWINCH was enough. When I first successfully attached to my fedora session, it was only a matter of minutes to migrate my other sessions from tmuxinator to tsess.

The reason why I took care of a couple TODOs in the documentation and performed the release on December 31th was simply to say that this journey to publishing my first Hare project belonged exclusively to 2023. Yes, I’m quite vain.

I’m not expecting anyone to use it, but I will continue tsess’ development and reach the scope I have in mind before cutting a 1.0 release. Just in case, you may find it on Sourcehut.

Closing words

As for Hare, I find it to be an interesting language striking an excellent balance in a spectrum going from C to Rust. I find the cursor between safety and agency to be where I want it most of the time, possibly where I should want it if I knew better. I just regret for now the lack of support for macros because code generation is one thing, but there are other use cases. This is for one may shoehorn generics in C, but I digress. Just like Rust, I will keep an eye on Hare, but unlike Rust I may play with Hare on a more regular basis, starting with exploring tsess until I can agree to make a 1.0 release.

I still haven’t formally introduced tsess, but I will assume that you can get the gist of it after seeing a configuration example and the command line usage. You can also Read The Fancy Manuals to learn more, but not much more.

POSIX interactive stupidity

Tue, 14 Jul 2020

Good old UNIX bashing

I’m a Fedora user, and Fedora is a GNU/Linux system. While GNU stands for “GNU’s Not UNIX” it is definitely a UNIX-like system and Linux is definitely a UNIX-like kernel. There are other UNIX-like systems in the wild, and there is even POSIX: the Portable Operating System Interface. Don’t ask me where the X fits in that acronym, my guess would be to rhyme with UNIX since POSIX is a UNIX-like family of standards that still carries stupid UNIX limitations from a different time in computing. POSIX had decades to fix those limitations, therefore I blame POSIX for the one I will cover today.

But before that, let’s digress a bit.

My stupid dial-up modem

I already mentioned that my dial-up modem is built on Fedora and coming from me that sounds amazing. The sad reality is that like most hardware, my modem’s firmware is crap.

Bullshitware

I can see without a problem the difference between hardware and software. For technical terms it’s a rare occurrence but I find the French equivalent even better. The French word “logiciel” is constructed like “logicware”, which to me is more expressive than software. But software works for me regardless.

Then you have different “kinds” of software, like free software or open source as defined respectively by the Free Software Foundation (FSF) and the Open Source Initiative (OSI). The rest to me is bullshitware.

I suppose it was a trend from the last century to name varieties of software somethingware: freeware (of course not to be confused with free software, I mean, who would fall for that?) or shareware. Then there are more legitimate entries that still register high on my bullshit-o-meter: malware, ransomware or even bloatware.

Finally, my personal (least) favorite is firmware. Firmware is software that controls hardware. I would probably need a whole post to explain why firmware is bullshit, but in one sentence it’s simply software that pretends to be special.

My modem’s firmware

My modem was unusable when I unpacked it. It failed to fulfill its main job, which was to connect me to the internet. I had to call the manufacturer to be redirected to their unmaintained (likely on purpose) support system, to be provided with a firmware update.

The firmware binary is hosted with sources and build instructions. Everything is present except the vendor’s web interface. You can see severely outdated and vulnerable packages. You can even see build instructions, and that’s where it gets really ugly for me: you need Fedora 5 to cross-compile the firmware.

When that particular model was released, I can assure you that Fedora already had two digits in its version numbers. So that is likely an unmaintained tool chain that they keep reusing from model to model. And don’t get me wrong, I’m not complaining about the act of reusing.

Stupid self-hosting

I’m going to need a new firmware update soon, and sadly there is nothing I can do besides hoping for the vendor to provide a patched firmware. My gut feeling tells me that this is a bug I could fix myself if I had access to the source code and the tool chain.

I wanted to self-host a service that turns out to also get a high stupidity rating but one full-blown digression is enough. After all this is a post about something POSIX, I could spend ages before getting to the point.

After some efforts and a lot of complaining I finally have a hosting setup that is almost satisfying. I try to access it from a remote location, and it looks fine. When I try from my laptop I get an invalid TLS certificate.

I configured the modem to forward TCP packets to ports 80 and 443 to the host where my service is installed, so what went wrong?

Stupid firmware

Much like any “modern” modem or set-top box, the modem can be configured via a web interface available on the gateway’s address. From there I can configure the DSL connexion, DNS resolvers, some aspects of the router like port forwarding or DHCP leases.

The DHCP customizations offered are way too simplistic and I had to configure the server with a fixed IP address, opting out of DHCP, to even enable port forwarding reliably.

The thing I didn’t expect was that port forwarding wasn’t based on the IP address but on which side of the modem the request comes from. So inside my network any request to the public IP address is handled as if the gateway IP address was used. I checked the packets with tcpdump and Wireshark.

So when I try to reach my self-hosted service from inside my private network I’m presented with the TLS certificate of the modem’s web interface.

Working around stupidity

As I am waiting for my new shiny support ticket to find a solution, I worked around the problem with iptables. It’s not pretty, but it works. It works, but I don’t want to manually enable or disable the workaround every time I switch to a different network.

It was a no-brainer for me that the solution would be a shell script, that I would manage as a systemd unit. Since systemd has a tight integration with DBus I thought it would be possible to have a unit that would be triggered by DBus messages. I couldn’t find anything of the sort, but I didn’t search hard.

Then I looked at how to integrate with DBus from a shell script that would monitor certain messages: if systemd can’t trigger executions for me I might as well have a simple service where the shell scripts listens to the events I’m interested in. What I could find looked too complicated for my taste so instead of listening to NetworkManager’s DBus API I simply used nmcli.

I’m also not too fond of nmcli, for example nmcli connection monitor NAME won’t tell me when I connect to or disconnect from NAME. It tells me nothing when disconnected and only tells me this three times when the connection is established:

NAME: connection profile changed

Not very useful for what I’m trying to do…

Poking at nmcli I finally find a combination of two commands to capture the events I’m interested in. First, nmcli connection show --active lets me know the current status, and nmcli device monitor wlp2s0 gives me connected and disconnected events for my wireless device.

At this point I’m still playing in my terminal and it’s trivial to process the output of both commands:

# wireless (dis)connected events
nmcli device monitor wlp2s0 | grep connected

# active NAME connection
nmcli connection show --active | grep -q '^NAME\s'

Now that it works I put a script together for my systemd service:

nmcli device monitor wlp2s0 |
grep connected |
while read line
do
	# disable iptables workaround
	if nmcli connection show --active | grep -q '^NAME\s'
	then
		# enable iptables workaround
	fi
done

I start the service and play with my connection while monitoring iptables rules and then… crickets. End of the longer-than-usual digression.

Interactive buffering

Finally, we can look at my POSIX peeve. Why is nothing happening? And why did it work in my terminal emulator?

It turns out to be a fairly easy answer that I knew but forgot after years of not running into this use case. Writing to the standard output is always buffered by default, and line-bufferred when writing to a tty. So what I was seeing from my terminal was very responsive, while the script feeding the while loop was waiting for grep to fill its buffer. It could have waited for a long long time…

Conditional buffering

It’s rather easy to reproduce in a terminal. I can create a small file and feed it to grep along with my terminal input:

$ cat >test.txt
a
b
c
d
^D
$ cat test.txt - | grep .
a
b
c
d
^D # hangs until I close cat's standard input
$ exit

Now, if the output of the grep command is not a tty, events happen in a different order:

$ cat test.txt - | grep . | sed ''
^D # hangs until I close cat's standard input
a
b
c
d
$ exit

The grep command will perform full buffering since its output goes to the sed command’s input.

Standard buffering

At this point I could come to the conclusion that grep is a stupid utility since it assumes that only a tty implies an interactive workload. My problem with this assumption is that my workaround script sits in limbo because of that.

That conclusion would be slightly incorrect because this assumption is made by the C runtime. There is a function called setvbuf() to control the buffering behavior and the defaults for the stdout stream are line-buffered for a tty and fully buffered otherwise.

I can even verify it with a simple program that will pipe stdin to stdout line by line:

#define _POSIX_C_SOURCE 200809L

#include <stdio.h>
#include <stdlib.h>

int
main(void)
{
	char *line = NULL;
	size_t n = 0;

	while (getline(&line, &n, stdin) >= 0)
		printf("%s", line);

	free(line);

	if (ferror(stdin)) {
		perror("getline");
		return (EXIT_FAILURE);
	}

	return (EXIT_SUCCESS);
}

I will call this program stupidcat to stay on brand and it will behave like grep regarding output buffering:

$ c99 -o stupidcat stupidcat.c
$ cat test.txt - | ./stupidcat
a
b
c
d
^D # hangs until I close cat's standard input
$ cat test.txt - | ./stupidcat | sed ''
^D # hangs until I close cat's standard input
a
b
c
d
$ exit

So as it stands, my stupidcat is as stupid as grep, but I could add an interactive mode that would enforce a specific buffering mode and not rely on defaults. That would slightly increase the amount of code in stupidcat.c but nothing scary or groundbreaking.

When sensible defaults turn stupid

My stupidcat is stupid by design, but it’s only here to illustrate that this behavior is not grep’s fault. So all I need now is to tell grep to do line buffering for my non-tty interactive needs.

And that’s where things become really unpleasant. There is no such option, and to be fair not only grep but also sed and cat have this problem. Of all the standard utilities I have used so far only cat has a -u option (which I assume stands for unbuffered).

Oddly enough, GNU cat defaults to unbuffered mode and as a result the -u options is only supported to be ignored:

$ cat test.txt - | cat | cat
a
b
c
d
^D # hangs until I close the leftmost cat's standard input
$ exit

While I tend to be unhappy about GNU extensions because they hurt portability, I’m also not surprised that GNU grep has a --line-buffered option, nor am I surprised that GNU sed has a -u option (sounds familiar) aliased to --unbuffered (sounds familiar) and in that case I have to be in the GNU camp even though more often than not I am in GNU’s camp, but I digress.

What appals me, is that POSIX 2017 still doesn’t have an option for grep and sed, and that the FUTURE DIRECTIONS section of the manuals simply says none. There might be other pipeline-friendly commands lacking the ability to change the buffering behavior, there is one that I know of that I will mention later.

A pseudo solution

Another solution would be to have a program that pretends to be a tty to make other commands buffer on a line-by-line basis. Such a program could openpty or forkpty a pseudoterminal and answer yes to the existential isatty question.

In other words my script could instead look like this:

nmcli device monitor wlp2s0 |
grep connected |
pty | # this line was added
while read line
do
	# disable iptables workaround
	if nmcli connection show --active | grep -q '^NAME\s'
	then
		# enable iptables workaround
	fi
done

It turns out though that such solutions exist like GNU stdbuf or the unbuffer command from the expect project but suddenly you need a TCL runtime. I’m not saying that those programs solve this by pretending to be a tty, but I’m definitely saying that I didn’t find a standard POSIX utility to solve this problem.

To solve this stupid problem, I can use grep --line-buffered connected and be done with it, but what if I want a portable solution to the general problem of interactive workloads not involving a tty?

An awk(ward) solution

A discussion on shell utilities mentioning grep and sed wouldn’t be complete if awk wasn’t somehow also covered given the overlap that can exist between the three commands, to which I will promptly remedy.

As promised, I found another utility that suffers from full buffering if its output is not a tty, and that command is awk. While awk is very different from grep and less but still quite different from sed, my solution was to emulate my grep command with a small awk script.

The WTF moment

A quick search quickly led me to the GNU awk documentation, and as usual you can always count on awk when a utility lets you down. According to the docs all my problems can be solved with the fflush() function:

fflush([filename])

Flush any buffered output associated with filename, which is either a file opened for writing or a shell command for redirecting output to a pipe or coprocess.

[…]

POSIX standardizes fflush() as follows: if there is no argument, or if the argument is the null string (""), then awk flushes the buffers for all open output files and pipes.

Wondering why searching for the “buf” pattern in the POSIX awk manual yields nothing this time I look for “flush” and still no result. I’m not sure where the GNU awk folks got the idea that this was in POSIX, maybe it was discussed and strongly considered, but it’s not in the POSIX 2017 docs.

More awkwardness

Since I was in the neighborhood I kept reading to find what knobs I had to control the standard output and quickly came across a very awkward section:

Output Statements

Both print and printf statements shall write to standard output by default. The output shall be written to the location specified by output_redirection if one is supplied, as follows:

> expression>> expression| expression

In all cases, the expression shall be evaluated to produce a string that is used as a pathname into which to write (for '>' or ">>") or as a command to be executed (for '|' ).

Despite the weird formatting on the POSIX end, I sensed that I was onto something so I check GNU awk and nawk manuals to come to the conclusion that I could emulate my grep statement like this:

awk '/connected/ {print | "cat"}'

In other words, for each line from the nmcli output awk forks a shell to execute the cat command and the cat command will naturally flush its output when it exits. And with that the cat is out of the bag, there is no global solution with awk and if you script has multiple print statements you’ll need to pipe them all! (or wrap that in a function.)

Closing words

I can’t remember when I last ran into buffering problems with a shell script, it must have been at least a few years. The difference between then and now is that nowadays I try to stick to portable code even when portability is not a requirement because I can always learn or rediscover something.

It’s a lot of waste to fork a shell and a cat process to emulate fflush(3) but that doesn’t bother me. The shell script doesn’t run a tight while loop and connecting to or disconnecting from a wireless network is not something that happens continuously. I also love how mentioning in a casual discussion that to solve a silly networking problem I forked a cat from shell in an awk script: it sounds like nonsensical plain English you could find in something like Alice in Wonderland, but I digress.

The GNU awk manual also states that forking a shell should be enough to flush:

awk '/connected/ {print; system("")}'

Even though nawk behaves similarly, and GNU awk flushes without forking a shell for maximum efficiency, I see nothing of the sorts in the POSIX awk manual, so I’ll stick to my guns and ignore this claim.

Lessons learned: non free firmwares are to be avoided like a plague and much like a plague it’s kinda hard to avoid, you can’t always gawk at a manual and trust what it says, and POSIX 2017 failed to identify a decades old common pitfall that can trap even experienced shell script authors.

Hardware tinkering

Sat, 30 May 2020

Hacking hardware

I have never been a hardware person, I may find my way around software but when it comes to hardware I have forgotten most of everything I learned about materials and electronics. It’s a shame because I had a knack for the latter.

And so I have never really hacked hardware or even software dealing directly with hardware, at best I may tinker with hardware. And even when hardware embeds some form of software, it’s so closed that today it’s most likely impossible to get any kind of hardware appliance loaded with 100% free (as in freedom) software.

I’ve had a couple recent hardware incidents that I feel are relevant to the topic of free software. This is the story of computers on my head, in my pocket, in a purse or on my desk and their unimpressive replacements despite the constant march of progress. The story of how hard it is to keep a working computer running.

The computer on my head

The sad thing about computing today is the diligent work from the market to remove computing from any computing discussion. Take the cloud for example, a weatherly word for hosting, probably coined because the word hosting was tainted by the collective idea that it has to suck, with FTP deployments of PHP files in a web server users have little control over.

You will see the media talk about digital and algorithms but not often about computers. Billions of computers were sold as phones, and yet such phones can run general purpose operating systems and often do. Everyday I wear a computer on my head, but we call that a bluetooth headphone.

The best worst bluetooth headphone

I’m usually not fond of naming and shaming but interactions Philips’ customer support were bitter at best. I own a Philips Fidelio M2BTBK headphone, I have used it for years to my satisfaction. It’s supposedly a beefier version of the Fidelio M1BT, twice as expensive, and yet the ear pads turned out to be a bit too fragile. Quick search, found some online, not too hard to change after seeing a video tutorial (you need take a leap of faith to remove them without breaking anything) and all is well that ends well.

The next time it broke, quick search, found nothing, longer search, found nothing. After a couple month ear pads were wearing out to the point where wearing the headphone was starting to hurt the skin of my ears. One last ditch effort, I ring the customer support and get a very helpful person on the line who opens a support case and moves the discussion from phone to email.

The support gives me a link to an online retailer selling ear pads, I order a pair and a couple weeks later it found its way into my mailbox. To my dismay it wasn’t even close to compatible with the headphone. The support’s answer was condescending, barely apologizing and condescendingly pointing the retailer’s return policy to me. This time, not one but two links to online retailers with the correct ear pads. I order to pairs this time, because I don’t want to run into this again next time they break, I want to secure an advance of one pair.

A couple weeks later, the new order makes it through again from China to my mailbox, visiting customs on the way. They definitely look like the ones I ordered myself the first time, and yet I fail to put them on. Looking closer at the product description, it says that those ear pads are compatible with M1BT headphones (and half a dozen other Philips headphones) but it specifically mentions not being compatible with the M2BT. Time to look for a new headphone.

The good, the bad, and the fugly

I had been happy until then with my headphone because it was one of a kind: it sounds decent, can fall back to wired audio, has great battery life, supports audio and phone calls, and can pause and resume audio if interrupted by a phone call. Nothing revolutionary, most recent competing products could do the same so what set it apart for me was the ability to connect to several devices concurrently. Pairing with several devices concurrently was the norm, but connecting to several devices was only something I had seen with bluetooth speakers so far.

The first problem is that Philips stopped selling them, and while I can only assume why, I’m convinced that this headphone has severe bugs. I used to have my laptop and phone connected, so a phone call would pause and resume any player running on my laptop. I achieved my custom setup using playerctl, but I digress…

I have witnessed bugs with the pause and resume feature, which I attribute to pausing directly from the laptop, desynchronizing the “state” of the headphone. My assumption is that it doesn’t base the state on whether the device was transmitting audio, but keeps track of its own internal state machine, via key presses to pause or play. I’m also convinced that software is involved in the design of the headphone and that this logic is not implemented 100% in hardware. Unfortunately there is no way to my knowledge to debug the alleged software, and no opportunity for firmware updates.

Another thing I suspect is that the M2BTBK doesn’t scale. I can connect up to 8 concurrent devices, which really sounds (pun intended) overkill, and bugs probably manifest themselves more often when more devices are competing: only one device at a time can be audible.

For a long time I also suspected the headphone to send spurious “back” commands, sending the player back to the beginning of a media. But having purchased a different product this is still happening to me on Android, which is where it always happened with the M2BTBK. This is very frustrating when it happens in the middle of a 2h podcast but again, I digress…

The worst part is probably that it has the same form factor as many Philips headphones that all share the same ear pads, but for some reason they singled this one out and gave it similar ear pads with incompatible connections. Even the audio jack has a custom connector, that has annoyed me from day one, worrying that if I lost it I would no longer be able to fall back to wired.

Most bluetooth headphones on the market have the same problems, those are locked down products that the average customer will never question. But Philips went the extra mile to ensure that their high-end headphone would turn into e-waste as soon as anything went wrong.

Slow clap.

The unimpressive replacement

After giving up my perfectly working headphone for lack of ear pads, I looked for a replacement and it looks like what made it unique back then still applies today. I couldn’t find an alternative that would connect to multiple devices simultaneously.

Obviously a new factor in my quest was the ability to purchase spare ear pads in addition to all the criteria I had from my Fidelio M2BTBK. I traded one feature with the ability to easily (VERY easily) replace pads with Marshall’s MID Active Noise Reduction headphone.

I had to give up on connecting to multiple devices, which to this day is still a pain in the ass since I frequently switch between devices when I use both but I gained a couple new neat features.

The first one is the ability to purchase ear pads directly from Marshall Headphones, and I ordered a pair along with the headphone. Being able to get them directly from the brand company gives more confidence than having to search online. The first time I found pads produced by a Chinese company just fine, the second time it had become impossible because Philips was no longer selling that product. Why would that company still produce spare parts for a discontinued product?

One very interesting feature from the MID ANC is the ability to get input from both bluetooth and jack audio simultaneously. I have yet to find a use case for this but well, that’s neat. I assume this is a side effect of an actual documented feature: the headphone can send the bluetooth audio via the jack wire, for example to another headphone, someone sitting next to you.

One thing I consider a smart improvement over my previous headphone is the independence of the noise reduction system. It isn’t tied to bluetooth audio and can be turned on or off even for jack audio. Noise reduction itself is not impressive (but I have yet to see one that is) and it’s mostly good with white noise, which might come in handy in plane, but I digress.

On the other hand it has one major (obscure pun intended) anti feature: a single knob controls everything. Pressing the button will turn the headphone on or off. Tapping the button will trigger a play or pause command over bluetooth. Pushing the button forward or backward will send backward and forward (yes, they got it backwards!) commands and interestingly keeping button in that direction will trigger fast backward or fast forward actions. Finally pressing up or down will send commands to turn the volume up or down.

The problem here is again, trying to change the volume in the middle of a 2h podcast is a recipe for disaster so no, less isn’t always more, having more controls on the headphone would have made it more usable. The worst part is that those control don’t feel like up/down or forward/backward with the headphone on but rather like diagonals, adding to the challenge, but I digress.

What I’m almost certain of is that this knob is the first thing that will physically break. I’m not counting ear pads because I can replace those. And I miss being able to play/pause with my shoulder when my hands aren’t free.

One more feature is the ability to double tap the button to invoke Apple’s stupid voice assistant, but on an Android phone (more on that later) it does nothing. Again, I’m convinced that some kind of software is embedded in the headphone and it would have been nice^Wdecent to have access to source code and be able to patch something different in, but that’s never going to happen…

Locked all the way down

Unlike last time when I purchased my Fidelio M2BTBK (and I don’t buy a new headphone every other year) I noticed that many manufacturers advertise high fidelity audio with aptX. I can only assume that Qualcomm® is aggressively marketing it since even my old headphone supports it and yet I never heard about it back then.

Digging a bit, it looks like Android supports it out of the box, but on my laptop there is no way Fedora would allow the PulseAudio plugin to be distributed. A quick search later I found the right package to install and my laptop was finally able to sound as good as my phone via bluetooth.

And at this point I can only assume (because I don’t care enough to verify) that aptX is patented, which by definition means that the codec description is publicly available and free software implementations can be distributed in countries where software patents aren’t a thing, like mine.

At this point I’m not sure whether I even own half of the headphone, but less than a month after switching headphones another kind of lock down started hitting the globe.

The computer in my pocket

I often carry a special computer in my pocket. What’s so special about it? Maybe the fact that it can make phone calls, or maybe the fact that it belongs to one of the most locked down family of products:

carriers may lock what kind of SIM card is accepted
phone manufacturers may lock the bootloader
the baseband processor runs its own operating system
- sometimes to work around legal requirements
the user-facing operating system doesn’t grant full privileges
- and I’m not suggesting we need that for regular operations
it may not be possible to install third-party software
it may not be possible to remove unessential applications
the multimedia stack may enforce DRMs

And if you go the extra mile you might find that:

components trend toward not being removable
- even batteries can no longer be replaced once they die
MicroSD card slots are a thing of the past
dedicated chips may prevent alternative OSs
screws may be non standard

The list probably goes on much much further than I can think of, but the gist of it remains the same even when after years it’s time to find a replacement.

The software in my pocket

It became apparent quite early in the commercial life of Android that things weren’t looking good on the software front. Manufacturers would come up with their own graphical layer on top of Android to set themselves apart from the competition and at some point it became common wisdom (in my computing circles at least) that only Nexus phones would offer you a proper experience across the board.

Besides fiddling with the OS itself they might also pre-install applications. Carriers might add yet another layer of OS changes and even more applications, and sometimes they would even deface preexisting applications. All that crap would significantly increase delays for important software updates.

I had an HTC Desire purchased via Orange at the time when I decided that my next phone would be a Nexus. I was able to compare it to a stock HTC Desire and that’s when I realized that the audio player was missing an important feature that had been awkwardly be replaced by Orange’s online music store. I did not hesitate to flash my phone with a stock HTC image to get rid of all this crap.

I also was able to compare it with a Nexus One (and Wikipedia tells me there are more differences than I thought) but before I could even do that I left Android for the not-so-happy world of Firefox OS.

The web is the platform

A new contender on the mobile market took me by surprise, it was Mozilla. One of those alternatives to Android and iOS that never survive, because most popular applications are not free and not interested by free (still as in freedom) platforms.

This system ticks all the boxes that should have driven me away. The market was already saturated by Google and Apple, leaving room for nobody else. The whole operating system was basically a browser engine and the user space so to speak was made of web applications (I refer to myself as a web hater). The system itself was based on AOSP and thus very dependent on Android.

And reviews were overall poor.

Some people rightfully noted that while Mozilla was (legitimately) complaining about iOS’ arbitrary restriction that forced them to use a different browser engine, they also built a competing system that made it impossible to even have a different browser. The scariest thing Mozilla did was to reinforce the idea that the web works hard at superseding the underlying operating system, but I digress.

Then a few things happened: I was able to take part of a hackathon at Mozilla Paris’ and play with a Firefox OS Flame phone. A Firefox OS emulator directly available in the desktop browser turned out to be very usable. Even better, I had a work laptop with a touch screen.

So in 2014 my venerable HTC Desire was replaced by a Firefox OS Flame phone which (to my surprise) I did appreciate. I wanted to believe in it, and I even considered JavaScript development to build applications. That break from Android was actually beneficial, I really enjoyed my time with my Flame phone but eventually parts of the screen started to become unresponsive and Mozilla announced the death of the project.

Nexus pain

I needed a new phone but Firefox OS was no longer on the proverbial table, and iOS never stood a chance in that regard. There were other systems then, but nothing convincing enough to take another leap of faith. Back to Android then, and finally back to the common wisdom of sticking to a Nexus phone.

There was one problem though. The market kept moving to larger phones to the point where “phablet” was coined: tablet-sized phones that had the unique feature of turning a simple phone call into an awkward scene. While not as ridiculous the Nexus 6 was larger than the Nexus 5 that was in turn larger than the Nexus 4 that was in turn larger than my Firefox OS Flame, itself much larger than my HTC Desire.

I purchased a Nexus 4, only because I found one of the last new models. One year later I ordered a very cheap refurbished Nexus 4, but let’s not jump ahead too much.

The main pain point besides finding a “pure” Android phone that is not too large is simple: updates. Turning on an old phone for the first time, I had to wait for several OS updates before I could start using the device. Thanks to a new trend of catchy names for security vulnerabilities I was aware that my device was affected by Stagefright and disabled MMS support. I also reviewed past Android security bulletins and to keep track of future ones to make sure to disable as many things as possible.

Going back to Android was already painful enough (which again, surprised me considering Firefox OS’ limitations) and having to deal with that kind of crap didn’t help. Since my phone was no longer supported it was hard to tell most of the time whether vulnerabilities were relevant or not. Then Blueborne was too much, I needed to look at alternatives.

Android in Wonderland

Alice wants to establish a secure connection with Bob, but both of them have Android “smart” phones. What can they do about it? Nothing. The closest is to keep devices up to date, but eventually they run out of support, incredibly fast actually. Welcome to the world of non free software with a tint planned obsolescence.

But AOSP is free software, right?

Indeed, but AOSP is not Android. And most manufacturers ship Android devices. But AOSP is free software, so there are mostly-compatible alternatives to Android. Unfortunately at the time CyanogenMod was in the middle of a storm that gave birth to LineageOS and overall it didn’t feel like the right moment to join either bandwagon.

Around the same time, a new player based on AOSP appeared out of the blue. Ubuntu Touch was a hybrid system that could turn phone into a desktop workstation once plugged (wired or over the air) to an external display and a keyboard. Canonical had “invented” the Convergence®©™ that everyone else was already pitching as the future of mobile. And like Firefox OS this exercise in futility didn’t last long.

The prospect of flashing a different OS was quickly overtaken by events, and not setting up an automated reminder didn’t help. To make matters worse, Google threw the Nexus common wisdom down the shitter by replacing its Nexus product line with Pixel devices. I’m not interested in high-end phones, only usable ones with as little as custom software as possible and this was no longer an option.

I can’t remember how that happened, but I found a very cheap refurbished Nexus 4 as I mentioned earlier. The USB port on the phone was in a poor state, and I had to bend the cable to help it recharge the battery. When I was offered with a refund I changed my mind and decided to keep the spare phone for spare parts just in case I’d keep the device alive a while longer.

What a foolish reasoning…

The unimpressive replacement

As I was back to using an outdated Android system on my Nexus 4, I was looking for a sustainable replacement from which I could squeeze years of mileage.

Thanks to Greg Kroah-Hartman who brought one during a conference I was able to look at concept device for Project Ara, a sort of pocket computer that would work like a PC in which you could change parts as you wish, to some extent. An interesting take on upgrading phone parts each at their own pace. The parts were overall large and as I remember it the assembled phone was too big for my tastes. I wanted to believe this project had a future but as far as I was concerned the writing was on the wall, it would go nowhere. Sadly I was apparently right.

The other thing I was aware of was the Fairphone 2. I don’t think I had heard of their first device but the second one made some heavy marketing noise and partnered with iFixIt who gave them a 10/10 rating for their second device. Fairphone, it turns out, is trying to change the electronics industry to move towards longer-lived devices and a better treatment for all people involved in the supply chain. So you essentially buy an expensive device, but instead of paying with a bit of money and a lot of blood, you pay slightly more money than a comparable device would cost with slightly less blood on the balance.

There were rumours that a Fairphone 3 was around the corner but I bought a Fairphone 2 nevertheless as my next device. I had a ton of updates that brought me from Android 5 to Android 7, a huge leap forward, and since then I have seen a handful of minor updates including security fixes. It’s far from perfect and I had to remove the camera module to perform device encryption but iFixIt didn’t lie: even with my crappy screwdrivers (two of them were enough since most screws are the same) I was able to tear it down and build it up again. Twice of course, to put the camera module back, and apparently my reward for being late in the game was that I ended up with the enhanced second version of the camera module but its kernel driver prevents the phone to reach the needed state for the encryption process, and as usual I digress.

Other than quirks that a technical savvy person like me can bear with, the only real problem with this phone is that it’s even larger than the Nexus 6 I thought was way too large… The software issues the Fairphone 2 faces today are mostly related to the lack of support from Qualcomm® that forces the team supporting that device to operate blindly, or so I’m led to believe. But I think this is to date the least closed computer I have put in my pocket.

And by the way yes, Qualcomm®, the same company that makes the bluetooth chip for the computer on my head. It’s a small world after all.

The computer in someone else’s purse

My girlfriend’s Android phone had been misbehaving for a while and when I switched phones she considered buying one too. After leaving two lives before and after my Firefox OS escapade I suggested that maybe it could start a third cycle and lo and behold, she accepted.

She could bear with all the crap I had diligently removed over the years mostly because most of it was invisible, and because MMS didn’t matter since she has Apps®©™ to send or receive pictures. She did notice that I had lent her a phone with not many apps, mostly because I had disabled many of the unremovable apps, including system apps. With Wi-Fi at home the lack of 4G connectivity went unnoticed for a while. Overall a device that doesn’t crash the foreground application every 10 minutes probably felt already like a huge improvement.

That reminds me that I was really not happy to see that I could no longer disable system applications on my brand new Fairphone 2. I had gotten used to removing a lot of cruft, but it turned out that they are no longer visible by default and it’s easy to show them. As a result I have a very small list of applications in the launcher, despite installing a few of them from F-Droid (some of them replacing disabled applications). My phone doesn’t react if I tell it “OK Google” and overall battery lasts a bit longer, but I digress…

Later on a few things happened: Fairphone released a third device, and half of the human population (allegedly) went on lock down.

Smack My Snitch Up

In the middle of its third life, my beloved Nexus 4 died after 6 years of interrupted service. Its screen was accidentally smashed, and my initial thought was that it was fine. I have seen many people use a phone with a broken touch screen and it’s all right. Let me unlock the phone with my thumb, why does this hurt? Why can I see small cuts on my skin? Oh, it’s as good as dead.

I treat my computers with a lot of respect. I would normally say care and respect but I’m so clumsy that care would be an overstatement. While the intent is here, I’d say I accidentally dropped that phone probably a hundred times. My girlfriend drops it once after several months, only once, and she breaks it. My luck probably beats my clumsiness, according to statistics.

We quickly set contingency measures in motion, ordering a new phone and pulling her old phone back from the grave. One problem though, we needed to back up the contents of ThatOneApp™ and restore it on the active phone to avoid losing StuffThatAppManages™ since it doesn’t support concurrent access on multiple devices for a given account. Yes, that application is certainly not free software, and not something I use myself.

Cornered as I was, that’s when I remembered my spare Nexus 4 from which I would salvage parts (like a screen) if something were to happen. She agreed to stop using ThatOneApp™ for the time being and give me a chance to save the day.

iFeelStupid

You don’t find guides to tear down all devices on iFixIt and that’s one more advantage of Nexus or Pixel devices, they tend to be covered. The Nexus 4 has a 7/10 rating for repairability, therefore it should be fine.

Of course it is not.

My crappy set of precision screwdrivers is really really crappy. There are screws I can’t even remove. Compared to the Fairphone 2 the more the progress the more I feel that I’m not going to make it. And of course there’s this ominous comment from the guide:

The glass is fused to both the display and the display frame. So don’t crack the glass unless you’re good with a heat gun, or you’re fond of replacing the glass, display, and frame together ($$$).

As I was sinking into moderate despair, I decided to take another leap of faith. I would order screwdrivers from iFixIt and give it a try, which I found a bit expensive. On the other hand they provided me with a free guide (not counting the Fairphone 2 guide since the companies partnered) so I would bear with it. On second thought I wouldn’t, for a few more euros I could get their Essential Electronics Toolkit with all the individual tools I needed plus a lot more.

iFeelBetter

Disclaimer: I’m not sponsored by anyone to praise iFixIt, I’m just a happy customer.

I would advise anyone unsure whether they could tear down an electronics device to order at least that kit. Trying their screwdrivers I came to the conclusion that my cheap kit was crappy. Tearing down the Nexus 4 became trivial. The guide doesn’t say much about connectors, but in the end they were easy to remove and put back. Based on my experience with a 10/10 and a 7/10 device I’m tempted to say that 6/10 is probably quite hard to deal with, at least for someone like me.

After a quick search in the basement I dug up my spare Nexus 4 and moved the motherboard from one frame to the other and put everything back together. When I tried to turn the phone on none of the batteries seemed to make a difference and trying to charge the phone with either battery didn’t work. Only then did I remember what was wrong with my spare device. Tear down again, then swap daughterboards, rinse, repeat. Now it charges!

Then the mixed feeling of relief and something hard to explain when the device finally turned on, and we were able to solve the problem. The Nexus 4 was brought back to life but remained undead with no real purpose. Had I known that I would be able to do that, maybe we wouldn’t have purchased a new device.

The unimpressive replacement

I managed to convince my girlfriend to get herself a Fairphone 3. While I admire Fairphone for what they envision and want to congratulate them for what they achieved so far, I was nonplussed by their latest entry.

It’s again larger, a trend that seems impossible to fight, and modules appear not to be compatible with the previous device. I get that they have to source components somewhere and chip manufacturers move on, but that’s a device even more hungry for power than the Fairphone 2 and that’s disappointing. They make very clear that the majority of resource consumption happens in the supply chain for a device, but I would hope that they’d keep devices frugal. Also did I mention how big it is?

I can see that Android is making strides and the most noticeable change is the update system that applies while the device is running and switches to the update during the next boot. Other than that, I performed the ritual of disabling all the things I would identify as unremovable crapware without asking for permission and presented the device as ready to be used.

The undead computer on my desk

Now that I had a working Nexus 4 in limbo I was wondering what to do with it. The spare parts went back to the basement, in the box containing the HTC Desire and the Firefox OS Flame. That’s when it occurred to me that I could give a try to an alternative operating system.

The Android clone

LineageOS is still alive and kicking, and only CyanogenMod appears to be scarred by the controversy dating back to the hostile fork. Looking at the latest LineageOS release, and the latest Nexus 4 build lagging what looks like two major releases behind I was not feeling very confident. It would likely improve the situation but also freeze it in time, not much of an improvement.

The Flame still burns

Firefox OS was abandoned, but from its ashes KaiOS was born. Let’s look at the list of supported devices… Oh, no Nexus 4, not even my Flame phone? Next.

The GNU/Linux mad house

There’s one phone designed to run a general purpose Linux distribution, with specialized applications nevertheless for the phone factor to my knowledge. However I’m pretty sure there are no plans for PureOS to support the Nexus 4 hardware.

The other mad scientist project is PostmarketOS and I really recognize myself in their pitch:

We are sick of not receiving updates shortly after buying new phones. Sick of the walled gardens deeply integrated into Android and iOS. That’s why we are developing a sustainable, privacy and security focused free software mobile OS that is modeled after traditional Linux distributions. With privilege separation in mind. Let’s keep our devices useful and safe until they physically break!

Of course I don’t belong to the OS development part, but overall I agree with their goals. The technical details are also very interesting and development seems to bear much less overhead than what I expected. Oh and it works with Nexus 4 hardware!

Except that working is an overstatement, and it is not working with a mainline Linux kernel. They seem to be stuck with whatever source code we had during the device’s prime. It hasn’t reached the point where I could participate, period.

Not a GNU system, not a mainline Linux kernel, next.

But hey, I want to see both PureOS and PostmarketOS succeed!

The impressive replacement

Last try: when Canonical pulled the plug on Ubuntu Touch, it seems that some people really liked it and formed a community to continue its development. I heard a lot about Ubuntu Touch when Canonical was trying to enter the market, and the user interface sounded weird, but I never bothered to look at their concept of scopes.

The installation process was truly impressive. The installer is not available in Fedora repositories but a Snap package is present and recommended. Snaps had never worked for me on Fedora but maybe this time it would work. It did, and the installer was able to detect the Nexus 4 without a glitch. It took me around 5 minutes to install the snapd RPM, install the UBports Installer snap, flash my device and remove UBports installer and snapd from my system.

Truly impressive.

The version of the system was OTA-11 (which I assume means Over The Air update 11) and while I don’t understand the scope concept that didn’t stop me from installing applications and finding my way around. Navigation was a bit confusing, but I’m fully aware that I was primed by Android experience. I think it’s mostly fine, except that the back button is often around the top of the screen which can put some strain on the thumb. Ubuntu Touch on my Fairphone 2 would probably kill my hand on the spot.

The system was very responsive and I haven’t seen a glitch yet. At the same time that’s not my daily driver. But installing and updating applications worked just fine.

A couple weeks later I noticed that OTA-12 was available but it showed up as “Version 10” in the settings. The upgrade went smoothly, the only nitpick I have is the lack of information about what’s happening besides 3 blinking dots. The changelog is impressive and apparently scopes are gone, but I didn’t notice a major difference. I may never know what the fuss was all about.

I have yet to give the Convergence®©™ a try and for now my Nexus 4 will stay undead but it might start its fourth life at some point. Everything I tried worked like a charm, so finally this old device has up-to-date software. Or does it? The device build description seems to imply that the Android kernel was compiled in user debug mode, whatever that means, it dates back to 2016 and belongs to the 4.4 series.

That doesn’t look reassuring and I couldn’t find a definitive answer after a quick search, sigh. On the other hand I didn’t ask anyone from the project mainly because I don’t want to waste anyone’s time with my petty questions unless I’m planning to seriously ramp up my Ubuntu Touch usage which is not on my agenda, yet.

Oh and of course I had to download and try a game on the first day, and I was able to get my best score ever on the first try:

I wish UBports the best, they seem to do an amazing work and I really feel that Canonical was onto something with Ubuntu Touch.

The computer in my closet

If I managed to keep my Nexus 4 alive (though currently undead) for that long, with great difficulty, I can’t say the same for the laptop in my closet. This one didn’t get the basement treatment but it’s close enough.

Non free software end of life

My girlfriend has her own laptop, and it’s “powered” by Windows 7 (if you ask me I’d rather say “crippled”). For almost a year now we were in parleys over the incoming end of support and what to do next. She didn’t want to upgrade to Windows 10, but it would also be hard to convince her to use “my Fedora thing”.

In January when Windows 7 finally died we agreed to make an inventory of what she currently uses on this computer (not much besides a web browser and an office suite) and make sure the alternative OS would cover the current needs. Then we’d try a live CD to check hardware support and if stars aligned the next step would be a dual boot setup.

Since she’s a teacher we decided to follow the plan during the next school holidays, but before that could happen we’d have to deal with the COVID-19 pandemic. I didn’t have a contingency plan for that.

The impressive replacement

As she began to teach kids remotely problems started to crop up. The last straw was a microphone problem I couldn’t solve: on battery everything would be fine but once plugged a constant noise would be picked up by the microphone and it apparently only happens for this remote teaching web application.

Running out of ideas I had to resort to creative thinking again. I remembered having my 2011 laptop computer in my closet, so maybe it was worth a shot? Last time I had used it was around two years ago, so I had three major Fedora upgrades to perform over one weekend. They took forever, the hard disk drive is probably not going to last long but there is nothing important there. Since this is not a Nexus 4 device replacing that part will be easy when push comes to shove. On the next weekend I upgraded to Fedora 32, and for one week she had access to a perfectly fine Fedora 31 system.

Besides the slow, very slow hard disk, slowing down everything from boot to packages updates or first launches of applications, it is very, very, very (I insist) responsive. It might certainly not fit the bill for AAA gaming, and none of my devices do but it otherwise works almost perfectly fine. The only problem I encountered is the halting restart. Shutting down the computer works fine, trying to restart it halts it, in a state waiting to be powered down using the hardware switch. And of course it’s hard to get two hours of battery life without doing anything productive, and while it’s removable I’m not sure I can find a compatible replacement. Therefore this laptop can only be used like a good old desktop computer, but at least it has none of the problems the Windows laptop has.

Shocking: none of the hardware is supported by LVFS…

Luckily for me, two years ago I installed MATE in addition to Xfce and left MATE as the default even though Xfce is still my desktop environment of choice. And MATE seems to be intuitive enough for her, at least for me it’s slicker than Xfce but somehow utterly fails to convince me to switch. But I digress.

Computing being computing, maybe installing Fedora on her laptop won’t make a difference. Maybe some power chord is too close to the microphone’s cable and it creates hardware interferences? Or maybe it’s a software problem since it only happens with the remote teaching web application and not with other kinds of audio or video conferencing software?

I’m willing to bet on Fedora, fully aware of my own bias. This laptop has nothing really impressive for itself. What’s really impressive is that, who would have guessed, computers can last and we don’t need to replace them too often. Sadly laptop computers today are even harder to repair and have less moving parts, less parts that can be individually replaced. One would argue this is a requirement for very slim and light devices, but I think this is only an excuse to justify more planned obsolescence. And as professional dealing with servers we worry a lot more about hardware faults or bit rot and tend to forget that devices with less intensive usage can last longer.

Wrapping up

The positive result of my various hardware experiences of the past few month is that I still have some degree of liberty for a small subset of my hardware and I no longer need parleys to exorcise Windows from my home.

I will keep an eye on Ubuntu Touch and PostmarketOS, but really I would prefer to run Fedora on all my devices. We also have one more Windows device, a netbook, more than ten years old. If it has an x86 32bit processor, Fedora (or even any Linux-based system) might no longer be an option. We’ll see what we do with the computer in the drawer, but not today.

I’ll say it once more, if you think you might hypothetically need tools to tinker with electronics devices in the future, don’t wait until midden lands on the windmill. My recommendation goes to iFixIt for the quality of their tools and the helpful repair guides, whether they originate from a partnership or were authored by individuals not from the iFixIt staff.

And this longest post (to date) was also the one that took me the longest to write, over two months. As events unraveled I ended up with much more to tell than initially planned. The smaller the device, the harder it is to keep it alive over the years, and that’s the sad reality of computing today. Some companies, under the cult of innovation even treat hardware as a commodity and the market gives the same incentive to everyone else, but there are still a few folks trying to reverse or slow down the trend. For this reason I’m also keeping an eye on the /e/ project (but seriously next time work on a better name) and it supports the Nexus 4 but like LineageOS it seems to be stuck a couple major AOSP releases behind.

And well, I could keep rambling on this topic so maybe next time I will talk about the other computers on my desk like my NAS or my docking station that is simply a computer in disguise, my bluetooth speaker that can take firmware updates or my dial-up modem built on Fedora, or even my main and current laptop. Spoiler alert, none of them are any less closed and I’ll probably use the word crap or its variants as many times if not more.

Xfce presentation mode

Fri, 04 Oct 2019

Friendship

I remember from when I joined my current $DAYJOB, the CTO looked me in the eyes and explained that I could choose my own work environment with a caveat: if I picked Windows the rest of the team would be instructed to always point a finger in my direction and laugh whenever they see me.

I don’t remember my exact reaction, but I was probably trying hard to suppress a smile, maintaining a poker face and replying with what I hoped was a casual “cool”.

Today I’m writing about one of those topics that can end a friendship like a favorite operating system, text editor or desktop environment.

So of course my favorite operating system is Fedora, but I’m not using Fedora Workstation because Gnome 3 drives me mad whenever I get to use it. I started with Gnome 2 and was very satisfied with it, but I would run into magic corners and other features that weren’t compatible with me. I tried Unity and actually enjoyed what they were pushing but again, small usability quirks that didn’t work for me on a daily basis eventually led to an accumulation of frustration that was unbearable for me. And incidentally I was using Unity at a previous $DAYJOB and didn’t have much of a choice.

So instead I quickly settled for Xfce. It hardly gets in my way, it’s sort of minimalist and yet featureful, and its overall architecture makes sense to me on paper: I have no experience with desktop environment development. Except that of course it also manages to drive me mad under certain very narrow circumstances when I’m giving a presentation or training a team: when I press Alt+Tab to cycle between open windows it shows up on the presentation screen or projector.

Please note that any harsh words are not directed to a group of people, but merely stating how I feel about software. Let’s stay on friendly terms, pretty please 😊.

(My) Xfce in Fedora

While I’m a big fan of Xfce, I think that the stock experience leaves too much to desire. Thankfully I have all I need on Fedora to fix everything I don’t like. For example I use the Whisker Menu and previously used Docky to replace the dock-like panel from the stock setup until it went defunct. Actually further than its retirement, I used it until it would block a major Fedora upgrade. That’s when I discovered Plank that thankfully worked as a drop-in replacement. Despite my distaste for Gnome, I still prefer their display manager over Xfce’s, so I happily take what works for me. I know, I’m praising the same folks who make a desktop environment that “drives me mad” to quote myself.

I switch themes every couple years if I manage to get enough bored to look for a new one so all in all Xfce despite a bad stock experience, it is with little customization effort that I got myself a very serviceable desktop environment that has pleased me for a while now, including some of the stock applications.

A year ago I tried the MATE desktop and found it amazing and a lot more polished than Xfce and yet it wasn’t enough to make me switch. I also ran into a lot of bugs when the Xfce maintainer for Fedora polled our mailing list to find whether we should follow the unstable 4.13 branch. A lot of bugs here means a couple, because yes, even running into a few bugs is more than I’m used to with Xfce. This was also their transition from GTK+ 2 to GTK+ 3, and others on the mailing list were less lucky and ran into many severe bugs.

But I digress here, now that Xfce 4.14 is available on Fedora 30, I’m saw what’s on the other side of the 4.13 tunnel and while it was pleasant to get the new features incrementally, I didn’t get one small change I’ve wanted for a few years already. I had a short window of free time where I felt I could look into that, and so I did.

Hacking Xfce (on Fedora)

To be clear, I’m mostly documenting this for myself just in case I revisit Xfce in the future. If you find this helpful, don’t hesitate to let me know.

Trial and (mostly) error

My first thought was to build my own RPM in /usr/local and change some systemd configuration to point to /usr/local/bin/xfwm4. I could easily revert that change without network access or further packaging update (likely a downgrade) if I ended up bricking my desktop.

In hindsight I’m really happy I couldn’t figure how my system knew which window manager to pick (if you know, feel free to email me) so I decided to look at Xfce’s documentation and found nothing really helpful. I did come across xfce-test and besides being in unfamiliar Ubuntu territory (I only realized later that a Fedora 29 branch exists) it also was denied execution by the default SELinux policy. I might shave this yak someday, but this was enough to convince me not to engage this route: besides pleasing SELinux I ultimately needed to also learn how to write automated tests.

My next step was to try inside a virtual machine, something I for some reason have trouble doing with a Fedora guest. I turned to my familiar tools, Vagrant and Virtual Box. A quick search taught me how to start a VM with multiple monitors (something I wasn’t even sure xfce-test supported) and looking through Vagrant’s atlas I found official Fedora images hosted by the Fedora Project itself (yay!) but the only one available was the Cloud (meh!) edition.

First success

I started a VM, struggled to install Xfce (more on that later), struggled again to install my modified copy of xfwm4, and then checked that it contained my modifications. At this point it was only an arbitrary string checked with the strings and grep utilities. So far so good, time to reboot the VM and check that this no-op change didn’t break everything. I’m prompted for a text mode login, and from there I have no idea how to actually start Xfce (much like I have no idea how xfwm4 is started). A quick search without leaving the VM and I quickly find the magic incantation:

sudo systemctl start graphical.target

And voilà, in my two VirtualBox windows the Xfce wallpaper shows up and after a couple eternities the desktop is fully functional! Time to automate the process, which is easily done with the following Vagrantfile:

$install_xfce = <<-EOF
dnf -y upgrade # somehow fails
dnf -y upgrade # succeeds
dnf -y group install 'Xfce Desktop'
EOF

Vagrant.configure("2") do |config|
	config.vm.box = "fedora/30-cloud-base"
	config.vm.provision "shell", inline: $install_xfce, privileged: true
	config.vm.provider :virtualbox do |vb|
		vb.gui = true
		vb.customize ["modifyvm", :id, "--monitorcount", "2"]
	end
end

You may notice that I needed to upgrade the package set twice to bring my VM up to speed. Initially I didn’t understand why, and I didn’t understand why I couldn’t install my xfwm4 package. The answer was unpleasant.

Fedora constrained

I should probably mention that I was slightly sleep-deprived as I was spending my first evening trying to crack this nut. I made my first visible change to the window manager and still didn’t realize that something was very wrong:

DNF crashed with a mere “Killed” in its output
Xfce took forever to start default applications
Once started the UI was functional but unresponsive

Several brain cycles later it occurred to me that the VM wasn’t fundamentally slow, it was just swapping like hell. A quick inspection shows defaults of 512MB of RAM and a single CPU with that Vagrantfile.

I find it sad that DNF is so resource hungry that it can’t both update its index and compute a transaction with limited resources. I’m wondering how the Fedora IoT project copes with that, maybe they use rpm-ostree instead. One more thing that happened with DNF is that I couldn’t install my modified package:

sudo dnf upgrade /vagrant/xfm4*.rpm

No matter what I would try, it would crash. So instead I opted for rpm -U and it happened almost instantly. DNF is way too hungry, and I understand that it adds value on top of RPM, but I can see the spectre of bloat looming above. DNF is a black hole: when RPM transactions enter time flies around them and resources get sucked in by its gravity.

Clunky workflow

Once I realized what was wrong, I was ready to start hacking in the source code. I wrote a small shell script that would build a make dist archive from source, build an RPM, upload it to the VM using SSH and upgrade the package. Then with a simple restart of the VM I could give a try to my changes. In this case I made a change in a label, just to validate the workflow. I updated my Vagrantfile to provision a usable system:

$install_xfce = <<-EOF
dnf -y upgrade
dnf -y group install 'Xfce Desktop'
EOF

Vagrant.configure("2") do |config|
	config.vm.box = "fedora/30-cloud-base"
	config.vm.provision "shell", inline: $install_xfce, privileged: true
	config.vm.provider :virtualbox do |vb|
		vb.gui = true
		vb.memory = 2048
		vb.cpus = 2
		vb.customize ["modifyvm", :id, "--monitorcount", "2"]
	end
end

At this point I don’t need to ask for an upgrade twice… I specifically don’t enable the graphical.target unit in systemd because I don’t know how to test the changes without a restart, so for a given VM I get one free try without a restart. In theory I only need to provision the VM once before I iterate with my changes, but in practice I rebuilt it from scratch a couple times before and after I was completely done.

The only problem with this workflow is that VirtualBox’s Linux kernel modules aren’t all part of the mainline releases, and because of Fedora’s policy on out-of-tree modules I get VirtualBox from RPM Fusion but I digress…

Working with xfwm4

This was not my first dive in the xfwm4 code base, I tried to look at this problem a couple years ago and didn’t get anywhere. I never really had enough spare time I was willing to allocate to this and it had already taken me some time to figure which one of the Xfce projects was responsible for the Alt+Tab window. This time I was ready to give it all my spare time until the problem was solved or until I’d throw the towel. After one evening spent on setting up a test environment, I was pleasantly surprised that it would only take me one evening to write a satisfying patch.

All out war

If a discussion about desktop environments or text editors can break a friendship, arguing about code style can lead to war. Even simple questions as indentation can become complicated beyond just spaces vs tabs, because I mean… How many spaces? Where do you align line continuations? For languages with a C-like syntax, where do you open curly brackets? Some code styles like FreeBSD’s simply troll you by having both spaces and tabs, open curly brackets both on the same or next line etc… And yet it’s the C code style I enjoy.

Thankfully, xfwm4 doesn’t follow the GNU code style for C, which I find deeply unreadable and that is probably a pain to write unless you have very good support in your text editor. I’m not fond of xfwm4’s style, and especially the inconsistencies across files, but it was fine.

Working without safety

In recent years I have been spoiled by my $DAYJOB where I work on a code base with extensive functional testing, and between 5 to 10% of the code being assertions shipped to production. This contributes to increasing my confidence in making changes, because many mistakes tend to manifest themselves quickly during the development process and in a helpful way. And of course once you build enough domain knowledge it becomes easier to deal with an increasingly familiar code base.

So leaving my comfort zone to work on a window manager did not bring much confidence to begin with. Building the code and then running make check doesn’t seem to do anything. One more reason why xfce-test wouldn’t help is that even if I built an SELinux module to deal with the denial I wasn’t even sure I could even deal with emulating multiple monitors.

And yet…

The code wasn’t that hard to browse and adding this feature only took a second evening, two in total. Working without my beloved seat belts (plenty of tests and assertions) turned out to be fine even though the project was throwing a lot of unfamiliar things at me: new code base, GTK+, Glade etc. In no time I was able to piece things together with reasonable confidence in the absence of automated and comprehensive testing. I need to stress this point: the GTK+ documentation was easy to browse (winking in the general direction of the Varnish project) and I found all my answers on the first try. I think I visited 3 pages of API descriptions in total, way to go!

Working on Fedora

One thing I love about Fedora is its packaging policy, and in general working with tooling that allows me to easily build and install my own packages and not feel that I’m throwing random files in random locations that I may forget about after a couple days.

I eventually installed the same RPM I was building for the development VM on my own system and it just worked fine also on bare metal. After submitting the patches upstream I decided to build the xfwm4 RPM that Fedora ships and simply incorporate them in the RPM spec. It didn’t work…

Being familiar with autotools, I quickly realized that a make dist archive wouldn’t consider the updated Glade files, since those are meant to be processed by maintainers. The solution was to --enable-maintainer-mode in the ./configure invocation and bring in the missing build dependencies. The point of processing them beforehand being that downstream maintainers don’t have to bother with architecture-independent build dependencies.

With that I was able to build a COPR repository and make it available for anyone to give it a try on Fedora very easily. Unfortunately I collected no test feedback, and the patches have been sitting in Bugzilla limbos ever since I submitted them, but it seems to be the case for a lot of xfwm4 tickets.

Future contributions

I’m still impressed by the tools we get for free and how it has become possible to easily test a change to an operating system without risking to brick a device. At least in this very case.

Packaging software for Fedora has taken me outside of my comfort zone more than I can remember, and after this first contact with Xfce code I think I may attempt more drive-by contributions.

TLS sandboxing

Wed, 28 Aug 2019

Reinventing TLS

After being a month away from $DAYJOB I couldn’t help myself not thinking about TLS sandboxing. But with limited spare time to give this proper thought I had to come up with something easier to grasp and implement. And as such I came up with my own SOCK_STREAM secure layer called MXC.

It’s very simple: you first agree on a session key using MUL operations, and at the end of the handshake the client can send encrypted traffic using XOR operations. MXC stands for Multiplication and eXclusive-or enCryption. I’m waiting for a pending security audit to complete before I can submit MXC to a standards body. It might take some time considering how long it took for TLS 1.3 to settle.

Since this is a thought experiment, I also decided to skip error handling and keep things to a minimum. So assert(3) is used where it wouldn’t make sense at all to make further progress outside the happy path, and otherwise return values are ignored using a (void) cast.

The protocol primitives are implemented in a short C header library called mxcrypt.h.

Needless to say, this implementation has many shortcomings, and the handshake won’t work unless both client and server share the same endianness, but I digress.

The client

The protocol library was designed so that both clients and servers could almost fit on a slide:

#include "mxcrypt.h"

void
main(int argc, char * const *argv)
{
	struct mxc_sess sess[1];
	char buf[256];
	ssize_t len;
	uint16_t port;
	int fd;

	fd = mxc_socket(argc, argv);
	port = mxc_port(fd);
	LOG("connected via port %hu", port);

	mxc_sess_init(sess);
	mxc_hello_client(sess);
	mxc_hello_send(sess, fd);
	mxc_hello_recv(sess, fd);
	mxc_sess_key(sess);
	LOG("computed session key 0x%08x", sess->key);

	do {
		len = read(STDIN_FILENO, buf, sizeof buf);
		assert(len >= 0);
		mxc_sess_send(sess, fd, buf, len);
	} while (len > 0);

	(void)close(fd);
}

The client is very straightforward: first it initializes a session and generates a client hello for the handshake, then it sends its own hello to the server and waits for the server hello in the reply. At this point the key can be derived and the client proceeds to forward its standard input encrypted over the network.

The server

Again, thanks to how the library was designed the server is equally straightforward:

#include "mxcrypt.h"

void
main(int argc, char * const *argv)
{
	struct mxc_sess sess[1];
	char buf[256];
	ssize_t len;
	uint16_t port;
	int sock_fd, sess_fd;

	sock_fd = mxc_socket(argc, argv);
	port = mxc_port(sock_fd);

	LOG("listening to port %hu", port);

	sess_fd = accept(sock_fd, NULL, NULL);
	assert(sess_fd != -1);

	mxc_sess_init(sess);
	mxc_hello_recv(sess, sess_fd);
	mxc_hello_server(sess);
	mxc_hello_send(sess, sess_fd);
	LOG("computed session key 0x%08x", sess->key);

	do {
		len = mxc_sess_recv(sess, sess_fd, buf, sizeof buf);
		assert(len >= 0);
		(void)write(STDOUT_FILENO, buf, len);
	} while (len > 0);

	(void)close(sess_fd);
	(void)close(sock_fd);
}

It requires one additional socket compared to the client, one for the port it is listening to and one for the single connection it will accept. Then it’s almost the same dance as the client. Initialize the session and wait for the client hello, then generate and send the server hello. At this point the key was already derived, courtesy of the MXC implementation. Finally the server decrypts the client’s contents and forwards them to its standard output.

The problem

Now that I have working client and server implementations of MXC I want to improve the handshake by adding the possibility to present a certificate for authentication purposes. But there is a little problem with the protocol so far. Actually there are so many problems with both the protocol and implementations that your average crypto person might already be suffering a heart attack.

I’ll focus on one problem though, and the astute reader may have already guessed that this is going to be about sandboxing.

The main problem here is that an MXC might do too many things:

require root privileges depending on which port it uses
deal with the private key of the certificate during the handshake
process untrusted input from potentially adversary parties

The first problem is technically not a problem. Unfortunately dropping privileges is often a privilege, so sandboxing might be more effective with system administration rights in the first place.

The real problem is that currently if there is an application-level bug below MXC, the “bug” is in the same address space as the certificate’s private key, and could do its damage with the aforementioned root privileges.

Incremental solutions

There is one straightforward solution which consists in having a proxy that terminates MXC traffic and forwards clear traffic to the application. It is also not entirely satisfying as a solution because it requires twice as many sockets on the system, and is both a resource and operational overhead. It could also not be an operational overhead if it terminates MXC for multiple services, but that still leaves one undesirable detail. Despite address space isolation between processes we still end up sending CLEAR traffic between them. We can mitigate the problem by co-locating the proxy and application on the same system, and using a Unix-domain socket with proper permissions, but then it becomes harder to amortize the operational overhead. The resource overhead is however here to stay.

If we end up with both the proxy and the application on the same host we can get rid of the clear traffic problem. Unix-domain sockets have one interesting property of inter-process communication, they can pass file descriptors from one process to another. In order to do that, there needs to be a new protocol between the proxy and the application so that forwarding the connection itself instead of its traffic can happen.

Next, we can solve the operational overhead by teaching the application how to do the sandboxing. Much like we can create a pipe(2) before forking a process, we can also create a Unix-domain socketpair(2) to create a private channel before two processes.

MXC sandboxing

Now that I have a working implementation of a client and a server, I can write a new server with sandboxing and check that it still works with the original client.

The admin process

The main() function would look like this:

void
main(int argc, char * const *argv)
{
	pid_t acceptor_pid, server_pid;
	int pair_fd[2];

	assert(socketpair(PF_UNIX, SOCK_STREAM, 0, pair_fd) == 0);

	acceptor_pid = fork();
	if (acceptor_pid == 0) {
		(void)close(pair_fd[0]);
		main_accept(argc, argv, pair_fd[1]);
		return;
	}
	assert(acceptor_pid != -1);

	server_pid = fork();
	if (server_pid == 0) {
		(void)close(pair_fd[1]);
		main_serve(pair_fd[0]);
		return;
	}
	assert(server_pid != -1);

	(void)close(pair_fd[0]);
	(void)close(pair_fd[1]);

	assert(waitpid(acceptor_pid, NULL, 0) == acceptor_pid);
	assert(waitpid(server_pid, NULL, 0) == server_pid);
}

Each process gets one end of the socket pair, but unlike a pipe(2) it is bidirectional so care needs to be taken to make sure no privilege escalation is possible between the server process (that receives application traffic) and the acceptor process (that manipulates certificates and their private keys).

The acceptor process takes the original command line arguments to figure which port it will listen to while the server only gets the socket from which it will receive MXC sessions.

The acceptor process

The main_accept() function is simple enough:

void
main_accept(int argc, char * const *argv, int server_fd)
{
	struct msg_sess ms[1];
	uint16_t port;
	int sock_fd;

	sock_fd = mxc_socket(argc, argv);
	port = mxc_port(sock_fd);

	LOG("listening to port %hu", port);

	ms->sess_fd = accept(sock_fd, NULL, NULL);
	assert(ms->sess_fd != -1);

	mxc_sess_init(ms->sess);
	mxc_hello_recv(ms->sess, ms->sess_fd);
	mxc_hello_server(ms->sess);
	mxc_hello_send(ms->sess, ms->sess_fd);
	LOG("computed session key 0x%08x", ms->sess->key);

	send_msg_sess(ms, server_fd);

	(void)close(ms->sess_fd);
	(void)close(sock_fd);
}

It does everything the previous server did up to the handshake, then it sends the MXC session to the server and finishes. After getting a hold of the TCP port and access to certificates, this process could drop many privileges. It could even drop privileges prior to accessing certificates if they are made available beforehand to a dedicated unprivileged user.

The server process

It should be easy to guess from the first server implementation and the acceptor process what this one looks like:

void
main_serve(int acceptor_fd)
{
	struct msg_sess ms[1];
	char buf[256];
	ssize_t len;

	recv_msg_sess(ms, acceptor_fd);

	do {
		len = mxc_sess_recv(ms->sess, ms->sess_fd, buf, sizeof buf);
		assert(len >= 0);
		(void)write(STDOUT_FILENO, buf, len);
	} while (len > 0);

	(void)close(ms->sess_fd);
}

This process could drop its privileges right away, unless the application needs to acquire resources before that. In this example of course, we already inherited the standard output so in theory we could drop everything at the very beginning.

Passing the MXC session

Passing a file descriptor between processes is only one thing we need to do. We can see in the server process that we still need a valid session to decrypt the traffic. Thankfully, it is possible to do this somewhat atomically: the session can be serialized and sent as the payload whereas passing a file descriptor to a unix-domain socket uses the control payload of a message.

In this case, we use a private socket and the MXC session is self-contained so it can be sent as its in-memory representation:

void
send_msg_sess(struct msg_sess *ms, int fd)
{
	struct msghdr msg[1];
	struct cmsghdr *cmsg;
	struct iovec iov[1];
	union msg_buf buf[1];

	iov->iov_base = ms->sess;
	iov->iov_len = sizeof ms->sess;

	(void)memset(msg, 0, sizeof msg);
	msg->msg_iov = iov;
	msg->msg_iovlen = 1;
	msg->msg_control = buf;
	msg->msg_controllen = sizeof(buf);

	cmsg = CMSG_FIRSTHDR(msg);
	cmsg->cmsg_level = SOL_SOCKET;
	cmsg->cmsg_type = SCM_RIGHTS;
	cmsg->cmsg_len = CMSG_LEN(sizeof(int));
	(void)memcpy(CMSG_DATA(cmsg), &ms->sess_fd, sizeof(int));

	assert(sendmsg(fd, msg, 0) > 0);
	LOG("sent session for file descriptor %d", ms->sess_fd);
}

The struct msg_sess type conveniently carries both session state and the file descriptor, in other words the data and control payloads:

struct msg_sess {
	struct mxc_sess	sess[1];
	int		sess_fd;
};

The recv_msg_sess() function isn’t too interesting to look at at this point because it mostly looks like send_msg_sess().

MXC sandboxing benefits

This scheme greatly reduces the overall attack surface. The main “admin” process very likely running as root doesn’t touch any application traffic in this example. The acceptor process that would have access to certificates and private keys if I were to extend the MXC protocol would have very limited interactions with untrusted input, namely handshakes. And finally the actual server implementing some sort of application logic would only have access to the ephemeral session key.

It would also have less overhead because with one program we may now both terminate encrypted traffic and process it at the application level. We’d still have three processes, but coming from the same executable file they’d share more and waste less. Finally we don’t need more sockets on the host system.

One nice twist of the second server implementation is the unidirectional traffic over the unix-domain socket pair. This way there is little risk of having a compromised application send commands to the acceptor process to leak MXC private keys (even though I didn’t implement certificates). This only works if the protocol no longer needs the certificate once the handshake completes. Otherwise one might design the sandbox differently, or have a two-way dialog between the acceptor and the server.

Also, in the likely event that the session state would not be of a fixed size like the MXC protocol so far, the acceptor may have read data past the handshake and should serialize any data in the pipeline along with session state.

TLS sandboxing

Circling back to $DAYJOB, I sometimes start pondering the two times where HTTPS support was considered in Varnish, which resulted in the Hitch TLS terminator proxy.

The first one says the following that stuck in my head:

Will that be significantly different, performance wise, from running a SSL proxy in separate process ?

No, it will not, because the way varnish would have to do it would be to … start a separate process to do the SSL handling.

There is no other way we can guarantee that secret krypto-bits do not leak anywhere they should not, than by fencing in the code that deals with them in a child process, so the bulk of varnish never gets anywhere near the certificates, not even during a core-dump.

In other words, if we wanted native HTTPS support in Varnish we would require the kind of sandboxing implemented as a thought experiment for MXC.

The second take points to a keyless TLS implementation that uses private keys stored remotely during the handshake, validating the MXC sandboxing model and adding one more layer of security. If the MXC (or at this point TLS) handshake can be exploited, at least it will be significantly harder to steal private keys.

A keyless handshake also makes the whole fork(2) with socketpair(2) dance somewhat obsolete, but it reintroduces operational overhead: instead of a TLS terminator proxy the application needs a key server and a new can of security worms to deal with. On the other hand it can probably be amortized rapidly for CDN setups, but I digress.

OpenSSL sandboxing

Now that TLS could hypothetically be sandboxed safely, allowing the handshake to happen in a separate process, possibly using private keys remotely without ever fetching them, comes the question of a TLS implementation.

The obvious candidate for multiple reasons would be OpenSSL, but can it do that as of today? It is possible to serialize a session to an ASN1 or base64 representation, so this could be passed alongside the file descriptor to the cache process (Varnish’s “server” process). We already use this for session resumption in Hitch, but can it be done for a live ongoing connection?

If it is doable, is it portable across all the platforms we care about in the Varnish project? Would the TLS handshake sandboxing create a new bottleneck? My intuition tells me that an application would still perform better without a separate proxy. But it comes with further challenges and may require that we modularize more parts of Varnish like the acceptor subsystem. Would OpenSSL integrate nicely in our delivery pipeline? Many questions I’m not covering for a short thought experiment in the middle of the night.

Bonus MXC demo

I wanted to give an interactive demo of the MXC PoC, so I looked at what was available on Fedora and settled for asciinema. In this scenario I build the client, the sandboxing server, and use a shell script to create a man-in-the-middle proxy. The client passes a secret key to the server and the netcat-based proxy dumps traffic on disk. In order to show all three actors semi-simulteanously I used tmux to split the terminal:

You can see what the client and server are logging, and that the server logs from two different PIDs. When the client and servers are rebuilt with the MXC_CLEAR macro defined, the traffic is sent non-encrypted and the MITM proxy successfully intercepts the private payload.

All the code is available:

Finally, one last reason I was thinking about HTTPS support in Varnish is that HTTP/3 will likely be based on QUIC and specified as secure-only so keyless or not we may want to eventually sandbox handshakes. I think we should still support a complete handshake and not force users to have a key server, like we force Varnish users to use proxies for both TLS termination and tunneling today in end-to-end HTTPS setups.

Taking reverse engineering to the next level

Sun, 31 Mar 2019

Learning new languages

Last October I wrote about my progress and the first release of bjxa. I published a reference implementation of a BandJAM XA decoder in C able to reproduce bit-for-bit the same audio output as xa.exe. After that I reviewed DTXMania’s code, which to this date I still can’t build completely on Linux with Mono.

I remember seeing years ago a piece of advice for developers, encouraging them to learn one new language each year, to challenge their comfort zone. While I never followed this advice, I think that learning a new language can be counter-productive if their isn’t an actionable project to back the learning. Porting bjxa to C# was a perfect excuse, since the scope is small enough and the language is somewhat familiar to Java which I already know.

My conclusions after conducting this work is that C# is indeed a much more pleasant language than Java. I’m still not satisfied with the code I came up with but it was enough to even challenge my C implementation and drive some improvements.

I then sent an email to the DTXMania maintainer in November to raise his awareness of BJXA and was greeted with deafening silence. This was of course my own fault for not creating an account and create a ticket on the project tracker, but that’s a consequence of a distinct lack of confidence.

That made me wonder about the languages I learned over the years and whether the total amount would average around one per year. Here is a list I could produce off the top of my head, in alphabetic order:

AWK
Ada
C#
C99
Go
Groovy
Java
JavaScript
POSIX shell
Perl
PHP
Python
Ruby
Rust
SED
SQL
VCL
m4
vimscript
x86 assembler (Intel syntax)

That’s a fair amount of languages, but I’m a beginner in most of them. For some of them I only learned them to write exactly one program or library for a transient need and never touched them again. Some I use out of necessity every once in a while but never became proficient, but I digress…

Reverse script kidding

Tools keep making astonishing progress in all areas, and even a beginner can get results pretty fast with the help of IDEs or similar tools. And yet, as I settled in programming my personal trend was to move away from such tools and instead learn underlying tools they tend to abstract away.

Using IDEs is fine, learning the fundamental tools is critical. Keeping IDEs despite being proficient with the underlying tools is also fine, although I personally prefer to avoid IDEs.

But as a beginner it is sometimes much easier to use graphical tools that try to be more user friendly by hiding the dirty details. Despite my success in uncovering the BJXA codec and documenting it thoroughly there are still unknown parts. The reverse engineering of xadec.dll was easy even for an x86 beginner because it came with a header file that gave a significant head start. The fact that it was a shared library also gave me by definition access in the assembly to all the public functions.

Now to find the answers to the remaining mysteries of the codec I would need to decompile xa.exe but there is a catch: it’s a statically linked program and the only starting point I have is the entry point of the program, which is responsible for calling the main() function of a C program. In addition it’s a Win32 executable, which I’m not familiar with.

Just when I decided to live with those mysteries, the NSA knocked on my proverbial door and convinced me to give a try to their IDE.

Here be dragons

After decompiling xadec.dll I was convinced that it was originally written in C and I suspected xa.exe would be too. Finding the main() function would in theory allow me to unravel the encoding code and see how it treats the undefined behavior of the codec.

Trying to use retdec didn’t work out because it failed to identify the calling convention and I gave up (without reaching out for help) when I couldn’t figure out how to let it know. The result was even worse with xa.exe.

Trying to decompile the assembly by hand resulted in a total failure to even locate the main() function. Even the knowledge of xa.exe’s command line arguments I failed to use a reference to the usage code and find my way back to the argv parsing:

$ wine xa.exe
WAV to XA v1.22 Copyright 2000-2001 bandjam.net
Usage  : xa.exe <option> [filename<.wav/.xa>]
Option : -e[n]    Encode[WAV->XA](Default) / n:BitCount(4/6/8)
         -d       Decode[XA->WAV]
         -p       Play File
         -o[dir]  Output Directory
         -u       Update

Then the NSA announced it would open source one of their reverse engineering tools and eventually made it available, although the source code isn’t yet. Ghidra is a graphical tool that is for reverse engineering the equivalent of an IDE for development.

In less than 2 minutes (I timed it!) I was able to find the main() function and confirm that it used a C-like signature taking the two famous arguments int argc and a char **argv.

I think that Ghidra’s main advantage over retdec was its ability to find the right calling convention from the get go. I managed to find the equivalent of xadec.dll’s xaDecodeOpen by luck and found the decompiled code to be quite elegant.

Overall, I think the NSA deserves to be congrat… to be cong… I think the NSA did an OK job.

Unfortunate obfuscation

One reason why I had a hard time finding the main() function and its command line parsing is again an incarnation of the robustness principle. With no hint from the usage description I couldn’t guess that options were case-insensitive and that they also supported the Windows style for options:

$ wine xa.exe -d square-mono-8.xa
WAV to XA v1.22 Copyright 2000-2001 bandjam.net
square-mono-8.xa -> square-mono-8.wav : 44100Hz / monaural / 8bits.
1 Files Decoded.
Completed.

$ wine xa.exe /d square-mono-8.xa
WAV to XA v1.22 Copyright 2000-2001 bandjam.net
square-mono-8.xa -> square-mono-8.wav : 44100Hz / monaural / 8bits.
1 Files Decoded.
Completed.

$ wine xa.exe /D square-mono-8.xa
WAV to XA v1.22 Copyright 2000-2001 bandjam.net
square-mono-8.xa -> square-mono-8.wav : 44100Hz / monaural / 8bits.
1 Files Decoded.
Completed.

I also couldn’t have guessed that the command line parsing would intertwine options parsing and a lot of processing. Thanks to Ghidra I have a semi-clean switch statement and was able to figure that while some processing is done immediately when an option is identify, some checks are deferred until needed. For example I can ask to encode something and ultimately say that I want to do decoding instead and it won’t complain:

$ wine xa.exe -e5 square-mono-8.wav
WAV to XA v1.22 Copyright 2000-2001 bandjam.net
bitcount error.

$ wine xa.exe -e5 -d square-mono-8.xa
WAV to XA v1.22 Copyright 2000-2001 bandjam.net
bitcount error.

$ wine xa.exe -e8 square-mono-8.wav
WAV to XA v1.22 Copyright 2000-2001 bandjam.net
square-mono-8.wav -> square-mono-8.xa : 44100Hz / 16bit / monaural
1 Files Encoded.
Completed.

$ wine xa.exe -e8 -d square-mono-8.xa
WAV to XA v1.22 Copyright 2000-2001 bandjam.net
square-mono-8.xa -> square-mono-8.wav : 44100Hz / monaural / 8bits.
1 Files Decoded.
Completed.

I also found an interesting undocumented -l option and its effects are puzzling to say the least… Looking at the shape of the code, it looks as though xa.exe was written in C++ and if I’m right that’s probably Visual C++.

I hope to solve the remaining mysteries in the BandJAM XA codec with the help of Ghidra and so far it looks much more within reach. Unfortunately that’s a project I will have to shelve for later, but I digress.

19 years of xadec.dll

Eventually, I received a response from the DTXMania maintainer regarding my struggles when it comes to running it on Linux using Wine, and especially the problem of xadec.dll. The response was overwhelmingly positive and I learned that they were struggling too because of it and the fact that so many songs playable for the game use that ancient codec.

They wanted to maintain the capability to read BandJAM XA files, but that prevented 64bit builds of the game. Out of nowhere came bjxa and it got them closer to solving the problem for good (they still have an ancient OGG and MP3 decoder similar to xadec.dll: no source code).

I also learned in the process that the original DTXMania maintainer was working on a DTXMania2 but unfortunately that one is even harder to run with Wine. Without much help from my side they managed to integrate libbjxa in their projects and the fact that bjxa.exe was available to see how to use the library probably helped.

When I got the first response, the maintainer was apologetic because it had taken a couple months to reply to my inquiry. Working on free and open source software I have experienced the demanding tone of some users. I have found myself in demanding positions too, and have probably given that impression more than once even when it wasn’t the case. I find it a bit sad though when maintainers apologize when they shouldn’t and keep doing so even after being told that all is fine, but I digress…

No matter how I thanked them they would thank me even more. I’d like to thank them here once more and I hope to hit them hard with more demanding patches as I hope to iron out the remaining problems inside Wine that I identified.

19 years of buffer overflows

I found two interesting classes of bugs in xadec.dll, one of which could lead to a security vulnerability. The first one is that most tainted pointer dereferences are done without a proper null check. The second one is a buffer overflow that can be triggered easily with a specially crafted XA file. Can it result in arbitrary code execution? I don’t know and even if I managed to own a DTXMania process running in Wine that wouldn’t prove that the same exploit could be used on Windows and vice versa. And considering that I won’t install Windows to run the game and introduced myself to reverse engineering just for the sake of running the game on Fedora, I won’t shave that yak.

If you would like to see what real reverse engineering looks like, I strongly recommend this conference from people that could have probably done in minutes what initially took me hours. My plan is to level up my game (all kinds of puns intended) by cheating. I will rely on Ghidra, my new power tool, but before that I have other things to tend to and hopefully I will write on other topics in the future.

Beating the reverse engineering game

Sun, 07 Oct 2018

Reverse motivation

After losing my annotated assembly listings, it’s been hard to find motivation to take the most complicated part of the code, reverse engineer it again and comment it here. And this is code I really want to comment because there are interesting findings inside.

So instead I decided to work on the actual project, that goes beyond the quick and dirty reverse engineered code. One thing I learned over the years is that it’s easier to start than finish things, for a given definition of finish. In this case that would be a working decoder, a specification of the codec, all that wrapped nicely inside a source repository, and a first release with all the bells and whistles. And as such bjxa 0.1 was released, matching this definition of finish with some hidden fine prints.

As a result I decided to open a projects section with the result of this reverse engineering self challenge as its first entry.

Thank Wine, again

In order to produce a decoder that would produce a byte for byte identical WAVE file I needed to somehow run xadec.dll but previous attempts at cross compiling sample.c to run it under Wine failed for reasons beyond my tinker spirit.

But since I found that xa.exe program from the same author, I was able to produce WAVE files that would infringe on nobody’s copyright, convert them to XA and back again to WAVE. Giving me an XA to WAVE baseline to check my work. Unsurprisingly Wine had no issues running this (statically linked) program.

I studied the differences between the PCM samples produced by my code and the PCM samples produced by xa.exe and noticed that most PCM samples were very much identical and when they were not, only the least significant byte of the samples were different.

Double wrong means not so wrong

I have yet to disassemble that part again, and it was the hardest bit to reverse engineer for me so it will take some time before I pull it off. But in short, I was right. The fact that the BandJAM author had shipped an encoder made me think that my assumption was wrong that this was not an original codec. But I was actually correct after my initial 30 hours once I got a working C decoder after my mental decompilation of the assembly: BandJAM XA is CDROM XA, the same audio codec as the PlayStation!

Why would I notice that in the first place? It’s a little embarrassing but sometimes I get bored and read other people’s source code. As I said, I’m probably a cat serial killer at this point because my curiosity extends to such lengths that I would sometimes read the source code of things that look magic to me and in this case an open source emulator, but I digress…

CDROM XA has the notion of a block profile and depending on the profile chosen for a block, a gain may be added to the PCM samples of a block, and I already determined in part 5 the size of XA blocks. And since the LSB of the decoded samples was sometimes off, I tried to figure why they would stop being off.

It turns out they’d stop diverging at block boundaries, and one of the profiles doesn’t adjust the gain and as a result is not influenced by past blocks. Quick check: yes, when PCM samples diverge, they converge again on such blocks. Following my intuition I updated my code to decode CDROM XA blocks and here goes nothing: the audio output is a perfect match!

Audio and unicorns

With this in mind, I have to revisit the decompilation of xadec.dll. Either I made some mistakes, or there is a bug in the library and I have reasons to keep this as a possibility. Another reason could be that xadec.dll doesn’t use the same code as xa.exe. Unfortunately, I lost that decompilation, but I’m still planning to redo it once I find my lost motivation.

This got me thinking: why would my code for CDROM XA work better than my decompiled code, especially since the former uses floating point arithmetic and the latter relies only on integers? I looked into this and found a rabbit hole so big that I shouldn’t even start.

Short story: you can find many of the CDROM specifications online and CDROM XA also known as CDROM eXtended Architecture, also known as the Green Book. The Green Book is an extension to the Yellow Book standard and apparently CDROM folks love colors. There are other books and the collection is sometimes referred to as the Rainbow Books.

I found a copy of the Green Book online and it’s about 1000 pages long. The quality of this book should humble most open source projects, I have spent hours reading through. This is a very interesting read, you have been warned.

Robust size

The digression above was brought to you by amazing engineers from the 90s. I believe I left off at the perfect match. Well, almost perfect: there is a bug in the RIFF header generated by xa.exe. So let’s start by decompiling the xaDecodeSize function:

10001140: mov    ecx,DWORD PTR [esp+0x4]        ; ecx = hxas;
10001144: test   ecx,ecx                        ; if (hxas == NULL) {
10001146: jne    0x1000114b                     ;
10001148: xor    eax,eax                        ;       eax = 0;
1000114a: ret                                   ;       return (0);
                                                ; }
1000114b: mov    eax,DWORD PTR [esp+0x8]        ; eax = slen;
1000114f: xor    edx,edx                        ; edx = 0;
10001151: div    DWORD PTR [ecx+0x28]           ; eax /= hxas->blk_sz;
10001154: mov    ecx,DWORD PTR [esp+0xc]        ; ecx = pdlen;
10001158: inc    eax                            ; eax++;
10001159: shl    eax,0x6                        ; eax *= 64;
1000115c: mov    DWORD PTR [ecx],eax            ; *pdlen = eax;
1000115e: mov    eax,0x1                        ; eax = 1;
10001163: ret                                   ; return (1);

It can feel confusing to see that div takes only one operand. It’s only a matter of knowing that in this case the dividend is the combination of eax for the LSB and edx for the MSB. This explains why edx is zeroed out of the blue.

Considering the signature of the function, it can be decompiled as such:

BOOL
xaDecodeSize(HXASTREAM hxas, ULONG slen, ULONG *pdlen)
{
	if (hxas == NULL)
		return (FALSE);

	*pdlen = 64 * (1 + slen / hxas->blk_sz);
	return (TRUE);
}

So, progress here, we have a null check for hxas. On the other hand we lack a null check for pdlen and the arithmetics look bonkers. The additional increment (pun intended) is here to compensate partial blocks. This is that kind of reasoning that leads to the RIFF header bug in xa.exe.

Fortunately, audio players tend to be more robust than that and will agree to play a broken WAVE file if it’s only that kind of corruption. Sure the RIFF size is in most cases smaller than what ends up declared, so this means we can safely ignore anything beyond the WAVE data.

This, is a good example of the robustness principle:

Be conservative in what you do, be liberal in what you accept from others.

Sure, let’s accept everyone else’s bugs or non standard extensions to the point where we can’t move a protocol or file format forward without breaking everyone’s code. This, is the sad reality of the Internet: a slow-moving behemoth that nevertheless keeps on growing faster than it can move. But I digress…

Building up motivation

Another reason why I started this series has yet to come. I’m not teaching x86 assembly or how to reverse engineer code. But I hope it will show that it is possible for someone to start that kind of endeavour by themselves. I was lucky to find an old C library either too old to lose me in optimizations or too simple to be aggressively optimized. And I found more than just code to disassemble and decompile, as cheesy as it may sound. I hope to publish a next post soon, I should only need a couple more to cover everything I wanted to report.

Expanding the XA puzzle

Sun, 12 Aug 2018

The reverse engineering games

This post should have started with the words “last week” but unfortunately I was not able to follow the weekly cadence I had planned. Instead of just the trivial matters I announced then I will also cover more topics than initially planned, going from a very short entry to hopefully an entry that still qualifies as short.

The reason why I started the xadec.dll reverse engineering game was as I said in the first post of this series the prospect of living in a bigger place with enough space to use my electronic drum kit. One thought led to another and I remembered the fun I had playing GuitarFreaks and DrumMania. The reason why this post is late is because in the process of moving my laptop was stolen.

I’m not too worried about the contents of the laptop because it should be a brick to anyone but me. Considering the current trend among politicians against strong cryptography, I’m very glad to be in the other camp. However having the laptop stolen is not the only reason for a whole month delay: I also lost some of my backups, including the sources of this blog.

I have a NAS at home for local backups, and copies online (or as the sheeps say these days, in the cloud). Some of my things are inherently backed up, because hosted online: emails, source code. Stuff that is not important tend to live on only one device until I archive it just in case, that’s obviously gone. The big bummer is a couple of local backups that fail to restore, a procedure I usually test when I switch laptops (that is, not often enough) so the local backups are useless and the on-device archives are gone.

If you paid attention, source code should not be a problem, but I lost the blog sources. I forgot for some reason to host a clone online and I had to reverse engineer the HTML web site and figure all the changes I had made to the theme to get it back. I guess I should talk about vim in a future post because it helped a lot in the process, but I digress… And with that this post is already longer than the original one I had in mind.

The music games

Anothernother reason for being so late is the fact that I eventually moved, and this was the occasion to revisit my games collection.

On this picture you should see the boxes containing the guitar controller I ordered before Guitar Hero happened, the Guitar Hero controller I then bought along with the first game, two original games from Japan and one more controller. The Playstation 2 game is GuitarFreaks and DrumMania V3, and missing from that picture are the defunct drums controller, the defunct modded Playstation 2 and the defunct modded Wii.

So unless I manage to use those controllers on a PC beefy enough to emulate those consoles and manage to run such an emulator, at best they can be seen as geeky decoration, otherwise dead weight that belongs in the basement. But surely, running those games using an emulator would be piracy, right?

The music game

The game I’m interested in (DTXMania) is officially a piece of software that anyone can use to practice their Yamaha DTX electronic drum kits. If you look closer though, it’s a DrumMania and GuitarFreaks clone, because yes you can also play the guitar. But to be fair DTXMania goes beyond the arcade nature of DrumMania and can support many more elements on a drum kit, and even elements with multiple states such as a hi hat that can be either open or closed.

Out of curiosity I looked at the source code beyond the presence of xadec.dll and found a well maintained copyright statement of its bundled dependencies. It mentions that xadec.dll can only be used free of charge, so any paid for game would not be allowed to ship and use it. It’s a bit challenging to find information in a code base maintained in Japanese, and that goes for the web site too, but there are some English resources. In there I found xa.exe apparently also shipped by the BandJam developer, and still accessible thanks to projects like DTXMania. xa.exe can both encode and decode this XA format, and I was probably wrong to think that the BandJam author was not behind this format. Now the question is the following: should I also reverse engineer the encoder?

Disassembly of a static program

Unlike a library that needs an entry point for each public symbol of its API, a quick try on xa.exe gives no starting point. Worse, even though I reinstalled the same OS, objdump no longer gives me the useful all-in-one output I was used to. That in itself is very puzzling.

One thing I could hope is to find the xadec.dll code in xa.exe since it can both encode and decode. The problem is that even if the code is virtually the same with identical optimizations for a similar output, it is likely that the choice of registers would differ, and it is even more likely that addresses wouldn’t match.

That would make an interesting coding exercise, trying to find the same code in two binaries with possibly slight differences, but for now I have other plans, like maybe playing some drums at some point, maybe?

Pouring some wine

Finally, I got hold of older versions of DTXMania, and managed to run them! At this point I should stop this reverse engineering game and fulfill my original goal of playing the game with my electronic drum kit. But that would imply sanity, a trait I do not claim. After all, when I initially tied to run the game I naturally went for the latest builds to no avail. I searched online and found that people tried to use Wine before and it failed miserably.

Well, Wine today can run DTXMania 067b just fine, even the version 061. In fact, those are versions that predate the open-sourcing of the game. And since I fail to build it on Fedora, I can’t assess when things broke today’s Wine.

$ dnf info wine
Installed Packages
Name         : wine
Version      : 3.13
Release      : 3.fc28
Arch         : x86_64
Size         : 0.0
Source       : wine-3.13-3.fc28.src.rpm
Repo         : @System
From repo    : updates
Summary      : A compatibility layer for windows applications
URL          : https://www.winehq.org/
License      : LGPLv2+
Description  : Wine as a compatibility layer for UNIX to run Windows applications. This
             : package includes a program loader, which allows unmodified Windows
             : 3.x/9x/NT binaries to run on x86 and x86_64 Unixes. Wine can use native system
             : .dll files if they are available.
             :
             : In Fedora wine is a meta-package which will install everything needed for wine
             : to work smoothly. Smaller setups can be achieved by installing some of the
             : wine-* sub packages.

All I can say is that it broke somewhere between 067b and 087. Maybe in a couple years Wine will catch up and run it like a champ! But I’m not here to reverse engineer the Windows APIs, so in the next post I will resume my report on xadec.dll but I don’t expect to stick to one post a week. While I still have my decompiled source code, I lost the side-by-side decompilation with the assembly.

Starting the xadec.dll puzzle

Sun, 08 Jul 2018

Thinking like a compiler

In the previous post I came to the conclusion that xadec.dll turns an XA file into a PCM stream and describes that PCM stream with a WAVEFORMATEX data structure. After looking at the valuable insights present in xadec.h it’s now time to look at the assembly and for the first time encounter machine code that _really_ doesn’t look like what a sane human being would come up with.

The xaDecodeOpen function does two main things: set up the decoder state and fill the WAVEFORMATEX structure. And that’s where the riddle begins: given an opaque data structure, how to make sense of the values it holds?

10001000: push   ebx                             ; esp offset to eip is 4
10001001: push   ebp                             ; esp offset to eip is 8
10001002: mov    ebp,DWORD PTR [esp+0xc]         ; ebp = pxah; /* arg0 */
10001006: push   esi                             ; esp offset to eip is 12
10001007: cmp    DWORD PTR [ebp+0x0],0x3144574b  ; if (pxah->id != _XAID)
1000100e: jne    0x1000110a                      ;         return (0); /* error */
10001014: push   0x34
10001016: push   0x40
10001018: call   DWORD PTR ds:0x10006010         ; eax = GlobalAlloc(64, 52);
1000101e: push   eax
1000101f: call   DWORD PTR ds:0x1000600c         ; eax = GlobalLock(eax); /* no-op */
10001025: mov    ebx,eax                         ; ebx = eax; /* hxas = calloc(1, 52); */
10001027: test   ebx,ebx                         ; if (ebx == NULL)
10001029: je     0x1000110a                      ;         return (0); /* error */
1000102f: [...]

Being completely new to this, and lacking any clear methodology I made things up as I was making progress. I would for example call arg0 the first argument to a function, or replace it with its name (if it’s known or I understand its purpose). In this case arg0 is called pxah in xadec.h and its id field known, otherwise I would have called written down something like arg0->dw0 in the if statement where dw0 means a 32-bit integer at the offset 0.

I have yet to make sense of what the opaque data structure fields mean, but at least I know its size: 52 octets. Yikes, that’s a lot of bytes to keep track of.

I note that a failing GlobalLock after a succeeding GlobalAlloc will leak 52 bytes of memory (plus the allocator’s overhead for alignment and tracking) but I also note something interesting. I once read that the first rule of compiler optimizations is that the compiler is smarter than you, and the second rule is that the compiler is smarter than you. Apparently xadec.dll was either compiled without (much) optimizations or comes from a time where compilers were less smart. In this case I’m not sure how moving eax’s value to ebx helps. Testing a null eax would have the same effect of unwinding the stack and returning an error:

10001104: [...]
1000110a: pop    esi                              ; esp offset to eip is 8
1000110b: pop    ebp                              ; esp offset to eip is 4
1000110c: xor    eax,eax                          ; eax = 0;
1000110e: pop    ebx                              ; esp offset to eip is 0
1000110f: ret                                     ; return (eax); /* error */

You could even save some processing by moving the xor operation before the other pop operations and jump straight after the xor when GlobalLock fails and eax is already null. Unless pop esi or pop ebp has a side effect I’m not aware of on eax, which sounds highly unlikely to me. I’ve had this idea for a while that most software problems can be reduced to graph theory problems. The use (and reuse) of a limited set of registers is akin to graph coloring and that’s a tough nut to crack, but I digress…

Thinking more like a compiler

Let’s move on and stay focused on what’s happening in the code. It’s already complicated enough, and too early too think about optimizing anything. It is more likely that I may have to mentally deoptimize code unless I’m lucky…

10001029: [...]
1000102f: xor    eax,eax                         ; eax = 0;
10001031: mov    al,BYTE PTR [ebp+0xe]           ; eax = pxah->nBits;
10001034: sub    eax,0x4                         ; eax -= 4;
10001037: je     0x10001063                      ; if (eax == 0) goto 0x10001063;
10001039: sub    eax,0x2                         ; eax -= 2;
1000103c: je     0x10001053                      ; if (eax == 0) goto 0x10001053;
1000103e: sub    eax,0x2                         ; eax -= 2;
10001041: jne    0x10001071                      ; if (eax != 0) goto 0x10001071;
10001043: mov    DWORD PTR [ebx+0x28],0x21       ; hxas->dw40 = 31;
1000104a: mov    DWORD PTR [ebx+0x2c],0x10001450 ; hxas->dw44 = 0x10001450;
10001051: jmp    0x10001071                      ; goto 0x10001071;
10001053: mov    DWORD PTR [ebx+0x28],0x19       ; hxas->dw40 = 25;
1000105a: mov    DWORD PTR [ebx+0x2c],0x100013e0 ; hxas->dw44 = 0x100013e0;
10001061: jmp    0x10001071                      ; goto 0x10001071;
10001063: mov    DWORD PTR [ebx+0x28],0x11       ; hxas->dw40 = 17;
1000106a: mov    DWORD PTR [ebx+0x2c],0x100013a0 ; hxas->dw44 = 0x100013a0;
10001071: [...]

So now the compiler is outsmarting us for a greater good, unless xadec.dll was coded in this style which I highly doubt. It is commonly known that computers are good at comparing things to zero and sometimes simply turning a for loop upside down so that it converges towards zero can provide a noticeable performance boost. An explanation could be that some architectures like x86 may set the zero flag (ZF) for instructions not primarily designed to make comparisons. Arithmetic and bitwise operations set the flag if the result is zero and unset it otherwise, and sub can be followed by a conditional jump.

So conceptually we get this:

int nBits = pxah->nBits;

if ((nBits -= 4) == 0) {
	hxas->dw40 = 31;
	hxas->dw44 = 0x10001450;
}
else if ((nBits -= 2) == 0) {
	hxas->dw40 = 25;
	hxas->dw44 = 0x100013e0;
}
else if ((nBits -= 2) == 0) {
	hxas->dw40 = 17;
	hxas->dw44 = 0x100013a0;
}
else {
	/* hxas->dw40 and hxas->dw44 are both zero */
}

After a mental deoptimization I may write this:

struct xastream *
xaDecodeOpen(struct xaheader *pxah, struct waveformatex *wfx)
{
	struct xastream *hxas = NULL;

	if (pxah->id == _XAID)
		hxas = calloc(1, 52);

	if (hxas == NULL)
		return (NULL);

	if (pxah->nBits == 8) {
		hxas->dw40 = 31;
		hxas->dw44 = 0x10001450;
	}
	else if (pxah->nBits == 6) {
		hxas->dw40 = 25;
		hxas->dw44 = 0x100013e0;
	}
	else if (pxah->nBits == 4) {
		hxas->dw40 = 17;
		hxas->dw44 = 0x100013a0;
	}

	/* ... */
}

On the other hand retdec ended up with something confusing, looking roughly like this once stripped (really really) down:

// Address range: 0x10001000 - 0x1000110f
char * xaDecodeOpen(char * a1, int16_t * a2, int32_t a3) {
    int32_t v1 = (int32_t)a1;
    if (*(int32_t *)a1 != 0x3144574b) {
        return NULL;
    }
    char * v2 = GlobalLock(GlobalAlloc(64, 52));
    int32_t pMem2 = (int32_t)v2;
    if (v2 != NULL) {
        unsigned char v3 = *(char *)(v1 + 14);
        int32_t v4 = v3;
        if (v3 != 4) {
            if (v4 == 6) {
                *(int32_t *)(pMem2 + 40) = 25;
                *(int32_t *)(pMem2 + 44) = 0x100013e0;
            } else {
                if (v4 == 8) {
                    *(int32_t *)(pMem2 + 40) = 33;
                    *(int32_t *)(pMem2 + 44) = 0x10001450;
                }
            }
            /* ... */
            return (char *)pMem2;
        }
        *(int32_t *)(pMem2 + 40) = 17;
        *(int32_t *)(pMem2 + 44) = 0x100013a0;
        /* ... */
        return (char *)pMem2;
    }
    return NULL;

You may note the presence of a third argument, and this is likely a problem caused by the lack of known calling convention and my inability to find how to tell retdec which one it is. It also has the tendency to follow the assembly too closely and for example when pxah->nBits is moved as a byte and then computed as a dword, we end up with two distinct variables v3 and v4. And that’s when we’re lucky: not accounting for the /* ... */ code duplication there were 12 other local variables and in total the library decompilation grew 119 global variables.

Breaking the spell

With a partial function decompilation we already end up with a handful of magic numbers, so it’s best to start now. I’m leaving 0x3144574b out because it was ostensibly waiting to be picked up in xadec.h and seemed like the first thing you’d check in a function called xaDecodeOpen. Well, don’t mind me, that would be the second thing to check, right? Did you notice the lack of null check of arg0 aka pxah?

Next, the mysterious numbers 17, 25 and 33. In order to understand their meaning, I need to solve a couple equations and for the sake of brevity I will mention when I get to that code what confirmed the supposed meaning.

f(x) = ???
f(8) = 33
f(6) = 25
f(4) = 17

It doesn’t take much time to notice the pattern, at least it didn’t take me long to realize that those figures were “almost” recognizable:

blockSize(nBits) = 4 * nBits + 1
blockSize(8) = 33
blockSize(6) = 25
blockSize(4) = 17

Praise my mighty arithmetic powers!

Next I need to make sense of 0x10001450, 0x100013e0 and 0x100013a0. They look awfully like function addresses, and tracking the use of this 44 offset confirms it:

100011c4: push   edx
100011c5: push   ebx
100011c6: call   DWORD PTR [edi+0x2c]

One can assume that nBits refers to the number of bits per samples, so the addresses probably refer to callbacks that uncompress the samples, and it looks like the callbacks take two arguments.

Moving forward in the decompilation I find more magic numbers, but fortunately less magic and more straightforward from now on:

1000106a: [...]
10001071: xor    eax,eax                         ; eax = 0;
10001073: mov    al,BYTE PTR [ebp+0xf]           ; eax = pxah->nChannels;
10001076: dec    eax                             ; eax--;
10001077: je     0x10001085                      ; if (eax == 0) goto 0x10001085;
10001079: dec    eax                             ; eax--;
1000107a: jne    0x1000108c                      ; if (eax != 0) goto 0x1000108c;
1000107c: mov    DWORD PTR [ebx+0x30],0x10001210 ; hxas->dw48 = 0x10001210;
10001083: jmp    0x1000108c                      ; goto 0x1000108c;
10001085: mov    DWORD PTR [ebx+0x30],0x10001190 ; hxas->dw48 = 0x10001190;
1000108c: [...]

This is quite similar to the nBits callbacks, here we have what looks like one callback for mono and one callback for stereo conversion. Again, using the same trick with dec instead of sub to compare pxah->nChannels to 1 and 2. xadec.dll is a 32bit library, so it’s no surprise to see function pointers fit in a dword.

Boardom is bliss

I’m a big fan of The IT Crowd, a silly British sitcom that manages to be hilarious at times. There’s an episode on board games, and they make it really sound like bored games, and sometimes boredom is the best thing that could happen to you, but I digress. Thankfully at this point it’s all very boring.

There is nothing complicated in this function and the decompilation is very straightforward. Except maybe for a couple magic numbers that were easy to understand, there was really only one surprising thing. The next boring step is to check that we have both callbacks, so in other words a valid combination of sample size and mono or stereo sound.

10001085: [...]
1000108c: mov    eax,DWORD PTR [ebx+0x2c]        ; eax = hxas->dw44;
1000108f: test   eax,eax                         ; if (hxas->dw44 == NULL)
10001091: je     0x100010f0                      ;         goto 0x100010f0;
10001093: mov    eax,DWORD PTR [ebx+0x30]        ; eax = hxas->dw48;
10001096: test   eax,eax                         ; if (hxas->dw48 == NULL)
10001098: je     0x100010f0                      ;         goto 0x100010f0;
1000109a: [...]

I’m omitting the error handing at 0x100010f0 but it is positioned just before the bit I already decompiled and frees hxas before returning zero (aka an error in this case).

Next is a bit of a surprise, although not puzzling. Thanks to prior exposure to this construct I was able to follow the logic:

10001098: [...]
1000109a: push   edi                             ; esp offset to eip is 16
1000109b: mov    ecx,0x8                         ; ecx = 8;
100010a0: mov    esi,ebp                         ; esi = pxah
100010a2: mov    edi,ebx                         ; edi = hxas
100010a4: rep movs DWORD PTR es:[edi],DWORD PTR ds:[esi]
                                                 ; memcpy(hxas, pxah, sizeof *pxah);
100010a6: [...]

Copy 8 dwords (32 bytes) from pxah to hxas, in other words embed a copy of the XA header in the opaque data structure, removing at once 32 bytes from the 52 bytes puzzle. What. A. Relief.

Surf the WAVE

Surf is one of those board games for which I wonder how it all started. At some point someone thought it would be fun to ride waves, and turned out to be correct. But how does such an idea come to someone’s mind? In a sense, this reverse engineering journey is that kind of improbable idea, and after a poor start I now feel like everything is going by itself according to plan. World domination will ensue.

The last bit of good news came from the last bit of decompiled code. At this point where everything succeeded, all we need (barring one more missing null check) is to fill in the wave file format:

100010a4: [...]
100010a6: mov    ecx,DWORD PTR [esp+0x18]        ; ecx = wfx; /* arg1 */
100010aa: xor    edx,edx                         ; edx = 0;
100010ac: pop    edi                             ; esp offset to eip is 12
100010ad: pop    esi                             ; esp offset to eip is 8
100010ae: mov    WORD PTR [ecx],0x1              ; wfx->wFormatTag = WAVE_FORMAT_PCM;
100010b3: movzx  ax,BYTE PTR [ebp+0xf]           ; ax = pxah->nChannels;
100010b8: mov    WORD PTR [ecx+0x2],ax           ; wfx->nChannels = pxah->nChannels;
100010bc: mov    dx,WORD PTR [ebp+0xc]           ; dx = pxah->nSamplesPerSec;
100010c0: mov    DWORD PTR [ecx+0x4],edx         ; wfx->nSamplesPerSec = pxah->nSamplesPerSec;
100010c3: xor    edx,edx                         ; edx = 0;
100010c5: movzx  ax,BYTE PTR [ebp+0xf]           ; ax = pxah->nChannels;
100010ca: shl    eax,1                           ; eax /= 2;
100010cc: mov    WORD PTR [ecx+0xc],ax           ; wfx->nBlockAlign = pxah->nChannels / 2;
100010d0: mov    dx,WORD PTR [ebp+0xc]           ; dx = pxah->nSamplesPerSec;
100010d4: and    eax,0xffff                      ; eax = ax; /* no prior xor */
100010d9: pop    ebp                             ; esp offset to eip is 4
100010da: imul   eax,edx                         ; eax = pxah->nSamplesPerSec * pxah->nChannels / 2;
100010dd: mov    DWORD PTR [ecx+0x8],eax         ; wfx->nAvgBytesPerSec = eax;
100010e0: mov    eax,ebx                         ; eax = hxas;
100010e2: mov    WORD PTR [ecx+0xe],0x10         ; wfx->wBitsPerSample = 16;
100010e8: mov    WORD PTR [ecx+0x10],0x0         ; wfx->cbSize = 0;
100010ee: pop    ebx                             ; esp offset to eip is 0
100010ef: ret                                    ; return (hxas);
100010f0: [...]

The decoder always produces 16bit PCM at the same rate as the XA file, and this code confirms that the WAVEFORMATEX structure pointer is an “output” argument, not a regular argument asking to produce “that kind of WAVE data”. It means that the rest of the code should remain simple as well and that’s quite motivating to follow this through the end.

Collecting my precious

With all that, my first pieces of the puzzle:

typedef TODO xa_inflate_f(TODO, TODO);
typedef TODO xa_convert_f(TODO, TODO);

struct xastream {
        struct xaheader hdr;
        uint32_t        dw32;
        uint32_t        dw36;
        uint32_t        blk_sz; /* 40 */
        xa_inflate_f    *inflate_cb; /* 44 */
        xa_convert_f    *convert_cb; /* 48 */
}; /* 52 */

Unfortunately that uncovered more pieces, but most of the opaque structure is known and that is a very significant progress. In the next post I will take a step back and look at more trivial matters.

First contact with "the" XA codec

Sun, 01 Jul 2018

Cracking the codec open

I may have complained in a past blog post that retdec as a decompiler didn’t produce readable or helpful code, I may have even said that the code was horrible, but I also showed that it was rather good at figuring library calls. So in order to understand that the four Windows function calls could be reduced to a plain free(3) call I had to both look at the decompiled code for xaDecodeOpen and documentation of the functions and their flags on the MSDN.

These days I’m way more comfortable with the man utility, but in a sense I also think that online docs are very important. In this regard, I always found Microsoft’s MSDN a very good place to browse documentation. It has always been a terrible place to start browsing though. If it weren’t for search engines, I would never find the starting point.

In order to decompile xaDecodeClose I needed to look at xaDecodeOpen to enumerate all memory management functions:

GlobalAlloc
GlobalFree
GlobalHandle
GlobalLock
GlobalUnlock

In portable C code that can be safely reduced to calloc(3) and free(3). No big surprise here, it is the memory management of the opaque structure, the “handle”. In other words, stuff hidden under the compiler rug that I will need to uncover, gathering clues like a detective, trying to fit all the pieces together. And that first clue was waiting there in the open to be picked up: xaDecodeOpen’s second argument.

The big bad WAV file

One misconception about WAV files is that they are big because uncompressed, another is that they are WAV files. WAV files are really one kind of RIFF file, a generic container that specialized in audio and video. A RIFF file is made of chunks and a WAV file is a RIFF file with a WAVE chunk describing how its audio data is encoded, and then a data chunk containing the aforementioned audio data. You could for example use an MP3 codec in a WAV file, and you would get a file approximately the same size as an MP3 file. It’s probably even possible to add things like track information or subtitles to a RIFF file, similar to MP3s ID3 tags but at this point, I digress…

The reason most WAV files are big is because of their number one (literally) and probably most widely used codec: PCM or Pulse-Code Modulation. To keep it short (or digression-free) the signal is represented with a fixed duration for samples and each samples has a fixed value. In stereo mode, samples are interleaved, with one left sample followed by one right sample for each point in time. Interleaving is streaming-friendly as you may read the audio without the need to jump back and forth in the audio stream and can avoid pseudo random accesses. Arguably, picking the smallest granularity (individual samples) may not be that efficient, but I can’t help it, I digress.

From XA to PCM

Looking at sample.c, xadec.h and some MSDN documentation we roughly see this:

typedef struct _XAHEADER {
	ULONG	id;
	ULONG	nDataLen;
	ULONG	nSamples;
	USHORT	nSamplesPerSec;
	UCHAR	nBits;
	UCHAR	nChannels;
	ULONG	nLoopPtr;
	SHORT	befL[2];
	SHORT	befR[2];
	UCHAR	pad[4];
} XAHEADER;

typedef struct {
	WORD	wFormatTag;
	WORD	nChannels;
	DWORD	nSamplesPerSec;
	DWORD	nAvgBytesPerSec;
	WORD	nBlockAlign;
	WORD	wBitsPerSample;
	WORD	cbSize;
} WAVEFORMATEX;

int
main()
{
	FILE *fp;
	XAHEADER xah;
	WAVEFORMATEX wfx;
	XASTREAMHEADER xash;

	fp = fopen("sample.xa", "rb");
	fread(&xah, 1, sizeof(XAHEADER), fp);
	hxas = xaDecodeOpen(&xah, &wfx);

	/* ... */
}

At this point we could assume that xadec.dll will turn an XA file into a WAV file but no. While I claimed that this XA file format doesn’t look like something the Bandjam author came up with, the use of WAVEFORMATEX on the other hand looks like a Windows-oriented choice in API design. The wfx structure is passed as an “output” parameter, only to neatly pack information about the resulting audio in a single data structure. We technically already have all we need in the XAHEADER structure, provided that we know one more secret about xaDecodeOpen (hint: it’s hidden in the assembly.)

The XA header

The fread(3) call gives us one vital piece of information: the XA header is located at the beginning of the file and is serialized using little endian integers. It means that at this point I’m able to dump the header of an XA and keep track of how the header fields are used using their offset, and that should help me dig further.

I’m always baffled that ISO C99 doesn’t offer anything to deal with the byte order of a given architecture. The byte orders big and little endian define in which order a “word” is laid out in memory. Confusingly enough, endian refers to one end or the other of the “word” once broken down into octets, so in this case values from the XA header start from the little end. It’s as confusing as finding the shutdown function in the start menu on Windows.

Apparently the name was borrowed from Johnathan Swift’s Gulliver’s Travels in which rebels would break their eggs from the big end in a political opposition to the Lilliputian king who’s tyrannic rule imposed breaking them from the little end. So in computing we find architectures relying on either ordering (and in some cases even both I’m told). It can get even more messy with “word” order.

Network protocols tend to favor big endian representation over the wire so it’s often referred to network byte order too. And for no apparent reason C was left with nothing to deal with architectural differences besides the usual undefined behavior. Instead we have cryptic function names from another time like ntohs and ntohl to convert numbers back and forth between network and host (CPU architecture) byte order. Nothing part of the C standard to go back and forth between both byte orders. Either those functions are no-op on a big endian host, or they swap the ordering. But I digress.

The header looks mostly straightforward: a magic number, audio information, some padding to align to a power of 2 byte size (32) and the number of bytes (nDataLen) after the header. Then there are less obvious fields like nSamples that can probably be computed from (and sanity-checked against) the other values and the befL and befR arrays. They will turn out to be very important clues that I only understood in retrospect. I’m not sure whether I could have figured out their purpose beforehand though, but they definitely confirmed my interpretation of the codec.

However, this is a topic for later. In the next post I will describe the decompilation of xaDecodeOpen and how encouraging it was.

My first function decompilation

Sun, 24 Jun 2018

First x86 contact

My hopes to offload as much work as possible were shattered as time was running out and I wasn’t finding the right tool for the job that would be both easy, efficient and not require too much investment from me. If decompilation is not a silver bullet, then there’s no getting around learning some assembly to understand the target code and figure what the original source code may have looked like. Once you have working code, you may be happy and move on to using it but I’m way too curious to not try to also understand how things work beyond mechanically turning a byte stream into a different byte stream.

They say curiosity killed the cat, so I’m probably a serial pet killer at this point and have always been. Of course it’s not my fault, it never is. I can easily shift the blame to public school that gave me the taste to learn and the lack of responsible adults when I was a kid to tell me how I would endanger an entire feline species. But I digress…

So soon after I discovered programming, I tried to learn more about computers in general and eventually I got my virtual hands onto a file called nasm.chm that would keep me occupied for a while. I would mainly write small useless programs using a handful of x86 and x87 instructions to compute well known functions using their Taylor series approximation.

Learning x86

To be honest, I don’t remember whether I was using nasm or fasm, but either way it came with a CHM file: a web based self-contained file with an index and the possibility to jump between sections via hyper links. You’d even find trivia like how many CPU cycles each instruction would take.

These days I’m way more comfortable with the man utility, but in a sense CHM can be viewed as a GUI alternative to GNU info. While I understand the rationale and terminal appeal behind info, I’m always confused whenever I need to look up something not mentioned in the standard manual. In that CHM file I would easily find everything and that probably played a part in the learning process, but I digress.

So let’s look at a very small and simple function:

10001110: push   %esi
10001111: mov    0x8(%esp),%esi
10001115: test   %esi,%esi
10001117: jne    0x1000111d
10001119: xor    %eax,%eax
1000111b: pop    %esi
1000111c: ret
1000111d: push   %edi
[...]
1000113f: ret

This looks nothing like what I learned about x86 assembly. Well it does look familiar but I don’t understand the notation. It turns out this is the AT&T notation and even though I’ve been living on the UNIX-like side of the fence for years now I’ll stick to the Intel notation:

objdump --disassemble-all --disassembler-options=intel xadec.dll

Much better, I can now start my mental decompilation:

10001110: push   esi                      ; esp offset to eip is now 4
10001111: mov    esi,DWORD PTR [esp+0x8]  ; esi = arg0
10001115: test   esi,esi                  ; if (arg0 == NULL) {
10001117: jne    0x1000111d               ;     /* else jump */
10001119: xor    eax,eax                  ;     eax = 0
1000111b: pop    esi                      ;     esp offset to eip is now 0
1000111c: ret                             ;     return (0);
                                          ; }
1000111d: push   edi                      ; esp offset to eip is now 8
[...]
1000113f: ret

Isn’t it funny that physicists accidentally called atoms as such because they were thought to be indivisible? It turns out they are made of electrons, protons and neutrons. And we can even break some further down to what we now think are elementary particles although at this point, I digress…

I was lucky that xadec.dll contained only one instruction I had never encountered before, so the learning curve was almost flat. Instructions are like atoms, the smallest decomposition of work we can give an x86-compatible processor although depending on the operands they will yield different opcodes. The opcodes and their argument bytes don’t bring any value to the objdump output so I cut them out.

Prior exposure to x86 code helped a lot. This way I knew immediately that xor eax, eax is a common alternative to mov eax, 0 and wasn’t bewildered that esi would be tested against itself. Why though, I don’t know, but I suspect this may simply be more efficient or result in smaller machine code. I may be curious, but I can also enjoy a bit of mystery.

However, how can I tell that arg0 is a pointer and not simply a scalar? How can I assume that this is a null check?

The first clues

The snippets above are from the xaDecodeClose function, one of the 4 public symbols I can find in the disassembly:

Export Address Table -- Ordinal Base 1
        [   0] +base[   1] 1110 Export RVA
        [   1] +base[   2] 1170 Export RVA
        [   2] +base[   3] 1000 Export RVA
        [   3] +base[   4] 1140 Export RVA

[Ordinal/Name Pointer] Table
        [   0] xaDecodeClose
        [   1] xaDecodeConvert
        [   2] xaDecodeOpen
        [   3] xaDecodeSize

Looking at other properties of the file I can make sense of the address 1110 of the first table, and deduce that xaDecodeClose is located at 10001110:

BaseOfCode              00001000
BaseOfData              00006000
ImageBase               10000000

All public symbols seem to land in the code section of the library. The C code generated by retdec also confirms my amazing deduction of the location of xaDecodeClose with my mighty arithmetic powers:

// Address range: 0x10001110 - 0x1000113f
int32_t xaDecodeClose(char * pMem, int32_t a2) {
    // 0x10001110
    if (pMem == NULL) {
        // 0x10001119
        return 0;
    }
    // 0x1000111d
    GlobalUnlock(GlobalHandle(pMem));
    GlobalFree(GlobalHandle((char *)(int32_t)pMem));
    return 1;
}

I said earlier that retdec produces horrible C code, this is quite readable. Don’t be fooled though, because retdec was confused and got the signature wrong. It failed to figure the calling convention of the library but that didn’t bother me too much, I knew.

How did I know that? And how did I know that arg0 is a pointer prior to peeping at the decompiled C?

Open source to the rescue, again!

Even though there’s no source code for xadec.dll, it has been used by more than one project so they need to know how to use it. Thanks to DTXMania being open source, I not only found the bundled DLL in the source tree but also a xadec.h file and even better: a sample.c file.

After wasting a fair amount of time failing to cross-compile sample.c to run it in Wine, hoping to get a sample output to compare to what my code would provide, I gave up. I had everything: xadec.dll, xadec.lib, xadec.h, a tool chain to build Windows binaries and Wine to run sample.exe but it would always fail to link and at some point I ran out of the 2-hour budget I had allocated for that. I realized later that it wouldn’t be a problem.

Not too long ago, we discovered that one of our fellow Varnish developers is a beekeeper on his spare time. That reminded me of an old joke of mine that for some reason never caught on: API culture. That was supposed to be a pun: “I’m doing apiculture” was suppose to mean “I’m designing the API”. In French “apiculture” means beekeeping. It’s a mystery why this joke never caught on.

Anyway, thanks again to DTXMania I could sum up the API to this:

typedef struct _XASTREAMHEADER { ... } XASTREAMHEADER;
typedef struct _XAHEADER { ... } XAHEADER;
typedef HANDLE HXASTREAM;

HXASTREAM __cdecl xaDecodeOpen(XAHEADER *, WAVEFORMATEX *);
BOOL __cdecl xaDecodeClose(HXASTREAM);
BOOL __cdecl xaDecodeSize(HXASTREAM, ULONG, ULONG *);
BOOL __cdecl xaDecodeConvert(HXASTREAM, XASTREAMHEADER *);

This is a gold mine:

2 structure definitions
1 opaque structure
4 function signatures
1 calling convention

After a quick MSDN search and a quick look at other Win32 functions used in xaDecodeOpen I was able to decompile the function:

void
xadec_close(struct xahandle *hdl)
{

	free(hdl);
}

My actual code doesn’t look like that, but in essence I don’t care about the return value and free(3) already does a null check for me. I can now move to the next low-hanging fruit because thanks to my past self I didn’t need more research before proceeding. In the next post I will discuss audio file formats and the WAVE file format in particular.

Prep work for xadec.dll

Wed, 20 Jun 2018

Enter the craft

I don’t know where I got this piece of advice, but one should consider that a super hero might be someone’s first super hero. In other words, before talking about something in depth one should make sure the audience has enough context to follow. Speaking of context, this is my second post on a series about reverse engineering of xadec.dll, a library used in an open source game I happen to like but missing one crucial thing: its source code.

So after a lengthy origin story that could be summed up to “I used to play fake drums on a home console and then on a Windows desktop but these days I only use a laptop running Fedora” it is now time to briefly introduce reverse engineering with the assumption that maybe the reader knows nothing or little about the topic.

Reverse engineers may not be super heroes, but after spending overall 30 hours on a small and simple library I can confirm this is no easy task. Much like Batman and all his bat tools, it would take forever to figure anything out without standing on the shoulder of giants.

Speaking of Batman, I love the irony behind the name. While it’s obviously the combination of “bat” and “man” it also results in an actual word. This turns Alfred into Batman’s batman, considering Bruce’s war on crime and Alfred’s Britishness and military background. But I digress…

Compilation is a lossy process

It’s easier to go from source code to target code. The other way around it requires additional efforts to guess anything that was lost during compilation since your processor cares little about type systems or variables and has to fit whatever high level instructions one may write into a limited set of instructions and registers. Turn optimizations on and you end up with even more obfuscation of the original code.

Disassembly is very easy, it’s only a matter of finding where the code is in a container. On Linux that would usually be somewhere in an ELF (Executable and Linkable Format) file, while on Windows it’s more likely to be found in a PE (Portable Executable) file. It may not only apply to native code though, a JVM will look for byte code in class files, and in essence any target code can easily be disassembled.

Decompilation on the other hand is very hard. The more we lose during the compilation process, the harder it becomes to perform the inverse operation. Debug information can probably greatly help, but xadec.dll doesn’t include any so going from native code to C code requires a fair amount of guessing. As an x86 library, it will map 1-1 with x86 assembly. Much like Java byte code almost maps 1-1 with Java source code, it’s usually easy to disassemble. Now try decompiling Scala code for example, and you will probably get some Java pidgin you’ll need to mentally map to proper Scala (unless tools made progress in this area).

Hammer time

I’ve been involved in the Varnish Cache project for a fair amount of years now and it has become my solution to most problems. Granted, most problems my day job bring to the table tend to be directly related but at some point any problem looks like it could be solved with a cache. Amusingly, one of my customers reached the support with interesting questions involving a disassembly of code generated by Varnish while I was knee-deep in this topic so that sure was fortunate for me to have spent some time reading assembly listings and be able to not only follow but contribute to the support case. But I digress…

When all you have is a hammer, all problems look like nails eh? One reason for the reverse engineering was also the prospect of doing something new and safely leave the comfort zone. Failing would bear no consequences other than losing hobby time and succeeding would be quite the payoff (although in retrospect it wasn’t that much).

It took me around two weeks to evaluate different tools, but evaluate may be an overstatement. The first one I tried is an online tool called ODA that scared and misled me into thinking I needed to know things like the endianness and word size of the library in addition to its calling convention. Maybe because I replied incorrectly to those questions the interface would show me many functions and symbols but nothing very useful. At this point I was already thinking that I was heading straight to failure.

I then searched for disassemblers in my package manager and found a couple of them available on Fedora. I tried examiner, a tool that adds value on top of objdump, but it would only give me this:

Binary file is not a known executable type.
xadec.dll: PE32 executable (DLL) (GUI) Intel 80386, for MS Windows

So I gave up on both of them and went back online to look for more tools.

Next I moved to Github for a search because most of the results on Fedora were either libraries or frameworks for disassembly. So maybe I would find actual programs making use of them that nobody maintains in Fedora. And I found half a dozen projects, some looking more alive than others. The only one I managed to build and run is called Plasma. Hobby time is scarce, and after spending so much time looking for tools and trying to learn this one I turned to the dark side again and looked for more online tools.

Giving up on tools

The first thing I found was that a recent-enough version of objdump could indeed disassemble PE32 executables but looking at the blog post’s date, published years ago, I decided to give it a try:

$ objdump --disassemble-all xadec.dll | wc -l
9435

Good… I guess? That sure looks like a lot of work. At least I don’t need to install a custom build of GNU binutils.

Then I shifted to decompilers and found one available online: the Retargetable Decompiler aka retdec. Not only was it able to produce C code, it also came with SVG graphs to follow the code flow and it was smart enough to give their proper names to 3 out of 4 of the public functions. And even better, I could download the entire archive with all the goodies available from my web browser. At the moment I’m writing this, retdec is no longer available online, I was very lucky!

However the largest graphs (incidentally most of them) were not built on purpose, probably for resource savings (and I won’t blame a free service for that) so as soon as I noticed the Github link I went ahead and cloned the repository to build a copy locally.

During the lengthy build process I read some of the documentation and watched a video presenting the project. It is very interesting, I recommend at least looking at the architecture. Once the build was done, I was able to run it locally and produce the C code. Bonus point: the master branch was able to map the function names of the 4 public functions. I was also able to figure that much with objdump but it’s nice to get a confirmation from retdec. On the other hand I didn’t get any fancy graphs. I gave up and kept the C code around just in case. The code is understandably horrible because of the loss of information, but it would sometimes help to compare my observations with retdec’s deductions. I barely used the decompiled C, and in the end never made use of those nifty SVG graphs…

And that’s how after spending a good 10 hours over the course of two weeks I decided to proceed because I could otherwise spend many more hours trying to find a tool I’d ultimately need to spend more time learning. Leaving the comfort zone sucks sometimes, but in the next post I will show that I had a couple tricks up my sleeves that kept the whole endeavor not that uncomfortable.

Reverse Engineering xadec.dll

Sun, 17 Jun 2018

Playing music

The year is 2005, a game called Guitar Hero comes out and makes 2006 the year when music games become mainstream worldwide. Surfing on the wave then came Rock Band in 2007 and you could play more than just the guitar, but also have other instruments including a drum kit!

Rewind to 1997, Konami started releasing series of musical games of various kinds under the Bemani umbrella. Those were a huge hit in Japan and in 1999 came the first releases of the GuitarFreaks and DrumMania games that could be played together and became mainstream in Japan many years before the rest of the world would enjoy music video games.

I remember back when Guitar Hero came out, some people would smugly remark how pointless those games were and how you’d be better off learning the real instruments in the first place. Living in France, I could always reply that they should stop playing football (soccer) on their home console and just play on the field. That would usually make my point, but I digress.

Thanks to a friend I discovered the Bemani games but unfortunately it was quite expensive to import any of it so the best I could afford was one game, one guitar controller and one drum kit controller. I couldn’t afford a Japanese PlayStation so instead I modded the one I already had and all was fine.

Playing actual music

Soon after that same friend enrolled me in an actual band and I learned the hard way the difference between arcade and simulation. It turns out playing drums involves more than hitting the right drum or cymbal at the right time. But that’s why those games are successful. You don’t need to learn an actual instrument to play or cover songs, you’re supposed to have fun!

So I kept having fun with a GuitarFreaks and DrumMania clone available for Windows. The final piece of hardware I needed was a PS1 to USB adapter and suddenly I had access to more songs than just the ones present on the one game I purchased.

I eventually moved to a place where there was not enough space to keep the game controllers or an electronic drum kit around and moreover ditched Windows in favor of Fedora, because free software. So all I was left with was an actual band enabling me to compose and play actual drum scores, poor me.

Enter xadec.dll

Years have passed and soon I should have enough space to take my electronic drum kit out of its box every now and then and the very thought made me research this topic lost in time.

It turns out the game I was playing at the time is open source! I wouldn’t have guessed that I could actually get the source code for DTXMania but yes, it is available. But no luck, it builds only for Windows and doesn’t run in Wine.

After a quick search, I came across DigiBand, another GuitarFreaks and DrumMania clone that can even play DTXMania songs. Sadly the project looks dead and I haven’t tried to build it yet because at first glance it looked like a painful process (at least on Fedora).

Then my butterfly senses triggered when I read a disturbing note in the cook book for Windows:

To play alot of the already existing songs for DigiBand, you will need XA decode. XA decode is a decoder written by a Japanese programmer for an application called bandjam. It has long gone extinct, however DTXMania, Session stream, and DigiBand all require the use of XA Decode so the developer released the XA decoder as a binary only library. This is why XA will not work for DigiBand under linux. We are however, moving to deprecate this library with the use of OpenAL. The XA format was originally used because MP3 and wav had inconsistant overhead and latency when playing back on demand.

I checked in the DTXMania code repository and there is indeed a xadec.dll library bundled with the rest of the code. So apparently a 32bit DLL became crucial but its source code was lost because bandjam (another GuitarFreaks and DrumMania clone) was not open source when it disappeared.

So that’s how I decided to take a stab at it. I would try to undo a wrong and at least give the opportunity to DTXMania to ditch xadec.dll and enter the 64bit world. Who knows, maybe then I will be able to run it in Wine?

This is a good example of why free software and open standards matter. By reverse engineering xadec.dll I was able to confirm its focus on latency, but I think another goal was compression and because the PlayStation already had its own XA format I’m guessing that having a different audio codec may have been a requirement to prevent piracy (with the usual effectiveness) because of licenses involved in a music game with lots of titles. Considering the hardware capabilities of the first PlayStation I wouldn’t be surprised if this was actually the format found in the games rather than a thing the bandjam developer came up with.

In the next posts of this series on xadec.dll I will share the results of my very first reverse engineering journey.