Reverse motivation

After losing my annotated assembly listings, it’s been hard to find motivation to take the most complicated part of the code, reverse engineer it again and comment it here. And this is code I really want to comment because there are interesting findings inside.

So instead I decided to work on the actual project, that goes beyond the quick and dirty reverse engineered code. One thing I learned over the years is that it’s easier to start than finish things, for a given definition of finish. In this case that would be a working decoder, a specification of the codec, all that wrapped nicely inside a source repository, and a first release with all the bells and whistles. And as such bjxa 0.1 was released, matching this definition of finish with some hidden fine prints.

As a result I decided to open a projects section with the result of this reverse engineering self challenge as its first entry.

Thank Wine, again

In order to produce a decoder that would produce a byte for byte identical WAVE file I needed to somehow run xadec.dll but previous attempts at cross compiling sample.c to run it under Wine failed for reasons beyond my tinker spirit.

But since I found that xa.exe program from the same author, I was able to produce WAVE files that would infringe on nobody’s copyright, convert them to XA and back again to WAVE. Giving me an XA to WAVE baseline to check my work. Unsurprisingly Wine had no issues running this (statically linked) program.

I studied the differences between the PCM samples produced by my code and the PCM samples produced by xa.exe and noticed that most PCM samples were very much identical and when they were not, only the least significant byte of the samples were different.

Double wrong means not so wrong

I have yet to disassemble that part again, and it was the hardest bit to reverse engineer for me so it will take some time before I pull it off. But in short, I was right. The fact that the BandJAM author had shipped an encoder made me think that my assumption was wrong that this was not an original codec. But I was actually correct after my initial 30 hours once I got a working C decoder after my mental decompilation of the assembly: BandJAM XA is CDROM XA, the same audio codec as the PlayStation!

Why would I notice that in the first place? It’s a little embarrassing but sometimes I get bored and read other people’s source code. As I said, I’m probably a cat serial killer at this point because my curiosity extends to such lengths that I would sometimes read the source code of things that look magic to me and in this case an open source emulator, but I digress…

CDROM XA has the notion of a block profile and depending on the profile chosen for a block, a gain may be added to the PCM samples of a block, and I already determined in part 5 the size of XA blocks. And since the LSB of the decoded samples was sometimes off, I tried to figure why they would stop being off.

It turns out they’d stop diverging at block boundaries, and one of the profiles doesn’t adjust the gain and as a result is not influenced by past blocks. Quick check: yes, when PCM samples diverge, they converge again on such blocks. Following my intuition I updated my code to decode CDROM XA blocks and here goes nothing: the audio output is a perfect match!

Audio and unicorns

With this in mind, I have to revisit the decompilation of xadec.dll. Either I made some mistakes, or there is a bug in the library and I have reasons to keep this as a possibility. Another reason could be that xadec.dll doesn’t use the same code as xa.exe. Unfortunately, I lost that decompilation, but I’m still planning to redo it once I find my lost motivation.

This got me thinking: why would my code for CDROM XA work better than my decompiled code, especially since the former uses floating point arithmetic and the latter relies only on integers? I looked into this and found a rabbit hole so big that I shouldn’t even start.

Short story: you can find many of the CDROM specifications online and CDROM XA also known as CDROM eXtended Architecture, also known as the Green Book. The Green Book is an extension to the Yellow Book standard and apparently CDROM folks love colors. There are other books and the collection is sometimes referred to as the Rainbow Books.

I found a copy of the Green Book online and it’s about 1000 pages long. The quality of this book should humble most open source projects, I have spent hours reading through. This is a very interesting read, you have been warned.

Robust size

The digression above was brought to you by amazing engineers from the 90s. I believe I left off at the perfect match. Well, almost perfect: there is a bug in the RIFF header generated by xa.exe. So let’s start by decompiling the xaDecodeSize function:

10001140: mov    ecx,DWORD PTR [esp+0x4]        ; ecx = hxas;
10001144: test   ecx,ecx                        ; if (hxas == NULL) {
10001146: jne    0x1000114b                     ;
10001148: xor    eax,eax                        ;       eax = 0;
1000114a: ret                                   ;       return (0);
                                                ; }
1000114b: mov    eax,DWORD PTR [esp+0x8]        ; eax = slen;
1000114f: xor    edx,edx                        ; edx = 0;
10001151: div    DWORD PTR [ecx+0x28]           ; eax /= hxas->blk_sz;
10001154: mov    ecx,DWORD PTR [esp+0xc]        ; ecx = pdlen;
10001158: inc    eax                            ; eax++;
10001159: shl    eax,0x6                        ; eax *= 64;
1000115c: mov    DWORD PTR [ecx],eax            ; *pdlen = eax;
1000115e: mov    eax,0x1                        ; eax = 1;
10001163: ret                                   ; return (1);

It can feel confusing to see that div takes only one operand. It’s only a matter of knowing that in this case the dividend is the combination of eax for the LSB and edx for the MSB. This explains why edx is zeroed out of the blue.

Considering the signature of the function, it can be decompiled as such:

BOOL
xaDecodeSize(HXASTREAM hxas, ULONG slen, ULONG *pdlen)
{
	if (hxas == NULL)
		return (FALSE);

	*pdlen = 64 * (1 + slen / hxas->blk_sz);
	return (TRUE);
}

So, progress here, we have a null check for hxas. On the other hand we lack a null check for pdlen and the arithmetics look bonkers. The additional increment (pun intended) is here to compensate partial blocks. This is that kind of reasoning that leads to the RIFF header bug in xa.exe.

Fortunately, audio players tend to be more robust than that and will agree to play a broken WAVE file if it’s only that kind of corruption. Sure the RIFF size is in most cases smaller than what ends up declared, so this means we can safely ignore anything beyond the WAVE data.

This, is a good example of the robustness principle:

Be conservative in what you do, be liberal in what you accept from others.

Sure, let’s accept everyone else’s bugs or non standard extensions to the point where we can’t move a protocol or file format forward without breaking everyone’s code. This, is the sad reality of the Internet: a slow-moving behemoth that nevertheless keeps on growing faster than it can move. But I digress…

Building up motivation

Another reason why I started this series has yet to come. I’m not teaching x86 assembly or how to reverse engineer code. But I hope it will show that it is possible for someone to start that kind of endeavour by themselves. I was lucky to find an old C library either too old to lose me in optimizations or too simple to be aggressively optimized. And I found more than just code to disassemble and decompile, as cheesy as it may sound. I hope to publish a next post soon, I should only need a couple more to cover everything I wanted to report.