TLS sandboxing

Reinventing TLS

After being a month away from $DAYJOB I couldn’t help myself not thinking about TLS sandboxing. But with limited spare time to give this proper thought I had to come up with something easier to grasp and implement. And as such I came up with my own SOCK_STREAM secure layer called MXC.

It’s very simple: you first agree on a session key using MUL operations, and at the end of the handshake the client can send encrypted traffic using XOR operations. MXC stands for Multiplication and eXclusive-or enCryption. I’m waiting for a pending security audit to complete before I can submit MXC to a standards body. It might take some time considering how long it took for TLS 1.3 to settle.

Since this is a thought experiment, I also decided to skip error handling and keep things to a minimum. So assert(3) is used where it wouldn’t make sense at all to make further progress outside the happy path, and otherwise return values are ignored using a (void) cast.

The protocol primitives are implemented in a short C header library called mxcrypt.h.

Needless to say, this implementation has many shortcomings, and the handshake won’t work unless both client and server share the same endianness, but I digress.

The client

The protocol library was designed so that both clients and servers could almost fit on a slide:

#include "mxcrypt.h"

void
main(int argc, char * const *argv)
{
	struct mxc_sess sess[1];
	char buf[256];
	ssize_t len;
	uint16_t port;
	int fd;

	fd = mxc_socket(argc, argv);
	port = mxc_port(fd);
	LOG("connected via port %hu", port);

	mxc_sess_init(sess);
	mxc_hello_client(sess);
	mxc_hello_send(sess, fd);
	mxc_hello_recv(sess, fd);
	mxc_sess_key(sess);
	LOG("computed session key 0x%08x", sess->key);

	do {
		len = read(STDIN_FILENO, buf, sizeof buf);
		assert(len >= 0);
		mxc_sess_send(sess, fd, buf, len);
	} while (len > 0);

	(void)close(fd);
}

The client is very straightforward: first it initializes a session and generates a client hello for the handshake, then it sends its own hello to the server and waits for the server hello in the reply. At this point the key can be derived and the client proceeds to forward its standard input encrypted over the network.

The server

Again, thanks to how the library was designed the server is equally straightforward:

#include "mxcrypt.h"

void
main(int argc, char * const *argv)
{
	struct mxc_sess sess[1];
	char buf[256];
	ssize_t len;
	uint16_t port;
	int sock_fd, sess_fd;

	sock_fd = mxc_socket(argc, argv);
	port = mxc_port(sock_fd);

	LOG("listening to port %hu", port);

	sess_fd = accept(sock_fd, NULL, NULL);
	assert(sess_fd != -1);

	mxc_sess_init(sess);
	mxc_hello_recv(sess, sess_fd);
	mxc_hello_server(sess);
	mxc_hello_send(sess, sess_fd);
	LOG("computed session key 0x%08x", sess->key);

	do {
		len = mxc_sess_recv(sess, sess_fd, buf, sizeof buf);
		assert(len >= 0);
		(void)write(STDOUT_FILENO, buf, len);
	} while (len > 0);

	(void)close(sess_fd);
	(void)close(sock_fd);
}

It requires one additional socket compared to the client, one for the port it is listening to and one for the single connection it will accept. Then it’s almost the same dance as the client. Initialize the session and wait for the client hello, then generate and send the server hello. At this point the key was already derived, courtesy of the MXC implementation. Finally the server decrypts the client’s contents and forwards them to its standard output.

The problem

Now that I have working client and server implementations of MXC I want to improve the handshake by adding the possibility to present a certificate for authentication purposes. But there is a little problem with the protocol so far. Actually there are so many problems with both the protocol and implementations that your average crypto person might already be suffering a heart attack.

I’ll focus on one problem though, and the astute reader may have already guessed that this is going to be about sandboxing.

The main problem here is that an MXC might do too many things:

• require root privileges depending on which port it uses
• deal with the private key of the certificate during the handshake
• process untrusted input from potentially adversary parties

The first problem is technically not a problem. Unfortunately dropping privileges is often a privilege, so sandboxing might be more effective with system administration rights in the first place.

The real problem is that currently if there is an application-level bug below MXC, the “bug” is in the same address space as the certificate’s private key, and could do its damage with the aforementioned root privileges.

Incremental solutions

There is one straightforward solution which consists in having a proxy that terminates MXC traffic and forwards clear traffic to the application. It is also not entirely satisfying as a solution because it requires twice as many sockets on the system, and is both a resource and operational overhead. It could also not be an operational overhead if it terminates MXC for multiple services, but that still leaves one undesirable detail. Despite address space isolation between processes we still end up sending CLEAR traffic between them. We can mitigate the problem by co-locating the proxy and application on the same system, and using a Unix-domain socket with proper permissions, but then it becomes harder to amortize the operational overhead. The resource overhead is however here to stay.

If we end up with both the proxy and the application on the same host we can get rid of the clear traffic problem. Unix-domain sockets have one interesting property of inter-process communication, they can pass file descriptors from one process to another. In order to do that, there needs to be a new protocol between the proxy and the application so that forwarding the connection itself instead of its traffic can happen.

Next, we can solve the operational overhead by teaching the application how to do the sandboxing. Much like we can create a pipe(2) before forking a process, we can also create a Unix-domain socketpair(2) to create a private channel before two processes.

MXC sandboxing

Now that I have a working implementation of a client and a server, I can write a new server with sandboxing and check that it still works with the original client.

The admin process

The main() function would look like this:

void
main(int argc, char * const *argv)
{
	pid_t acceptor_pid, server_pid;
	int pair_fd[2];

	assert(socketpair(PF_UNIX, SOCK_STREAM, 0, pair_fd) == 0);

	acceptor_pid = fork();
	if (acceptor_pid == 0) {
		(void)close(pair_fd[0]);
		main_accept(argc, argv, pair_fd[1]);
		return;
	}
	assert(acceptor_pid != -1);

	server_pid = fork();
	if (server_pid == 0) {
		(void)close(pair_fd[1]);
		main_serve(pair_fd[0]);
		return;
	}
	assert(server_pid != -1);

	(void)close(pair_fd[0]);
	(void)close(pair_fd[1]);

	assert(waitpid(acceptor_pid, NULL, 0) == acceptor_pid);
	assert(waitpid(server_pid, NULL, 0) == server_pid);
}

Each process gets one end of the socket pair, but unlike a pipe(2) it is bidirectional so care needs to be taken to make sure no privilege escalation is possible between the server process (that receives application traffic) and the acceptor process (that manipulates certificates and their private keys).

The acceptor process takes the original command line arguments to figure which port it will listen to while the server only gets the socket from which it will receive MXC sessions.

The acceptor process

The main_accept() function is simple enough:

void
main_accept(int argc, char * const *argv, int server_fd)
{
	struct msg_sess ms[1];
	uint16_t port;
	int sock_fd;

	sock_fd = mxc_socket(argc, argv);
	port = mxc_port(sock_fd);

	LOG("listening to port %hu", port);

	ms->sess_fd = accept(sock_fd, NULL, NULL);
	assert(ms->sess_fd != -1);

	mxc_sess_init(ms->sess);
	mxc_hello_recv(ms->sess, ms->sess_fd);
	mxc_hello_server(ms->sess);
	mxc_hello_send(ms->sess, ms->sess_fd);
	LOG("computed session key 0x%08x", ms->sess->key);

	send_msg_sess(ms, server_fd);

	(void)close(ms->sess_fd);
	(void)close(sock_fd);
}

It does everything the previous server did up to the handshake, then it sends the MXC session to the server and finishes. After getting a hold of the TCP port and access to certificates, this process could drop many privileges. It could even drop privileges prior to accessing certificates if they are made available beforehand to a dedicated unprivileged user.

The server process

It should be easy to guess from the first server implementation and the acceptor process what this one looks like:

void
main_serve(int acceptor_fd)
{
	struct msg_sess ms[1];
	char buf[256];
	ssize_t len;

	recv_msg_sess(ms, acceptor_fd);

	do {
		len = mxc_sess_recv(ms->sess, ms->sess_fd, buf, sizeof buf);
		assert(len >= 0);
		(void)write(STDOUT_FILENO, buf, len);
	} while (len > 0);

	(void)close(ms->sess_fd);
}

This process could drop its privileges right away, unless the application needs to acquire resources before that. In this example of course, we already inherited the standard output so in theory we could drop everything at the very beginning.

Passing the MXC session

Passing a file descriptor between processes is only one thing we need to do. We can see in the server process that we still need a valid session to decrypt the traffic. Thankfully, it is possible to do this somewhat atomically: the session can be serialized and sent as the payload whereas passing a file descriptor to a unix-domain socket uses the control payload of a message.

In this case, we use a private socket and the MXC session is self-contained so it can be sent as its in-memory representation:

void
send_msg_sess(struct msg_sess *ms, int fd)
{
	struct msghdr msg[1];
	struct cmsghdr *cmsg;
	struct iovec iov[1];
	union msg_buf buf[1];

	iov->iov_base = ms->sess;
	iov->iov_len = sizeof ms->sess;

	(void)memset(msg, 0, sizeof msg);
	msg->msg_iov = iov;
	msg->msg_iovlen = 1;
	msg->msg_control = buf;
	msg->msg_controllen = sizeof(buf);

	cmsg = CMSG_FIRSTHDR(msg);
	cmsg->cmsg_level = SOL_SOCKET;
	cmsg->cmsg_type = SCM_RIGHTS;
	cmsg->cmsg_len = CMSG_LEN(sizeof(int));
	(void)memcpy(CMSG_DATA(cmsg), &ms->sess_fd, sizeof(int));

	assert(sendmsg(fd, msg, 0) > 0);
	LOG("sent session for file descriptor %d", ms->sess_fd);
}

The struct msg_sess type conveniently carries both session state and the file descriptor, in other words the data and control payloads:

struct msg_sess {
	struct mxc_sess	sess[1];
	int		sess_fd;
};

The recv_msg_sess() function isn’t too interesting to look at at this point because it mostly looks like send_msg_sess().

MXC sandboxing benefits

This scheme greatly reduces the overall attack surface. The main “admin” process very likely running as root doesn’t touch any application traffic in this example. The acceptor process that would have access to certificates and private keys if I were to extend the MXC protocol would have very limited interactions with untrusted input, namely handshakes. And finally the actual server implementing some sort of application logic would only have access to the ephemeral session key.

It would also have less overhead because with one program we may now both terminate encrypted traffic and process it at the application level. We’d still have three processes, but coming from the same executable file they’d share more and waste less. Finally we don’t need more sockets on the host system.

One nice twist of the second server implementation is the unidirectional traffic over the unix-domain socket pair. This way there is little risk of having a compromised application send commands to the acceptor process to leak MXC private keys (even though I didn’t implement certificates). This only works if the protocol no longer needs the certificate once the handshake completes. Otherwise one might design the sandbox differently, or have a two-way dialog between the acceptor and the server.

Also, in the likely event that the session state would not be of a fixed size like the MXC protocol so far, the acceptor may have read data past the handshake and should serialize any data in the pipeline along with session state.

TLS sandboxing

Circling back to $DAYJOB, I sometimes start pondering the two times where HTTPS support was considered in Varnish, which resulted in the Hitch TLS terminator proxy.

The first one says the following that stuck in my head:

Will that be significantly different, performance wise, from running a SSL proxy in separate process ?

No, it will not, because the way varnish would have to do it would be to … start a separate process to do the SSL handling.

There is no other way we can guarantee that secret krypto-bits do not leak anywhere they should not, than by fencing in the code that deals with them in a child process, so the bulk of varnish never gets anywhere near the certificates, not even during a core-dump.

In other words, if we wanted native HTTPS support in Varnish we would require the kind of sandboxing implemented as a thought experiment for MXC.

The second take points to a keyless TLS implementation that uses private keys stored remotely during the handshake, validating the MXC sandboxing model and adding one more layer of security. If the MXC (or at this point TLS) handshake can be exploited, at least it will be significantly harder to steal private keys.

A keyless handshake also makes the whole fork(2) with socketpair(2) dance somewhat obsolete, but it reintroduces operational overhead: instead of a TLS terminator proxy the application needs a key server and a new can of security worms to deal with. On the other hand it can probably be amortized rapidly for CDN setups, but I digress.

OpenSSL sandboxing

Now that TLS could hypothetically be sandboxed safely, allowing the handshake to happen in a separate process, possibly using private keys remotely without ever fetching them, comes the question of a TLS implementation.

The obvious candidate for multiple reasons would be OpenSSL, but can it do that as of today? It is possible to serialize a session to an ASN1 or base64 representation, so this could be passed alongside the file descriptor to the cache process (Varnish’s “server” process). We already use this for session resumption in Hitch, but can it be done for a live ongoing connection?

If it is doable, is it portable across all the platforms we care about in the Varnish project? Would the TLS handshake sandboxing create a new bottleneck? My intuition tells me that an application would still perform better without a separate proxy. But it comes with further challenges and may require that we modularize more parts of Varnish like the acceptor subsystem. Would OpenSSL integrate nicely in our delivery pipeline? Many questions I’m not covering for a short thought experiment in the middle of the night.

Bonus MXC demo

I wanted to give an interactive demo of the MXC PoC, so I looked at what was available on Fedora and settled for asciinema. In this scenario I build the client, the sandboxing server, and use a shell script to create a man-in-the-middle proxy. The client passes a secret key to the server and the netcat-based proxy dumps traffic on disk. In order to show all three actors semi-simulteanously I used tmux to split the terminal:

You can see what the client and server are logging, and that the server logs from two different PIDs. When the client and servers are rebuilt with the MXC_CLEAR macro defined, the traffic is sent non-encrypted and the MITM proxy successfully intercepts the private payload.

All the code is available:

• mxcrypt.h
• mxcrypt-client.c
• mxcrypt-server.c
• mxcrypt-server2.c
• mitm-proxy.sh

Finally, one last reason I was thinking about HTTPS support in Varnish is that HTTP/3 will likely be based on QUIC and specified as secure-only so keyless or not we may want to eventually sandbox handshakes. I think we should still support a complete handshake and not force users to have a key server, like we force Varnish users to use proxies for both TLS termination and tunneling today in end-to-end HTTPS setups.