Zig's two languages
A case study of using Zig's comptime
When writing about my first impressions of the programming language Zig, the comptime feature intruiged me. Now that I've written a sizable project in Zig, I've seen how comptime enables new code designs that aren't possible in other systems programming languages like Rust or C++ (as far as I know). Assuming you're somewhat proficient in Rust or C++, I'll try to highlight how Zig's take on compile time execution is different.
Note: The Zig code snippets in this example are simplified compared to the originals. In particular, error handling, allocators, and const/visibility modifiers are removed to make the code less cluttered. My goal is to bring across an intuition about
comptime, not introduce the entire Zig language.
A quick recap: What is comptime?
Comptime just stands for compile time. Before looking at Zig, let's see how Rust and C++ handle compile time execution. Basically, they have two ways of executing code during compilation:
Macros can generate new code at compile time. For example, if you add
#[derive(Eq)]to a Rust type, a macro will inspect the structure of your type during compilation and then generate an equality checking function specialized to that type. However, macros don't really integrate with the rest of the language – they are an entirely separate, new language on top. Most importantly, they can't interact with the language semantics and are instead constrainted to operating on the syntax level:In C and C++, macros only textually replace parts of the code.
In Rust, it depends on the type of macro: Builtin macros have only very narrow, targeted use cases, such as
cfg!for conditional compilation. Procedural macros are similar to C++ macros – they receive aTokenStreamand produce anotherTokenStream. Declarative macros work on a slightly higher level: the syntax tree. That prevents name clashes (making them "hygienic"), but ultimately, they still only operate on the syntax instead of the semantics of the code.
Constant folding is a technique that compilers use to optimize your code. For example, a compiler may replace
2 + 2with4. However, not all code can run during compilation – reading files, making network calls, or generally anything that requires performing syscalls has to happen at runtime. Recently, Rust and C++ have addedconst fn, a feature that allows you to explicitly mark a piece of code as only being allowed to use the subset of the language that can be used at compile time.
If you map out which part of Rust and C++ code can run at compile time vs. runtime, you end up with something like this:
Zig is different. There are no macros. Instead, the language itself has powerful features that let you inspect code, but they can only be used during compilation.
Turns out, Zig actually consists of two languages, the runtime language and the compile time language:
The Zig runtime language is what you usually think of as Zig. This language is statically typed, compiles to machine code, and executes efficiently. You can do all the I/O you want.
The Zig compile time language only runs during compilation. It uses a tree-walking interpreter, similar to how some scripting languages work. The interesting thing: You can opt in to dynamic typing! If you declare a variable of the type
anytype, it can store anything, similar toObjectin Java. You can also inspect the types of values, storetypes directly, and get information about them.
Using the comptime keyword, you can influence what runs at compile time and what doesn't. In this small example, it forces a Fibonacci number calculation to happen during compilation:
const result = comptime fibonacci(10);
Here is code that stores a type in a variable and calls the @typeName function (the @ at the beginning of the name indicates that this is a builtin function).
const MyIntType: type = u8;
const foo: MyIntType = 3;
const type_name = @typeName(u16);
After the compile time interpreter finishes executing, we are left with this program:
// MyIntType is optimized away because it's not used anymore.
const foo: u8 = 3;
const type_name: []u8 = "u16";
When writing Zig, it feels like you have the flexibility of a scripting language at compile time, while still retaining the efficiency of a systems programming language at runtime. You can use this to design really flexible APIs with complex compile time logic. For example, Zig's print function internally uses this formatType function:
fn formatType(value: anytype, writer: Writer) void {
const T = @TypeOf(value);
// If the value has a format method, use that.
if (std.meta.hasMethod(T, "format")) {
value.format(writer);
return;
}
// Otherwise, use the default formatting.
switch (@typeInfo(T)) {
.Void => formatBuf("void", writer),
.Bool => formatBuf(if (value) "true" else "false", writer),
...
}
}
There are a few things to unpack here: The anytype only exists at compile time, so the compiler will generate and type-check a new specialized version of the function for each type that it is used with. This allows the code inside the function to perform any operations on an anytype (such as calling value.format(writer)).
The code also uses several functions to inspect types:
@TypeOf(anytype)returns thetypeof the given value. Thistypecan then be used where a type annotation is expected.std.meta.hasMethod(type, []u8)checks if thetypehas a method with the given name.@typeInfo(type)returns astd.builtin.Type, which is an enum with information about the type.
After the compile time interpreter is done, the formatType function accepting a generic anytype no longer exists. Instead, you are left with a few specialized functions that only contain the necessary control flow. For example, calling formatType with a bool and later on with a User that has a custom format method makes the compiler generate these functions:
fn formatType(value: bool, writer: Writer) void {
formatBuf(if (value) "true" else "false", writer);
}
fn formatType(value: User, writer: Writer) void {
value.format(writer);
}
Unlike with C++ or Rust, optimizing an if with a known condition or a switch on a known value is not optional, but required for Zig to behave as it does. For example, for bool, the std.meta.hasMethod(T, "format") condition evaluates to false, so the content of the if is guaranteed to not be type-checked – otherwise, you'd get a compilation error that bool doesn't have a format method. That also means that this Zig code compiles:
if (false) {
"2" * 12.4; // this is nonsense
}
Soil
Let's use comptime! My programming language Martinaise compiles to a byte code called Soil. Soil is pretty low-level – it has a handful of registers and instructions that operate on them. To get you into the right mindset, here are some Soil instructions:
moveib a 21 | a = 21
moveib b 2 | b = 2
mul a b | a = a * b = 42
syscall 0 | exits with 42 (uses the a register as the error code)
Soil syscalls allow the byte code to hook into functionality that the VM provides – such as exiting in the snippet above. However, not all targets support all syscalls. For example, the server where this blog is hosted, runs Soil. It doesn't have a display system and therefore doesn't support syscalls related to UI rendering.
I ended up writing a small, reusable Zig library that implements everything of Soil except syscalls – those have to be provided. You use the library like this:
// By the way: Imports in Zig are also just a compile time function.
const std = @import("std");
const soil = @import("soil");
fn main() void {
const binary = read_file("my_file.soil");
soil.run(binary, Syscalls);
}
const Syscalls = struct {
fn not_implemented(_: *soil.Vm) callconv(.C) void {
std.debug.print("Syscall not implemented", .{});
std.process.exit(1);
}
fn exit(_: *soil.Vm, status: i64) callconv(.C) void {
const actual_status: u8 = @intCast(status);
std.process.exit(actual_status);
}
...
};
What's going on here?
You create a struct with all the syscall implementations – in this case, the Syscalls struct. Those have to follow a few criteria. For example, they have to use the C calling convention, accept a *soil.Vm as the first parameter as well as i64s (one for each register they want to read). You pass this Syscalls struct to soil.run, which then runs the binary, calling the appropriate function whenever a syscall instruction is executed. If you don't implement a syscall, not_implemented is called instead.
How does it work?
The run function behaves differently based on your CPU architecture. On x86_64, it compiles the byte code to x86_64 machine code. Otherwise, it uses a slower interpreter. This conditional compilation can be done using a regular switch – again, we use Zig's guarantee that only the chosen case is compiled.
fn run(binary: []u8, Syscalls: type) void {
comptime check_struct(Syscalls);
const file = parse_file(binary, alloc);
switch (builtin.cpu.arch) {
.x86_64 => {
const compile = @import("x86_64/compiler.zig").compile;
var vm = compile(file, Syscalls);
vm.run();
},
else => {
const Vm = @import("interpreter/vm.zig");
var vm = Vm.init(file);
vm.run(Syscalls);
},
}
}
The call to check_struct checks that the Syscalls struct contains a not_implemented function and that all functions with names of syscalls have the right signature. The comptime keyword ensures this check happens at compile time. The checks themselves are pretty straightforward:
fn check_struct(Syscalls: type) void {
// @compileError immediately aborts the compilation.
if (!std.meta.hasFn(Syscalls, "not_implemented"))
@compileError("The Syscalls struct doesn't contain a not_implemented function.");
check_syscall_signature(Syscalls, "not_implemented");
// All syscalls need to have good signatures.
for (0..256) |number| {
// name_by_number returns an optional result, and orelse handles the
// case where it returns nothing.
const name = name_by_number(number) orelse continue;
if (!std.meta.hasFn(Syscalls, name)) continue; // not implemented, that's fine
check_syscall_signature(Syscalls, name);
}
}
fn name_by_number(number: u8) ?[]u8 { // the ? indicates an optional return value
return switch (number) {
0 => "exit",
1 => "print",
...
else => null,
};
}
fn check_syscall_signature(Syscalls: type, name: []u8) void {
// @field is like a field access (such as Syscalls.name), but you can pass
// a compile-time known string as the field name.
const function = @field(Syscalls, name);
// @TypeOf returns the function type.
const FunctionType = @TypeOf(function);
// @typeInfo lets us reflect over the structure of that type as a
// std.builtin.Type. The .Fn means we know which enum variant the enum will
// have and we unwrap the payload.
const signature = @typeInfo(FunctionType).Fn;
// The signature is a std.builtin.Type.Fn that contains information about
// the function signature such as the calling convention, parameter types,
// and return type.
if (signature.is_generic)
@compileError("Syscall " ++ name ++ " is generic.");
... // many more checks
}
Because of these checks, if you try to compile the code with an ill-formed struct, you get an error. For example, if one of your syscall implementations takes a parameter of an unexpected type, the compiler will yell at you:
src/syscall.zig:74:17: error: All except the first syscall argument must be i64 (the content of a register). For the print syscall, an argument is main.Foo.
@compileError("All except the first syscall argument must be i64 (the content of a register). For the " ++ name ++ " syscall, an argument is " ++ @typeName(param_type) ++ ".");
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
src/syscall.zig:45:32: note: called from here
check_syscall_signature(Syscalls, name);
~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~
src/root.zig:12:49: note: called from here
comptime @import("syscall.zig").check_struct(Syscalls);
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~
referenced by:
main: src/main.zig:46:17
callMain: /snap/zig/11625/lib/std/start.zig:524:32
callMainWithArgs: /snap/zig/11625/lib/std/start.zig:482:12
main: /snap/zig/11625/lib/std/start.zig:497:12
Note that the stack trace consists of two parts: The upper part is the runtime compile-runtime stack trace of the comptime evaluation. The lower part tells you why the comptime expression was compiled in the first place (how it's reached from main).
The Interpreter
The interpreter basically consists of a tight loop with a switch over the instruction kind. The case for syscalls is interesting:
.syscall => |number| {
// We want to execute the syscall with the given number!
switch (number) {
inline else => |n| run_syscall(vm, Syscalls, n),
}
},
It takes the syscall number (a u8) and then switches over it. Instead of handling cases manually, it uses an inline else to make the compiler duplicate the else branch 256 times. Inside the else branch, it calls run_syscall with a compile-time-known n.
That function then retrieves the syscall implementation, reflects on its signature, and calls it:
fn run_syscall(vm: *Vm, Syscalls: type, comptime n: u8) void {
const fun = comptime syscall_by_number(Syscalls, n);
const signature = @typeInfo(@TypeOf(fun)).Fn;
const result = switch (signature.params.len) {
1 => fun(vm),
2 => fun(vm, vm.get_int(.a)),
3 => fun(vm, vm.get_int(.a), vm.get_int(.b)),
4 => fun(vm, vm.get_int(.a), vm.get_int(.b), vm.get_int(.c)),
5 => fun(vm, vm.get_int(.a), vm.get_int(.b), vm.get_int(.c), vm.get_int(.d)),
else => @compileError("handle syscalls with more params"),
};
// Move the return value into the correct registers.
switch (@TypeOf(result)) {
void => {},
i64 => vm.set_int(.a, result),
else => @compileError("syscalls can only return void or i64"),
}
}
The syscall_by_number function returns the syscall implementation for a given compile-time known number:
fn syscall_by_number(Syscalls: type, comptime n: u8) ... {
const name = name_by_number(n) orelse return Syscalls.not_implemented;
if (!std.meta.hasFn(Syscalls, name)) return Syscalls.not_implemented;
return @field(Syscalls, name);
}
Calling syscall_by_number(Syscalls, 0) results in Syscalls.exit, calling syscall_by_number(Syscalls, 1) in Syscalls.print, etc. But those functions all have different signatures, and thereby different types! So, what's the return value of this function? It depends.
Cue dependent types: Types that can differ based on values. In our case, the return type depends on the n. After lots of struggling, I ended up with the weirdest code I've ever written: A function that duplicates the logic of syscall_by_number, except all returns are wrapped with @TypeOf!
fn syscall_by_number(Syscalls: type, comptime n: u8) TypeOfSyscall(Syscalls, n) {
...
}
fn TypeOfSyscall(Syscalls: type, comptime n: u8) type {
const name = name_by_number(n) orelse return @TypeOf(Syscalls.not_implemented);
if (!std.meta.hasFn(Syscalls, name)) return @TypeOf(Syscalls.not_implemented);
return @TypeOf(@field(Syscalls, name));
}
I would have liked to use anytype as the return value (which should be okay because the function is only ever evaluated at compile time). But functions can't return anytype, so that's not possible. If someone ever finds a more elegant way to write this code, please contact me.
Thankfully, this was the last puzzle piece for a working interpreter. Now, when compiling our interpreter, Zig will go through all numbers from 0 to 256 at compile time, check if those are valid syscall numbers, get the name of the matching syscall, look up the function in the Syscalls struct, and call it with the right arguments. At runtime, a syscall instruction will be handled by code that looks something like this:
.syscall => |number| {
switch (number) {
0 => Syscalls.exit(vm),
1 => Syscalls.print(vm, vm.get_int(.a), vm.get_int(.b)),
...
256 => Syscalls.not_implemented(vm),
}
},
For platforms where not all syscalls are implemented, the unimplemented ones will instead call Syscalls.not_implemented. That's pretty elegant!
The Compiler
On x86_64, the Soil byte code is compiled to machine code. Soil registers are directly mapped to x86_64 registers – the a register lives in r10, the b register in r11, etc. This way, most Soil byte code instructions map to a single x86_64 machine instruction.
I wrote a machine code builder, where you can call methods for emitting instructions. Here's an example code of how to emit instructions that save 42 in the r10 register:
machine_code.emit_mov_soil_word(.a, 21); // mov r10, 21
machine_code.emit_mov_soil_word(.b, 2); // mov r11, 2
machine_code.emit_imul_soil_soil(.a, .b); // imul r10, r11
Compiling a syscall instruction starts the same way as it did in the interpreter: With an inlined switch. Like in the interpreter, inside the inlined else case, we get the corresponding syscall implementation.
.syscall => |number| {
switch (number) {
inline else => |n| {
const fun = syscall_by_number(Syscalls, n);
...
},
}
}
Now the cool part: The check_struct code already ensured that the syscall implementations all use the C calling convention. This means we can call them from assembly as long as the stack is aligned to 16 bytes, and the arguments are in the correct registers (rdi, rsi, rdx, etc.)
So for each syscall that you implement in Zig, the compiler can emit machine code that correctly calls that function!
const fun = syscall_by_number(Syscalls, n);
// Save all the Soil register contents on the stack.
machine_code.emit_push_soil(.a);
machine_code.emit_push_soil(.b);
machine_code.emit_push_soil(.c);
...
// Align the stack to 16 bytes.
...
// Move args into the correct registers for the C ABI.
// Soil C ABI
// Vm (rbx) -> arg 1 (rdi)
// a (r10) -> arg 2 (rsi)
// b (r11) -> arg 3 (rdx)
// c (r12) -> arg 4 (rcx)
// d (r13) -> arg 5 (r8)
// e (r14) -> arg 6 (r9)
const signature = @typeInfo(@TypeOf(fun)).Fn;
const num_args = signature.params.len;
if (num_args >= 1) machine_code.emit_mov_rdi_rbx();
if (num_args >= 2) machine_code.emit_mov_rsi_r10();
if (num_args >= 3) machine_code.emit_mov_rdx_r11();
if (num_args >= 4) machine_code.emit_mov_rcx_r12();
if (num_args >= 5) machine_code.emit_mov_soil_soil(.sp, .d);
if (num_args >= 5) machine_code.emit_mov_soil_soil(.st, .e);
// Call the syscall implementation.
machine_code.emit_call_comptime(@intFromPtr(&fun));
// Unalign the stack.
...
// Restore Soil register contents.
...
machine_code.emit_pop_soil(.c);
machine_code.emit_pop_soil(.b);
machine_code.emit_pop_soil(.a);
// Move the return value into the correct registers.
switch (signature.return_type.?) {
void => {},
i64 => machine_code.emit_mov_soil_rax(.a),
else => unreachable,
}
Conclusion
Zig's comptime is such an interesting language decision. Dynamic ducktyping, reflection, and first-level types at compile time remove the need for macros and generic types. Zig is a language that lets you just "do what you want" at compile time without the language getting in your way. I would have never thought that a systems programming language could feel so dynamic.
This also has drawbacks. If you see a Rust function signature, you immediately know what types are expected. Even for generic functions, traits tell you how the types need to behave. In Zig, a function signature may not immediately tell you all you need. In our case, the run function just expects a Syscalls: type – that doesn't tell you anything! Only when you try to compile the code, you get an error.
Of course, just because you can do arbitrary turing-complete checks at compile time doesn't mean you should. I feel obliged to say that the vast majority of functions in Zig have readable signatures with simple types. This blog post focuses on how Zig differs from other languages, so insane function signatures are disproportionately represented.
I think the comptime feature aligns very well with Zig's general vibe. It allows the language itself to be simple and uniform. If you're open to trying new things, I encourage you to play around with Zig! The combination of high-level dynamic scripting and low-level systems programming makes Zig just feel different from all the mainstream languages. Happy coding!