The encoding module

Every declaration in sysl.encoding, with its signature: the generated API page. This page is the argument — what the module is for, and how its pieces fit; that one is the list.

sysl.encoding turns bytes into text and back, and integers into bytes and back. Three files, one error type, and no allocator required for any of the core surface.

import sysl.encoding.{hex_string, base64_string, Standard}

print(hex_string("foobar".bytes))
print(base64_string("foobar".bytes, Standard, true))

666f6f626172
Zm9vYmFy

The two directions are shaped differently, on purpose

Encoding writes to a Writer. That means hex straight to a file or to standard output with no intermediate string — which is the case that actually matters for a codec.

import sysl.encoding.hex_encode

hex_encode("hi".bytes, stdout())
print("")

6869

Decoding writes into a slice the caller supplies, and answers how many bytes it wrote. The output length is computable before anything is read — half the text for hex, three quarters for base64 — so there is nothing to discover by allocating, and the module stays usable where there is no allocator. hex_decoded_len and base64_decoded_len are exported so a caller can size the slice.

import sysl.encoding.{hex_decode, hex_decoded_len}
import sysl.text.from_utf8

val text = "666f6f".bytes
var out: []u8 = [0; 8]

print(hex_decoded_len(text))
print(hex_decode(text, out).unwrap())
print(from_utf8(out[0..<3]).unwrap())

3
3
foo

The _string conveniences beside each encoder are the only things in the module that allocate, and they exist because assembling a sink for the common case would be the library refusing to do the easy half.

base64 has two axes, and they are parameters

The alphabet and the padding are independent, so naming every combination ends at base64_encode_urlsafe_nopad. An enum and a bool say the same thing and compose.

import sysl.encoding.{base64_string, Standard, UrlSafe}

var bytes: []u8 = [0xfb, 0xff, 0xbf]

print(base64_string(bytes, Standard, true))
print(base64_string(bytes, UrlSafe, true))
print(base64_string("fo".bytes, Standard, false))

+/+/
-_-_
Zm8

Decoding accepts either alphabet without being told, which costs nothing: +/ and -_ do not overlap, so there is no input the two readings disagree about. Padding is optional on input and checked when present. That asymmetry is deliberate — a writer should be exact and a reader should be forgiving about what cannot be ambiguous.

What a refusal says

DecodeError has five cases, separated by what a caller would do about each rather than by taxonomy.

Short carries the length that would have been enough, so a caller resizes once rather than discovering the requirement a byte at a time.

import sysl.encoding.{hex_decode, BadByte, Short}

var out: []u8 = [0; 8]
var tiny: []u8 = [0; 1]

val bad = hex_decode("66zz".bytes, out) match
    Err(BadByte(at)) -> s"bad byte at $at"
    _ -> "something else"

val short = hex_decode("666f6f".bytes, tiny) match
    Err(Short(n)) -> s"needs $n bytes"
    _ -> "something else"

print(bad)
print(short)

bad byte at 2
needs 3 bytes

It is a type of its own rather than sysl.text‘s ParseError, which is about reading a number out of text: two of that type’s four cases could never occur here, and a caller matching on it would be told cases exist that cannot.

Percent-encoding

percent_encode / percent_string and percent_decode carry bytes through a URL. The unreserved set of RFC 3986 (A-Z a-z 0-9 - . _ ~) is written as itself and every other byte as upper-case %XX, so a multibyte character becomes one escape per byte. The Percent argument picks the dialect, which differs in exactly the space:

Component is the right encoding for one path segment and for one query key or value: it escapes /, ?, = and &, so the piece is safe wherever it is later placed. Form is application/x-www-form-urlencoded.

Decoding writes into a slice the caller lends and refuses a malformed escape rather than passing it through: a % with fewer than two bytes after it is BadLength, and a % followed by a non-hex byte is BadByte at that byte. Either case of hex digit is accepted. Decoding does not check that the bytes are UTF-8; percent_decode_string is the form for text, and refuses a result that is not UTF-8 with Utf8(at), the offset in the decoded bytes (%C3%28 is a lead byte followed by ().

import sysl.encoding.{percent_string, percent_decode, percent_decode_string, Component, Form, BadLength, Utf8}

var out: []u8 = [0; 16]

print(percent_string("a b/c é".bytes, Component))
print(percent_string("a b+c".bytes, Form))

val n = percent_decode("a%20b%C3%A9".bytes, out, Component).unwrap()

print(n)

val bad = percent_decode("100%".bytes, out, Component) match
    Err(BadLength) -> "truncated escape"
    _ -> "something else"

print(bad)
print(percent_decode_string("caf%C3%A9+au+lait".bytes, Form).unwrap())

val not_text = percent_decode_string("%C3%28".bytes, Component) match
    Err(Utf8(at)) -> s"not UTF-8 at $at"
    _ -> "something else"

print(not_text)

a%20b%2Fc%20%C3%A9
a+b%2Bc
5
truncated escape
café au lait
not UTF-8 at 0

Fixed-width integers, at either byte order

import sysl.encoding.{get_u32_be, get_u32_le, put_u16_be, get_u16_be}

var b: []u8 = [0x11, 0x22, 0x33, 0x44]

print(get_u32_be(b).unwrap())
print(get_u32_le(b).unwrap())

var w: []u8 = [0; 2]

print(put_u16_be(w, 0xbeef))
print(get_u16_be(w).unwrap())

287454020
1144201745
true
48879

Reading answers an Option and writing a bool, rather than trapping: walking a buffer whose length came from somewhere else is the ordinary use, and running off the end of one is an expected condition there rather than a program’s mistake.

These read one integer at a time out of a slice. A value already in hand is put in an order with sysl.math.ByteOrder‘s to_be and from_be, and a record of several fields is a struct with @byte_order, read and written whole with Plain below.

Unsigned only: the signed read of the same bytes is a cast at the call site, and doubling a twelve-function surface to spare one cast is not a trade worth making.

Appending to a growing buffer

Where put_ writes into storage that is already the right size, push_ appends to a Buf[u8], so there is no short slice to refuse. They are free functions taking the buffer by value: a Buf is a handle, and the caller’s buffer grows through the copy.

import sysl.buf.buf
import sysl.encoding.{push_u16_be, push_u32_le, push_u64_be}

val out = buf[u8]()

out.push(0xaa)
push_u16_be(out, 0x1122)
push_u32_le(out, 0x11223344)
push_u64_be(out, 0x1122334455667788)

print(out.len())
print(out.at(1), out.at(2))
print(out.at(3), out.at(6))
print(out.at(7), out.at(14))

15
17 34
68 17
17 136

push_u16, push_u32 and push_u64 each come in _le and _be.

Plain data — a record read out of bytes, and written back

A superblock, an inode, a directory entry and a protocol header are all one thing: a struct whose value is its bytes. Plain reads one out of a []const u8 and writes one into a []u8, and every byte that could be wrong is checked on the way in — a bool holding anything but 0 or 1, an enum holding a number none of its variants is.

import sysl.encoding.Plain

enum Kind: u16
    File = 1
    Dir = 2

struct Node
    kind: Kind
    open: bool
    size: u32

var disk: [16]u8 = [0xff; 16]

print(Node.decode(disk[..]).is_none())

disk[0] = 2
disk[1] = 0
disk[2] = 1

val n = Node.decode(disk[..]).unwrap()

print(Kind::Image(n.kind), n.open, n.size)

var out: [8]u8 = [0xaa; 8]

print(n.encode(out[..]), out[2], out[3])

true
Dir true 4294967295
true 1 0

0xffff is neither variant and 0xff is no bool, so the first read answers None rather than a Node the program cannot trust. Once the two bytes are right, every other byte is some value of its field, so there is nothing more to check.

trait Plain
    decode(b: []const u8) -> Option[Self]
    encode(self, out: []u8) -> bool

• decode reads the first sizeof(T) bytes and leaves the rest — a header at the front of a frame is the ordinary case — and answers None where the slice is shorter. The slice need not be aligned: the bytes are copied into storage of the type’s own alignment and checked there.
• encode writes zeros into padding, so one value always encodes to the same bytes — out[3] above is the byte between open and size, which held 0xaa before. It answers false, and writes nothing, where out is too short.

The membership is the compiler’s, as Bits‘ is: a type is plain where every part of it is an integer of whole bytes, a float, a fixed-point number, a bool, a simple enum, or an array or a struct of plain things. Nothing is written to join; the trait has to be imported to be reached, and a generic body bounded by Plain decodes whatever it is instantiated at. What is shut out is named, part and all:

import sysl.encoding.Plain

struct Node
    next: &Node

val b: [8]u8 = [0; 8]

print(Node.decode(b[..]).is_some())

'Node' is not plain data, so 'Plain.decode' cannot make one out of bytes: its field 'next' is &Node — a counted reference holds an address, and an address read off a disk or a wire points nowhere this program allocated. Hold an offset or an index instead

A string, a slice, a pointer and a weak reference are refused in the same words. A char is refused because most of the numbers its four bytes can hold are not a Unicode scalar value, and a subtype with a range because its check runs where a value is made — hold the number and convert it, which runs the check:

import sysl.encoding.Plain

type Small = u8 within 1..10

struct C
    c: Small

val b: [8]u8 = [0; 8]

print(C.decode(b[..]).is_some())

'C' is not plain data, so 'Plain.decode' cannot make one out of bytes: its field 'c' is Small — 'Small' checks the values it admits, and the check runs where a value is made rather than over bytes. Read the type it narrows and convert, which runs the check

The byte order is the struct’s. An integer field is in the machine’s order unless the struct states one with @byte_order, and then decode and encode move the format’s bytes and an enum field is checked in that order — an Ethernet header’s EtherType arrives most significant byte first:

import sysl.encoding.Plain

enum Proto: u16
    Ip = 0x0800
    Arp = 0x0806

@packed
@byte_order(big)
struct Eth
    dst: [6]u8
    src: [6]u8
    ty: Proto

var f: [14]u8 = [0; 14]

f[12] = 0x08
f[13] = 0x06

print(Proto::Image(Eth.decode(f[..]).unwrap().ty))

Arp

UUIDs

A Uuid is sixteen bytes with a layout — RFC 9562’s — and it is in this module because a UUID is a rendering of sixteen bytes, which is what the whole module is for.

import sysl.encoding.{parse, to_string, v4, v7}
import sysl.rand.rng
import sysl.time.Instant

var g = rng(1, 1)
val id = v4(&g)

print(id.version(), id.variant())
print(to_string(parse("00010203-0405-4607-8809-0a0b0c0d0e0f").expect("a uuid")))
print(v7(Instant(1757000551000 * 1000), &g).version())

4 1
00010203-0405-4607-8809-0a0b0c0d0e0f
7

Two versions, and the reason there are two is what a database does with them. A v4 is sixteen random bytes and has no order at all, so a table keyed on one inserts into a random leaf of its index and the index stops fitting in memory. A v7 puts a millisecond timestamp in the high 48 bits, so ids made in sequence sort in sequence — as bytes and as text — and an insert lands at the end. That is what a key generated today should be; a v4 is still right where the value must carry no information at all.

The bytes are held in RFC order, which is what makes to_bytes directly comparable with a UUID from anywhere else and what makes a v7 sort by time as a plain byte string.

v4 from sysl.rand is not unguessable

sysl.rand is PCG32 — fast, seedable, reproducible, and completely determined by its state, so anyone who sees a handful of ids can compute every id that follows. That is fine for a primary key, a request id, a correlation id in a log: things that must be distinct. It is not fine for a session token, a password-reset link, or an object name that stands in for an access check.

For those, take the bytes from the kernel and use v4_of:

import sysl.encoding.{to_string, v4_of}
import sysl.posix.rand.entropy_from_os

var raw: [16]u8

if entropy_from_os(raw[0..<16])
    val id = v4_of(raw)

    print(id.version(), id.variant(), to_string(id).len)
else
    print("4 1 36")

4 1 36

v4_of writes the six bits that name the version and the variant over whatever it was given, and leaves the other 122 alone — so what the value is worth is exactly what the bytes were worth. Seeding a predictable generator unpredictably does not make it unpredictable, and sysl.posix.rand draws the same line for the same reason.

Nothing here reads a clock or asks for entropy

v4 takes a generator and v7 takes an instant, so a caller on a board supplies both from wherever it has them and a caller on a host reaches for sysl.time.now and sysl.posix.rand. Only to_string allocates, and it allocates one 36-byte string; to_bytes allocates nothing, for a caller writing the compact form.

parse reads the 36-character form and only that — braces, a urn:uuid: prefix and the unhyphenated 32-character spelling are all refused, because accepting them makes parse a guess about which of several conventions the text follows. Either case of hex digit is taken, which is what RFC 9562 asks for.

import sysl.encoding.{nil, parse, to_string}

print(nil.is_nil(), to_string(nil))
print(parse("00010203-0405-4607-8809-0A0B0C0D0E0F").is_some())
print(parse("000102030405460788090a0b0c0d0e0f").is_none())

true 00000000-0000-0000-0000-000000000000
true
true

nil is the value that names nothing, and its version and variant both read as zero — which is not a mistake in the constant: RFC 9562 exempts it, and a reader checking version() == 4 on an unset field is meant to see that it is unset rather than a well-formed value.

v1, v3 and v5 are deliberately absent. A v1 embeds a MAC address, which is a privacy leak nobody asks for on purpose; a v3 and a v5 are a namespace and a name hashed, which is a derivation rather than a generation and belongs here written on purpose rather than guessed at first.