The io module

Every declaration in sysl.io, with its signature: the generated API page. This page is the argument — what the module is for, and how its pieces fit; that one is the list.

sysl.io is the input half of the byte surface. It is a module rather than part of the core for a reason that reads as an asymmetry and is not: print is a keyword, so what it desugars onto cannot be behind an import a program has not written — while nothing in the language desugars onto reading. A program that never takes input never has these names, and one that does said so.

trait Reader: Fallible
    read(*self, into: []u8) -> []const u8

struct FdReader
    fd: int
    bad: bool

fd_reader(fd: int) -> FdReader
stdin() -> FdReader

find_byte(b: []const u8, c: u8) -> Option[usize]
line_text(b: []const u8) -> Result[string, Utf8Error]

read_all(r: *Reader) -> Buf[u8]
read_all_text(r: *Reader) -> Result[string, Utf8Error]
read_exact(r: *Reader, into: []u8) -> []const u8

enum LineEnding
    Lf
    CrOrLf

struct Lines
    getline(*self) -> Option[string]
    try_getline(*self) -> Option[Result[string, Utf8Error]]
    error(self) -> Option[Utf8Error]

lines(r: *Reader) -> Lines
console_lines(r: *Reader) -> Lines
lines_ending(r: *Reader, ending: LineEnding) -> Lines

That is the whole module. Its shape is Writer turned around, deliberately: one method on bytes, a latch rather than a Result, and *self on both so a source can be stateful and still object-safe for a raw trait object — which is what lets a reader need no allocator.

Reading the lot, and filling a slice

read answers whatever one call could fill, and a short answer is not the end of anything — a pipe with one write in it so far, a socket, and a terminal all answer short routinely. So “read once” and “fill this” are different requests, and three functions are the loops that turn one into the other:

import sysl.io.{bytes_reader_at_most, read_all, read_all_text, read_exact}

main()
    var r = bytes_reader_at_most("hello".bytes, 2)
    var into: [3]u8 = [0; 3]

    print(read_exact(&r, into).len)
    print(read_all(&r).len())

    var t = bytes_reader_at_most("hello".bytes, 1)

    read_all_text(&t) match
        Ok(s) -> print(s)
        Err(e) -> print("not text at byte", e.offset)

3
2
hello

bytes_reader_at_most is what poses the case: it hands back at most that many bytes per call whatever it was offered. Nothing in the block above would notice the difference against a reader that always fills, which is exactly why the loops are worth having written once — the two mistakes in a hand-rolled version are stopping at the first short read, and taking the length from the slice you offered rather than from the one you were handed.

read_all_text answers bytes that are not UTF-8 as an Err whose offset is into the whole stream. Nothing in sysl.io stops the program over its input — a line that is not text is answered too, as below.

read_exact hands back a prefix of into exactly as read does, so a caller compares got.len against what it asked for rather than being given a count it might forget to apply.

The latch is required, not declared

Reader requires Fallible rather than declaring a failed of its own, and so does Writer. That is what lets one type be a reader and a writer at once: a trait’s members become the implementing type’s, so two traits each declaring their own failed could not both be implemented for one file. An open file is both, which is why sysl.fs works at all.

Leaving it out is caught at the impl, with the line to write:

import sysl.io.Reader

struct Memory
    src: []const u8
    at: usize

impl Reader for Memory
    read(*self, into: []u8) -> []const u8 = into[0..<0]

'sysl.io.Reader' requires 'sysl.Fallible', so 'Memory' has to implement that too — write 'impl sysl.Fallible for Memory'

failed defaults to false, so a source that cannot fail — an in-memory buffer, a fixed device — writes impl Fallible for X with no block at all and says nothing about failure.

read answers with the prefix it filled

import sysl.io.Reader

struct Memory
    src: []const u8
    at: usize

impl Fallible for Memory

impl Reader for Memory
    read(*self, into: []u8) -> []const u8
        var n = self.src.len - self.at

        if n > into.len then n = into.len

        for i in 0..<n do into[i] = self.src[self.at + i]

        self.at += n
        into[0..<n]
    end read
end Memory

var m = Memory("alpha\nbeta".bytes, 0)
var r: *Reader = &m
var window: [4]u8

var got = r.read(window)

print(got.len, got[0], got[3])

var more = r.read(window)

print(more.len, more[0])
print(m.failed())

4 97 104
4 97
false

It hands back a slice rather than a count, and that is the one place this surface improves on read(2). A slice already is a length and a pointer: got.len is the count and got is what to look at, so there is no way to be handed the one and forget to apply it to the other — which is the mistake read(2)‘s signature invites every time it is called. It costs nothing, since a view is three words either way.

The caller supplies the storage, and it has to be writable:

import sysl.io.{Reader, stdin}

var r = stdin()
var frozen: []const u8 = "hi".bytes

var bad = r.read(frozen)

a '[]const byte' views elements it may not write, and a '[]byte' is a licence to write them — so the one does not become the other

That is the read-only view rule in the direction it does not travel. A []T goes where a []const T is wanted; the reverse would be a licence nobody granted.

Empty means the end; failed means it ended badly

Reading empty says end of input and says only that. Whether input ended badly is a separate question, and the two are separate because a caller who does not care should not have to ask, and a caller who does should not have to unwrap a Result at every read to find out.

That mapping is read(2)‘s own — zero for the end, -1 for a failure — so nothing is being conflated to make the surface tidy.

import sysl.io.{Reader, lines}

struct Flaky
    left: int
    bad: bool

impl Fallible for Flaky
    override failed(*self) -> bool = self.bad

impl Reader for Flaky
    read(*self, into: []u8) -> []const u8
        if self.left <= 0
            self.bad = true
            return into[0..<0]

        self.left -= 1
        into[0] = u8('x')
        into[1] = 10
        into[0..<2]
    end read
end Flaky

var f = Flaky(2, false)
var n = 0

for line in lines(&f)
    n += 1

print(n, f.failed())

2 true

The loop ended and the question is still answerable. That is the whole reason the next section’s cursor is shaped the way it is.

lines borrows its reader

import sysl.io.{Reader, lines}

struct Memory
    src: []const u8
    at: usize

impl Fallible for Memory

impl Reader for Memory
    read(*self, into: []u8) -> []const u8
        var n = self.src.len - self.at

        if n > into.len then n = into.len
        if n > 6 then n = 6

        for i in 0..<n do into[i] = self.src[self.at + i]

        self.at += n
        into[0..<n]
    end read
end Memory

var text = "alpha\nbeta\r\ngamma"
var m = Memory(text.bytes, 0)

for line in lines(&m)
    prints("[")
    prints(line)
    prints("]")

prints("\n")
print(m.failed(), m.at)

[alpha][beta][gamma]
false 17

This reader hands back at most six bytes at a time, so "beta\r\n" spans two reads and "gamma" arrives with no newline after it at all. Both come out whole, and the \r is gone.

A trailing \r leaves with the \n, so text written on either system reads the same. That is bufio.Scanner‘s choice rather than C getline‘s, and it is the right one for a language whose string is bytes: a program that did not strip it would find line == "beta" false on half the world’s files.

lines takes a *Reader, and the & is not decoration:

import sysl.io.{lines, stdin}

var r = stdin()

for line in lines(r)
    print(line)

a *sysl.io.Reader points at a value, so it needs an address — write '&' in front of the sysl.io.FdReader to take one

A for iterates a copy of its iterator. A Lines that owned its reader would latch a failure onto a copy the caller cannot reach, and failed would be decorative — you could ask it, and the answer would be about a cursor that no longer exists. Borrowing leaves the reader named in the caller’s scope, which is what makes r.failed() answerable after the loop, the only moment the question matters. The price is one line at the call site.

Three more decisions inside the cursor

It scans the slice read returned, not the one it offered. Those are the same memory for a reader that fills what it was given, so the distinction is free — and it is what lets a reader hand back a view of a buffer of its own, never touching the offered one, and still be read correctly. Taking the length from the answer and the bytes from the offer is how the two could have disagreed, and there is nowhere for them to.

A line that fits in one read is never copied. The cursor scans its buffer in place; only a line spanning two reads is gathered, into a Buf[u8] that is then reused. So the cost is proportional to the input rather than to the input times the number of refills.

A line is found with memchr, through find_byte — libc’s scans a word at a time where a sysl loop could not:

import sysl.io.find_byte

print(find_byte("a,b".bytes, u8(',')).unwrap(), find_byte("abc".bytes, u8('z')).is_none())

1 true

That function is what pointer difference is for: memchr answers where with an address, and an index is that address minus the first. It is public because a program scanning for a delimiter of its own wants exactly it. On a freestanding target it does not ask for memchr — not even where the manifest grants a C library (libc), since the choice is made when the library is compiled for the machine — so there it walks the bytes, and the answer is the same.

An empty read ends the cursor for good. A caller reading again past the end gets None every time rather than a second chance, which is right for the sources read(2) serves.

Validation happens where the bytes arrive

getline yields a string, not bytes — so input is checked for well-formed UTF-8 at the boundary it came in at, which is where the language puts that check. A line that is not text ends the cursor rather than the program: getline answers None there and on every call after, and the cursor’s error() says which byte of that line was wrong. A for loop over a cursor therefore stops at the bad line with every line before it handed over, and a program that wants to know why names the cursor and asks once the loop is done:

import sysl.io.{bytes_reader, lines}

val input: []const u8 = [u8('o'), u8('k'), 10, u8('a'), 255, u8('b'), 10, u8('z'), 10]

var r = bytes_reader(input)
var c = lines(&r)

for line in c
    print("[", line, "]")

c.error() match
    Some(e) -> print("not UTF-8 at byte", e.offset, "of the line")
    None -> print("all text")

[ ok ]
not UTF-8 at byte 1 of the line

It does not read past the bad line — z is never handed over — because a loop that skipped one silently would give a program a file with a line missing. Stopping the program was the old answer, and it was the wrong one: on a freestanding target exit is a halt with no supervisor to notice it, so one mistyped byte on a serial port hung the board. error() is None for a cursor that simply reached the end; failed() on the reader stays the question for input that ended badly.

line_text is the conversion on its own, for a program doing its own framing: the \r handling and the validation, answered as a Result.

Reading past a line that is not text

try_getline is the same walk answering each line’s own verdict and carrying on past a bad one. Two answers nest — the Option says whether there was a line at all, the Result says whether it was text, and an input that ends partway through a sequence answers Err with truncated set:

import sysl.io.{lines, stdin}

var r = stdin()
var c = lines(&r)

loop
    c.try_getline() match
        None -> break
        Some(line) ->
            line match
                Ok(s) -> print("[", s, "]")
                Err(e) -> print("not text, at byte", e.offset)

getline is written in terms of it, so there is one place that decides where a line ends and one that decides what to do about bytes that are not text.

A terminal does not end a line the way a file does

lines() splits on \n, which is right for a file and for a pipe. A terminal sends a bare \r when Enter is pressed, so a cursor splitting on LF never sees a line at all: it waits forever, prints nothing, and looks hung, with nothing to grep for, because nothing has gone wrong — the line has not ended.

console_lines is the same cursor taking CR, LF and CRLF alike. Which is right is a property of where the bytes came from, which the library cannot know: a \r in the middle of a pipe’s line is a \r, and a \r from a console is a line. So the caller says, and lines() is unchanged.

import sysl.io.*

struct Typed
    src: []const u8
    at: usize

impl Fallible for Typed

impl Reader for Typed
    read(*self, into: []u8) -> []const u8
        var n: usize = 0

        while n < into.len && self.at < self.src.len
            into[n] = self.src[self.at]
            self.at += 1
            n += 1

        into[0..<n]

var keys = Typed("one\rtwo\r\nthree\n".bytes, 0)

for line in console_lines(&keys)
    print("[", line, "]")

[ one ]
[ two ]
[ three ]

A bare \r ends the line immediately rather than waiting to see whether an \n follows, because on a bare-CR terminal it never does and waiting inside the read would hang until the next keystroke. What that costs is that the \n of a \r\n may arrive in a later read than its \r, so the cursor remembers it is owed one across the refill — which is the case a hand-rolled console reader gets wrong, producing a spurious empty line between every pair.

lines_ending(r, e) is the two written out, for a caller holding the policy as a value.

Echo and editing are not a line cursor’s business — one is about the terminal and the other about the line — and they have to come from somewhere, because a terminal in raw mode shows nothing as it is typed and a mistake cannot be corrected without them.

They come from sysl.term.edit, which is this trait’s other consumer: it edits where nothing else will, and hands back whole lines the same way a Lines does — an Iterate whose Item is string. So the two are interchangeable at a call site, as a *Iterate[string], and which one a program wants is decided by whether anything else is already doing the editing — the kernel, on a cooked terminal, and nothing at all over a serial cable.

Reading from memory, and writing to it — bytes_reader and bytes_writer

Every Reader and every Writer above is a file descriptor, and for a long time that was all there was — which made a whole class of code awkward to reach: anything taking a *Reader could only be exercised by arranging for real input to arrive on a real descriptor.

import sysl.io.{bytes_reader, bytes_writer, lines}
import sysl.text.from_utf8

main()
    var src = bytes_reader("one\ntwo\n".bytes)
    var out = bytes_writer()

    for line in lines(&src)
        out.write(line.bytes)
        out.write("|".bytes)

    print(from_utf8(out.view()).unwrap_or("?"))

one|two|

A bytes_reader borrows what it reads rather than copying it, so one over a large buffer costs three words; what that asks is that the bytes outlive the reader, which is the ordinary rule for a slice. A bytes_writer grows to fit, and view() hands back everything written so far — a caller wanting text calls from_utf8 on that view and gets to decide what an ill-formed sequence means, which is right here, because whatever wrote those bytes is the thing under suspicion.

bytes_reader_at_most(b, n) hands back at most n bytes per read, however much room it was offered, and it earns its place: anything reading a stream has cases that arise only when a unit of input straddles two reads — the two bytes of a \r\n, an escape sequence, the continuation bytes of one character. Those are exactly the cases a hand-rolled reader gets wrong, and a reader that always empties itself in one go can never produce them. It is also what a slow source honestly looks like: a serial port hands over what has arrived, not what was asked for.

FdReader and stdin

import sysl.io.{lines, stdin}

var r = stdin()

for line in lines(&r)
    print(line)

if r.failed() then print("input ended badly")

stdin() names the descriptor a program is started with, so a caller does not have to know it is zero, and fd_reader(fd) takes any other. FdReader is the only reader the library supplies — the mirror of ByteSink being the only writer that keeps what it is given.

That symmetry is what makes the freestanding story short. FdReader reads through one hook, sysl_io_read in sysl.io.sys, and flush pushes standard output out through another, sysl_io_flush; on a hosted target the library answers both over C’s read and fflush. A freestanding target answers them itself — a C library its manifest grants (libc) being no operating system, it changes nothing here — with an @export each, beside the putchar that print writes through — a read system call is usually the whole of the first, and the second answers zero where nothing is held back:

@export("sysl_io_read")
console_read(fd: int, into: *u8, room: usize) -> isize = 0

@export("sysl_io_flush")
console_flush(fd: int) -> int = 0

Each answers what it did — a count of bytes, or zero — or the code of what went wrong, negated, so a kernel call that already answers a negative error number is the hook as it stands; the one status UNSUPPORTED (sysl.sys, -38 on every platform, not the host’s ENOSYS) means the target cannot make the call at all, and -1 is EPERM. That is the contract of every hook module (sysl.fs.sys, sysl.process.sys, sysl.net.sys and sysl.env.sys answer the same way). A hook the program reaches and leaves unanswered is refused when it is compiled, naming every such hook and only those it reaches, rather than surfacing at the link as a symbol nobody wrote. Everything above the hooks is unchanged: every renderer, every Display, the whole of lines, and every impl Reader a program wrote for itself.

Next: sysl.fs — files and paths, and the capability that gates them.