sysl.text

What a program says when it means text as bytes: the validator, the two conversions either side of C, the character cursor, and the builder in the file beside this one.

A module of its own for the reason a submodule exists at all. None of this is what the language desugars onto – a string literal, +, and an interpolation are all the compiler’s own doing and reach none of these names – so a program that converts says so with an import, and one that never leaves string never sees them. The one exception is reached without being named: s.chars is a member the compiler provides, and it calls chars_of by key rather than by resolving the word, so the cursor moving costs a for over characters nothing at all.

Everything here reaches the standard module’s names freely – Option, Result, Iterate – which reference/modules.md § The module graph is acyclic allows so long as sysl does not name back. It could not have moved while the reading surface was still in sysl, since line_text validates.

Index

char_columns char_from_u32 char_indices chars_of columns cstring fields from_cstring from_utf8 from_utf8_lossy is_char_boundary join parse_bool parse_bool parse_int parse_int parse_int_base parse_int_base parse_long parse_long parse_long_base parse_long_base parse_real parse_real parse_uint parse_uint parse_ulong parse_ulong parse_ulong_base parse_ulong_base repeat replace_all split str_builder str_builder_with_capacity to_lower to_upper CharIndices Chars CString ParseError StrBuilder Utf8Error Ascii Search Ascii for char Ascii for u8 Display for ParseError Iterate for CharIndices Iterate for Chars Search for []const u8 Search for string

Functions

char_columns

char_columns(c: char) -> usize

How many terminal columns one character occupies – two for the East Asian wide and fullwidth forms, none for a combining mark or a format character, and one for everything else.

A control character answers zero, which is the honest answer to a question it does not really have: a terminal does not draw \n in a column, it acts on it. A caller laying out a field is therefore measuring text it has already decided is printable, and one that has not is asking about the wrong string rather than getting a wrong number.

char_from_u32

char_from_u32(u: u32) -> Option[char]

The fallible half of u32 -> char (reference/types.md § char). It is a free function because a scalar has no member namespace to hang a char.try on, and it needs no unchecked primitive underneath it: the guard it writes and the check char(u) already makes are the same two comparisons on the same value, so the cast on the far side of the guard can never trip and the optimizer folds it. Trading a primitive for a redundant compare in a cold branch is the better side of that bargain.

char_indices

char_indices(b: []const u8) -> CharIndices

chars_of

chars_of(b: []const u8) -> Chars

columns

columns(text: []const u8) -> usize

How many terminal columns a run of UTF-8 occupies, which is the sum of what its characters do.

It takes bytes rather than a string so that text being assembled – what a sink has gathered, what a slice of a larger buffer holds – can be measured without being copied into one first. s.bytes is the view a caller holding a string passes.

cstring

cstring(s: string) -> CString

fields

fields(s: string) -> []string

The runs of non-whitespace, with the whitespace between them discarded.

This is not split on a space: a run of several spaces separates two fields rather than producing empty ones between them, and leading or trailing whitespace produces nothing at all. It is what reading a line of columns wants, and writing it as a split and a filter would be slower and no clearer.

from_cstring

from_cstring(p: *u8) -> Result[string, Utf8Error]

A string copied out of the NUL-terminated bytes a C function handed back – the other direction of cstring, and the one every binding needs the moment a C library reports anything in words.

The copy is not optional. The bytes belong to C: a static buffer it reuses on the next call, or storage the caller is about to free. A string outlives the call that produced it, so it cannot be a view onto either.

But it is one copy, and it is from_utf8‘s. Slicing a *u8 names the bytes where C left them without taking any hold on them – a *T promises nothing about how long they stay – and validation is what turns that borrowed run into a string owning its own. Gathering them into a []u8 first, as this once did, is a second buffer for the same bytes that is freed on the next line.

Fallible for the same reason from_utf8 is, and it goes through it rather than around it: nothing about a char * promises well-formed UTF-8, and a C library reporting in a non-UTF-8 locale is the ordinary case rather than a corrupt one.

from_utf8

from_utf8(b: []const u8) -> Result[string, Utf8Error]

The one direction a string could not otherwise be reached from, and it is written here rather than in the compiler because only its last line needs to be: validation is ordinary sysl over a []u8, and the compiler supplies just from_utf8_unchecked, the primitive that says “these bytes are a string now”.

The validation is scan_at below, which is also what the lossy form walks – one table, so that the two can never come to disagree about which bytes are text.

from_utf8_lossy

from_utf8_lossy(b: []const u8) -> string

What a decoder is expected to do with bytes that are not text: keep what is well-formed and put U+FFFD where the rest was, rather than refusing the whole input.

It is the right answer for untrusted bytes and the wrong one for bytes that ought to be valid, which is why it sits beside from_utf8 rather than replacing it. Text coming off a wire, out of a file somebody else wrote, or from a serial port carries whatever was sent; a caller reading that wants a string it can show a person, and a Result there only moves the substitution one line up. A caller decoding its own output wants to be told, and from_utf8 tells it.

One replacement character per maximal ill-formed subsequence, which is what Unicode recommends and what scan_at reports: a truncated three-byte sequence at the end of the input is one U+FFFD and not two.

Valid input costs one walk and no allocation at all – the bytes are handed over as they are, which is the case a caller who cannot know has most of the time.

is_char_boundary

is_char_boundary(b: u8) -> bool

Whether a byte begins a character rather than continuing one. A continuation byte is the only one matching 10xxxxxx, so this is one mask and one comparison – and it is what a program walking backwards through text, or snapping an arbitrary offset onto a boundary, needs and would otherwise write inline. reference/strings.md § Granularity calls the same property “boundary-checked” where the compiler enforces it on s[a..b]; this is how a program asks the question for itself.

join

join(parts: []const string, sep: string) -> string

Every piece laid end to end with a separator between, which is split‘s inverse whenever the separator is one split would have found.

It sizes the buffer before it starts, since it can: the answer’s length is the sum of the pieces plus the separators, and knowing it turns a series of doublings into one allocation.

parse_bool

parse_bool(b: []const u8) -> Result[bool, ParseError]

"true" or "false", and nothing else.

Exactly the two spellings str(b) produces, so the pair round-trips and neither direction has an opinion the other lacks. "True", "yes" and "1" are refused deliberately: each is somebody’s convention, none is this library’s, and a program wanting one can say so in three lines that read as the policy they are.

parse_bool

parse_bool(s: string) -> Result[bool, ParseError]

parse_int

parse_int(b: []const u8) -> Result[int, ParseError]

parse_int

parse_int(s: string) -> Result[int, ParseError]

parse_int_base

parse_int_base(b: []const u8, base: int) -> Result[int, ParseError]

The same, narrowed to int. A value that is a perfectly good long and does not fit here is an Overflow, since what the caller asked for is an int and there is no honest int to hand back.

parse_int_base

parse_int_base(s: string, base: int) -> Result[int, ParseError]

parse_long

parse_long(b: []const u8) -> Result[long, ParseError]

parse_long

parse_long(s: string) -> Result[long, ParseError]

parse_long_base

parse_long_base(b: []const u8, base: int) -> Result[long, ParseError]

The whole signed range in the given base, MIN included.

The accumulation runs negative, and that is the whole trick. The signed range is asymmetric – the magnitude of the most negative value is one larger than the largest positive one – so a parser building the magnitude and negating at the end cannot represent MIN at any point and has to special-case it. Building on the negative side covers the entire range with one path, and the only value needing a word of its own is a positive result of MIN‘s magnitude, refused just before the final negation.

The overflow test happens before the arithmetic that would overflow, because integer arithmetic wraps rather than trapping (01): a product that has already wrapped is not a number a later comparison can learn anything from. limit is the largest magnitude surviving a multiply by the base, and cutoff is the largest digit that may still be subtracted once the accumulator is exactly there.

parse_long_base

parse_long_base(s: string, base: int) -> Result[long, ParseError]

parse_real

parse_real(b: []const u8) -> Result[real, ParseError]

A floating-point value, through the C library.

It goes to strtod for the reason the float half of str goes to snprintf: correctly rounded decimal-to-binary conversion is a hard algorithm to get right and an easy one to get subtly wrong, and the two directions must agree or a value will not survive being written and read back. Writing it in sysl waits for a target with no C library, which is the same condition reference/strings.md § Rendering a value puts on the rendering half.

It costs a copy and usually not an allocation, because strtod reads a NUL-terminated pointer and neither a slice nor a string carries a terminator. The copy goes into a buffer on the stack for any text short enough to fit one, which is every float anybody writes and every float str produces; longer input falls back to the heap rather than being refused, since a decimal expansion may legitimately run to hundreds of digits and every one of them can move the last bit of the result.

The end pointer is what turns C’s lenient parse into this library’s strict one: strtod stops at the first byte it cannot use and reports where, so anything left over is trailing garbage and "1.5x" is refused rather than read as 1.5.

parse_real

parse_real(s: string) -> Result[real, ParseError]

parse_uint

parse_uint(b: []const u8) -> Result[uint, ParseError]

parse_uint

parse_uint(s: string) -> Result[uint, ParseError]

parse_ulong

parse_ulong(b: []const u8) -> Result[ulong, ParseError]

parse_ulong

parse_ulong(s: string) -> Result[ulong, ParseError]

parse_ulong_base

parse_ulong_base(b: []const u8, base: int) -> Result[ulong, ParseError]

The unsigned range, which is not the signed one with the sign removed: "ffffffffffffffff" is a perfectly ordinary 64-bit mask and overflows every signed parse there is. A systems language that could not read one back would be missing the case its own literals are written in.

No sign is accepted, not even +. A leading - on an unsigned value is a question with no good answer – wrap, clamp or refuse – and refusing at the first byte is the one that cannot surprise.

parse_ulong_base

parse_ulong_base(s: string, base: int) -> Result[ulong, ParseError]

repeat

repeat(s: string, n: usize) -> string

The text repeated n times, and nothing when n is zero.

replace_all

replace_all(s: string, old: string, new: string) -> string

Every non-overlapping occurrence of old replaced by new, left to right.

Non-overlapping and left-to-right is the same rule count_of counts by, so the number of replacements made is exactly the number it reports – and the scan continues after what was substituted, so a replacement containing the pattern does not feed on itself.

An empty old matches nowhere rather than everywhere: the alternative inserts a copy of new between every pair of bytes, which for a string would cut multi-byte characters apart.

split

split(s: string, sep: string) -> []string

The pieces between each occurrence of a separator.

Adjacent separators yield an empty piece between them, and a separator at either end yields an empty piece outside it, so the count is always one more than the number of separators found and nothing about the input is silently dropped. A caller wanting the other behaviour wants fields.

An empty separator yields the whole string as one piece. Splitting on nothing has no meaningful answer here: the byte-level reading would cut multi-byte characters in half and hand back pieces that are not text, and the character-level reading is what s.chars already is.

str_builder

str_builder() -> StrBuilder

str_builder_with_capacity

str_builder_with_capacity(n: usize) -> StrBuilder

A builder that already has room for n bytes, for a caller who knows roughly how long the text will be – a line being assembled from fields, a buffer sized from what it is copying. What it saves is the reallocate-and-copy at each doubling on the way up to n.

It is a guess and nothing depends on it: too small and the buffer grows the way it always does, too large and the slack is freed with the rest when the builder goes.

to_lower

to_lower(s: string) -> string

to_upper

to_upper(s: string) -> string

ASCII case conversion.

It walks characters rather than bytes, and that is what keeps it safe. A byte map would be the faster loop and would need a way to put a raw byte into a builder – which is the one thing the builder deliberately does not offer, since a public write taking bytes is from_utf8_unchecked with a longer name (reference/strings.md § Making new bytes). Going through push_char instead means every way in still carries the UTF-8 guarantee and no unchecked primitive is named here at all. The decode and re-encode that costs is the right price for that.

It works because Ascii for char is total: a character outside the range comes back as itself, so a multi-byte character is re-encoded to exactly the bytes it arrived as. The buffer is sized to the input for the same reason – no ASCII case change alters a character’s width, and nothing else is changed at all.

It is ASCII and says so, like the trait it is written on. É is left as it is, and a program wanting otherwise wants a Unicode case table, which reference/strings.md § Granularity puts above this layer.

Types

CharIndices

struct CharIndices
    inner: Chars

The same walk, reporting where each character started as well as what it was.

This is the cursor a lexer actually wants, and its absence is why the JSON reader that used to sit in guide/ and the bytecode lexer beside it both index bytes by hand: a for walks a copy of a cursor, so for c in s.chars cannot be asked afterwards where it got to, and a program needing both the character and its offset had no form that gave it both. Rust calls this char_indices and it is here for the same reason.

The offset is the character’s first byte, so it is directly what s[a..b] takes – a slice from a reported offset lands on a boundary by construction.

It wraps a Chars rather than decoding for itself, so there is one decoder and the two cursors cannot come to disagree about a width.

Chars

struct Chars
    rest: []const u8
    at: usize

A cursor over the characters of a run of bytes, which is what s.chars reaches – the compiler names chars_of for itself, so this is a declaration the language depends on by name rather than one a program has to find.

Member	Signature	Description
offset	offset -> usize	Where the cursor is, in bytes from the start of the run it was made over.
peek	peek(self) -> Option[char]	The next character without moving to it.
count	count(self) -> usize	How many characters are left, which costs a walk of the remaining bytes – there is nothing in the representation that knows it, since the whole point of UTF-8 is that the count and the byte length are different numbers.

CString

struct CString
    bytes: []u8

A string copied into the NUL-terminated shape C reads, and the owner of that copy – a language with no manual free has to say who frees it, so the bytes belong to a value with a lifetime rather than to whoever remembers.

len is one less than the storage because the terminator is not part of the text, and ptr is what an extern taking a char * is given.

Member	Signature	Description
ptr	ptr -> *u8
len	len -> usize

ParseError

enum ParseError
    Empty
    BadDigit(at: usize)
    Overflow
    BadBase(base: int)

Why a parse refused, at the granularity a caller can act on.

BadDigit carries the byte offset for the same reason Utf8Error does: a message naming where is worth writing and cannot be reconstructed afterwards. The four are separated by what a caller would do about them and not by taxonomy – an empty field is often a default, a bad digit is a message to a user, an overflow is a wider type or a refusal, and a bad base is the program’s own mistake rather than the input’s.

StrBuilder

struct StrBuilder
    bytes: &Buf[u8]

Text gathered a piece at a time, handed back as one string.

It is here rather than in the compiler because only its last line needs to be underneath the language: everything above finish is ordinary sysl over the same Buf[u8] a ByteSink uses, and what no sysl body can write is turning owned bytes into a string, which is from_utf8_unchecked of bytes this builder has held all along.

It exists so that joining does not rebuild what it already has. a + b allocates a fresh buffer every time, so gathering n pieces that way costs the sum of every prefix; a builder pays for each piece once and grows geometrically.

Member	Signature	Description
len	len -> usize
is_empty	is_empty -> bool
push	push(*self, s: string)
push_char	push_char(*self, c: char)	A character’s bytes, through the library’s one encoder rather than string(c) – which would allocate a whole string per character only to copy its bytes back out.
push_int	push_int(*self, n: long)	The numbers, rendered straight into the buffer.
push_uint	push_uint(*self, n: ulong)
push_real	push_real(*self, x: real)	%g, so that a float gathered into a builder reads the same as the one str and print would have produced.
push_bool	push_bool(*self, b: bool)	Two immortal literals, so this allocates nothing either – and it is here rather than left to the caller so that the set of things a builder takes without a str is the set str itself renders directly.
clear	clear(*self)
finish	finish(self) -> string	Every byte that went in came from a string or from push_char, so what is held is well-formed by construction and there is nothing left to validate.

Utf8Error

struct Utf8Error
    offset: usize
    truncated: bool

What a refused decode reports: the offset 04 asks for, plus the one distinction a caller can act on – whether the input merely ended in the middle of a sequence, which more bytes would fix, or holds one that no continuation could rescue.

Traits

Ascii

trait Ascii
    code(self) -> u32
    to_upper(self) -> Self
    to_lower(self) -> Self
    is_ascii(self) -> bool
    is_digit(self) -> bool
    is_upper(self) -> bool
    is_lower(self) -> bool
    is_alpha(self) -> bool
    is_alnum(self) -> bool
    is_space(self) -> bool
    is_blank(self) -> bool
    is_hex_digit(self) -> bool
    is_punct(self) -> bool
    is_print(self) -> bool
    is_graph(self) -> bool
    is_control(self) -> bool
    digit_value(self, base: int) -> Option[int]

Asking a byte or a character what kind of thing it is, in the ASCII range.

It is named for the range it answers over, which the module this replaces in the older sysl was not: that one was called unicode and classified nothing above 127, so every caller read a promise the code did not keep. Everything here is a comparison against the ASCII table, and a value outside it answers false to every question rather than being guessed at. Real Unicode classification needs property tables, and reference/strings.md § Granularity puts those in a library above this one rather than underneath it.

A trait rather than two sets of functions, for the reason Float is one (sysl.math). Overloading would answer the naming half of it now (reference/declarations.md § Overloading) – it did not exist when this was written, so is_digit and is_digit_char were the alternative – but dispatch on the receiver is what makes b.is_digit() the same three words whichever of the two it is, and what lets a member written over the others be written once.

The pair is closed, which is what makes an impl the right mechanism here and the wrong one for the integers. sysl.math.Signed cannot be written this way because iN is an open family and five blocks would leave i128 out; a byte and a character are the only two types this question is ever asked of, so two blocks are the whole of it.

What each implementation supplies is three things, and the rest is written once. A code to compare against, and the two case conversions – which stay per type only because each has to answer in its own type, not because the arithmetic differs. Every classifier below is a default over code(), so adding one is one line here and none in either block.

Member	Signature	Description
code	code(self) -> u32	The receiver’s value as a code point, which is what every question below is asked of.
to_upper	to_upper(self) -> Self	The case conversions, which answer in the receiver’s own type and so cannot be shared.
to_lower	to_lower(self) -> Self
is_ascii	is_ascii(self) -> bool	Whether the receiver is in the range every other member here speaks for.
is_digit	is_digit(self) -> bool
is_upper	is_upper(self) -> bool
is_lower	is_lower(self) -> bool
is_alpha	is_alpha(self) -> bool
is_alnum	is_alnum(self) -> bool
is_space	is_space(self) -> bool	Space, tab, newline, vertical tab, form feed, carriage return – C’s isspace set, and the one a program hand-rolling the test tends to get wrong by stopping after the first two.
is_blank	is_blank(self) -> bool	The two characters that separate words on a line without ending it.
is_hex_digit	is_hex_digit(self) -> bool
is_punct	is_punct(self) -> bool	Printable and not a space or alphanumeric: the four runs the ASCII table leaves between its named groups.
is_print	is_print(self) -> bool	Something that occupies a column when written, the space included.
is_graph	is_graph(self) -> bool	Something that leaves a mark, which is is_print minus the space – the distinction C draws between isprint and isgraph, and the one a program measuring visible width needs.
is_control	is_control(self) -> bool	The C0 range and the delete, which is the one control character that is not below the space.
digit_value	digit_value(self, base: int) -> Option[int]	The receiver read as a digit in the given base, or None when it is not one.

Search

trait Search
    view(self) -> []const u8
    slice(self, lo: usize, hi: usize) -> Self
    is_empty(self) -> bool
    starts_with(self, prefix: Self) -> bool
    ends_with(self, suffix: Self) -> bool
    index_of(self, needle: Self) -> Option[usize]
    last_index_of(self, needle: Self) -> Option[usize]
    index_of_byte(self, b: u8) -> Option[usize]
    last_index_of_byte(self, b: u8) -> Option[usize]
    contains(self, needle: Self) -> bool
    has_byte(self, b: u8) -> bool
    count_of(self, needle: Self) -> usize
    trim_start(self) -> Self
    trim_end(self) -> Self
    trim(self) -> Self
    trim_start_matches(self, cutset: Self) -> Self
    trim_end_matches(self, cutset: Self) -> Self
    trim_matches(self, cutset: Self) -> Self

Searching text and trimming it – the half of the operations that makes no new bytes.

Everything here is allocation-free, and that is what the file boundary is drawn on. A search answers with an offset or a bool, and a trim answers with a view of what it was given, so nothing in this file needs an allocator and all of it is reachable under no alloc. What makes new text – joining, replacing, changing case – is in edit.sysl beside it, and needs one.

It is written once, over []const u8, and that is the whole design. The older sysl wrote this surface twice: std.strings over string and std.bytes over []byte, 1,630 lines of near-identical code, because it had no way to say “either of these”. A trait says it: each implementation supplies two members – the bytes to look at, and how to cut a piece of itself out – and every operation below is a default written against those two. So s.starts_with("//") and b.starts_with(prefix) are the same code, and adding an operation adds it to both.

A byte-level search over UTF-8 is not a shortcut, it is correct. UTF-8 is self-synchronizing: a continuation byte is distinguishable from a lead byte, so a well-formed needle cannot match starting anywhere but at a character boundary. That is why these can ignore encoding entirely, and why an offset one of them returns is always safe to slice at – which matters, since s[a..b] traps on a mid-codepoint bound.

The searches are the naive O(n*m) scan. A sub-linear one may replace them without anything observable changing, which is the reason to say so here rather than in each body.

Member	Signature	Description
view	view(self) -> []const u8	The bytes to look at.
slice	slice(self, lo: usize, hi: usize) -> Self	A piece of the receiver, as the receiver’s own type – what lets a trim answer with a view rather than a copy, and what a default cannot do for itself because it does not know how to build a Self.
is_empty	is_empty(self) -> bool	There is deliberately no len here: both implementations already have one the compiler provides, and a trait member of that name would hide it rather than agree with it.
starts_with	starts_with(self, prefix: Self) -> bool	The empty prefix is a prefix of everything, which falls out of the loop rather than being said: there is nothing to disagree about.
ends_with	ends_with(self, suffix: Self) -> bool
index_of	index_of(self, needle: Self) -> Option[usize]	Where the first occurrence starts, or None.
last_index_of	last_index_of(self, needle: Self) -> Option[usize]	The empty needle is found at the end here rather than at 0, which is the same convention read from the other side: the last place it occurs.
index_of_byte	index_of_byte(self, b: u8) -> Option[usize]
last_index_of_byte	last_index_of_byte(self, b: u8) -> Option[usize]
contains	contains(self, needle: Self) -> bool
has_byte	has_byte(self, b: u8) -> bool	Whether one byte occurs anywhere.
count_of	count_of(self, needle: Self) -> usize	How many non-overlapping occurrences there are, counted the way a replace_all would find them – so count_of and the number of replacements replace_all makes always agree.
trim_start	trim_start(self) -> Self	ASCII whitespace off the front, the back, or both.
trim_end	trim_end(self) -> Self
trim	trim(self) -> Self
trim_start_matches	trim_start_matches(self, cutset: Self) -> Self	The same three against a set of bytes the caller names, which is what strips quotes, slashes or padding rather than whitespace.
trim_end_matches	trim_end_matches(self, cutset: Self) -> Self
trim_matches	trim_matches(self, cutset: Self) -> Self

Implementations

Ascii for char

impl Ascii for char

A scalar value, which is what walking s.chars yields. Anything outside ASCII passes through both conversions untouched, which is what makes mapping one over arbitrary text safe rather than merely defined: é is not a letter this trait knows, so it is left alone instead of being shifted into something else.

Ascii for u8

impl Ascii for u8

A byte, which is what a program walking s.bytes or a []const u8 is holding. The case shift is the single bit ASCII sets between the two alphabets, but it is written as the arithmetic rather than as a mask: a mask would also change eleven punctuation characters, and the guard is what makes these total.

Display for ParseError

impl Display for ParseError

The rendering, so that a refusal can be printed without matching on it.

Iterate for CharIndices

impl Iterate for CharIndices

Iterate for Chars

impl Iterate for Chars

Search for []const u8

impl Search for []const u8

Bytes that are not text, or not yet: what a program reads off a socket or a file before it knows whether it is UTF-8. The same operations answer the same way, which is the point of the trait – and here there is no encoding to be careful of, since there is no guarantee to keep.

Search for string

impl Search for string

Text, which is the receiver these are nearly always written on. slice goes through the language’s own boundary-checked subscript, so a piece cut out of a string is a string with the same guarantee – and the bounds every operation above hands it are ones the search or the trim already knows are boundaries.