Standard library implementation
How the standard library is written, shipped inside the compiler, and extended
This page describes how the standard library is built today. What it offers is described in the Standard Library section, whose references are generated from the library (see its pages on the docs site). What is still open about it is on Compilation, the workstream it is part of.
An ordinary package
The standard library is an ordinary package named nanyx, written in Nanyx, with one module per file
under stdlib/nanyx/ at the repository root. A
program imports its modules like any other package’s: import nanyx/math, import nanyx/text, or
import nanyx/text as t for qualified access.
It ships inside the compiler. NanyxCompiler.fsproj embeds every stdlib/nanyx/*.nyx file as a
resource named nanyx/<file>.nyx, so the published single-file nanyx binary carries the library
and reads nothing from disk. A module is in the file named for it, nanyx/<module>.nyx, and may have more
files, each starting @partial module <module> (see partial modules): the
library’s modules are the names its files declare (StandardLibrary.moduleNames, read from each file’s first
module line). import nanyx/<module> reads the module’s files (StandardLibrary in
Nanyx.Compiler/Builtins.fs), compiles the module together with the program, and inlines its
definitions into the program the backend sees; importing a module the library does not have is a
compile error that lists the ones it does.
The compiler’s build checks every standard-library module once and writes the result beside the compiler, so a
compile checks none of them (#837). Nanyx.Compiler.CheckedLibrary,
a small program, calls Compiler.writeCheckedLibrary when it is built, which checks each module as an import of it
does and writes the checked modules to checked-library.bin (BinaryData writes the F# data, sharing what the modules
share; CheckedLibrary holds the format). It writes the file after CoreBuild, inside its Build target, not after
it: in a parallel build, a project that asks for Build while the writer is building gets the result as soon as
Build is done, before the targets that run after it, so the file has to be there by then
(#1648). compiler/CheckedLibrary.targets, which the CLI and the
test projects import, copies the file beside them and into the published nanyx. It builds the writer for the machine
that builds, without the program’s runtime or the publish properties that would pick one, and makes it a reference
of the program while restoring, so that building or publishing one project alone restores it
(#1656). Its syntax nodes and symbols have ids from
CheckedLibrary.firstId (2⁵⁶) on, which no process reaches, so they never meet a process’s own. A module read from it
is reported as reused, nanyx/prelude (cached) under --timings. The file is used only where it is what checking
would give: by the build of the compiler that wrote it (CheckedLibrary.build, the id of the compiler’s assembly, which
changes with its code and with the library files it embeds), and for a module only with the source it was checked
from. Anything else, as the editor’s unsaved prelude, is checked from its source.
The file is read on demand (#1852). It holds a list of the modules,
each with the hash of its source and the keys of what it imports, and then each module checked, once, in a part of its
own: BinaryData writes a Lazy as a part that a read skips, and reads it the first time the Lazy is asked for, from
the file’s bytes, which it keeps. A part may refer back to what was written before it, but nothing refers into a part,
so a module that refers to another’s types holds a copy of them, and the file is larger than one written whole. A
process reads the list the first time it imports a library module, and then only the modules it imports. The modules
those import, all the way down, come into the compilation as pending (CheckedModules.AddImported), each read only when
an import of it, or the type of a node inlined from it, first needs it: def x = 1 under the console host reads only
the modules it imports itself. A module’s typed items (TypedModule.Items), about half of what the file holds of it,
are a Lazy too, since only tools ask for them.
The file also holds what every compile needs that would otherwise take parsing many files: the standard-library modules
that export each name (CheckedLibraryFile.Exports), for the hint an unknown name gets and the definitions a library
module attaches to a built-in type (CheckedLibrary.standardLibraryExportPaths), and the module each source file of
the shipped packages declares, with the file’s text (CheckedLibraryFile.Files), which buildHostOf hands to
PackageModules.knowDeclaration before it reads the console host’s modules, so that a file that reads the same is not
parsed for its name. So the first compile of def x = 1 parses no library source and no file of the console host’s
packages.
BinaryData reads each type with a reader of its own, which it emits as IL (a DynamicMethod) the first time a read
meets the type, and keeps for the process: the reader reads each field straight from the file’s bytes, a primitive
without boxing it, and calls the record’s constructor or the union case’s with them, rather than going through
reflection, which would box each primitive and make an array of the fields for each value. A list is built from its
items as they are read. A map or set is written in order, so its tree is built from its items straight away, balanced,
with FSharp.Core’s own (internal) tree types (SortedMaps, SortedSets), rather than by adding the items one by one,
which would make a new path to each; items read out of order, which a key compared differently from when it was
written would give, are added one by one.
Each standard-library module the file does not have is checked once per process, the first time a compile imports
it, and every later compile reuses the result (checkedStandardLibrary in Nanyx.Compiler/Library.fs, keyed by the
module and its source).
The file has the modules of the shipped packages too: the console host’s package, nanyx.console, and the packages it
depends on, nanyx.console.core, nanyx.fs and nanyx.http.core (#1637). The writer checks each as
a loose script’s import of it does and writes what checkedSourceFiles keeps of it: the module, the modules it
imports, and the text of every file its check read. Their files are beside the compiler (ShippedPackages.directory),
so the file names a path there relative to that directory, and a path read back is in the reading compiler’s
(BinaryData.write and read take a function over each string). A compile takes such a module from the file when the
file is in its build’s host package or one that package depends on, so it resolves its imports as the writer’s check
did, and while every file the check read reads the same through the compile’s reader; else it checks it as a user’s
module. Under --timings, nanyx.console/src/console.prelude.nyx (cached). So a one-line nanyx check, even of
dbg("hello"), which imports the host, checks nothing but the file itself.
A user’s own
module that a compile imports is kept too, once it checks without errors (checkedSourceFiles), with the text of
every file its check read: its own, its other fragments’ and those of the modules it imports, all the way down. A
later compile reuses it while each of those files reads the same, through the compile’s own reader (the editor’s
buffer for a file it has open), and while the package’s module table is the same; otherwise it is checked again
(#1270). So typing in a small file that imports a large module
does not check the large one again on every change. A package build also hands each file’s compile the modules the
files before it checked (Package.build).
One module, nanyx/prelude, is never imported by name: the compiler imports it, unqualified, into every
other module, so its definitions are what a file sees with no imports at all. A module’s own definitions
and its explicit imports take precedence over the prelude’s, so a module can define its own proc.
The prelude is split by topic across prelude.nyx and the files beside it that start @partial module prelude:
prelude-contexts.nyx (equality, ordering, hashing, joining and text), prelude-numbers.nyx (int and float,
Add, Sub, Num, powers and bitwise functions on ints), prelude-sized-numbers.nyx, prelude-conversions.nyx, prelude-results.nyx (results,
options and proc), prelude-lists.nyx, prelude-sequences.nyx (lazy sequences and ranges), prelude-maps.nyx
(maps and sets) and prelude-arrays.nyx (reading an array, and what collection literals are built with). prelude.nyx keeps the module’s doc
comment and what fits no topic: crash, ignore, booleans and memory blocks.
What is written
What each module holds is in the Standard Library section, generated from the library’s
doc comments. println, Console and Env aren’t in the standard library: they are in the console host’s interface
package, nanyx.console.core, which the compiler ships beside it, and the console host’s prelude brings them into every
module of a program (see Hosts).
Its pages on the docs site
The library is the source of truth for its pages in the Standard Library section: each
file of the library that exports a name has a page, docs/src/content/stdlib/<file>.md with a - for each . in
the file’s name (host.runtime.nyx is on host-runtime.md), so the prelude has a page for each of its files. A page’s
reference, between the two marker comments on it, is generated; the rest of the page, its introduction and longer
examples, is written by hand.
The reference lists what the file exports, in the order it declares it: each type, context and def, attached ones
included, under a heading of its name, with its signature as hover writes it and its doc comment; a context’s members
with theirs; and, on the page of the file named for the module, the module’s doc comment first. A custom pattern is
headed by its tags, #Matches, and written as declared, pattern string -> string -> #Matches(Match) | _. Names the library
reserves, those that start with __, are left out. It is read from the compile of each module the compiler embeds, as an import of it reads it, so no module is checked
again (Compiler.compileStandardLibraryModuleReporting),
and of each public module of the packages it ships beside it (ShippedPackages), not from a parse of its own (StandardLibraryDocs in Nanyx.Compiler/StandardLibraryDocs.fs), and a link in a doc
comment, [`name`], leads to the name’s entry, on its page or another.
A shipped package’s public module, such as nanyx.console.core’s console and env, or nanyx.fs’s file, has a page in the same section,
named for its file as a library file’s is, and its reference is generated in the same way from the package’s files in
packages/. A host’s modules (nanyx.console’s console.host and console.prelude) have none: a program doesn’t
import them.
A published package’s public module, such as nanyx.json’s json, has a page too, which pins the version its
reference is from in its front matter, package: nanyx.json 0.1.0 (StandardLibraryDocs.pinOf). stdlib-docs reads
that version from the package cache, downloading it from the index first if it isn’t there, as a script’s
::dependency nanyx.json 0.1.0 is, and writes its reference as for a shipped package, after a marker that names the
package and version (beginMarkerOf). A pin that can’t be read, a page whose reference is from another package than
the one it pins, and a pin written wrong are errors. The test reads the version from the tests’ index on disk, so it
needs no network (packagesForTests in the Wasm tests’ TestHelpers.fs, from the .nyxpkg in their Packages
directory). To show a newer version’s reference, change the pin, give the tests’ index that version, and write the
references again.
After changing a doc comment, or what the library exports, write the references again from compiler/:
dotnet run --project Nanyx.Compiler.Cli/NanyxCompiler.Cli.csproj -- stdlib-docs ../docs/src/content/stdlib
stdlib-docs also writes nominal-types.json beside the pages: the names of the protected and private types and
contexts the library and those packages export. The docs site reads it to show those names in italics in its code
blocks, as the editor does (rehypeNominalTypes in docs/site/rehype-plugins.mjs), together with the nominal types a
page’s own code blocks declare. It finds them in the text, as the site’s build has no compiler; once the playground’s
browser compiler is released, the build could ask it which names are nominal instead, which would be exact.
--check writes nothing and lists the pages, and the list, that are out of date. A test in the WebAssembly tests,
StandardLibraryReferenceTests.fs, fails, showing the difference, when a page’s reference, or the list of nominal
types, is not what the library gives, when a file that exports names has no page, and when a page with a reference has no file. The nanyx blocks
on these pages outside the reference are compiled and run, as the tour’s are
(Tour examples).
So explain a function in its doc comment, where hover shows it too, and keep a page’s prose to what is about the module as a whole. The open questions about the generated references are on Compilation.
Intrinsics
The library is written over a handful of compiler intrinsics: operations that cannot be expressed in
Nanyx itself, such as writing to standard output or reading a byte of a string. They are listed with
their types in one place, Intrinsics in Nanyx.Compiler/Builtins.fs, are only in scope while a
standard-library module is being checked (user code cannot call them), and are lowered by the backend
onto helper functions it emits with the module (runtimeHelpers in Nanyx.Compiler.Wasm/WasmRuntime.fs). The few that are one
or two instructions, the wrapping arithmetic and the bitwise operations on ints, are emitted inline instead
(wrappingOperations and bitwiseOperations in Nanyx.Compiler.Wasm/CoreIrEmit.fs).
Their names all start with __nanyx_.
Adding to it
To add a module: create stdlib/nanyx/<name>.nyx declaring module <name>, mark what it offers with
export, and rebuild; the glob picks it up and import nanyx/<name> resolves. To split a module, start
stdlib/nanyx/<name>.nyx and each other file of it (stdlib/nanyx/<name>.<part>.nyx, say) with
@partial module <name>. Add an intrinsic
only when a function genuinely cannot be written in Nanyx, or not without copying far more than it needs
to: __nanyx_string_join joins a list of strings with one allocation (its body is in
Nanyx.Compiler.Wasm/StringJoin.fs), where ++ would copy the result so far for every piece, so a
string built from many pieces is built as a list of them and joined once. Give it a name and type in Intrinsics, and
a WebAssembly body of the same name in runtimeHelpers, and list it, with the helpers it calls, in
helperDependencies (the order of that list is the order helpers are emitted in, so add new ones at the end).
Format what you write before committing it: CI fails if any file under stdlib/ is not formatted (see
AGENTS.md).