The nanyx command
How the nanyx CLI is built
Every command and option is described for users on The nanyx command. This page
says how the CLI is built. What its package commands still lack is on Packaging, and
nanyx test is proposed on Testing.
nanyx is a small C# CLI over the compiler pipeline (Nanyx.Compiler.Cli), implemented with Consolo. Its
entrypoint and help text are generated by Consolo from the command methods and their doc comments in Program.cs and
PackageCommands.cs. Each command’s work is in a method that takes a Shell (RunIn, CheckIn, …), the directory
and the streams it runs with, so the Wasm tests run commands in their own process. The exit codes are UsageError in
Program.cs, tested by ExitCodeTests.fs.
run --check-memory sets WasmOptions.CheckMemory (see Memory management), and the Wasm tests
run every program this way, so a leak anywhere fails a test. run --memory does too, and then writes the report of
MemoryReport.cs from the checked allocator’s counters; the two are MemoryCheck.Check and MemoryCheck.Report, which
RunIn and RunScript take. build and wasm write the program as a ProgramFormat, the binary format unless --wat
asks for text; WriteProgram assembles it (see the binary format). run --no-inline and build --no-inline clear
WasmOptions.Inline (see Inlining); the tour examples are run both ways, and must print the same.
Runtime errors
A program that traps stops with Runtime error: … and exit code 1, written by ReportRunError in RunProgram.cs from the
RunError the runner gives. WasmRunner.trapped in Runner.fs reads the trap: an unreachable in the allocator is
out of memory (see Memory management), and running out of stack names the function (see
Tail calls). Arithmetic the program can’t finish is a RunError.ArithmeticError
with an ArithmeticFault, by the trap’s code and the runtime helper it happened in: integer overflow, naming the type
when the helper is one of the integer helpers __nanyx_<type>_<op> (__nanyx_int_add, __nanyx_i8_mul); division
by zero; a float that can’t be an int, from __nanyx_float_to_int (which traps as an overflow on an infinity and as a
bad conversion on NaN); and an int out of a sized type’s range, from __nanyx_<type>_from_int. Each names the innermost
function on the stack that the source names, as running out of stack does, where inlining left one.
WasmRunner.describeFault words the message, for the CLI and the HTTP host. Any other trap is reported as Wasmtime
gives it, with its backtrace.
Profiling
run --profile <file> runs the program with Wasmtime’s guest profiler (GuestProfiler in Profiling.fs), which
wasmtime-dotnet doesn’t wrap: the runtime calls Wasmtime’s C API from the native library wasmtime-dotnet ships
(wasmtime_guestprofiler_new, _sample and _finish, in Wasmtime 48), with the engine, store and module handles
wasmtime-dotnet keeps internal, read by reflection. The engine is made with epoch interruption; a timer moves the epoch
on every 10 ms, and the store’s epoch deadline callback takes a sample each time. Once the program stops, however it
stops, the profile is written. Its frames are named from the module’s name section. The profiler gives each frame the
address of Wasmtime’s compiled code, not an offset in the module, so the table of places (see
Crash stacks) can’t give a frame the line it ran. It gives each function the line it is defined on
instead: the backend starts each function’s body with a definition marker, ;;@def main.nyx:3, from the line its def
starts on (a lambda’s own line for a lambda), and WasmBinary writes each into the table by the function’s name.
ProfilePlaces.withDefinitions (in Profiling.fs) then fills in the fileName and lineNumber of each function of
the profile’s funcTable that the table has a definition for. Functions the compiler makes up, such as runtime
helpers, have none.
The stopwatch
The compiler reports to the CLI through a CompileListener (CompileListener.fs): a package build says which file
it is on, and every compile says when each module’s check starts and ends, by the module’s full path or
nanyx/<module> for the standard library. Package.buildReporting, Script.compileFileReporting and the other
…Reporting functions take a listener; the functions without the suffix tell nobody. A standard-library module is
read from the library the build checked (see the standard library), so a compile
reports it reused; one the build did not check is checked once per process, so only the first compile that needs it
reports its check, and later ones report it reused.
CompileProgress.cs draws the stopwatch, ten times a second, only when stderr is a terminal. Its line is cut to the
terminal’s width, so it never wraps onto a line it can’t rewrite.
Timings
--timings records each module’s check in CompileTimings, and ModuleCheck.tree writes the tree: self is the
check’s total less its imports’ totals. Where a module is needed again, or is the library’s, the compiler tells the
listener ModuleReused, and the line is (cached). Emitting the program to WebAssembly happens per file after
checking, not per module, so it is one line of its own after the tree.
The live view is built by ModuleCheck.live from CompileTimings.SoFar, the checks finished and under way at a
moment, and CompileProgress draws it with ANSI escapes (cursor up, clear line), cutting each line to the terminal’s
width so none wraps. A finished check is one line, without what it needed, so the live view is the chain of checks
under way, each with its finished imports. On Windows it turns on virtual terminal processing for the console first;
where that can’t be done, the stopwatch stays the single line it is without --timings.
The run’s box
RunBox.cs (tested by RunBoxTests.fs) gives the compiler’s lines a │ gutter: the stopwatch, the timings tree
and its live view, diagnostics, Emitted … and Running with …. There is no right border, so a long line or a
resized terminal can’t break the box, and the rules span the terminal’s width (or 80 columns when the console can’t
say). On a terminal in colour, the gutter, the rules and the compiler’s own lines are dimmed; diagnostics keep their
colours, and their underlines stay aligned, since every line moves two columns right. Where the console’s code page
has no rounded corners, as code page 437 on Windows hasn’t, the corners are ┌ and └. When the program doesn’t run,
because of an error that can’t be deferred, the box ends with a plain rule.
The compiler doesn’t know whether it has more to say than Compiled … in until the compile has finished, so the
stopwatch leaves its final line open, without its line ending: if the compiler writes nothing more, the line is ended
before the program runs, and if it writes a line, the top rule is written over the stopwatch’s and the stopwatch’s
line moves into the box. With --timings, the tree always follows, so the top rule is written when the compile starts
and the live view is drawn inside the box.
Packages
nanyx migrate is Migration.fs, nanyx pull is PackageDependencies.pull, nanyx install is
PackageDependencies.install, nanyx pack is PackageApi.pack (PackageArchive.packWith, with the package’s api.json), nanyx bump is PackageApi.bump, and nanyx api-check is PackageApi.checkVersions on the api.json of each
.nyxpkg (PackageApi.ofArchive). nanyx publish and nanyx yank are PublishCommands.cs: the GitHub requests they send are
GitHubApi.fs, through a GitHubClient the Shell’s GitHub gives (a fake one in the Wasm tests, otherwise GitHub
signed in with GITHUB_TOKEN or gh auth token), and the files of the index’s repository they write are
Publishing.fs. publish checks everything it can before it creates the release: that the account can push to the
package’s repository (GitHubApi.canPush) and, for a first version, where it will open the registration from
(GitHubApi.indexRoute: a branch of nanyx-lang/packages if it can push there, otherwise its fork, made and waited for,
since GitHub makes a fork in the background). GitHubApi.createRelease finishes a release a publish left part way, one
with the same asset (by the digest GitHub records) or none yet, so running publish again finishes it. The index
those pull requests change is described on The package index. The cache and the index a command uses are the
Shell’s Packages, PackageStore.user () (NANYX_PACKAGES, NANYX_INDEX); the Wasm tests give their own.
A manifest is read by Manifest.fs as a data-only Nanyx record. It is Nanyx rather than TOML because every package
name has a dot, which a TOML table header must quote, and because settings a package declares can be checked as Nanyx
values. A manifest’s Dependency says where the package is found with a DependencySource: AtPath, Shipped, FromIndex
or OnGitHub. nanyx migrate, nanyx install and nanyx new hold the entries they write as that union, and write
them with Manifest.entryText, which escapes their strings.
nanyx pull brings nanyx.lock up to date and downloads what it locks (PackageDependencies.update,
PackageIndex.download); nanyx build and nanyx run call the same update first (TryUpdateLockfile), and
--locked makes it refuse a lockfile that doesn’t satisfy the manifest (Resolution.unsatisfied). nanyx install
checks the manifest’s new text and resolves it before writing either file, so a conflict leaves the manifest as it was.
A script has no lockfile: each ::dependency on a package from the index pins an exact version. nanyx run on a
script (TryPullScript) and nanyx pull script.nyx download them first (Script.pull, ScriptDependencies.pull):
Resolution.resolvePinned resolves as resolve does, but offers the solver only the pinned version of each pinned
package, which may be yanked, and PackageIndex.download checks each download against the index’s SHA-256. A build
of the script reads the cache alone (ScriptDependencies.cached, the same solve over the versions in the cache), and
the versions it finds are the lockfile Package.scriptModules reads, so a pin that isn’t downloaded is NYX0041. A
package from the index with no version is NYX0009, whose fix writes the newest version the index has
(Resolution.newest, kept for five minutes, since the editor checks a script as it is typed).
A dependency on GitHub (github = in the manifest, github: in a script’s ::dependency) is read and checked, and
resolved through GitHubFetcher (GitHub.fs). A build’s store fetches with GitHub.cached, which reads only the
cache and fails with NotPulled (NYX0041); nanyx pull wraps the shell’s store in PackageStore.downloading,
which fetches with GitHub.downloading. HTTP and unzipping are Downloads.fs, shared with the index. The pin is the
commit, not a checksum of the file, since a commit identifies the repository’s whole state, which a whole-package
reference needs, and a file at a commit can’t change; the idea for single files comes from
Paket. A GitHub dependency never touches the index, so
its ownership rules don’t apply to it. It is for trying a fork or a fix that has not been released, not a second way
to distribute packages, so nanyx publish refuses a package that has one (NYX0039).
Choosing versions is Resolution.resolve. It gathers the dependencies named by
their version of the package and of each package it reaches by path or on GitHub (Resolution.requirements), reads
each package’s versions from the index as it is reached (PackageIndex.versions, with their dependencies and minimum
compiler), and hands them to VersionSolver.solve: PubGrub
as pub builds it, with a term’s versions a finite set of those the index lists. A version that is yanked (unless the
lockfile locks it), missing from GitHub or too new for the compiler is ruled out by an incompatibility of its own, so
that an explanation can name it, and a dependency that no listed version satisfies is one too. The version a lockfile
locks is chosen whenever it is allowed. A conflict is NYX0047, whose message is the solver’s explanation, written as
pub writes it: a sentence for each step, and a number for a step used twice.
A package in the cache, from the index or GitHub, never changes, so a build doesn’t check it again: its checked modules
are kept in a file of their own in the cache, .checked/<name>-<version>-<build>.bin (CheckedPackages,
#2167), in the format of the standard library’s checked library
(Implementing the standard library). Package.fs gives such a package its
CheckedFile. The first time a compilation meets a file of one (Compiler.fromCheckedPackage), it reads the package’s
checked file, or, if there is none it can use, checks every module of the package from source and writes it
(writeCheckedPackage), and then takes the module from the file, with the package’s other modules. A path dependency
never has one. The file is named for the compiler’s build, and is used only while the package’s files hash as they did
when it was written; a file that doesn’t read, or whose bytes aren’t those written, is written again.
Every int64 a checked module holds is the id of a syntax node or a symbol, which a process numbers from 1, so a module
written by one process would share ids with what another checks. The package is checked with both counters moved past
every id given so far (CheckedPackages.startIds), and as the file is written (BinaryData.writeWith) every id given
while it was checked moves into a range of the file’s own, from 2⁶² up, chosen by the file’s path; an id given before,
which a module this process checked itself holds, stops the file being written. A module from the file names the ids of
what it imports as the files it was checked against have them, so the file records, for each module outside the package
that its checks imported, which checked file it came from (CheckedFileOrigins: the checked library, or another
package’s file), and a compilation uses the file only when those modules are here as the same files have them. A
package that depends on another in the cache is checked against the other’s checked file, written first if need be.
Using nanyx.json from the cache, nanyx check of the JSON cookbook’s first example takes about 0.9 s against 1.3 s
(Release): the package’s module is read in about 30 ms rather than checked in 0.86 s.