Nanyx
PlaygroundPackages

The nanyx command

How the nanyx CLI is built

Every command and option is described for users on The nanyx command. This page says how the CLI is built. What its package commands still lack is on Packaging, and nanyx test is proposed on Testing.

nanyx is a small C# CLI over the compiler pipeline (Nanyx.Compiler.Cli), implemented with Consolo. Its entrypoint and help text are generated by Consolo from the command methods and their doc comments in Program.cs and PackageCommands.cs. Each command’s work is in a method that takes a Shell (RunIn, CheckIn, …), the directory and the streams it runs with, so the Wasm tests run commands in their own process. The exit codes are UsageError in Program.cs, tested by ExitCodeTests.fs.

run --check-memory sets WasmOptions.CheckMemory (see Memory management), and the Wasm tests run every program this way, so a leak anywhere fails a test. run --memory does too, and then writes the report of MemoryReport.cs from the checked allocator’s counters; the two are MemoryCheck.Check and MemoryCheck.Report, which RunIn and RunScript take. build and wasm write the program as a ProgramFormat, the binary format unless --wat asks for text; WriteProgram assembles it (see the binary format). run --no-inline and build --no-inline clear WasmOptions.Inline (see Inlining); the tour examples are run both ways, and must print the same.

Runtime errors

A program that traps stops with Runtime error: … and exit code 1, written by ReportRunError in RunProgram.cs from the RunError the runner gives. WasmRunner.trapped in Runner.fs reads the trap: an unreachable in the allocator is out of memory (see Memory management), and running out of stack names the function (see Tail calls). Arithmetic the program can’t finish is a RunError.ArithmeticError with an ArithmeticFault, by the trap’s code and the runtime helper it happened in: integer overflow, naming the type when the helper is one of the integer helpers __nanyx_<type>_<op> (__nanyx_int_add, __nanyx_i8_mul); division by zero; a float that can’t be an int, from __nanyx_float_to_int (which traps as an overflow on an infinity and as a bad conversion on NaN); and an int out of a sized type’s range, from __nanyx_<type>_from_int. Each names the innermost function on the stack that the source names, as running out of stack does, where inlining left one. WasmRunner.describeFault words the message, for the CLI and the HTTP host. Any other trap is reported as Wasmtime gives it, with its backtrace.

Profiling

run --profile <file> runs the program with Wasmtime’s guest profiler (GuestProfiler in Profiling.fs), which wasmtime-dotnet doesn’t wrap: the runtime calls Wasmtime’s C API from the native library wasmtime-dotnet ships (wasmtime_guestprofiler_new, _sample and _finish, in Wasmtime 48), with the engine, store and module handles wasmtime-dotnet keeps internal, read by reflection. The engine is made with epoch interruption; a timer moves the epoch on every 10 ms, and the store’s epoch deadline callback takes a sample each time. Once the program stops, however it stops, the profile is written. Its frames are named from the module’s name section. The profiler gives each frame the address of Wasmtime’s compiled code, not an offset in the module, so the table of places (see Crash stacks) can’t give a frame the line it ran. It gives each function the line it is defined on instead: the backend starts each function’s body with a definition marker, ;;@def main.nyx:3, from the line its def starts on (a lambda’s own line for a lambda), and WasmBinary writes each into the table by the function’s name. ProfilePlaces.withDefinitions (in Profiling.fs) then fills in the fileName and lineNumber of each function of the profile’s funcTable that the table has a definition for. Functions the compiler makes up, such as runtime helpers, have none.

The stopwatch

The compiler reports to the CLI through a CompileListener (CompileListener.fs): a package build says which file it is on, and every compile says when each module’s check starts and ends, by the module’s full path or nanyx/<module> for the standard library. Package.buildReporting, Script.compileFileReporting and the other …Reporting functions take a listener; the functions without the suffix tell nobody. A standard-library module is read from the library the build checked (see the standard library), so a compile reports it reused; one the build did not check is checked once per process, so only the first compile that needs it reports its check, and later ones report it reused.

CompileProgress.cs draws the stopwatch, ten times a second, only when stderr is a terminal. Its line is cut to the terminal’s width, so it never wraps onto a line it can’t rewrite.

Timings

--timings records each module’s check in CompileTimings, and ModuleCheck.tree writes the tree: self is the check’s total less its imports’ totals. Where a module is needed again, or is the library’s, the compiler tells the listener ModuleReused, and the line is (cached). Emitting the program to WebAssembly happens per file after checking, not per module, so it is one line of its own after the tree.

The live view is built by ModuleCheck.live from CompileTimings.SoFar, the checks finished and under way at a moment, and CompileProgress draws it with ANSI escapes (cursor up, clear line), cutting each line to the terminal’s width so none wraps. A finished check is one line, without what it needed, so the live view is the chain of checks under way, each with its finished imports. On Windows it turns on virtual terminal processing for the console first; where that can’t be done, the stopwatch stays the single line it is without --timings.

The run’s box

RunBox.cs (tested by RunBoxTests.fs) gives the compiler’s lines a │ gutter: the stopwatch, the timings tree and its live view, diagnostics, Emitted … and Running with …. There is no right border, so a long line or a resized terminal can’t break the box, and the rules span the terminal’s width (or 80 columns when the console can’t say). On a terminal in colour, the gutter, the rules and the compiler’s own lines are dimmed; diagnostics keep their colours, and their underlines stay aligned, since every line moves two columns right. Where the console’s code page has no rounded corners, as code page 437 on Windows hasn’t, the corners are ┌ and └. When the program doesn’t run, because of an error that can’t be deferred, the box ends with a plain rule.

The compiler doesn’t know whether it has more to say than Compiled … in until the compile has finished, so the stopwatch leaves its final line open, without its line ending: if the compiler writes nothing more, the line is ended before the program runs, and if it writes a line, the top rule is written over the stopwatch’s and the stopwatch’s line moves into the box. With --timings, the tree always follows, so the top rule is written when the compile starts and the live view is drawn inside the box.

Packages

nanyx migrate is Migration.fs, nanyx pull is PackageDependencies.pull, nanyx install is PackageDependencies.install, nanyx pack is PackageApi.pack (PackageArchive.packWith, with the package’s api.json), nanyx bump is PackageApi.bump, and nanyx api-check is PackageApi.checkVersions on the api.json of each .nyxpkg (PackageApi.ofArchive). nanyx publish and nanyx yank are PublishCommands.cs: the GitHub requests they send are GitHubApi.fs, through a GitHubClient the Shell’s GitHub gives (a fake one in the Wasm tests, otherwise GitHub signed in with GITHUB_TOKEN or gh auth token), and the files of the index’s repository they write are Publishing.fs. publish checks everything it can before it creates the release: that the account can push to the package’s repository (GitHubApi.canPush) and, for a first version, where it will open the registration from (GitHubApi.indexRoute: a branch of nanyx-lang/packages if it can push there, otherwise its fork, made and waited for, since GitHub makes a fork in the background). GitHubApi.createRelease finishes a release a publish left part way, one with the same asset (by the digest GitHub records) or none yet, so running publish again finishes it. The index those pull requests change is described on The package index. The cache and the index a command uses are the Shell’s Packages, PackageStore.user () (NANYX_PACKAGES, NANYX_INDEX); the Wasm tests give their own. A manifest is read by Manifest.fs as a data-only Nanyx record. It is Nanyx rather than TOML because every package name has a dot, which a TOML table header must quote, and because settings a package declares can be checked as Nanyx values. A manifest’s Dependency says where the package is found with a DependencySource: AtPath, Shipped, FromIndex or OnGitHub. nanyx migrate, nanyx install and nanyx new hold the entries they write as that union, and write them with Manifest.entryText, which escapes their strings. nanyx pull brings nanyx.lock up to date and downloads what it locks (PackageDependencies.update, PackageIndex.download); nanyx build and nanyx run call the same update first (TryUpdateLockfile), and --locked makes it refuse a lockfile that doesn’t satisfy the manifest (Resolution.unsatisfied). nanyx install checks the manifest’s new text and resolves it before writing either file, so a conflict leaves the manifest as it was. A script has no lockfile: each ::dependency on a package from the index pins an exact version. nanyx run on a script (TryPullScript) and nanyx pull script.nyx download them first (Script.pull, ScriptDependencies.pull): Resolution.resolvePinned resolves as resolve does, but offers the solver only the pinned version of each pinned package, which may be yanked, and PackageIndex.download checks each download against the index’s SHA-256. A build of the script reads the cache alone (ScriptDependencies.cached, the same solve over the versions in the cache), and the versions it finds are the lockfile Package.scriptModules reads, so a pin that isn’t downloaded is NYX0041. A package from the index with no version is NYX0009, whose fix writes the newest version the index has (Resolution.newest, kept for five minutes, since the editor checks a script as it is typed).

A dependency on GitHub (github = in the manifest, github: in a script’s ::dependency) is read and checked, and resolved through GitHubFetcher (GitHub.fs). A build’s store fetches with GitHub.cached, which reads only the cache and fails with NotPulled (NYX0041); nanyx pull wraps the shell’s store in PackageStore.downloading, which fetches with GitHub.downloading. HTTP and unzipping are Downloads.fs, shared with the index. The pin is the commit, not a checksum of the file, since a commit identifies the repository’s whole state, which a whole-package reference needs, and a file at a commit can’t change; the idea for single files comes from Paket. A GitHub dependency never touches the index, so its ownership rules don’t apply to it. It is for trying a fork or a fix that has not been released, not a second way to distribute packages, so nanyx publish refuses a package that has one (NYX0039).

Choosing versions is Resolution.resolve. It gathers the dependencies named by their version of the package and of each package it reaches by path or on GitHub (Resolution.requirements), reads each package’s versions from the index as it is reached (PackageIndex.versions, with their dependencies and minimum compiler), and hands them to VersionSolver.solve: PubGrub as pub builds it, with a term’s versions a finite set of those the index lists. A version that is yanked (unless the lockfile locks it), missing from GitHub or too new for the compiler is ruled out by an incompatibility of its own, so that an explanation can name it, and a dependency that no listed version satisfies is one too. The version a lockfile locks is chosen whenever it is allowed. A conflict is NYX0047, whose message is the solver’s explanation, written as pub writes it: a sentence for each step, and a number for a step used twice.

A package in the cache, from the index or GitHub, never changes, so a build doesn’t check it again: its checked modules are kept in a file of their own in the cache, .checked/<name>-<version>-<build>.bin (CheckedPackages, #2167), in the format of the standard library’s checked library (Implementing the standard library). Package.fs gives such a package its CheckedFile. The first time a compilation meets a file of one (Compiler.fromCheckedPackage), it reads the package’s checked file, or, if there is none it can use, checks every module of the package from source and writes it (writeCheckedPackage), and then takes the module from the file, with the package’s other modules. A path dependency never has one. The file is named for the compiler’s build, and is used only while the package’s files hash as they did when it was written; a file that doesn’t read, or whose bytes aren’t those written, is written again.

Every int64 a checked module holds is the id of a syntax node or a symbol, which a process numbers from 1, so a module written by one process would share ids with what another checks. The package is checked with both counters moved past every id given so far (CheckedPackages.startIds), and as the file is written (BinaryData.writeWith) every id given while it was checked moves into a range of the file’s own, from 2⁶² up, chosen by the file’s path; an id given before, which a module this process checked itself holds, stops the file being written. A module from the file names the ids of what it imports as the files it was checked against have them, so the file records, for each module outside the package that its checks imported, which checked file it came from (CheckedFileOrigins: the checked library, or another package’s file), and a compilation uses the file only when those modules are here as the same files have them. A package that depends on another in the cache is checked against the other’s checked file, written first if need be. Using nanyx.json from the cache, nanyx check of the JSON cookbook’s first example takes about 0.9 s against 1.3 s (Release): the package’s module is read in about 30 ms rather than checked in 0.86 s.