Components
How the compiler writes the core module of a WASI 0.3 component, and how the componentizer wraps it
The compiler can write a WebAssembly component that targets WASI 0.3, so that a program runs under Wasmtime, or any
runtime that implements WASI 0.3, and not only under nanyx’s own runtime. The design, and the stages still to come,
are on Components in Work in progress. nanyx build writes a component for a program
whose host has a component root, as the console and HTTP hosts do; --core writes a core module, which is all nanyx run,
nanyx test, the debugger and the profiler run.
A program can export functions of its own from its component, marked @component, for components in other languages
to call (A program’s own exports), and import theirs, marked @import
(A program’s own imports).
A component is built in three steps:
- The program is checked under its host’s component root (
Package.asComponent, see Component roots). - The backend writes a core module in wit-component’s conventions (
WasmOptions.Component). - The componentizer wraps it as a component of the root’s world (
Nanyx.Runtime.Componentizer.componentize).
The core module
WasmOptions.Component changes three things about the module the backend writes (ComponentAbi.fs). It holds the
entry the module exports, a ComponentEntry, which the world says (ComponentEntry.ofWorld): HttpHandler for
wasi:http/service, and CliRun for any other.
-
The entry is exported under the world’s name for it, not run when the module is instantiated:
CliRuniswasi:cli/run@0.3.0#run. It calls the start function, if the program has one, and returns 0, which isok.runis anasync funcin WIT, lifted synchronously: the instance blocks in each call it makes, as it does undernanyx run.HttpHandleriswasi:http/handler@0.3.0#handle, lifted async with a callback (The handler).
-
cabi_reallocis exported, for the host to allocate in the module’s memory through. It is the program’s own allocator: a new block of the size asked for, with the old block’s bytes copied in and the old block freed. The allocator’s blocks are only 4-byte aligned, so when the host asks for more (8, for alist<u64>, the most any WIT type needs), it takes the block fromallocAligned8, which the allocator describes. -
The runtime’s own writes and exits go through
wasi:cli, not WASI preview 1. The runtime’s helpers,dbgand a crash write with preview 1’sfd_write, and exit with itsproc_exit. A component’s module defines both functions itself, with the same signatures, so every caller is unchanged:fd_writewrites each buffer towasi:cli/stdout(fd 1) orwasi:cli/stderr(fd 2). Each write makes astream<u8>([stream-new-0]write-via-stream), gives its readable end towrite-via-stream, and writes to its writable end until the host has taken every byte. Then it drops the writable end and waits on the futurewrite-via-streamgave, so that the host has written the bytes before anything else is: stdout and stderr come out in the order they were written.- The write and the future’s read are lowered async (
[async-lower]), and when one returns BLOCKED the module waits for it at once, in a waitable set of its own ([waitable-set-wait]). Wasmtime turns the synchronous forms ofstream.writeandfuture.readoff by default, and the async form followed by a wait runs with its defaults. - The waits write their event at addresses 16 to 31, below the string literals, where nothing else reads or writes.
proc_exitiswasi:cli/exit’sexit-with-code. Awasi:http/servicecomponent can’t exit, so forHttpHandlerit traps, and the host answers the request with an error.
A foreign function of preview 1’s
fd_writeorproc_exitis the same function, as it is in a module the runtime runs (Foreign functions), soprintlnwrites throughwasi:clitoo.
The module’s imports are named as wit-component expects: the interface as the module and the function as the name, and
the canonical built-ins prefixed with what they are, such as [stream-write-0]write-via-stream, whose 0 is the
stream’s place among the types in write-via-stream’s signature. The waitable-set built-ins come from $root.
A component’s module has no probes, no memory checks and no nanyx:debug dbg, whose imports and exports no WASI world
has. A crash stops the program with a trap, as it does under nanyx run, having written why to stderr.
The handler
wasi:http/handler’s handle takes a request and returns result<response, error-code>. The host reads the
response’s body from a stream after handle returns, so a handler lifted synchronously, which returns only when its
core function does, can’t write a body bigger than the stream’s buffer: the host isn’t reading it yet. So handle is
lifted async, with the callback ABI ([async-lift] and [callback][async-lift]), and gives its result with
task.return before it is done:
- It raises backpressure (
[backpressure-inc]), so that the host gives the instance no other request until it is done, and runs the program’s start function, which answers the request through the component root’srun. - The root takes the request’s handle from
argument, and givesresultthe response it made, the writable ends of its body’sstream<u8>and its trailers’ future, and the address and length of the body, whichresultcopies.argumentandresultare foreign functions ofnanyx:component(Hosts.componentModule), which the module defines itself (ComponentAbi.externs) rather than imports; only a host’s package, or a host library it depends on, may declare one (NYX0276). handlereturns the response with[task-return]handle, then writes the body’s copy to its stream, waiting while a write blocks, asfd_writedoes, and drops it; writesok(none)to the trailers’ future and drops it; frees the copy; lowers backpressure; and returns EXIT.
The program runs synchronously, as under CliRun: a call it waits on is waited on where it is made, and the callback
is only called if the host cancels the task, which traps. The start function runs afresh for each request an instance
is given: it sets the program’s cached values up and tears them down again, so nothing the program computes is kept
from one request to the next.
The componentizer
The wrapper around the core module is wit-component’s, the library nearly every language uses for this. Nothing in .NET writes components, so the compiler runs wit-component itself, compiled to WebAssembly:
compiler/Nanyx.Componentizeris a Rust crate of a few lines on wit-component: given a core module, a WIT directory and a world’s name, it embeds the world’s types in the module (embed_component_metadata) and builds the component (ComponentEncoder), validating what it builds. It is a WASI preview 1 command, built forwasm32-wasip1, about 2 MB.- The built
.wasmis committed beside the crate, asnanyx-componentizer.wasm, andNanyx.Compiler.Runtime.fsprojembeds it in the assembly, so building the compiler doesn’t need Rust. After a change to the crate,dotnet build -p:BuildComponentizer=truerebuilds it with Cargo (which needs Rust and thewasm32-wasip1target) and.github/scripts/componentizer-sources.sh --writerecords the hash of the sources it was built from innanyx-componentizer.wasm.sources. The Compiler workflow fails when that hash isn’t the sources’. Componentizer.componentizeruns it on the Wasmtime the runtime already embeds, with WASI preview 1: the core module in a temporary directory it may write to, and the WIT directory read-only. It gives the component, or wit-component’s message saying why it can’t build one, such as an import the world doesn’t have.- The compiled componentizer is cached. Compiling the 2 MB module takes a second or two, so the first run
serializes it into the user’s cache (
NANYX_CACHE, or.nanyx/cachein their home directory), named by a hash of its bytes, and later runs read it from there. A cached file this Wasmtime can’t read is compiled again.
The WIT directory is a WIT package, with the packages it uses under deps/, as wit-component reads it. The tests use
wasi:cli@0.3.0 itself, with the WIT of WASI’s v0.3.0 tag
(compiler/Nanyx.Compiler.Wasm.Tests/Features/Components/wit), and wrap modules as its world command.
Component roots
A host’s manifest names its component root and the world it is written against beside its root
(component = (root = "console.host.wasi", world = "nanyx:console/command")), and keeps the world’s WIT in its wit
directory. HostPackages reads it into Host.Component.
- A component build swaps the root.
nanyx buildtakes the package asPackage.asComponentgives it, whose host’sModuleis the component root (Host.AsComponent). Everything after that is a build as any other: the entry point is passed to that root’srun, so it is checked against the contexts the component root provides, and the backend compiles the program withWasmOptions.Component.Host.IsComponentsays whether a host is one a component build sees. The componentizer is given the host’switdirectory and the world. - It provides what the core root does. When the component root’s own module is checked, in a component build,
Library.checkModuleimports the core root and compares the contexts the tworuns provide; one only one of them provides isNYX0275(Hosts.componentRootErrors). So a program fits the one root when it fits the other, and an entry point that requires a context neither provides is the same error,NYX0237, naming it, in either build. - CI runs them. The Compiler workflow builds each package in
examples/as a component, runs it underwasmtime run, and checks that it prints whatnanyx rundoes. A package under the HTTP host,hello.server, is served bywasmtime serveinstead, and checked with requests in turn, which one instance answers.
The console host’s component root
console.host.wasi provides Console, Env, FileSystem, Http, Clock, Random and Async on WASI 0.3 alone, in
the world nanyx:console/command, which the package’s wit/console.wit declares: wasi:cli/command@0.3.0 with
wasi:http/client@0.3.0. wit-component imports only the interfaces the core module uses, so a program that makes no
requests has no wasi:http import. It is Nanyx, on
@externs named as wit-component expects, and it lowers the canonical ABI by hand with nanyx/host.memory:
- Writes to stdout and stderr go as the runtime’s own do (The core module): a
stream<u8>given to the stream’swrite-via-stream, written until the host has taken every byte, then dropped, and its future waited on. The stdout and stderr imports are a record of functions (Output), so the onewriteToserves both. Imports the runtime’s code also declares, such as[stream-new-0]write-via-stream, are imported twice, which WebAssembly and wit-component allow. - Standard input is one stream, which
wasi:cli/stdin’sread-via-streamopens the first time the program reads: its result is a tuple, written to memory the host is given the address of. Each read fills the buffer in the host’s block from it. Taking lines and the rest of the input out of the buffer isconsole.stdin’s, shared with the core root, which fills the buffer with WASI preview 1’sfd_read. - Arguments and environment variables are
list<string>andlist<tuple<string, string>>, returned in memory: an address and a count, each string an address and a length. The host allocates them withcabi_realloc, which is the program’s allocator, and the root frees each once it has copied it out. - The clocks and random numbers are
wasi:clocks/system-clock’snow, aninstantof seconds and nanoseconds written to memory,monotonic-clock’snow, nanoseconds as au64, andwasi:random’sget-random-u64. - Exiting is
wasi:cli/exit’sexit-with-code, which takes 0 to 255; any other code exits with 1. - Files are
wasi:filesystem’s, inconsole.host.wasi.files.nyx, a@partialfragment of the root. The preopens come fromget-directoriesonce, and are kept in the host’s block; a path is found in one as the core root finds it in WASI preview 1’s (within, inconsole.files, which both roots share, withbytesAt). A file is opened withopen-at, read throughread-via-streamfrom an offset, and written throughwrite-via-streamorappend-via-stream, which areOutputs as stdout is, sowriteTowrites a file too; its future’sresult<_, error-code>says whether the write went through.stat,stat-at,create-directory-at,unlink-file-atandread-directory, astream<directory-entry>read 64 entries at a time, do the rest. Each descriptor is dropped ([resource-drop]descriptor) once the call is done with it.- These are
async funcs in WIT, lowered synchronously, which blocks, and runs with Wasmtime’s default settings. - A result in memory is laid out as the canonical ABI lays it out: a
result<descriptor-stat, error-code>has its case at 0 and its payload at 8, sincedescriptor-statholds 64-bit fields; anerror-codeis its case, thenother’soption<string>at 4, whose string the host allocated and the root frees. - An
error-codebecomes theFsErrorthe core root gives for the same WASI error:no-entryis#notFound, and so on (errorAt).
- These are
- HTTP requests are
wasi:http/client’ssend, in the host librarynanyx.wasi(nanyx.wasi/http), which the HTTP host’s component root shares. ARequestbecomes awasi:http/typesrequest: its headers afields, with aContent-Lengthfor a body that doesn’t name one, its body astream<u8>, its trailers a future writtenok(none), its timeoutrequest-options’ connect and first-byte timeouts, and its URL split into scheme, authority and path with query (urlParts).sendis lowered async ([async-lower]send), so that the root can write the body while the host sends it; the root then writes the trailers and waits on the subtask until it has returned ([subtask-drop]after).- The response’s status and headers (
copy-all) are read, and its body from the streamconsume-bodygives; the futureconsume-bodytakes is writtenokonce the body is read, since a future’s writable end can’t be dropped unwritten. - An
error-codebecomes anHttpError: a timeout#timeout, a DNS or connection failure#connectionFailed, an invalid URI#invalidUrl, and anything else#other.
A wait blocks: each async-lowered call that returns BLOCKED is waited on at once, in a waitable set of its own, as the runtime’s own writes are.
The HTTP host’s component root
http.host.wasi provides Http, Clock, Random and Async, as the console host’s root does, in the world
wasi:http/service@0.3.0. The package’s wit/http.wit is a package of its own with nothing in it, which the
componentizer needs as the root of the directory, and wit/deps holds WASI 0.3’s WIT. Its run answers the request
handle was given (The handler) through nanyx.wasi/http.service:
- The request is read from the
wasi:http/typesrequestthatargumentgives. Its method is amethodvariant, whose case is aMethod; oneRequesthas no case for,CONNECT,TRACEor another, is answered with a 400 without calling the handler. Its URL is put together from its scheme,httpunless it isHTTPS, its authority, and its path with its query. Its headers are copied out of itsfields(headersOf, shared with the client), and its body is read from the streamconsume-bodygives (bodyFrom, shared too), whose future is then writtenok. - The response is a
responsemade withresponse.newfrom afieldswith its headers (fieldsOf), a newstream<u8>and a new future for its trailers, whose readable ends go to the host with it, and its status set withset-status-code. The futureresponse.newgives, which says how sending the response went, is dropped. A response whose headers can’t be sent is a 500 with no headers and no body.
A program’s own exports
A program exports functions of its own from its component, for a component in another language to call, by marking
them @component (#2241). The component’s WIT is generated from
their Nanyx types: Nanyx is the source of truth, and the WIT is written for other languages to generate their bindings
from.
module shapes
@component
export def areaOf: float, float -> float = { width, height -> width * height }
Which defs
@componentcomes before an exported def, with no argument (Attributes.applyAttributes); before a def its module doesn’t export, it isNYX0008. The def is exported in the WIT interface its module names: the name its file declares withmodule, or else its file’s name, ormain.- A program exports the defs of every module it imports, directly or not, and its own. Each module’s checked form
keeps its own and those its imports carry (
TypedModule.ComponentExports), each by the name it is inlined under in the importer (ModuleLinking.componentExportsForImport), and inlining keeps them though the program never calls them (ModuleLinking.reachedItems). A module the program doesn’t import exports nothing. - Its type must cross a component’s boundary (
ComponentInterface.errors, run as each module is checked). A function, a context, a type variable, a record or tag union with no name, which WIT must name, a union of types other than tags, a recursive type, which WIT can’t write, and a def that isn’t a function at all areNYX0277. What the compiler can’t take across yet isNYX0301, naming it: a function that requires contexts.
The WIT
ComponentInterface writes each type as the WIT type it is, and each name in kebab case (areaOf is area-of, the
module geo.shapes the interface geo-shapes), escaping a WIT keyword with %:
| Nanyx | WIT |
|---|---|
bool | bool |
I8, I16, I32, int | s8, s16, s32, s64 |
U8, U16, U32, U64 | u8, u16, u32, u64 |
F32, float | f32, f64 |
string | string |
list(a) | list<a> |
a tuple, (int, string) | tuple<s64, string> |
a named record, type Point = (x: int, y: int) | record point { x: s64, y: s64 } |
a named tag union, type Shape = #circle(float) | #empty | variant shape { circle(f64), empty } |
a function’s () | no parameters, or no result |
- A record or tag union is named by the type that names it, and defined in the interface of each function that
uses it, once, after the types it is made of. A generic one is named with its arguments, so
Pair(int)ispair-s64, andPair(list(Point))ispair-list-point. A named type that stands for anything else, such astype Id = int, is written as what it stands for, and so is a nominal type. - Every tag union is a
variant, with a case for each tag, as written, and a payload of the tag’s type, a tuple for a tag of several.#some(a) | #noneis a variant too, not anoption, and#ok(a) | #error(e)not aresult. - A record’s fields are in the order they are written, and a variant’s cases in the order its tags are.
- A parameter is named as the def’s lambda names it.
The package’s WIT package is named from its manifest (ComponentInterface.packageName): acme.shapes at 0.1.0 is
acme:shapes@0.1.0, the first segment the namespace, and a name of one segment is both, greeter:greeter@0.1.0. For a
program that exports anything, nanyx build writes its WIT to out/wit beside the component
(ComponentInterface.write): component.wit, with an interface for each module and a world, component, that includes
the host’s world and exports them; and under deps, the host’s own package (deps/host) and the packages its world is
made of. The componentizer is given that directory and world, rather than the host’s.
The core module
The backend gets the functions and the names they are exported under (WasmOptions.ComponentExports), each the
interface’s qualified name, # and the function’s, acme:shapes/shapes@0.1.0#area-of. The function of each export’s
def is a root for inlining and for what the module keeps (CoreIrInline.inlineCalls, CoreIrReach.reachedFrom), and
ComponentExports writes a function that the module exports under that name, lifted synchronously:
ComponentValues writes the code that lifts a value of each WIT type into a Nanyx value of the type the backend
lowered its Nanyx type to, and lowers one back, following the canonical ABI’s layout in memory and its flattening to
core values (CanonicalABI.md).
A WIT record’s fields are a Nanyx record’s by name, a tuple’s by position, and a variant’s cases a union’s tags by
name, so the two needn’t be in the same order.
- The arguments come as core values: numbers as they are, since Nanyx holds each in the core type the canonical
ABI passes it as, a sized integer narrower than 32 bits extended from its width; a string or a list as an address
and a length; a record or a tuple as its items’ values in turn; and a variant as its discriminant, then the values
of its payload in places its cases share, each the type that holds what each case puts there, an
f64in ani64read back withf64.reinterpret_i64. Arguments that come to more than 16 core values come in a block the host allocated, laid out as a tuple of them, which is freed once they are lifted. - A string’s bytes and a list’s elements are in blocks the host allocated through
cabi_realloc, which the module then owns. Each is copied into a Nanyx string (stringFromMemory) or list, made from its last element, and the block freed with what its elements point to. - An argument the def’s function doesn’t take over (
CoreIrCounting) is dropped after the call. - A result that comes to at most one core value is returned as it: a number, or a variant whose tags have no
payloads, as its discriminant. Any other is written into a new block, its strings and lists into new blocks of their
own, and the block’s address returned. The Nanyx result is dropped once it is written. The module exports
cabi_post_and the export’s name too, which the host calls once it has copied the result, and which frees the block and every block it points to.
An export runs on whatever the instance has set up: a top-level value is built the first time something reads it,
and the start function tears it down again at the end of run, so an export called before run, after it, or with no
run at all works as one called during it.
A program’s own imports
A program calls a function another component exports by declaring it, marked @import, with its type and no value
(#2241):
@import("acme:geometry/shapes@1.0.0", "area")
def area: Shape -> float
- The attribute names the WIT interface in full, its package, its name and its version, and the function, whose
name is the def’s in kebab case if it is left out (
Attributes.applyAttributes). The def’s value becomes a call of__componentImportwith the two (ComponentImportSyntax), which the type checker takes to be of the def’s written type, as it does a foreign function’s. Its type must cross a component’s boundary, as an export’s does (ComponentInterface.errors). - A program imports what it calls. An import is inlined into the modules that use it as any def is, so
ComponentInterface.importsfinds those the program reaches. The worldnanyx buildwrites imports each interface they name, and the WIT of each interface’s package is written from the program’s declarations of it underout/wit/deps(ComponentInterface.importedPackages): the component model checks types by their structure, by the names of fields and cases, not by the names of types, so no WIT of the other component’s is read. - A call of one is a call of a foreign function (
CallImport), of the interface and the function, whose parameters and result are the core types the Nanyx values are held in (ComponentInterface.externOf). The module defines that function itself (ComponentImports), and imports the interface’s function under another name, with the canonical ABI’s signature: the arguments as their core values, or the address of a block of them past 16, then the address of a block for a result of more than one core value; and a result of one as it. - The function lowers its arguments, which it borrows, as an export lowers its result: into a new block, their
strings and lists into blocks of their own, passed as their core values (
Generator.FlatFromMemory), a variant’s payload widened into the places its cases share, or by the block’s address. It frees all of it after the call. It lifts the result as an export lifts its arguments, from the core value or from the block it allocated for it, whose strings and lists the callee allocated throughcabi_realloc, and frees those blocks. - Outside a component, nothing provides an import. In a core module, as
nanyx runandnanyx testrun andnanyx build --corewrites, a call of one that the program reaches isNYX0278, naming it, at its def.
Names at the boundary
A component’s WIT is what another language’s bindings are generated from, and the component model checks more than a
value’s shape: an interface’s types, a record’s fields, a variant’s cases and a function’s parameters are matched by
name. So the other component’s WIT must be the WIT nanyx build writes, as it is for the Rust components below. An
import’s parameters have no names in its Nanyx type, so they are written p0, p1 and on, and the other component’s
WIT names them so too.
Testing with other languages
compiler/ComponentInterop holds a Nanyx component (shapes) and two Rust components: a command (app) that calls
the Nanyx component’s exports, and a component (units) whose exports the Nanyx component imports. run.sh builds the
three, checks that the Rust components’ WIT is what nanyx build wrote, composes them with wac plug, and runs the
result under wasmtime run, which must print expected.txt. The Compiler workflow runs it on Linux, with Wasmtime,
wac and Rust’s wasm32-wasip2 target installed at pinned versions.