Hosts
How the compiler builds hosts, the console host and the HTTP host
Choosing a host, the console, HTTP and playground hosts, writing a host package, foreign functions and host memory are described for users in Hosts. This page says how the compiler builds hosts, and why they have the shape they do. The hosts still to come and the open questions are on Hosts in Work in progress.
How it’s built
- A host declares itself. A host package’s manifest has
host = (root = "...", prelude = "...")in itspackage(Manifest.checkHostDeclarationchecks it,HostPackages.ofManifestreads it intoManifest.Declares).Host(Hosts.fs) is what the compiler knows of one: its package, its root module, its prelude and its directory. The compiler keeps no table of hosts:HostPackages.tryShippedreads the manifest of one it ships, from its directory beside the compiler (ShippedPackages, see Built-ins as packages), andHostPackages.consoleis the console host, which a loose script runs under unless it names another. The host is imported asnanyx.console/console.host(Host.Import), and its prelude asnanyx.console/console.prelude(Host.PreludeImport). - A package chooses its host.
Manifest.checkreads the manifest’shost,(package = "nanyx.console", version = "0.28.0")or(package = "acme.host", path = "../acme.host"), intoManifest.Host(aHostEntry; none without ahostfield: a library, whose@entryAttributes.applyAttributesreports asNYX0031). A host the compiler doesn’t ship, named without a path, isNYX0023, saying how to give its path; a host with both a path and a version is an error. Itssettingsare a record, checked against the settings the host declares, with their defaults, in its own manifest (Host.Settings,HostSettings.errors) once the host is read.Package.ofManifestreads the host’s own manifest (Package.hostIn): its name must match, and it must declare itself a host. The host’s package travels with the package’s module table (PackageModules.HostandPackageModules.HostPackage), so every compile of a package’s file, the language server’s included, passes@entryto that host’srun, and the host’s imports resolve in its own package.--hostonrun,buildandcheckreplaces it with a host the compiler ships or the directory of one (Package.withHost); on a script it is a usage error. - A script chooses its host.
Script.readreads a script’s::hostand::dependencydirectives into the entries a manifest’shostanddependencieswould be (Manifest.hostEntry,Manifest.dependencyEntries), so they are checked as a manifest’s are, with the same errors, at the word each is about.Package.scriptModulesreads the packages they name, the console host when there is no::host, into a module table for the script’s build (PackageModules.Script): it has no files of its own, so every file of the build in no package it depends on is the program’s (PackageModules.isProgramFile), and its imports are found by path, but for an import of a dependency’s public module,acme.util/strings.Script.compileSourceReadingcompiles a script in that build, fornanyx runand the language server alike. A script with neither directive compiles in no build. - Which modules see the host. A module of the program itself runs under the build’s host: it sees the host’s
prelude and may declare the
@entry. A module of a package the program depends on, of the host’s own package, or of the standard library has no host (checkModuleinLibrary.fs). - The host’s prelude. A module with a host imports its prelude after the standard library’s, as an implicit import
that the module’s own imports win over. The contexts the prelude re-exports are those the host provides
(
providesContextsinImportMerging.fs): importing the module that declares one, as the prelude does, binds the names of its members, so they are ambient (see below). An import that selects a context, as the prelude’sexport nanyx.console.core/console (Console, ...)does, selects its members too. @entrycalls the host’srun.Attributes.applyAttributesimports the host under an alias of its own (__host), so that it can’t clash with the program’s names, and appends__host.run(main)to the module as its start. The entry point passed torunis at the attribute’s span, so a mismatch there is the entry not fitting the host:NYX0237, reported at@entry, with the type the host expects and the one the entry has.runignores what the entry returns. Its parameter is[Console] () -> a, not() -> (), so that an entry whose block ends in an expression (dbg(x)) fits, and its value is dropped.HostRuntimeis declared innanyx/host.runtime, a context with no members: the capability that lets code call foreign functions and usenanyx/host.memory. With no members, a function that requires it is passed nothing, and the type checker treats it as available wherever it is required, soHosts.runtimeErrorsreportsNYX0238in any module but one of the build host’s own package or of a host library it depends on (isHostinLibrary.checkModule,PackageModules.HostLibrary),nanyx/host.runtimeandnanyx/host.memory, by what each call calls: at each top-level def that requires it, as its written contexts say or a call in it does, other than a foreign function, which any module may declare, and at each call at the top level that requires it, other than the call ofrunthat@entrymakes. A module that calls a host’srunitself needsHostRuntimefor it, so that is an error too.- Where the entry’s
Consolecomes from. Every entry is passed the members of theConsolethatrunuses, so it reads and writes through the host. An entry that doesn’t write its type, which is most programs, infersConsoleas an ambient context, because the host’s prelude bringsConsole’s names in. A def that infers an ambient context takes its members as a def that writes it does:InferModuleadds the contexts it inferred to the def’s annotation for the backend ([Console]on its own, or after the parameterised contexts it was given), and the backend gives a function named as a value the contexts without parameters it requires (CodeGen.carryPlainContexts), sorun(main)passesmainits members rather than closing it over the module that declaresConsole. Nothing wiresConsoleto the entry by name. - Scripts run under the host. A module without an
@entrywhose top level has statements is a script, and its statements are its entry point:Scripts.asEntrymakes them the body of a lambda, followed by(), and appends__host.run({ ... })to the module, importing the host (a script sees the host’s prelude, as any program’s module does). The lambda is not givenConsole’s members as a lambda passed torunwould be (Contexts.extendEnvForLambdaArgument): its statements are the top level, and see the script’s own names and ambient contexts as top-level code does. A script’s value that needs a context, directly or through the functions and values it names (def line = readLine()), is inferred in a scope of its own (ScriptValueScope) that records what it needs. A lambda written in the value is not run by building it, so what it calls is its own type’s, not the value’s (def handlers = (log = { m -> println(m) })needs nothing). Once the module is inferred, those values (InferState.ScriptEntryValues) move into the entry point as local defs, in the order written, so each is computed once (#1257). A value that needs no context stays at the top level, where a function can name it, and is built on its first read (Memory management). A function that names a value that moved, or a statement that reads one before its def, isNYX0239. The ambient contexts the statements and the moved values need through unannotated functions are only known once the module is inferred, so the lambda’s type is given them then (InferModule.withEntryContexts), and the host passes their members. Once typed, the entry point is a def of its own,def __script: [Console] = { ... }, which__host.run(__script)names (Scripts.asDefinition), so a lambda in a statement captures its members as a lambda in any def does. Statements beside an@entryare a script entry too, run before the@entry’s. A value that needs a context in a module with no statements, which nothing would run, is an error at the value. A script carries only the members of the host’s contexts that it calls, as an entry program does (seeEnvbelow). Consoleonly comes from the host.nanyx.console.core/consoledeclares the context andprint,println,eprintandeprintln, but no instance: the line reading lives in the console host alone. Importing a module whose contexts the host provides (those its prelude re-exports) binds the names of its plain contexts’ members,readLineand the rest, each a symbol of its own (ImportMerging.mergeImport), soConsoleis still an ambient context: a script’s statements and an unannotated def can callreadLine(), and the def infersConsole. A member of a context a def declares, or auseloads, shadows an imported function of the same name, and takes none of its contexts (Contexts.contextRequirementsOf): a def’s ownprintlnmember is not the prelude’sprintln. A function named as a value carries it in its type (#1288), and an unannotated one carries those it was inferred to require, its own and those of the functions it calls (Contexts.inferredAmbientContexts), found once for each function it reaches however many paths reach it (#1458). A def that writes[c.Console]for an aliased import is passed the members as[Console]is: the backend reads the layout of the context’s own name (CodeGen.layoutOf), and a member called under the alias,c.write, is written by its own name for the backend (ModuleLinking.writeMemberAliases).- A context the host doesn’t provide. An imported function may require a context the module can’t name, as a
library’s
[Console]function does under a host whose prelude doesn’t bringConsolein. Where the imported module has it as an ambient context, it is carried in (InferState.CarriedAmbientContexts). Under a host, one that isn’t carried in can only come from the host, which doesn’t provide it, soImportMerging.mergeImportrecords it inInferState.UnprovidedContexts, and a call to a function that requires it isNYX0205, naming the contexts and the host (Contexts.ensureContextAvailable, #1811).HostRuntime, which only a host may require, is left out: it has an error of its own,NYX0238. Envis declared innanyx.console.core/env(args,variable,exit), andrunuses it besideConsole, withFileSystem,Http(see below),RandomandClock, so its parameter is[Async, Clock, Console, Env, FileSystem, Http, Random] () -> a; an entry that requires fewer contexts fits. The host’s prelude re-exportsEnv, so its members are ambient names in every module of the program. Its members are functions of the host’s own over WASI’sargs_sizes_getandargs_get,environ_sizes_getandenviron_get, andproc_exit, declared with@extern, with their buffers allocated withnanyx/host.memory(see Foreign functions).runpasses the members of both contexts to every entry, but the inliner inlinesrunand the entry into the start (#1469), the entry whatever its size, since the module calls it once (#1883), so a member nothing calls is never built, and a program contains only the members it calls and imports only what they call (#1497): hello world importsfd_writealone. A program that readsargs()imports noenviron_*function, and one that reads onlyvariablenoargs_*one (#1766):entriesOf, which reads either list, is passed the list’s two WASI functions, each wrapped in a lambda the inliner removes, rather than a tag it matches on, which built both lists’ code into every caller.nanyx runpasses what follows the file or package as the arguments, after the program’s name, and the process’s environment variables (seenanyx run); anexit(0)part-way is a clean end, which--check-memorydoesn’t count as a leak. A script’s entry point requires only the plain contexts that are ambient in its module (Scripts.withAmbientContextsOnly): a statement that calls a function whose type is an error would otherwise leave it requiring all thatrunprovides,Envtoo in a script that doesn’t import it.FileSystemis declared in the shipped packagenanyx.fs, which the console host depends on; its prelude re-exportsFileSystem, withPathandFsError, the types its members name: a program module seesrun’s parameter type, so it must be able to name the members’ types. A program reaches its functions by importingnanyx.fs’s modules, with::dependency nanyx.fsor a manifest’sdependencies.FileSystem’s members are named for what they do on their own (readFile,writeFile,pathExists,listDirectory, …), since the prelude makes them ambient names.listDirectorygives names in any order, anddir.listsorts and joins them, so another host has less to do.- The file system on WASI. The console host writes
FileSysteminconsole.host.files.nyx, a@partialfragment of its root moduleconsole.host(a host package’s modules are assembled from their fragments as a program’s are,PartialModules.otherFragmentssearching the host’s package too), on WASI’sfd_prestat_getandfd_prestat_dir_name,path_open,fd_read,fd_pread,fd_write,fd_close,fd_filestat_get,fd_readdir,path_filestat_get,path_unlink_fileandpath_create_directory. A path is found in the runtime’s preopened directories: a relative one in the preopen named., an absolute one in the preopen whose name starts it (compared without case when it starts with a drive), and one in none is#permissionDenied, as is a relative path that climbs out of., which WASI refuses (ENOTCAPABLE). WASI’s error numbers map toFsError.WasmRunner(Nanyx.Compiler.Runtime/Runner.fs) takes the directories a program may reach, the current directory first: it preopens that one as.and by its absolute path, and each other by its absolute path, written with/.nanyx runpasses the shell’s directory and the one--allow-dirnames (Program.RunIn).file.linesreads 64 KB at a time through a member of its own,readFileChunk(at, offset, length), which opens the file, reads the chunk withfd_preadand closes it again, so a sequence abandoned before its end holds no file, and pulling a sequence again reads the same lines again. The host’sreadFileChunkgives an error, not an empty chunk, for an offset past the file’s end, so a file cut short between chunks is caught. Asyncmarks a function that can wait. It is declared in the standard library’snanyx/asyncasprotected (), so nouseoutside that module can make one, and it has no members. Both hosts provide it: each prelude re-exports it, and eachrunrequires it in its own type,[HostRuntime, Async], and passes it to the entry, whose parameter type requires it. A context with no members that a host’s prelude exports is ambient by its name, as one whose members the prelude stands in for is by theirs (Contexts.isAmbientMemberless): the program’s top level has it, and supplies it to the start’s call ofrunas the runtime suppliesHostRuntime; an unannotated def that calls something requiring it infers it; and an annotated def that doesn’t declare it isNYX0211. A memberless context has nothing to pass at run time, soAsynccosts nothing in the generated code.nanyx.http.core’ssendRequestandnanyx.fs’s members, and the functions written on them, require it. Waits are blocking host calls, each in its instance’s own thread: the console host has one instance, and the HTTP host runs each request’s instance on a thread of its own (WasmServer.serve), so a handler that waits on an outbound request holds only its own (see Async).Httpis declared in the shipped packagenanyx.http.core, which the console host depends on; its prelude re-exportsHttpandHttpError, withRequest,MethodandResponse. Its one member,sendRequest, is written on the runtime’s own importnanyx:http, not on WASI, which has no networking (see Foreign functions).nanyx.http.core’s modulewirewrites and reads requests and responses as the runtime’s imports pass them, so the console host and the HTTP host share it.Randomis declared in the standard library’snanyx/random, and both hosts’ preludes re-export it. Each host writes its one member,bits, on WASI’srandom_get, which fills 8 bytes thatreadI64then reads: both callsystemBitsin the host librarynanyx.wasi(nanyx.wasi/preview1), which their packages depend on.random.seededis the library’s ownRandom, SplitMix64 over amutcell, so a test host, when there is one, can give its tests a fixed seed by running each under it.Clockis declared in the standard library’snanyx/clock, withnow, Unix milliseconds, andticks, milliseconds from a start the host chooses that never go backwards; both hosts’ preludes re-export it. Each host writes both members on WASI’sclock_time_get,nowon the realtime clock (0) andtickson the monotonic one (1), each giving nanoseconds in 8 bytes of the host’s state thatreadI64reads and the host divides by a million. It has nosleep: waiting belongs withAsync.- The HTTP host.
packages/nanyx.httpdeclares itself a host withhost = (root = "http.host", prelude = "http.prelude", settings = (port = 3000)). Itsruntakes the handler,uses anHttpwritten onnanyx:httpas the console host’s is, takes the request from the runtime, calls the handler, and gives the runtime the response; a request the runtime wrote wrongly is answered with a 400 without calling the handler. The host imports the runtime’s ownnanyx:serve, andnanyx runserves any program whose module imports it (WasmServer.serves), whatever its host: it compiles the module once, listens onhttp://localhost:<port>/with .NET’sHttpListener, and answers each request in a store and an instance of its own, on a thread of its own. An instance that traps or exits without responding is answered with a 500, the reason goes to standard error, and the server carries on.Request.pathtakes the path from the whole URL the runtime passes. Its component root,http.host.wasi, whichnanyx buildbuilds awasi:http/servicecomponent from, is on Components. - The playground host.
packages/nanyx.playgrounddeclares itself a host withhost = (root = "playground.host", prelude = "playground.prelude"), and depends only onnanyx.console.core. Itsruntakes[Clock, Console, Random] () -> aanduses aConsole, aRandomand aClockwritten on imports of its own,nanyx:playgroundwrite,writeError(each given the address and byte length of the text in the module’s memory),randomBits(64 bits, as ani64),nowandticks(milliseconds, asi64s), andreadLineandtakeLine, which a page implements, not on WASI. ItsConsole’sreadLinecalls the page’sreadLine, which waits for a line and gives its length in bytes, or -1 when the input has ended (#error(#endOfInput)), then allocates a buffer that long and hastakeLinewrite the line into it; itsreadToEndreads lines until the input ends, each followed by a line break (#2151). Its prelude re-exportsConsolewithprint,println,eprintandeprintln,RandomandClock, so a console program that needs only those compiles under it unchanged, and one that needs a context it doesn’t provide isNYX0205naming it, by the check above (#2150). The runtime’s own helpers still import WASI’sfd_writeandproc_exit, for a deferred error or crash, so whatever runs the module provides those two as well. - The playground host’s runner.
packages/nanyx.playground/runneris the JavaScript that runs a module compiled for the playground host; it isn’t a source of the package (itssourcesaresrc/**/*.nyx), and the compiler built for the browser publishes it with the page.playground.js’srun(wasm, { seed, onStdout, onStderr, onDbg, onRead })startsworker.jsin a web worker and gives{ done, stop, input, closeInput }:stopterminates the worker, anddoneresolves to{ code }, withtrapwhen the program trapped, or{ code: null, stopped: true }. The worker implementsnanyx:playground(writeandwriteErrorpost the text, decoded from the module’s memory;readLinewaits for the page’s next line, below, andtakeLinecopies it into the module’s memory;randomBitsis SplitMix64 over the page’s seed, so a seed gives whatrandom.seededgives with it;nowisDate.now()andticksthe milliseconds since the run started, fromperformance.now()),nanyx:debugdbg(posts the file, line, column and text, and says it has dealt with it), and WASI’sfd_write(fd 1 is stdout, any other stderr) andproc_exit, which throws, so the program unwinds to the call of its start, which gives the exit code. A trap is exit code 1. The module is compiled with its start exported (WasmOptions.ExportStart) anddbggiven to the page (WasmOptions.DbgToEmbedder), so the worker instantiates it, then calls__nanyx_startonce the memory can be read; a program with nothing to run at its top level, only definitions, has no start, and ends at once with 0. The playground page uses it as Playground says. - Input in the playground. A program’s
readLineblocks, which a page’s main thread can’t, so the worker waits for the line: the page gives it aSharedArrayBuffer, threeInt32s (answered, the line’s length or -1 for the end of the input, and whether the line goes on) and then 64 KiB for the line’s bytes. To read, the worker postsreadand blocks inAtomics.waituntil the page answers; a longer line comes in parts. The page answers from the lines given toinput(text), in order; when it has none it callsonRead(), where a page shows its input line, and answers when a line arrives, or with the end of the input aftercloseInput(). A page has aSharedArrayBufferonly when it is cross-origin isolated, served withCross-Origin-Opener-Policy: same-originandCross-Origin-Embedder-Policy: require-corp, asdocs/vercel.jsonserves the playground’s route andserve.mjsserves the compiler’s page;canReadInputsays whether it is, and when it isn’t, the worker gets no buffer and a program’s input has ended before it starts. - The browser’s default host.
Script.compileUndercompiles a script that runs under a shipped host of the caller’s choosing unless its::hostnames another (Script.readFromUnder, andPackage.scriptModules, which take the default host);nanyx runand the language server pass the console host.BrowserCompiler.Compile(compiler/Nanyx.Browser) passes the playground host (Hosts.playgroundPackage), compiling the page’s program as the script/playground/main.nyxin the runtime’s in-memory file system, with a store that finds only the packages the compiler ships. The build writespackage-files.txt, the shipped packages’ files the page hands the compiler (wwwroot/compiler.js).wwwroot/run.htmlcompiles and runs a program, andplayground-test.mjsdrives it in headless Chromium with Playwright: the tour’s hello world,Randomfrom a seed,Clock, a crash,dbg, a program that needsFileSystem, two lines typed into the page’s input line and then Ctrl+D,readToEndand a line longer than the buffer, and Stop; the Compiler workflow’sBrowser testjob runs it. - Where
dbgprints.dbgneeds no context, and a host package does nothing for it: the runtime that runs the host’s program decides. A modulenanyx runruns gives eachdbgtonanyx:debugdbgfirst, with its file, line, column and text, and the handler says whether it has dealt with it (Probing.defineDbg). The console runner (WasmRunner) leaves it to the program, which writes[file:line] textto its stderr through WASIfd_write, in order with what else it writes there; the HTTP server (WasmServer) writes it to its log,dbg (GET /path): ..., and never into the response. A module written bynanyx buildornanyx wasmhas no such import and writes it to stderr itself, so another runtime can run it (Debug probes, #2058). - Host settings. A host declares the settings it takes, with their defaults, in its own manifest’s
host; a program’s manifest sets them in itshost’ssettings, and one the host doesn’t take, or of another kind than its default, is an error at the setting (HostSettings, checked whenPackagereads the host). A script’s::hosthas no settings, so it gets the defaults.
Why it’s built this way
A command-line tool, an HTTP service, a browser app and a test run are the same program model under different hosts. A host answers three questions for a program:
- What shape must the entry function have?
- Which contexts are available to it?
- Who owns the process: the main loop, concurrency, signals, shutdown and the exit code?
Different kinds of program need different answers. An HTTP service wants its entry to be a Request -> Response
handler and shouldn’t have to write its own accept loop. A sandboxed plugin should get fewer capabilities than a
CLI tool. A test run wants fake or recording implementations of the same contexts the real program uses. Hosts make
these answers explicit and swappable, and let the compiler check them.
- A host is a function that takes the entry and runs it, under
uses of the contexts it provides. Checking an entry against a host is checking it againstrun’s parameter type, and a library that requires a context the host doesn’t provide fails the same check. Nothing about hosts needs a check of its own. - One entry shape per host. The console host’s is
() -> (). A program’s arguments, environment and exit code come from theEnvcontext, not from a second shape. A handler may require fewer contexts than its host provides: a function type with fewer contexts fits where one with more is expected. - Capability security comes with it. Libraries never assume a host. They declare what they need, such as
[FileSystem], and a host that doesn’t provideFileSystemguarantees that the program, and every library it uses, can’t touch the file system. - Hosts are ordinary packages that the compiler ships but doesn’t treat specially, so anyone can write one.
- The HTTP handler is synchronous, and the host provides concurrency. A server can block, so the host runs many
calls of the handler at once, one instance per request, as
wasmtime serve, Spin and Fastly run components of thewasi:httpworld. An outbound call waits in its request’s instance only. A trap ends only that request.
Hosts are not frameworks
A host owns the process lifecycle. A framework is something that needs effects or produces an entry:
- The HTTP host listens on a socket, runs the request lifecycle, manages concurrency and shuts down. It expects a
Request -> Response. - A web framework provides routing, middleware and templating. It is a library that produces a
Request -> Response.
The test is: if it needs to own the main loop, it’s a host; if it only needs effects, it’s a library that requires contexts. That way any framework runs on any HTTP host, and swapping hosts (local server, serverless, WASI) doesn’t mean rewriting the application. Python’s WSGI/ASGI and Ruby’s Rack show how this split works out, and what happens without it.
Interface packages
A library that only makes sense with a certain kind of host, such as middleware for the HTTP host, depends on what a host provides, its contexts and types, never on the host itself. So each host is two packages:
- An interface package that declares the types and contexts:
nanyx.http.coredeclaresRequest,ResponseandHttp;nanyx.console.coredeclaresConsoleandEnv. It has no code beyond those declarations and the functions on them. - The host that implements it, its
runbuilt on foreign functions. It depends on its interface package.
Middleware is a function from handler to handler, (Request -> Response) -> (Request -> Response): it needs the
types, not the host that serves them. A library depends only on the interface package, and the capability check does
the rest. The same interface can have more than one host, so a third-party HTTP host is another implementation of the
same interface, with every library already written for it. This is the split WASI makes between an interface (WIT)
and its implementation, and that WSGI makes between a server and a framework.
Prior art
- Roc: platforms and applications. This is the closest existing design. The platform owns
mainand all effects; the application is pure. Roc’s platforms are written in a systems language, and making effects available to libraries in a platform-agnostic way was a long-running problem. Nanyx’s contexts address that directly. - Elm:
Browser.sandbox,Browser.elementandBrowser.applicationare hosts that each require a different entry shape. - WASI: worlds describe the imports and exports an embedder provides, which corresponds to a host’s contexts and entry shape.
- WSGI / ASGI / Rack / OWIN: the split between server and framework.
- .NET Generic Host: host configuration, lifetime and composition of hosted services.