eddy — an async runtime from scratch

eddy is a from-scratch async runtime in Rust: explicit task state, custom wakers, current-thread and multi-thread (work-stealing) schedulers, readiness I/O (epoll / kqueue / IOCP), a hierarchical timer wheel, synchronization primitives, a Linux io_uring completion backend, runtime instrumentation, and TUI + web consoles.

Quickstart

fn main() {
    eddy::Runtime::new().block_on(async {
        let handle = eddy::spawn(async { 40 + 2 });
        assert_eq!(handle.await.unwrap(), 42);
    });
}

cargo test -p eddy --all-targets
cargo clippy -p eddy --all-targets -- -D warnings
cargo fmt --all -- --check
RUSTFLAGS='--cfg loom' cargo test -p eddy --lib

Where to look

• Architecture — the event → waker → queue → poll cycle.
• Task representation — allocation layout, state machine, refcounting.
• Why Pin exists — the intrusive waiter as the worked example.
• Cancel safety — per-future cancellation analysis.
• Cross-platform verification — which gates run where (Linux containers, QEMU ARM, Windows, Miri, ASan, loom).
• Roadmap — all 19 phases complete; Phase 12 covers the io_uring backend including multishot accept/recv.

Consoles

• eddy-console (TUI): task list, worker heatmap, wake causality, poll histograms — see crates/eddy-console.
• eddy-console-web + console-ui/: browser dashboard over a WebSocket bridge.
• Recorded demos live under docs/assets/; docs/assets/demo.sh drives a synthetic event stream through the TUI for recording. Rebuild GIFs with: python3 docs/assets/generate_demo.py && python3 docs/assets/render_gifs.py.

Eddy Architecture

Eddy has one execution path for an asynchronous operation:

event -> waker -> run queue -> task poll -> event registration

The reactor and timer driver do not execute user futures. A readiness event or timer expiry wakes the task's Waker. The waker performs the task state transition and submits the task to the current worker, the global injector, or the worker that owns the task. A worker claims the task, polls it once, and then either completes it, leaves it idle waiting for the next event, or requeues it when a wake raced with the poll.

Runtime Flavors

Builder::new_current_thread uses one owner thread and a local queue. It can run !Send futures. Builder::new_multi_thread starts workers with local queues, a LIFO fast path, a global injector, and stealing from other workers. The caller of Runtime::block_on drives the root future; worker threads drive spawned tasks.

The scheduler checks the injector periodically before local work. This bounds the time an externally spawned task can remain behind a self-replenishing local queue. The cooperative budget also forces hot resource loops through the queue instead of allowing one task to monopolize a worker.

I/O and Timers

io::Registration owns the platform registration and stores readiness waiters. The driver waits in the operating-system poller, then wakes registered tasks. The timer driver uses the same wake path: the timing wheel selects expired entries and wakes their task. This keeps polling policy in the scheduler and event translation in the driver.

Ownership Invariants

• A task is polled by at most one worker at a time.
• A task has at most one queued notification; a wake during RUNNING records NOTIFIED for the poller to requeue.
• The scheduler, join handle, and cloned wakers hold independent task references.
• A current-thread future and its destruction stay on the owner thread.

The implementation points are crates/eddy/src/task/state.rs, task/harness.rs, scheduler/multi_thread/worker.rs, and io/driver.rs.

Eddy Tasks

Each spawned future is stored in one Box<Cell<F, S>>. Cell is repr(C) and contains three regions:

Header: state, vtable, ownership/instrumentation metadata, queue link
Core:   scheduler and Stage<F> (Running, Finished, or Consumed)
Trailer: join-handle waker

The scheduler holds a type-erased RawTask pointer. Its vtable knows how to poll, schedule, read output, shut down, and deallocate the concrete cell, so the queues do not need to be generic over every future type.

State Machine

Header::state packs flags into the low bits of one atomic word and the task reference count above them. The important transitions are:

spawn -> NOTIFIED + queued reference
queue -> RUNNING + remove NOTIFIED
poll Pending -> IDLE, or NOTIFIED when a wake raced with poll
poll Ready -> COMPLETE, then output is read by JoinHandle
abort -> CANCELLED; the owner finalizes the future safely

RUNNING prevents concurrent polls. NOTIFIED closes the lost-wakeup window: if a reactor or another task wakes a running task, the current poll observes the bit and submits another queue entry before becoming idle.

References

The initial task allocation accounts for the join handle and the initial run-queue entry. A waker clone adds a reference. By-value wake consumes its waker reference; wake_by_ref does not. The allocation is released only when all scheduler, join, and waker references are gone. Current-thread destruction is deferred to the owner when a non-owner drops the last usable reference.

The size-budget test in task/raw.rs guards the concrete Cell<F, S> layout without making the internal scheduler or allocation layout part of the public API.

Why Pin Exists

An async function is a state machine. After it has been polled, one state may contain both a value and a reference into that value. Moving the state machine would invalidate the reference, so polling takes Pin<&mut Future> rather than &mut Future.

Waiter Example

The semaphore acquire future contains its waiter node. When it cannot acquire a permit, it puts a pointer to that node into the semaphore's intrusive FIFO list. The node must remain at the same address until it is removed. Its poll and drop paths therefore require the future to remain pinned:

#![allow(unused)]
fn main() {
let acquire = semaphore.acquire();
let mut acquire = std::pin::pin!(acquire);
// Polling `acquire` may link its in-future waiter into the semaphore.
}

Dropping the pinned future removes the node before its storage is destroyed. Moving a registered future would turn the list pointer into a dangling pointer. PhantomPinned makes accidental moves impossible for this waiter. Notify uses the same lifetime rule for its registration state; its waiter payload is reference counted while the future itself remains pinned.

Pinning is not a promise that a future will complete. It is an address stability guarantee for the future's self-references and for data structures that temporarily point into the future.

Cancellation Safety

Dropping a future is Eddy's cancellation operation. select!, a dropped join set, task abort, and runtime shutdown can all drop an in-flight future. An operation is cancel safe when dropping it at any await point does not lose or duplicate the logical item it was processing.

Shipped Operations

Designing a Select Branch

Use cancel-safe receive operations directly in a select. For a non-safe operation, make progress explicit: retain the buffer and offset outside the future, use a single low-level operation at a time, or finish the operation before entering another cancellation point. The runtime cannot recover bytes already consumed by the operating system or a peer.

The low-level contracts live beside the APIs in crates/eddy/src/io/mod.rs; they are part of the behavior users must rely on, not merely implementation notes.

Cross-Platform Verification

The native macOS build is the source of truth for kqueue behavior. Containers on macOS run inside a Linux virtual machine, so they are useful for Linux epoll/io_uring and Linux ARM checks, but they cannot validate kqueue or IOCP.

Linux on macOS

Podman and Docker both provide a Linux VM on macOS. Use two vCPUs, 2-4 GiB of RAM, and a small disk for the lowest-cost loop:

podman machine init --cpus 2 --memory 2048 --disk-size 10 eddy-linux
podman machine start eddy-linux
podman run --rm -v "$PWD:/src" -w /src rust:1.87-bookworm \
  cargo test -p eddy --features io-uring io::uring::tests::

The equivalent Docker command is:

docker run --rm -v "$PWD:/src" -w /src rust:1.87-bookworm \
  cargo test -p eddy --features io-uring io::uring::tests::

The ring probe treats ENOSYS and EPERM as an unavailable kernel feature; that is a valid result for a restricted container and should fall back to the epoll driver. Do not grant --privileged merely to make this probe pass.

Multishot accept and receive use a separate multi-completion state machine. AcceptMultishot yields owned descriptors, while RecvMultishot uses an owned provided-buffer pool and copies each completed packet before returning its buffer to the kernel. The stream retains orphaned buffers through cancellation and waits for the cleanup CQE before releasing them.

ARM

On Apple Silicon, a native aarch64 Linux VM/container is preferable to amd64 emulation. For the weak-memory check, CI runs aarch64-unknown-linux-gnu under qemu-user with the cross linker. That is slower than a local container but validates the target ABI and memory model without requiring a second machine.

Windows

Linux containers cannot exercise Windows IOCP, WSARecv, or Windows handle semantics. Keep the Windows job on windows-latest (or a Windows VM) and run cargo test --target x86_64-pc-windows-msvc there. A Linux container can only check platform-independent code and must not be used as evidence for IOCP.

Miri and sanitizers

Miri requires the nightly toolchain and the miri component. It is a host tool, not a substitute for an io_uring container: Miri cannot model epoll, kqueue, IOCP, or the kernel writing into a mapped ring. Run the pure task, timer, queue, and synchronization tests under Miri, and run the owned-buffer CQE cancellation test under Linux with ASan or Valgrind where available.

CI runs the pure-module Miri selection with MIRIFLAGS=-Zmiri-strict-provenance. The Linux ASan job uses nightly -Zsanitizer=address and only the dropping_a_read_orphans_it_until_the_cqe io_uring test. This keeps kernel availability and sanitizer failures isolated from the portable test gate. The test is ignored by default because shared GitHub kernels have returned from io_uring_enter(GETEVENTS) without delivering a matching CQE; run it with --ignored on a dedicated Linux kernel to validate orphan-buffer reclamation. It rebuilds the Linux standard library with -Zbuild-std and uses an explicit target so sanitizer flags do not affect host build scripts or procedural macros.

The repository CI already separates these checks: Linux native tests, aarch64/qemu, Windows, loom, strict-provenance Miri, ASan, and the feature-gated io_uring job. The io_uring job is non-blocking on shared GitHub runners because those runners currently do not provide reliable CQE delivery; it remains a visible diagnostic and must be repeated on a dedicated Linux kernel for release validation.

Phase 17 Tests

The cancellation, bounded soak, watchdog, and Tokio differential tests are ordinary integration tests and run in the normal package test job:

cargo test -p eddy --test cancel_safety
cargo test -p eddy --test stress --test watchdog --test differential

The differential test compares full-capacity try_send, async backpressure, sender and receiver closure, timer ordering/expiration, interval deadlines, and a cancelled select! receive against Tokio. Timer elapsed durations are checked for no early wakeup rather than compared byte-for-byte, since host scheduling is inherently variable.

The soak is intentionally bounded for CI. It uses 64 concurrent tasks and a five-second progress deadline rather than claiming a ten-minute wall-clock soak; longer RSS and deadlock runs belong in a scheduled stress job.

Phase 18 Benchmarks

Run the complete Criterion suite with:

make bench-vs-tokio

Criterion writes measured HTML reports under target/criterion/report/ and machine-readable estimates under each benchmark directory. The benchmark harness does not include checked-in numbers: record results from the host and toolchain that produced them. For a real instrumentation comparison, run:

make bench-report BASELINE=phase18

The first command saves the feature-disabled measurement. The second runs the same benchmark IDs with the instrumentation feature and compares against that saved baseline. Use a different BASELINE name when retaining multiple host or compiler runs. This is intentionally a measured comparison rather than a claim that instrumentation overhead is below a threshold.

When publishing results, use this template and include the exact command, commit, host, target, Rust version, and whether the CPU was otherwise idle:

Benchmark date:
Commit:
Host / CPU:
Target:
Rust / Cargo:
Command:

| Benchmark | Eddy | Tokio | Ratio or delta | Notes |
| --- | ---: | ---: | ---: | --- |
| spawn-join | measured | measured | measured | |
| ping-pong | measured | measured | measured | |
| channels/bounded | measured | measured | measured | |
| channels/unbounded | measured | measured | measured | |
| mutex-contention | measured | measured | measured | |
| timer-churn | measured | measured | measured | |
| echo-server | measured | measured | measured | Unix only |
| work-steal-latency | measured | measured | measured | |

Instrumentation baseline:
| Mode | Median | Change |
| --- | ---: | ---: |
| feature disabled | measured | baseline |
| feature enabled | measured | measured |

Sources:

• https://docs.podman.io/en/latest/markdown/podman-machine.1.html
• https://docs.docker.com/desktop/features/vmm/
• https://doc.rust-lang.org/cargo/commands/cargo-miri.html

Roadmap

The full 19-phase roadmap lives in CHECKLIST.md at the repository root. All phases are complete; Phase 12 covers the io_uring backend including multishot accept/recv.

Specification

The runtime specification lives in SPEC.md at the repository root. Start with Architecture for the event → waker → queue → poll cycle, then read the SPEC sections it cites.

Contributing

Contribution rules live in CONTRIBUTING.md at the repository root: one file per commit, track/* branches, green CI before merge, and the unsafe / loom safety rules.

Changelog

Release notes live in CHANGELOG.md at the repository root, currently tracking the Phase 12 multishot release and the 0.1.0 workspace baseline.