bethel-nz
/Simdriver
/project2026
/project2026

2026

Simdriver

One open driver that gives agents the whole iOS Simulator — frames, touch, and accessibility.

Visit project ↗
Role
A driver that gives AI agents the iOS Simulator
Problem
Every agent IDE that wants a simulator pane rebuilds a piece of CoreSimulator in private. simctl can't send a touch, and XCUITest needs a runner.
Constraint
Real HID input and live frames with nothing installed in the simulator, no daemon, and no crash that can take the host app down.
Outcome
One Rust core with three small helpers: 60 fps frames over shared memory, touch through the simulator's own digitizer service, and accessibility snapshots — open, and drivable from Node, Rust, or a CLI.

The gap every agent IDE keeps re-solving

Watch an agent IDE drive iOS and you'll notice each one built its own glass. Claude's app has an iOS pane. Codex had a picture-in-picture simulator. Every one of them reverse-engineered the same frameworks to get pixels in and touches out, and none of that work is reusable.

The tools underneath don't cover it. simctl can boot, install, and launch, but it cannot send a tap. XCUITest drives real input, but it needs an xcodebuild runner built and signed before the first tap lands, and it still can't reach the system gestures an agent needs — the home indicator, the app switcher. Meta's idb gets closer, but its HID surface carries a single contact and no edge flag, so swipes from the screen's edge die quietly.

So simdriver is the piece I kept wishing existed: one open driver a host embeds — an Electron app, a Rust window, a CLI — and the whole simulator just works inside it. The loop was already proven at 60 fps inside a prototype app I built a week earlier; simdriver's v1 was deliberately boring. Package what worked, change nothing.

Look, then touch

An agent that can't see the screen guesses at coordinates. The fix is a read-before-write loop, and simdriver makes each half of it real.

snapshot() waits for the screen to settle, then returns every accessibility element with a ref — e1, e2, e4 — plus the same picture annotated with boxes drawn around what it found. First snapshot costs a second or two; the next ones take tens of milliseconds. Targets can be a ref, a label like "Log in", or a raw point, and screens with no accessibility at all — games, maps — fall back to a lettered grid.

The rule that makes it safe: a ref is re-validated the moment it is used. Tap e4 and simdriver re-reads the live tree, matches the element by identity, and refuses instead of guessing when the screen has moved on:

rust
1let root = self.device.accessibility()?;
2let (fresh, _) = marks::elements(&root, screen);
3let now = fresh
4 .iter()
5 .filter(|f| f.same_identity(el))
6 .min_by(|a, b| a.moved_by(el).total_cmp(&b.moved_by(el)))
7 .ok_or_else(|| {
8 AgentError::Stale(format!(
9 "{target} {:?} is no longer on screen; take a new snapshot",
10 el.name()
11 ))
12})?;
13let moved = now.moved_by(el);
14if moved > MOVE_TOLERANCE {
15 return Err(AgentError::Stale(format!(
16 "{target} {:?} moved {moved:.0} pt since snapshot {}; take a new snapshot",
17 el.name(),
18 last.id
19 )));
20}
21self.touch(target, &now, root, screen, hold, double)

The refusals are the API. stale, covered, not-found — each one tells the agent nothing was touched and what to do next, the same way a failed edit refuses when you didn't read the file first. fill goes further: it types, then reads the field back over accessibility and only reports success when the committed value matches, so an agent never believes a keystroke the field swallowed.

Everything composes into steps() — a batched run that stops at the first refusal and returns the trailing snapshot — and record, which samples the frame stream at 25–30 fps into contact sheets an agent skims in order. Recording never adds load: frames are judged as they arrive and skipped, not queued. The CLI stays daemon-free; each invocation restores the last snapshot's refs from a session file and exits.

simdriver9:41Welcome backSign in to your workspace to continue.Email address••••••••ShowContinueForgot password?New here?Create an account9:41HomeTuesday, November 8TODAYDDesign reviewToday · 9:41SStandup notesYesterdayRRelease checklistTuesdayB1:1 notesLast weekHomeCardsSettingstestbethelBettyqwertyuiopasdfghjkl⇧zxcvbnm⌫123return

Run the loop an agent runs.

Swipe up from the home indicator, then tap the preview to return. You can also use the controls below.

ready · run a step, or run the flow

Fig.Run the loop an agent runs against the device: snapshot lists the elements as refs, fill types and watches the committed value, tap re-reads its element just before touching, and a drag from the bottom edge reaches the system through the edge flag.

A driver, not a screen recorder

The design decision underneath everything: pixels, not an overlay. The driver hands you frames — about 60 a second — and you draw them wherever you like. An Electron pane, a Rust window, a terminal cell. The host keeps its UI; the simulator is just content.

Three tiny Objective-C helpers carry the private-API surface, one process each, supervised by the Rust core with restart and backoff:

  • Frames. The simulator's own GPU surface, read through CoreSimulator's private screen objects and packed straight into a three-slot shared-memory ring under a seqlock. The newest frame wins; a slow consumer skips, never queues. This is not a screen capture — the framebuffer is the device's own surface.
  • Touch. The simulator's digitizer service (dtuhidd) over XPC — the same path Simulator.app uses, so input lands as real HID events, not synthesized clicks. When another client takes the digitizer, simdriver steps aside and reports it, then reclaims on your next touch; frames never stop.
  • Accessibility. A snapshot of the guest's element tree through the private translation framework, with nothing installed inside the simulator.

The gesture detail cost the most and is worth naming. iOS decides a swipe is a system gesture from an edge flag that must ride on every contact sample, not just the first. idb's transport can't send it, which is why edge swipes die there. In simdriver it's one field on the wire:

rust
1HidCmd::Touch { phase, x, y, edge } => format!(
2 r#"{{"t":"touch","p":"{}","x":{:?},"y":{:?},"e":{}}}"#,
3 phase.wire(),
4 x,
5 y,
6 edge.code()
7),

Gesture interpretation — which zone a touch started in, pinch versus pan — lives in the core as a pure state machine, so every host behaves identically. The private-header shapes the helpers compile against come from Meta's idb, vendored with a NOTICE: everyone loves open source, and attribution is how it stays open.

The same driver already renders a live simulator inside a terminal pane — frames over shared memory, touch included — which is the strongest proof the pixels-not-overlay bet was right.

What holds, what does not

Working today:

  • 60 fps frames with p99 gaps around 23 ms on the dev machine; the integration tests fail the build under 55 fps or 34 ms
  • about 9 ms median from tap to the first frame that shows it
  • real HID touch, hardware buttons, and keyboard, with system gestures working
  • accessibility snapshots with re-validated refs, batched steps, and load-free recording
  • nothing installed in the simulator, no daemon, and no private framework loaded into the host

Still rough:

  • the helpers compile against private frameworks, so every Xcode release is a compatibility check — that table lives in the README and lags first, by design
  • no biometrics, no device orientation, and two fingers is the limit
  • non-ASCII fill breaks on the current Xcode's pasteboard path
  • key() and text() need "Connect Hardware Keyboard" enabled

The honest summary: the pixels and the touch are solved problems now. The maintenance contract with Apple's private frameworks is the part that never finishes — and packaging it once, in the open, is how the cost gets shared instead of paid by every agent IDE again.