Overview

The constraint above is the entire brief: capture audio, transcribe it, work out what the user is actually looking at, and paste a clean result back into whatever field they were typing in — all on one Mac, with nothing sent off the device at any step. There is no cloud model to fall back on when the local one is slow, and no shortcut that ships a few seconds of audio to a server for a better transcript. Everything below is a chain of native macOS subsystems — audio capture, on-device speech recognition, screen and clipboard context, the Accessibility API — wired together by a pipeline that has to keep working when any single link changes state mid-session: a permission gets pulled, a headset connects, focus moves to a different window. The engineering problem isn't transcription accuracy in isolation; it's making five independent, individually brittle OS subsystems behave like one dependable tool.

On-device transcription with whisper.cpp

The foundation is a speech-to-text model that runs entirely inside the app's process, with no network call anywhere in the transcription path. whisper.cpp is a C/C++ implementation of OpenAI's Whisper model built specifically to run efficiently on commodity hardware — including Apple Silicon's CPU and Neural Engine — without a Python runtime or a GPU cluster behind it. That makes it the right fit for a desktop tool: it ships as a linkable library, loads a quantized model file once at launch, and runs inference against short audio buffers fast enough to feel like live dictation rather than a batch job.

Audio capture sits below the model, feeding it a steady stream of samples rather than one long recording processed after the fact. A low-level tap on the input device pulls raw PCM frames into a rolling buffer as the user speaks; the pipeline slices that buffer at natural pause points and hands each slice to whisper.cpp, so a transcript starts appearing well before the user stops talking rather than only after they release a key.

audioEngine.inputNode.installTap(onBus: 0, bufferSize: 1024, format: recordingFormat) { buffer, _ in
    ringBuffer.append(buffer)
    if ringBuffer.hasReachedSilenceBoundary() {
        let chunk = ringBuffer.drain()
        transcriptionQueue.async {
            let segment = whisper.transcribe(chunk)
            onPartialResult(segment)
        }
    }
}

Chunked capture feeding incremental transcription, rather than one blocking call at the end

Running inference off the audio thread matters as much as the chunking strategy: a model invocation that blocks the tap callback would stall capture itself, dropping audio rather than merely delaying the transcript.

Why not use a cloud dictation API

A cloud speech API will, model for model, usually out-transcribe a quantized local model — more parameters, more compute, no thermal or battery budget to respect. None of that is the deciding factor here. The moment audio leaves the device, the tool is making a promise on the user's behalf about where their voice — and whatever they're dictating over it — is allowed to go, and for anything touching sensitive material that promise is not the app's to make. A cloud path also adds a round trip on the critical path of a feature that is supposed to feel instantaneous: network latency, provider-side queuing, and a retry policy for when the connection drops all sit between speaking and seeing text appear, on a feature where a half-second of extra lag is the difference between "invisible" and "annoying." Local-first removes both problems at once instead of trading one for the other: no data leaves the machine, and there is no network hop to add latency or fail.

Context gathering: screen, clipboard, and selection

Transcription alone tells the pipeline what was said, not what the user was looking at while they said it, and a lot of real dictation is shorthand that only makes sense with that context — "reply to this," "make it shorter," "fix the typo above." Before the transcript is finalized, the pipeline gathers three sources of on-screen state: a screen capture run through OCR to recover visible text, whatever is currently sitting on the system clipboard, and the text currently selected in the frontmost application, read through the Accessibility API rather than a copy-paste round trip that would visibly disturb the clipboard.

Those three signals are assembled into a compact context block and handed, along with the raw transcript, to an optional LLM cleanup pass. That pass does two things a transcription model alone can't: it resolves references the user made to what's on screen, and it repairs the small disfluencies — false starts, filler words, a mid-sentence correction — that show up in real speech but not in written prose. The pass is optional by design, not just by configuration; a user working with nothing sensitive on screen, or one who wants the rawest possible transcript, can skip it entirely and still get usable, if less context-aware, output straight from whisper.cpp.

This is also where local-first pays off a second time: the same on-device model that keeps audio off the network can be reused to keep the screen context off the network as well, so context-awareness doesn't quietly reopen the privacy question the audio path just closed.

Accessibility-API paste injection

A transcript is only useful once it lands in the field the user was actually typing into, and that turns out to be its own small integration problem. The pipeline doesn't know in advance which app has focus, what kind of text field it is, or whether that field even supports the Accessibility API's richer value-setting calls — a modern native text view usually does, but plenty of web-based editors and some cross-platform apps expose only a partial or nonstandard accessibility tree.

The pipeline first asks the currently focused element, via AXUIElementCopyAttributeValue, whether it accepts a direct value write. Where that succeeds, the transcript is set as the field's value in one atomic call — no visible keystrokes, no risk of a dropped character under system load. Where the target element doesn't support that path, the pipeline falls back to synthesizing keyboard events through CoreGraphics' event-tap APIs, effectively typing the transcript character by character the way a very fast, very accurate typist would.

if let focusedElement = AXUIElementCopyFocusedElement(),
   AXUIElementIsAttributeSettable(focusedElement, kAXValueAttribute) {
    AXUIElementSetAttributeValue(focusedElement, kAXValueAttribute, cleanedTranscript as CFString)
} else {
    synthesizeKeystrokes(for: cleanedTranscript)
}

Preferring a direct accessibility write, falling back to synthetic keystrokes

The fallback path is slower and, in principle, interruptible mid-paste if the user starts typing at the same moment — a real edge case, not a theoretical one, given that the whole point of the tool is to sit alongside normal typing rather than replace it. Preferring the direct write wherever it's available is what keeps paste injection reliable across the very different text-field implementations a general-purpose dictation tool actually meets in the wild.

Coordinating five macOS permissions

None of the above works without the user granting it, individually, in System Settings, and macOS gates each capability behind its own prompt: Microphone access for audio capture, Screen Recording for the OCR context pass, Accessibility for reading the current selection and for paste injection, Input Monitoring for the global keyboard shortcut that starts and stops dictation, and Automation for querying which application is currently frontmost. Each of those is a separate entry in the system's privacy database, requested and revocable independently of the other four, and macOS does not offer a single "trust this app" toggle that covers all five at once.

That has two consequences the pipeline has to design around. First, the app has to be able to start in a state where only some permissions are granted — a fresh install has none of them — and offer a degraded but working mode rather than refusing to run: dictation without screen context still works with only Microphone and Accessibility granted, for instance. Second, and less obvious, any of the five can be revoked by the user at any point after being granted, not just denied up front, so a permission check performed once at launch is not sufficient; the pipeline re-checks the relevant permission immediately before the capability that needs it runs, rather than trusting a cached "granted" state from minutes or hours earlier.

Graceful degradation under device and state changes

The chain above is long enough that something in it changing mid-session isn't a rare event — it's a Tuesday. The clearest case is audio hardware: connecting a Bluetooth headset, or unplugging a USB microphone, mid-recording used to be treated by early versions of the pipeline as fatal, tearing down the whole capture session and losing whatever had been buffered. The fix was to subscribe to Core Audio's device-change notifications and treat a hot-swap as a routine event: the pipeline flushes whatever audio is already buffered through the model, tears down and rebuilds the audio tap against the new default device, and resumes capture — audibly, to the user, as a brief gap rather than a crash or a silently truncated transcript.

The same posture extends to the other subsystems. A permission revoked mid-session — the user opens System Settings and turns off Screen Recording while dictation is active — degrades that one context source rather than aborting the whole pipeline; the transcript still completes, just without screen OCR feeding the cleanup pass that run. A focus change to a different application between "recording started" and "paste ready" is handled by re-resolving the focused element at paste time rather than caching it at record time, so the transcript lands in whatever field is actually focused when it's ready, not wherever focus happened to be a few seconds earlier.

NotificationCenter.default.addObserver(forName: .AVAudioEngineConfigurationChange, object: audioEngine, queue: nil) { _ in
    let pending = ringBuffer.drain()
    if !pending.isEmpty { transcriptionQueue.async { whisper.transcribe(pending) } }
    audioEngine.stop()
    configureTap(on: audioEngine.inputNode)
    try? audioEngine.start()
}

Treating a device change as a resume point, not a teardown

The design principle underneath all of this is the same one that shows up in the permission handling: every subsystem the pipeline depends on is assumed to be able to fail or change state independently of the others, and the pipeline's job is to keep as much of the remaining chain working as it can rather than collapsing the whole session over one link.

Tradeoffs and limitations

None of this comes for free, and it's worth being explicit about what stays unsolved:

• On-device inference has a real accuracy and speed ceiling relative to the largest cloud speech models — the tradeoff for keeping audio local is accepting that ceiling rather than chasing state-of-the-art transcription quality.
• Screen OCR context only helps when the relevant text is actually visible and legible on screen; a reference to something scrolled off-screen, in an image, or in an app that renders text as a bitmap gets none of the benefit.
• Five independently revocable permissions mean five separate places a session can be interrupted by the user changing a setting, not one — more moving parts to design a graceful path for, not fewer.
• The optional LLM cleanup pass is itself a place errors can be introduced — a model repairing disfluencies can also over-correct, dropping or rephrasing something the user actually meant to say.
• Graceful degradation reduces crashes, not confusion; a context source silently dropping out because a permission was revoked mid-session is safer than a crash, but the user still has to notice their output got less context-aware than usual.
• Running a speech model locally has a real resource cost — CPU and Neural Engine time that a cloud call wouldn't consume on-device — which matters more on older or thermally constrained Macs than on the hardware it was built against.

References and further reading

• whisper.cpp — on-device Whisper inference in C/C++
• Apple Developer — Accessibility documentation
• Apple Developer — Core Graphics documentation
• Apple Developer — ScreenCaptureKit documentation
• Apple — Platform Security guide