SpellType
On-device AI autocomplete that stays in the flow of writing.
A shipped macOS autocomplete system built around a locally running OLMo 2 1B model. The work joined model adaptation and blinded evaluation with a low-latency MLX runtime, native product behavior, and the unglamorous release engineering required to deliver an on-device model safely.
Visit SpellType
Project evidence
- <200 ms
- median suggestion latencyMeasured with the deployed local runtime.
- 10+
- supported writing apps
- 1,680
- curated fine-tuning examplesLoRA adaptation across all 16 transformer layers.
- ~3:1
- preference over the baselineDeployed 8-bit model versus the unadapted 8-bit model in a blinded 200-case evaluation.
The product constraint
Autocomplete has to disappear into the act of writing.
A writing assistant can be technically impressive and still fail if it interrupts typing. SpellType needed to watch a rapidly changing context, generate useful continuations, and get out of the way when the user kept typing.
I built the product as a native macOS system that runs OLMo 2 1B locally on 8 GB Apple Silicon and works across more than ten writing applications. Suggestions reached a sub-200 ms median latency in the deployed runtime.
Adaptation and evaluation
Improve the model, then make the comparison blind.
I fine-tuned OLMo 2 1B with LoRA across all 16 transformer layers using 1,680 curated training examples. The goal was not a larger model; it was a small model whose behavior better matched short, useful autocomplete.
In a blinded 200-case evaluation, the deployed 8-bit model was preferred roughly three to one over the unadapted 8-bit baseline. That result provided a concrete reason to ship the adapted model rather than relying on subjective demos.
System map
A preemptible, on-device suggestion loop
Typing updates the local context; cache reuse and token healing reduce repeated work; cancellation prevents superseded generations from reaching the interface.
- 01Typing context
Text changes across supported macOS apps
- 02Context preparation
Token healing · idle prewarm
- 03Persistent KV cache
Reuse valid prefix computation
- 04OLMo 2 1B on MLX
Local 8-bit inference
- 05Preemption
Cancel work made stale by new input
- 06Native suggestion
Present only the current completion
- Typing context connects to Context preparation.
- Context preparation connects to Persistent KV cache.
- Persistent KV cache connects to OLMo 2 1B on MLX.
- OLMo 2 1B on MLX connects to Native suggestion.
- Typing context connects to Preemption: input changed.
- Preemption connects to OLMo 2 1B on MLX: preempt.
First-party product material
SpellType, in product.
The hot path
Reuse work, preempt aggressively, never show a stale thought.
Continuous typing makes redundant prefill expensive and turns yesterday's generation into today's UI bug. The MLX runtime keeps a persistent KV cache, heals the token boundary, and uses idle time to prewarm likely work.
Swift Concurrency-based cancellation and preemption stop obsolete generations as input changes. The result is not just faster inference; it is product behavior that remains coherent under rapid typing.
- Persistent KV-cache reuse to reduce repeated prefill work
- Token healing at the changing text boundary
- Idle prewarming for the next likely request
- Cancellation and preemption to prevent stale suggestions
Production delivery
The model was only useful once the whole product could ship.
I owned verified model distribution, Developer ID signing, notarization, and signed auto-updates alongside the inference and interface work. Those release boundaries are part of the system, not an afterthought.
Business-context autocomplete is now in development. That work is exploring retrieval-augmented generation with embeddings and vector databases so suggestions can be grounded in approved company content and workflow context; it is not presented here as a shipped capability.
Private production code
About the source code.
The production repository is private. I'm happy to walk through relevant source code, architecture, testing strategy, trade-offs, and implementation decisions during a technical interview.