All work VOICE AI / MOBILE & WEB in progress

04  /  VOICE AI / MOBILE & WEB

Aura Voice Companion

You talk.
It remembers.

A companion you talk to instead of typing at. The memory it keeps never leaves the phone.

July – September 2026FlutterDartPythonLiveKitGemini Live / OpenAI RealtimeDrift / SQLiteObjectBox HNSWNext.js
46things it can do by voice, from reminders to dialling

The challenge

I wanted a companion you talk to rather than type at, one that remembers months of conversation and keeps working with the screen off. The models were the straightforward part. Everything around them broke: it stopped mid-sentence to listen to whoever else spoke in the room, it answered "I am an AI, I do not have feelings", it forgot the previous session, and an early five-minute call cost about two dollars. Selling it was a separate problem, because a page of static text cannot demonstrate a product whose entire interface is a voice answering you.

What I built

I built a Flutter app of twenty-four screens, paired with a Python agent worker I run myself against LiveKit Cloud. The phone holds the only copy of the memory, an encrypted SQLite database and an on-device vector index, and publishes a short retrieval pack to the agent over the data channel at session start. I keep no database on the server and no transcript log: conversation passes through an in-memory ring and is discarded, apart from a few turns I deliberately write to a file when someone corrects the companion or dismisses a safety screen. The agent exposes forty-six voice tools, from reminders and lists to calendar writes, Gmail drafting, health readings and dialling.

How it works

ON THE PHONE ON THE SERVER The only copyencrypted SQLite + index Retrieval packsent at session start The agentno database, no transcript In-memory ringconversation is discarded The safety checka tier, a label, no text Unless you correct itor dismiss a safety screen On “forget that”, the vector goes first the other order orphans an embedding Written to a file the only exception
The mechanism, drawn from the build.

What it looks like

Aura Voice Companion — screenshot
Running locallyVOICE AI / MOBILE & WEB

The detail

01

Hybrid recall

Recall fuses two legs, SQLite FTS5 keyword search and an ObjectBox HNSW vector index, with reciprocal rank fusion at k = 60 over an over-fetch of 100. ObjectBox cannot load its native library under flutter test, so I unit-test the fusion arithmetic against an exact brute-force store, and an on-device integration test asserts that HNSW agrees with brute force on ordering. On a spoken "forget that", I delete the vector before the row, because the other order orphans an embedding with nothing left to find it from.

02

A bystander bug

Anyone else speaking in the room would make the companion stop and listen to them. Lowering the provider's onset sensitivity did reject bystanders, and I took it back out five days later because it also dropped the user's own quiet turns: eight to twelve seconds of dead air each time the watchdog rescued one. I have the gate pinned back to high, my current attempt sits behind a flag that is off by default, and the framework knobs that look relevant are dead code, because a realtime model doing its own turn detection reaches interruption through a path with no duration test in it.

03

Classified, not spoken

Every user turn is classified on the server, launched beside the model turn so it never blocks a reply. The verdict type has no field that can carry the transcript, so a log line or a crash report cannot acquire one by accident; it returns a tier and a label from a closed vocabulary. What the user hears is a scripted constant the app already holds, never generated text.

04

Consent at egress

Thirteen capabilities, from cloud voice and always-listening microphone to calendar, inbox, health and contacts, pass through a consent gate before any outbound call. A denied capability means the call is not made, rather than made and its result discarded, and the enforcement test asserts a call count of zero. I keep calendar read and calendar write deliberately separate, because cancelling a meeting emails guests who never installed the app.

05

Selling a voice

The marketing site I built under the working name Willster plays the product instead of describing it: one continuous scripted morning, thirty beats and twenty-six pre-rendered clips over 1:48, with a receipt filling in beside it as the conversation runs. A voice-mode model can paraphrase a line, and a paraphrase silently desynchronises every caption, so my build script checks the returned transcript against the script before accepting a clip, measures its real duration with ffprobe and writes the timings back into source. Captions, animation and voice therefore run off one clock, and I bound the page by the product's own rules as well, taken from statute rather than taste: no countdown, no waitlist counter, no scarcity in the pricing section.

The outcome

I use it on my own iPhone and a sideloaded Android APK; the last field report is a battery measurement I took off the phone in mid-September. The last recorded gate is 2,667 Dart tests, 39 goldens and 645 Python tests green with zero analyzer issues, and three integration tests red, one of which I flagged as a possible real defect rather than a stale assertion. The worker still runs on a one-vCPU droplet shared with an unrelated product, where a spoken reply pins the box and drains the agent's 200 ms egress buffer into audible word-by-word stutter. I have that diagnosed down to the cgroup's throttle counter and the CPU quota that causes it, with a migration runbook written and not yet executed. Neither surface is published: installs are sideloaded, and the site builds fully static but is not deployed.