Adeline Cook / Nova
An open-source Smart Home Control Surface designed to be installed and extended by the user's existing frontier coding agent, for the things a household uses every day: lights, climate, reminders, energy, media and voice.
Product overview
Nova sits above Home Assistant and turns a collection of devices and services into one household interface. It provides the framework the owner's existing frontier coding agent uses to build the controls required for the actual devices in the house, rather than shipping a fixed catalogue of device panels or its own installer agent. The resulting zones, controls and themes stay consistent across the wall panel, browsers and Apple TV. Voice processing remains local. Services can run on one machine or be distributed across several, while the household sees one coherent control surface. A public demo appears further down the page.
Private by design
The voice assistant runs inside the house.
Wake-word detection, transcription, the language model, speech synthesis and speaker recognition all run together on a 2018-era GPU. Conversations are not sent to a cloud service, an internet outage does not remove the core voice path, and there is no metered inference bill attached to ordinary use. The complete conversational stack is tuned to fit within 7 GB of VRAM so the system can run on capable hardware the household already owns.
Adaptive speaker recognition
Enrolment happens through normal conversation.
A household member can identify themselves during a conversation and Nova begins an acoustic profile. Subsequent speech refines that profile across different rooms and microphones, capturing enough environmental variation for reliable recognition throughout the house. Enrolment behaviour, recognition thresholds and individual profiles are all adjustable from the configuration workspace.
An interruption-aware dialogue controller preserves the active turn as a request develops. Late additions can be folded into the current statement; when they materially change the intent, Nova cancels and restarts handling against the revised request. Follow-ups retain conversational context without another wake word, and cross-room arbitration ensures that only one satellite responds.
Self-hosted · local inference · used daily
What it does
Lighting & zones
Lights, switches, climate, fans, covers and humidifiers are organised by room and zone. A single control can affect one fixture, a room or the whole house. Schedules handle routine changes such as lowering lights in the evening, while exclusions and presets prevent broad commands from touching the wrong device.
Voice
Small voice satellites can be placed around the house. Follow-ups, interruptions and late additions remain part of the active request, while arbitration chooses one room to answer when several microphones hear it. Noise suppression and echo cancellation prevent playback from re-entering the conversation.
Climate
A planner coordinates air conditioners and panel heaters around the household's target temperatures. Commands are queued and debounced so the dashboard respects device cool-down and polling periods instead of repeatedly correcting the same state.
Voice engines
The dashboard supports preset voices, quick voice cloning from a small set of reference clips, and fully fine-tuned voices. Samples, training and voice selection are managed from the same configuration screen, along with accent, mood, speed and pitch where the selected engine supports them.
Power & network
Live electricity use is combined with the household's actual tariff to show cost, estimates and account history alongside raw consumption. Network and host health remain available without filling the main household screen with infrastructure detail. A separate observability pipeline is used while developing and validating Nova; it is development tooling, not a feature deployed into every household installation.
Household information
Weather, rain radar, a local map, tasks, iCloud calendars, reminders and recent camera footage share the wall panel. Due items surface without being buried in a menu, while system health remains available without dominating the screen.
Mobile automation
Nova does not stop at the wall panel. Through Tailscale, iPhone Shortcuts and Android Tasker can connect to the dashboard wherever a household member happens to be.
That makes the house feel more considered: a personal routine can prepare a room before arrival, settle things after departure, or put the controls that matter in reach from a phone. The same calm control surface follows the household beyond the front door.
Zones
These screens are captured from the wall panel. Drag to explore a screen or open it at full size.
Zones can contain other zones, allowing the same control pattern to cover a fixture, a room or a whole branch of the house. Devices can also be excluded from their parent's broad commands. Membership and nesting are configuration, so the interface follows the household rather than forcing the household into a fixed plan.
An inverter air conditioner and a plug-in panel heater share this screen. The air conditioner gets modes, a timer, fan speed and a fresh-air damper. The heater gets on and off. Each control comes from the capabilities reported for that device, avoiding dead controls and special-case screens.
Speech engine, wake words, pronouns, conversational style and delivery controls save as one assistant profile. Speaker profiles and recognition tuning live alongside it. Changes take effect without requiring a service restart.
Current draw is resolved against the household's real retailer tariff, turning the current draw into an estimated cost per hour. The same screen shows projections, retailer history, modelled base load and a per-device view of current demand.
Timed and recurring reminders, with today and upcoming views, marked done from the wall or by voice. Due reminders can be announced in the relevant room, and the complete set can be imported or exported as data.
How it fits together
Nova's clients render controls from a shared schema and the capabilities reported by each service or device. The wall kiosk, configuration workspace and native Apple TV app do not carry separate lists of supported hardware. When a capability is added, the relevant controls appear wherever they belong.
The same approach keeps three very different speech engines behind one interface. One provides expressive preset voices, one clones from a small set of clips, and one runs a fine-tuned voice model. Each publishes the settings it supports. That makes the choice of speed, likeness or delivery a household setting, while the clients remain independent of the engine underneath.
One answer, in the right place
Several rooms may hear the same sentence. A household-wide arbiter selects one satellite to own the turn, while layered echo defence prevents the system from hearing its own reply as a new request.
Installed by your frontier agent
Structured operations tools and packaged skills are the installation and extension interface. Your existing frontier coding agent can provision one host or several, then add the device-specific controls the household requires. Nova does not bundle an installer agent.
Interaction design
The wall panel is read from across the room and touched in passing. The clock and status orb own the resting view, while the zone grid stays in place as controls change beneath it. Large touch targets suit one-handed use. Due reminders surface across the interface instead of waiting behind a tab. During an interaction, incoming device updates pause briefly so the control does not jump under a finger. If the network drops, the last good reading remains visible. Lite mode keeps the same layout on weaker hardware and removes only the expensive visual effects.
Shared behaviour
One versioned schema defines zones, thresholds, capabilities and design tokens for both runtimes. Validation and a dry-run import catch incompatible configuration before it reaches a device. A change is made once and interpreted by each client in its own native controls, reducing drift without forcing the two interfaces to share an implementation.
A native app for one household
The tvOS client is open source and agent-extensible. A frontier coding agent can adapt its controls, layout and behaviour for a particular installation, then build and install it directly on the household's Apple TV through Xcode. Apple supports local device testing with a free Personal Team, so this path does not require an App Store release or paid developer membership. Nova's current Xcode-managed development profile is valid for one year; when it expires, the same local build and deployment can be repeated. The result is boutique, agent-built software for the home: specific to the household, locally deployed and under its control.
Try it yourself
The whole dashboard runs in your browser.
A real Nova build runs on GitHub Pages with simulated data and no access to household devices or personal information. Change the theme, move through the zones and open the configuration workspace.
Source
One project, three public repositories.
Nova is maintained across its web dashboard, voice runtime and native Apple TV client. Together they include the portable configuration schema, wake-word pipeline, turn arbitration, speech engines and tvOS control surface. The demo build allows the interface and automated tests to run without live household devices.
Installation & extension
Nova does not ship a catalogue of fixed control panels. It ships the open framework: a shared schema, capability model, operations surface and packaged skills. The intended installer is the user's existing frontier coding agent, working from the devices, rooms and computers available in a particular household. Nova does not ship with an agent capable of installing itself. The owner provides clarity of purpose, household constraints and direction on the choices that materially affect them.
Have a ChatGPT Plus or Claude subscription? You already have the frontier coding agent Nova expects you to use. It installs and configures Home Assistant where needed, inspects the available hosts, resolves dependencies, assigns dashboard, voice, media and monitoring roles, connects the declared hardware and runs the deployment. The owner weighs in on important choices; the agent handles the setup and verification. The process is repeatable and reviewable whether Nova is installed on one machine or distributed across several.
Extension uses the same interface. For a new device, the owner's agent can add the adapter, capability description, controls and verification needed for that device instead of waiting for a prebuilt panel to exist. The shared contract then carries the new support to each relevant client. Configuration validation and dry-run imports make the proposed result inspectable before it reaches the live household.
Recommended stack
These are Nova's current, tested choices—not a bundled appliance. The owner's frontier agent installs the relevant parts, pins compatible versions and adapts the topology to the available hardware.
Home & application
Home Assistant is the device layer, with Mosquitto and Matter Server where the household's hardware needs them. The control surface uses Node.js and Next.js; voice and operational services use Python. FFmpeg handles media capture and DVR work, with systemd managing long-running services on Linux hosts.
Access, memory & source
Tailscale is the recommended private access layer between the owner's machines. MemPalace gives the frontier agent durable project and household context. Git carries the public software and specification, while the specification keeps credentials and private household data in designated stores outside version control.
Voice understanding
The local voice path uses Silero VAD, NVIDIA Nemotron Speech Streaming EN 0.6B for speech recognition, Qwen3.5-4B Q4_K_M through llama.cpp for interpretation, and NeMo TitaNet-Large for local speaker identity. DeepFilterNet3 provides CPU-side noise suppression. Wake words are matched in the speech transcript; Nova does not run a separate acoustic wake model.
Voice output
The selectable output engines are Qwen3-TTS 12Hz 0.6B CustomVoice for preset voices, dots.tts MeanFlow for zero-shot cloned voices, and GPT-SoVITS v2 for fine-tuned voices. Exactly one TTS engine is resident at a time; they are alternatives behind one Nova contract, not an ensemble that loads every model for each response.
Development observability
NATS and Grafana support the observability pipeline used to measure, diagnose and validate Nova during development. This pipeline is a development tool, not a Nova feature that installers are expected to deploy in the household.
Design priorities
A household system earns trust quietly: it preserves privacy, behaves predictably and recovers without turning ordinary users into operators.
Privacy
Keep speech inside the house
Conversational audio and inference stay on local hardware. Public repositories contain the software, while household addresses, hostnames, credentials and device details remain outside version control. The specification directs agents where sensitive information must be stored, so the repositories are safe to fork, commit and push.
Familiarity
Keep the layout stable
Rich and lite modes share the same information hierarchy. Web and tvOS use the same zones, capabilities and design tokens, allowing visual treatment to change without moving the household's controls around.
Resilience
Fail without losing the whole interface
The dashboard retains the last trustworthy state through short network interruptions. Voice and media paths have watchdogs and bounded recovery steps, so one unhealthy service does not require the household to restart the stack.
Extensibility
Describe capabilities once
Device and service capabilities flow through a shared schema. Adding supported hardware is primarily a configuration task, and clients render only the controls that make sense for that hardware.
Operability
Use observability during development
The development observability pipeline collects metrics and logs while Nova is measured, diagnosed and validated. It is development tooling, not a deployed household feature. Production deployments still coordinate dependent services and count as complete only after the running system has been verified.
Development approach
Coding agents contribute implementation across Nova. Architecture, acceptance criteria, review and release decisions remain explicit human responsibilities.
Agent-assisted development increases the amount of work that can be attempted in parallel, which makes vague requirements and weak review more expensive. Nova keeps the behavioural specification in the repository and requires it to move with the implementation. Golden snapshots protect dashboard state, end-to-end tests run against the public demo, and source checks guard feature parity between rich and lite clients.
A shared task log records ownership and blocks overlapping changes. Dashboard and voice releases move through one dependency chain, so a partial deployment cannot be mistaken for a finished one. Packaged operational instructions give both people and automation the same deployment contract.
The development loop is deliberately ordinary: define the behaviour, make a change, run the relevant checks, inspect the result and update the specification when the design has genuinely changed. Agent tooling makes that loop faster; it does not replace it.
Measure before and after
Performance work starts with a real bottleneck and ends with the same workload measured again. Findings are folded back into the specification and operations tooling so the improvement survives the next change.
For example, moving the media capture path from software processing to a hardware-accelerated pipeline returned most of the host CPU to the rest of Nova:
Verification
Unit, component, planner and end-to-end coverage can run against simulated data without access to live household devices.
Release safety
Dependency-aware coordination prevents dashboard and voice changes from being released as incompatible partial updates.
Privacy
Household specifics live outside version control. The software is public; addresses, hostnames, credentials and keys are not.
Ongoing project
Nova continues to evolve.
The public demo and three open-source repositories track the working web and tvOS control surfaces, voice runtime, configuration model and agent-facing extension tools.
Voice Agent Harness
Nova does not hand a transcript to an LLM and wait for a result. The harness compiles audio, transcript revisions, room state and conversation state into one governed turn; runs one schema-constrained interpretation; validates the proposed actions against fixed policy and live device state; then executes, verifies and terminates inside explicit latency and retry budgets.
Foreground turn compiler
Every foreground interaction advances through
capture → endpoint → contextualize → interpret → authorize
→ execute → verify → render → speak → commit.
Input and context are hashed into revisions; policy decisions, tool results,
verification evidence, response revisions, cancellation decisions and elapsed time
are recorded before the terminal state is committed.
01 / Deterministic input control
Satellites submit ordered 20 ms PCM frames. The activity gate retains 400 ms of pre-roll and an 800 ms silence tail; the server then scores falling or continuing cadence and transcript punctuation. A complete clause closes immediately. An unfinished clause can extend the endpoint once, up to a hard 3.5 second ceiling.
02 / Speculative read path
A prefix must contain at least three words and survive two interim observations before it can prefetch room context and likely tools. The result carries context and model-state hashes. It is reused only when the final transcript extends the exact token prefix; divergence cancels the prefetch. This path has no policy, persistence, execution, rendering or TTS access.
03 / Hybrid interpretation
One local structured pass combines the final transcript, confidence, room, active goal, compact live state, prosody and bounded conversation history. It may return zero to four ordered semantic actions. Deterministic cue repair then blocks negated commands, self-intention, abandonment and contradictory decisions; live target resolution, JSON schemas, identity thresholds and provider policy remain authoritative.
04 / Bounded state validation
After a device action, the verification loop never resends the mutation. It checks the immediate state, then performs at most 20 authoritative reads inside an eight-second deadline. Cheap deterministic checks run on every read; a throttled JSON-only LLM check may resolve an ambiguous still-settling state. The loop exits on proof, iteration cap or wall-clock deadline.
Voice stack flow
02 → 02. An unfinished cadence waits once for more PCM, bounded by 3.5 seconds. A stable interim prefix may prefetch read-only context in parallel.
02–05 → 02. New speech may cancel, merge PCM and replace the
prior transcript only while the turn remains before_side_effects.
07 → 07. Re-read authoritative state until confirmed, capped or timed out. This edge never returns to 06, so the mutation is never resent.
08 → one lookup → 08. An addressed epistemic failure may run
exactly one policy-checked web.ask and one grounded re-render. No recursion.
Authority implementation 3acd5bf
· test_identity_classes_enforce_deterministic_base_capabilities
· test_scoped_grant_honors_target_schedule_budget_expiry_and_revocation
Backward correction
before_side_effects may cancel. Old and new PCM are concatenated; stale prefetch is discarded.If a provider mutation may already have started, rewind is rejected. The new speech becomes a contextual follow-up, so completed work is not duplicated. If response audio has already started, playback is cancelled at a chunk boundary while the verified device result remains intact.
Race control
Household-wide source election and turn claims suppress duplicate microphones. A per-room service lock serializes interpretation and mutation. A resumed segment carries the newest transport and arbitration handles, while task-to-transcript mappings are pruned on completion to prevent stale revisions.
Response gating
True barge-in, backchannel, cross-talk and false interruption are classified without model latency. Only true barge-in sets the playback cancellation event. Queued actions may stop; an in-flight mutation completes and verifies unless its provider explicitly declares that cancellation is safe at any time.
Latency envelope
The response-text to first-audible-PCM span is measured on the native satellite path. The first codec chunk bypasses the steady-state frame floor and the default playback buffer is 700 ms, providing an enforceable sub-800 ms TTFB target at the delivery edge. The separate end-of-speech gate remains p50 ≤1.8 s and p95 ≤3.0 s, including STT, interpretation and rendering.
Observability without foreground coupling
The voice runtime emits content-reduced stage timings, policy outcomes, tool journals, verification revisions, interruption classes and first-audible latency. Dashboard-side operational events are written atomically to a fire-and-forget spool, drained by the host into NATS JetStream and then VictoriaLogs/VictoriaMetrics for Grafana. A monitoring failure cannot delay or fail the device command that produced the event.
ForegroundTurnStateMachine + immutable TurnTrace
test_foreground_turn_emits_complete_immutable_trace
d07d4f9
request_turn_continuation + merge_pending_turns
test_continuation_replays_only_before_side_effects
e0712a6
supersedes_announce_id + dedup survivor replacement
test_resumed_pending_audio_is_appended_and_uses_newest_turn_claim
bacfe3c
TurnCancellationController + declared cancellation safety
test_mutation_finishes_and_verifies_after_in_flight_cancel_request
d07d4f9
verify_loop.run: deterministic polls + throttled LLM confirmation
test_max_iterations_bounds_the_loop + deadline/LLM tests
bbb37c5
HouseholdAuthority + durable role/grant records + audit replay
test_scoped_grant_honors_target_schedule_budget_expiry_and_revocation
3acd5bf
d07d4f9+
Why this is harness engineering
The model proposes a typed plan; infrastructure owns every state transition.
The LLM does not decide when speech is final, whether speculative work may commit, whether an action is authorized, whether an old turn may be rewound, whether a mutation can be cancelled, when observed state proves success, or when the turn is terminal. Those controls are implemented as deterministic state, clocks, schemas, provider contracts and replayable tests. That is an agentic harness, not a passive model wrapper and not a permanent reasoning loop.
Primary implementation d07d4f9 · e0712a6 · bacfe3c · bbb37c5
Development Post-Mortem
LLM-backed code generation produced much of Nova's implementation, but code volume was not the limiting variable. The engineering work was defining cross-repository contracts, supplying environment-specific topology, measuring live runtime behaviour and translating each correction into versioned specifications, schemas, tests and deployment gates.
The central lesson
Loose context boundaries produce locally valid code that violates global architecture.
The primary engineering bottleneck was not syntax generation. It was maintaining invariant schemas across services, constraining write authority and proving behaviour against physical hardware, live network timing and observable device state.
The human contribution
I defined and maintained the system architecture, including service boundaries, integration with the household deployment topology and runtime validation against physical devices. Model-assisted implementation operated inside those explicit constraints.
01 / System invariants
Defining a device-agnostic configuration schema, engineering a local low-latency voice inference pipeline, and maintaining state sync across independent web/tvOS runtimes. These architectural constraints must be hardcoded by the developer; a repository-scoped context window does not contain the global topology, deployment roles or physical-device invariants required to derive them.
02 / Context
Repository context did not include mDNS dropouts, transient wall-kiosk lag, delayed Home Assistant convergence or races between air-conditioner polling cycles. Physical hardware validation supplied those missing inputs and exposed where internally consistent code failed under real network and device timing.
03 / Evaluation
Compilation and isolated tests did not verify runtime behaviour. Evaluation combined NATS/Victoria/Grafana telemetry with physical replay: VAD endpoint timing, the <800 ms response-text-to-first-audible target, duplicate turn claims, UI event-loop races during concurrent device updates and state jitter in the local inference path. Designs that increased latency or widened race surfaces were removed or rewritten.
04 / Governance
Runtime corrections were converted into explicit system prompts, strict JSON Schema parameters, Playwright integration tests and deployment checks. Every later code-generation session loaded the same static constraints before editing; repeated failure modes therefore became test failures or schema errors rather than another unbounded prompt-and-repair cycle.
Reading the Git record
Git records the implementation outcomes, not every prompt, observation or rejected idea. Read alongside the product decisions, however, it shows a clear progression: rapid expansion, repeated correction, stronger verification, and finally explicit boundaries around what belonged in a public repository.
20–21 April / Direction becomes software
A baseline, then immediate product shaping
The earliest retained history moves from a dashboard baseline to configurable controls, refined navigation and interaction polish. The pattern was established early: code-generation throughput was high, while product topology, interaction invariants and acceptance criteria still required explicit architectural decisions.
Git baseline → configurable controls → navigation → polish
May / Physical validation begins
The household deployment became the acceptance target
Tasks, energy, climate, a watchface and Apple TV settings arrived. So did direct evidence that technically complete work was not necessarily correct: one commit says the UI was “still not doing as asked”; the next records finally fixing the air-conditioner state highlighting. Human observation closed the gap between the implementation and its acceptance criteria.
Git 9e9116b → f2cf681
June–July / Runtime signals force structural reversals
Refactor, observe, simplify, repeat
A large refactor introduced Playwright coverage, followed by offline behaviour, monitoring and hardware video acceleration. The history records exact structural reversals: a stateful climate planner was abandoned and completely replaced by a stateless debounced queue to prevent device polling loops. Additionally, a persistent theme-token rendering bug took multiple iterations to isolate until an aggressive refactor of the underlying CSS layout revealed a hidden race condition in the theme-token engine. Both reversals came from tracing state custody and event ordering rather than adding another implementation layer.
Git e469b7b · fd80c4f · 1dcf924 · 811113d
18 July / Publication reset
The public/private boundary was introduced after implementation
Before the repository went public, its history had to be squashed into a clean root because early generation sessions lacked a strict public/private data boundary. The new root imports 421 files and explicitly states that public history begins there; deployment-specific hostnames, addresses and account names now live in a git-ignored private reference. After the reset, authority, retained context, sensitive confirmations and operational separation became visible infrastructure rather than assumptions.
Git de52162 · 186,847 inserted lines · “Public history begins here”
Persistence & coordination
A model context window is temporary; repository constraints and accepted decisions cannot be. A local, provider-neutral context store records prior decisions, failed approaches and proven workflows so each new session starts from the same governed project state regardless of model provider.
What we made persistent
The operating contract, in six rules
Deployment task orchestrator
The orchestrator turns coordination rules into shared machine-readable state. Each task records its owner, model and scope; longer epics add milestones and an explicit dependency map. Unrelated lanes can run in parallel, while collisions, stale task claims and blocked dependencies are surfaced before they become conflicting changes.
Because the contract lives outside any one conversation, this works across Codex, Claude, model families and providers. Linked work waits at the release boundary; the final task owner claims the combined deployment and verifies the running system. Coordination depends on the shared task record, not on access to another session's hidden context.
Lessons carried forward
Failure traces and accepted revisions produced four connected practices: specify the architecture, validate against deployed state, change the generation method when results plateau, and persist each verified correction.
While iterative implementation prompts caused compounding code churn, dedicating upfront tokens to a pseudo-code architecture phase mapped data-flow dependencies, state custody, cancellation boundaries and edge cases first. Verifying the execution path in a markdown specification before generating active files reduced code regressions and stopped repeated edits at the wrong abstraction layer.
Become the primary user and take a low-trust approach. Living in the household that ran the software surfaced timing failures, device quirks and awkward interactions that static reviews and simulated tests missed. Acceptance testing meant locating unstated assumptions in generated code, then proving or rejecting them with telemetry, recorded replay and physical device state.
Keep a working comparison and fallback. Google Home and ordinary switches kept the house usable while revealing what Nova still could not do. Those gaps changed the roadmap in unexpected ways, including iOS Shortcuts that bridged information and control Nova could not yet cover itself.
Generated code can be syntactically correct while its scope, state custody or service boundary violates the global contract. The corrective action was to simplify, narrow the problem or discard an implementation that satisfied its local tests but violated the system contract. Repeated implementation without new runtime evidence was treated as churn.
There was no consistent hierarchy of models—not even between two attempts at the same task. One provider's output could fail a layout or architecture gate while a different sampled candidate passed the same bounded comparison. Sometimes a stronger specification resolved the defect; sometimes a different provider supplied a useful alternative. Dedicated review passes inspected environment, retained context, tool access and procedure, then proposed changes that still passed the same governance gates.
A useful counter-test supplied working code and the same acceptance suite to a fresh model context, then compared the alternative implementation against the baseline. This was a bounded evaluation, not an open-ended polishing loop.
Fresh-context comparison: same contract, independent candidate.
Low generation cost made frequent candidate evaluation practical, including experiments outside the active release scope. Each candidate was deployed or simulated, measured against acceptance criteria and either retained, revised or removed. This kept iteration rapid without removing technical review.
Broad context improved those evaluations. Career requirements influenced the portfolio structure; the portfolio determined which Nova evidence needed to be explicit; the repository and operational traces supplied that evidence. Keeping those materials adjacent exposed useful cross-project relationships while explicit scope kept write authority narrow. The implementation session also retained staging and commit responsibility within the publication rules, preserving one auditable chain from requirements to Git.
Each accepted correction was encoded into MemPalace, shared skills, repository rules, specifications, tests or deployment gates. Conversation-local context affected one session; durable workflow context made repeated experiments reproducible.
The takeaway
Model-assisted development is an architecture and orchestration problem.
Low-cost code generation increases the importance of structural boundaries. Reliable output depends on invariant schemas, sandboxed context execution, explicit state ownership, NATS-backed observability, runtime acceptance budgets and automated verification that rejects regressions before deployment.