My research spans inference-time reasoning and evaluation, acoustic intelligence, agent coordination, compute, and recovery. The practical question is how to make capable AI act reliably under real constraints.
My frontier-model research focuses first on inference-time compute, verifier reliability, and evaluation, followed by controlled model adaptation and learning experiments. The projects below range from completed experiments to early-stage proposals; they do not yet form one integrated autonomous system. I do not claim frontier-scale pretraining or demonstrated frontier-model performance.
How I help
If you are buying, building or investing in a voice or language-model system, I can tell you which parts are real.
- Review an AI, agent or speech system. An independent look at quality, cost, provider failures and recovery, including voice products where the demo works and the field deployment does not. You receive a written report with prioritized findings and a remediation plan.
- Technical due diligence. Before an acquisition or investment, a review that goes past the vendor’s slide deck to what actually runs. You receive a written report with risk-tiered findings and a hold or proceed recommendation.
- Check code a model wrote. The AI Code Provenance Audit is free to use online or as a PDF. On a live deal, CodeProvenance.ai is the fast first read: the target runs my scanner on its own machine, and I write a five-page memo 48 to 72 hours after the bundle arrives.
Prices, scope and current availability are on the services page.
Build, measure, recover
I connect the architecture to the operating requirements: quality, latency, cost, security, and recovery. A model response is one component. A working system also needs clear ownership, observable execution, and checks that establish whether the task succeeded.
- Build: connect models, tools, state, infrastructure, and physical interfaces.
- Measure: record task outcomes, model behavior, costs, and the conditions behind each result.
- Recover: test failed workers, unreliable compute, noisy inputs, and unavailable operating systems.
How I test a research claim
I start with a testable hypothesis, compare it with simpler alternatives, and keep the evidence that changes my conclusion.
- Define the comparison. State the hypothesis, task and failure criterion. Compare simpler baselines under matched information and budgets; distinguish my implementation work from upstream models, data and methods.
- Freeze conditions before outcomes. Record exact source, model and data identities, splits, scoring rules, stopping decisions and resource limits. Keep development and calibration separate from held-out evaluation.
- Retain and expose the evidence. Keep failures, abstentions, unrun cases and unknown costs in the registered accounting. Link versioned artifacts and descriptive uncertainty, and state which records an authorized reviewer can actually rerun.
- AMBIE: revise the assumptions: The acoustic evidence changed planning priorities; third-party benchmark results do not establish AMBIE-trained accuracy.
- Verifier selection: inspect the headroom: A completed engineering pilot exposed a limited candidate pool. It did not test the separate joint-verifier hypothesis.
- Echo: separate coordination mechanisms from outcomes: The reused-fixture coding diagnostic remains negative; separate bounded path-lease and crash-recovery checks now test coordination primitives, not coding-task benefit.
- Information limits: state the assumptions: A short fixed-pool note proves a ceiling on what any selector can gain from verifier signals and lists seven ways it fails to transfer to real policies. It reports no model result.
A source-only guard, a load-qualified runtime and a completed task experiment support different claims. My self-published reports and same-workflow reviews remain separate from outside peer review and independent reproduction.
Four projects to start with
Echo
Coding agents coordinate across machines and supported vendors through durable messaging, leased tasks, persisted results and advisory file reservations. Self-hosted deployments require a user-managed NATS server and credentials for participating machines. Echo coordinates workers; it does not sandbox their processes or synchronize repositories. Persistent records alone do not restart workers after a crash or reboot, and interrupted turns are not replayed as if they had completed. A result record still needs task-specific correctness checks.
- Headline result
- Historical reused-fixture diagnostic: single agent 8 of 16, ordinary two-worker messaging 0 of 16, Echo 0 of 16.
- Does not establish
- Improved coding-agent task success: the historical fixture predates shared path leases and was reused. New local checks establish path-lease behavior and bounded guest reboot-resume, not coding-task improvement or physical-host recovery.
TuneHarness
Checks that rented GPUs can compute before starting work, monitors runs, and retains a cost ledger after instances disappear. It also supports documented model experiments, including fine-tuning and tool-calling evaluations from its former VastOps period.
- Headline result
- Provisioning ledger, 11 July to 15 September 2026: 75 of 121 rented GPU instances (62%) never delivered a verified usable GPU. Those instances cost $4.96 of $41.62 total.
- Does not establish
- A failure rate for either marketplace, or proof that training scales to any size. The model experiments are documented under Systems in detail, with their limits.
Compute and cost evidence → · Completed model comparison and limitations →
Bootscry
Reads a KVM console and sends keyboard and mouse input when the target OS is unavailable. Used for autonomous setup and rescue across single-board computers and older PCs.
- Headline result
- Two ARM boards brought up, one older PC installed with Debian and another rescued. Watchdog resets and a machine hang needed manual power interventions, and the records keep them.
- Does not establish
- An unattended success rate. The evidence is a small set of documented sessions, with OCR errors retained.
Noise or Voice
Investigates whether interpretable acoustic features can help distinguish synthetic speech across generators, speakers and recording conditions. The working prototype analyzes complete recordings. Internal evaluations retain failures; its scores are uncalibrated and do not authenticate speakers.
- Headline result
- In a developer-selected, known-label 168-trial comparison, the signal-processing (DSP) detector was correct on 123 and a third-party self-supervised baseline on 140.
- Does not establish
- General model superiority: the audio windows differed and training exposure is unresolved. Earlier historical-detector diagnostics retained failures and do not validate the current detector.
Research methods and detector limits → · Read the evidence →
Project status and evidence
Status reviewed 21 October 2026 (Pacific). A published website is not, by itself, evidence of a deployed capability or an independently validated result.
| System | Status | What it does | Evidence boundary |
|---|---|---|---|
| Echo | Pre-release | Agent messaging, task leases, advisory reservations, and persisted results | Local transport measurements and internal recovery qualification. No coordination benefit established; task diagnostics and limits are in Systems in detail, and a fresh task study remains planned |
| TuneHarness | Operating tooling and ML experiments | GPU preflights, run monitoring, evaluation gates, cost accounting and documented model experiments | Dated provisioning sample; not a marketplace-wide failure estimate or proof of arbitrary training scale |
| Bootscry | Working internal tooling | Out-of-band KVM control for machine setup, installation and rescue | Documented ARM-board and x86 use; OCR errors, hangs and manual power interventions documented |
| Model Gateway & Router | Internal tooling · E4 empirical result withdrawn | One interface to local and hosted models, with tracing, an evaluation runner, an escalation ladder and trace mining | The earlier E4 comparison used a seeded routing simulation with generated task outcomes and assumed prices; it did not measure model quality or savings. Its empirical interpretations are withdrawn. The simulator tests selector mechanics only; a real-outcome evaluation remains planned. Read the corrected status |
| Mail triage agent | Evaluation built; service not built (added 8 October 2026) | Tiered triage of mail into proposed calendar items, with code checks and a person accepting each proposal | One mailbox, model-made labels and no human spot-check. Read-only ingest has run; the model tiers are not in service. Figures are in Gateways & Agent Orchestration |
| Personal archive analysis | Working internal tooling; some stages unfinished (added 8 October 2026) | Local models over 17 years of my mail, photos and activity records, with a privacy screen | No independent accuracy check, and several stages are unfinished. Details and limits are in Compute & Training Substrate |
| AMBIE | Live passthrough; model research | Commercial speech-engine access and research on noisy speech | The benchmark below evaluates third-party models; no AMBIE model accuracy is claimed |
| Tacetto | In development (added 8 October 2026) | Many small nodes sharing what they sense, with a room unit (wake word, microphone array, speaker, camera) and a speech and language service | Only a single room unit exists. One timing of the open-source wake-word model on a Raspberry Pi 5, run on synthetic noise: 10 ms per 80 ms of audio. The shared memory across nodes is a design with no implementation or test. No microphone, accuracy, latency or end-to-end result yet |
| Noise or Voice | Experimental acoustic research | Complete-recording analysis of synthetic-speech indicators | Selected internal samples with documented failures; not speaker authentication or independent blind validation. Figures and limits are under Acoustic & Voice Intelligence |
| CodeProvenance | Scanner and advisory offering | Local code metrics, encrypted bundles, and a human-written acquisition assessment | Scan metrics inform diligence; they do not prove authorship or that a team understands its code |
| LLM Lab | Educational implementation | Browser-based neural-network training and visible model mechanics | Small-model education, not frontier-model research |
| ThermoCog | Whitepaper and design | Research questions about computation and energy | No hardware or software implementation and no measured results |
How I got here
Oldest first. The note at the end of each row is what the field was doing at the time, so the dates mean something.
- 2008
The Netflix Prize
I joined about a year and a half after it opened. Part time, a few weeks, on an old desktop. I got to somewhere in the low 200s on the leaderboard. Teams from whole universities, with clusters, had not reached the 10% target in that time, and I wanted to know why. I found out quickly. My own ceiling was compute, and I had no more of it. My curiosity was satisfied, so I stopped.
The lesson held up. Netflix later wrote that the extra accuracy from the winning blend “did not seem to justify the engineering effort needed to bring them into a production environment.” Accuracy and a working system are different things. That gap is most of what I get paid to look at.
The field: the prize opened 2 October 2006 and closed 26 July 2009. 5,169 teams made 44,014 submissions.
- About 2012
Roger
A soft walkie-talkie with speech recognition built in. The first of the speech products.
The field: in November 2012 four research groups published a joint paper showing deep neural networks outperforming the older acoustic models on a range of benchmarks. Recognition was about to change direction.
- 2017 to 2018
SmartEar
CTO of a hearing-technology company. Hands-on across the mobile clients, desktop, firmware, the API and SDK, and the server. It taught me that speech is a whole-system problem. The microphone, the firmware, the network and the server all decide whether the right words come out.
The field: in 2016 WaveNet showed that synthesized speech could sound clearly more natural than earlier systems.
- 2018 to 2024
Voice AI for federal agencies
Co-founder and CTO of a Bay Area voice AI company. We deployed speech recognition for the US Coast Guard and DHS, in the places where standard recognition fails. Radio audio is narrow and noisy before any model hears it. I also designed the passive adapter that let the software key a radio transmitter with an audio tone, so it could send as well as receive.
A company-era patent from this period, US 11,924,717, covers analytics for walkie-talkie networks and was granted 5 March 2024.
The field: Whisper was released on 21 September 2022, trained on 680,000 hours of multilingual and multitask audio. Open model access broadened speech-recognition development; the noisy-speech measurements below show why application-specific evaluation still matters.
- 2025 to now
Research, then agents
AMBIE is the research on noisy speech. The rest of the recent work is the tooling around language models, sovereign multi-modal pipelines, synthetic audio defense, and the agents that coordinate across them. It is all below, organized by layer.
Systems in detail
Operating tools, internal implementations, and prototypes, organized by layer. Research projects follow separately.
Compute & Training Substrate
Local unified-memory silicon, GPU fleet harnesses, deterministic OCR, and bare-metal console agents.
TuneHarness
An operating GPU-rental and training-run ops layer for vast.ai and RunPod. It checks GPU delivery and bf16 computation, monitors runs, applies an idle watchdog, and keeps an append-only cost ledger. Evaluation gates check declared metrics before a model is accepted. It also contains documented specialist-model experiments, including work under its former VastOps name: fine-tuning, tool-calling evaluations, single-node multi-GPU execution and speech benchmarks. Each experiment retains its dataset, hardware and evaluation limits; the whitepaper records the experiment history and current scope. The LibriSpeech benchmark harness behind the table below is part of this project. Tooling and documentation →
My ledger sample covers 11 July to 15 September 2026: 121 instances created, of which 75 (62%) never delivered a verified usable GPU. Those instances lived about twelve minutes on average and cost $4.96 of $41.62 total. This describes my provisioning sample, not the failure rate of either marketplace as a whole.
A completed reconstructed names-only tool-caller comparison and a separate load qualification of a small tool-calling model are documented with their limits. A separate native comparison retained 400 records per arm: base 228/360 call and 0/40 no-call correct; checkpoint-600 0/360 call and 40/40 no-call. Its native invocation collapse and eight truncated outputs limit interpretation.
A post-hoc audit replayed all 800 native labels unchanged. On a separate strict-JSON endpoint removing one terminal <eos> only at stop ID 1, checkpoint-600 scored 159/360 calls and 36/40 no-calls, pooled 195/400 (48.75%). Exact stored JSON scored zero for both arms. This endpoint-specific result does not establish causal improvement or equivalent full-schema performance.
A 160-case authored full-schema comparison, with fixtures written after the earlier findings, scored the reconstructed names specialist 160/160 against the publisher’s 1.7B-parameter model at 132/160, and original parsed, typed and strict-JSON-conjoined correctness agreed. Read the full method and limits.
A source audit found that the earlier 504-task / 4,032-record Pillar 1 sweep was a seeded synthetic policy simulation: candidate correctness, test outcomes, verifier scores, costs and latency were generated by code, with no model calls or benchmark tests. The reported covariance, accuracy, domain-shift and token-conservation values are withdrawn as empirical evidence; all three simulation labels share the same 14 domain names, and the 0.85 stopping threshold is unreachable. The separate 120-task HumanEval engineering pilot is descriptive; the proposed H1 study remains unrun. Read the correction and current evidence boundary →
Imagine
An internally operated image-generation and editing pipeline on an AMD Strix Halo APU, combining third-party diffusion models and an upscaler. Processing stays on local hardware and incurs no per-request API charge; electricity and hardware still have costs. Its outputs demonstrate pipeline integration, not new model architectures. Speed and energy per token for local language models on the same machine are measured and published at roamingpigs.org.
Deterministic OCR
Local text extraction from console images for automation and the Bootscry feedback loop. A non-generative reader can still misread text; recovery actions need screen-state checks and a record of misses. Latency depends on the capture and processing path.
Local APU Inference Economics
The local profiler combines a real model/power-meter acquisition path with calculated hardware and cost scenarios. The previous performance and energy headline came from a formula-generated row. Three limited summaries labeled live remain separate; raw power arrays are absent from the named ledger. Calculated bandwidth, efficiency and GPU-cost comparisons are assumptions rather than measured superiority. Read the evidence correction and next measurement →
Bootscry
Working out-of-band KVM tooling used to autonomously configure and install physical machines. Current evidence includes two ODROID-M1S ARM boards brought up with Armbian and rescue cards, and Debian installation on a 2010 mini PC and rescue boot on another x86 machine. I also used Bootscry to set up a Libre Computer SBC; the SD card was written on another host, and this setup is separate from the linked KVM session records. The ARM bring-up record is published with its method and limitations. The GL.iNet Comet is the KVM device exercised. Watchdog resets and a machine hang required manual power interventions; the records retain those failures and OCR mistakes. Watch the sessions and read the limitations →
Seventeen years of my own data
Local models read my own Google export: over half a million emails, tens of thousands of photos, and a dozen other kinds of record. Everything ran on my own network, and the photos never left it. The mail was indexed for full-text search in about three minutes. Records were joined by date, and each place is labeled by how it is known: measured by a device, taken from a booking, read from a photograph or carried forward from the day before. A coordinate shows where a device was, not where a person was.
The photo work chains image-text and self-supervised embeddings, calibrated tagging, face detection and grouping, vision-language captioning, text reading, and a two-stage privacy screen that holds photos showing documents, cards or credentials. Most candidate tags failed their per-tag threshold and were dropped. In an audit of six low-score holds, two had been held wrongly, and I chose not to loosen the screen. Names never enter a prompt, and a model’s guess stays a candidate until confirmed.
Three things went wrong. A first attempt to group faces put 98% of them in one cluster, because small faces acted as hubs that chained unrelated people together. A mutual nearest-neighbour graph fixed it, with the largest group falling to 4%, at the cost of a deliberately over-split result. One captioning pass failed on 19% of photos, mostly from my own bugs, including a check that rejected accented letters. And sender history from the mail archive gave no measurable gain for triage: on the 186-email development set, 63 of 64 actionable emails were found with or without it.
Because the thresholds were set against the archive’s own labels, the precision figures are floors, not independent accuracy. Captions and face groups are guesses, and library-wide captioning, near-duplicate clustering and face naming are unfinished. The photos, face data and mail stay private.
Gateways & Agent Orchestration
Routing, dispatch, coordination, and persistent execution across local processes and cloud endpoints.
Model Gateway & Router
Internal model-access and routing tooling spans local and hosted providers, with retries, budgets and tracing. Its companion routing comparison currently uses seeded, generated task outcomes and assumed prices. That simulation tests selector mechanics and establishes no measured model-quality, savings or production-routing advantage. A real-outcome evaluation remains planned. Method and evidence boundary →
echo.cc
Pre-release coordination for coding agents across vendors and machines: durable messaging, a shared task list, leased ownership, advisory file reservations, and persisted result records. Reservations depend on cooperating agents and do not prevent filesystem writes. On 21 September 2026, three local 50-message runs measured queue-to-deliver p50 of 83 to 103 ms and throughput of 355 to 402 messages a second on the server machine. Those are transport measurements, not cross-host throughput or evidence of better task completion. A newer 29 September worker and fan-out benchmark documents API coordination under its own host and workload conditions; it also does not establish task correctness. Project and waitlist →
Task diagnostics are separate from those timings. An initial synthetic run retained 48 attempts with no accepted pair. A later run with a revised tool contract completed 48 more attempts on reused fixtures: a single agent 8 of 16, ordinary two-worker messaging 0 of 16, and Echo 0 of 16, with 13 merge conflicts retained across the run. The 48 attempts retain complete usage for 385 requests (432,571 tokens). The companion MAST script does not supply independent human or model annotations; the earlier independent two-pass coding and kappa 0.941 claims are withdrawn. These development diagnostics establish no coordination benefit, equivalence, fresh benchmark performance or isolated causal improvement. Read the coordination failure report →
Handhelm
Handhelm replays declared browser flows in an operator-controlled signed-in session, checks page state before acting, and pauses for human approval before consequential actions. It makes no model calls at runtime. Private development; no public download yet. Product design and status →
SmartChannels
A self-hosted assistant project spanning seven communication channels, with persistent state and retrieval. Local-first execution gives control over storage and deployment; privacy also depends on the configured model providers and external channel services. It revives the name of my 1998 company. Project details →
Abundera
A financial-advocacy product family with passkey authentication, Plaid integration, and vertical applications for billing, debt, insurance, and credit disputes. AI assists analysis and dispute-package drafting. The production estate demonstrates product and infrastructure integration; request volume across my sites is not a count of users or successful AI outcomes. Product family →
Mail triage agent
A tiered design that turns incoming mail into proposed calendar items. Rules set a priority floor, a local open-weight model decides whether an email needs action, and a hosted model sees only the cases that code escalates. Code checks every quoted date against the email, and a person accepts each proposal. I built the evaluation first. Read-only ingest has run, and the model tiers are not in service.
On 15 September 2026 I scored more than ten local configurations on a 186-email development set. A 35-billion-parameter open-weight teacher found 75 of 78 actionable emails, at 74% precision, and 71 of 72 on a separate 155-email held-out set, at 83%. A 9-billion-parameter student trained on its labels found 76 of 78 on development, against 65 of 78 for a prompted model of the same size, and 71 of 72 on held-out. Keyword rules found 46 of 72 on held-out, at 58% precision. One 20-billion-parameter candidate returned no valid output on any email.
Two model labelers and an adjudication step made the labels, with class agreement of 0.95 and 0.94. Shared model bias inflates that agreement, and no human spot-check is recorded. I revised the development labels and prompt after seeing model errors and chose the student’s training epoch on development, so held-out is the cleaner figure. One person’s mail does not transfer to other mailboxes. The mail, labels and weights stay private.
AI Research Ingestor
An internal workflow for collecting and organizing research papers and model releases into a searchable ledger. Summaries help identify work to investigate; conclusions still need checking against the source paper and, where practical, reproduction.
Acoustic & Voice Intelligence
Synthetic voice detection, conversation diarization, on-premise speech synthesis, and high-noise radio recognition.
Noise or Voice
An experimental acoustic research prototype using interpretable acoustic features to investigate synthetic speech. Its selected internal tests include substantial failures on an In-the-Wild subset. Historical-detector diagnostics retained failures: ODSS test 10/16 human false flags and 4/16 synthetic misses; ASVspoof 5 evaluation 22/48 human false flags and 8/48 synthetic misses (66/96 correct). Neither validates the current deployed detector. A separate developer-selected, known-label 168-trial paired diagnostic retained DSP 123/168 correct versus an author-checkpoint SSL baseline 140/168; differing audio windows and unresolved training exposure prevent a general model-superiority claim. Dated diagnostic and limitations Selected-168 DSP/SSL diagnostic artifact. Scores are uncalibrated and verdicts support further review; they do not authenticate a speaker. Streaming and telephony integrations remain proposed designs. Measured results, failure cases, and methodology →
Auralytic
A conversation-intelligence project for speaker diarization, sentiment, and review of recorded audio. It extends the voice-analytics thread of my earlier radio work. A published project description does not establish validated compliance decisions or measured accuracy. Project details →
Tacetto
An AI for places, designed as many small nodes on one local network, from single sensors to a room unit with a microphone array, speaker and camera, that share what they sense so any node can answer about what another noticed. The room unit’s software is written and installed on one board. The microphone array is not yet fitted, the speech services are not built, and the shared memory between nodes is a design, not yet an implementation. The design is meant to scale from a home to larger sites, which would bring workplace recording, biometric and safety rules that have not been studied here beyond a first reading. No accuracy, latency or barge-in result exists. Project page →
Voice & Speech Studio
An internal local speech pipeline using third-party Kokoro synthesis and faster-whisper recognition models. Local inference avoids sending audio to a hosted model API; deployment privacy and latency need checking against the actual configuration and workload. This is integration work, not a claim to have trained those models.
Tactical Speech & Codec Benchmarks
Earlier pilot: I retained six third-party speech models’ outputs on a small authored synthetic input set. Ungated non-speech decoding produced 24–55 words per minute overall; 160 was a condition-cell peak. Energy gating reduced overall output by 19.5–25.0%, leaving nonempty outputs on 108/144 non-speech evaluations. A separate five-utterance codec pilot covered 30 declared conditions; mean word error rate reached 129.4% at 0 dB. These diagnostics do not establish genuine-voice preservation, codec compliance or field-radio accuracy. Earlier report and limitations.
Full-study update — 11 October 2026 Pacific: I completed a separate registered evaluation of six third-party models: 5,760 E2, 1,620 E1 and 1,440 speech-cost records from 1,050 authored fixtures. These are repeated model/condition evaluations, not 8,820 independent samples.
For each model, the pipelines emitted normalized words on 71/71 distinct non-speech clips without gating and 50/71 with energy gating. The neural gate rejected all 71 distinct non-speech clips before ASR; this is a gating result, not model robustness on those inputs. On synthetic speech, pooled word error rose from 58.85% to 67.47% with neural gating; the paired bootstrap resamples 40 authored utterances, giving a 95% interval of 1.62–16.00 percentage points for the increase. Energy gating left the pooled point estimate unchanged. These fixture intervals establish no population inference.
Local snapshot replay reproduced the analysis and report, and all input-file hashes matched. This was verification within my workflow, not independent replication or peer review; it establishes neither field nor proprietary-model performance. Full-study results and limits · Codec study.
Verification, Education & Diligence
Code audits for M&A and browser-based neural network mechanics.
CodeProvenance.ai
A fixed-fee preflight for software and AI acquisitions. The target runs my scanner locally and sends encrypted metrics rather than source code. I interpret the findings and write a five-page assessment. Metrics support questions about dependencies, test debt, provenance, and operating costs; they cannot establish authorship, ownership, or team comprehension by themselves. Scanner and advisory offering →
LLM Lab
A classroom in the browser. Train a real character-level network live, follow tokenization through backpropagation, watch a KV cache work, and run a small language model in the tab. It teaches; it is not a product. Part of the Logical Art studio. Live at watchitlearn.com.
What I got wrong
A page that only shows wins is an advertisement. This section is the reason to trust the rest.
A number with nothing behind it
This site once printed an accuracy range for AMBIE. No measurement stood behind it. I took it down on 29 August 2026. The rule since then is that a number ships with a dataset, a noise condition, a metric and a date, or it does not ship. Every benchmark figure on this page follows that rule.
Reconsidering the original bets
The 9 September 2026 AMBIE evidence review challenged five planning assumptions. It combined published research, engineering economics and a small benchmark of third-party models. That review informed what I would pursue; it was not a controlled experiment that proved four universal propositions false.
- Size target reconsidered: off-the-shelf models made compression a less compelling starting point. The review estimated Moonshine Tiny at about 27 MB using one byte per parameter; it did not produce or measure that int8 export or qualify it on a phone.
- Architecture reconsidered: the review challenged choosing a Conformer as the parent. No measured WER for AMBIE’s own Conformer was available to establish a head-to-head result.
- Economics reconsidered: the self-hosting case was unattractive under the scale and cost assumptions reviewed. That is a planning assessment, not a general result against owning models.
- Specialization reconsidered: I questioned a separate acoustic model per industry. The benchmark did not compare trained industry experts against a general model.
- Privacy retained as a hypothesis: on-device processing remained a reason to investigate the project, with deployment feasibility still requiring evidence.
What the benchmark showed
The measured evidence was narrower: third-party recognizers on the sampled LibriSpeech recordings and synthetic babble conditions. It motivated testing noise robustness, not a claim that model compression is solved or that no small recognizer can handle noise.
| Model | Size | No added noise | 10 dB babble | 5 dB babble |
|---|---|---|---|---|
| moonshine-tiny | ~27 MB | 11.3% | 36.2% | 64.5% |
| whisper-tiny.en | ~38 MB | 14.3% | 35.5% | 76.8% |
| moonshine-base | ~62 MB | 8.3% | 23.8% | 44.2% |
| whisper-small.en | ~242 MB | 7.2% | 16.5% | 33.6% |
How to read it. Measured on 9 September 2026 with LibriSpeech test-other, the harder split, using about 150 utterances picked with a fixed seed and scored with Whisper’s own text normaliser, on 4 CPU threads. The noise is synthetic babble, six overlapping talkers added at the signal-to-noise ratio shown. Sizes are estimates for an 8-bit build. These are other people’s models. None of them is an AMBIE model. The harness is part of TuneHarness above.
A later run on a stricter protocol found problems in this harness: the babble was drawn from the pool of scored utterances, model revisions were not pinned, and the text normaliser could fall back silently. They do not show these figures are wrong, so read the table as a first, directional measurement. On a different 12-speaker sample from the cleaner split, the same small model scored 107.61% at 5 dB, and insertions dominated the six-speaker subset. The later run and its limits.
Two checks kept me honest. I ran whisper-tiny.en on the unmodified split and got 14.3%, against a published 14.6%, which supports a harness sanity check. An earlier sample I had drawn by taking the first utterances from a file ordered by speaker flattered the same model to 11.7%, so I threw it away.
Then the finding. At 5 dB the best model in this sample, at an estimated 242 MB, has 33.6% word-error rate. Larger models in this table reduce the error, but the result still falls short of the task I had in mind. This led me to prioritize noise robustness over compression; it does not establish a limit for every model in this size class. Babble is only directionally right for a loud room, and I have not yet run AMBIE’s own models through this harness. That is the next thing to do.
A judge that changed its mind
I use model judges as deploy gates for content. On 31 July 2026 a trivia pack passed a small hosted judge model once and failed it twice, on identical text, and a temperature of zero with a fixed seed reduced the flapping without removing it. Across consecutive runs three packs changed verdict without voting and none did with three-sample majority voting. Separately, the judge blocked six packs for errors it could not know about, because the events were recent. Giving it the current date, permission to abstain and the cited sources cleared four. These are single-session observations, not rates. I have no measure of how often the judge agrees with a human reader, so its verdict blocks a deploy and I do not call it accurate.
A prune that broke the tools
On 11 July 2026 a model contest pruned 13 local models. Two coding tools still named models that were gone, and they stayed broken for about three weeks. On 27 August I found another internal service with 8 of its 10 model references dead, including its default, and a router set to a host with no model server. The cause each time was a model name pinned inside a tool. Tools now ask for a role, and one file maps each role to a fallback chain.
Independent AI research
I am an independent AI systems researcher focused on reliable machine agency, grounded in decades of hands-on systems engineering. How can models become systems that perceive, reason, act, coordinate and recover under real constraints, with verification and measured cost? My work connects acoustic intelligence, tool use, verifier reliability, agent coordination and deployment infrastructure. Each project retains its own stage, method and evidence.
Research projects and experiments
- Verifier reliability — Research proposal · 120-task engineering pilot complete; H1 study unrun. Under a fixed total inference budget and task-family shift, does modeling cross-verifier error dependence improve hidden-test success when selecting candidates and deciding when to stop, beyond strong single-judge, fixed-consensus and matched-retention filter baselines? The 120-task HumanEval engineering pilot is separate from the H1 study: first-candidate success was 109/120 and the higher-cap result was 91.39%, with a descriptive interval including zero gain and only one task of candidate-pool headroom. An earlier 504-task/4,032-record companion is a deterministic synthetic simulation, not a benchmark evaluation; its covariance and policy outputs are not model results, its splits share the same 14 domains, and its stated early-stop threshold is unreachable. The 576-slot H1 collection remains unrun. Read the evidence and method. Separate 120-task HumanEval pilot: methods, results and limits.
- AMBIE — Model research · Bounded synthetic-input speech diagnostics recorded. Which acoustic and deployment choices make noisy-speech recognition more reliable under radio and environmental constraints? E1 retains 27 authored synthetic input clips across 6 models and two decoding regimes (324 output evaluations). Overall per-model ungated rates are 24–55 words per minute; 160 is a condition-cell peak. The gate reduces overall WPM by 19.5–25.0%, leaving hallucinations on 108/144 non-speech evaluations. E2 retains 5 authored synthesized utterances across 30 declared conditions and 6 models (900 model-by-condition records). Mean WER across recorded models and paths is 129.4% at 0 dB. These are repeated views of a small synthetic input set, not independent samples or authentic Coast Guard recordings. No population confidence interval, genuine-voice preservation, codec-compliance guarantee, operational-radio accuracy or proprietary AMBIE model accuracy is established. Read the evidence and method.
- ThermoCog — Whitepaper and design · Implementation pending. What does machine intelligence cost in energy, and which proposed computational mechanisms can be tested against physical constraints? No software or hardware implementation and no measured results. Proposed energy figures are not demonstrated performance.
- Noise or Voice — Experimental acoustic research · DSP and SSL diagnostics completed. Can interpretable acoustic features help distinguish synthetic speech across generators, speakers and recording conditions? A developer-selected 168-trial comparison favored a third-party self-supervised baseline over the DSP detector, and the exposed, related-source sample prevents general superiority claims. The full diagnostics and limits are under Acoustic & Voice Intelligence. Read the evidence and method. Selected-168 DSP/SSL diagnostic artifact.
- TuneHarness — Operating research infrastructure · Documented ML experiments. How can specialist-model training, adaptation and evaluation improve task outcomes within measured compute budgets? Completed specialist-model comparisons measured task-specific differences and failures; the native, rescored and 160-case results are set out under Compute & Training Substrate. These exposed, task-specific diagnostics and unresolved training lineage establish neither causal training benefit nor general superiority, frontier transfer or production readiness. Read the evidence and method.
Next step
If one of these sounds like your situation, read the services page or book a Reality Check.
Read or download my research and systems portfolio: contributions, completed experiments, artifacts and next studies in one dated document.
Sources
- Netflix Prize dates and team counts: Wikipedia, checked 21 September 2026.
- Netflix on not deploying the winning blend: Netflix Tech Blog, April 2012.
- Hinton et al., Deep Neural Networks for Acoustic Modeling in Speech Recognition, IEEE Signal Processing Magazine, November 2012.
- van den Oord et al., WaveNet, 12 September 2016.
- Radford et al., Whisper, 6 December 2022.
- Panayotov et al., LibriSpeech: An ASR corpus based on public domain audio books, ICASSP 2015.
- The benchmark is my own, described in full above.