Talking My Blog Back to Life

There's a huge gap between my last blog post and this one, which is very telling. It sort of shows the priority for me. As a dyslexic, I struggle with writing, it's a difficult job, and I always tend to put it off for too long.

I could get an agent to actually write these posts for me, but it's my blog and I want it in my voice. I want it to be me writing these things. So there is an in-between, a kind of happy medium, where I build an assistant who can take dictation and help me present what I'm actually saying as text. It's quicker, it's easier, and it's a lot less work than the hours it would take for me to write this by hand.

To get me to the point I wanted to be, I started building my own private voice server, and made some good headway on it. But as interesting as the technical side of it was, the need to actually have a finished product I could use outweighed the need to build the project myself. So I've used what's detailed here as a short circuit to get me to the end product, and that way I can go back and revisit my own project at a later date.

I'm still very much focused on local AI, keeping my AI stack on my own machine. But for a lot of the work I'm doing, both professionally and personally, I'm using Claude Code, so it seems like the best place for an in-between stage. Adding voice interaction with the computer, going through Claude Code while I sort out the details of my private voice server, is a super power. It definitely stands as a good intermediate between my own software stack and this hybrid, where it's part local and part on Claude's cloud.

ℹ️ The stack

Everything runs locally on one machine, with nothing sent to the cloud for speech. VoiceMode is a plugin that gives Claude Code a converse tool: Claude speaks a reply, then listens for mine.

• VoiceMode (v8.12.0) is the glue: a Claude Code plugin and MCP server that records, sends audio off to be transcribed, and plays the reply.
• Whisper, run as the whisper.cpp server, turns my speech into text.
• Kokoro, an 82-million-parameter text-to-speech model served by Kokoro-FastAPI, turns Claude's reply back into speech. It has British voices, and I picked Lily.
• Claude Code is the agent I'm talking to.
• PipeWire is the Linux sound system that plays the audio out over HDMI.

The first session gave me a good baseline. What was coming back was useful and I could see it definitely had potential, but there was a bit of frustration. It cut bits out and it missed the start of conversations. At one point it heard me as Welsh, interesting as I am a Welshman, although I don't actually speak the language. But this certainly gave me the idea that this was going to work as a stack.

When things were slow, my first thought was that the stack was picking the external GPU. I've got this GPU harnessed to the machine for some distributed workloads I was trying in the past, and unfortunately I found it's actually slower than the machine it's connected to. As this was my last bit of tinkering, that was the first thing that popped into my head. I didn't realise at the time that it was actually just running on the CPU and had a hard cap there.

With all of these issues, I was starting to think I'd have to go back to writing my code and finish the actual project, which is fine, it just would have taken me longer. I want the utility more than I wanted the experience of writing the code myself, so when I managed to sort out all of these issues, it gave me the platform that I'm using right now.

🔵 What was actually wrong

None of it was the GPU. Kokoro never uses one here, and it doesn't need to.

Four separate things were wrong:

1. Both speakers were muted. The speech played perfectly, into silence.
2. Kokoro was capped to one CPU core. The service file VoiceMode installs includes CPUQuota=100%. PyTorch starts 16 threads, they used up that single-core allowance almost immediately, and then all 16 sat waiting for the next slice. Run directly, Kokoro generated speech 17.7 times faster than real time. Inside the capped service it managed 0.7 times, slower than it could be spoken. Removing the cap took a long paragraph from 15.7 seconds to 0.78.
3. The same cap caused the gaps mid-sentence. VoiceMode starts playing the first chunk while the next one is still being made. Chunks holding about 10 seconds of speech were arriving 14 to 19 seconds apart, so playback ran dry and went quiet until the next one turned up.
4. The HDMI output was falling asleep. Linux suspends an idle audio output, and HDMI takes about a second to wake, so the first second of every reply was lost. One rule tells it to stay awake.

And the Welsh? Whisper guesses the language of every clip, and the small model guessed wrong. It is now told to expect English.

All of this did make me think of somebody approaching this who is not technically minded, and who may not have questioned the issues they're finding. They could have thought that this was just a faulty stack and not really worth pursuing. It didn't work off the shelf the way I wanted it to, and it did need tinkering, and people may very well be put off by these hurdles.

I've got a bad habit of reinventing the wheel. I get an idea into my head and I want to build it from the ground up, rather than looking for what's already out there. But there were already systems out there that I could plug together with Claude. There are still things that this can't do that I really want: a voice satellite I can give a wake word to, that then sparks up, takes my thoughts down and puts them into notes somewhere for me. So in that regard, my private voice server is still in play as a project. I just feel less urgency about it now, because I have this setup and it works and it's doing what I wanted it to.

My advice: if you need this kind of tool to help you do the things that you want to do, then have a look around and see what's out there. Don't give up at the first hurdle. Have a dig through and see if you can sort out what the problems actually are. A lot of these things are open source projects that people have put their time and passion into, but they may not work exactly the way you want them to, or on every machine.

These tools are extremely powerful, although we are still in the early stages, so you may not find a polished solution for what you want. So go out, have a look at what's available to you, and build something that works for you.

ℹ️ Addendum: installing it

This is what worked on Ubuntu Linux with PipeWire, on 2026-09-26. Whisper and Kokoro both run on the CPU.

1. Install the VoiceMode plugin in Claude Code. Add the marketplace first, or the install fails with "not found in marketplace".

claude plugin marketplace add mbailey/voicemode
claude plugin install voicemode@voicemode

2. Install the voicemode command. It's a separate Python package; the plugin doesn't include it.

uv tool install voice-mode

3. Install the system audio libraries in a normal terminal, since it needs sudo. Skip the pulseaudio package some guides list if you're on PipeWire.

sudo apt install -y libasound2-dev libasound2-plugins libportaudio2 portaudio19-dev pulseaudio-utils libsdl2-dev

4. Install Whisper and Kokoro as background services. Kokoro's first start takes about five minutes while it downloads its libraries.

voicemode service install whisper
voicemode service install kokoro

5. Restart Claude Code, then run /voicemode:converse.

Then apply the fixes, because the defaults gave me every problem in this post:

• Remove Kokoro's one-core cap. Create ~/.config/systemd/user/voicemode-kokoro.service.d/cpu.conf containing:

  [Service]
  CPUQuota=

  Then run:

  systemctl --user daemon-reload
  systemctl --user restart voicemode-kokoro

• Stop the HDMI output sleeping. Create ~/.config/wireplumber/wireplumber.conf.d/51-hdmi-no-suspend.conf with a monitor.alsa.rules entry that matches your HDMI output's node.name (find it with wpctl status) and sets session.suspend-timeout-seconds = 0. Then run systemctl --user restart wireplumber.
• Tell Whisper it's English, and give yourself time to think. Add these to ~/.voicemode/voicemode.env, then reconnect VoiceMode with /mcp in Claude Code:

  VOICEMODE_WHISPER_LANGUAGE=en
  VOICEMODE_SILENCE_THRESHOLD_MS=3000
  VOICEMODE_INITIAL_SILENCE_GRACE_PERIOD=5.0

• Check that nothing is muted: wpctl status.