Speech recognition on a microcontroller usually means a short list of fixed commands. Boumedine Billal has gone much further: his open-source project runs a full speech-to-text model live on an ESP32-P4, a chip that costs about $5. You talk into the board's microphone and any English sentence appears on its screen as you speak, with no Wi-Fi, no cloud and no neural accelerator. On the standard LibriSpeech test-clean set it gets about 4 words wrong in 100 (4.34% word error rate), close to the 3.94% the same model scores on a PC.
Why this should not fit
The model is Kroko by Banafo, a 65.7-million-parameter streaming Zipformer2 transducer. As released it is 71 MB, and even in 8-bit form it would not fit in the ESP32-P4's 32 MB of PSRAM. That rules out ESP-DL and TensorFlow Lite Micro, whose smallest weights are 8 bits, and Espressif's own ESP-SR only recognises a fixed list of up to 200 commands.
So he wrote his own engine in C. The weights are stored in 2 and 4 bits, with the bit width picked per matrix by how sensitive it is, which brings them down to 26.65 MB without retraining. They load from a microSD card in 2.9 seconds at boot, because they still do not fit in the 16 MB flash. The maths is integer everywhere, and the hot loops are hand-written in assembly for PIE, the P4's SIMD extension that does 16 int8 multiply-adds in one instruction.
The first port took 1.3 seconds to process 1.28 seconds of audio, which is slower than real time. After the integer and PIE work, each 1.28-second chunk takes about 0.9 seconds live on the newer ESP32-P4X (chip revision v3, 400 MHz), screen drawing included. That is faster than you speak, so it keeps up however long you talk, with fixed memory. A word is final about 2 seconds after you say it.
Bit-exact with the PC
One detail engineers will appreciate: the C maths libraries on the PC and on the chip gave slightly different results for functions like log, exp and cos. He replaced them with his own deterministic versions, so the chip now computes exactly what the PC build computes, bit for bit. That is why the accuracy numbers measured on the full test sets on a PC are also the chip's numbers.
Key details
- Chip: ESP32-P4X (revision v3), two RISC-V cores at 400 MHz, 32 MB PSRAM, no NPU
- Board used: Espressif ESP32-P4X-Function-EV-Board (ES8311 mic codec, 1024x600 screen, microSD)
- Model: Kroko streaming Zipformer2, 65.7M parameters, 2/4-bit weights, 26.65 MB
- Accuracy: 4.34% WER on LibriSpeech test-clean, 10.26% on test-other
- Speed: each 1.28 s of audio in about 0.9 s live; words final about 2 s after speaking
- Extras: custom hotwords from a text file, and six offline voice-control demos (robot vacuum, smart home, car cabin and more) that turn spoken orders into actions on the chip without an LLM
- Code: esp32-pie-asr on GitHub (Apache-2.0), flashable from the browser
Limits
It needs the v3 chip (ESP32-P4X); older v1.x chips use different PIE instruction encodings, so their binaries are not compatible yet. It is English only, the accuracy figures are for clean read speech, noisy rooms and strong accents are still hard, and power consumption has not been measured.
Our take
This is one of the best examples we have seen of what careful low-level work can squeeze out of a cheap microcontroller. The voice-control demos are the practical part: a robot, appliance or car dashboard that understands full sentences offline, without sending audio anywhere, on a chip that costs about $5.
Image credit: Boumedine Billal, frame from the YouTube video, cropped and resized.
About the author: Aswinth Raj is an electronics engineer and the editor of CircuitDigest, where he has published more than 150 hands-on projects and tutorials across Arduino, PIC, ESP32, Raspberry Pi and EV battery systems. He selects the stories on this page for what engineers and makers can actually learn or build from. Connect on LinkedIn
Add a comment
No account needed. Your comment appears once a moderator has reviewed it.