How to Build an ESP32 Offline Speech Recognition

User Avatar Anand D
Author
Share
ESP32 Offline Speech Recognition

Speech recognition models like OpenAI’s Whisper offer high levels of accuracy, but their base model itself consumes up to 140MB and running them on a low-end device like an ESP32 is difficult. In this tutorial, we are going to run Cactus Compute’s Whistle Speech-to-Text Model launched on 2nd of October 2026. The actual size of the model is 16.9MB, which won't fit into an ESP32 with 16MB Flash. In this tutorial, we will compress the 16.9MB Cactus Whistle down to 14.52MB, so that it fits into the ESP32 Flash memory. ESP32 offline speech recognition is now practical without any cloud service.

This setup reads audio via an INMP441 I²S digital microphone, records audio as we hold down the push button, stops recording on releasing the button and triggers live actions on an onboard RGB LED that's connected to GPIO pin 48, all this 100% offline with zero cloud API dependencies. Several other cloud providers offer STT services as well. We have already covered one such tutorial using Wit.ai with minimal latency. If interested, you can check out our ESP32 Speech-to-Text using Wit tutorial as well. Now, let's check out the components required to build this project.

Quick answer: Yes, an ESP32 can do offline speech recognition. An ESP32-S3 with 16 MB flash and 8 MB PSRAM can run a four-layer version of Cactus Whistle (about 14.5 MB) using the whistle-esp32 firmware from GitHub. It transcribes up to 30 seconds of speech in seven languages on the chip, taking roughly 6 seconds for a short command in our test.

What Is Cactus Whistle and How Is It Compatible with ESP32?

Cactus Whistle is a small speech recognition model from Cactus Compute, designed to be run on processors in devices such as smartphones, smartwatches, and robotics. It converts audio in a 16 kHz mono format with a maximum length of 30 seconds, detects the language of the audio signal automatically, and provides word-level timestamps. Cactus does not sell an ESP32 version, so the whistle-esp32 was created as the author's own independent implementation of the engine.

How the 16.9 MB Model Fits into 16 MB of Flash

» Layered Decoder: The teaching of the receiver was done in a way that every level from 2 and above works independently. This means that the interface can be removed without the necessity to repeat the training process.
» Layer Removal: The whistle_strip.py script saves only the first four layers, thus decreasing the size from 16.9 MB to around 14.52 MB.
» Complete Flash Partition: The partition starts at the offset of 0x210000 and finishes at the end of 16 MB, leaving less than 100 KB to spare.
» INT8 Acceleration: For computation purposes, matrix multiplication can be performed as 8-bit integer operations on the ESP32-S3 SPE vector unit, with 16 multiplications and additions being performed simultaneously.
» 8 MB PSRAM (pseudo-static RAM): The octal PSRAM acts as an audio buffer and working tensor which wouldn’t fit in the SRAM installed in the chip.

Components Required for ESP32 Offline Speech Recognition

Below are the components required to build this project.

ESP32-S3 board, INMP441 I2S microphone, push button, breadboard and jumper wires for ESP32 offline speech recognition Components used for the ESP32-S3 offline speech recognition project

First, we have the ESP32-S3 board with 16MB of Flash and 8MB of PSRAM. This specific module is the ESP32-S3 CAM Module that comes with a camera. But we are not utilising the camera part in this project. Then we have the INMP441 microphone to record audio. Hold the push button to record the audio and release it to stop recording. We will be assembling the whole circuit on a breadboard, as it is the easiest way to get things started. Jumper wires will be used to make all the connections. Below is the list of components, neatly provided in a table for easy reference.

S.noItem                      Description
1ESP32-S3To be used as the main microcontroller
2INMP441 I2S MICTo record the audio
3Push ButtonTo trigger audio recording and to stop recording
4BreadboardWhere we will be setting up the circuit
5Jumper WiresTo connect all the parts as per the circuit diagram

Circuit Diagram for ESP32-S3 Offline Speech Recognition

The following is the circuit diagram for this project.

Circuit Diagram for ESP32 Speech Recognition

As in the circuit diagram, there are mainly 3 components. One is the ESP32-S3 Board itself, which has 16MB of Flash and 8MB of PSRAM, where we will be running the STT model. Then we have the INMP441 I2S microphone to capture the audio and then a push button pressing on which, it starts recording and releasing which it stops recording. We will be controlling the onboard RGB LED, which is connected to GPIO48 of the ESP32, using this setup, so we don’t need to connect any external output devices. But the code can be completely modified to meet the project requirements. Along with the onboard RGB LED response, we will have a response in the TUI(Terminal User Interface) Monitor as well.

Hardware Connections: INMP441 and Push Button to ESP32-S3

Connect the INMP441 microphone and push button according to the following pinout. If you’re someone who wants offline speech recognition but doesn't want to use an ESP32 for that, we have a dedicated tutorial on Low-Cost Offline Voice Recognition using a low-cost module. Now, let's take a look at the connection required to set this up.

Component PinESP32-S3 Board PinDescription
INMP441 VDD3V3Do NOT connect to 5V
INMP441 GNDGNDGround
INMP441 L/RGNDSelects left channel
INMP441 SCKGPIO 4I²S bit clock
INMP441 WSGPIO 5I²S word select
INMP441 SDGPIO 6I²S serial data
Push Button Pin 1GPIO 7Uses internal pull-up
Push Button Pin 2GNDGround
WS2812 RGB LEDGPIO 48Onboard LED (Default)

How to Set Up ESP32 Offline Speech Recognition from GitHub

Step 1⇒ Install ESP-IDF 5.5 or Later

We are going to use ESP-IDF. Download and install it from the official website. Here, in this tutorial, ESP-IDF Version 5.5.5 is being used. Version 5.5 or above is required. Once installed, open it from the Start menu and go to the desired directory where we plan to clone the repo. The following is the directory where we will be working. Let's CD into it.

cd $HOME\Documents\Projects

Step 2⇒ Clone the whistle-esp32 GitHub Repository

Clone the Whistle-ESP32 repository and CD into the project directory using the following commands.

git clone https://github.com/andrisgauracs/whistle-esp32.git
cd whistle-esp32

Step 3⇒ Create a Python Virtual Environment

Let's create a virtual environment named “venv” so that we can install all the required packages in that venv.

python -m venv .venv

Activate the newly created venv using the following command.

.\.venv\Scripts\Activate.ps1

If it throws an error, run this command to unblock scripts for the current PowerShell session.

Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope Process

Install the required tools.

pip install numpy rich pyserial huggingface_hub 
pip install esptool

Now, let's download & Strip the Whistle Model. Make sure we are in the following directory:
C:\Users\Pavilion\Documents\Projects\whistle-esp32

Step 4⇒ Download and Strip the Whistle Model

mkdir -Force model\whistle

Download the model from Hugging Face: 

python -c "from huggingface_hub import hf_hub_download; import shutil; shutil.copy(hf_hub_download('Cactus-Compute/whistle','whistle.cact'), 'model/whistle/whistle.cact')"

Strip the model down to 4 layers (14.5 MB):

python tools\whistle_strip.py model\whistle\whistle.cact 4 model\whistle\whistle_d4.cact
Cactus Whistle model size of 16.9 MB before and 14.52 MB after stripping to four decoder layers Whistle model size before (16.9 MB) and after (14.52 MB) stripping

In the image above, we can clearly see the actual downloaded model size, which is 16.9MB and the size after stripping down the model, which is 14.52MB.

Step 5⇒ Export the ESP-IDF Environment

Make sure we are in the following directory :
C:\Users\Pavilion\Documents\Projects\whistle-esp32\esp32\whistle_demo
We don't want to activate the venv as of now, as the project's local virtual environment (.venv) may conflict with ESP-IDF's own dedicated Python environment. If it's activated by any chance, use the following command to deactivate it.

deactivate

Now, the default installation directory for ESP-IDF 5.5 on Windows is usually inside the user directory or under C:\Espressif, rather than C:\esp-idf. We need to find that out.
Run this command in PowerShell to automatically find where export.ps1 is located on your machine:

Get-ChildItem -Path "C:\Espressif", "C:\Users\$env:USERNAME" -Filter "export.ps1" -Recurse -ErrorAction SilentlyContinue | Select-Object -ExpandProperty FullName

How to Export ESP-IDF Based on the Location

Once the script outputs the path, run it with a dot operator at the beginning. For example:

If it is inside C:\Espressif:

. C:\Espressif\frameworks\esp-idf-v5.5.5\export.ps1

Or if it is in your User directory:

. C:\Users\Pavilion\esp\esp-idf\export.ps1

Step 6⇒ Build and Flash the Firmware

Now, let's proceed with the set-target and build commands:

idf.py set-target esp32s3
idf.py build

Flash the firmware using the following command. Replace the “x” with the actual COM Port of your ESP32. It can be seen in Device Manager on Windows.

idf.py -p COMx flash

Step 7⇒ Flash the Whistle Model to Partition 0x210000

Go back to the project root
cd ..\..
Reactivate the project venv
.\.venv\Scripts\Activate.ps1
Flash the 14.52MB model binary to partition 0x210000
esptool.py --chip esp32s3 -p COM6 -b 921600 write_flash 0x210000 model\whistle\whistle_d4.cact

esptool output confirming the Whistle speech model was flashed to ESP32-S3 at offset 0x210000 esptool confirms the Whistle model has been written to the ESP32-S3 flash

The above message will show up once the model is successfully flashed onto our ESP32.

Run the TUI(Terminal User Interface) Monitor

Once the model write reaches 100%, launch the TUI monitor to test your board. 
At this point, we need to connect the ESP32-S3 board to its 2nd Type-C port labelled as USB-OTG instead of USB-UART. This is because the USB-UART port connects to the external CP2102 or CH340 USB-to-Serial bridge chip attached to the hardware UART0 pins (TX0/RX0).
Since the whistle-esp32 repository connects directly to the native USB peripheral inside the ESP32-S3 silicon, we need to connect it to the USB-OTG port.

Before running the following command, make sure to replace the “x” with the actual COM Port of the ESP32.

python tools\needle_tui.py --whistle --serial COMx
Needle TUI monitor showing ESP32-S3 connected over USB-OTG for offline speech recognition The TUI monitor showing the ESP32-S3 connected on COM8

The above will be the TUI Monitor interface that will pop up immediately when we run the above command. It even shows ESP32-S3 is connected to COM8. Once we are at this level, we can proceed to the next steps and see the results.

Testing Results: Offline Voice Commands on the ESP32-S3

Below are some of the commands that can make some actions with this setup. The firmware transcribes any speech but only acts on the colour commands (red, green, blue, yellow, purple, white) plus flash and blink.  As of now, we don't have too many I/O devices connected to it and programmed, so below are some valid commands. It even converts any random text that is spoken to it and displays it in the TUI Monitor.

SNoCommandAction
1Change colour to RedRed light flashes
2Flash green light for 5 secondsGreen colour flashes for 5 seconds
3Blink purple every half secondpurple blinks for half a second
4PurplePurple colour flashes

What the RGB LED Status Means

StageTUI StatusLED Indication
Button heldRecordingSolid white
Button releasedTranscribingSlow white blink
No speech detectedEmpty resultTwo red blinks
Command recognisedTranscript and action shownLED performs the command
TUI monitor showing Recording status while the ESP32-S3 captures audio from the INMP441 microphone Recording: audio is captured while the button is held

When the push button is pressed, the TUI Monitor shows the status as “Recording” and starts recording the audio that is spoken to the INMP441 Mic. The recording keeps going until the button is released. At this point, the onboard RGB LED shows white.

TUI monitor showing Transcribing status as the ESP32-S3 converts speech to text offline Transcribing: the ESP32-S3 converts the recorded speech to text on-chip

Once the button is released, the "Recording" status changes to “Transcribing”, meaning the spoken audio is being transcribed to text. This process typically takes 6 seconds. At this point, the onboard RGB LED blinks white slowly, indicating that transcription is going on. If no text is detected, it immediately blinks Red colour twice. The possible reason for this may be that the MIC is not good, or the MIC connection is wrong.

TUI monitor displaying the transcribed voice command and RGB LED action on the ESP32-S3 Result: the transcript appears and the LED turns red

Once the transcription is done, we can see the output in the TUI Monitor. It displays the response to the action. For this command, the onboard RGB LED turns red. It behaves as per the command provided. It's also visible that this action took less than 6 seconds. 

Demo: Run Offline Voice Recognition on a $10 ESP32-S3

Watch the ESP32-S3 turn spoken commands into RGB LED actions in real time, with no Wi-Fi or cloud connection. Hold the push button, say a command like “change colour to red”, and the chip records, transcribes and acts on it in about six seconds. The video shows the full push-to-talk flow, from recording to transcribing to the LED changing colour, all running on the stripped 14.5 MB Cactus Whistle model.

Conclusion

The successful deployment of Cactus Compute’s Whistle model on the ESP32-S3 demonstrates that modern, Transformer-based speech-to-text is no longer locked behind cloud APIs or high-power Linux single-board computers. By combining structural layer pruning with INT8 vector quantisation, a $10 microcontroller can run multilingual, offline speech recognition locally. Now, if the transcribed text needs to be converted into audio, we have a dedicated tutorial that teaches how to build an ESP32 Text-to-Speech offline system so that this can also be integrated into this project. Text-to-speech can also be done using our ESP32-C3 Text-to-Speech Using Wit tutorial as well. For more ESP32-based projects, check out our collection of ESP32 projects.

Official Andrisgauracs GitHub Repository

The whistle-esp32 repository by Andrisgauracs has everything you need to run Cactus Whistle on an ESP32-S3: the ESP-IDF firmware, the whistle_strip.py model-stripping script and the needle_tui.py terminal monitor. Clone it, follow the README, and you can have offline speech recognition working on a 16 MB ESP32-S3 board in under an hour.

ESP32 Offline Speech Recognition GitHub RepoESP32 Offline Speech Recognition Download Zip

This collection explores the development of smart, offline embedded systems using Arduino and ESP32 microcontrollers. It covers accessing Wikipedia without an internet connection, converting text into speech using Arduino, and implementing offline voice recognition using Edge Impulse. Together, these projects demonstrate how microcontrollers can enable intelligent, voice-controlled, and speech-based applications without relying on continuous internet connectivity.

DIY Raspberry Pi Pico Text-to-Speech Using Wit.ai

DIY Raspberry Pi Pico Text-to-Speech Using Wit.ai

In Raspberry Pi Pico TTS setups, the Pico sends text to an online service where speech is generated and returned as audio. The device then plays the sound through a speaker, allowing clear speech output without overloading the hardware or increasing system complexity.

Arduino based Text to Speech (TTS) Converter

Arduino-based Text-to-Speech (TTS) Converter

In this tutorial, we will learn how to make a text-to-speech converter using Arduino. This step involves the conversion of the symbols, numbers and abbreviations into words that can be read by the machine.

ESP32 Offline Voice Recognition Using Edge Impulse

ESP32 Offline Voice Recognition Using Edge Impulse

We’ll build a mini yet expandable ESP32 voice assistant using Edge Impulse, with the ESP32 as the main microcontroller. The goal of ESP32 offline speech recognition is to keep it simple while still leaving room for future upgrades.

Have any question related to this Article?

Add a comment

No account needed. Your comment appears once a moderator has reviewed it.

Never shown publicly.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.