Fish-Following Robot Using Raspberry Pi 5 and OpenCV

Share
Fish Following Robot Using Raspberry Pi

Most robots move because someone presses a button or a program tells them where to go. This fish following robot takes its input from a fish. A red Betta swims in a glass tank on top of a four-wheeled rover; a camera watches the tank from above, and the rover moves based on the fish's movements. The idea came from a reel of a fish-driven vehicle, and we wanted to build our own version.

In this project, we built a fish-following robot on an aluminium frame. A Raspberry Pi 5 with a Pi Camera Module 3 looks down into the tank from a mast, finds the fish with OpenCV, and determines which part of the frame it is in. It then sends a movement command over USB to a NodeMCU, which drives four 12V DC motors through two L298N motor drivers. The rover drives, and the final move, stop and wait behaviour ran with the fish on board. However, the final OpenCV tracking and NodeMCU motor code aren't in the repository yet, so this guide provides the full published YOLO vision pipeline and describes the rest of the fish-following rover. You can also check out similar robotics projects and Raspberry Pi projects done previously.

What is a Fish Following Robot?

A fish-following robot is a four-wheel rover where the position of the fish in the camera frame is the input, in place of a joystick or a remote. When the fish swims into one corner of the tank, the rover moves toward that corner, and when it comes back to the centre, the rover stops. Keep the tank small, as every extra litre of water is weight for the frame and motors to carry, which is also why we chose a fighter fish. The Raspberry Pi ball tracking robot works on a similar idea, following a ball by its colour. If interested, do check it out.

How Does the Fish-Following Robot Work?

The work is split between two boards. The Raspberry Pi 5 reads frames from the Pi Camera Module 3, finds the fish with an OpenCV HSV colour mask and checks which part of the frame its centre falls in. The NodeMCU receives commands such as forward, left or stop from the Pi over a USB cable and switches the two L298N motor drivers, which run the four 12V Johnson DC motors. The Pi also serves a dashboard on port 5000, so the camera view and the robot status can be watched from a laptop on the same Wi-Fi network. Below is the block diagram of the Fish Following Robot.

Block Diagram of AquaNav Fish Following Rover

The repository does not show how the Raspberry Pi itself is powered, so the diagram leaves that out. The Pi-to-NodeMCU link is a plain USB serial connection, the same kind used in our Arduino and Raspberry Pi serial communication tutorial. Below is the flowchart of the movement logic.

The camera frame is split into four zones: top-left, top-right, bottom-left and bottom-right, with a centre zone in the middle. If the fish is in the centre, or not found at all, the rover stays still. If it is in one of the four zones, the rover moves according to that zone for 5 to 8 seconds, stops, and waits for about 10 seconds before it checks again. When the rover moved for as long as the fish stayed outside the centre, it kept on driving, so we settled on the timed cycle. Which movement each zone triggers is set in the final tracking code, which is not in the repository yet. TODO: zone-to-movement mapping.

Components Required for the Fish Following Robot

Below is the list of components required to build this project

Components Required for AquaNav Fish Following Rover
S.NoComponentDescriptionQuantity
1. Raspberry Pi 5Fish tracking and dashboard1
2.Raspberry Pi Camera Module 3Watches the tank from the mast1
3.NodeMCU ESP8266Motor controller, on USB to the Pi 1
4.L298N Motor DriverDual H-bridge, two motors each 2
5.12V Johnson DC Geared Motor
TODO: RPM
Drives one wheel each4
6.Wheel TODO: diameterSturdier than 60 mm Mecanum wheels, which
could not carry the load
4
7.Battery TODO: final pack and
Rocker Switch
Motor power1 each
8.Custom Glass AquariumHolds the fish1
9.Aluminium FrameRover base 1
10.3D-Printed ShaftJoins each wheel to its motor 4
11.Ultrasonic Sensor TODO: modelTested on its own; not yet in the final circuit1
12.Perfboard and Wires Final soldered circuitAs
required
13.Red Betta (Fighter Fish)Decides where the rover goes1

The Raspberry Pi 5 runs the camera, the tracking and the dashboard, so the NodeMCU only has to switch the motor drivers. We started with an ESP32-S3 board and moved to the NodeMCU after the ESP32's USB port got damaged. The L298N boards are the common red modules with a heatsink. To learn how the L298N works, refer to our L298N motor driver with Arduino tutorial.

Wiring Diagram of the Fish Following Robot Using Raspberry Pi

Below is the wiring diagram for the Fish Following Robot.

Wiring Diagram of AquaNav Fish Following Rover

The table below lists every connection. The NodeMCU to L298N pin map is not in the repository, so the diagram shows those lines without GPIO numbers. The battery feeding both L298N boards through the switch with a common ground is the usual way to wire this, but the build notes do not show it, so do confirm it on your build. Never connect the cells straight to a controller board. Our
ESP32 crashed the moment we did that, so feed any board from the battery through a buck converter, as shown in our DC-DC buck converter circuit article. That is all about the wiring.

Connection Table

ConnectionFromToNote
CameraPi Camera Module 3Raspberry Pi 5 CSI port Ribbon cable
CommandsRaspberry Pi 5 USB port NodeMCU micro-USBSerial over USB
Motor inputsNodeMCU GPIO pinsIN1 to IN4 on both L298N
boards
TODO: NodeMCU to L298N
pin map
Motors 1 and 2L298N 1, OUT1/OUT2 and
OUT3/OUT4
Two Johnson motorsOne motor per output pair
Motors 3 and 4L298N 2, OUT1/OUT2 and
OUT3/OUT4
Two Johnson motorsOne motor per output pair
Motor powerBattery through the rocker
switch
12V and GND on both
L298N boards
Assumed, with a common
ground
NodeMCU
power
Raspberry Pi 5 USBNodeMCUAssumed, same cable as the
commands

Building the Fish Following Robot Base

The base has to carry the frame, the electronics and a tank full of water. Below are the three bases we tried, from left to right.

Rover Bases Tried for AquaNav

Skip furniture-type bases. A 3-level shoe rack could not safely hold the aquarium, and plywood and wooden bases were only good enough for early wheel tests. An aluminium frame gave the rover the support and grip it needed. Ours had been made for a college event two years earlier, and Jayasurya gave it to us after the aluminium shops we asked would not build one to order. Choose the wheels for the full load as well. Below are the 60 mm Mecanum wheels we started with, on the wooden base, with the 3D-printed shafts that join each wheel to its motor.

AquaNav Mecanum Wheels and 3D-Printed Shafts

These wheels could not handle the weight of the setup, so we moved to a sturdier set of four wheels. Print the shafts in small steps and test the fit each time, as ours were only a few millimetres apart, some too loose and some too tight, before one fitted. The shaft STL is not in the repository. It's only a CAD file, Aquanav2.STL is a 155 × 210mm plate up to 10mm thick, and the build notes do not say which part it is.

Preparing the Aquarium and the Fish

Below is the custom aquarium, with the Betta still in its bottle while it gets used to the water.

The aquarium shops near us mostly had round tanks, so we had a rectangular one built by Glass World, a shop near Tinkerspace, to our own dimensions. TODO: aquarium dimensions. It weighs around TODO: aquarium weight kg, so check that your base and wheels can carry it once it is filled. Clean and fill the tank, keep the fish in its bottle inside the aquarium for some time so that it adjusts to the water temperature, and then release it.

Hardware Assembly of AquaNav

Below is the aluminium frame with the motors and electronics on it.

Fix the four Johnson geared motors along the sides of the frame and fit the wheels on the printed shafts. Put the controller, the two L298N boards and the cells in the middle, and test the drive on a breadboard first. With the ESP32 in charge, our rover went forward, backwards, left and right, did a U-turn and stopped on command, and it moved with the aquarium on top as well. 

Once the drive works, move everything from the breadboard to a perfboard and solder it, then complete the circuit with the battery and the switch. Below is our soldered perfboard.

AquaNav Perfboard Circuit

Finally, place the aquarium on the frame and mount the Raspberry Pi and the camera on a mast above the tank, with the camera looking down into the water.

Setting Up the Raspberry Pi and the Fish Detector

The vision code runs on Raspberry Pi OS with the system Python. Follow the steps below in order.

Step 1⇒ Connect to the Pi and Check the Camera

We worked on the Pi over SSH from PowerShell on a laptop. Check the camera with these commands before writing any tracking code.

rpicam-hello # check that the camera is detected
rpicam-hello -t 5000 # preview for 5 seconds
rpicam-still -t 2000 -o test.jpg # capture an image after 2 seconds
rpicam-vid -t 10000 -o test.h264 # record a 10-second video

Step 2⇒ Install the Packages

sudo apt update
sudo apt install -y python3-pip python3-venv python3-picamera2 --no-install-recommends
python3 -m venv --system-site-packages ~/aquanav-venv
source ~/aquanav-venv/bin/activate
pip install ultralytics flask opencv-python-headless numpy

The --system-site-packages flag lets the virtual environment see Picamera2, which comes from apt and is tied to libcamera.

Step 3⇒ Arrange the Code Folders

The files sit flat in the code folder of the repository, but the imports expect subfolders. Put settings.py in config/, camera_stream.py in camera/, fish_detector.py and visualizer.py in vision/, and stream_server.py in web/, as listed in code/README.md.

Step 4⇒ Collect and Label the Fish Images

Run collect_fish_images.py with the fish in the tank. It captures 1000 photos at 640 × 480, one every 0.5 seconds, into the folder set in SAVE_DIR, so change that path to your own. We labelled more than 1000 images by hand, and the README says the labelling was done in Edge Impulse.

Step 5⇒ Train the Model

from ultralytics import YOLO
model = YOLO("yolo11n.pt")
model.train(data="data.yaml", epochs=50, imgsz=640, batch=16, name="fish_tracking")
# data.yaml: train: images/train, val: images/val, names: {0: fish}

Copy the trained best.pt to vision/models/best.pt, as the model is not in the repository, and run python check_model.py /path/to/best.pt to print its class names. Set FISH_CLASS_NAME in settings.py to match.

Step 6⇒ Test One Stage at a Time

python test_camera.py # camera only, saves stage1_test_frame.jpg
python test_detection.py # camera + YOLO, prints 10 detections
python main.py # full pipeline + live stream on port 5000

Once main.py is running, open http://<pi-ip>:5000 in a browser on the same network. Below is our trained model tracking the fish, with its grid position printed next to the box.

Controlling the Motors from the Raspberry Pi

Connect the NodeMCU to the Raspberry Pi with a USB cable and send the movement commands from the Pi as serial data. Below is the control page on the Pi, with the NodeMCU connected over USB.

AquaNav Raspberry Pi Control Page

The page has forward, left, stop, right and backward buttons and shows whether the NodeMCU is connected. Its code and the NodeMCU motor sketch are not in the repository yet.

Testing the Fish Tracking

Once the YOLO model had to drive the wheels, it kept giving us trouble, so we moved to OpenCV, which gave better results. The rover only needs to know where the fish is, and an HSV colour mask finds that as one blob. Our OpenCV version tracks the fish as a dark blue blob, and prints the fish's horizontal and vertical zone, its centre and its area.
The red fish showed up dark blue because of a colour-order slip in the code, covered in the code explanation below, so fix that first and then set the HSV range for red. Check that the printed zone and centre follow the fish around the tank before you connect the tracking to the motors. If you want to try the dataset route with Edge Impulse instead, refer to our ESP32-CAM object detection using Edge Impulse project.

Vision Code Explanation

The code is written in Python for the Raspberry Pi 5. The final OpenCV tracking, the grid movement and the NodeMCU motor code are not in the repository yet. TODO: add the final tracking, movement and NodeMCU code, so this section explains the YOLO pipeline that is.

Settings

CAMERA_RESOLUTION = (640, 480)   # keep low for real-time inference on Pi 5
CAMERA_FRAMERATE = 30
MODEL_PATH = "vision/models/best.pt"
CONFIDENCE_THRESHOLD = 0.5
FISH_CLASS_NAME = "fish"         # set this from the Stage 0 model audit
WEB_HOST = "0.0.0.0"
WEB_PORT = 5000

All the values that may need tuning sit in settings.py. The camera runs at 640 × 480 to keep inference fast on the Pi 5, detections below a confidence of 0.5 are ignored, and the web stream listens on port 5000 on every interface.

Camera Wrapper

       cam_config = self.picam2.create_preview_configuration(
           main={"size": resolution, "format": "RGB888"}
       )
       self.picam2.configure(cam_config)
   def get_frame(self):
       """Return one frame as a numpy array (H, W, 3) in RGB order."""
       return self.picam2.capture_array()

CameraStream is a thin wrapper around Picamera2. It sets up a preview configuration at the configured size in the RGB888 format, and get_frame() returns each frame as a NumPy array. Picamera2's RGB888 frames are actually stored in blue, green, red order, which is what OpenCV expects, while the docstring here assumes RGB. main.py then converts RGB to BGR, which swaps red and blue, and that is the most likely reason our red fish looked dark blue.

Fish Detection

       results = self.model(frame, conf=self.confidence, verbose=False)[0]
       detections = []
       for box in results.boxes:
           x1, y1, x2, y2 = box.xyxy[0].tolist()
           conf = float(box.conf[0])
           cx = int((x1 + x2) / 2)
           cy = int((y1 + y2) / 2)

detect() runs the YOLO model on one frame with the confidence threshold from settings.py. For every box, it reads the corners and the confidence and works out the centre by averaging the corners, returning a list of Detection objects holding plain numbers.

Main Loop and Live Stream

       while True:
           frame = camera.get_frame()  # RGB
           bgr_frame = cv2.cvtColor(frame, cv2.COLOR_RGB2BGR)
           detections = detector.detect(bgr_frame)
           annotated = draw_detections(bgr_frame, detections)
           if detections:
               d = detections[0]
               print(f"Fish center: ({d.center_x}, {d.center_y}) "
                     f"conf={d.confidence:.2f}")
           update_frame(annotated)

main() starts the camera and the Flask stream server on its own thread, then loops forever. draw_detections() adds a green box, a red dot at the centre and the confidence to each frame; the fish centre is printed to the terminal, and update_frame() hands the frame to stream_server.py, which serves it as an MJPEG stream.

Live Video Demonstration

Below is the final AquaNav run. A team member walks alongside, holding the white cable that runs up to the Raspberry Pi on the mast.

Output of the Fish Following Robot 

Below is the finished Fish Following Robot, with the red Betta in the tank and the Raspberry Pi and camera on the mast above it.

AquaNav Fish Following Rover Final Build

In the dashboard clip, the Robot Status panel shows the fish state, how far the fish moved between frames in pixels and the motor command, with the rules of that stage: fish moves → FORWARD, fish stationary → STOP, and fish not found → STOP. When the fish has moved only 1.0 px, the motor shows STOP. The grid strategy, with its move, stop and wait cycle, is the final behaviour we kept.

Troubleshooting of Fish Following Robot 

Symptom Likely CauseFix
The red fish shows up bluePicamera2's RGB888 frames are already in B, G, R order, and the extra RGB to BGR conversion swaps red and bluePass the frame to OpenCV without the conversion, then set the HSV range for red
ModuleNotFoundError: No
module named 'camera'
The files are flat in the code folder, but the imports expect subfoldersArrange camera/, config/, vision/ and web/ as per code/README.md
The Raspberry Pi switches off while the motors runThe supply cannot hold up under the motor loadUse a battery that meets the measured need of around 14V and 2A
The ESP32 crashes as soon as the battery is connectedThe cells were wired straight to the boardFeed the board through a buck converter
The NodeMCU web page works only while it is plugged into the laptopWe did not find the causeConnect the NodeMCU to the Pi over USB instead
The wheels struggle once the aquarium is on60 mm Mecanum wheels cannot carry the loadUse a sturdier set of four wheels on a metal frame
The printed shaft is loose or will not go onThe bore is a millimetre or two offReprint in small steps and test the fit each time

Lessons From Our Build

Size the power supply before you change motors or wheels. After we moved to the bigger wheels and motors, three 3.75V cells drained in about five minutes, and we measured the need at around 14V and 2A. We kept testing on an external power supply and then a borrowed battery, and even then the Raspberry Pi kept switching off.
Test the wheels, the ultrasonic sensor and the fish tracking separately before you join them. All three worked on their own for us, and connecting the tracking directly to the wheels was much harder than testing each part.

Use a wired link between the Pi and the motor controller like our NodeMCU web server tutorial, but it lagged, and with the Pi and the NodeMCU on the same Wi-Fi network, the NodeMCU only behaved while it was connected to the laptop. The USB serial link had no problems.

Conclusion

Fish Following Robot moves according to where a red Betta swims, using a Raspberry Pi 5, OpenCV and a NodeMCU. To take a build like this further, the open items are a battery sized for the measured 14V and 2A, the final tracking and NodeMCU code in the repository, and the colour order fixed so that the fish shows up red. The team's next idea is a Guided Mode, where the rover maps a place such as a shopping mall with LiDAR and guides a person to the spot they ask for on a screen or by voice. Further, one can use the same kind of colour tracking on a flying platform, just like this ESP32 drone with object tracking using the LiteWing Tutorial.

GitHub Repository

The AquaNav_ repository has the YOLO vision code, the dataset collection script, and the build notes with their photos and videos, and one CAD file, Aquanav2.STL.

AquaNav GitHub RepositoryAquaNav Downloadable Zip

Frequently Asked Questions About The Fish Following Robot 

⇥ 1. Can Fish Following Robot follow any fish?
Yes, but the tracking is set up for our red Betta. A fish of a different colour needs a new HSV colour
range, and a YOLO model would need new images of that fish as well.

⇥ 2. How long does AquaNav run on a battery?
With three 3.75V cells, it ran for about five minutes after we changed to the bigger wheels and
motors. The runtime on the final battery was not recorded. TODO: runtime with the final battery.

These projects explore robotics, computer vision, and wireless control using platforms such as Raspberry Pi, Arduino, and ESP32. They demonstrate real-time object detection and tracking, autonomous drone movement, and Wi-Fi-based robot control.

Raspberry Pi Ball Tracking Robot using Processing

Raspberry Pi Ball Tracking Robot using Processing

In this project, let use the power of Raspberry Pi to build a Robot that could track ball and follow it just like the robots that plays football.

ESP32 Drone with Object Tracking Using LiteWing

ESP32 Drone with Object Tracking Using LiteWing

We built this project to see if we could make a simple LiteWing drone behave intelligently using just a laptop and a basic camera. By placing the camera above the room ceiling, the floor becomes a large coordinate map.

Hand Gesture Control Robot Using OpenCV

Hand Gesture Control Robot Using OpenCV

This project demonstrates a gesture-controlled robot using a laptop webcam, OpenCV, and MediaPipe for real-time hand gesture control.

Have any question related to this Article?

Add a comment

No account needed. Your comment appears once a moderator has reviewed it.

Never shown publicly.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.