For this IoT & Edge AI project challenge I want to make a project for the farmers cause India is a country where most of the people are depended on the farming so to make their life easy and encourage them towards getting more yield so that’s why I come up with this idea.
Here is my project Advanced Automated Farm, as in its name I automated the farm by using the Maixduino Development board got from the Digi-key. This project solves the major problems faced by the farmers when they are not available at the farm to maintain and other stuff to do so I came up with this idea.
Impact Statement
The Advanced Automated Farm project is designed to support farmers managing large fields and multiple farms. This innovative system enables farmers to monitor and control their operations from anywhere in the world through a specially designed mobile app. By using AI voice commands, farmers can easily manage tasks in their regional languages, making it accessible for those who may not be familiar with technology.
This low-maintenance project is simple to install and use, ensuring that even illiterate farmers can benefit from its features. With real-time data and automation, farmers can optimize their resources, streamline their operations, and ultimately increase their yields. The Advanced Automated Farm project aims to transform the agricultural sector by empowering farmers to work more efficiently and effectively, ensuring they can thrive in a competitive market. This initiative envisions a future where technology helps every farmer reach their full potential.
Components Required
1. Sipeed Maixduino AI & IoT kit
2. ESP32
4. 2-Channel 5V relay modules (2)
7. Led lights
8. 18650 Batteries
9. Wires
10. AAF APP
11. 3D Printed Parts
Watch the full demonstration by clicking on the YouTube video below :
FEATURES OF THE AAF:
There are 3 main features in this project:
Surveillance and Realtime monitoring
AAF app
Voice Control
1. Surveillance And Realtime Monitoring:
I came up with this feature cause at the Night times and when the farmer is far away from the farm there is no one to look after the farm so this Surveillance feature helps to monitor and control the camera to know what’s happening in the farm at any time and anywhere from the AAF app in their mobile.
2.AAF App:
I designed a simple and easy to use mobile app by using MIT APP INVENTOR to monitor and control all the activities in the farm from anywhere in the world through their mobile.
This app has so many features that helps the farmer to simplify their work. It can control the electrical devices like Motors, Lights, Fly traps, Rain Sensing mechanism in the farm and also has ability to turn on and off the voice commands and finally it can control the camera and shows the Realtime video footage in this app.
3.Voice Control:
I included the voice control thanks to the MEMS microphone in the Maixduino board and has an ability to Recognise the voice commands and work with them. This feature helps the farmer when his mobile is not available or any other person want to look after the farm. I trained the voice commands in English as well as in my regional language this will helps the farmer who are poor in the English and easily control the farm.
Components Selected:
1.Sipeed Maixduino Devlopment Kit:
I choose this Maixduino board because it has powerful Ai K210 processor and a build in MEMS microphone so this board will be the perfect choice for this AAF project.
2.Esp32:
Even though there is an on board ESP32 in the Maixduino dev board, I choose to use an external one due to there is only one UART communication path in between the K210 and ESP32 so we need 3 UART communication paths to carry the Voice commands, App data and the camera data captured in k210 so it is better to use the external Esp32.
3.Relays:
Here I am using the 2 channel relays (2) which are available to me it is better to use the 4-channel relay to fulfill this project. These relays act’s as the switches to control the devices in the farm.
4.Rain Sensor:
I used the basic and most popular rain sensor HL-83 in this project to detect the rain and triggers an alarm in the farm.
5.Buzzer:
I used this passive buzzer to generate alert signal when the rain I detected.
6.Submersible Water Pumps:
I am using the 5V small water pumps to represent the actual water pumps in the farm these are connected to the relays and can controlled by them.
7.LEDS:
I am using the 3mm 3.3v leds along with the 220ohm resistors to represent the actual lights and the Fly traps in the farm. These are also connected to the relay to be controlled by them.
8.Batteries:
This project runs on the battery power to simplify the reparability and also helpful when there are numerous Power-cuts. Here I am using 2 3.7V 18650 batteries to power the entire system.
Circuit Explanation:
I made this circuit diagram using the Fritzing and MS paint. This is the Overall circuit diagram of this project.
I connected the camera and Display to their respective slots. Then I connected the 3,4,5,6, pins of Maixduino to the relays. 6th pin of Maixduino is connected to the VCC of the rain sensor to power it on when required. The 7th pin of Maixduino is connected to the Buzzer Positive terminal. A0 analog pin of Maixduino is connected to the A0 pin of Rain sensor to read the data from it.
Then I established 3 UART connections in between the K210 and ESP32 to send and receive the APP data, Camera data and Voice command data. By establishing the 3 connections the data doesn’t interrupt with each other so this project will run smoothly.
Then I connected the devices like motors and lights to the relays. The entire project is powered by using the 2 18650 batteries and devices are powered using another external source.
AAF APP:
I made this AAF app by using the MIT APP INVENTOR to control the entire project. Let us see how I made this app.
I added the firebase component to this app to be the database and storage in between the ESP32 and AAF app. I connected the Firebase to the app by using the Firebase token and Firebase URL. I also added the BASE64 component to this app to decode the base64 string of the captured image data.
These are the block code for the each and every Switch in the app, when a switch is ON in the app then it will write “1” in the Firebase otherwise ”0”. Then ESP32 takes this data from Firebase and sends the required signal to the K210.
This is the block code for the Firebase, whenever there is some change in data in Firebase then this program will make sure to update the switches position and everything in the app.
This is the layout for the screen-2 which displays the live video when camera switch is ON.
This is the Block code for the screen-2 and Decoding of the Base64 data into the image.
FIREBASE SETUP:
I used the Firebase as the Realtime database to store the device values and image data. This firebase is connected to the AAF app and also the ESP32 these two exchange the data by using this Firebase Realtime database feature.
After creating a Firebase basic account then I created a project with name AAF_APP.
After successfully creating the project then I added all the required variables to my project to hold the status of the devices. This will acts as mediator between the app and the ESP32.
VOICE MODEL TRAINING:
I trained the voice model to work with the voice commands, I trained the 13 commands in 2 languages ENGLISH & TELUGU to control the entire farm.
Here is the program that is used and it is from the GitHub repository from the SIPEED MAIXDUINO and i also given this program in my GitHub repository, it is pretty simple program whenever it we gave a voice command it will take the command and convert into MFCC codes.
#include "Arduino.h"#include "Maix_Speech_Recognition.h"#include "voice_model.h"SpeechRecognizer rec;void setup(){ rec.begin(); Serial.begin(115200); Serial.println("start rec..."); if( rec.record(0, 0) == 0) //keyword_num, model_num { rec.print_model(0, 0); } else Serial.println("rec failed");}void loop(){}
By using this program it gives the MFCC codes of your command along with a FRAME NUM, then we can use this code and frame number in our voice_model.h header file in our program
Then i pasted the frame number and also the MFCC code array.
NOTE/TIP: I faced the problem of only getting NOICE while training the commands then I changed the NOICE level in the header file "Maix_Speech_Recognition.cpp" from "10000" to "50000" this helps us in solving this problem
this is in line number:210
Similarly i changed in other place in the same file, it is for while running the program or working with sensors i got the same issue so it will be resolved by changing in here too.
This is in line number:295
By making this changes we can solve this problem.
After adding all the voice models then i use header file voice_model.h in my program. This is how i created my own voice model to this project.
3D PRINTED PARTS:
I designed the case for this project by using the online platform ON SHAPE.
I designed this case into two parts where each part has mounts to hold the Components into it.
where the top part holds the Maixduino Dev board, display and other parts, the bottom half holds the relays and ESP32.
Finally I designed a simple case for my project with compact design within 120 X 100 X 50 mm.
Then i designed the Flytraps 3D model using the Blender here i made a sample of 3 Pieces just to showcase in this project.
ASSEMBLY:
These are the images taken when i am assembling the project.
This is after the completion of assembly.
WORKING:
Now let us see how this project works when we use different features
When I control the devices in the farm using the AAF mobile app, the app writes the data to the Firebase and then ESP32 takes the data from the Firebase and sends the respective command to the K210 via UART to control the device, then the K210 control the respective device either turning ON or OFF the device.
When I turn ON the camera on the app then the screen-2 is opened and also K210 takes a snapshot and sends it to the ESP32 via UART with the resolution of 320 X 160 pixels and then the ESP32 converts the data to the BASE64 and writes it on the Firebase, then the AAF app takes the BASE64 String from the Firebase and decodes it into the image and displays it on the screen.
Similarly when we use the Voice commands then the data get updated in the app in this similar way.
Code Explanation:
I wrote 2 codes for this project one for the Maixduino K210 and one for ESP32 by using the PlatformIO cause it is more powerful than the Arduino IDE.
Now let us see these codes:
1.Maixduino K210 Code Explanation:
Before writing the code I added the Maixduino board to the PlatformIO then created a project K210_PROGRAM then proceed to write the program.
CODE:#include <Arduino.h>#include <Sipeed_ST7789.h>#include <Sipeed_OV2640.h>#include <SPI.h>#include <Maix_Speech_Recognition.h>#include <voice_model.h>#include "FreeRTOS.h"#include "task.h"//UART PINS#define K210_CMD_UART_TX 13 // TX pin for K210#define K210_CMD_UART_RX 12 // RX pin for K210
#define K210_CAM_UART_TX 11 // TX pin for K210#define K210_CAM_UART_RX 10 // RX pin for K210
#define K210_VCMD_UART_TX 9 // TX pin for K210#define K210_VCMD_UART_RX 8 // RX pin for K210
// Define Pin Constants#defineM1_PIN2#defineM2_PIN3#defineLIGHT_PIN4#defineFTRAP_PIN5#define RSENSOR_PIN 6#define BUZZERPIN 7#define READ_PIN A0
// LCD Display Constants#define LABEL_TEXT_SIZE 3 // Font size multiplier#define BG_COLOR COLOR_DARKCYAN
//SPEECH RECOGNITION CONSTANTSSpeechRecognizer rec;int VCMDS;
// SPI and LCD InitializationSPIClass spi_(SPI0); // Must be SPI0 for Maix series on-board LCDSipeed_ST7789 lcd(320, 240, spi_, SIPEED_ST7789_DCX_PIN, SIPEED_ST7789_RST_PIN, DMAC_CHANNEL2);Sipeed_OV2640 camera(FRAMESIZE_QQVGA, PIXFORMAT_RGB565);
// Function to print centered text on the LCDvoid printCenterOnLCD(Sipeed_ST7789 &lcd_, const char *msg, uint8_t textSize = LABEL_TEXT_SIZE) { lcd_.setCursor((lcd_.width() - (6 * textSize * strlen(msg))) / 2, (lcd_.height() - (8 * textSize)) / 2); lcd_.print(msg);}
//FUNCTION TO DISPLAT THE LOGOvoid displayLogo(Sipeed_ST7789 &lcd_) { lcd_.fillScreen(COLOR_BLACK); // Set the background color // Display large "AAF" text lcd_.setTextColor(COLOR_WHITE); lcd_.setTextSize(6); // Calculate position to center "AAF" int16_t x = (lcd_.width() - (6 * 3 * 6)) / 2; int16_t y = (lcd_.height() - (8 * 6)) / 3; // Set cursor to the calculated position and print "AAF" lcd_.setCursor(x, y); lcd_.print("AAF"); // Display smaller "FARMING MADE SIMPLE" text below the "AAF" lcd_.setTextSize(2); int16_t x_small = (lcd_.width() - (6 * 18 * 2)) / 2; int16_t y_small = y + (8 * 6) + 30; lcd_.setCursor(x_small, y_small); lcd_.print("FARMING MADE SIMPLE");}
// Global variablesint camera_state = 0; // This controls the camera state globallyunsigned long lastCaptureTime = 0;const unsigned long captureInterval = 200; // Capture every 200 msvoid setup() { Serial1.begin(230400, K210_CMD_UART_RX, K210_CMD_UART_TX); // To send COMMANDS Serial2.begin(230400, K210_CAM_UART_RX, K210_CAM_UART_TX); // To send CAM DATA Serial3.begin(115200,K210_VCMD_UART_RX,K210_VCMD_UART_TX); // To send Voice CMD Serial.begin(115200); // For communicating // Initialize GPIO Pins pinMode(M1_PIN, OUTPUT); pinMode(M2_PIN, OUTPUT); pinMode(LIGHT_PIN, OUTPUT); pinMode(FTRAP_PIN, OUTPUT); pinMode(RSENSOR_PIN, OUTPUT); pinMode(BUZZERPIN,OUTPUT); pinMode(READ_PIN,INPUT); // Initialize LCD if (!lcd.begin(15000000, BG_COLOR)) { Serial.println("LCD initialization failed."); while (1); // Halt if LCD fails } lcd.setTextSize(LABEL_TEXT_SIZE); lcd.setTextColor(COLOR_WHITE); //TO PRINT TAG LINE ON LCD WHEN POWRED ON displayLogo(lcd); delay(2000); lcd.fillScreen(BG_COLOR);
rec.begin(); // Load voice models for commands (the actual commands will vary based on the data you are using) rec.addVoiceModel(0, 0, motor1_on_0, fram_num_motor1_on_0); // TURN ON MOTOR1 rec.addVoiceModel(0, 1, motor1_on_1, fram_num_motor1_on_1); rec.addVoiceModel(0, 2, motor1_on_2, fram_num_motor1_on_2); //IN REGIONAL LANGUAGE rec.addVoiceModel(0, 3, motor1_on_3, fram_num_motor1_on_3); rec.addVoiceModel(1, 0, motor2_on_0, fram_num_motor2_on_0); // TURN ON MOTOR2 rec.addVoiceModel(1, 1, motor2_on_1, fram_num_motor2_on_1); rec.addVoiceModel(1, 2, motor2_on_2, fram_num_motor2_on_2); //IN REGIONAL LANGUAGE rec.addVoiceModel(1, 3, motor2_on_3, fram_num_motor2_on_3); rec.addVoiceModel(2, 0, flytrap_on_0, fram_num_flytrap_on_0); // TURN ON FLY TRAPS rec.addVoiceModel(2, 1, flytrap_on_1, fram_num_flytrap_on_1); rec.addVoiceModel(2, 2, flytrap_on_2, fram_num_flytrap_on_2); //IN REGIONAL LANGUAGE rec.addVoiceModel(2, 3, flytrap_on_3, fram_num_flytrap_on_3); rec.addVoiceModel(3, 0, rainsensor_on_0, fram_num_rainsensor_on_0); // TURN ON RAIN SENSOR rec.addVoiceModel(3, 1, rainsensor_on_1, fram_num_rainsensor_on_1); rec.addVoiceModel(3, 2, rainsensor_on_2, fram_num_rainsensor_on_2); //IN REGIONAL LANGUAGE rec.addVoiceModel(3, 3, rainsensor_on_3, fram_num_rainsensor_on_3); rec.addVoiceModel(4, 0, lights_on_0, fram_num_lights_on_0); // TURN ON LIGHTs rec.addVoiceModel(4, 1, lights_on_1, fram_num_lights_on_1); rec.addVoiceModel(4, 2, lights_on_2, fram_num_lights_on_2); //IN REGIONAL LANGUAGE rec.addVoiceModel(4, 3, lights_on_3, fram_num_lights_on_3); rec.addVoiceModel(5, 0, camera_on_0, fram_num_camera_on_0); // TURN ON CAM rec.addVoiceModel(5, 1, camera_on_1, fram_num_camera_on_1); rec.addVoiceModel(5, 2, camera_on_2, fram_num_camera_on_2); //IN REGIONAL LANGUAGE rec.addVoiceModel(5, 3, camera_on_3, fram_num_camera_on_3); rec.addVoiceModel(6, 0, motor1_off_0, fram_num_motor1_off_0); // TURN OFF MOTOR1 rec.addVoiceModel(6, 1, motor1_off_1, fram_num_motor1_off_1); rec.addVoiceModel(6, 2, motor1_off_2, fram_num_motor1_off_2); //IN REGIONAL LANGUAGE rec.addVoiceModel(6, 3, motor1_off_3, fram_num_motor1_off_3); rec.addVoiceModel(7, 0, motor2_off_0, fram_num_motor2_off_0); // TURN OFF MOTOR2 rec.addVoiceModel(7, 1, motor2_off_1, fram_num_motor2_off_1); rec.addVoiceModel(7, 2, motor2_off_2, fram_num_motor2_off_2);//IN REGIONAL LANGUAGE rec.addVoiceModel(7, 3, motor2_off_3, fram_num_motor2_off_3); rec.addVoiceModel(8, 0, flytrap_off_0, fram_num_flytrap_off_0); // TURN OFF FLY TRAPS rec.addVoiceModel(8, 1, flytrap_off_1, fram_num_flytrap_off_1); rec.addVoiceModel(8, 2, flytrap_off_2, fram_num_flytrap_off_2); //IN REGIONAL LANGUAGE rec.addVoiceModel(8, 3, flytrap_off_3, fram_num_flytrap_off_3); rec.addVoiceModel(9, 0, rainsensor_off_0, fram_num_rainsensor_off_0); // TURN OFF RAIN SENSOR rec.addVoiceModel(9, 1, rainsensor_off_1, fram_num_rainsensor_off_1); rec.addVoiceModel(9, 2, rainsensor_off_2, fram_num_rainsensor_off_2); //IN REGIONAL LANGUAGE rec.addVoiceModel(9, 3, rainsensor_off_3, fram_num_rainsensor_off_3); rec.addVoiceModel(10, 0, lights_off_0, fram_num_lights_off_0); // TURN OFF LIGHTS rec.addVoiceModel(10, 1, lights_off_1, fram_num_lights_off_1); rec.addVoiceModel(10, 2, lights_off_2, fram_num_lights_off_2); //IN REGIONAL LANGUAGE rec.addVoiceModel(10, 3, lights_off_3, fram_num_lights_off_3); rec.addVoiceModel(11, 0, camera_off_0, fram_num_camera_off_0); // TURN OFF CAMERA rec.addVoiceModel(11, 1, camera_off_1, fram_num_camera_off_1); rec.addVoiceModel(11, 2, camera_off_2, fram_num_camera_off_2); //IN REGIONAL LANGUAGE rec.addVoiceModel(11, 3, camera_off_3, fram_num_camera_off_3); rec.addVoiceModel(12, 0, vcmd_off_0, fram_num_vcmd_off_0); // TURN OFF VOICE COMMANDS rec.addVoiceModel(12, 1, vcmd_off_1, fram_num_vcmd_off_1); rec.addVoiceModel(12, 2, vcmd_off_2, fram_num_vcmd_off_2); //IN REGIONAL LANGUAGE rec.addVoiceModel(12, 3, vcmd_off_3, fram_num_vcmd_off_3); }void loop() { for(int i=0;i<9;i++){ // Check for serial data to control the camera and other components if (Serial1.available()) { String CMDESP32 = Serial1.readStringUntil('n'); CMDESP32.trim(); Serial.println(CMDESP32); // Control logic based on commands if (CMDESP32 == "CAM_ON" && camera_state == 0) { camera_state = 1; // Set camera state to ON camera.begin(); // Initialize the camera only when turning it ON Serial.println("Camera started"); } else if (CMDESP32 == "CAM_OFF" && camera_state == 1) { camera_state = 0; // Set camera state to OFF camera.end(); // Stop the camera when turning it OFF lcd.fillScreen(BG_COLOR); // Clear LCD screen Serial.println("Camera stopped"); } else if (CMDESP32 == "M1_ON") { digitalWrite(M1_PIN, LOW); } else if (CMDESP32 == "M1_OFF") { digitalWrite(M1_PIN, HIGH); } else if (CMDESP32 == "M2_ON") { digitalWrite(M2_PIN, LOW); } else if (CMDESP32 == "M2_OFF") { digitalWrite(M2_PIN, HIGH); } else if (CMDESP32 == "LIGHTS_ON") { digitalWrite(LIGHT_PIN, LOW); } else if (CMDESP32 == "LIGHTS_OFF") { digitalWrite(LIGHT_PIN, HIGH); } else if (CMDESP32 == "RSENSOR_ON") { digitalWrite(RSENSOR_PIN, HIGH); } else if (CMDESP32 == "RSENSOR_OFF") { digitalWrite(RSENSOR_PIN, LOW); } else if (CMDESP32 == "FTRAP_ON") { digitalWrite(FTRAP_PIN, LOW); } else if (CMDESP32 == "FTRAP_OFF") { digitalWrite(FTRAP_PIN, HIGH); } else if (CMDESP32 == "V1") { VCMDS = 1; } else if (CMDESP32 == "V0") { VCMDS = 0; } } } delay(1000); // Only run the camera if it's turned ON if (camera_state == 1) { unsigned long currentMillis = millis(); if (currentMillis - lastCaptureTime >= captureInterval) { lastCaptureTime = currentMillis; // Capture and send the image camera.run(true); uint8_t *img = camera.snapshot(); uint16_t width = camera.width(); uint16_t height = camera.height(); if (img == nullptr) { Serial.println("Camera snap failed"); } else { lcd.drawImage(0, 0, width, height, (uint16_t *)img); // Update the LCD display Serial2.write((uint8_t *)&width, sizeof(width)); // Send width Serial2.write((uint8_t *)&height, sizeof(height)); // Send height Serial2.write(img, width * height * 2); // Send image data } } } delay(10); // Avoid task starvation // RAINDETECTION AND ALERT SIGNAL GENERATION if(analogRead(READ_PIN) > 500){ digitalWrite(BUZZERPIN,HIGH); } else{ digitalWrite(BUZZERPIN,LOW); } //VOICE COMMANDS if(VCMDS == 1){ // Start the voice recognition system int res = rec.recognize(); Serial.printf("Recognized command: %d --> ", res); lcd.fillScreen(BG_COLOR); // Respond based on recognized result switch (res) { case 0: // TURN ON MOTOR1 Serial3.println("M1_ON_ACK"); digitalWrite(M1_PIN, LOW); printCenterOnLCD(lcd, "Motor1 On"); break; case 1: // TURN ON MOTOR2 Serial3.println("M2_ON_ACK"); digitalWrite(M2_PIN, LOW); printCenterOnLCD(lcd, "Motor2 On"); break; case 2: // TURN ON FLY TRAP Serial3.println("FTRAP_ON_ACK"); digitalWrite(FTRAP_PIN, LOW); printCenterOnLCD(lcd, "Fly Trap On"); break; case 3: // TURN ON RAIN SENSOR Serial3.println("RSENSOR_ON_ACK"); digitalWrite(RSENSOR_PIN, LOW); printCenterOnLCD(lcd, "Rain Sensor On"); break; case 4: // TURN ON LIGHTS Serial3.println("LIGHTS_ON_ACK"); digitalWrite(LIGHT_PIN, LOW); printCenterOnLCD(lcd, "Lights On"); break; case 5: // TURN ON CAMERA Serial3.println("CAM_ON_ACK"); printCenterOnLCD(lcd, "Camera On"); break; case 6: // TURN OFF MOTOR1 Serial3.println("M1_OFF_ACK"); digitalWrite(M1_PIN, HIGH); printCenterOnLCD(lcd, "Motor1 Off"); break; case 7: // TURN OFF MOTOR2 Serial3.println("M2_OFF_ACK"); digitalWrite(M2_PIN, HIGH); printCenterOnLCD(lcd, "Motor2 Off"); break; case 8: // TURN OFF FLY TRAP Serial3.println("FTRAP_OFF_ACK"); digitalWrite(FTRAP_PIN, HIGH); printCenterOnLCD(lcd, "Fly Trap Off"); break; case 9: // TURN OFF RAIN SENSOR Serial3.println("RSENSOR_OFF_ACK"); digitalWrite(RSENSOR_PIN, HIGH); printCenterOnLCD(lcd, "Rain Sensor Off"); break; case 10: // TURN OFF LIGHTS Serial3.println("LIGHTS_OFF_ACK"); digitalWrite(LIGHT_PIN, HIGH); printCenterOnLCD(lcd, "Lights Off"); break; case 11: // TURN OFF LIGHTS Serial3.println("LIGHTS_OFF_ACK"); digitalWrite(LIGHT_PIN, HIGH); printCenterOnLCD(lcd, "Lights Off"); break; case 12: // TURN OFF VOICE COMMANDS Serial3.println("VCMD_OFF_ACK"); printCenterOnLCD(lcd, "Voice Commands Off"); break; default: Serial1.println("Command not recognized"); printCenterOnLCD(lcd, "Unrecognized Command"); break; } } delay(1000);}
Explanation:
Firstly I added the Required libraries for this project,#include <Arduino.h>#include <Sipeed_ST7789.h>#include <Sipeed_OV2640.h>#include <SPI.h>#include <Maix_Speech_Recognition.h>#include <voice_model.h>#include "FreeRTOS.h"#include "task.h"
then I started by configuring the UART communication lines, which are essential for communication between the K210, the ESP32, and other peripherals. For each line, I assigned specific TX and RX pins:#define K210_CMD_UART_TX 13#define K210_CMD_UART_RX 12 For the commands #define K210_CAM_UART_TX 11#define K210_CAM_UART_RX 10 For the Camera#define K210_VCMD_UART_TX 9#define K210_VCMD_UART_RX 8 For the Voice commands
This setup ensures reliable data exchange between the K210’s command, camera data, and voice commands.
For controlling farm devices like motors, lights, fly traps, and sensors, I mapped the necessary components to specific GPIO pins:
#define M1_PIN 2 // Motor 1#define M2_PIN 3 // Motor 2#define LIGHT_PIN 4 // Lights#define FTRAP_PIN 5 // Fly traps#define RSENSOR_PIN 6 // Rain sensor#define BUZZERPIN 7 // Buzzer#define READ_PIN A0 // For analog inputs
This allows easy control of devices like motors and lights by simply setting these pins high or low in the code.
Next, I handled the LCD display by initializing it using SPI communication. I used the Sipeed_ST7789 library for displaying graphics and messages.
SPIClass spi_(SPI0); // Must be SPI0 for Maix series on-board LCDSipeed_ST7789 lcd(320, 240, spi_, SIPEED_ST7789_DCX_PIN, SIPEED_ST7789_RST_PIN, DMAC_CHANNEL2);Sipeed_OV2640 camera(FRAMESIZE_QQVGA, PIXFORMAT_RGB565);
I created a function named printCenterOnLCD() to display the commands at the center of the screen to know what device is turned ON or OFFvoid printCenterOnLCD(Sipeed_ST7789 &lcd_, const char *msg, uint8_t textSize = LABEL_TEXT_SIZE) { lcd_.setCursor((lcd_.width() - (6 * textSize * strlen(msg))) / 2, (lcd_.height() - (8 * textSize)) / 2); lcd_.print(msg);}
To make the user interface appealing, I created a displayLogo() function that presents a startup logo when the system is powered on:
void displayLogo(Sipeed_ST7789 &lcd_){
lcd_.fillScreen(COLOR_BLACK); // Clear the screen to black lcd_.setTextColor(COLOR_WHITE);
lcd_.setTextSize(6); // Display "AAF" in large text lcd_.setCursor(centerX, centerY); // Position text in the center lcd_.print("AAF");
lcd_.setTextSize(2); // Display the tagline in smaller text lcd_.setCursor(centerXSmall, centerYSmall);lcd_.print("FARMING MADE SIMPLE");
}
This provides branding and a welcome screen before entering the main control functionality.
then I add few variables to control the camera
int camera_state = 0; // This controls the camera state globallyunsigned long lastCaptureTime = 0;const unsigned long captureInterval = 200; // Capture every 200 ms
Initialization In Setup()
Then in the setup() function I initialize the UART Serial communication with their respective baud rates Serial1.begin(230400, K210_CMD_UART_RX, K210_CMD_UART_TX); // To send COMMANDS Serial2.begin(230400, K210_CAM_UART_RX, K210_CAM_UART_TX); // To send CAM DATA Serial3.begin(115200,K210_VCMD_UART_RX,K210_VCMD_UART_TX); // To send Voice CMD Serial.begin(115200); // For communicating
Then i Initializes the pins of the devices // Initialize GPIO Pins pinMode(M1_PIN, OUTPUT); pinMode(M2_PIN, OUTPUT); pinMode(LIGHT_PIN, OUTPUT); pinMode(FTRAP_PIN, OUTPUT); pinMode(RSENSOR_PIN, OUTPUT); pinMode(BUZZERPIN,OUTPUT); pinMode(READ_PIN,INPUT);
Now i initialize the LCD display and displays the Logo for like 2 seconds// Initialize LCD if (!lcd.begin(15000000, BG_COLOR)) { Serial.println("LCD initialization failed."); while (1); // Halt if LCD fails } lcd.setTextSize(LABEL_TEXT_SIZE); lcd.setTextColor(COLOR_WHITE); //TO PRINT TAG LINE ON LCD WHEN POWRED ON displayLogo(lcd); delay(2000); lcd.fillScreen(BG_COLOR);
For the key feature of speech recognition, I loaded multiple voice models to recognize different languages. This helps farmers use the system in their native languages. For instance, I mapped different speech models for controlling Motor 1:
rec.addVoiceModel(0, 0, motor1_on_0, fram_num_motor1_on_0); rec.addVoiceModel(0, 2, motor1_on_2, fram_num_motor1_on_2);
By doing this, the system becomes more accessible and inclusive for regional users.
Main Logic In Loop()
Now in the loop() function I am reading the data continuously from the ESP32 with the Serial1, after getting the data then K210 controls the devices based on the commands it got from the ESP32.
for(int i=0;i<9;i++){ // Check for serial data to control the camera and other components if (Serial1.available()) { String CMDESP32 = Serial1.readStringUntil('n'); CMDESP32.trim(); Serial.println(CMDESP32); // Control logic based on commands if (CMDESP32 == "CAM_ON" && camera_state == 0) { camera_state = 1; // Set camera state to ON camera.begin(); // Initialize the camera only when turning it ON Serial.println("Camera started"); } else if (CMDESP32 == "CAM_OFF" && camera_state == 1) { camera_state = 0; // Set camera state to OFF camera.end(); // Stop the camera when turning it OFF lcd.fillScreen(BG_COLOR); // Clear LCD screen Serial.println("Camera stopped"); }//similarly for every device.....}
After getting the values then it will check the camera function if camera_state ==1 then the K210 takes the snapshot and sends it to the ESP32 via Serial2 with the resolution of 320 X 160// Only run the camera if it's turned ON if (camera_state == 1) { unsigned long currentMillis = millis(); if (currentMillis - lastCaptureTime >= captureInterval) { lastCaptureTime = currentMillis; // Capture and send the image camera.run(true); uint8_t *img = camera.snapshot(); uint16_t width = camera.width(); uint16_t height = camera.height(); if (img == nullptr) { Serial.println("Camera snap failed"); } else { lcd.drawImage(0, 0, width, height, (uint16_t *)img); // Update the LCD display Serial2.write((uint8_t *)&width, sizeof(width)); // Send width Serial2.write((uint8_t *)&height, sizeof(height)); // Send height Serial2.write(img, width * height * 2); // Send image data } } }
Then it will check for the rain if the read value is less than 350 then the alarm will start
// RAIN DETECTION AND ALERT SIGNAL GENERATION if(analogRead(READ_PIN) < 350){ digitalWrite(BUZZERPIN,HIGH); } else{ digitalWrite(BUZZERPIN,LOW); }
Then whenever the voice commands is ON through the mobile app then the system will takes the Voice commands and control the devices by them.//VOICE COMMANDS if(VCMDS == 1){ // Start the voice recognition system int res = rec.recognize(); Serial.printf("Recognized command: %d --> ", res); lcd.fillScreen(BG_COLOR); // Respond based on recognized result and sending acknowledge switch (res) { case 0: // TURN ON MOTOR1 Serial3.println("M1_ON_ACK"); digitalWrite(M1_PIN, LOW); printCenterOnLCD(lcd, "Motor1 On"); break; case 1: // TURN ON MOTOR2 Serial3.println("M2_ON_ACK"); digitalWrite(M2_PIN, LOW); printCenterOnLCD(lcd, "Motor2 On"); break;similarly for all the commands.....}
After this i added the 1second delay in between each cycle this will make sure to run our program properly.
This is a simple explanation of this Maixduino program. Now let us see the ESP32 program and its explanation.
2. ESP32 Code Explanation:
I wrote the code for the ESP32 where the Major work is to communicate with the K210 and the Firebase
CODE:
#include <Arduino.h>#include <WiFi.h>#include <FirebaseESP32.h>#include <Base64.h>#include <SoftwareSerial.h>#define ESP32_CMD_UART_RX 16 // RX pin for ESP32 CMD#define ESP32_CMD_UART_TX 17 // TX pin for ESP32 CMD#define ESP32_CAM_UART_RX 26 // RX pin for ESP32 CAM#define ESP32_CAM_UART_TX 27 // TX pin for ESP32 CAMSoftwareSerial serial3(18,19); //Serial for Voice command communication// FIREBASE REQUIRED CONSTANTS#define FIREBASE_HOST "https://aaf-app-9ba60-default-rtdb.firebaseio.com/"#define FIREBASE_AUTH "AIzaSyCpWLG8aPMLBZtiCaysBdtvKzL8WKDAbP4"const char *ssid = "Siva";const char *password = "1234567890";FirebaseData fbd;FirebaseJson json;FirebaseConfig config;FirebaseAuth auth;// VARIABLES TO GET VALUES FROM FIREBASEString m1 ;String rs ;String m2;String cam;String lts;String ftr;#define ON 1#define OFF 0void setup() {Serial.begin(115200); // For debuggingSerial1.begin(230400, SERIAL_8N1, ESP32_CMD_UART_RX, ESP32_CMD_UART_TX); // For receiving CMD dataSerial2.begin(230400, SERIAL_8N1, ESP32_CAM_UART_RX, ESP32_CAM_UART_TX); // For receiving CAM dataserial3.begin(115200);Serial.println();// CONNECTING ESP32 TO WIFISerial.print("[WiFi] Connecting to ");Serial.println(ssid);WiFi.begin(ssid, password);while (WiFi.status() != WL_CONNECTED) { delay(500); Serial.print(".");}Serial.println("");Serial.println("WiFi connected.");Serial.println("IP address: ");Serial.println(WiFi.localIP());
config.host = FIREBASE_HOST;config.signer.tokens.legacy_token = FIREBASE_AUTH;//INITIALISING FIREBASEFirebase.begin(&config, &auth);Firebase.reconnectWiFi(true);}void loop() { Serial.println("loop is running"); //GETTING VALUES FROM FIREBASE String LIGHTS, CAM, FTRAP, M1, M2, RSENSOR, VCMD; if (Firebase.getString(fbd, "/AAF_APP/lights")) LIGHTS = fbd.stringData(); else Serial.println("Failed to get LIGHTS"); if (Firebase.getString(fbd, "/AAF_APP/cam")) CAM = fbd.stringData(); else Serial.println("Failed to get CAM"); if (Firebase.getString(fbd, "/AAF_APP/fly_trap")) FTRAP = fbd.stringData(); else Serial.println("Failed to get FTRAP"); if (Firebase.getString(fbd, "/AAF_APP/motor_1")) M1 = fbd.stringData(); else Serial.println("Failed to get MOTOR_1"); if (Firebase.getString(fbd, "/AAF_APP/motor_2")) M2 = fbd.stringData(); else Serial.println("Failed to get MOTOR_2"); if (Firebase.getString(fbd, "/AAF_APP/rain_sensor")) RSENSOR = fbd.stringData(); else Serial.println("Failed to get RSENSOR"); if (Firebase.getString(fbd, "/AAF_APP/voice_cmds")) VCMD = fbd.stringData(); else Serial.println("Failed to get RSENSOR"); // Send commands to K210 via Serial1 and update Firebase only if necessary if (M1 == "1") { Serial1.println("M1_ON"); } else { Serial1.println("M1_OFF"); } if (M2 == "1") { Serial1.println("M2_ON"); } else { Serial1.println("M2_OFF"); } if (LIGHTS == "1") { Serial1.println("LIGHTS_ON"); } else { Serial1.println("LIGHTS_OFF"); } if (RSENSOR == "1") { Serial1.println("RSENSOR_ON"); } else { Serial1.println("RSENSOR_OFF"); } if (FTRAP == "1") { Serial1.println("FTRAP_ON"); } else { Serial1.println("FTRAP_OFF"); } if (CAM == "1") { Serial1.println("CAM_ON"); } else { Serial1.println("CAM_OFF"); } if (VCMD == "1") { Serial1.println("V1"); } else { Serial1.println("V0"); } delay(200); if (Serial2.available() >= sizeof(uint16_t) * 2) { // Check if enough bytes for width and height uint16_t width, height; // Read width and height from camera data Serial2.readBytes((uint8_t*)&width, sizeof(width)); Serial2.readBytes((uint8_t*)&height, sizeof(height)); Serial.print("Width: "); Serial.print(width); Serial.print(", Height: "); Serial.println(height); // Prepare buffer to receive image data uint8_t* img = new uint8_t[width * height * 2]; //buffer for image data // Read image data Serial2.readBytes(img, width * height * 2); // Encode image data to base64 string String base64Image = base64::encode(img, width * height * 2); Firebase.setString(fbd, "/AAF_APP/cam_image", base64Image); // Send base64 image string to Firebase Realtime Database if (Firebase.setString(fbd, "/AAF_APP/cam_image", base64Image)) { Serial.println("Image sent to Firebase"); } else { Serial.println("Failed to send image to Firebase"); } delete[] img; // Free the allocated buffer } delay(100); // Avoid task starvation //RECEVING COMMAND FROM K210 WHEN VOICE COMMANDS ACTIVATED if(serial3.available()){ String K210CMD = serial3.readStringUntil('n'); K210CMD.trim(); Serial.println(K210CMD); if(K210CMD == "M1_ON_ACK"){ Firebase.setString(fbd,"/AAF_APP/motor_1","1"); } else if(K210CMD == "M1_OFF_ACK"){ Firebase.setString(fbd,"/AAF_APP/motor_1","0"); } else if(K210CMD == "M2_ON_ACK"){ Firebase.setString(fbd,"/AAF_APP/motor_2","1"); } else if(K210CMD == "M2_OFF_ACK"){ Firebase.setString(fbd,"/AAF_APP/motor_2","0"); } else if(K210CMD == "FTRAP_ON_ACK"){ Firebase.setString(fbd,"/AAF_APP/fly_trap","1"); } else if(K210CMD == "FTRAP_OFF_ACK"){ Firebase.setString(fbd,"/AAF_APP/fly_trap","0"); } else if(K210CMD == "RSENSOR_ON_ACK"){ Firebase.setString(fbd,"/AAF_APP/rain_sensor","1"); } else if(K210CMD == "RSENSOR_OFF_ACK"){ Firebase.setString(fbd,"/AAF_APP/rain_sensor","0"); } else if(K210CMD == "LIGHTS_ON_ACK"){ Firebase.setString(fbd,"/AAF_APP/lights","1"); } else if(K210CMD == "LIGHTS_OFF_ACK"){ Firebase.setString(fbd,"/AAF_APP/lights","0"); } else if(K210CMD == "CAM_ON_ACK"){ Firebase.setString(fbd,"/AAF_APP/cam","1"); } else if(K210CMD == "CAM_OFF_ACK"){ Firebase.setString(fbd,"/AAF_APP/cam","0"); } else if(K210CMD == "VCMD_OFF_ACK"){ Firebase.setString(fbd,"/AAF_APP/cam","0"); } }}
Explanation:
I start by including essential libraries that enable the ESP32 to perform different functions:
#include <Arduino.h>#include <WiFi.h>#include <FirebaseESP32.h>#include <Base64.h>#include <SoftwareSerial.h>
Arduino.h: This is the core Arduino library that provides basic functionality.
WiFi.h: I use this library to handle Wi-Fi connections.
FirebaseESP32.h: This library allows the ESP32 to interact with Firebase services.
Base64.h: I utilize this library for encoding image data into Base64 format.
SoftwareSerial.h: This library enables serial communication on other pins, which is useful for the voice command module.
Next, I define the RX and TX pins for two different serial communications: one for the command interface of the ESP32 and another for the camera interface.
#define ESP32_CMD_UART_RX 16 // RX pin for ESP32 CMD#define ESP32_CMD_UART_TX 17 // TX pin for ESP32 CMD#define ESP32_CAM_UART_RX 26 // RX pin for ESP32 CAM#define ESP32_CAM_UART_TX 27 // TX pin for ESP32 CAMSoftwareSerial serial3(18, 19); // Serial for voice command communication
Here, I set the pins for communication with the K210 chip and the camera. I also initialize a SoftwareSerial instance for handling voice commands on pins 18 and 19.
Firebase And Wi-Fi Configuration
I define constants for Firebase host and authentication:
#define FIREBASE_HOST "https://aaf-app-9ba60-default-rtdb.firebaseio.com/"
#define FIREBASE_AUTH "AIzaSyCpWLG8aPMLBZtiCaysBdtvKzL8WKDAbP4"
const char *ssid = "Siva";
const char *password = "1234567890";
FIREBASE_HOST:The URL of the Firebase Realtime Database.FIREBASE_AUTH: The API key needed for authentication.ssidandpassword:My Wi-Fi credentials, which I use to connect to the network.
Initialization In Setup()
In the setup() function, I initialize serial communication and connect to Wi-Fi and Firebase:
void setup(){Serial.begin(115200); // For debugging
Serial1.begin(230400, SERIAL_8N1, ESP32_CMD_UART_RX, ESP32_CMD_UART_TX); // For receiving CMD dataSerial2.begin(230400, SERIAL_8N1, ESP32_CAM_UART_RX, ESP32_CAM_UART_TX); // For receiving CAM data
serial3.begin(115200);
Serial.println();
I set the baud rates for different serial interfaces. The main Serial is for debugging purposes, while Serial1 and Serial2 are for command and camera data, respectively.
Next, I connect to Wi-Fi:
Serial.print("[WiFi] Connecting to ");
Serial.println(ssid);
WiFi.begin(ssid, password);
while (WiFi.status() != WL_CONNECTED) {
delay(500);Serial.print(".");
}
I repeatedly check the Wi-Fi connection status until it successfully connects, at which point I print the local IP address.
Firebase Initialization
After establishing the Wi-Fi connection, I configure Firebase:
config.host = FIREBASE_HOST;config.signer.tokens.legacy_token = FIREBASE_AUTH;
Firebase.begin(&config, &auth);
Firebase.reconnectWiFi(true);
Here, I set the Firebase host and authentication token and initialize Firebase using Firebase.begin().
Main Logic In Loop()
In the loop() function, I continuously check for commands and update device states:
void loop(){Serial.println("loop is running");// GETTING VALUES FROM FIREBASE
String LIGHTS, CAM, FTRAP, M1, M2, RSENSOR, VCMD;
I declare string variables to store the states of various devices. I retrieve these values from Firebase:
if (Firebase.getString(fbd, "/AAF_APP/lights")) LIGHTS = fbd.stringData();else Serial.println("Failed to get LIGHTS");if (Firebase.getString(fbd, "/AAF_APP/cam")) CAM = fbd.stringData();
else Serial.println("Failed to get CAM");
if (Firebase.getString(fbd, "/AAF_APP/fly_trap")) FTRAP = fbd.stringData();else Serial.println("Failed to get FTRAP");if (Firebase.getString(fbd, "/AAF_APP/motor_1")) M1 = fbd.stringData();else Serial.println("Failed to get MOTOR_1");
if (Firebase.getString(fbd, "/AAF_APP/motor_2")) M2 = fbd.stringData();
else Serial.println("Failed to get MOTOR_2");
if (Firebase.getString(fbd, "/AAF_APP/rain_sensor")) RSENSOR = fbd.stringData();
else Serial.println("Failed to get RSENSOR");
if (Firebase.getString(fbd, "/AAF_APP/voice_cmds")) VCMD = fbd.stringData();else Serial.println("Failed to get RSENSOR");
For each device state, I call Firebase.getString(), and if successful, I assign the value to the respective variable. If it fails, I print an error message.
Sending Commands To Devices
Once I have the device states, I send commands to the K210 through Serial1 based on the values retrieved:
if (M1 == "1") {
Serial1.println("M1_ON");}
else {
Serial1.println("M1_OFF");
}
if (M2 == "1") {
Serial1.println("M2_ON");
Similarly for ecery command....
This section sends the appropriate command based on the state of each device. For instance, if M1 is "1", it sends "M1_ON" to turn on motor 1; otherwise, it sends "M1_OFF".
Handling Camera Data
I check if there’s available data from the camera and process it accordingly:
delay(200);if (Serial2.available() >= sizeof(uint16_t) * 2) { // Check if enough bytes for width and height
uint16_t width, height;// Read width and height from camera data
Serial2.readBytes((uint8_t*)&width, sizeof(width));
Serial2.readBytes((uint8_t*)&height, sizeof(height));Serial.print("Width: ");
Serial.print(width);Serial.print(", Height: ");Serial.println(height);
Here, I ensure that enough bytes are available to read the image dimensions. After reading the width and height, I prepare to receive the image data.
uint8_t* img = new uint8_t[width * height * 2]; // Buffer for image data// Read image dataSerial2.readBytes(img, width * height * 2);// Encode image data to base64 string
String base64Image = base64::encode(img, width * height * 2);
Firebase.setString(fbd, "/AAF_APP/cam_image", base64Image);
I allocate a buffer for the image data and read the image bytes into it. After that, I encode the image to a Base64 string and send it to Firebase.
// Send base64 image string to Firebase Realtime Databaseif (Firebase.setString(fbd, "/AAF_APP/cam_image", base64Image)) {
Serial.println("Image sent to Firebase");
} else {
Serial.println("Failed to send image to Firebase");
}delete[] img; // Free the allocated buffer}delay(100); // Avoid task starvation
After sending the image to Firebase, I free the allocated memory to prevent memory leaks.
Receiving Commands From K210
Lastly, I check for incoming commands from the K210 when voice commands are activated:
// RECEIVING COMMAND FROM K210 WHEN VOICE COMMANDS ACTIVATEDif (serial3.available()) {
String K210CMD = serial3.readStringUntil('n');K210CMD.trim();
Serial.println(K210CMD);
If there’s data available, I read the command sent by the K210, trim any whitespace, and print it for debugging.
Next, I handle specific commands received from the K210:
if (K210CMD == "M1_ON_ACK") {Firebase.setString(fbd, "/AAF_APP/motor_1", "1");}else if (K210CMD == "M1_OFF_ACK") {Firebase.setString(fbd, "/AAF_APP/motor_1", "0");}else if (K210CMD == "M2_ON_ACK") {Firebase.setString(fbd, "/AAF_APP/motor_2", "1");}Similarly for Every command....}
In this block, I check for specific acknowledgment commands sent by the K210. For example, if the K210 acknowledges that motor 1 is turned on with "M1_ON_ACK", I update the Firebase database to reflect that motor 1 is now active by setting it to "1". Similarly, I do this for other devices like motors, fly traps, rain sensors, lights, and the camera. This two-way communication ensures that the ESP32 stays in sync with the state of the devices.
This is the simple code explanation of the CODE for this project.
Battery Consumption:
Finally when i check the Battery consumption I got more than 1.5 days ( And i used 2 18650 3.7V batteries of 9.23Whr) and here is still some power left in the battery but i stopped the Calculation of the battery as per my assumption it will last for a week( roughly ).
Conclusion:
In conclusion, my Advanced Automated Farm (AAF) project is designed to revolutionize how farmers manage their fields by integrating smart automation with accessible technology. Through voice commands and a user-friendly app, I've created a system that simplifies complex tasks, offers real-time monitoring, and boosts productivity, all while being easy to install and maintain. By focusing on the needs of farmers, especially those with large land areas, I aim to make farming more efficient, sustainable, and accessible, paving the way for a more innovative future in agriculture.
To view the code and schematics of the project, please click on the GitHub icon below.
