Edge AI-Powered Surveillance: Intelligent Threat Detection

Brainstorming for Required configurations

We spent a significant amount of time in determining the specifications and features we would want to incorporate so as to ensure the most optimal solution.

Implementation of all the peripherals and features is not possible due to time and cost constraints. Considering this, a list of most important ones is included in developing the test plan for the same. 

After extensive iterations and careful selection of microcontrollers, two nodes have been established for the project. The first node, named the TACT (Threat Analysis and Crisis Tracking) sensor node, utilizes a Sipeed Maixduino procured from Digi-Key. This node is designed for deployment at remote locations to detect potential threats. The second node is a mobile slave unit composed of an NodeMCU(ESP8266) equipped with LoRa technology, positioned with police officers within a 10 km radius of any detected incident threat. 

TACT Node- Threat Analysis and Crisis Tracking

• Camera-Based Surveillance - Equipped with an integrated camera for capturing images or video of detected threats, which can be processed locally for faster threat assessment.

• Edge AI-Based Threat Detection - Uses trained machine learning model on the sensor node itself to detect potential threats such as weapons or suspicious behaviour in real-time, minimizing response delays.

• Real-Time Alerts using LoRa and Wi-Fi - Supports both LoRa for long-range, low-power communication and Wi-Fi for faster data transfer, enabling flexibility in network deployment.

• Geolocation Tracking - Monitors and reports the precise location and Node-ID of each node, helping to pinpoint the exact location of detected threats for faster intervention.

• Scalable architecture - Multiple nodes can be deployed to cover larger areas, forming a scalable network that communicates wirelessly to ensure robust, distributed threat detection.

Mobile Slave Node

• Robust and Secure Communication - LoRa integrated with NodeMCU(ESP8266) microcontroller board ensures secure and interference-resistant communication, essential for maintaining uninterrupted data flow even in challenging environments.

• Real-time threat tracking - The node provides officers with real-time data on detected threats, displaying the Node-ID and location on an OLED screen for quick and precise identification of the threat source.

• Single-Multi point communication - Use of LoRa ensures each mobile slave node can capture information simultaneously from multiple sensor node connected on the LoRaWAN network.

• Compact and Portable Design - The node is built using an NodeMCU(ESP8266), making it lightweight and portable. It can be easily carried by police officers without adding bulk to their equipment.

Component Selection

Sipeed Maixduino RISC V Microcontroller board

The Sipeed Maixduino RISC-V Kit is an excellent choice for this Edge AI-powered surveillance project due to its specialized features and capabilities, which align well with the objectives of the proposed real-time threat detection and response system:

RISC-V Architecture for Efficient AI Processing: 

• Open-Source and Customizable Architecture: RISC-V is an open-source instruction set architecture (ISA), allowing developers to customize it for specific applications like Vision-based-learning. The Maixduino’s K210 processor running at 400 MHz, based on RISC-V, includes custom extensions optimized for machine learning tasks. This makes it highly efficient for running Machine Learning models, such as TensorFlow Lite, on a microcontroller without unnecessary overhead.

• Low Power Consumption: RISC-V processors are known for their energy efficiency. The lightweight nature of the instruction set allows for simpler designs, reducing the power required to execute tasks.

• Parallel Processing Capabilities: RISC-V allows for parallel processing, which is essential for handling AI workloads efficiently. The dual-core design in the K210 processor enables it to handle multiple tasks simultaneously—such as processing camera input, running AI inference, and handling data transmission via LoRa and Wi-Fi without overwhelming the system.

• Integrated Camera Support and LCD support: The board contains 24P 0.5mm FPC connector allowing us to easily integrate camera modules to monitor live video feed for weapon detection. Similar 24P 0.5mm FPC connector is present for supporting LCD interfacing using SPI.

• TensorFlow Lite Compatibility: It supports TensorFlow Lite, allowing us to deploy lightweight deep learning models for weapon detection on the microcontroller itself. This enables efficient real-time inference on the device.

• Integrated Wi-Fi and Bluetooth: The Maixduino supports Wi-Fi (2.4 GHz, 802.11 b/g/n) and Bluetooth 4.2, enabling wireless communication for data transmission and remote access. This is crucial for connecting surveillance nodes to central systems or cloud platforms without the need for physical cables, enhancing deployment flexibility in different environments. Wi-Fi allows utilization of MQTT protocol to connect to a remote Node-RED dashboard.

• SPI and UART Functionality: The LoRa SX1278 module communicates using SPI, enabling long-range, low-power communication between devices. The Maixduino’s SPI interface allows seamless integration of the LoRa module for transmitting threat data from edge devices to a central gateway or cloud-based platform. The SPI interface also enables
  the use of an SD card for local data storage. This allows the system to retrieve the threat detection model stored on SD card, allowing creation and dynamic linking of modular micropython files. Also, the UART interface allows for easy debugging of code and real-time monitoring of the system's operation.

Sipeed Maixduino Pinout

LoRa SX1278

LoRa SX1278 module despite being the cheapest LoRa module perfectly fits for the project due to its robust features, high compatibility and capabilities that align with the project goals.

• Long-Range Communication: The SX1278 operates using LoRa (Long Range) modulation, enabling reliable communication over several kilometres (up to 8-10km in open environments). This is crucial for a distributed surveillance system where edge nodes (like cameras and sensors) might be deployed far from the central control system.

• Robust and Secure Data Transmission: The LoRa SX1278 offers secure and interference-resistant data transmission. It uses spread spectrum technology, which provides better immunity to interference, noise, and multipath fading.

• Supports Point-to-Multipoint Communication: The SX1278 allows for point-to-multipoint communication, which is especially advantageous in this project, where multiple edge nodes (such as cameras and sensors) need to transmit data to a number of mobile devices carried by policemen.

• License-Free ISM Band Operation: The SX1278 operates in the license-free ISM band (typically 433 MHz or 915 MHz), which makes it cost-effective for deployment.

• High Sensitivity and Penetration: With a high sensitivity of -148 dBm and +20 dBm power amplifier, the SX1278 can send and detect weak signals among distant nodes, ensuring reliable communication even in challenging conditions (e.g., behind walls, in basements, or across large areas).

 LoRa XL1278-SMT contains SX1278

NodeMCU for Gateway Node

• Microcontroller capability: The NodeMCU is equipped with a microcontroller (ESP8266) that can handle basic processing tasks and control operations in your surveillance system. These operations include retrieving Node-ID and location of the threat detecting sensor node.

• LoRa Integration: Readily available LoRa libraries could be used enable long-range, low-power communication between devices, which is crucial for remote surveillance applications.

• Cost-Effective Solution: Costing at just ₹150 the NodeMCU is an affordable microcontroller platform, making it accessible for cheap monitoring solution when equipped with LoRa. This allows creation of a great monitoring device which could be utilized in developing mobile solutions for police and defence personnel.

NodeMCU pinout

Circuit Diagram

TACT Node Circuit Diagram

  Mobile Slave Node Circuit Diagram

Architecture

Code

To test the working of the entire system, micropython code and modules has been written for the TACT node and have been dumped onto the MicroSD-XC card. These files are dynamically linked to the Kendryte model developed (which is also dumped on the MicroSD card). For the master slave node code written in Arduino IDE has been dumped on the NodeMCU.

Note: For integrating the LoRa code on NodeMCU, library developed by Sandeep mistry has been utilized.

TACT Node:

• Detect Threats: Use cameras based detection with trained AI model to identify potential threats.

• Send Data Over LoRa: Transmit threat alerts in terms of Node-ID, weapon type, location and threat level to the Mobile slave node via LoRa.

• Send Image and threat data Over Wi-Fi: Detected threats image along with the above stated threat alerts are also sent via Wi-Fi to the Node-RED dashboard on a common Wi-Fi.

Master Slave Node:

• Receive Threat Alerts: Collect threat alerts sent from the TACT node via LoRa integrated on the NodeMCU and display these alerts on the OLED display.

Node-RED dashboard:

• Image and threat parameters received: NodeMCU Wi-Fi is connected to the Node-RED dashboard on a common network IP address and sockets are utilized to receive image and threat alerts over Wi-Fi.

• Intuitive UI: An intuitive UI for the Node-RED dashboard has been developed showcasing the image of the detected threat along with different threat alerts sent by TACT node through Wi-Fi.

• Multi-point Accessibility: The Node-RED allows large number of devices to be connected using the same server IP address.

Reference Libraries and Documentation:

1. https://github.com/sandeepmistry/arduino-LoRa

2. https://github.com/adafruit/Adafruit-GFX-Library

3. https://github.com/adafruit/Adafruit_SSD1306

4. https://github.com/kendryte/kendryte-doc-datasheet

5. https://github.com/sipeed/maix_train

6. https://github.com/sipeed/MaixPy-v1_scripts

TACT Node Explanation

Library Imports and Initialization: 

• Libraries used: sensor, image, lcd, time, KPU, machine, gc, sys, fpioa_manager, maix, micropython, XL1278(custom made by us).

• Declare constants and variables: define input size of image, labels (‘rifle’, ‘handgun’) and threshold.

• Setting up LoRa: Predefined NodeID, location, pin assignment for LoRa using SPI at 10 MHz.

• Wi-Fi Credentials: SSID, password, socket.

LoRa Communication (sendLora): 

• Payload Setup: Contains threat information (severity, weapon type) and concatenate with NodeID and location and send payload via LoRa to receiver (mobile slave node).

Establish Esp32 Wi-Fi: 

• Use predefined credential with wifi.connect() to connect to SSID stated above.

• Retry upto 5 times if connection fails.

Communication over UART (Comm): 

• The Comm class handles sending detection results over UART, encoding the detected objects' details such as position, probability, and label.

Image Transmission Over Wi-Fi (send_image_over_socket):

• Sends a captured image over Wi-Fi to a specified IP address and port.

• Compresses the image and sends it in packets (in form of bytes) over a socket connection, with error handling for broken sockets.

Payload Transmission Over Wi-Fi (send_payload_over_socket): 

• Sends a payload consisting of the NodeID, threat level, weapon type, and location via socket over Wi-Fi.

• Connects to a specific address and sends data in byte format.

Object Detection and Main Function: 

• The YOLOv2 model is loaded and used for object detection. Detected objects ('handgun', 'rifle') are classified and their bounding boxes are drawn on the image.

• The main function continuously captures frames from the camera and processes them for object detection. Detected objects trigger LoRa or Wi-Fi communication, sending both the image and threat information to the receiver using the different functions and custom libraries stated above.

Maixduino Code Link

Maixpy IDE

Contribution to LoRa SX1278 micropython library:

Using Sandeep Mistry’s library for LoRa SX1278 for Arduino we have developed the micropython counterpart compatible with Maixduino architecture. Logic analyzer based testing to find the register address being written and various iterative practices to ensure high compatibility.

Testing LoRa using Logic Analyzer

Mobile Slave Code Explanation

Library Imports: 

• Libraries used: spi, wire, Adafruit_gfx, Adafruit_SSD1306, LoRa (by Sandeep Mistry).

• Setting up OLED screen dimensions.

SSD1306 class: 

• An instance ‘display’ of this class is declared for setting up the OLED screen width, height and reset.

Setup (): 

• Initialize LoRa, SSD1306 OLED.

• LoRa uses license-free radio bands allowing cheap deployment. 433 MHz operation allows longer distance operation for LoRa with a directional antenna. Also it is less crowded than the 2.4 GHz band.

Loop (): 

• The data sent from the TACT node is a list which is parsed on reception by the LoRa on the mobile slave node.

• Segregation of parsed data and storing it in a array comprising various parameters such as Node-ID, weapon type, threshold and location.

• Information is then displayed on the OLED display and is updated upon latest threat detection data.

• To avoid data corruption observed earlier, a new function named classifyAndStore() is created.

ClassifyAndStore():

1. Conditional statements are used to prevent data corruption when displaying information on the OLED screen.

   1. This ensures that the correct data values are assigned to their respective variables and are not accidentally overwritten or misallocated to the wrong variables.  

NodeMCU Code Link

Arduino IDE

Node-RED Server dashboard Explanation

There are numerous ways to establish an IoT system with database. The Node-RED provides an open source functionality to automate and scale the IoT system while maintaining the cost component. The Node-RED establishes a socket connection with the IP and port to retrieve the image in the form of bytes from the TACT node. The received image is stored in a particular folder on the server and displayed on the dashboard.

Since it’s only for demonstration purposes we have not used a database server but our PC acts as a server and stores the threat images A different port is used to receive the payload data, this data is further displayed on the dashboard. Data received is converted into JSON format and then stored into a CSV file. This file contains details and location of threat detected along with timestamps.

Currently the dashboard is locally running on: https://192.168.1.231:1880/ui

Node-RED Code Link

Our Node-RED flow:

Gets Images from TACT Node and stores on server

Display Images on Dashboard

 Gets Payload data from TACT Node and save in a CSV file

Node-RED dashboard Styling

 For further information about use of Node-RED flows together refer the GitHub link provided above.

Dataset Creation and Model Training

The YOLOv2model was trained on a dataset of 2,083 images, categorized into two classes: ‘rifle’ and ‘handgun’. The dataset was generated using Roboflow, which annotated the images with bounding boxes. These images were captured from a live video and split into a 70:30 ratio for training and testing. To enhance detection efficiency, several preprocessing techniques were applied, including dynamic cropping, auto-orientation, and resizing the images to 224x224 pixels

Image preprocessing

—optimal for YOLOv2’s MobileNetV1 backbone. This backbone was chosen due to its compact size, ideal for deployment on the Maixduino K210, which has 6MB of general-purpose RAM. Both the model weights and parameters are stored within this memory limit.

After creation of dataset, it has been downloaded from Roboflow in Pascal VOC format. Our institute’s Tesla V100 GPU has been used to train this dataset and hence a docker container has been created with all required dependencies on the GPU server. A jupyter notebook named “CreateDataset” has been utilized to convert Pascal VOC format rearranged into two parts one containing the images and the other containing the bounding box related text (XML files), this is done to avoid errors during training. (standard procedure for training a Maixduino model). This is followed by training using Yolov2 (You look only once) algorithm. Yolov2 is a single state real-time object detection model which improves upon its predecessor Yolov1. Our Yolo model utilizes mobilenetV1 backbone (only MobilenetV1 is supported on the Maixduino board). The referenced jupyter notebook has been modified to train the final model. The model has been trained for 50 epochs, reaching peak detection accuracy of 90.9% at the final epoch, indicating optimal performance at this point. 

Python Code Link

Python Code

Validation Accuracy and Validation Loss vs Epoch

Working Setup

This TACT node system can be deployed in presence of any 2.4GHz Wi-Fi network and the LoRa functionality just requires presence of another LoRa module present with Mobile Slave node.

TACT Node Setup:

Initially the Maixduino board has been flashed with v0.6.2_4 version bin file using K-flash GUI tool to install kendryte firmware allowing connection to Maixpy IDE.

The final setup of the TACT node involves connecting the battery and LoRa module to the Maixduino board. An SD card is used to store the Kendryte model, along with MicroPython modules and the main code. Upon execution, the code is dynamically linked to facilitate accurate detection of objects like rifles and handguns. Once a threat is detected, the payload and corresponding image are transmitted to the mobile slave node and displayed on the Node-RED dashboard using LoRa and Wi-Fi protocols, respectively.

TACT Node Components

Final Assembly of TACT Node

Mobile Slave Node Setup:

The final setup includes a NodeMCU board, powered by an independent battery and connected to a LoRa module and an OLED display. When the NodeMCU successfully receives the threat payload via LoRa, the information is displayed on the OLED screen. This setup enables remotely located policemen to quickly access crucial details such as the Node-ID and the location of the TACT node that detected the threat.

Mobile Slave Node Components

Final Assembly of Mobile Slave Node

Node-RED Dashboard:

Node-RED Dashboard UI

Other Photos:

Dashboard Visible on Different Devices Simultaneously

Excel sheet where Data from Payload is being stored

Images simultaneously being stored on server as they arrive along with timestamp

Team Members

1. Kartik Khandelwal

2. Viren Sharma

Thank You!!

Click on the GitHub Image to view or download code