In a previous article, we looked into the Maixduino Development kit and learned how to use it with the Arduino IDE. In this article, we will learn how to use Micropython in the Maixduino development kit. First, we will learn how to flash the Micropython firmware to the Maixduino and then how to set up Sipeed’s Maixpy IDE and use AI and Machine Learning with the board.
What is MaixPy?
MaixPy is a port of Micropython specifically for the K210 SoC. It not only supports generic MCU functions but also integrates hardware-accelerated AI machine vision and microphone array-related algorithms. Keep in mind that the Maixduino supports the MaixPy V1, but there is another version of MaixPy called MaixPy V4 which is for the newest Sipeed product called MaixCam and doesn’t support Maixduino. The main benefit of using MicroPython is that it is much easier and faster to do development. So let’s start with the basics and learn how to prepare the Maixduino board to use with MaixPy.
Installing the USB driver
As we know the Maixduino features two SoCs onboard a K210 AI SoC and an ESP32-Wroom Module. Since both of these doesn’t have any native USB support, the manufacturers used a USB to UART bridge for communication and firmware updates. Since we need two separate UARTs for the SoC, Sipeed chose a custom solution that uses a CH552 MCU with dual serial firmware. By doing this they were able to implement the communication between the computer and both of the onboard SoCs through a single USB port. Since they are using a custom solution, it is necessary to install any required drivers for it to be able to communicate with the computer.
In a Linux environment, we don’t need to install any drivers. The operating system will automatically detect the hardware and assign a generic driver that is already a part of the Linux system. All we need to do is note the port numbers. For that open the terminal window and type the following command ‘ls /dev/ttyUSB*’ and hit enter. A list of available USB devices and corresponding ports will be displayed. Note down the appropriate port number for further use. In Windows, it is necessary to install the specific driver. For that go to the USB driver download page, and download the driver file. There will be multiple files on the download page. The easier method would be to download the zip file with the setup in the name, extract it, and run the driver installer. It will automatically install the driver. Another way is to download the driver files from the download page and manually install the driver from the device manager. Once the driver is successfully installed and the MAxiduino is connected to the computer, open the device manager and expand the Ports (COM and LPT) section. You will find two comports that will be shown only when the Maxiduino is still connected to the PC. Note down the port number, by default the first port number will be for the K210 SoC and the second will be for the ESP32.
Installing the MaixPy firmware
So before coding, we must install the Maixpy firmware to the Maxiduino so that it will accept the micropython code and execute it. For the first make sure you have installed the driver as per the instruction above and note down the port number. In Linux and Mac OS execute ls /dev/ to see the port numbers and in Windows use the device manager. So to start with we need to download the precompiled firmware file. For that first go to the MaixPy firmware page. In there, select the latest version folder and in it, you will find multiple firmware files with either .bin or .kfpkg extensions. Use the following table to select the appropriate firmware for your application. Once selected download the firmware binary to your computer
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Once the firmware file is downloaded, the next step is to download the firmware flashing utility. For flashing or upgrading the firmware we are going to use the kflash_gui, for that download it from the kflash_gui download page. Extract the firmware and run the kflash_gui application. For Windows, it is recommended to run it using “run as administrator”.
Once the app is opened select the previously downloaded firmware. As you may observe the address range will be automatically populated and you don’t need to change it. Select the proper COM port(by default the first COM port of the two that will appear when the board is connected.) and click on download the flash utility will flash the firmware to the board and once it's done the board will reset and the MaixPy splash screen will be displayed on the LCD display.
Installing the MaixPy IDE
The next step is to install the MaixPy IDE. For that go to the MaixPy IDE download page and download the appropriate binary for your operating system. For Windows run the installer as usual and follow the onscreen instructions. For Mac use the DMG file and install the application. For Linux use the following commands to give permission and to install the IDE.
chmod +x maixpy-ide-linux-x86_64-0.2.2.run
./maixpy-ide-linux-x86_64-0.2.2.run
Once installed open MaixPy IDE, and select the model of the development board in the Tool menu. And click on the connect button in the bottom left corner of the IDE window. It will automatically connect to the Maixduino board.
When opening the IDE for the first time it opens with a test code already in it. So to test the code just click on the Run button located below the Connect button as shown below. The code will be loaded into the Maixduino board and executed. The example code initializes the camera module and display and continues to display the video stream from the camera on the display. You can also see the video preview in the MaiixPy IDE.
To stop running the code click on the stop button(same as the run button). To upload the files to the board use the Send file option from the Tools menu.
Using the Maixduino With Serial Terminal
After flashing the MaixPy firmware we can also use the Maixduino through any serial terminal without the need of the IDE if needed. For this, we can use any serial terminal that we are comfortable with, for example in Windows we can use Putt, mobaxterm , xshell , or mpfshell-lite and in Linux we can use the pyserial. The MaixPy IDE itself has a built-in serial terminal and we can also use that too. Sipeed recommends mpfshell-lite and more details about how to use them can be found on the mpfshell-lite page. If we want to run a micropython script, open the serial terminal, press CTRLl+E and paste the following code
import sensor, lcd
sensor.reset()
sensor.set_pixformat(sensor.RGB565)
sensor.set_framesize(sensor.QVGA)
sensor.run(1)
sensor.skip_frames()
lcd.init(freq=15000000)
while(True):
lcd.display(sensor.snapshot())
Press Ctrl+D on the keyboard to start running the code. In the code, you can see that we imported the necessary libraries for the camera and display using the import function. Later we initialised the camera and configured it. After that, we have initialised the display. Then using the while loop we display the live video feed from the camera on the TFT display.
Maixduino File System
The Maixduino has a total of 16MB of onboard storage and an SD card slot for external storage. The file system structure of the Maixduino is illustrated in the image below.
The internal storage is divided into three parts: namely the MaixPy.bin firmware area, the xxx.kmodel model area, and the file system area. As the name suggests the MaixPy.bin area is for the MaixPy firmware storage and the xxx.kmodel area usually starts at 0x300000 and is used for the trained AI model. For the generic filesystem, the Maixduino uses SPIFFS. The SD card must be formatted with FATFS for the Maixduino to be able to access it. If the trained model is larger than the xx.kmodel area, we can also use the SD card to store it.
Managing the File System and Uploading the Code
As we know in micropython all the scripts are stored as .py files, to modify the scripts it is necessary to have filesystem access for creating, editing or deleting those files. So with Maixduino, we have multiple ways to do these file handlings. The first method includes using the Micropython Editor (pye) editor which is built into the MaixPy firmware. We can use the serial terminal for the Pye editor. You can use os.listdir()to view the files in the current directory and pye("hello.py")to create a file and enter edit mode. After editing the file you can press Ctrl+S to save, and Ctrl+Q quit editing. You can find more details about the Micropython editor on the Micropython editor GitHub repo.
The second method is for when we are using the MaixPy IDE. In the IDE we can choose to save the opened file as boot.py from the tool menu to save the content in the IDE windows as the boot.py file.
The third method is to use uPyLoader. The uPyLoader gives you an FTP client like user interface where you can add, remove or execute very easily.
If you face any error when trying to transfer the file for the first time use the Init transfer files from the file menu.
Executing the Python Scripts
If you want to execute a Python script in the flash memory you can do that in various ways. The first way of course is through the serial terminal. For that first goto the directory where the file is stored using the os.chdir() command, for example, os.chdir("/flash"). Then you can execute the scripts using the import command, for example, import helloworld. This method is simple and easy to use, but it should be noted that the import command can only be used once. If we use the import command for the second time, the file will not be executed again.
Another way is to use the exec() function to execute. Here is a sample code snippet that shows the use of the exec() function.
with open("hello.py") as f:
exec(f.read())
Another way is to run the program from the MaixPy IDE as we have mentioned before. But with this method, the program is only temporarily running, it will not be saved on the device. You can also execute codes using the uPyLoader. After connecting, select the file and click execute the button to execute the file.
Automatically Running the Code on StartUp
The system will create the boot.py file and main.py in the /flash or /sd (preferred) directory. When booting, it will automatically execute boot.py first, and then main.py (if the SD card is detected, the file in the SD card will be executed). Edit the contents of these two scripts to achieve self-starting. If you write an infinite loop (While True) program in boot.py, main.py will not be able to run. Conventionally, the boot.py is mainly used to configure hardware and only needs to be configured once and the main.py is used for running the main program. So edit those scripts according to your needs.
Board Configuration File
To make the programming much easier we can use the board configuration file. It is nothing but a board definition file with pin mapping for easier understanding. Even though it is not necessary it will help the programming much easier when using the GPIO and onboard peripherals. To use it all you need to do is to run the config_maix_duino.py script once. Which will create a config.json file within the flash and can be used later. Using it is much easier, just import the board_info parameter from the config file and you are good to go. Here is an example where we are turning on the red element of the onboard RGB LED, the pin connected to the red is defined as LED_R in the config file. We can directly use it without checking the schematics for the exact pin number.
from Maix import GPIO from fpioa_manager import fm from board import board_info print(board_info.LED_R) fm.register(board_info.LED_R, fm.fpioa.GPIO0, force=True) led_r = GPIO(GPIO.GPIO0, GPIO.OUT) led_r.value(0)
Similarly, all the pins are mapped in a more convenient manner and can easily be used in our code. To know the exact pin mapping you can either open the config_maix_duino.py or the config. You can also refer to the image below, where it shows the Arduino like pin map.
Maixduino MicroPython Bacis
Covering all the basics of Maxduino MicroPython would take multiple articles and nonetheless Sipeed has detailed documentation about that. You may refer the the Sipeed’s Maixduino Specific MaixPy basics documentation for more details about it.
Maixduino AI Neural Network Applications
As we know the main selling points of the K210 AI SoC that is used in the Maixduino are its AI capabilities including Convolutional Neural Network based machine vision and machine learning. So to understand the AI capabilities of the Maixduino board we will look at some AI examples using the pre-trained AI models Sipeed provides.
Face Detection Example
As the name suggests, in this example we will look at the face detection AI model that Sipeed provided. The model will find a face in a picture and frame it and it uses YOLO V2 to detect the faces. To use it make sure to flash the normal or default MaixPy firmware to the Maixduino as instructed in the MaixPy firmware installation section. The next step is to download the pre-trained AI model. For that go to Sipeed’s AI model download page and download the face_model_at_0x300000.kfpkg model file. Once the file is downloaded, download it to the Maixduino’s flash memory using the kflash_gui utility or put it in an SD card. Since reading from the flash memory is always faster than reading from the SD card, it is recommended to load the AI model file to the flash memory, as long as the model file size is within the limit.
Once it's done we can move forward with the micropython script. You can load the AI model using task = kpu.load(0x300000). In this, the memory address is specified since the model is stored in the flash memory. If you are using the SD card to store the AI model, you can load the AI model into the script using task = kpu.load(0x300000). Then you can set the anchor points as anchor = (1.889, 2.5245, 2.9465, 3.94056, 3.99987, 5.3658, 5.155437, 6.92275, 6.718375, 9.01025). The anchor point parameter is consistent with the model parameter. For each model, this parameter is fixed and bound to the model (determined when the model is trained). It cannot be changed to other values. Later you can initialize the kpu network object using kpu.init_yolo2(task, 0.5, 0.3, 5, anchor). Since this model is used YOLO V2, we used init_yolo2 to initialize the model. This function has a total of five parameters. Those parameters are:
- kpu_netKPU.load(): kpu network object, that is, the returned value of the loaded model object
- threshold: Probability threshold. The result will be output only if the probability of this object is greater than this value. The value range is: [0, 1]
- nms_value: box_iou threshold, in order to prevent the same object from being framed by multiple boxes, when two boxes are framed on the same object, if the ratio of the intersection area of the two boxes to the total area occupied by the two boxes is less than this value, the box with the highest probability is selected.
- anchor_num: The number of anchor points, fixed here as len(anchors)//2
- anchor: As mentioned earlier this parameter is fixed and bound to the model
After the initialisation, you can input the image data and run the model as follows.
code = kpu.run_yolo2(task, img)
This will analyse the given image data and will give you the result. Here is a full example code in which the Maixduino will detect the face from the camera feed in real-time and creates a frame on the preview displayed in the LCD display.
import sensor, image, lcd, time
import KPU as kpu
import gc, sys
def lcd_show_except(e):
import uio
err_str = uio.StringIO()
sys.print_exception(e, err_str)
err_str = err_str.getvalue()
img = image.Image(size=(224,224))
img.draw_string(0, 10, err_str, scale=1, color=(0xff,0x00,0x00))
lcd.display(img)
def main(model_addr=0x300000, lcd_rotation=0, sensor_hmirror=False, sensor_vflip=False):
try:
sensor.reset()
except Exception as e:
raise Exception("sensor reset fail, please check hardware connection, or hardware damaged! err: {}".format(e))
sensor.set_pixformat(sensor.RGB565)
sensor.set_framesize(sensor.QVGA)
sensor.set_hmirror(sensor_hmirror)
sensor.set_vflip(sensor_vflip)
sensor.run(1)
lcd.init(type=1)
lcd.rotation(lcd_rotation)
lcd.clear(lcd.WHITE)
anchors = (1.889, 2.5245, 2.9465, 3.94056, 3.99987, 5.3658, 5.155437, 6.92275, 6.718375, 9.01025)
try:
task = None
task = kpu.load(model_addr)
kpu.init_yolo2(task, 0.5, 0.3, 5, anchors) # threshold:[0,1], nms_value: [0, 1]
while(True):
img = sensor.snapshot()
t = time.ticks_ms()
objects = kpu.run_yolo2(task, img)
t = time.ticks_ms() - t
if objects:
for obj in objects:
img.draw_rectangle(obj.rect())
img.draw_string(0, 200, "t:%dms" %(t), scale=2)
lcd.display(img)
except Exception as e:
raise e
finally:
if not task is None:
kpu.deinit(task)
if __name__ == "__main__":
try:
main( model_addr=0x300000, lcd_rotation=0, sensor_hmirror=False, sensor_vflip=False)
# main(model_addr="/sd/m.kmodel")
except Exception as e:
sys.print_exception(e)
lcd_show_except(e)
finally:
gc.collect()
After running this code you can see the result on the LCD display as shown below
You can download the python script for this example from our GitHub Repository. https://github.com/Circuit-Digest/Maixduino-AI-Projects/tree/main/Face%20Detection
Object Classification Example
For this example, we need to load the minimal firmware to Maixduino since the model itself is a little bigger in size. This model can classify up to 1000 different objects, since the bigger model. After flashing the minimal firmware, download the mobilenet_0x300000.kfpkg model from the download page. Once the file is downloaded, download it to the Maixduino’s flash memory using the kflash_gui utility. Also, download the labels.txt file and save it to the file system. Since the minimum firmware does not support IDE, you can use uPyloader to download the file to the flash. We also need to reduce GC heap size. To do so just run the following script.
from Maix import utils import machine utils.gc_heap_size(256*1024) machine.reset()
Once it's done we can move forward with the main micropython script. Use the following script.
import sensor, image, lcd, time
import KPU as kpu
import gc, sys
def main(labels = None, model_addr="/sd/m.kmodel", lcd_rotation=0, sensor_hmirror=False, sensor_vflip=False):
gc.collect()
sensor.reset()
sensor.set_pixformat(sensor.RGB565)
sensor.set_framesize(sensor.QVGA)
sensor.set_windowing((224, 224))
sensor.set_hmirror(sensor_hmirror)
sensor.set_vflip(sensor_vflip)
sensor.run(1)
lcd.init(type=1)
lcd.rotation(lcd_rotation)
lcd.clear(lcd.WHITE)
if not labels:
raise Exception("no labels.txt")
task = kpu.load(model_addr)
try:
while(True):
img = sensor.snapshot()
t = time.ticks_ms()
fmap = kpu.forward(task, img)
t = time.ticks_ms() - t
plist=fmap[:]
pmax=max(plist)
max_index=plist.index(pmax)
img.draw_string(0,0, "%.2f\n%s" %(pmax, labels[max_index].strip()), scale=2, color=(255, 0, 0))
img.draw_string(0, 200, "t:%dms" %(t), scale=2, color=(255, 0, 0))
lcd.display(img)
except Exception as e:
sys.print_exception(e)
finally:
kpu.deinit(task)
if __name__ == "__main__":
try:
with open("labels.txt") as f:
labels = f.readlines()
main(labels=labels, model_addr=0x300000, lcd_rotation=0, sensor_hmirror=False, sensor_vflip=False)
# main(labels=labels, model_addr="/sd/m.kmodel")
except Exception as e:
sys.print_exception(e)
finally:
gc.collect()
As you can see, at first we imported all the necessary modules, including sensor, image, LCD and time libraries. Along with that we have also imported the KPU neural network module, garbage collector and system modules. Later you can see the function called main is being declared. This function handles all the image processing and neural network procedures. When the script is run it will first read the labels.txt files is first read and the listed labels from the file is loaded into an identifier called labels. After that the main function is called with five arguments. The first argument points to the labels identifier while the second argument points to the model's memory location. The third argument is to set the display rotation and the fourth and fifth arguments are used to set the horizontal mirror and vertical flip of the camera image.
The main function initialises the camera, display and neural network modules using the provided arguments. After initialisation, the main function will get the image from the camera, run the object identifier model on it and if an object id detected it draws a frame around it and prints the corresponding label. The following line of code is used for object detection.
fmap = kpu.forward(task, img) plist=fmap[:] pmax=max(plist) max_index=plist.index(pmax)
Once the object is detected the result is printed using the following line of code.
img = img.draw_string(0, 0, "%.2f : %s" %(pmax, labels[max_index].strip()), color=(255, 0, 0)) lcd.display(img, oft=(0,0)) print(fps)
As you can see the draw_string function is used to add the label to the image prior to displaying it on the screen. Here is the demonstration of the above script.
You can download the python script for this example from our GitHub Repository. https://github.com/Circuit-Digest/Maixduino-AI-Projects/tree/main/1000%20Object%20Detection
What is MaixHub and how to use it?
Sipeed also has an online platform called Maixhub AI models and training. The Maixhub not only allows you to download pre-trained models but also gives you the option to train your own model. To use it go to the Maixhub page and register a new account, If you are already registered log in to the Maixhub platform.
If you click on the Models tab on the top of the page it will redirect you to the page where you can find a ton of pre-trained models to try out. If you want to use them you can open that particular model page and download it. Most of the models would also have the required instructions on the model page itself.
Training Your Own AI Model Using MaixHub
To train sour own AI model first go to the training page on MaixHub.Click on Create to create a new project. Give a project name and select the type, whether it's image classification or image detection. If you just need to identify objects, then select image classification. If you need to identify object categories and output the coordinates of the recognized objects, then choose image detection. It's recommended to train an image detection model first. Image detection training involves dataset annotation, so mastering image detection training means you also master image classification.
Once the project is created, the next step is to create a dataset. The created dataset can be reused for other projects if needed. Later select the dataset we have created and click confirm.
The next step is to collect training images. We can either upload images, import from datasets, import from device or copy from other datasets. And there is also an option for automatically generate annotated pictures from video.
Images uploaded to MaixHub can be annotated within. Select upload images or compressed packages and remember to click start upload.
Once the images are uploaded, we can move forward with annotation. Annotating in MaixHub is very easy. First, create labels, and then click on New button or press w on the keyboard to annotate. Click save or press s for saving the annotation. For annotating the next image either click on next or press d on the keyboard.
Once all the images are annotated, let’s move to the training. For that click on the Create Task from the left panel menu.
In the task creation page choose the nncase as model for k210, which is the main SoC in the Maixduino, then create a training task and wait for the task to complete.
After the model training is completed, click on deploy.In the deployment page choose manual deployment, and click on download to download the model file. The model file with the extension .kmodel will be downloaded to your computer. Use this model on your project.
Now as we gone through the process of using the Maixduino board for Ai projects with MaixPy enjoy creating new projects.
