Bao, AI Pet Dog

I wanted to build a Pet Robot Dog for my daughter. She loves dogs and would love to have a pet dog. As I can't get her an actual dog so thought I will get her something to play.
This robot dog responds to "Hi Bao" and "Hi Ping" for names (wakeup words).
Below are the features of Bao

1. Face Recognition: Bao will detect registered faces and greet them.

2. Offline Voice Commands: Bao responds to commands like go front, go back, turn etc.

3. Chat bot: Bao will be like Alexa, where you can interact with it.

4. Personal Assistant: Bao can be a personal assistant with alarms and reminders.

5. Friendly Face: Bao will use a display to show its mood and perform random actions.
   Bao is built on 3d printed custom chassis. The chassis is multi-layer. The below deck is for mobility and top deck is for power bank and AI.

6. Bluetooth Connectivity: This also has a Bluetooth connectivity so that Bao can be controlled by an app with Bluetooth from your mobile device. This is very useful as sometimes the voice commands fail due to noisy environments and accent issues.

I hope she will like it.

Components Required

• SIPEED MAIXDUINO FOR RISC-V AI + : This doesn't have display. Need to procure the display separately 
• ESP32 WROVER Dev Module
• Ai-Thinker VC-02 module
• SG90 9gm Servo *2 with required Horns
• N20 geared motors * 4
• N20 Wheels
• L298N H-Bridge motor driver
• Li-PO Batteries 18650 * 2
• Battery Holder for 18650*2 Lipo 
• Toggle Switch for main power
• Male Female Headers & Male Female Wires for header connections & Other Wires
• Heat Shrink tubes
• SD Card for storing captured images
• Type C and micro USB cables used for programming
• Perf Boards to create circuit board *3 (base deck, Second deck and for Maixduino connections.
• Misc. M3 nuts and bolts
• Leds and resistors 
• 3d Printed Parts for Chassis and Pan / Tilt of Head, N20 motor mounts

Below are images of the different build stages and here I would like to present the work that has gone in this project till now.

Construction

I started by designing a base plate as a chassis. It is 13*11 cm and 3 mm in thickness. This was a rough figure. Also made a bracket for n20 motors. Then 3d printed them. I also printed extra plates with 2 mm thickness that will act as the second deck of Bao's body.

Then using the mounting brackets and m3 nut bolts attached 4 * N20 motors. Here 2 motors in each side are connected parallelly. This small vehicle with this motor configuration is built like a tank and can climb up to 45 degrees gradient. The wheels were pre bought and has rubber grips for good friction.

Next step was to use a perf board for connecting and mounting all the components which are part of the base deck. Here we will have L298N driver, ESP32 Dev Module and AI thinker VC 02 kit which comes with its speaker and microphone. I also added the main power switch here. Used some tape and header wires to create a modular connection between the different boards so if something goes wrong, we will be able to replace it like a plug and play.  By the way, I am not sure why, but I did blow up 2 ESP32s :-)

Perf board with circuit, ESP32 mounted, VC 02 kit, L298N driver, speakers, switches

Here we can see fully assembled base deck. Do note the 4 metal screws next to each Tyre. These will serve as a mounting stud for the second level deck.  Which we will see next. Also added some jumpers which I can remove to disconnect individual boards while programming updates and won't need to dismantle it. I tried to follow a modular and a scalable design as much as possible.

Here we have now mounted the second deck plate. we will install the main battery bank which is 2 * 18650 Lithium Polymer (LiPo) cells connected in series. This has a peak voltage of 8.4 volts. This is connected to L298N driver board which a 5volt regulated output. This 5v output is used to drive all boards like Maixduino, ESP32 and VC 02 kit. In future if required we can have a separate buck convertor installed to have a proper regulated supply.

We also have the second perf board here to connect ESP32 header out wiring to the SG90 9gm servos. This also has the mounting bracket for Pan and Tilt Functionality. All the parts of the Pan and tilt functionality has been 3d printed.

We can now see the Pan and tilt head assembly connected. On this we need to mount the Maixduino AI Kit. I was not able to get this kit with a display due to unavailability. In the future it will have the display. The display will be used for showing emotions, configurations, setup and interactions. I have used another perf board to mount it, since this needs to be further connected to ESP32 for total hardware control based on AI logic. Like if a friendly face is recognized then BAO can show some happy emotions and run around or wag its tail.

 Here we see a completed assembly and Bao is "Ready to play". Do note the camera popping out from its right ear.

 My daughter gave it a face. This will be replaced by a Led display in the future.

Currently we can use Voice command like Go front, go back, turn Right, Turn left. We can add dog specific commands like wag your tail or speak / bark etc.

Bluetooth Connectivity

Using the Bluetooth capabilities of the ESP32, I have also connected to it using an APP. Currently this app provides some basic movement just like the voice commands.

We can further enhance it control functions, configure some settings etc.

Focus on AI

Let us have a quick look on the main functionality which is based on Maixduino. I flashed the default face recognition model (KPKG) file with the help of SIPEED OEM flasher. Then wrote a micropython program to run it. As I don't have a display I am using the laptop as a debugger. I am running MaixPy IDE to develop and debug all the code written for this board.

The program identifies a face i.e. face detection model. Then draws a rectangle on the face. As I don't have a display, I have used the micro-SD card connected to the board to save images and also for logging and debug info that I may need. The logic is to identify a familiar face (registered face) and send a command to ESP32 on its serial port. The ESP32 will further perform actions like wag tail, move Bao's head, make do some random movements etc.

The face detection is working very nicely and we can see below that all the faces are either obscured, have glasses on, at the very corner or even hazy still the face is correctly identified and rectangle drawn is also correct.

 The console is showing coordinates with respect to the camera where the face is identified, and it saves the image in the SD card.

below is the code running in MaixDuino. it can be found here in my GitHub repository.

Below is the MicroPython Code being run on MaixDuino.

Clone URL : https://github.com/Anupkg1/BaoPetDogMain.git

How does it all work?

1)When powered ON, some system checks are performed and Bao performs some demo commands. We can hear Bao greeting us "Ready to play" and visually we can see if all is ok. It will move front, go back, turn right and left

2) The basic physical action of Bao starts with the AI thinker VC 02 module. It has been preprogramed with voice commands. Once a voice command is identified it relays a trigger in the serial port to ESP32. ESP32 receives this signal and based on type of the signal further makes a decision on the next action. This action could be to perform going front, going back etc.
ESP32 then runs a subroutine using ON OFF signals to L298N motor driver.
L98N in turn drives the N20 motors. Here we are using the Offline voice model AI of the AI thinker module to perform most of the physical actions that are done by the pet dog.
Here using the Bluetooth capability of ESP32 we also connect to ESP32 and can perform some basic actions.

In the future we will adding more functionality to the Bluetooth.

3) Maixduino has the Vision model flashed on its memory. A micro python program runs in it to detect faces. On face detect it draws a rectangle around it and pinpoint the face location. Till here the functionality is already implemented.
In the future we will also be able to register a face and then Bao will be able to do face recognition. Here Bao can perform, some actions showing his excitement.

4) A display like an OLED or TFT will be used to show BAO's emotions. Since I was not able to get Maixduino with a display, I will be working on this in the future. This will give Bao its personality and is very critical to achieve my end goal for this project.

Roadmap and Things to do

1) Complete the face recognition code and give appropriate commands to ESP32 to perform actions.

2) Add Display to MaixDuino for showing emotions and can be used for configuration and alerts also. Like showing battery level.

3) Connect to Open AI service and add Speech to text and text to speech feature to give it Alexa like functionality.

4) Using the Open AI service or edge computing have some basic personal assistant features like setting alarms and reminders.

5) Make it look like a pet dog cosmetically and also smoothen out it movement.

lots of work still to do..............

Click on the GitHub Image to view or download code