An open-source project called ESP32 iPhone Agent lets an AI model control a real iPhone using a low-cost ESP32-S3 development board. Created by Henry Yuan, the project can perform actions such as tapping, swiping, typing text, drawing paths and pressing keys. It can also return to the home screen or complete a task. The Seeed Studio XIAO ESP32-S3 board costs around $7, and the project does not require jailbreaking the iPhone. However, the updated version requires an iOS app called ESP32 Agent.
The updated system combines AirPlay screen mirroring, USB HID and AI vision to control the iPhone. The XIAO ESP32-S3 creates its own Wi-Fi hotspot and acts as an AirPlay receiver for the iPhone's mirrored screen. The ESP32 Agent app runs on the iPhone and works with the board to process screen images using an AI vision model and determine the next action. Commands are sent back to the ESP32-S3, which acts as a USB mouse and keyboard to interact with the phone. Initial setup requires a Mac with Xcode to flash the board and build and install the iOS app. A free Apple ID can be used to sign the app, but the signature must be renewed every seven days.
What makes this ESP32-S3 iPhone automation project interesting is that it uses an inexpensive USB input device and AI vision to let an AI agent interact with an iPhone through its normal interface, without modifying iOS. The project is open source under the MIT licence, with its firmware and software available on GitHub. The updated setup requires the iOS app and initial access to a Mac with Xcode. Some limitations of the earlier version may not apply to the updated implementation, so details such as typing support, screen-mirroring restrictions and AssistiveTouch behaviour should be checked against the current code before publication. The project offers an unusual way to explore AI-powered smartphone automation using low-cost hardware.
Add a comment
No account needed. Your comment appears once a moderator has reviewed it.