An open-source project called ESP32 iPhone Agent lets an AI model control a real iPhone using a low-cost ESP32-S3 development board. Created by Henry Yuan, the project can perform actions such as tapping, swiping, typing text, drawing paths and pressing keys. It can also return to the home screen or finish a task. The setup costs around $7 for the Seeed Studio XIAO ESP32-S3 board and does not require jailbreaking the iPhone, installing an app on it or using an Apple developer account.
The system combines AirPlay screen mirroring, USB HID and AI vision to control the iPhone. A Mac or Linux computer runs UxPlay to receive the mirrored screen, along with a Python server and the AI agent logic. The AI examines the screen image and decides which action to take next. The computer then sends commands over Wi-Fi to the ESP32-S3, which is connected to the iPhone's USB port and acts as a mouse and keyboard. With AssistiveTouch enabled, mouse clicks work as taps, while mouse movements control the on-screen pointer. The phone and computer connect through the same Wi-Fi network, and the ESP32-S3 joins that network to receive commands.
What makes this ESP32-S3 iPhone automation project interesting is that it uses a simple USB input device to let an AI agent interact with an iPhone through its normal interface, without modifying iOS. The project is open source under the MIT licence, with its firmware and software available on GitHub. However, it has some limits: typing supports ASCII characters only, AssistiveTouch tracking speed may need recalibration, and some screens, including passcode entry, Apple Pay and DRM-protected video, cannot be mirrored. The iPhone must also remain unlocked because screen mirroring stops when it locks. Despite these limits, the project offers an unusual way to explore AI-powered smartphone automation with inexpensive hardware.
Add a comment
No account needed. Your comment appears once a moderator has reviewed it.