ESP32 iPhone Agent: A $7 Chip Lets AI Control a Real iPhone

A $7 microcontroller board can now give an AI model eyes and hands on a real, unmodified iPhone. No jailbreak, no Apple Developer Account, and after initial setup, no computer either. The project is called ESP32 iPhone Agent, built by a developer going by Henry Yuan (GitHub handle onehumanbeing), and it's doing the thing every viral GitHub repo dreams of: going from 12 stars to over 3,000 in about a week.

I went through the README, the Hackaday writeup, and a pile of coverage sites to figure out what's real here and what's getting rounded up for clicks. Short version: it's real, it's clever, and it's also a lot more limited than the headlines make it sound.

What is the ESP32 iPhone Agent, exactly?

It's an open-source rig that lets any OpenAI-compatible vision model look at an iPhone's screen and control it like a person would, by tapping, swiping, and typing. The whole thing runs on a Seeed Studio XIAO ESP32S3 board that costs about $7 (the plain 8MB PSRAM version, not the Sense or Plus variants, that distinction matters if you're ordering one).

The board does two jobs at once. It mirrors the iPhone's screen over AirPlay so the AI can see what's happening, and it pretends to be a USB mouse and keyboard so the AI can act on what it sees. An iPhone app handles the thinking: it grabs frames, sends them to a vision model, and turns the model's decisions into mouse and keyboard commands.

How does it see the screen without jailbreaking anything?

The ESP32-S3 opens its own Wi-Fi hotspot and shows up in your iPhone's Screen Mirroring list as "ESP32 Agent," same as any AirPlay receiver. You mirror your phone to it like you would to an Apple TV. The board grabs that H.264 video and forwards it over Bluetooth to the companion iOS app, which decodes it and hands screenshots to the vision model.

This is standard Apple functionality being repurposed, not a bypass. Nothing on the phone is exploited. That's also why it has real limits: iOS refuses to mirror passcode entry, Apple Pay sheets, and DRM-protected video, by design. The agent is blind during exactly the moments Apple doesn't want a screen recorded.

How does it actually tap and type?

Once the ESP32-S3 is plugged into the iPhone over USB (USB-C models connect directly, Lightning phones need a Lightning-to-USB Camera Adapter), it enumerates as a standard USB HID mouse and keyboard. iOS has supported external mice since iOS 13, through AssistiveTouch, so you turn that on and suddenly mouse movements become an on-screen pointer, clicks become taps, and drags become swipes.

The iOS app runs a planner/actor loop: look at the screenshot, decide the next action, send it, check if the screen changed the way it expected, and retry if it didn't. It tracks pointer position with self-correction, since USB mice report relative movement and drift happens. Every run gets logged with a step-by-step replay, which is a nice touch if you're debugging why your agent kept tapping the wrong button.

What do you actually need to build one?

• A Seeed Studio XIAO ESP32S3, roughly $7
• An iPhone on iOS 18 or later with cellular data (the phone's Wi-Fi gets taken over by the board's hotspot while the agent runs, so internet access has to come through the cellular connection)
• Any OpenAI-compatible vision API: a cloud provider, or a model you run yourself
• A Mac with Xcode, but only once, to flash the board's firmware and build the SwiftUI app

That last point is where the marketing ("no Mac!") gets a little ahead of the README. You need a Mac and Xcode to set the thing up. You don't need one running in the background while the agent works, and that's the actual improvement over earlier automation rigs that needed a laptop babysitting a screen-mirroring session the whole time. Worth knowing before you promise your team a zero-computer setup.

What can't it do yet?

This is the part the headlines skip, and it's the part I'd actually read before building one.

• Resolution and frame rate are rough. Bluetooth Low Energy tops out around 300-500 kbps, so the mirrored screen is capped at 480x1040 at 5 frames per second or less. Fine for reading a to-do list, not great for anything fast-moving.
• It can't see protected screens. Passcode entry, Apple Pay, and DRM video are invisible to it, because iOS won't mirror them to anyone.
• Keyboard input is ASCII only. No emoji, no non-Latin scripts, at least not yet.
• Portrait only. Apps that run sideways aren't supported in the current build.
• No internet on the phone's own Wi-Fi while connected, since that radio is busy being the board's hotspot.

None of these are fatal, but they rule out a lot of "fully autonomous iPhone assistant" fantasies people are projecting onto this project.

How does this stack up against other ways to automate an iPhone?

The honest pitch for this project isn't "better than everything." It's "almost as capable as a cloud phone farm, for the price of a cheap USB cable, if you can live with 5 fps."

Is it actually safe to use?

Mostly, yes, with the usual open-source caveats. The code is MIT licensed, except for the AirPlay portion ported from UxPlay, which is GPL-3.0, so the compiled firmware binary ends up GPL-3.0 as a whole. Read that before you build it into a commercial product.

Security-wise, it needs physical possession of your phone and a USB connection, it's not a remote attack tool. The bigger thing to watch is what you point the agent at. It has full AssistiveTouch control of your phone, so a buggy prompt or a bad vision call could tap something you didn't want tapped. I wouldn't point this at a banking app on day one.

Should you build one?

If you're into home automation or just like weird hardware projects, yes, it's a fun weekend build and genuinely clever engineering. Henry Yuan's demo videos show it setting repeating alarms and drawing a smiley face in a whiteboard app, which is a good illustration of the real trick here: it works through the same interface a human uses, so it doesn't need a custom integration for every single app.

If you're trying to build a production iPhone automation pipeline, I'd wait. The frame rate and ASCII-only keyboard are going to bite you fast, and the project is barely two weeks old as of this writing. Yuan has said he's working on an App Store release and testing free vision models through OpenRouter, so the "$7 and a free API key" version of this story is probably coming. It's just not quite here yet.

For more on where AI agents are actually ready to run unattended versus where they still need a human in the loop, our breakdown of building a support bot with GPT-4o covers a similar gap between demo and production. If you're scouting what else is worth your weekend, our roundup of AI tools worth trying in 2026 has more projects in this same "clever hack, early days" category.

Watch it in action