Building a Better Computer-Use Extension for Pi: 3.26× Faster with Python

Computer-use agents spend a surprising amount of time talking about clicking things. Observe a page, choose a field, call a tool, wait for the model, type something, observe again. Repeat until either the form is complete or we have all aged considerably.

I built pi-computer-use-exec to implement the best practices for computer use with GPT-6 in Pi: persistent Python, Playwright, and optional desktop control through PyAutoGUI, inside Pi's existing model conversation.

On a local form-filling benchmark, it completed the task in 19.89 seconds versus 64.93 seconds for @injaneity/pi-computer-use@0.5.1. That is a 3.26× speedup, using 2 tool calls instead of 13. Both extensions completed all three measured runs correctly.

My goal was to build the best computer-use extension for how I use Pi: let a capable model express a short sequence of actions as code, keep the browser alive, and spend its next turn checking the result. The benchmark supports that design on this task. It does not establish a universal winner across every website and desktop application.

Code-driven interaction is becoming the default

OpenAI's computer-use guide explicitly recommends code execution for GPT-6 Astra: give the model a script-execution tool, preserve the environment between calls, and return observations including screenshots. That is the pattern I wanted this extension to bring into Pi.

I think this will become the new normal. Newer models seem to be increasingly reinforcement-learned to generate code for UI interaction: inspect the screen, write a short program, execute it, inspect the result, and correct course. That is my reading of their behaviour and the direction of the tooling, rather than a claim that I know the providers' private training recipes.

The same direction is visible with Claude. Anthropic's programmatic tool calling lets Claude orchestrate tools through code, using loops and conditionals while keeping intermediate results out of the model conversation. It is broader than UI automation, but the principle transfers: let code handle a sequence of understood operations and bring the model back when another judgment is needed.

My hypothesis is that this same code-generation and feedback loop is also a source of the advances we are seeing in these models' 3D capabilities. A model can write a Blender script to construct geometry, render the scene, inspect the image, and revise its code. Coding ability becomes a practical way to create and manipulate a 3D world. That does not establish that UI-focused RL caused the spatial gains; it explains why better code generation and visual feedback can unlock more capable 3D workflows. Anthropic's robotics experiments similarly show how strongly results depend on the control interface, including whether a model can write controller code.

For Pi, I wanted an interface that fits that direction: persistent execution, short grounded action sequences, and visual verification.

Give the agent a small program

The extension exposes one main tool:

exec_py({ code, timeout_ms? })

It runs synchronous Python in a persistent worker. Variables survive between calls, along with the browser session. Pi handles the model conversation and authentication; the worker handles execution. There is no separate model client or extra API key for this extension.

The basic loop is:

1. Inspect the current page or screen.
2. Run a short sequence of actions grounded in that observation.
3. Verify the resulting interface and return useful text or a screenshot.

For example, the first call can open the dedicated browser and inspect the form:

start_browser('http://127.0.0.1:8080')
log(page.locator('body').aria_snapshot())
display(page.screenshot())

Once the agent has observed the controls, it can fill several fields in the next call:

page.get_by_label('Full name', exact=True).fill('Ada Lovelace')
page.get_by_label('Email', exact=True).fill('ada@example.test')
# Continue with the other fields observed on this page.
display(page.screenshot())

The agent still needs to inspect the next state when a page changes unexpectedly. Batching works well when the actions are already understood. A long script built on guesses simply fails more efficiently.

Playwright supplies browser locators, form input, screenshots and accessibility snapshots. Desktop mode adds PyAutoGUI for mouse and keyboard interaction. The dedicated Chromium browser is visible, so I can watch what the agent is doing.

The benchmark

I compared the extension against the unmodified published @injaneity/pi-computer-use@0.5.1 on a local registration form. It required five text values and a checkbox, followed by a Register action.

A separate loopback server validated every required value and recorded submissions. Success meant exactly one correct submission, and the agent also had to observe the visible completion message. A confident “done” was insufficient. A useful policy for software in general.

The recorded runs were made on September 12, 2026, using github-copilot/gpt-6-astra with medium thinking,

The most interesting row is time inside the tools. My backend took longer to execute its tool calls in aggregate. The end-to-end improvement came primarily from needing fewer model/tool round trips and fewer observation and recovery steps.

The extension consistently used one call to launch and inspect the form, then another to fill the fields, check the box, submit and verify the result. The baseline also supported batching and batched field edits, but used more calls to discover references, inspect the checkbox and confirm completion.

This is why I think the interface matters. The model's ability to plan several steps only helps when the tool lets it execute those steps together.

A longer task: DMV appointments

The project's README also records a DMV appointment-task comparison:

That is approximately 1.80× faster with about half as many tool calls.

Persistent state needs a lifecycle

Keeping Python and a browser alive makes the interface useful, but it also introduces state to manage. Calls are serialized, ordinary Python exceptions preserve variables for correction, and timeout, cancellation or a PyAutoGUI failsafe stops the worker. A reset discards the runtime. Resuming a conversation does not restore the old browser or Python variables.

Desktop input needs particular care. On Windows, focus_browser() requests native foreground focus and verifies it before physical keyboard input. It refuses ambiguous targets or denied focus. Bringing a Playwright page to the front does not guarantee that the operating system will send keystrokes there.

That check happens at call time; another window can still steal focus later. The PyAutoGUI failsafe stays enabled.

This is also unrestricted local Python running with the user's permissions. A worker process and dedicated browser profile do not create a security sandbox. Browser and desktop modes choose the available convenience helpers; they do not prevent arbitrary imports. For untrusted tasks, use a disposable desktop or VM, and keep sensitive submissions, purchases and external messages under explicit user control.

Try it

Install the extension into Pi:

pi install npm:pi-computer-use-exec

Or install from the repository:

pi install git:github.com/husain-zaidi/pi-computer-use-exec

The installer is set up to create a Python environment, install the dependencies and download Playwright's Chromium. Inside Pi, choose the mode:

/computer-use browser
/computer-use desktop
/computer-use reset

Browser mode is the default. Desktop mode adds the PyAutoGUI helpers. Linux desktop control requires X11; macOS requires the relevant Accessibility and Screen Recording permissions. The recorded comparison and native focus helper were tested on Windows.

The result I care about is straightforward: on the measured form, a model using two short Python calls finished correctly with substantially less waiting. Persistent execution gives the agent room to use what it already knows about the page. The next step is to test that advantage across more varied, longer tasks.

Install pi-computer-use-exec from the Pi package directory →

Extension source →