[Feature Request] Browser visual Interaction based on pixels

[nitpicker55555]

Required prerequisites

• I have searched the Issue Tracker and Discussions that this hasn't already been reported. (+1 or comment there if it has.)
• Consider asking first in a Discussion.

Motivation

More and more model providers are releasing computer-use models that support pure pixel-based clicking and page interactions. We need to add support for this mode of operation, which relies purely on visual input to interact with pages using pixel coordinates and UI elements.

Solution

No response

Alternatives

No response

Additional context

No response

[NadeebaAtiqui]

Hi! I'm interested in contributing to this issue.
I have a clarifying question: Are you looking to support pixel-based interactions as an additional mode alongside existing interaction methods, or should this replace/extend the current approach? Also, are there specific model providers or frameworks you'd like to prioritize for initial support?

[nitpicker55555]

Hi! I'm interested in contributing to this issue. I have a clarifying question: Are you looking to support pixel-based interactions as an additional mode alongside existing interaction methods, or should this replace/extend the current approach? Also, are there specific model providers or frameworks you'd like to prioritize for initial support?

@NadeebaAtiqui Hi! Thanks for your interest
We need to introduce a new mode called pixel_interaction. In this mode, it should behave with full_vision_mode (this mode already exists, but we need to verify whether it truly never returns snapshots for any operation), in which no operation should return a snapshot. Additionally, in pixel_interaction mode, click and type actions must operate using pixel coordinates only. This requires setting dynamic docstrings: in this mode, docstrings that take ref as input should not be passed to the model.

Furthermore, we need to add an option to browser_get_som_screenshot that allows it to output a clean screenshot without SOM labels. In pixel_interaction mode, browser_get_som_screenshot should default to returning screenshots without SOM labels.