Live API does not perceive visual information and hallucinates

[franzscherr]

Description of the bug:

The current example script for the gemini live API (quickstarts/Get_started_LiveAPI.py) is unable to perceive visual information and hallucinates wildly.

Actual vs expected behavior:

Setting:

• Run the script as python Get_started_LiveAPI.py --mode screen or python Get_started_LiveAPI.py --mode camera, and
• query the model with "What do you see on the screen that I am sharing"

This will result in responses like "There is a man shown with dark hair..." or "A chess game...".

Expected response: "The screen shows a terminal with various commands..."

Hence one can conclude that vision currently does not work with live API?

Any other information you'd like to share?

No response

[moizulfiqar]

Hi! Is this issue resolved yet?

[Sravan1011]

Hi! Is this issue resolved yet?

Hey did you work on this issue ? May I know what is the expected output that should be returned when you run python Get_started_LiveAPI.py --mode screen