Skip to content

Fix eval tests #846

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
gspencergoog merged 4 commits into from
Apr 1, 2026
Merged

Fix eval tests #846

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
86fdfcb
Fix tests
gspencergoog Mar 31, 2026
c4337c8
Update packages/genui/lib/src/engine/surface_controller.dart
gspencergoog Apr 1, 2026
7d54ba5
Update packages/genui/lib/src/facade/prompt_builder.dart
gspencergoog Apr 1, 2026
b5f3853
Add comment, test turn counts
gspencergoog Apr 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 35 additions & 2 deletions examples/eval/test/simple_chat_test.dart
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,22 +90,48 @@ class _ChatSessionTester {
var content = 0;
var waiting = 0;
final errors = <String>[];

var currentTurnCreates = 0;
var currentTurnUpdates = 0;
var turnCount = 0;

void verifyTurn() {
if (turnCount > 0) {
reporter.expect(
currentTurnCreates <= 1,
'Turn $turnCount should create at most 1 surface',
);
reporter.expect(
currentTurnUpdates == currentTurnCreates,
'Turn $turnCount should have matching creates ($currentTurnCreates) '
'and updates ($currentTurnUpdates)',
);
}
}

for (final ConversationEvent event in events) {
switch (event) {
case ConversationSurfaceAdded():
created.add(event.surfaceId);
currentTurnCreates++;
case ConversationComponentsUpdated():
updated.add(event.surfaceId);
currentTurnUpdates++;
case ConversationSurfaceRemoved():
removed.add(event.surfaceId);
case ConversationContentReceived():
content++;
case ConversationWaiting():
verifyTurn();
turnCount++;
waiting++;
currentTurnCreates = 0;
currentTurnUpdates = 0;
case ConversationError():
errors.add(event.error.toString());
}
}
verifyTurn();

print('Conversation summary:');
print(' Created surfaces: $created');
Expand All @@ -117,9 +143,16 @@ class _ChatSessionTester {

reporter.expect(errors.isEmpty, 'No errors should occur');
reporter.expect(
updated.isEmpty,
'In chat setup surfaces should not be updated',
updated.length == created.length,
'In chat setup surfaces should not be updated after initial creation',
);
Comment on lines +146 to 148
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist bot Apr 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The check updated.length == created.length only verifies the total number of events. It doesn't ensure that each specific surface was updated exactly once. For example, if one surface is updated twice and another is never updated, this check would still pass. A more robust check would verify the update count per surface ID.

      updated.length == created.length,
      'In chat setup surfaces should not be updated after initial creation',
    );
    for (final id in created) {
      final updateCount = updated.where((u) => u == id).length;
      reporter.expect(updateCount == 1, 'Surface $id should be updated exactly once');
    }

for (final id in created) {
final int updateCount = updated.where((u) => u == id).length;
reporter.expect(
updateCount == 1,
'Surface $id should be updated exactly once',
);
}
}

void failIfIssuesFound() => reporter.failIfIssuesFound();
Expand Down
5 changes: 2 additions & 3 deletions packages/genui/lib/src/engine/surface_controller.dart
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -234,9 +234,8 @@ interface class SurfaceController implements SurfaceHost, A2uiMessageSink {
final DataModel model = _store.getDataModel(surfaceId);
model.update(path, value);

// Trigger generic update on surface to refresh UI
final SurfaceDefinition current = _registry.getSurface(surfaceId)!;
_registry.updateSurface(surfaceId, current);
// Note: We don't trigger a surface update here to avoid full UI refreshes
// on data changes. Components should listen to the DataModel directly.

case DeleteSurface(:final surfaceId):
_pendingUpdates.remove(surfaceId);
Expand Down
3 changes: 2 additions & 1 deletion packages/genui/lib/src/facade/conversation.dart
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,8 @@ final class ConversationContentReceived extends ConversationEvent {
final String text;
}

/// Fired when the conversation is waiting for a response.
/// Fired when a request is sent to the LLM and the conversation is waiting
/// for an AI response.
final class ConversationWaiting extends ConversationEvent {}

/// Fired when an error occurs during the conversation.
Expand Down
12 changes: 8 additions & 4 deletions packages/genui/lib/src/facade/prompt_builder.dart
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ abstract class PromptFragments {
/// [prefix] is a prefix to be added to the prompt.
/// Is useful when you want to emphasize the importance of this fragment.
static String acknowledgeUser({String prefix = ''}) =>
'''
'''
${prefix}Your responses should contain acknowledgment of the user message.
'''
.trim();
Expand Down Expand Up @@ -131,7 +131,7 @@ enum ProtocolMessages {
explanation: 'Creates a new surface.',
properties: '''
Requires `surfaceId` (you must always use a unique ID for each created surface),
`catalogId` (use the catalog ID provided in system instructions),
`catalogId` (use the catalog ID provided in system instructions),
and `sendDataModel: true`.
''',
// TODO: figure out why we instruct AI to always set sendDataModel: true,
Expand All @@ -142,15 +142,15 @@ and `sendDataModel: true`.
name: 'updateComponents',
explanation: 'Updates components in a surface.',
properties: '''
Requires `surfaceId` and a list of `components`.
Requires `surfaceId` and a list of `components`.
One component MUST have `id: "root"`.
''',
),
updateDataModel(
name: 'updateDataModel',
explanation: 'Updates the data model.',
properties: '''
Requires `surfaceId`, `path` and `value`.
Requires `surfaceId`, `path` and `value`.
''',
),
deleteSurface(
Expand Down Expand Up @@ -286,6 +286,10 @@ You can control the UI by outputting valid A2UI JSON messages wrapped in markdow
To create a new UI:
1. Output a ${ProtocolMessages.createSurface.tickedName} message with a unique `surfaceId` and `catalogId` (use the catalog ID provided in system instructions).
2. Output an ${ProtocolMessages.updateComponents.tickedName} message with the `surfaceId` and the component definitions.
''',
if (!update)
'''
IMPORTANT: DO NOT update or modify surfaces created in previous turns. If the UI needs to change, you MUST create a NEW surface with a new unique `surfaceId`. You may only use ${ProtocolMessages.updateComponents.tickedName} to populate the components of a freshly created surface.
''',
if (update)
'''
Expand Down
4 changes: 2 additions & 2 deletions ...ages/genui/test/facade/prompt_builder_test.golden/all_operations_with_dataModel_false.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -100,9 +100,9 @@ Supported messages are: `createSurface`, `updateComponents`, `deleteSurface`.
Properties:

- `createSurface`: Requires `surfaceId` (you must always use a unique ID for each created surface),
`catalogId` (use the catalog ID provided in system instructions),
`catalogId` (use the catalog ID provided in system instructions),
and `sendDataModel: true`.
- `updateComponents`: Requires `surfaceId` and a list of `components`.
- `updateComponents`: Requires `surfaceId` and a list of `components`.
One component MUST have `id: "root"`.
- `deleteSurface`: Requires `surfaceId`.

Expand Down
4 changes: 2 additions & 2 deletions packages/genui/test/facade/prompt_builder_test.golden/all_operations_with_dataModel_true.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,9 +101,9 @@ Supported messages are: `createSurface`, `updateComponents`, `deleteSurface`, `u
Properties:

- `createSurface`: Requires `surfaceId` (you must always use a unique ID for each created surface),
`catalogId` (use the catalog ID provided in system instructions),
`catalogId` (use the catalog ID provided in system instructions),
and `sendDataModel: true`.
- `updateComponents`: Requires `surfaceId` and a list of `components`.
- `updateComponents`: Requires `surfaceId` and a list of `components`.
One component MUST have `id: "root"`.
- `deleteSurface`: Requires `surfaceId`.
- `updateDataModel`: Requires `surfaceId`, `path` and `value`.
Expand Down
4 changes: 2 additions & 2 deletions ...s/genui/test/facade/prompt_builder_test.golden/create_and_update_with_dataModel_false.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,9 +99,9 @@ Supported messages are: `createSurface`, `updateComponents`.
Properties:

- `createSurface`: Requires `surfaceId` (you must always use a unique ID for each created surface),
`catalogId` (use the catalog ID provided in system instructions),
`catalogId` (use the catalog ID provided in system instructions),
and `sendDataModel: true`.
- `updateComponents`: Requires `surfaceId` and a list of `components`.
- `updateComponents`: Requires `surfaceId` and a list of `components`.
One component MUST have `id: "root"`.

To create a new UI:
Expand Down
4 changes: 2 additions & 2 deletions ...es/genui/test/facade/prompt_builder_test.golden/create_and_update_with_dataModel_true.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -100,9 +100,9 @@ Supported messages are: `createSurface`, `updateComponents`, `updateDataModel`.
Properties:

- `createSurface`: Requires `surfaceId` (you must always use a unique ID for each created surface),
`catalogId` (use the catalog ID provided in system instructions),
`catalogId` (use the catalog ID provided in system instructions),
and `sendDataModel: true`.
- `updateComponents`: Requires `surfaceId` and a list of `components`.
- `updateComponents`: Requires `surfaceId` and a list of `components`.
One component MUST have `id: "root"`.
- `updateDataModel`: Requires `surfaceId`, `path` and `value`.

Expand Down
6 changes: 4 additions & 2 deletions packages/genui/test/facade/prompt_builder_test.golden/create_only_with_dataModel_false.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,14 +99,16 @@ Supported messages are: `createSurface`, `updateComponents`.
Properties:

- `createSurface`: Requires `surfaceId` (you must always use a unique ID for each created surface),
`catalogId` (use the catalog ID provided in system instructions),
`catalogId` (use the catalog ID provided in system instructions),
and `sendDataModel: true`.
- `updateComponents`: Requires `surfaceId` and a list of `components`.
- `updateComponents`: Requires `surfaceId` and a list of `components`.
One component MUST have `id: "root"`.

To create a new UI:
1. Output a `createSurface` message with a unique `surfaceId` and `catalogId` (use the catalog ID provided in system instructions).
2. Output an `updateComponents` message with the `surfaceId` and the component definitions.

IMPORTANT: DO NOT update or modify surfaces created in previous turns. If the UI needs to change, you MUST create a NEW surface with a new unique `surfaceId`. You may only use `updateComponents` to populate the components of a freshly created surface.
-----CONTROLLING_THE_UI_END-----

-------------------------------------
Expand Down
6 changes: 4 additions & 2 deletions packages/genui/test/facade/prompt_builder_test.golden/create_only_with_dataModel_true.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -100,15 +100,17 @@ Supported messages are: `createSurface`, `updateComponents`, `updateDataModel`.
Properties:

- `createSurface`: Requires `surfaceId` (you must always use a unique ID for each created surface),
`catalogId` (use the catalog ID provided in system instructions),
`catalogId` (use the catalog ID provided in system instructions),
and `sendDataModel: true`.
- `updateComponents`: Requires `surfaceId` and a list of `components`.
- `updateComponents`: Requires `surfaceId` and a list of `components`.
One component MUST have `id: "root"`.
- `updateDataModel`: Requires `surfaceId`, `path` and `value`.

To create a new UI:
1. Output a `createSurface` message with a unique `surfaceId` and `catalogId` (use the catalog ID provided in system instructions).
2. Output an `updateComponents` message with the `surfaceId` and the component definitions.

IMPORTANT: DO NOT update or modify surfaces created in previous turns. If the UI needs to change, you MUST create a NEW surface with a new unique `surfaceId`. You may only use `updateComponents` to populate the components of a freshly created surface.
-----CONTROLLING_THE_UI_END-----

-------------------------------------
Expand Down
2 changes: 1 addition & 1 deletion packages/genui/test/facade/prompt_builder_test.golden/update_only_with_dataModel_false.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ Supported messages are: `updateComponents`.

Properties:

- `updateComponents`: Requires `surfaceId` and a list of `components`.
- `updateComponents`: Requires `surfaceId` and a list of `components`.
One component MUST have `id: "root"`.

To update an existing UI:
Expand Down
2 changes: 1 addition & 1 deletion packages/genui/test/facade/prompt_builder_test.golden/update_only_with_dataModel_true.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ Supported messages are: `updateComponents`, `updateDataModel`.

Properties:

- `updateComponents`: Requires `surfaceId` and a list of `components`.
- `updateComponents`: Requires `surfaceId` and a list of `components`.
One component MUST have `id: "root"`.
- `updateDataModel`: Requires `surfaceId`, `path` and `value`.

Expand Down
Loading