Skip to content

e2e_tests: migrate droid benchmarks#12159

Draft
andrea-reale wants to merge 14 commits intomainfrom
andrea/migrate_benchmarks
Draft

e2e_tests: migrate droid benchmarks#12159
andrea-reale wants to merge 14 commits intomainfrom
andrea/migrate_benchmarks

Conversation

@andrea-reale
Copy link
Member

Why

We are consodlidating redap-component-tests into e2e_redap_tests.

What

The first step is moving the droid group into e2e_redap_tests. This change:

  • Moves the tests and encodes them as pytest-benchmarks
  • Adds CLI options and fixtures for the tests
  • Adds markers to select the tests (cloud_only and slow)

What next

  • Add telemetry (metrics, traces)
  • Port the flask service for on demand services
  • Missing vecbench group

@github-actions
Copy link

github-actions bot commented Dec 10, 2025

Latest documentation preview deployed successfully.

Result Commit Link
89f4e32 https://landing-alhlhvw78-rerun.vercel.app/docs

Note: This comment is updated whenever you push a commit.

@github-actions
Copy link

github-actions bot commented Dec 10, 2025

Web viewer built successfully.

Result Commit Link Manifest
89f4e32 https://rerun.io/viewer/pr/12159 +nightly +main

View image diff on kitdiff.

Note: This comment is updated whenever you push a commit.

@andrea-reale andrea-reale force-pushed the andrea/migrate_benchmarks branch 2 times, most recently from 14698b8 to f6a4806 Compare December 11, 2025 13:29
@andrea-reale andrea-reale added the exclude from changelog PRs with this won't show up in CHANGELOG.md label Dec 11, 2025
@andrea-reale andrea-reale force-pushed the andrea/migrate_benchmarks branch 9 times, most recently from 99d7d88 to b10dcf1 Compare December 12, 2025 10:38
@andrea-reale andrea-reale requested a review from abey79 December 12, 2025 11:05
@andrea-reale andrea-reale marked this pull request as ready for review December 12, 2025 11:05
@andrea-reale andrea-reale force-pushed the andrea/migrate_benchmarks branch from 2b63b30 to aff6f48 Compare December 12, 2025 13:05
Signed-off-by: Andrea Reale <andrea@rerun.io>
* Telemetry setup
* Metrics publication

Signed-off-by: Andrea Reale <andrea@rerun.io>
Signed-off-by: Andrea Reale <andrea@rerun.io>
Signed-off-by: Andrea Reale <andrea@rerun.io>
Signed-off-by: Andrea Reale <andrea@rerun.io>
Signed-off-by: Andrea Reale <andrea@rerun.io>
Signed-off-by: Andrea Reale <andrea@rerun.io>
Signed-off-by: Andrea Reale <andrea@rerun.io>
Signed-off-by: Andrea Reale <andrea@rerun.io>
Signed-off-by: Andrea Reale <andrea@rerun.io>
Signed-off-by: Andrea Reale <andrea@rerun.io>
@andrea-reale andrea-reale force-pushed the andrea/migrate_benchmarks branch from aff6f48 to 548eea9 Compare December 16, 2025 10:15
Signed-off-by: Andrea Reale <andrea@rerun.io>
Copy link
Member

@abey79 abey79 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

great work!

Comment on lines +11 to +16
logger = logging.getLogger(__name__)

if TYPE_CHECKING:
from collections.abc import Iterator

from rerun.catalog import CatalogClient, DatasetEntry
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit

Suggested change
logger = logging.getLogger(__name__)
if TYPE_CHECKING:
from collections.abc import Iterator
from rerun.catalog import CatalogClient, DatasetEntry
if TYPE_CHECKING:
from collections.abc import Iterator
from rerun.catalog import CatalogClient, DatasetEntry
logger = logging.getLogger(__name__)

if: ${{ inputs.PLATFORM == 'linux-arm64' }}
# Workaround for OpenCV TLS issue, see https://github.com/opencv/opencv/issues/14884#issuecomment-815632861
run: cd rerun_py/tests && LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libgomp.so.1 pixi run -e wheel-test-min pytest -c ../pyproject.toml
run: cd rerun_py/tests && LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libgomp.so.1 pixi run -e wheel-test-min pytest -c ../pyproject.toml -m "not cloud_only"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this needed since it's disabled by default? Also, do we want to force opt-in to slow? (genuine question, not sure it makes sense here)

Comment on lines +61 to 75
droid_group = parser.getgroup("droid dataset options")

droid_group.addoption(
"--droid-dataset",
action="store",
default=None,
help="URI prefix for test resources (e.g., 's3://bucket/path/' for remote resources). "
"If not provided, local file:// URIs to the resources directory will be used.",
default="droid:sample50",
help="Dataset for Droid benchmarks (options: droid:raw, droid:sample50, etc.)",
)

droid_group.addoption(
"--droid-preregister-dataset",
action="store_true",
default=False,
help="should the test runner register the droid dataset before running tests",
)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be slightly more readable if the adoption code was colocated with the getoption fixtures. Would it make sense to move that to the other conftest.py?

@andrea-reale andrea-reale marked this pull request as draft January 12, 2026 13:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

exclude from changelog PRs with this won't show up in CHANGELOG.md

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants