Pydantic package organization episode 4: "Scope is in (scope)" #412

vcschapp · 2025-10-28T03:41:07Z

tl;dr

This is part #4. Part #5 is hoped for in the next week or two if all goes according to plan.

Done in this part: re-organized the addresses and places packages to their (hopefully) final form, finish dynamic scoping implementation (@scoped decorator) and apply it to names and transportation.

Merge guidance: Rebase

Unlike parts 1, 2, and 3, this iteration has a well-written commit history. It should ideally be rebase-merged onto the pydantic branch rather than squashed.

Journey

Part 1 is in PR Pydantic package organization episode 1: "Making huge mess" #397.
Part 2 is in PR Pydantic package organization episode 2: "The mess unfolds" #400.
Part 3 is in PR Pydantic package organization episode 3: "System update complete" #403.

Destination

Packages.
- ✅ The package setup will be organized as diagrammed on PR Pydantic package organization episode 1: "Making huge mess" #397 (Part 1) and PR Pydantic package organization episode 2: "The mess unfolds" #400 (Part 2).
- ⬜️ Package dependencies and pyproject.toml will be standardized.
Documentation.
- ✅ All doctests will pass and automatically be enforced to pass by make check.
- ▶️ All documentation will pass pydocstyle with the NumPy convention. (As of now, the tool reports too many complaints to add it into make check.)
- ⬜️ pydocstyle will be enforced by make check.
- ⬜️ docformatter will be run by make check.
- ▶️ Package-level README content will migrate into the package's __init__.py docstring.
- ⬜️ There will be a doc target in the Makefile that does uv run pdoc to a useful place.
Modules.
- ▶️ Re-exports will be standardized and everything of value will be re-exported into the main module.
- ▶️ In theme packages, each feature type will be within its own non-exported module within the package root.
- ⬜️ In theme packages, if a field contains a theme-specific "deep model" then it will be given its own module, e.g. perspectives.

Frozen-ness: - The frozen/immutable characteristics of the parent model are perpetuated into the "when" clause. - I initially thought this was needed to make the parent model behave as expected when contained in a list with a `UniqueItemsConstraint`, but this is not so because `UniqueItemsConstraint` is well-written and can handle non-hashable items. - But it is still needed for correctness: if a model is marked as frozen then it will only be truly so if the sub-models that compose it are also frozen.

mojodna

Nothing blocking, but I have lots of questions about scopes (surprise, surprise).

mojodna · 2025-10-28T04:08:13Z

packages/overture-schema-places-theme/pyproject.toml


 [project.entry-points."overture.models"]
-"places.place" = "overture.schema.places.place.models:Place"
+"places.place" = "overture.schema.places:Place"


Note that I made a similar change in the CLI PR. In that case, it may refer to an indirect alias, but either way, the intent is to register the shortest useful import for the class (because that path will be displayed when listing available types).

mojodna · 2025-10-28T04:09:20Z

PYDANTIC_GUIDE.md

Thank you for updating this!

mojodna · 2025-10-30T17:29:14Z

packages/overture-schema-core/src/overture/schema/core/scoping/__init__.py

+...
+>>> hotspots=[
+...     WiFiHotspot(ssid="always_on"),
+...     WiFiHotspot(ssid="daytime_use_only", when=WiFiHotspot.When(during="06:00-22:00"))


This is confusing. Where does WiFiHotspot.When() come from? (Update: found it below)

Reading the description above, I would expect this to be:

WiFiHotspot(ssid="daytime_use_only", when=dict(during="06:00-22:00"))

mojodna · 2025-10-30T17:33:00Z

packages/overture-schema-core/src/overture/schema/core/scoping/__init__.py

+Geometric range scope (linear events):
+--------------------------------------
+Geometric range scoping allows a value to be tied to a sub-segment of a linear path using linear
+referencing. When a model is decorated with geometric range scoping, a `between` field is


Why are point and linear events and side treated differently from other scopes? Specifically, at / between / side at the top level where the others are all nested under when.

Historical reasons.

The scoping concept came together somewhat organically (inductive reasoning). The existence of when can be explained by the fact that some fool thought "it read nicer in JSON", i.e. { "between": [0, 0.5], "when": { "mode": ["foot"] } }...

In retrospect, the existence of when seems regrettable. We should probably phase it out. But for now, it is what it is...

I'm not sure how to phrase it, but we should create an issue (and probably a label for breaking changes we want to make generally) to capture phasing this out, even if we don't have a good idea for how to migrate it yet.

mojodna · 2025-10-30T17:38:54Z

packages/overture-schema-core/src/overture/schema/core/scoping/__init__.py

+>>> island = BusLoadingIsland(
+...     id='island_001',
+...     theme='bus_terminal',
+...     type='loading_island',
+...     version=1,
+...     geometry=Geometry.from_wkt('LINESTRING (0 0, 1 1)'),
+...     stops=[
+...         BusStop(route='15', side=Side.LEFT, at=0.10),
+...         BusStop(route='B-Line', side=Side.RIGHT, at=0.25),
+...     ]
+... )


I'm thinking through how I'd expand on this by modeling standalone bus stops... It seems like I'd need a different, non-scoped BusStop that extends from OvertureFeature if I want to model them directly. (I might mix in fields that are common to all bus stops, like route.)

How would I model bus stops along road segments (without changing the RoadSegment model, unless there's a generic point_events field or similar to capture these)? I suppose I could reference the road segment with an at (to indicate where on the road segment the stop is). It would kind of look like the equivalent of a point event even though it's not.

Modeling them along road segments might be done in several ways but they're all variations of two options.

If you can change the road segment model then you probably add a bus_stops field similar to how it's down in the example.

If you can't change the road segment model, it probably means that bus stops are independent point features. If we want them to reference the road network you end up with something like (maybe):

@scoped(Scope.GEOMETRIC_POSITION) class BusStop(OvertureFeature[Literal["transportation"], Literal["bus_stop"]]): geometry: Annotated[ Geometry, GeometryTypeConstraint(GeometryType.POINT) ] road_segment_id: Annotated[ Id | None, Reference(Relationship.CONNECTS_TO, RoadSegment) ]

That's what I was envisioning, but I was thinking 1 step further: a linear reference to the road segment where it not only describes what it connects to, but where.

We could do something like this:

@scoped(Scope.GEOMETRIC_POSITION) class BusStop(OvertureFeature[Literal["transportation"], Literal["bus_stop"]]): geometry: Annotated[ Geometry, GeometryTypeConstraint(GeometryType.POINT) ] road_segment_id: Annotated[ Id | None, Reference(Relationship.CONNECTS_TO, RoadSegment) ] road_segment_at: LinearlyReferencedPosition | None = None

but the combination of the CONNECTS_TO and linear references implies to me that some form of nested structure would be valuable, especially if both are required at the same time.

packages/overture-schema-core/src/overture/schema/core/scoping/lr.py

mojodna · 2025-10-30T18:05:52Z

packages/overture-schema-core/src/overture/schema/core/scoping/scoped.py

+                return (
+                    "The linearly-referenced position on the geometry, "
+                    "specified as a percentage displacement from the start "
+                    f"of the geometry, that the containing {parent} applies to."
+                )


Can / should these pull from enum docstrings or related doc properties? It would help centralize the strings.

These get added as Pydantic descriptions on the fields that are generated by the @scoped decorator. So code generation software just has to traverse the model as it normally does and it'll see the descriptions.

I was thinking more about the maintenance benefits of centralization than code generation.

packages/overture-schema-core/src/overture/schema/core/scoping/scoped.py

mojodna · 2025-10-30T18:08:08Z

packages/overture-schema-core/src/overture/schema/core/scoping/scoped.py

+    At least one `Scope` must be given either in `optional` or `required` or both. A scope may be
+    optional or required, but not both.


How does one indicate that a scope must be provided without indicating which one?

That's currently not possible.

I didn't think it was needed for any existing use case, but if I'm wrong let me know so I can fix it.

(Assuming we don't use that construct today: I can imagine it's something we'll need in the future. The best thing to do would be to drop when and just use @require_any_of to indicate this constraint. With when in the mix, it's messy.)

SpeedLimitWhenClause is what I was thinking of, with @min_fields_set(1)

Got it. Yeah, that was a faithful modeling of this hand-written JSON Schema. At least for now it's still just trying to block multiple ways of the when clause saying the same thing.

packages/overture-schema-core/src/overture/schema/core/scoping/scoped.py

vcschapp added 8 commits October 20, 2025 12:11

Organize addresses theme

4db381a

Organize places theme

82e1b01

Add DocumentedEnum for __doc__ metadata on enum members

7fe1d9d

Apply @scoped to names

e0f7060

Re-home naming types to module

acdc7fe

Freshen Pydantic guide based on recent changes

a89ba7b

Apply @scoped to transportation models

5219d2b

vcschapp added the change type - cosmetic 🌹 Cosmetic change label Oct 28, 2025

vcschapp deployed to staging October 28, 2025 03:41 — with GitHub Actions View deployment

vcschapp requested a review from mojodna October 28, 2025 03:41

vcschapp changed the title ~~Pydantic package organization episode 4: "Subtitle required"~~ Pydantic package organization episode 4: "Scope is in (scope)" Oct 28, 2025

mojodna approved these changes Oct 30, 2025

View reviewed changes

vcschapp merged commit 4315747 into pydantic Oct 31, 2025
6 checks passed

vcschapp deleted the pp4 branch October 31, 2025 22:24

This was referenced Oct 31, 2025

[Pydantic] overture-schema CLI for type listing, JSON Schema generation, and validation #406

Merged

Pydantic package organization episode 5: "Only mostly dead" #415

Merged

		At least one `Scope` must be given either in `optional` or `required` or both. A scope may be
		optional or required, but not both.

Pydantic package organization episode 4: "Scope is in (scope)" #412

Pydantic package organization episode 4: "Scope is in (scope)" #412

Uh oh!

Conversation

vcschapp commented Oct 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

tl;dr

Merge guidance: Rebase

Journey

Destination

Uh oh!

mojodna left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mojodna Oct 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

vcschapp commented Oct 28, 2025 •

edited

Loading

mojodna Oct 31, 2025 •

edited

Loading