dft street manager odp 0019 transition
In arranging street works, promoters co-operate with many authorities who co-operate with many promoters.
Authorities and promoters will not all be able to cut over from legacy EToN to Street Manager (SM) at the same time: the risk of something going wrong for some organisation is just too high.
However, as soon as one organisation moves, it impacts many organisations that it co-operates with. This impact continues until all organisations have moved.
This is what we mean by the transition problem.
This ODP describes the challenges of staggered transition and engineering options for Street Manager to ease those problems.
Some of the challenges of transition to Street Manager also apply as organisations move between successive versions of the Street Manager API as DfT tightens the data contract on organisations. So one option discussed below is to separate the problem of centralization from the problem of tightening the data contract.
Start with the assumption of no support for a staggered transition, where production Street Manager supports works processes (permitting, noticing) only if both interlocutors are using it as a system of record. Then organisations face the following changes.
- Routing. Each organisation needs to know whether its interlocutor has moved over to Street Manager or not. Promoters instigate the EToN process; authorities just need to continue each case in the system it started in.
- Differing data representations: dealing with both XML for EToN and JSON for Street Manager
- Fragmented view. Each authority that has moved over will deal with a mix of promoters that have moved over (case managed in Street Manager) and those that have not (case managed in legacy EToN tools). The authorities somehow have to bring these two data sets together, e.g. to check occupancy of the highway. Utilities may also need to maintain a single view.
- Differing data standards. We have assumed that EToN 6 will still be used alongside Street Manager, but Street Manager will have stricter data contracts. Each organisation will have to support both data contracts.
In the following, by "legacy EToN package", we mean EToN packages as they currently function, with no SM API integration. DfT could decide to provide the following kinds of support.
SM can act as a message switch and as a protocol translator between SM and EToN messages.
- Where neither party to a case uses SM API/UI, SM forwards messages.
- Where one party (X) to a case uses SM API/UI (Y) and the other uses legacy EToN package, SM acts as an EToN package for Y, delivering messages to X based on Y's SM API calls, and updating the state of works in SM based on X's messages.
If organisation X uses SM API/UI, SM can consolidate a view of all EToN messages for X. There are two ways to achieve this.
- Online. SM is a copy recipient for all messages from X and to X.
- Inline. SM acts as a message switch (as above) and updates its view whenever it successfully delivers a message.
The disadvanages of the "online" option are:
- Need to co-ordinate copy recipient configuration across organisations.
- The risk of inconsistency. Copies may not always be delivered. (Additionally, it appears from spec as though copies may be delivered before originals are delivered.)
Set against that, the cost of meeting non-functionals is lower for "online" than "inline".
Both options require SM to support differing data standards, as discussed below.
For supporting theory, see Liskov Substitution Principle.
Leaving aside translation between representations, there are two possible types of difference between EToN data and Street Manager data, which apply equally well to successive versions of the Street Manager API.
- New rules are tighter than old rules. Variants include making a message field mandatory, making a free text field into an enumeration.
- New rules are looser than old rules. E.g. dispensing with a field altogether. (We have no instances so far, but it could happen.)
During transition, there are some unavoidable constraints:
- Recipients of messages cannot rely on tighter rules until everybody is able to send according to those rules. Example: accept free text rather than enumerated values. (Strictly speaking, a recipient can "flip" when it knows all its possible senders are sending according to the new rules. For example, a water company deals with a subset of authorities.)
- Senders cannot rely on looser rules until all receivers are able to accommodate those rules. Example: continue to send some value for an old mandatory field. (Strictly speaking, a sender can "flip" when it knows all its possible receivers can cope.)
Let U be the data schema that "just" permits both old rules and new rules (e.g. includes all fields in either). Then Street Manager must be able to store data that conforms to U and to translate as needed.
| Sender | Receiver | SM translation |
|---|---|---|
| Old | Old | None |
| Old | New(U) | Formatting only |
| New(U) | Old | Formatting only |
| New | New | None |
When organisations have all "flipped" then Street Manager can enforce the new model on them all. Note that Street Manager will retain history, so that its internal schema will still be U.
This design gives routing and interoperation. It does not store a "unified view" of messages exchanged between EToN participants.
- To use the Hub all production EToN instances send messages to the Hub, regardless of their real destination.
- Records for all EToN organisations and their districts have been created in Street Manager.
- The Hub's config database of routing details and credentials is populated. This contains the SM IDs of all EToN organisation and EToN addresses of all SM organisations. It also stores credentials that it can impersonate Street Manager organisations in EToN and EToN organisations to Street Manager.
As more organisations transition to Street Manager, the Street Manager run team adjusts the routing details in the config database. We do not anticipate building a friendly UI for this.
See also editable draw.io diagram
The Hub stores copies EToN messages that have recipients in Street Manager, into Street Manager. It transmits EToN messages for all Street Manager events that have EToN participants. Where all participant organisations are using EToN, the Hub just forwards EToN messages to their recipients. Where all participant organisations are using Street Manager, the Hub takes no action. The Hub has no long-term storage for works and messages.
For each incoming EToN message:
- Hub determines from config whether the main recipient is currently using EToN or Street Manager.
- If Street Manager, and the current copy of the message is the one for the main recipient, then Hub saves an equivalent Street Manager event with sender, recipient and copy recipients as specified by the EToN message.
- If EToN, then Hub just forwards the message.
Note that the Hub may receive multiple copies of the same EToN message, because of copy recipients, but it only loads the message once into Street Manager.
The Hub subscribes to events in Street Manager, using the same Street Manager subscription API that organisation API clients use. For each new Street Manager event:
- The Hub determines which recipients of the event are still on EToN.
- For each such, the Hub translates to EToN XML and sends the message.
EToN Receiver Service
Accepts incoming EToN messages intended for any recipient, selects whether they are for organisations on Street Manager or EToN and queues them appropriately.
Street Manager Translator Worker
Turns EToN XML into Street Manager JSON and queues for loading into Street Manager.
Street Manager Sender Worker
Loads JSON into Street Manager acting on behalf of sender organisations still on EToN and recipient organisations already on Street Manager.
Street Manager Subscriber Worker
Subscribes for new Street Manager events on behalf of recipient organisations still on EToN. Queues them.
EToN Translator Worker
Turns Street Manager JSON into EToN XML and queues for sending.
EToN Sender Worker
Sends messages to recipients still on EToN on behalf of senders already on Street Manager.
The EToN Receiver Service may be implemented as a Lambda triggered by API Gateway. The worker containers may be implemented by Lambdas triggered by CloudWatch Events. (SQS does not trigger Lambdas; SQS queues have to be polled.) It is important to set maximum concurrency on these lambdas so that they do not dogpile the Street Manager API and database.
Fallback: k8s containers.
Table comprising EToN organisation, EToN district, SM organisation, SM district, flag indicating whether that district is on EToN or SM, SM credentials, plus, if still on EToN, true EToN address. There must be one row for every EToN district.
This could be implemented as a simple PostgreSQL RDS instance, with indexes on both the EToN organisation and district and the SM organisation and district. Alternatively it could be implemented as a pair of DynamoDB tables (one clustered by EToN organisation and district and the other by SM organisation and district) with sufficient RCU provisioning.