Shared Data Server PoC - with OAuth bonus

[bitbeckers]

Github repo

Loom demo

Goal

This branch established a proof of concept for shared authorites on a single repository. The approach was to fork the pds package into a sds package and replace the single auth check (userDid === repoDid) with a RBAC system based on claims assigned to DIDs stored in the database of the SDS server.
SDS Implementation

Development

To support SDSes the following was implemented:

Lexicons for:

• organisations
• organisation create
• organisation listing
• repository permissions fetching
• repository permission granting
• repository collaborator listing
• repository revoke access

The PoC was scoped to org create & listing, permissions fetching & granting

Agents (services) for:

• doing (un)authenticated CRUD calls to the SDS server leveraging the users PDS OAuth if needed.

DB migrations for:

• shared repository permissions
• permission management audit log

Repo changes for developement

The development environement was expanded with a SDS test instance for local testing.

The lexicons files were updated with the lexicons needed for the SDS functionality

A sds-demo package was created that showcases OAuth based flows of SDS creation and collaborator management.

[aspiers]

Awesome, great work - thanks a lot for sharing, and for answering my questions on the Loom.

In general I think this is a really worthwhile line of research, and that supporting shared write access to repositories should be beneficial not just for Hypercerts stakeholders, but most likely for the "ATmosphere" ecosystem as a whole. (For example, it could allow corporations to manage their official Bluesky accounts much more securely by giving multiple users access and then being able to revoke access per user, rather than sharing a single set of credentials or having to delegate authorization to a third-party product like Hootsuite.)

I just replied with some follow-up questions there, but maybe it makes sense to move those discussion threads here.

1. Architecture clarification

I got the strong impression that your PoC involves

1. a normal PDS which users bitbeckersa and bitbeckersb authenticate against, and
2. a separate SDS and corresponding frontend where most of the demo happens.

However I didn't hear this explicitly stated, so it would be helpful to confirm that just to make sure we're all on the same page.

2. Unauthenticated calls

You mentioned an unauthenticated call to create the shared repo, but it's not 100% clear to me what you mean by "unauthenticated" in this context, or how repo creation could be unauthenticated, because surely the PDS needs to assign an owner to the new repo just after creating it.

I have other questions about this PoC:

3. Minimal delta

I see from your initial IMPLEMENTATION-PLAN.md (which was really great, BTW) that a minimal delta was one of the key goals, by preferring composition over modification. This makes a ton of sense, e.g.

• making it as easy as possible to understand
• adhering to the KISS principle
• maximising maintainability
• maximising compatibility with existing PDS
• maximising the chances of getting it merged upstream

However, it seems the AI agent/LLM let you down in this case, because it ended up being a huge delta, I suspect with a ton of duplicated code. For example, these two are identical:

• https://github.com/bitbeckers/atproto/blob/feat/shared_data_server_poc/packages/pds/src/actor-store/actor-store.ts
• https://github.com/bitbeckers/atproto/blob/feat/shared_data_server_poc/packages/sds/src/actor-store/actor-store.ts

Presumably this duplication is just a vibe-coding artifact which can be cleaned up?

4. Lexicons

I see you created some new lexicons: https://github.com/bitbeckers/atproto/tree/feat/shared_data_server_poc/lexicons/com/sds

Presumably we'll need to do some work to ensure that these slot in with the new Hypercerts data schema and shared lexicon repo which we have been discussing.

5. Authorization (RBAC) model

I appreciate that this PoC is only a sketch of an RBAC model and that we need to think further on how authorization needs to work. My intuition is that it may make sense to have a base set of roles and permissions which Hypercerts defines (and these could be very simple indeed), and then different stakeholders can choose to extend these as they see fit.

Have you looked into UCANs at all? I'm wondering whether it might make sense to use UCANs from the beginning, as my first impressions suggest they could be a really flexible and well-decentralized foundation for the Hypercerts authorization layer. Plus they are already used by other components we are looking at, e.g.

https://docs.storacha.network/concepts/ucans-and-storacha/

Or maybe we should start with something simpler and then potentially migrate to this later. I'm not sure yet, but I think it's worth considering at least.

[bitbeckers]

@aspiers thanks for the kind words and reviewing this 🙏

1. Yes :) Maybe these diagrams help for a better understanding?

2. It's changed by now. But for the PoC we get the user's DID from their OAuth token and pass that to the sds repo create call to set the initial owner

3. no, this was intentional. At first I tried to inherit and extend the PDS as a class, but it let to too many conflicts and quircks while setting up the PoC. So I just copied the pds package and turned it into a sds.

4. From my perspective, these should not be owned by Hypercerts. This is an extension of ATProto and should be treated as such.

5. Yes, but would like a deeper convo on the actual auth and OAuth, but it looks like we're setting these up in other channels already.

[aspiers]

1. Yes, HUGE help thanks!
2. OK but I'm still curious what you meant by unauthenticated. Was it "unauthenticated to the SDS but authenticated to the PDS"? I guess that would tie in with your second sentence here. Does it validate the OAuth JWT before creating the shared repo? Otherwise there's a spam attack where anyone could anonymously create shared repos.
3. Ah 🫤 This could be a major problem. I would be very surprised if upstream was willing to consider a PR with such a large amount of duplicated code, as it creates a substantial maintenance problem. For a PoC it's OK, but I think we would really need to solve this for it to be a viable long-term solution. Also it makes it harder for reviewers like me to see the real delta. I guess I need to do a diff -ur packages/{pds,sds} then?
4. Sure, but in that case we need to ensure they fulfill the needs of Hypercerts in modelling organisations and facilitating appropriate RBAC.
5. Yep 👍

[aspiers]

BTW really nice job with the Mermaid diagrams. How did you do those so quickly - with LLM assistance I guess? If you have any good tricks please share with the team as in general I think we need to diagram more.

[zuckymiller256]

This is a solid step forward great to see RBAC replacing the simple auth check for more flexible control. Forking into an SDS package and adding OAuth-based flows makes it much more scalable. The study consultant in islamabad lexicons and test instance setup are super helpful for local dev. Really looking forward to seeing this evolve past the PoC stage!

[aspiers]

Another question is how to handle OAuth JWT revocation, e.g. if the user logs out of the PDS, can they still create new shared repos in the SDS? The normal solution would either be the SDS checking the PDS for revocation which is an extra round-trip either per action or at least on a regular basis, or issuing JWTs which expire quickly and requiring refresh. Both approaches are well-documented but not ideal (and defeat the main benefit of JWTs being stateless and self-verifiable), which is why many people recommend avoiding JWTs for session tokens ... but then if that's how ATproto OAuth works then we probably don't have much choice in which case it's not worth worrying about too much 😅