fix(dag-jose): update to match implementations

- Specify that `payload` of JWS and plaintext of JWE ciphertext must be
CIDs
- Use `Bytes` instead of `String` {String:Any} for authenticated headers
- Add clarifying prose around purpose of the encoded and decoded schemas
- Add clarifying prose for encrypted padding
- Add Go implementation

Author: alexjg

block-layer/codecs/dag-jose.md

@@ -2,125 +2,99 @@
 
 **Status: Descriptive - Draft**
 
-JOSE is a stanard for signing and encrypting JSON objects. The various specifications for JOSE can be found in the [IETF datatracker](https://datatracker.ietf.org/wg/jose/documents/). 
-
-DAG JOSE supports the full [IPLD Data Model](../data-model-layer/data-model.md) (within the payload).
+JOSE is a standard for signing and encrypting JSON objects. The various specifications for JOSE can be found in the [IETF datatracker](https://datatracker.ietf.org/wg/jose/documents/). 
 
 ## Format
 
-The are two main ways to represent a JOSE object. As a JWS ([json web signature](https://datatracker.ietf.org/doc/rfc7515/?include_text=1)) and JWE ([json web encryption](https://datatracker.ietf.org/doc/rfc7516/?include_text=1)). These two formats  acts as the primitives in JOSE and can be used to create JWT and JWM objects etc. This specification describes how to encode JWS and JWE as an IPLD format.
+The are two kinds of JOSE objects: JWS ([JSON web signature](https://datatracker.ietf.org/doc/rfc7515/?include_text=1)) and JWE ([JSON web encryption](https://datatracker.ietf.org/doc/rfc7516/?include_text=1)). These two objects are primitives in JOSE and can be used to create JWT and JWM objects etc. The IETF RFCs specify a JSON encoding of JOSE objects. This specification maps the JSON encoding to CBOR. Upon encountering the `dag-jose` multiformat implementations can be sure that the block contains dag-cbor encoded data which matches the IPLD schema we specify below.
+
+### Mapping from the JOSE general JSON serialization to dag-jose serialization
+
+Both JWS and JWE supports three different serialization formats: `Compact Serialization`, `Flattened JSON Serialization`, and `General JSON Serialization`. The first two are more concise, but they only allow for one recipient. Therefore DAG JOSE always uses the `General Serialization` which ensures maximum compatibility with minimum ambiguity. Libraries implementing serialization should accept all JOSE formats including the `Decoded Representation` (see below) and convert them if necessary.
+
+To map the general JSON serialization to CBOR we do the following:
 
-### Representation
+- Any field which is represented as `base64url(<data>)` we map directly to `Bytes` . For fields like `header` and `protected` which are specified as the `base64url(ascii(<some json>))` that means that the value is the `ascii(<some json>)` bytes.
+- For JWS we specify that the `payload` property MUST be a CID, and we set the `payload` of the encoded JOSE object to `Bytes` containing the bytes of the CID. For applications where an additional network request to retrieve the linked content is undesirable then an `identity` multihash should be used.
+- For JWE objects the `ciphertext` must decrypt to a plaintext which is the bytes of a CID. This is for the same reason as the `payload` being a CID, and the same approach of using an `identity` multihash can be used, and most likely will be the only way to retain the confidentiality of data.
 
-The layout of a decoded JOSE object is described by the IPLD schema defined below. We will refer to this layout as the `Decoded Representation`. 
+Below we present an IPLD schema representing the encoded JOSE objects. Note that the `EncodedJOSE` union is not in fact a valid IPLD schema as there is no valid discriminator. The actual wire format is a single struct which contains all the keys from both the `EncodedJWE` and the `EncodedJWS` structs, implementors should follow [section 9 of the JWE spec](https://tools.ietf.org/html/rfc7516#section-9) and distinguish between these two branches of the union by checking if the `payload` attribute exists, and hence you have a JWS; or the `ciphertext` attribute, hence you have a JWE.
+
+**Encoded JOSE**
 
 ```ipldsch
-type Signature struct {
+type EncodedSignature struct {
   header optional {String:Any}
-  protected optional {String:Any}
+  protected optional Bytes
   signature Bytes
 }
 
-type JWS struct {
-  payload Any
-  signatures [Signature]
-}
-
-type Recipient struct {
+type EncodedRecipient struct {
   encrypted_key optional Bytes
   header optional {String:Any}
 }
 
-type JWE struct {
+type EncodedJWE struct {
   aad optional Bytes
   ciphertext Bytes
   iv optional Bytes
-  protected optional {String:Any}
-  recipients [Recipient]
+  protected optional Bytes
+  recipients [EncodedRecipient]
   tag optional Bytes
   unprotected optional {String:Any}
 }
 
-type JOSE union {
-  | JWS jws
-  | JWE jwe
-} representation kinded
+type EncodedJWS struct {
+  signatures [EncodedSignature]
+  payload optional Bytes
+}
+
+type EncodedJOSE union { EncodedJWE | EncodedJWS }
 ```
 
-### Serialization
+## Padding for encryption
+
+Applications may need to pad the plaintext when encrypting to avoid leaking the size of the plaintext. This raises the question of how the application knows what part of the decrypted plaintext is padding. In this case we use the fact that the plaintext MUST be a valid CID, implementations should parse the plaintext as a CID and discard any content beyond the multihash digest size - which we assume to be the padding.
 
-Both JWS and JWE supports three different serialization formats: `Compact Serialization`, `Flattened JSON Serialization`, and `General JSON Serialization`. The first two are more concise, but they only allow for one recipient. Therefore DAG JOSE always uses the `General Serialization` which ensures maximum compatibility with minimum ambiguity. 
 
-The implementation of the serialization function should accept all JOSE formats including the `Decoded Representation` and convert them if necessary. 
+## Decoded JOSE
 
-#### General JSON Serialization
+Typically implementations will want to decode this format into something more useful for applications. Exactly what that will look like depends on the language of the implementation, here we use the IPLD schema language to give a somewhat language agnostic description of what the decoded representation might look like at runtime. Note that everything which is specified as `base64url(ascii(<some JSON>))` in the JOSE specs - and which we encode as `Bytes` in the wire format - is here decoded to a `String`. We also add the `link: &Any` attribute to the `DecodedJWS`,  which allows applications to easily retrieve the authenticated content.
 
-Below the  `General JSON Serialization` can be observed. Note that all data represented as `String` here is data that has been encoded using `base64url`. Converting `Compact Serialization` and `Flattened JSON Serialization` to the general serialization is trivial.
+Also note that - as with the encoded representation - the `DecodedJOSE` union is not valid IPLD schema as there is no way to discriminate between them. How exactly this would be represented will depend on the language of the implementation. For example in Typescript this would be a straightforward `type DecodedJOSE = DecodedJWE | DecodedJWS`.
 
 ```ipldsch
-type GeneralSignature struct {
+type DecodedSignature struct {
   header optional {String:Any}
   protected optional String
   signature String
 }
 
-type GeneralJWS struct {
+type DecodedJWS struct {
   payload String
-  signatures [GeneralSignature]
+  signatures [DecodedSignature]
+  link: &Any
 }
 
-type GeneralRecipient struct {
+type DecodedRecipient struct {
   encrypted_key optional String
   header optional {String:Any}
 }
 
-type GeneralJWE struct {
+type DecodedJWE struct {
   aad optional String
   ciphertext String
-  iv optional String
-  protected optional String
-  recipients [GeneralRecipient]
-  tag optional String
+  iv String
+  protected String
+  recipients [DecodedRecipient]
+  tag String
   unprotected optional {String:Any}
 }
 
-type GeneralJOSE union {
-  | GeneralJWS jws
-  | GeneralJWE jwe
-} representation kinded
+type DecodedJOSE union { DecodedJWE | DecodedJWS }
 ```
 
-##### Serializing the Decoded Representation
-
-When serializing a JOSE object from the `Decoded Representation` special care needs to be taken with the `payload` property as well as the `protected` properties. 
-
-###### Protected
-
-The `protected` property in JWE and JWS have the type `{String:Any}`. This means that it may include data with *Link Kind* and *Bytes Kind*. These should be converted into pure JSON in the same way as it's done in [DAG-JSON](./dag-json.md). However, the properties should **not** be sorted since that would cause any integrity check on the JOSE data to fail. Once in JSON format the `protected` property should be converted into `base64url` using the method described in the JOSE spec  (`BASE64URL(UTF8(data))`). 
-
-###### Payload
-
-The payload property of JWS can be of either `Bytes` or  `{String:Any}` types. If the former it's simply just encoded as `base64url`. If the latter, it should be encoded in the same manner as the `protected` property.
-
-Note that any change in the ordering of the properties of the payload at this point would cause potential validation of the JOSE object to fail. Good signature libraries will sort the payload before the signature is applied.
-
-#### Ordering
-
-Once the data has been converted to the `General Serialization`, codec implementors **MUST** use the same sorting algorithm as [DAG-JSON](./dag-json.md) to sort the data to ensure hashes consistently match for the same block data.
-
-## Additional information
-
-### Reccomended JOSE creation strategy
-
-When creating a JOSE object there are some suggested approaches of how to format the data that is being signed / encrypted / authenticated that will keep you out of trouble. The main thing to keep in mind is that signatures / data authentication could be invalidated if the order of the properties in the JOSE object changes. It's therefore a good idea to sort the properties before any signature / authentication is added. The best way to do this is simply to use the same strategy employed by [DAG-JSON](./dag-json.md), which will also convert `Link` and `Bytes` to JSON representation. 
-For JWS the relevant properties to do this for is `protected` and `payload` since the signature is done over  `ASCII(BASE64URL(UTF8(JWS Protected Header)) || '.' || BASE64URL(JWS Payload))` according to the [JWS specification](https://datatracker.ietf.org/doc/rfc7515/?include_text=1).
-
-For JWE it is `protected` and the cleartext before it is encrypted into `ciphertext`.
-
-### Decryption of JWEs
-
-Similar to the `payload` of JWS, the decrypted data of a JWE may be encoded as [DAG-JSON](./dag-json.md) as described above. The implementation of the decryption function should account for this if neccessary to allow the data be interpreted as an IPLD dag node. In the future the decryption itself could be described using an [Advanced IPLD schema layout](../../schemas/advanced-layouts.md). 
-
-### Implementations
+##Implementations
 
-* [Javascript](https://github.com/oed/js-dag-jose)
+- [Javascript](https://github.com/oed/js-dag-jose)
+- [Go](https://github.com/alexjg/go-dag-jose)