SDPX 3.0 Support

[mlieberman85]

The SPDX 3.0 spec has been officially released. This library should add support for SPDX 3.0.

What would this look like?

• Add 3.0 models (in spdx/v3/v3_0)
• Update the referenced core models in spdx to be the latest 3.0 version
• Add any necessary conversion functions from 2.3 -> 3.0
  • see: go-struct-converter and a potential change to support backwards conversion if we also want to support 3.0 -> 2.x conversion
  • Add at least one fairly complete 2.3 -> 3.0 conversion test
• Add support in deserialization calls for 3.0:
  • JSON (SPDX 3 only supports JSON-LD presently)
  • YAML
  • Tag Value (also: writer)
• Add tests against some reference 3.0 documents (to be identified)

[kzantow]

@mlieberman85 there are plans to add SPDX 3 support at some point, but there isn't currently a timeframe for it. I hope you don't mind but I've updated the text of this issue to use it a bit more placeholder issue for adding support.

[kzantow]

There have been some people looking in to how we can add support for SPDX 3 to this library. There are some distinct challenges -- namely the SPDX 3 model is based on single-inheritance, which isn't really directly supported in Golang. To share some potential solutions, I created this PR for discussion so anyone interested in what the future of this library looks like can comment on the ideas, suggest others, etc.: #240

[kzantow]

A prototype is available for feedback here: #247

[janjaeschke]

I would like to push this topic or ask for a current state. :)
The linked draft is 3 month old and for me it looks like the work sadly stopped. :(

[kzantow]

Hi @janjaeschke 👋 I have paused working on this until there is some resolution to a serious problem between the current SPDX 3 specification and the implementations.

I believe the draft PR was left in a fully functional state serializing to and deserializing from JSON-LD 1.1. However, documents created from the draft PR do not validate with the online tools and are not compatible with the other SPDX 3 implementations because the other implementations do not accept JSON-LD but rather only JSON conformant to the SPDX 3 JSON schema.

The serializations section of the SPDX 3 spec, however, indicates that JSON-LD is a valid serialization format, but the current implementations do not actually support JSON-LD, but rather a subset that no JSON-LD libraries are able to produce and as such requires a custom serializer, which I am not planning on writing without the spec indicating I should. Here is the text copied verbatim from that page:

RDF serialization
Since the data model is based on RDF, any SPDX data can be serialized in any of the multiple RDF serialization formats, including but not limited to:

• JSON-LD format as defined in JSON-LD 1.1;
• Turtle (Terse RDF Triple Language) format as defined in RDF 1.1 Turtle;
• N-Triples format as defined in RDF 1.1 N-Triples; and
• RDF/XML format as defined in RDF 1.1 XML Syntax.

As it stands I believe the spec is misleading and should have all references to data formats removed that are not going to be compatible and should only indicate JSON compliant to the schema is allowed OR the implementations should support full JSON-LD, in which case the draft PR is fairly close to being done, albeit requiring lots more testing to be performed and more unit tests to be added.

In the meantime, there is another library that only supports SPDX 3 using the custom serialization format you are able to use, but last I saw does not support conversions from 2.x and has an API that is not what I hope to provide in tools-golang here. Since golang was not really considered when designing SPDX 3, the lack of inheritance in the language ends up making the SPDX object model fairly cumbersome to deal with in practice, but I think I came up with a reasonable set of utilities in the draft PR to make people's lives easier, such as listing all items by type within each collection, pervasively through the model. I don't believe the same is true for the library noted above, so you will need to type assert almost everything to be able to use it.

Of course, I'd love to have any help testing the implementation in the draft PR, too, if someone was interested in moving it forward.

[goneall]

@kzantow - Shortly after you raised the issue of not specifying the specific serialization for SPDX 3 (which many of us agreed was an issue), we created this PR: spdx/spdx-spec#1197

Here's the current description of the serialization spec: https://spdx.github.io/spdx-spec/v3.0.1/model/Core/Vocabularies/ExternalIdentifierType/

If this isn't sufficient, please provide some details or perhaps a clarifying PR to address so we can move forward on SPDX 3 support.

I'll be on the implementers call this morning if you'd like to discuss. There are quite a few people who would really like to update this library for SPDX 3 and your contributions to the library are very much appreciated by the community.

[JPEWdev]

In the meantime, there is another library that only supports SPDX 3 using the custom serialization format you are able to use, but last I saw does not support conversions from 2.x and has an API that is not what I hope to provide in tools-golang here. Since golang was not really considered when designing SPDX 3, the lack of inheritance in the language ends up making the SPDX object model fairly cumbersome to deal with in practice, but I think I came up with a reasonable set of utilities in the draft PR to make people's lives easier, such as listing all items by type within each collection, pervasively through the model. I don't believe the same is true for the library noted above, so you will need to type assert almost everything to be able to use it.

The spdx-go-model bindings are low level generated code bindings; the intention is that these are easily generated based on the SPDX spec, so they can be kept up to date. Projects are free to use these bindings if they desire and they should work just fine; they do take steps to guide users into creating correct documents, providing (incomplete) compile time and runtime validation. However, they are missing some QoL features that tools-golang is better suited to do. The intention is that tools-golang take the generated bindings and add these QoL features on top (for example, conversion from SPDX 2 -> SPDX 3, etc.)

Also, I'm not a go programmer, but I really didn't get a lot of help writing the code generation for go bindings (the automatic generation of the code bindings is IMHO paramount for our maintenance of the SPDX spec), so I do sincerely apologize for how non-go-like they are. If a fluent go programmer would like to make more go-like bindings that are more comfortable, please feel free to make a new generator for shacl2code; my only request is that you use the existing test benches to validate the new bindings (the existing go bindings should be helpful to show how to do that).

[janjaeschke]

@kzantow thank you for the brief explanation, that totally explains why you got stuck here.
I hope the mentioned changes described by @goneall will solve at least a part so you might continue or even solve the entire issue.

Sadly I have way too little experience to support here as any kind of developer :(