New Feature: Support Readium Annotation format (import/export)

[panaC]

Goal : Import/export in Thorium of JSON serialized user annotations in Readium Annotation format

Resources :

• https://docs.google.com/document/d/1pJu27gUKBsw5-oFw60JJwhXSz77JYPIkJagoyr1NZjk/edit?pli=1
• https://www.w3.org/TR/annotation-model/

Use Cases :

• A user annotates a publication. When he closes the publication, his annotations are saved in Thorium. As soon as he re-opens the publication, his annotations re-appear.

• A user decides to export from Thorium an EPUB publication he has previously annotated. He selects the option to save his annotations in the publication. The exported EPUB contains his annotations. If he imports the publication in another reading application which supports this specification, his annotations re-appear.

• A user decides to export from his reading application the annotations he has created in a publication. He chooses a file name and a destination folder and validates his choice. An annotation file is created on his computer. If he imports this file into another reading application which supports this specification, and if the reading application already contains this publication, his annotations now appear in the publication.

• A teacher reads an ebook and prepares annotations. He exports an annotation file and shares this file with his students. The students import the ebook and the detached file into their reading application: the annotations made by the teacher now appear in the ebook. The students can now add their own annotations to the ebook, and these student annotations will not be mixed with the annotations created by their teacher.

Key Features :

• Support of the Readium Annotation Format, subset of the W3C annotation model, cf google docs

• User annotation serialized in JSON Readium Annotation Format is tied to the publication (ebook). Export of all publication user annotations in one JSON file is not possible.

• Export to JSON Readium Annotation of a file saved in the folder chosen by the user with the name: <title of the collection>.annotation.

• Import of the file .annotation only in Library Window by drag and drop or click on the import annotation button in catalog menu of the publication.

• Export of the file .annotation only in Library Window by click on the export annotation button in catalog menu of the publication

• Export of the publication (ebooks) with Readium Annotation Set saved in it by click on the the "save publication as" button of the catalog menu.

• When importing, a matching phase between the publication ID and the publication identification ID of the Readium Annotation document is necessary. In case of identification conflict, the choice must be left to the user.

• When importing a Readium Annotation file, a modal appears to the user to identify the corresponding publication, the number of annotations to import, the management of conflicts between 2 uniquely identified annotations, the name of the collection (title of the collection by default) and finally the possibility to continue or cancel the import.

• When exporting a Readium Annotation file, a modal appears allowing you to choose the set of annotations to export by their tag identifier, (annotation not identified by tags included).

• Tag management by annotation in reader window. Add/Remove/Update tags for each annotation in the reading window. Filter annotation by tags name.

[panaC]

Current implementation in Thorium :

Thorium currently support the export of all annotations set from a publication in a first version of the Readium Annotation format, based on this model .
Export save the annotation file on filesystem directory chosen by user, under the name <uuid>.annotation.

The export of annotations is possible for each publication and is accessible by a button in the catalog menu of the library window. This button opens a system modal window to choose the folder in which the annotation will be saved. Implementation here

Readium Annotation format exported from children litterature publication :

{
  "@context": "http://www.w3.org/ns/anno.jsonld",
  "id": "urn:uuid:79fba412-0ebe-45dd-8ad5-99699cbdfc72",
  "type": "AnnotationSet",
  "generator": {
    "id": "https://github.com/edrlab/thorium-reader/releases/tag/v3.1.0-alpha.1",
    "type": "Software",
    "name": "Thorium 3.1.0-alpha.1",
    "homepage": "https://thorium.edrlab.org"
  },
  "generated": "2024-09-03T13:54:04.287Z",
  "label": "Annotations set",
  "about": {
    "dc:identifier": [
      "urn:isbn:code.google.com.epub-samples.cc-shared-culture"
    ],
    "dc:format": "application/epub+zip",
    "dc:title": "Creative Commons - A Shared Culture",
    "dc:publisher": [
      "Creative Commons"
    ],
    "dc:creator": [
      "Jesse Dylan"
    ],
    "dc:date": "",
    "dc:source": "urn:uuid:74f0d5ae-5e83-48b9-a08c-701d7911eb9d"
  },
  "total": 1,
  "items": [
    {
      "@context": "http://www.w3.org/ns/anno.jsonld",
      "id": "urn:uuid:181819eb-c7bd-4310-a00c-d64954d86b02",
      "created": "2024-09-03T13:54:04.288Z",
      "modified": "2024-09-03T13:54:04.288Z",
      "type": "Annotation",
      "hash": "",
      "body": {
        "type": "TextualBody",
        "value": "The work of Creative Commons",
        "format": "text/plain",
        "color": "#fad0c3"
      },
      "target": {
        "source": "EPUB/xhtml/p50.xhtml",
        "meta": {
          "headings": [
            {
              "level": 2,
              "txt": "a new kind of folk culture"
            }
          ],
          "page": ""
        },
        "selector": [
          {
            "type": "TextQuoteSelector",
            "exact": "The work of Creative Commons",
            "prefix": "d get the law out of the way. ",
            "suffix": " is really about laying\n                the in"
          },
          {
            "type": "ProgressionSelector",
            "value": 0.14235545602605862
          },
          {
            "type": "DomRangeSelector",
            "startContainerElementCssSelector": "p:nth-child(2)",
            "startContainerChildTextNodeIndex": 0,
            "startOffset": 133,
            "endContainerElementCssSelector": "p:nth-child(2)",
            "endContainerChildTextNodeIndex": 0,
            "endOffset": 161
          },
          {
            "type": "FragmentSelector",
            "conformsTo": "http://www.idpf.org/epub/linking/cfi/epub-cfi.html",
            "value": "epubcfi(/4/2/4,/1:133,/1:161)"
          }
        ]
      }
    }
  ]
}

The 4 selectors are generated here

Taking Annotation

model

To create an annotation select/highlight text while reading the publication.
Press the "take annotation" button in the top bar (or keyboard shortcut Ctrl+Shift+Alt+A). a modal appears and persists the annotation in the [application] database(

thorium-reader/src/renderer/reader/components/ReaderHeader.tsx

Line 1333 in 91b1aa7

saveAnnotation: (color: IColor, comment: string, drawType: TDrawType) => {

). (redux). The persistence is attached to the publication, then persisted by the main process (reader process to main process). The annotation is listed in the annotation control panel. This control panel does not allow adding tags or filtering annotations.

[panaC]

This a Work In Progress Issue description. I will append comments here to describe each point.

[panaC]

Importing annotations

Laurent quote from GDocs :

To simplify the association of an imported annotation set and its associated publication, a Reading System MUST offer a way to select a publication, and then select an annotation set. A drag&drop of an annotation set into a Reading System MAY also be proposed, but finding the proper publication from the metadata present in the annotation set is more difficult.

If during the import of an annotation set, an annotation (which is uniquely identified) is re-imported, the Reading System MUST offer to the user the choice to override or not the existing annotation with the new one, or abort the import of the annotations set.

If different annotation sets are imported sequentially and refer to the same publication, a Reading System SHOULD displays a message with the title of the annotation set and the number of annotations in the set. The Reading System MUST offer to the user the choice to abort the import.

When a user imports an annotation set, he SHOULD be offered an additional tag for each imported annotation. The proposed tag is by default the title of the Annotation Set. The user can edit this label (or replace it by a string of his choice). He can also choose to import the annotation set with no added tag.

The advantage of doing this is that every annotation in a set titled “Grade B 2024” can for instance be tagged by the user as “teacher”. It will become easy to only display “teacher” annotations. This will stay true if the teacher has added other tags to his annotations.

flowchart TD
    A["Import file by button"]
    B["Parsing Annotation set"]
    C["Matching between annotation id and publication id"]
    D["Looking for id conflicts"]
    E["User modal confirmation"]
    F1["Abort"]
    F2["Confirm import"]
    G["New annotations imported to publication database"]

    A --> B
    B --> C
    C --> D
    D --> E
    E --> F1
    E --> F2
    F2 --> G

Loading

Implementation flowchart:

In main process triggered by an action with the pubId :

1. open file picker
2. read the file
3. parse the file to readium annotation format
4. check if the publication identifier or publication name from the file is equal to the current publication
5. If no -> show an alert dialog : "do you want to continue?"
6. start a loop of matching with each newly imported annotation with existing annotation set
7. If conflicts with a newly imported annotation -> show an alert dialog with 4 buttons : "Do you want to replace it?" : "Yes" , "Yes for all", "No", "No for all".
8. dispatch a toast notification with a failure or a success

[panaC]

exporting annotations

Laurent quotes from GDocs :

When a user decides to export an Annotation Set from a reading system, he is proposed to filter the annotation set by tags (multiple choice) and he can enter a title for the Annotation Set (which can be left empty).

The filtering is an opt-in. “Untagged” MUST be a possible choice and ALL MUST be an option.

The advantage of this practice is that a user can export his personal annotations (untagged) but no “teacher” annotation.

Thanks to this PR #2538 (#2521 (comment)). We can now filter annotations list with tags name, color, and drawType (highlight). The goal is to use the tags filter component to choose a set of annotations to export. In the annotations list header, there are 3 buttons: "filter", "delete", "export". Filter to open the filter component modal. Delete to open the delete confirmation modal and remove the annotation set currently filtered. Export to open the export annotation set modal and allows you to choose the title name, the file format and the save directory.

Implementation details of the Pull Request merged : #2551 :

1. Export button added to the readerMenu annotations list
2. trigger the showSaveFilePicker
3. User can for the moment only choice .annotation (json) file format (html, csv, will be added later)
4. generate the readium annotation set in json format
5. write the content to the file with the web api
6. dispatch a toast notification with a failure or a success

thorium-reader/src/renderer/reader/components/ReaderMenu.tsx

Lines 821 to 838 in f7b410e

	onClick={async () => {
	try {
	const fileHandle = await (window as any).showSaveFilePicker({ excludeAcceptAllOption: true, id: publicationView.identifier.slice(0, 32), suggestedName: "myAnnotationSet.annotation", types: [{ description: ".annotation", accept: { "application/rd-annotations+json": [".annotation"] } }] });
	const writable = await fileHandle.createWritable();
	const annotations = annotationList.map(([, anno]) => anno);
	const contents = convertAnnotationListToW3CAnnotationSet(annotations, publicationView);
	const jsonData = JSON.stringify(contents, null, 2);
	await writable.write(jsonData);
	await writable.close();
	dispatch(toastActions.openRequest.build(ToastType.Success, __("catalog.exportAnnotationSuccess", {fileName: fileHandle.name})));
	} catch (e) {
	dispatch(toastActions.openRequest.build(ToastType.Error, __("catalog.exportAnnotationFailure", { errorTxt: e?.toString() })));
	}

[panaC]

Annotation tags management

Laurent quotes in GDocs

The addition of tags to annotations enables filtering out unwanted annotations.
For instance, a user can temporarily hide his personal (untagged) annotations and only display “teacher” annotations.

Use Cases

• A user create a new annotation, set a tag, and save the annotation. The annotation is saved and visible in navigation menu. The user car see and edit/remove tags of the annotation.

• A user want to filter annotation by tag name to create a collection. He can open the navigation menu under "annotation" section and filter by click on tag name from any annotation. The user can also filter tag collection with a select component at the top of the annotation panel. The annotation panel remain paginated even with a filter on specific tags.

• A user has filtered annotation panel with specific tags and lock the panel as dock view instead of modal view. A user navigate on the publication and click on any annotation highlight, the annotation panel focus on the annotation selected and remove any tags filter, to focus on the right annotation clicked.

PR #2529
Issue #2524

[llemeurfr]

About the import of an annotation set, after parsing the file, Thorium should display a modal containing:

• the title of the annotation set,
• the number of annotations in the set
• an indication that X annotations in the set are already present in Thorium (if conflicts are found)
• a text input for an optional tag

The user has the choice to abort the import or confirm.

[llemeurfr]

The update spec is in https://github.com/readium/annotations