User Manual refresh using Markdown and MkDocs

[jodygarnett]

Meeting

Demo / discussion:

1. Review of docs/manual/README.md build and deploy instructions
2. Deploy with mike was cool, but more research needed.
   Open question how to have 4.4-SNAPSHOT docs? latest redirects to 4.2.5 and is not separate
3. Open question on handling of translations (whole document or POT files)

Decisions:

• decide to deploy to github pages, or update website folder? GitHub Pages ...
• Do we want a CNANEM for http://docs.geonetwork-opensource.org/ ? Yes ..
• IMPORTANT: Do not do any structure changes on docs so redirect is easy (navigation does not change URLs)

merge planning:

• jody: some small structure changes from User Manual refresh content updates #7363
• jody: enable language chooser (required site_url to work correctly Unclear how to work with mkdocs-static-i18n and mike versioning at the same time ultrabug/mkdocs-static-i18n#265)
• Jose: update mkdocs.yml site_url to http://docs.geonetwork-opensource.org just before merge
• Jose: merge to git 4.2.5 published, mike publish 4.2.5
• Jose: backport to the 4.2.x branch? and remove what is not needed
• Jose: backport to the 3.12.x branch? and remove what is not needed
• Jose: Check what is different between https://github.com/geonetwork/doc/tree/3.12.x and https://github.com/geonetwork/doc/tree/4.0.x ?
• Jody: should we run the script on 2.10.x branch https://github.com/geonetwork/core-geonetwork/tree/2.11.x/docs? Good for for the version chooser? Or keep the 2.10.x manual online as it is ... (try experiment, keep static files as backup)
• Juan: setup subdomain
• Juan: check about redirect to GitHub pages
• Jose: existing geonetwork links to documentation (https://github.com/geonetwork/core-geonetwork/blob/main/web/src/main/webapp/WEB-INF/data/data/resources/config/manual.json) this would need update if there are any structure changes.

• Jose: Create new GitHub action to build and publish tags
• Jose: Turn off build jobs when complete

Final Review Update

Review:

• links and titles for references
• links and titles for docs
• toctree content
• version publishing with mike plugin (see readme below)
• warning, info, seealso, since version, etc...
• code-blocks: careful review of xml and javascript examples
• origional docs have strange things, do not fix at this time!
• maven integration with build
• maven integration with web app (including online help)

How-to included in PR:

• https://github.com/GeoCat/core-geonetwork/blob/mkdocs/docs/manual/README.md
• https://github.com/GeoCat/core-geonetwork/blob/mkdocs/docs/manual/translate/README.md

IMPORTANT:

The translate script handles sphinx-build content and references, this information will be lost once we are done.
Until the migration is complete it is important to correct content (broken links) and formatting (for indenting, accidental block quotes) in the origional rst files.

See pull-request: geonetwork/doc#245

Code-complete update

1. Preview available here https://jodygarnett.github.io/core-geonetwork/

   • We are interested in review of the content

   • Want to ensure the xml and javascript code examples are not harmed in the transition for example

2. The docs are now pretty much 100% convertible using Jody's translate script wrapper on pandoc:

   python3 -m translate rst manual/**/*.rst

3. With this script working so well it is now "easier" to fix the sphinx-build content to correct formatting for indenting, accidental block quotes and the like.

4. For later:

   • Any restructuring/improvement of content
   • Decision about translating using POT (weblate or transfix) or whole document conversion (deeply, google, apple)

Initial code-sprint proposal

The following work is proposed: https://github.com/geonetwork/core-geonetwork/wiki/Proposal-mkdocs.md

• pandoc conversion of existing docs from sphinx-build rich-structured-text to mkdocs Markdown format. This lowers barrier of entry as Markdown is much more popular than rich-structured text.
• core-geonetwork codebase docs/ folder to directly manage user-manual alongside codebase (PRs changing UI required to change docs.)
• Expand the Quickstart section into an end-user manual "online help" for visitors
• inclusion of translated files as entire documents (removing the use of Transfix for documentation)
• To help a python translate script provided to automate the creation of new translation files using deepl (even if it just provides a starting point that is fine)

Keep in mind:

• Material for MkDocs and PanDoc (used for translation script) have different flavours of Markdown and Markdown Extensions.
• A strict writing guide documents what works today with both mkdocs and translation via deepl. This script can improve over time if more of the mkdocs extensions are wished

Compare:

• mkdocs end-user manual: https://jodygarnett.github.io/iso19139.ca.HNAP/
• sphinx user end-user manual: https://metadata101.github.io/iso19139.ca.HNAP/

Example page:

[jodygarnett]

Code sprint notes:

• branch: https://github.com/GeoCat/core-geonetwork/tree/mkdocs

• help online help for anonymous users, end-user focused content with no cross links to admin
• manual port of sphinx-build docs (does not conflict with manuals git submodule)

Discussion:

• user manual "reference" (what things do) vs "operations" (how to use instructions)

[jodygarnett]

Note we will need to wish to update the sample data for screen snaps (some web services have changed and we need an update in order to have working Add to map demo):

• https://data.apps.fao.org/map/catalog/srv/eng/catalog.search#/metadata/e54e2014-d23b-402b-8e73-c827628d17f4

[MichelGabriel]

It would be nice to add an extra menu to the GeoNetwork menu bar that points to parts of the documentation.

[jodygarnett]

There is a link in the footer presently and a build option to include docs in release.

I hope we can include a small “online help”’for visitors. If we do the online help doc theme right it can pick up the logo of the running geonetwork and have a similar menu bar appearance.

[jodygarnett]

Okay the script is doing great for converting files unattended, however references are a pain to update manually.

Looks like we have 366 items to update:

grep --include=\*.rst -rnw . -e ":ref:`.*`" | wc -l                                                                                                                                         366

Reviewing examples:

./overview/change-log/version-3.10.1.rst:10:* CSW / Configuration of the capabilities is now done using a service metadata record :ref:`csw-configuration`
./annexes/standards/dublin-core.rst:25:* :ref:`dublin-core-view-default`
./annexes/standards/iso19115-3.2018.rst:121:See :ref:`iso19115-3.2018-elem-mdb-identificationInfo-7b3f7b7fbb8a986c92658058fe54f876`

So while may end up being links to page, or header, references. The last example seems unnecessarily difficult!

The online manual has https://geonetwork-opensource.org/manuals/4.0.x/en/annexes/standards/iso19115-3.2018.html?#iso19115-3-2018-tab-identificationinfo

The annexes/standards/iso19115-3.2018.rst source has:

.. _iso19115-3.2018-elem-mdb-identificationInfo-7b3f7b7fbb8a986c92658058fe54f876:
      
Identification info
===================

:Name:
    
    mdb:identificationInfo

:Description:
  
.. raw:: html

   Basic information about the resource(s) to which the metadata
         applies

So something seems off, perhaps these things linked to the external

[jodygarnett]

How many anchors do we have:

 grep --include=\*.rst -rnw . -e "^.. _.*:$" | wc -l                                                                                                                                    
    2105

Well that is intimidating:

./api/index.rst:1:.. _api-guide:
./overview/change-log/version-3.10.7.rst:1:.. _version-3107:
./annexes/standards/dublin-core.rst:164:.. _dublin-core-elem-dc-contributor-0974cafd6cf5302fe8501874dbe3b3ac:

The vast majority are in the annexes, most of the other are simple page lookup.

[jodygarnett]

Processing by hand:

grep --include=\*.rst -rnw . -e "^.. _.*:$" > anchors.txt

Grep search and replace (I used an editor):

• ^\./ -->
• ^([\w\-/\.\d_]+)\.rst:\d*\:\.\.\s+_([\w\-\d_\s\./]+):$ --> \2=/\1.md#\2

Result anchor=absolute markdown reference

toc=/index.md#toc
administrator-guide=/administrator-guide/index.md#administrator-guide
...

With one leftover:

annexes/standards/iso19139.rst:24731:.. _iso19139-cl-gmd-MD_ScopeCode-/ancestor--node()[name()='gmd-MD_Metadata']/gmd-identificationInfo/srv-SV_ServiceIdentification:

See attached: anchors.txt

[fxprunayre]

@jodygarnett some documentation pages are created from the schema plugins

So Annexes > Standards > *
and https://geonetwork-opensource.org/manuals/4.0.x/en/customizing-application/editor-ui/creating-custom-editor.html
can probably be excluded from your scripts
and we can probably replace rst writer by the new format in
https://github.com/geonetwork/core-geonetwork/tree/main/docs/schema-doc/src/main/resources

[jodygarnett]

Well this is disheartening - @MichelGabriel updated the version of mkdocs-i18n-static to version 1.0.x and we have encountered a bug: ultrabug/mkdocs-static-i18n#258

I do hope these plugins do not change often, but I was using a <1.0 release of mkdocs-18n-static so change should be expected.

I think the nice thing to do for now is to leave the files as text, and include them in one big long change log. Renaming the files would be unpleasant.

[jodygarnett]

The PR is now code complete, preview available for visual review here https://jodygarnett.github.io/core-geonetwork/

[jodygarnett]

We can probably replace rst writer by the new format in https://github.com/geonetwork/core-geonetwork/tree/main/docs/schema-doc/src/main/resources

Thanks for that information - ticket here: #7342

[MichelGabriel]

It looks like some of the relative links are broken, they always point to the file in the same directory, which is not always the case and the named anchor seems to end up in the filename somehow.

[jodygarnett]

Updated the bug description, all links are fixed and the build is clean with no warnings.

I am getting private feedback on content - I would prefer of content updates are in a second pull request.

[jodygarnett]

I enabled the language chooser, but here is some strange interaction with using mike to publish versions.
I opened a ticket here ultrabug/mkdocs-static-i18n#265 in hopes of guidance.

Update: resolved by reading the manual (site_url was required)

[jodygarnett]

I think this ticket can now be closed?