Document use of Elasticsearch 7.6.2 via Apache License 2.0, with a note on license change for 7.11.0 onward

[jodygarnett]

See https://www.elastic.co/pricing/faq/licensing for Elasticsearch license change from 7.11 onward.

Update: 7.10.2 has a known date_time issue, alternatives include:

• https://github.com/elastic/elasticsearch/tree/v7.9.3 has date_range fix
• https://github.com/elastic/elasticsearch/tree/v7.6.2 Already used across the codebase, so it must be tested
• https://github.com/elastic/elasticsearch/tree/v7.5.2 Used in some early examples

[josegar74]

The current version used for ES is 7.11.1:

core-geonetwork/pom.xml

Line 1447 in d2141aa

<es.version>7.11.1</es.version>

We need to have clear how the license change affects GeoNetwork, to check if we can continue with this version or need to downgrade to 7.10.2 or even change to https://github.com/opensearch-project/OpenSearch

[jodygarnett]

The use of ES version 7.11 and greater requires GeoCat products, notably GeoNetwork Enterprise, and GeoCat Live to use a non-open source component in order to function. This does not align with our company values.

It is prudent to hold the version at 7.10.2 while the PSC or geonetwork user meeting assesses this matter.

[jodygarnett]

I went to use main today and noticed this was not yet merged

[josegar74]

According to this: https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/java-rest-high-compatibility.html

The High Level Client is guaranteed to be able to communicate with any Elasticsearch node running on the same major version and greater or equal minor version. It doesn’t need to be in the same minor version as the Elasticsearch nodes it communicates with, as it is forward compatible meaning that it supports communicating with later versions of Elasticsearch than the one it was developed for.

Afaik, that means we can use the High Level Client version 7.10.2 as a dependency in GeoNetwork and use ES 7.10.2 or later version. Seems ES 7.11 introduced some new features for Kibana security & anonymous access that are required by some users, but I don't think are directly used from GeoNetwork (seem more like configuration options for Kibana to restrict the access).

[jodygarnett]

I recommend accepting this pull request (both to have consistent documentation on what to run) and to provide some space for the PSC to make a plan.

[jodygarnett]

todo: Add curl command to troubelshooting section:

 curl -XPUT http://localhost:9200/_cluster/settings -H 'Content-Type: application/json' -d '{ "transient" : { "cluster.routing.allocation.disk.threshold_enabled" : false } }' 

[jahow]

This is definitely necessary in my opinion, thanks @jodygarnett.

[jodygarnett]

Can this be merged please?

[fxprunayre]

From my experience, unfortunately version 7.10.1 and 7.10.2 are not good choices to work with GeoNetwork 4 and will probably in most catalog content cases fail to index properly the full content due to an issue on Elasticsearch (related to update of mapping with range type). On my side I've projects working fine with version from 7.5.x to 7.12.x; but you should consider not fixing the version to 7.10.2.

[jodygarnett]

@fxprunayre I am not sure what to say…

• The existing instructions used random version of elasticsearch, this PR at least made them consistent
• Contains important trouble shooting information that I keep needing to refer to.

If the PR is not going be merged, or is here approved and not merged, please reject it so I can get on with life. I cannot manage to work on main due to these issues, and would prefer to work with the rest of the team.

[jodygarnett]

Like what version should we put here that works?

[fxprunayre]

Like what version should we put here that works?

Here in the PR we have 2 topics:

• the license
• the version

For the version, I'm only saying that most of GN users will probably end up on elastic/elasticsearch#69012 if it is set to 7.10.2. I've not tested all versions, only raising a warning on this one.

[jodygarnett]

Okay I get it a known bug with 7.10.2.

What is a good revision you would recommend (ignoring the license issue for now, I can always make a new PR - but I need to be able to work on main in order to contribute to GN4)

[jodygarnett]

Checking release history:

• 7.10.2 - you indicate this won’t work
• 6.8.14
• 7.11.0 - has license change

So the challenge is to determine if the date_range bug is fixed in 6.8.14?

[jodygarnett]

I am searching to find the exact commit for the date_range bug fix so we can figure out if it is available in an apache licensed branch. I am not sure if all the branches changed license, or just 7.11.0 onward ...

elastic/elasticsearch#69012 (comment) indicates 7.9 did not have this issue as of Feb 16th 2021...

• The release that matches is v7.9.3 :) - we have a Oct 2020 candidate
• You indicate 7.5 worked, v7.5.2 is from Jan 2020
• docker/docker-compose.yml uses 7.6.2 so it must be tested

[jodygarnett]

@fxprunayre thanks for the discussion, changed this PR to make 7.6.2 consistent across the documentation and various ways of starting up. I selected 7.6.2 as it was used by most of the codebase (so I assume it is tested).

[jodygarnett]

The PSC has decided to go with Easlsticsearch, Amazon has since setup Opensearch which is a viable fallback if we run into issue with Business Software License conflicts.

The troubleshooting advise form this PR has been added to #7866