[Onboarding] [2.10.0] Install custom-codecs plugin (zstd compression) in 2.10 distribution

[andrross]

Is your feature request related to a problem? Please describe

See this issue for context, but the tl;dr is that consensus formed around the idea to move the ZSTD compression implementation to a plugin, but have it installed by default. This will require some kind of change in the distribution assembly logic.

Describe the solution you'd like

There was continuing discussion on the linked issue about specifically where the plugin should live (namely an separate repo vs inside the /plugins folder of the OpenSearch repo), but as of this moment the zstd implementation is in the custom-codecs plugin in the OpenSearch repo. That plugin will need to be installed by default in the OpenSearch distributions per the agreed upon plan.

Describe alternatives you've considered

No response

Additional context

No response

[peterzhuamazon]

New Repo:

• https://github.com/opensearch-project/custom-codecs

[peterzhuamazon]

We might need to do these since zstd is not bundled directly in zip:

1. We bundle zstd entirely in our zip
2. If not, we need clear docs telling users what to do if zstd not on their systems or not pre-installed
3. Any queries or API calls to test if this feature is running correctly after starting a cluster?

[gaiksaya]

Adding @nknize who might have more info.

[dblock]

AFAIK we're doing this in 2.10 to prevent users that have enabled ZSTD in 2.9 from being DOA when upgrading to 2.10.

[peterzhuamazon]

Offline talk with @reta on the confusions and questions about the zstd lib:

How do I install zstd lib.

The way it works is that the plugin bundles JAR and JAR bundles SOs (native libs) for supported platforms
so when plugin is initialized, it unpacks SOs into /tmp folder and loads native library

Everytime the zstd process initialized it will unpack the so/dll from zstd-jni-1.5.5-5.jar to tmp dir, and load in mem to use.
If the process stopped and restart the extraction happens again.

Any queries or API calls to test if this feature is running correctly after starting a cluster?

Yes

  curl -X PUT -H "Content-Type: application/json"     http://localhost:9200/test     -d '{
          "mappings": {
              "properties": {
                  "tag": { "type": "keyword" }
              }
          },
          "settings": {
              "index.codec": "zstd"
          }
      }'

Then there is a problem since it is native.
you dont have to install a separate zstd package?
do you bundle different so and even dll for linux x64/arm64, win x64?

1. Yes, 2. Yes

@andrross confirms the jar file have all the supported lib files within

$ jar tf zstd-jni-1.5.5-5.jar
...
darwin/
darwin/aarch64/
darwin/aarch64/libzstd-jni-1.5.5-5.dylib
darwin/x86_64/
darwin/x86_64/libzstd-jni-1.5.5-5.dylib
freebsd/
freebsd/amd64/
freebsd/amd64/libzstd-jni-1.5.5-5.so
freebsd/i386/
freebsd/i386/libzstd-jni-1.5.5-5.so
linux/
linux/aarch64/
linux/aarch64/libzstd-jni-1.5.5-5.so
linux/amd64/
linux/amd64/libzstd-jni-1.5.5-5.so
linux/arm/
linux/arm/libzstd-jni-1.5.5-5.so
linux/i386/
linux/i386/libzstd-jni-1.5.5-5.so
linux/loongarch64/
linux/loongarch64/libzstd-jni-1.5.5-5.so
linux/mips64/
linux/mips64/libzstd-jni-1.5.5-5.so
linux/ppc64/
linux/ppc64/libzstd-jni-1.5.5-5.so
linux/ppc64le/
linux/ppc64le/libzstd-jni-1.5.5-5.so
linux/s390x/
linux/s390x/libzstd-jni-1.5.5-5.so
win/
win/amd64/
win/amd64/libzstd-jni-1.5.5-5.dll
win/x86/
win/x86/libzstd-jni-1.5.5-5.dll

[reta]

Here is the reference https://github.com/luben/zstd-jni#binary-releases

[gaiksaya]

Hi @reta @andrross ,

Do we have integration tests enabled for custom-codec so that we can add in the test manifest and that can be run in automated way? https://github.com/opensearch-project/opensearch-build/blob/main/manifests/2.10.0/opensearch-2.10.0-test.yml
If not, can you please create an issue to add them later?

Thanks!

[reta]

Hi @gaiksaya ,

We do have internal cluster tests [1], technically those are the integration ones, I will make sure with have test manifest that runs properly. Thanks for heads up!

[1] https://github.com/opensearch-project/custom-codecs/tree/main/src/internalClusterTest