[MD] Issue connecting to data source cluster with self-signed certificates

[zhongnansu]

I had problems creating a data source connection to an OpenSearch with HTTPS protocol (created with a self-signed certificate). I received this problem:

Response:

{
    "statusCode": 400,
    "error": "Bad Request",
    "message": "Data Source Error: unable to verify the first certificate",
    "attributes": {
        "error": "Data Source Error: unable to verify the first certificate"
    }
}

observation:
I tested the communication in the opensearch-dashboard terminal.
I executed $ curl -k https://opensearch-b:9200 and worked it.

---

When I configure a new data source connection to an OpenSearch with HTTPS protocol with a valid certificate, it works as expected. It also works when I configure OpenSearch with HTTP protocol.

---

Do you have some tips or suggestions? Is it a requirement that the data source connection with HTTPS must have a valid certificate? I did not find this information in the documentation.

• OpenSearch: 2.8.0
• OpenSearch-Dashboards: 2.8.0
• Tests made on docker-compose

Originally posted by @carlos-neto-trustly in #2829 (comment)

[zhongnansu]

I am able to reproduce the issue. Even with opensearch.ssl.certificateAuthorities: ['path/to/root-ca.pem'] configured in opensearch-dashboards.yml. It will still fail. The issues are,

1. Datasource clients is configured to always validate certificate by default.
2. The ssl config in� opensearch-dashboards.yml is not being passed to data source client

related code:

OpenSearch-Dashboards/src/plugins/data_source/server/client/client_config.ts

Lines 22 to 25 in 24bde32

	ssl: {
	requestCert: true,
	rejectUnauthorized: true,
	},

Those issues can also be part of #2009, that we should have a way to support configuration of datasource clients, including SSL configuration

[c-neto]

Hello! @zhongnansu

I would like to share my collaboration with this Issue.

I created a docker-compose lab to reproduce the problem described in this Issue.

• https://github.com/augustoliks/lab-opensearch/tree/main/multi-data-source-bug

FYI: I'm @carlos-neto-trustly.

[Martin-Kemp]

I have the same issue. The setting opensearch.ssl.verificationMode: none is clearly not being used for datasources.

[Martin-Kemp]

Workaround as described by carlos-neto-trustly here:

Set and environment variable NODE_EXTRA_CA_CERTS=/path/to/CAs when starting your container.

Node docs: https://nodejs.org/api/cli.html#node_extra_ca_certsfile

This worked for me.

[uzhinskiy]

Hi.
We are using 2 similar opensearch-clusters v2.10 - 'clusterA' and 'clusterB'. The 'clusterB' runs with enabled option plugins.security.ssl.http.clientauth_mode: REQUIRE.
When we try to create a data source in opensearchdashboardA we get an error:

Data Source Error: C0475B41D77F0000:error:0A000412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:../deps/openssl/openssl/ssl/record/rec_layer_s3.c:1605:SSL alert number 42

Settings for opensearch_dashboards.yml in the "clusterA":

    opensearch.ssl.verificationMode: none
    opensearch.ssl.alwaysPresentCertificate: true
    opensearch.ssl.certificateAuthorities: [ "/usr/share/opensearch-dashboards/config/elk-root-ca.pem", "/usr/share/opensearch-dashboards/config/remote-root.pem" ]
    data_source.enabled: true

We also tried setting env for opensearch-dashboards in clusterA, but without success:

NODE_TLS_REJECT_UNAUTHORIZED=0
NODE_EXTRA_CA_CERTS=/usr/share/opensearch-dashboards/config/remote-root.pem

The remote-root.pem contains chain of certificate for clusterB's opensearch.

However, adding the datasource works correctly with plugins.security.ssl.http.clientauth_mode: clientauth_mode: OPTIONAL in clusterB and set NODE_EXTRA_CA_CERTS variable .

Is it possible to set up 'datasource' when a remote cluster using plugins.security.ssl.http.clientauth_mode: REQUIRE?

[cwperks]

I ran into this issue while looking into setting up multiple datasources when the security-dashboards-plugin is installed. I left a comment on the issue in the security-dashboards-plugin repo that has a zip containing a docker setup with 2 clusters that demonstrates the issue. opensearch-project/security-dashboards-plugin#1782 (comment)

[derek-ho]

+1 if we can prioritize a fix that would be awesome! I think ideally the client should be using the same settings as the connection to the local cluster specified in the dashboards configuration.

[BionIT]

This is due to ssl config for rejectUnauthorized is hardcoded as true for now in data source client config, tested that changing this to false will not validate the certificate. Thus to solve this, we can expose a property data_source.ssl.rejectUnauthorized which will pass over to ssl configuration so that this behavior can be configured. The opensearch.ssl.verificationMode:none is the same as configuring ssl.rejectUnauthorized as false, see ref

OpenSearch-Dashboards/src/core/server/opensearch/client/client_config.ts

Line 137 in 735424b

ssl.rejectUnauthorized = false;

The reason I think a new config is better is because opensearch.ssl.verificationMode is for local cluster setting, while we are actually configure for connected data sources here. But open to other suggestions, @derek-ho @cwperks

[derek-ho]

This is due to ssl config for rejectUnauthorized is hardcoded as true for now in data source client config, tested that changing this to false will not validate the certificate. Thus to solve this, we can expose a property data_source.ssl.rejectUnauthorized which will pass over to ssl configuration so that this behavior can be configured. The opensearch.ssl.verificationMode:none is the same as configuring ssl.rejectUnauthorized as false, see ref

OpenSearch-Dashboards/src/core/server/opensearch/client/client_config.ts

Line 137 in 735424b

ssl.rejectUnauthorized = false;

The reason I think a new config is better is because opensearch.ssl.verificationMode is for local cluster setting, while we are actually configure for connected data sources here. But open to other suggestions, @derek-ho @cwperks

Thanks. That should work, and I agree that we may want different values for different datasource that is not the backing one. However, this begs this question - would this apply to all datasources? If that is the case and not specific to a datasource, I think we should call it out in the documentation that we are assuming all datasources are compatible with the ssl setting. I know that @cwperks had the same question for TLS settings - are we assuing/calling it out that a built in assumption is that the config works for all datasources?

[cwperks]

@BionIT For ultimate flexibility, TLS would need to be configured on a per data-source basis. Would data_source.ssl.rejectUnauthorized: false disable SSL verification for all connected data-sources?

I am thinking about TLS configuration in 2 ways:

1. TLS configuration is common across all configured data-sources so it only needs to be configured once
2. If flexibility is needed per-datasource then there would also need to be a way to configure TLS per data-source.

[cwperks]

When configuring TLS for Dashboards, there are 2 separate traffic flows that need to be secured:

1. From the user's browser to OpenSearch Dashboards
2. From Dashboards to the backend data-source

For 2, ultimate flexibility would mean configuring this per data-source, but there could be a common configuration for multiple data-sources.

The settings to configure 2) are:

SSL Verification Mode: This setting allows you to specify how OpenSearch Dashboards verifies the certificates from OpenSearch nodes.

opensearch.ssl.verificationMode: full

Possible values for this setting include:

• full: (Default) Verifies that the provided certificate is signed by a trusted authority (CA), that the certificate is not expired, and that the common name (CN) or subject alternative name (SAN) matches the requested hostname.
• certificate: Verifies that the provided certificate is signed by a trusted authority (CA), but does not perform hostname verification.
• none: Does not perform any verification.

SSL Certificate Authorities: If you're using self-signed certificates or certificates signed by your organization's CA, you need to specify the path to the CA certificate(s) used to sign the certificates for OpenSearch nodes.

opensearch.ssl.certificateAuthorities: ["/path/to/your/ca.pem"]

---

This next one is only required if clientcert auth is set to REQUIRED.

Client Certificate and Key: Specify the paths to the certificate and key files that OpenSearch Dashboards will use to authenticate itself to the OpenSearch cluster.

opensearch.ssl.certificate: "/path/to/your/client.crt"
opensearch.ssl.key: "/path/to/your/client.key"

[seraphjiang]

close as per #6171