[HA Proxy] TSDB Enablement

[agithomas]

Test Environment Setup

• Creation of Test Environment

Datastream : info

• Add dimension fields - [HA Proxy]Add dimension mapping to info datastream #7201
• Add metric_type mappings - [HA Proxy] Add metric_type mapping to info datastream #7164

Verification and validation

• Verification of data in visualisation after enabling the TSDB flag in kibana
• Verification of the count of documents (before & after TSDB enablement) in Discover Interface
• Verify if field mapping is correct in the data stream template.
• Verify if the TSDB can be disabled using the API method
• Verify if the TSDB can be re-enabled after 4 hours, following disablement using API method, making use of package upgrade.
• Add screenshots of dashboards (Recommended), if applicable.

Datastream : stat

• Add dimension fields - [HA Proxy] Add dimension fields for stat datastream #7323
• Add metric_type mappings - Add metric_type mapping to stat datastream of HA Proxy #7183

Verification and validation

• Verification of data in visualisation after enabling the TSDB flag in kibana
• Verification of the count of documents (before & after TSDB enablement) in Discover Interface
• Verify if field mapping is correct in the data stream template.
• Verify if the TSDB can be disabled using the API method
• Verify if the TSDB can be re-enabled after 4 hours, following disablement using API method, making use of package upgrade.
• Add screenshots of dashboards (Recommended), if applicable.

TSDB Release PR

• Fix dashboard to use the correct aggregation function - [HA Proxy ]Corrected incorrect aggregation function used in visualisation and dashboard #8204
• PR to enable TSDB - TSDB enablement for HAProxy datastreams #8224
• Update the tracking sheet (GA release version, github link, status)

Blocked by

• [haproxy] Correct the mapping of source.address to keyword type instead of text #7440
• Document how to reindex a tsdb data stream elasticsearch#99176

[agithomas]

haproxy stats datastream has field having text type mapping. This would leads to issue as below when on doing elastic-package install

Install the package
Error: can't install the package: can't install the package: could not install package; API status code = 500; response body = {"statusCode":500,"error":"Internal Server Error","message":"illegal_argument_exception\n\tRoot causes:\n\t\tillegal_argument_exception: field [haproxy.stat.source.address] of type [text] doesn't support synthetic source unless it is stored or has a sub-field of type [keyword] with doc values or stored and without a normalizer"}

[agithomas]

The above mentioned issue is related to

elastic/elasticsearch#96254
elastic/elasticsearch#97039

[agithomas]

Below are the details of dashboard testing report post enabling the TSDB & metric_type mapping. The visualisation library elements marked as Not Ok must be corrected to use the appropriate aggregation function.

Requests - OK
Health Check - Not Ok
Connections - Not Ok
Downtime seconds - Not Ok
Traffic Volume - Not Ok
Average Connection time - Ok
Servers per connection - Not Ok
Backends per connection - Not Ok
Average response time - Ok
Connections per server - Not Ok
Frontends per connection - Not Ok
HTTP response codes - Not Ok
Number of Server connections - Not Ok
Active servers in backend - Ok
Average time in queue - Ok

[agithomas]

Health Check Metrics
Before

After

Connections
Before

After

Downtime seconds

Before

[agithomas]

Traffic Volume
Before

After

Servers per connection

Before

After

Backend per connection

Before

After

[agithomas]

Connections per server
Before

After

Frontend per connection

Before

After

HTTP Response Code

Before

Number of Server connections

Before