mysql: Add script processor to fix grok behaviour of ES 8.7.x+

[shmsr]

• Bug

What does this PR do?

Changes deduplicate the list and only keep the unique elements. After this, if only 1 element remains, then it's added to the result as a single element (a string) and if not, the list is kept in the result with unique elements. This handling is done to manage the new behaviour of the grok processor in 8.7.x+. Read the related issue for more details.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Related issues

• Closes MYSQL/IIS log file issue in Integrations #6016

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-06-19T15:32:32.584+0000

• Duration: 26 min 57 sec

Test stats 🧪

Test	Results
Failed	0
Passed	63
Skipped	0
Total	63

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (2/2)	💚
Files	100.0% (2/2)	💚 2.89
Classes	100.0% (2/2)	💚 2.89
Methods	92.857% (26/28)	👍 0.643
Lines	94.34% (150/159)	👍 2.079
Conditionals	100.0% (0/0)	💚

[shmsr]

cc: @HiDAl I've seen you work on similar issues in the past. Could you please also take a look?

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 574dd97

History

• 💔 Build #214 failed 0b3b8e1

cc @shmsr

[HiDAl]

Hey @shmsr, just to be clear: I haven't worked previously on fixing this kind of issue, I just changed the behavior of the Grok processor elastic/elasticsearch#92586.

Now, I don't know the whole business logic (Beats side, since I'm not part of the team), but looking at the code your code does not have a consistent output. In case thread_id is different at any point in the log lines, the processor will return a list of different thread_ids, which is not tested (I don't know whether is possible to have distinct thread_id per log block, though). With your code change, you've just made the current test pass. In case the thread_id might be different within a log block, I think this change is better: https://github.com/elastic/beats/pull/35276/files

[andrewkroh]

Please add a description and a tag to the script processors. This helps in debugging.

The description field documents what the script does which helps other developers and it is displayed in the Ingest Pipeline editor in Kibana. The tag field is included in the error message metadata and ingest node stats to help distinguish between script processors.

[shmsr]

Sure @andrewkroh!

[ishleenk17]

@shmsr : You can add the tests for this change and update the expected json file.

[ishleenk17]

Hey @shmsr, just to be clear: I haven't worked previously on fixing this kind of issue, I just changed the behavior of the Grok processor elastic/elasticsearch#92586.

Now, I don't know the whole business logic (Beats side, since I'm not part of the team), but looking at the code your code does not have a consistent output. In case thread_id is different at any point in the log lines, the processor will return a list of different thread_ids, which is not tested (I don't know whether is possible to have distinct thread_id per log block, though). With your code change, you've just made the current test pass. In case the thread_id might be different within a log block, I think this change is better: https://github.com/elastic/beats/pull/35276/files

In the PR shared in this comment, for any new entry of thread_id in the grok, a change is needed in the painless script.
I suppose the change made here should be fine. It returns list of distinct values when thread ids are different and a single value if thread id is same.

[shmsr]

@shmsr : You can add the tests for this change and update the expected json file.

@ishleenk17 I did add them but had to remove them because of unavailability of setting kibana version for specific tests. Here: 3a656c9

[ishleenk17]

Looks good!