Add support in e2e-testing repo for Logstash and add test for Logstash output in the Elastic Agent (Standalone and/or Fleet mode).

[ph]

The Logstash team is currently working on a new output that will write to data_stream and will use the fields defined in the received events to define which data_stream it needs to write to.

The current test architecture in stand-alone assumes the following.

Elastic Agent -> Elasticsearch -> assert indices.

Using the Logstash output in the Elastic-Agent should produce the following architecture.

Elastic Agent -> Logstash -> Elasticsearch -> assert indices.

Using the Logstash output should produce the same results at the end, since by default without any filter it should just act as a proxy.

Tasks:

• Add logstash to the end2end.
• Elastic agent should only be able to send data to basic Logstash
• Allow the standalone suite to run with two different environment.

@mdelapenya I would expect the work to be done by the Logstash team if possible, but it would be good to know what we need to do on our side to support that feature and how we want to support it.

cc @jsvd @colinsurprenant

[jsvd]

@ph quick question, is logstash output supposed to work at this point in Elastic Agent? While it's possible to set it in the agent's configuration, it seems agent periodically does a healtcheck HTTP request to the configured output, that logstash beats input doesn't support (not a lumberjack protocol message).

[ph]

This is strange, I don't think we do any special code that would do that. do you have logs?

[jsvd]

[2020-10-15T05:14:42,727][INFO ][org.logstash.beats.BeatsHandler][main][e0de183fb9815c84f2e9fd5606c3ae5dd0b1f867c2dd346dc87813c21a785aa2] [local: 127.0.0.1:3333, remote: 127.0.0.1:51238] Handling exception: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 71
[2020-10-15T05:14:42,729][INFO ][org.logstash.beats.BeatsHandler][main][e0de183fb9815c84f2e9fd5606c3ae5dd0b1f867c2dd346dc87813c21a785aa2] [local: 127.0.0.1:3333, remote: 127.0.0.1:51238] Handling exception: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 69

71 and 69 are the GE characters at the start of a GET / request:

[ph]

Payload is empty? I am surprised that you see that, we don't have healthcheck logic on the output, you dont have that behavior with filebeat directly?

@blakerouse @michalpristas @ruflin any idea?

[jsvd]

this is likely from metricbeat doing healthchecks, perhaps uptime checks?

metricbea 46072 joaoduarte   19u     IPv4 0xf9401d09453b75d         0t0      TCP localhost:51655->localhost:3333 (ESTABLISHED)

[michalpristas]

do you have monitoring enabled? monitoring works only with ES. (maybe we could change that in this release ph). but it might end up with type elasticsearch and configuration body from what you provided.
but aside from that Logstash should work. we dont do anything to output just pass it down.

[jsvd]

@michalpristas that's it! I disabled monitoring (on by default) and logstash doesn't get http calls anymore. Metrics themselves seem to be processed correctly in Logstash with or without monitoring.

+1 on having distinct logic depending on the output type. btw it's fine that this wont be a priority since ES is the only documented output for now

[ph]

@michalpristas Can you create an issue for that for 7.11?

[ph]

created elastic/beats#22051 targetted for next release.

[roaksoax]

cc @andsel

[ph]

I've added the wrong link above elastic/beats#22051

[EricDavisX]

@roaksoax @andsel wanted to follow up, did we ever end up with an e2e-test for Logstash output from Elastic Agent in a different repo / ci, or are we still intending to use this ticket and this e2e-testing repo for it? The Fleet Support is scheduled for a future release, it would be great to have a test for that or stand-alone Agent mode if the team can budget it in.

[EricDavisX]

I'm updating the expectations / short description of the ticket to allow (not require) us to support this from the Fleet mode testing, if we should choose. Development will soon complete a POC for Fleet mode and if we automate against that it is a more inclusive test, while still covering Agent / Logstash side.

FYI: The effort to support multiple outputs (including Logstash) in Fleet is coming in a future release tracked here:
elastic/kibana#108968

The remaining Beats repo (Agent) issue is here:
elastic/beats#24020

While @adam-stokes is assigned, and can help with the framework side, we likely need a Logstash member to help support the test design. I'll ping folks in slack.

[EricDavisX]

I wanted to update (funny timing) I just hear from the Fleet team that this support is going to take longer than I may have made it sounds above. We can talk off-line if we want to do anything more than the standalone mode work, we should probably prioritize that to ensure some coverage on Agent side.

[jlind23]

As discussed with @faec and after a workshop with the robots, we will close this issue for now as we won't add Logstash e2e-tests for beats inside the e2e repository.
Other Logstash tests will be added by @lykkin following the work done on the Logstash output under Elastic Agent.