Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

set "keepalive timeout", closing the splunk connection every 5 second… #354

Merged
merged 3 commits into from
Mar 18, 2024

Conversation

metskem
Copy link
Contributor

@metskem metskem commented Oct 5, 2023

First of all, we have been using the splunk-firehose-nozzle (sfn) for many years and are very happy with it.
Since we are getting ever more data being sent to our Splunk environment, we noticed very unequal loads on the heavy forwarders.
Our Splunk setup consists of 6 HFs with an Azure Load Balancer in front.
Because the sfn tries to keep the connection to splunk open (less or more) forever, the Azure Load Balancer doesn't get a chance to properly balance the load to the HFs.
This is due to this draining the response body all the time.

This PR adds a (hardcoded) interval where it will close the connection every 5 seconds.
We have tested this in all our environments, including production and it solves the issue, we see new TCP connections every 5 seconds, and the load is now nicely equally distributed over all HFs.

For simplicity I choose a hardcoded 5 second interval, we can also opt for :

  • a toggle to switch this on/off
  • an option make the interval configurable

Please advise...

…s fixing loadbalancing issues to the Splunk HFs
Copy link
Collaborator

@sbylica-splunk sbylica-splunk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! I'm going to merge it and add configuration options later, before releasing it in v.1.3.1

@sbylica-splunk sbylica-splunk merged commit a68a289 into cloudfoundry-community:develop Mar 18, 2024
@metskem
Copy link
Contributor Author

metskem commented Mar 18, 2024

Great, thanks for the merge!

sbylica-splunk added a commit that referenced this pull request Mar 29, 2024
* Refactoring sonarqube issues (#355)
* Refactoring various functions to lower complexity
* Set "keepalive timeout", closing the splunk connection every 5 second - with configuration (#354)
* Keepalive timeout config (#361)
* Updated dependencies (#363)
* Version bump (#365)
* Edge processor configuration
* Upgrade to golang 1.20 and upgrade compatible dependencies

---------

Co-authored-by: VihasMakwana <121151420+VihasMakwana@users.noreply.github.com>
Co-authored-by: Vihas Splunk <vhiteshbhai@splunk.com>
Co-authored-by: Harry Metske <harry.metske@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants