feat: Add the custom headers syslog http drain RFC #1228
Conversation
beyhan
requested review from
a team,
rkoster,
beyhan,
stephanme,
ameowlia and
ChrisMcGowan
and removed request for
a team
|
Please ensure that doc updates are included as part of this work. |
|
|
||
| We propose the following enhancements: | ||
|
|
||
| 1. **Extend the syslog drain binding specification** to allow a `headers` field (key-value map). |
There was a problem hiding this comment.
Could you please provide a link to the current syslog drain binding specification for reference? Additionally, I think we should mention that it should be extended in a backwards compatible way.
There was a problem hiding this comment.
@beyhan We don't have a formal Syslog Binding specification except this paragraph in the docs. Of course, the API will be extended in a backward compatible way.
Here are the is the whole flow with the affected components:
-
An app developer will make a cf cups cli call with a
-pparameter which will contain JSON with an object of headers among the other parameters like certificates for mTLS for example. -
The Cloud Controller will have store the data in its database. We will have to adjust the syslog_drain_urls_controller from the cloud controller's internal API.
-
The Syslog Agent will have to be adjusted to use possible set headers for HTTP drains when sending data to a Syslog Server. The Syslog Agent must pass the headers transparently without any modification. It will also possibly do some validation of the HTTP Headers passed if needed. The app developer will be informed about the possible problems with Syslog Drain configuration (we are preparing a PR for that) , the errors will be written into the Syslog Agent's error log and metrics about problematic drains will be emitted.
There was a problem hiding this comment.
this looks as a good draft to describe the concept and also mentions the involved CF components. It will be great to incorporate this into the RFC as requested to describe the impacted components.
There was a problem hiding this comment.
We already have documented what the Syslog Agent does here, but we can always improve the docs.
|
|
||
| 1. **Extend the syslog drain binding specification** to allow a `headers` field (key-value map). | ||
| 2. Modify Metron Agent and Syslog Adapter components to **inject the custom headers** into outbound HTTPS syslog requests. | ||
| 3. Implement **validation logic** to block prohibited/unsafe headers (such as `Host`, `Content-Length`). |
There was a problem hiding this comment.
Since the proposal is based on user-provided services: I don't see how a validation of the credentials can be implemented by CLI or CF API (beside validating json structure). The validation could happen on the loggregator side, e.g. you implement an block or allow-list for headers. Same as for syslog drain CAs.
There was a problem hiding this comment.
Yes, we cannot validate things except in the Syslog Agent. Everything will be done as the syslog drain CAs (mTLS workflow). See the detailed steps in my comment above.
There was a problem hiding this comment.
@ZPascal Please address the requests from @beyhan and @stephanme and add a description how this feature will work.
Description
This proposal introduces support for specifying custom HTTP headers in HTTPS syslog drains for Cloud Foundry's Loggregator system. It enables native, secure integration with third-party logging systems that require authentication via bearer tokens or custom headers, such as Splunk HEC, Datadog, and similar modern log routing solutions.
Involved Working Groups:
@cloudfoundry/toc
@cloudfoundry/wg-app-runtime-platform-logging-and-metrics-approvers
@cloudfoundry/wg-app-runtime-platform-logging-and-metrics-reviewers