Skip to content

feat: Add the custom headers syslog http drain RFC #1228

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat: Add the custom headers syslog http drain RFC #1228

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 41 additions & 0 deletions toc/rfc/rfc-draft-custom-http-headers-for-syslog-https-drains.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# Meta
[meta]: #meta
- Name: Support Custom HTTP Headers for Syslog HTTPS Drains
- Start Date: 2025-06-30
- Author(s): ZPascal
- Status: Draft
- RFC Pull Request: [community#1228](https://github.com/cloudfoundry/community/pull/1228)

## Summary

This proposal introduces support for specifying **custom HTTP headers** in **HTTPS syslog drains** for Cloud Foundry's Loggregator system. It enables native, secure integration with third-party logging systems that require authentication via bearer tokens or custom headers, such as **Splunk HEC**, **Datadog**, and similar modern log routing solutions.

## Problem

Modern observability platforms increasingly rely on HTTP-based ingestion with authentication through bearer tokens or custom headers (Splunk, Datadog). Currently, Loggregator supports HTTPS drains but **does not allow adding custom headers**, making integration with key services (e.g., Splunk HEC, Datadog, New Relic, Elastic) difficult or impossible without proxies or custom sidecar containers. Using proxies adds complexity and operational burden, while putting authentication in drain URLs is insecure and often not compatible.

## Proposal

We propose the following enhancements:

1. **Extend the syslog drain binding specification** to allow a `headers` field (key-value map).
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please provide a link to the current syslog drain binding specification for reference? Additionally, I think we should mention that it should be extended in a backwards compatible way.

Copy link
Contributor

@chombium chombium Jul 8, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@beyhan We don't have a formal Syslog Binding specification except this paragraph in the docs. Of course, the API will be extended in a backward compatible way.

Here are the is the whole flow with the affected components:

  1. An app developer will make a cf cups cli call with a -p parameter which will contain JSON with an object of headers among the other parameters like certificates for mTLS for example.

  2. The Cloud Controller will have store the data in its database. We will have to adjust the syslog_drain_urls_controller from the cloud controller's internal API.

  3. The Syslog Agent will have to be adjusted to use possible set headers for HTTP drains when sending data to a Syslog Server. The Syslog Agent must pass the headers transparently without any modification. It will also possibly do some validation of the HTTP Headers passed if needed. The app developer will be informed about the possible problems with Syslog Drain configuration (we are preparing a PR for that) , the errors will be written into the Syslog Agent's error log and metrics about problematic drains will be emitted.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this looks as a good draft to describe the concept and also mentions the involved CF components. It will be great to incorporate this into the RFC as requested to describe the impacted components.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We already have documented what the Syslog Agent does here, but we can always improve the docs.

2. Modify Metron Agent and Syslog Adapter components to **inject the custom headers** into outbound HTTPS syslog requests.
3. Implement **validation logic** to block prohibited/unsafe headers (such as `Host`, `Content-Length`).
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since the proposal is based on user-provided services: I don't see how a validation of the credentials can be implemented by CLI or CF API (beside validating json structure). The validation could happen on the loggregator side, e.g. you implement an block or allow-list for headers. Same as for syslog drain CAs.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we cannot validate things except in the Syslog Agent. Everything will be done as the syslog drain CAs (mTLS workflow). See the detailed steps in my comment above.

4. Ensure the headers field is visible in drain metadata available to platform operators and log integration tools.

### Example configuration

#### Example CUPS call
```bash
cf create-user-provided-service DRAIN-NAME -l SYSLOG-URL -p '{"headers": {"Authorization": "Bearer abc123xyz", "X-Splunk-Request-Source": "cloudfoundry"}}'
```

#### Forwarded JSON details
```json
{
"headers": {
"Authorization": "Bearer abc123xyz",
"X-Splunk-Request-Source": "cloudfoundry"
}
}
```