Transform user set headers to lowercase

[alextes]

There's one thing that should happen which is to lowercase all the headers passed so no duplicate lowercase keys can exist. Because 1. we have logic that reads and writes headers, but 2. we have no clue what node will do when we pass it duplicate header keys.

The reason this is hard to test is that we use cacheable-request and by the time the request event fires the headers have already been merged, with keys set later in time overwriting ones set earlier in time. So I test by setting some header in uppercase I know with got's current implementation is followed by a key set in lowercase before everything is passed to cacheable-request. That's brittle. If someone has a better idea let me know.

Fixes #612 .

[brandon93s]

I advise strongly against this approach. Instead, we should update code in got that directly modifies (adds, removes, edits) headers to handle case sensitivity (e.g. find existing header in a case insensitive manner).

Headers in HTTP/1.1, per RFC 7230, are case insensitive. The problem is, any server that is checking for a specific header (case-sensitive) value, could now fail because we are forcing the user's headers to be all lowercase. I have experienced this in a production application and had resulting outages.

HTTP/2 makes this more complicated: headers are required to be all lowercase. If got is to ever adopt native HTTP/2 support, this would need to be accounted for at that time.

Example:

// Client
await got('/api', {
  headers: {
      'X-My-Custom-Header': 'unicorn'
  }
})


// API
if (request.headers['X-My-Custom-Header']) {
   // this fails now if headers are forced to lowercase
}

This very scenario is one of several issues with legacy applications migrating to HTTP/2 support in the wild.

[brandon93s]

Related: https://github.com/request/caseless

[sindresorhus]

@brandon93s I strongly disagree. We should not add complicated logic just to work around faulty servers. HTTP1 specifies headers as case-insensitive and HTTP2 enforces lowercase headers, so the most future-proof way forward is to lowercase headers. Got has always lowercased headers and it has not been a problem. It was only accidentally left out recently.

[brandon93s]

Fair enough. If this was the previous behavior, it shouldn't be a surprise to any current users.

[gajus]

@brandon93s I strongly disagree. We should not add complicated logic just to work around faulty servers. HTTP1 specifies headers as case-insensitive and HTTP2 enforces lowercase headers, so the most future-proof way forward is to lowercase headers. Got has always lowercased headers and it has not been a problem. It was only accidentally left out recently.

This is a bummer.

I have been using got to make HTTP requests in a proxy service.

This proxy service interacts with tens of thousands different websites.

After recent Got upgrade, I got reports of many requests failing.

After some digging, it was traced back to Got lower-casing the HTTP request headers.

I am digging the argument let's keep it simple. However, this literally makes the library unusable for those of us who need to interact with Internet at large, not just select websites that respect the standard, and as our experience shows, there are still websites (some with copyright dates of pre-2000s era) that require headers in a certain way.

I have published a library that abstracts working with headers in case-insensitive way.

https://github.com/gajus/http-header

Happy to raise a PR.

[szmarczak]

@gajus PR won't be necessary. Got just follows the standard. I agree with @sindresorhus - we shouldn't add some bloat because of the faulty servers. That's not our fault - it's done right here.

[issuefiler]

But getting the body is not all there is to requests. We also do tests with them.

If everyone starts (or more precisely, “is forced”) to request only with lowercase headers, how would you test an HTTP<2 server if it well accepts uppercase headers as well, as per the specification you guys are talking about? How would you test a web server with uppercase, lowercase, mixed, and fuzzed headers?

[szmarczak]

How would you test a web server with uppercase, lowercase, mixed, and fuzzed headers?

Raw TCP sockets and precomputed payloads and expected outputs.

Got is an HTTP client, not a server testing tool.