Consecutive php-fpm calls return response for wrong request

[jjmontgo]

Hello there!

I've been testing Bref out, and so far I love it. But I've run into an issue that is a definite show-stopper. I'm hoping someone might have some guidance here because it's pretty unusual.

I have a web page that makes 3 XHR requests to the same endpoint with different URL parameters. On that endpoint, I print out the content of the $_REQUEST array. Here are the URL parameters of each request, followed by the output from the $_REQUEST array:

== REQUEST 1 ==

Params:
?id=3442&categoryId=setup

$_REQUEST:
Array
(
[id] => 3442
[categoryId] => setup
)

== REQUEST 2 ==

Params:
?id=3442&categoryId=instructions

$_REQUEST:
Array
(
[id] => 3442
[categoryId] => safety
)

== REQUEST 3 ==

Params:
?id=3442&categoryId=safety

$_REQUEST:
Array
(
[id] => 3442
[categoryId] => safety
)

As you can see, the second request's URL parameters and $_REQUEST array don't match up. This happens persistently.

When I print out $_SERVER['REQUEST_URI'], I see ?id=3442&categoryId=safety

But the browser request is /ajax/activity/loadBlocks/?id=3442&categoryId=instructions

I also printed out the content of $_SERVER['LAMBDA_INVOCATION_CONTEXT'], and the last two requests share exactly the same context information.

If I make a direct request to the endpoint in the browser (not through AJAX), I see the results I would expect.

It's very unusual. Can anyone shed some light on how this could happen and what could be done about it?

[mnapoli]

I would go as far as to say this is close to impossible that this is what's actually happening 😅

I mean, it is much more likely that you are sending the wrong requests in JS? How could request 2 have the data of request 3 when request 3 is not even sent?

[jjmontgo]

Indeed, that's exactly what I would think.

The production application is running in a typical environment with a load balancer and multiple EC2 instances. I'm using identical code, an identical database, and the only difference here is the bref/lambda execution environment. If it were something wrong with the JS alone, it would be happening with both the lambda and the EC2 environment.

Here are screenshots of the requests in the console, with the content of $_GET printed out from the PHP. The request query is highlighted with the response on the right.

This only seems to happen when the 3 requests are chronologically close together. I can put the URLs in the browser address bar and see correct output.

I've tried using both ${bref:layer.php-73-fpm} and ${bref:layer.php-74-fpm}, but the result is the same. I'm using the "serverless-lift" plugin, so the request/response is also going through CloudFront.

Is there anything I can output that would explain why this is happening?

[t-richard]

so the request/response is also going through CloudFront

This might be the problem, I know Cloudfront can behave weirdly if cache behaviours are configured incorrectly. Cloudfront should set a response header for cache saying Miss from Cloudfront, Hit from Cloudfront or something similar. Do you have a hit on the incorrect response?

[jjmontgo]

And, you're right.

The first request and the last request are cache misses, while the middle is a hit.

So this means it isn't a lambda problem, but a Cloudfront problem, and I'm super relieved about that.

I don't know why the middle request is cached in Cloudfront when the other two aren't, but now I can focus on that.

Thank you so much @t-richard and @mnapoli

[t-richard]

I think it's because those are "same second requests" so cloudfront tries to use cache on them even though the cache policy default to a TTL of zero. This almost look like a bug on cloudfront side.

You could try to include the query string in the cache behavior to see if it changes something?

You could also try to set cache headers explitly in your code to disable caching.

[mnapoli]

Ahh, good point. Here is the config for the official "Caching Disabled" policy (https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/using-managed-cache-policies.html):

Maybe what we need to do is explicitly add "minimum" and "maximum" TTL to 0 in Lift's config?

[jjmontgo]

Yes, that was the first thing I tried:

There was only one google search result which matched this error message, and it had to do with an origin request policy.

I had to remove the Authorization header from the cache key settings, and then I could save the policy.

I believe this made the custom policy equivalent to the "Caching Disabled" policy.

[mnapoli]

OK, we could actually maybe switch to the "Caching Disabled" official policy then. Someone else opened a similar issue here too: getlift/lift#144

[mnapoli]

Update: I dug into that again and could not reproduce it with a "normal" application.

Looking again at your original screenshot, I see your app seems to emit an expires header. I think this is what triggers caching by CloudFront.

So my conclusion is: this is expected, if your app emits a caching header, CloudFront will respect that.

I'll be updating getlift/lift#144 as well to consolidate all discussions.

cc @t-richard FYI

[mnapoli]

I'm sorry, I was wrong!

Maybe in your situation it was the expires header, but it does look like we have a problem. I've posted an update in getlift/lift#144 (comment) let's continue the discussion there.