Playmaker doesn't update etag; therefore Fdroid thinks no repo update available

[Yannik]

Hi,
I had the problem of the fdroid app keeping the same "last updated" date for my playmaker repository (and also not providing new app versions).
I debugged this and found the reason.

These are the headers returned by the playmaker repo when requesting index-v1.jar before updating the repo:

date: Thu, 01 Feb 2018 13:40:32 GMT
server: TornadoServer/4.5.3
strict-transport-security: max-age=31536000
content-type: application/java-archive
accept-ranges: bytes
etag: "4087f73ea76e66962006245fb4aa4ecb"
last-modified: Thu, 01 Feb 2018 12:27:21 GMT
content-length: 5489

Now I updated the repo and got these headers returned:

date: Thu, 01 Feb 2018 13:40:50 GMT
server: TornadoServer/4.5.3
strict-transport-security: max-age=31536000
content-type: application/java-archive
accept-ranges: bytes
etag: "4087f73ea76e66962006245fb4aa4ecb"
last-modified: Thu, 01 Feb 2018 13:40:46 GMT
content-length: 5494

You can clearly see that the last-modified header is updated correctly, but the etag stays the same.

This is a huge issue, as fdroid uses the etag to determine whether a repository needs to be updated or not: https://gitlab.com/fdroid/fdroidclient/blob/master/app/src/main/java/org/fdroid/fdroid/net/HttpDownloader.java#L95

[Yannik]

Looking at http://www.tornadoweb.org/en/stable/_modules/tornado/web.html#RequestHandler.compute_etag, we can clearly see why it doesn't return anything useful.

Apache (see https://httpd.apache.org/docs/2.4/de/mod/core.html#fileetag) uses

INode
    The file's i-node number will be included in the calculation
MTime
    The date and time the file was last modified will be include
Size
    The number of bytes in the file will be included

to calculate the etag. I just tried using apache to serve the repo, it worked without issues. To fix this in playmaker, we should definitely use the modified datetime and size to calculate the etag. (I wouldn't use the inode to ensure persistent etags between docker deploys)

[NoMore201]

Thanks for reporting, I never noticed this problem. Need to check if there's a way in RequestHandler to get the filepath which is currently being served through tornado, so that we could use [1] and [2]

[1] https://docs.python.org/3/library/os.path.html#os.path.getmtime
[2] https://docs.python.org/3/library/os.path.html#os.path.getsize

[Yannik]

@NoMore201 Look at this: https://gist.github.com/andreadipersio/7526464#file-tornado-disable-auto-etag-py-L12
Edit: Sorry, I misread your comment. Yes, there needs to be a way to get the filepath to correctly implement etag generation.

[j5k]

I can report this problem too. Thanks for fixing.

[Yannik]

@NoMore201 Any updates on this? :)

[apexo]

A manual workaround in f-droid: disabling and re-enabling the repository in f-droid seems to force a refresh. This kind of defeats the purpose of automatic updates, but at least it is still possible to update apps.

[apexo]

Also, when using a reverse proxy, it can be configured to hide the ETag, e.g. for nginx:

proxy_hide_header ETag;

[Yannik]

@apexo I don't think that's useful, as fdroid will re-download the repo list every time, as it thinks it has changed.

[apexo]

It's not a perfect solution, but at least you'll get updates.

[Yannik]

Hey @NoMore201, have you abandoned this project? :-(

[Findus23]

@Yannik, @apexo

I just had an idea for an workaround (if you are using nginx in front of playmaker on the docker host)

As the fdroid repo is just tornado serving the fdroid/repo/ directory, you can completly circumvent playmaker and make nginx directly serve that directory (and therefore correctly report etags)

location /fdroid {
        alias /srv/fdroid/repo/;
}