Dirty tiles and cache invalidation

[pnorman]

When updating databases with OpenStreetMap data, you generally get a tile expiry list, a list which contains tiles that have changed (e.g. osm2pgsql --expire-tiles). When pre-rendering the world these tiles need to be re-rendered async to serving. When not pre-rendering, it is more complicated and most options involve "dirty tiles" or similar.

What options are there with t-rex for dealing with changed data?

[pka]

That's an important missing feature. My plan was to add an –older [timestamp|now] option to the seed command.
In the meantime you can only delete tiles with commands like find /path/to/cache -mtime +7 -delete
How does an expiry list look like?

[pnorman]

That's an important missing feature. My plan was to add an –older [timestamp|now] option to the seed command.
In the meantime you can only delete tiles with commands like find /path/to/cache -mtime +7 -delete

This would be based on tiles, not time. rm would possibly work, of course, race conditions might be a problem. And you'd then need to re-render

How does an expiry list look like?

The standard format is X Y Z separated by spaces, with each tile separated by newlines.

For what needs to be implemented, I'd look to render_list from renderd

$ render_list --help
Usage: render_list [OPTION] ...
  -a, --all            render all tiles in given zoom level range instead of reading from STDIN
  -f, --force          render tiles even if they seem current
  -m, --map=MAP        render tiles in this map (defaults to 'default')
  -l, --max-load=LOAD  sleep if load is this high (defaults to 16)
  -s, --socket=SOCKET  unix domain socket name for contacting renderd
  -n, --num-threads=N the number of parallel request threads (default 1)
  -t, --tile-dir       tile cache directory (defaults to '/var/lib/mod_tile')
  -z, --min-zoom=ZOOM  filter input to only render tiles greater or equal to this zoom level (default is 0)
  -Z, --max-zoom=ZOOM  filter input to only render tiles less than or equal to this zoom level (default is 20)
If you are using --all, you can restrict the tile range by adding these options:
  -x, --min-x=X        minimum X tile coordinate
  -X, --max-x=X        maximum X tile coordinate
  -y, --min-y=Y        minimum Y tile coordinate
  -Y, --max-y=Y        maximum Y tile coordinate
Without --all, send a list of tiles to be rendered from STDIN in the format:
  X Y Z
e.g.
  0 0 1
  0 1 1
  1 0 1
  1 1 1
The above would cause all 4 tiles at zoom 1 to be rendered

The difference with renderd is that it has "dirty" tiles, so it can have an outdated tile still in its store.

For an overview of the different methods, it's worth reading osm2pgsql-dev/osm2pgsql#709 and osm2pgsql-dev/osm2pgsql#747

I suppose the first questions are

1. Will t-rex support continually updating data when pre-rendering all tiles
2. Will t-rex support continually updating data when not pre-rendering all tiles

To me the first is more important and easier to understand + implement.

[pka]

Would the options --tile-list <file> and --execept-tile-list <file> for t_rex generate cover both cases? I assume the list can get pretty big, so maybe gzip compression had to be supported.

[pnorman]

Would the options --tile-list and --execept-tile-list for t_rex generate cover both cases?

You'd have t_rex serve running in the background, then do t_rex generate with those options?

I don't see a need for --execept-tile-list, nothing else uses it.

I assume the list can get pretty big, so maybe gzip compression had to be supported.

It can be relatively big, but I don't know of anyone gzipping it. The list never needs to be transmitted over the network.