Expiry and diff support

[pnorman]

I would open this at https://github.com/mapzen/TileStache, but it doesn't have tickets, and this is sort of a higher level question.

OSM data differs from other data in that it can be updated minutely. How is expiry of vector tiles handed in vector-datasource? The traditional raster stacks use a notion of a "dirty" tile that is rendered but not up to date. There are no public vector tile stacks with full diff support, but some providers are rendering the entire planet, then re-rendering any dirty tiles.

https://github.com/mapzen/vector-datasource/wiki/Mapzen-Vector-Tile-Service makes no reference to diffs, including reading them with osm2pgsql.

[rmarianski]

We also fall into the group of providers that render the full planet, and then re-generate dirty tiles.

We don't have much in the way of documenting how all the services talk to each other, but we do have a vagrant image that should demonstrate how all the pieces fit together, in a purely local setting. I added some details to the vector-datasource wiki page you linked to, and the vagrant image repository can be found here:

https://github.com/mapzen/vagrant-tiles

[bcamper]

We do use osm2pgsql's tile expiry list. We can add that info to the docs.

[pnorman]

We also fall into the group of providers that render the full planet, and then re-generate dirty tiles

I thought tilestache didn't support "dirty" tiles and had no concept of serving old data that it knew to be outdated?

[bcamper]

Right - we are just using TileStache as the web server to generate the tiles. They are then stored in S3 + Fastly (in our install, could use other services in place of those), and the actual expiry and re-render process is handled outside of TileStache. @rmarianski will have more info :)

[rmarianski]

Yup, that's correct. When osm2pgsql applies the update, we ask it to generate an expiry list, which we enqueue onto a work queue (sqs). That work is pulled off, and processed by a tool called tilequeue. That tool also handles enqueueing the work itself from the expiry list.