Cannot generate expire tiles list when using undocumented option IMPOSM3_SINGLE_DIFF

[plblueraven]

Context

I'm going to create CI process which runs every day and updates OSM data in postgis used to generate tiles and generates expired tiles list.

Expected Behavior

Using run command to update database should generate (at least) similar results with same run but with IMPOSM3_SINGLE_DIFF set to true executed X times.

Actual Behavior

When I use import & update (simple run command) then I got list of expired tiles, as expected.

On the other hand when I do the same except setting IMPOSM3_SINGLE_DIFF to true before update and run that kind of update 15 times then expired tiles list isn't even created.

Possible Fix

I'm not go developer, but maybe it is connected with order of caches closing vs tile list flushing?

Steps to Reproduce

import & update:

1. imposm import -mapping /imposm/mapping.yml -read /osm_tmp/lodzkie-230501.osm.pbf -write -connection postgis://user:pass@osm-tools-db:5432/test -overwritecache -diff -diffdir /tmp/diff -cachedir /tmp/cache -config /imposm/replication-config.json -deployproduction
2. imposm run -mapping /imposm/mapping.yml -connection postgis://user:pass@osm-tools-db:5432/test -diffdir /tmp/diff -config /imposm/replication-config.json -expiretiles-dir /imposm/expire-tiles
3. ls -la /imposm/expire-tiles -> contains folder named as current date (e.g. 20230516)

---

import & update + IMPOSM3_SINGLE_DIFF:

1. imposm import -mapping /imposm/mapping.yml -read /osm_tmp/lodzkie-230501.osm.pbf -write -connection postgis://user:pass@osm-tools-db:5432/test -overwritecache -diff -diffdir /tmp/diff -cachedir /tmp/cache -config /imposm/replication-config.json -deployproduction
2. export IMPOSM3_SINGLE_DIFF=true
3. imposm run -mapping /imposm/mapping.yml -connection postgis://user:pass@osm-tools-db:5432/test -diffdir /tmp/diff -config /imposm/replication-config.json -expiretiles-dir /imposm/expire-tiles
4. imposm run -mapping /imposm/mapping.yml -connection postgis://user:pass@osm-tools-db:5432/test -diffdir /tmp/diff -config /imposm/replication-config.json -expiretiles-dir /imposm/expire-tiles
5. imposm run -mapping /imposm/mapping.yml -connection postgis://user:pass@osm-tools-db:5432/test -diffdir /tmp/diff -config /imposm/replication-config.json -expiretiles-dir /imposm/expire-tiles
6. imposm run -mapping /imposm/mapping.yml -connection postgis://user:pass@osm-tools-db:5432/test -diffdir /tmp/diff -config /imposm/replication-config.json -expiretiles-dir /imposm/expire-tiles
7. imposm run -mapping /imposm/mapping.yml -connection postgis://user:pass@osm-tools-db:5432/test -diffdir /tmp/diff -config /imposm/replication-config.json -expiretiles-dir /imposm/expire-tiles
8. imposm run -mapping /imposm/mapping.yml -connection postgis://user:pass@osm-tools-db:5432/test -diffdir /tmp/diff -config /imposm/replication-config.json -expiretiles-dir /imposm/expire-tiles
9. imposm run -mapping /imposm/mapping.yml -connection postgis://user:pass@osm-tools-db:5432/test -diffdir /tmp/diff -config /imposm/replication-config.json -expiretiles-dir /imposm/expire-tiles
10. imposm run -mapping /imposm/mapping.yml -connection postgis://user:pass@osm-tools-db:5432/test -diffdir /tmp/diff -config /imposm/replication-config.json -expiretiles-dir /imposm/expire-tiles
11. imposm run -mapping /imposm/mapping.yml -connection postgis://user:pass@osm-tools-db:5432/test -diffdir /tmp/diff -config /imposm/replication-config.json -expiretiles-dir /imposm/expire-tiles
12. imposm run -mapping /imposm/mapping.yml -connection postgis://user:pass@osm-tools-db:5432/test -diffdir /tmp/diff -config /imposm/replication-config.json -expiretiles-dir /imposm/expire-tiles
13. imposm run -mapping /imposm/mapping.yml -connection postgis://user:pass@osm-tools-db:5432/test -diffdir /tmp/diff -config /imposm/replication-config.json -expiretiles-dir /imposm/expire-tiles
14. imposm run -mapping /imposm/mapping.yml -connection postgis://user:pass@osm-tools-db:5432/test -diffdir /tmp/diff -config /imposm/replication-config.json -expiretiles-dir /imposm/expire-tiles
15. imposm run -mapping /imposm/mapping.yml -connection postgis://user:pass@osm-tools-db:5432/test -diffdir /tmp/diff -config /imposm/replication-config.json -expiretiles-dir /imposm/expire-tiles
16. imposm run -mapping /imposm/mapping.yml -connection postgis://user:pass@osm-tools-db:5432/test -diffdir /tmp/diff -config /imposm/replication-config.json -expiretiles-dir /imposm/expire-tiles
17. imposm run -mapping /imposm/mapping.yml -connection postgis://user:pass@osm-tools-db:5432/test -diffdir /tmp/diff -config /imposm/replication-config.json -expiretiles-dir /imposm/expire-tiles
18. ls -la /imposm/expire-tiles -> contains nothing

---

Initial import from: https://download.geofabrik.de/europe/poland/lodzkie-230501.osm.pbf

/imposm/replication-config.json is:

{
  "replication_interval": "24h",
  "replication_url": "https://download.geofabrik.de/europe/poland/lodzkie-updates/"
}

and /imposm/mapping.yml is:

areas:
  area_tags: [buildings, landuse, leisure, natural, aeroway]
  linear_tags: [highway, barrier]
generalized_tables:
  landusages_gen0:
    source: landusages_gen1
    sql_filter: ST_Area(geometry)>16000000
    tolerance: 2000.0
  landusages_gen1:
    source: landusages
    sql_filter: ST_Area(geometry)>4000000
    tolerance: 1000.0
  landusages_gen2:
    source: landusages
    sql_filter: ST_Area(geometry)>1000000
    tolerance: 500.0
  landusages_gen3:
    source: landusages
    sql_filter: ST_Area(geometry)>60000
    tolerance: 125.0
  waterareas_gen0:
    source: waterareas_gen1
    sql_filter: ST_Area(geometry)>500000
    tolerance: 200.0
  waterareas_gen1:
    source: waterareas
    sql_filter: ST_Area(geometry)>50000
    tolerance: 50.0
  waterways_gen0:
    source: waterways_gen1
    sql_filter: type = 'river'
    tolerance: 120
  waterways_gen1:
    source: waterways
    sql_filter: type IN ('river', 'pier', 'breakwater', 'groyne')
    tolerance: 60.0
  roads_gen0:
    source: roads_gen1
    sql_filter: class = 'highway' and type in ('motorway', 'trunk')
    tolerance: 900.0
  roads_gen1:
    source: roads_gen2
    sql_filter: (class = 'highway' and type IN ('motorway', 'trunk', 'primary')) OR (class = 'railway' and type IN ('funicular','light_rail','narrow_gauge'))
    tolerance: 450.0
  roads_gen2:
    source: roads_gen3
    sql_filter: (class = 'highway' and type IN ('motorway', 'motorway_link', 'trunk', 'trunk_link', 'primary', 'primary_link', 'secondary', 'secondary_link')) OR (class = 'railway' and type IN ('funicular','light_rail','narrow_gauge'))
    tolerance: 300.0
  roads_gen3:
    source: roads
    sql_filter: type IN ('motorway', 'motorway_link', 'trunk', 'trunk_link', 'primary', 'primary_link', 'secondary', 'secondary_link', 'tertiary', 'tertiary_link') OR class = 'railway'
    tolerance: 30.0
tables:
  admin:
    fields:
      - name: osm_id
        type: id
      - name: geometry
        type: geometry
      - key: name
        name: name
        type: string
      - key: name:en
        name: name_en
        type: string
      - key: int_name
        name: int_name
        type: string
      - name: type
        type: mapping_value
      - key: admin_level
        name: admin_level
        type: integer
      - name: area
        type: webmerc_area
    mapping:
      boundary:
        - administrative
    type: polygon
  boundary:
    fields:
      - name: osm_id
        type: id
      - name: geometry
        type: geometry
      - name: type
        type: mapping_value
      - key: admin_level
        name: admin_level
        type: integer
    mapping:
      boundary:
        - administrative
    type: linestring
  amenities:
    fields:
      - name: osm_id
        type: id
      - name: geometry
        type: geometry
      - key: name
        name: name
        type: string
      - key: name:en
        name: name_en
        type: string
      - key: int_name
        name: int_name
        type: string
      - key: access
        name: access
        type: string
      - key: religion
        name: religion
        type: string
      - key: denomination
        name: denomination
        type: string
      - key: power_source
        name: power_source
        type: string
      - key: ele
        name: score
        type: integer
      - name: type
        type: mapping_value
    type: geometry
    type_mappings:
      points:
        aeroway:
          - helipad
          - aerodrome
        tourism:
          - artwork
          - alpine_hut
          - camp_site
          - caravan_site
          - chalet
          - wilderness_hut
          - guest_house
          - hostel
          - hotel
          - motel
          - information
          - museum
          - viewpoint
          - picnic_site
        amenity:
          - shelter
          - atm
          - bank
          - bar
          - bicycle_rental
          - bus_station
          - cafe
          - car_rental
          - car_wash
          - cinema
          - clinic
          - community_centre
          - fire_station
          - fountain
          - fuel
          - hospital
          - ice_cream
          - embassy
          - library
          - courthouse
          - townhall
          - parking
          - bicycle_parking
          - motorcycle_parking
          - pharmacy
          - doctors
          - dentist
          - place_of_worship
          - police
          - post_box
          - post_office
          - pub
          - biergarten
          - recycling
          - restaurant
          - food_court
          - fast_food
          - telephone
          - emergency_phone
          - taxi
          - theatre
          - toilets
          - drinking_water
          - prison
          - hunting_stand
          - nightclub
          - veterinary
          - social_facility
          - charging_station
        shop:
          - __any__
        leisure:
          - water_park
          - playground
          - miniature_golf
          - golf_course
          - picnic_table
          - slipway
          - dog_park
          - sauna
        man_made:
          - mast
          - water_tower
          - lighthouse
          - windmill
          - cross
          - obelisk
        natural:
          - peak
          - volcano
          - saddle
          - spring
          - cave_entrance
        historic:
          - memorial
          - monument
          - archaeological_site
          - wayside_cross
        highway:
          - bus_stop
          - elevator
          - traffic_signals
          - ford
        power:
          - generator
      polygons:
        aeroway:
          - helipad
          - aerodrome
        tourism:
          - artwork
          - alpine_hut
          - camp_site
          - caravan_site
          - chalet
          - wilderness_hut
          - guest_house
          - hostel
          - hotel
          - motel
          - information
          - museum
          - viewpoint
          - picnic_site
        amenity:
          - shelter
          - atm
          - bank
          - bar
          - bicycle_rental
          - bus_station
          - cafe
          - car_rental
          - car_wash
          - cinema
          - clinic
          - community_centre
          - fire_station
          - fountain
          - fuel
          - hospital
          - ice_cream
          - embassy
          - library
          - courthouse
          - townhall
          - parking
          - bicycle_parking
          - motorcycle_parking
          - pharmacy
          - doctors
          - dentist
          - place_of_worship
          - police
          - post_box
          - post_office
          - pub
          - biergarten
          - recycling
          - restaurant
          - food_court
          - fast_food
          - telephone
          - emergency_phone
          - taxi
          - theatre
          - toilets
          - drinking_water
          - prison
          - hunting_stand
          - nightclub
          - veterinary
          - social_facility
          - charging_station
        shop:
          - __any__
        leisure:
          - water_park
          - playground
          - miniature_golf
          - golf_course
          - picnic_table
          - slipway
          - dog_park
          - sauna
        man_made:
          - mast
          - water_tower
          - lighthouse
          - windmill
          - cross
          - obelisk
        natural:
          - peak
          - volcano
          - saddle
          - spring
          - cave_entrance
        historic:
          - memorial
          - monument
          - archaeological_site
          - wayside_cross
        highway:
          - bus_stop
          - elevator
          - traffic_signals
          - ford
        power:
          - generator
  barrierpoints:
    fields:
      - name: osm_id
        type: id
      - name: geometry
        type: geometry
      - key: name
        name: name
        type: string
      - key: name:en
        name: name_en
        type: string
      - key: int_name
        name: int_name
        type: string
      - name: type
        type: mapping_value
    mapping:
      barrier:
        - block
        - bollard
        - cattle_grid
        - chain
        - cycle_barrier
        - entrance
        - horse_stile
        - gate
        - spikes
        - lift_gate
        - kissing_gate
        - fence
        - 'yes'
        - wire_fence
        - toll_booth
        - stile
    type: point
  barrierways:
    fields:
      - name: osm_id
        type: id
      - name: geometry
        type: geometry
      - key: name
        name: name
        type: string
      - key: name:en
        name: name_en
        type: string
      - key: int_name
        name: int_name
        type: string
      - name: type
        type: mapping_value
    mapping:
      barrier:
        - city_wall
        - fence
        - hedge
        - retaining_wall
        - wall
        - bollard
        - gate
        - spikes
        - lift_gate
        - kissing_gate
        - embankment
        - 'yes'
        - wire_fence
    type: linestring
  buildings:
    fields:
      - name: osm_id
        type: id
      - name: geometry
        type: geometry
      - key: name
        name: name
        type: string
      - key: name:en
        name: name_en
        type: string
      - key: int_name
        name: int_name
        type: string
      - name: type
        type: mapping_value
      - key: amenity
        name: amenity
        type: string
      - key: aeroway
        name: aeroway
        type: string
      - key: building
        name: building
        type: string
    mapping:
      building:
        - __any__
    type: polygon
  housenumbers:
    fields:
      - name: osm_id
        type: id
      - name: geometry
        type: geometry
      - key: name
        name: name
        type: string
      - name: type
        type: mapping_value
      - key: addr:street
        name: addr_street
        type: string
      - key: addr:postcode
        name: addr_postcode
        type: string
      - key: addr:city
        name: addr_city
        type: string
      - key: addr:unit
        name: addr_unit
        type: string
      - key: addr:housename
        name: addr_housename
        type: string
    type: geometry
    type_mappings:
      points:
        addr:housenumber:
          - __any__
      polygons:
        addr:housenumber:
          - __any__
  housenumbers_interpolated:
    fields:
      - name: osm_id
        type: id
      - name: geometry
        type: geometry
      - key: name
        name: name
        type: string
      - name: type
        type: mapping_value
      - key: addr:street
        name: addr:street
        type: string
      - key: addr:postcode
        name: addr:postcode
        type: string
      - key: addr:city
        name: addr:city
        type: string
      - key: addr:inclusion
        name: addr:inclusion
        type: string
    mapping:
      addr:interpolation:
        - __any__
    type: linestring
  landusages:
    fields:
      - name: osm_id
        type: id
      - name: geometry
        type: geometry
      - key: name
        name: name
        type: string
      - key: name:en
        name: name_en
        type: string
      - key: int_name
        name: int_name
        type: string
      - name: type
        type: mapping_value
      - key: religion
        name: religion
        type: string
      - name: area
        type: pseudoarea
        key: null
      - key: layer
        name: layer
        type: integer
      - args:
          values:
            - land
            - island
            - heath
            - railway
            - industrial
            - commercial
            - retail
            - residential
            - quarry
            - zoo
            - vineyard
            - orchard
            - scrub
            - hospital
            - place_of_worship
            - theatre
            - cinema
            - nature_reserve
            - parking
            - fuel
            - baracks
            - library
            - college
            - school
            - university
            - golf_course
            - allotments
            - common
            - pitch
            - sports_centre
            - garden
            - recreation_ground
            - village_green
            - wetland
            - grass
            - meadow
            - wood
            - farmland
            - farm
            - farmyard
            - cemetery
            - forest
            - park
            - dog_park
            - playground
            - footway
            - pedestrian
        name: z_order
        type: enumerate
    mapping:
      aeroway:
        - runway
        - taxiway
      amenity:
        - university
        - school
        - college
        - library
        - fuel
        - parking
        - bicycle_parking
        - motorcycle_parking
        - cinema
        - theatre
        - place_of_worship
        - hospital
        - kindergarten
        - grave_yard
        - prison
        - clinic
      barrier:
        - hedge
      highway:
        - services
        - rest_area
      landuse:
        - park
        - forest
        - residential
        - retail
        - commercial
        - industrial
        - railway
        - cemetery
        - grass
        - farmyard
        - farm
        - farmland
        - orchard
        - vineyard
        - wood
        - meadow
        - village_green
        - recreation_ground
        - allotments
        - quarry
        - military
        - cemetery
        - garages
        - greenhouse_horticulture
        - brownfield
        - landfill
        - construction
      leisure:
        - park
        - garden
        - playground
        - golf_course
        - sports_centre
        - pitch
        - stadium
        - common
        - nature_reserve
        - recreation_ground
        - miniature_golf
        - picnic_table
        - track
        - dog_park
        - swimming_pool
        - water_park
        - marina
      man_made:
        - pier
      military:
        - danger_area
      natural:
        - wood
        - land
        - scrub
        - wetland
        - heath
        - sand
        - scree
        - shingle
        - bare_rock
        - beach
        - shoal
        - grassland
        - wood
        - sand
        - bare_orock
        - mud
        - marsh
      place:
        - island
      power:
        - station
        - sub_station
        - substation
        - generator
      tourism:
        - zoo
        - attraction
        - camp_site
        - caravan_site
        - picnic_site
      railway:
        - station
    type: polygon
  places:
    fields:
      - name: osm_id
        type: id
      - name: geometry
        type: geometry
      - key: name
        name: name
        type: string
      - key: name:en
        name: name_en
        type: string
      - key: int_name
        name: int_name
        type: string
      - name: type
        type: mapping_value
      - key : capital
        name: capital
        type: string
      - args:
          values:
            - locality
            - suburb
            - hamlet
            - village
            - town
            - city
            - county
            - region
            - state
            - country
        name: z_order
        type: enumerate
      - key: population
        name: population
        type: integer
    mapping:
      place:
        - country
        - state
        - region
        - county
        - city
        - town
        - village
        - hamlet
        - suburb
        - locality
    type: point
  roads:
    fields:
      - name: osm_id
        type: id
      - name: geometry
        type: geometry
      - name: class
        type: mapping_key
      - name: type
        type: mapping_value
      - key: highway
        name: highway
        type: string
      - key: name
        name: name
        type: string
      - key: name:en
        name: name_en
        type: string
      - key: int_name
        name: int_name
        type: string
      - key: tunnel
        name: tunnel
        type: boolint
      - key: bridge
        name: bridge
        type: boolint
      - key: oneway
        name: oneway
        type: direction
      - key: ref
        name: ref
        type: string
      - key: layer
        name: layer
        type: integer
      - key: access
        name: access
        type: string
      - key: service
        name: service
        type: string
      - key: horse
        name: horse
        type: string
      - key: bicycle
        name: bicycle
        type: string
      - key: construction
        name: construction
        type: string
      - key: surface
        name: surface
        type: string
      - key: tracktype
        name: tracktype
        type: string
      - key: layer
        name: z_order
        type: wayzorder
      - key: highspeed
        name: highspeed
        type: string
      - key: usage
        name: usage
        type: string
    filters:
      exclude_tags:
        - - area
          - 'yes'
    mappings:
      aeroway:
        mapping:
          aeroway:
            - runway
            - taxiway
      railway:
        mapping:
          railway:
            - rail
            - tram
            - tram-service

      railway:
        mapping:
          railway:
            - rail
            - tram
            - tram-service
            - light_rail
            - subway
            - narrow_gauge
            - preserved
            - funicular
            - monorail
            - disused
            - miniature
            - construction
            - platform
            - turntable
      roads:
        mapping:
          highway:
            - motorway
            - motorway_link
            - trunk
            - trunk_link
            - primary
            - primary_link
            - secondary
            - secondary_link
            - tertiary
            - tertiary_link
            - road
            - path
            - track
            - service
            - footway
            - bridleway
            - cycleway
            - steps
            - pedestrian
            - living_street
            - unclassified
            - residential
            - raceway
            - platform
            - construction
          man_made:
            - pier
            - groyne
    type: linestring
  transport_areas:
    fields:
      - name: osm_id
        type: id
      - name: geometry
        type: geometry
      - key: name
        name: name
        type: string
      - key: name:en
        name: name_en
        type: string
      - key: int_name
        name: int_name
        type: string
      - name: type
        type: mapping_value
    mapping:
      aeroway:
        - aerodrome
        - terminal
        - helipad
        - apron
      railway:
        - station
        - platform
    type: polygon
  transport_points:
    fields:
      - name: osm_id
        type: id
      - name: geometry
        type: geometry
      - key: name
        name: name
        type: string
      - key: name:en
        name: name_en
        type: string
      - key: int_name
        name: int_name
        type: string
      - name: type
        type: mapping_value
      - key: ref
        name: ref
        type: string
    mapping:
      aeroway:
        - aerodrome
        - terminal
        - helipad
        - gate
      highway:
        - motorway_junction
        - turning_circle
        - bus_stop
      railway:
        - station
        - halt
        - tram_stop
        - crossing
        - level_crossing
        - subway_entrance
    type: point
  waterareas:
    fields:
      - name: osm_id
        type: id
      - name: geometry
        type: geometry
      - key: name
        name: name
        type: string
      - key: name:en
        name: name_en
        type: string
      - key: int_name
        name: int_name
        type: string
      - name: type
        type: mapping_value
      - name: area
        type: area
    mapping:
      amenity:
        - swimming_pool
      landuse:
        - basin
        - reservoir
      leisure:
        - swimming_pool
      natural:
        - water
        - glacier
      waterway:
        - riverbank
        - dock
        - canal
    type: polygon
  waterways:
    fields:
      - name: osm_id
        type: id
      - name: geometry
        type: geometry
      - key: name
        name: name
        type: string
      - key: name:en
        name: name_en
        type: string
      - key: int_name
        name: int_name
        type: string
      - name: type
        type: mapping_value
      - key: tunnel
        name: tunnel
        type: string
      - key: intermittent
        name: intermittent
        type: bool
      - key: bridge
        name: bridge
        type: string
      - key: lock
        name: lock
        type: string
    mapping:
      barrier:
        - ditch
      waterway:
        - stream
        - river
        - canal
        - drain
        - ditch
        - dam
        - weir
        - lock_gate
        - derelict_canal
        - wadi
      man_made:
        - pier
        - breakwater
        - groyne
    type: linestring

Your Environment

• Version used: 0.0.0-dev, instaled today (33e15acf7b3e0df1c4af25c5eda312842ef0cbb4) by go install github.com/omniscale/imposm3/cmd/imposm@latest
• Environment name and version (e.g. PostgreSQL x.y.z with PostGIS a.b.c): postgres (PostgreSQL) 14.5 (Debian 14.5-1.pgdg110+1) with PostGIS "3.2 USE_GEOS=1 USE_PROJ=1 USE_STATS=1"