Improve the human readility of the osm2pgsql-replication
#1975
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lonvia
reviewed
Jun 19, 2023
scripts/osm2pgsql-replication
Outdated
| _base_url, _seq, current_local_timestamp = cur.fetchone() | ||
|
|
||
| total_applied_changes_duration_sec = (current_local_timestamp - initial_local_timestamp).total_seconds() | ||
| LOG.info("It took %s (%d sec) to apply %s (%d sec) of changes. this is a speed of ×%.1f.", |
There was a problem hiding this comment.
Maybe the first three lines can be rolled into one: Applying sequence number 5619810 to 5619815 (2023-06-15T09:42Z to 2023-06-15T09:47Z)
amandasaurus
force-pushed
the
repl-output-more-human
branch
from
bc5e2b6
to
4df7b0f
Compare
|
On Mon, 19 Jun 2023 17:42 +02:00, Sarah Hoffmann ***@***.***> wrote:
This is starting to become a bit too verbose for my taste but will let
me be convinced by good arguments to the contrary.
Would it better if they were a lower log level?
Or merged into 1 line?
e.g.:
```
Applying 5 sequence numbers, from 5619810 to 5619815 (2023-06-15T09:42Z to 2023-06-15T09:47Z , 5 minute(s) of changes)
```
*(BTW: typo fixed & rebased onto current `master` (master... 🤨))*
--
Amanda
|
If you want to have non-approximate times, I've made a PR #1977 to show that. Apply that or not, it's your call. |
amandasaurus
force-pushed
the
repl-output-more-human
branch
from
4df7b0f
to
bfb8f84
Compare
|
If we move it to a higher log level only accessible with '-v' then it would be fine. However, this makes pyosmium more verbose as well which may or may not be an issue. |
amandasaurus
force-pushed
the
repl-output-more-human
branch
from
bfb8f84
to
afdd1e4
Compare
• No OSM server does subminutely data updates, so don't include seconds in outputted timestamps. • Pretty format the interval, rather than just numbers. • Show the local & remote status of replication • Show how long it took to apply the changed & the rate
amandasaurus
force-pushed
the
repl-output-more-human
branch
from
afdd1e4
to
e998bb7
Compare
|
I've made the changes as requested (& merged all into one commit). Most of the new output is only at w/o w/ |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
A handful of minor improvements to the
osm2pgsql-replicationoutput in this PR.Here's what the (new) output looks like. changed lines in bold:
2023-06-15 11:48:39 [INFO]: Using replication service 'https://planet.openstreetmap.org/replication/minute/'.
2023-06-15 11:48:39 [INFO]: Local database is at sequence 5619810, which is data up until 5 minute(s) ago (354 seconds ago, 2023-06-15T09:42Z).
2023-06-15 11:48:39 [INFO]: Remote server is at sequence 5619815, which is data up until <1 minute ago (54 seconds ago, 2023-06-15T09:47Z).
2023-06-15 11:48:39 [INFO]: There are 5 sequence(s) on the remote server which will be applied locally. This represents 5 minute(s) (300 sec) of changes.
2023-06-15 11:48:40 osm2pgsql version 1.4.1
2023-06-15 11:48:40 Database version: 12.9 (Ubuntu 12.9-0ubuntu0.20.04.1)
2023-06-15 11:48:40 PostGIS version: 3.0
2023-06-15 11:48:40 Node-cache: cache=800MB, maxblocks=12800*65536, allocation method=11
2023-06-15 11:48:40 Setting up table 'planet_osm_point'
2023-06-15 11:48:40 Setting up table 'planet_osm_line'
2023-06-15 11:48:40 Setting up table 'planet_osm_polygon'
2023-06-15 11:48:40 Setting up table 'planet_osm_roads'
[324B blob data]
2023-06-15 11:48:57 Processed 14086 nodes in 3s - 5k/s
2023-06-15 11:48:57 Processed 1526 ways in 5s - 305/s
2023-06-15 11:48:57 Processed 78 relations in 9s - 9/s
2023-06-15 11:48:58 Going over 725 pending ways (using 4 threads)
[66B blob data]
2023-06-15 11:49:00 Processing 725 pending ways took 2s at a rate of 362.50/s
2023-06-15 11:49:00 Going over 146 pending relations (using 4 threads)
[66B blob data]
2023-06-15 11:49:02 Processing 146 pending relations took 2s at a rate of 73.00/s
2023-06-15 11:49:02 node cache: stored: 13297(100.00%), storage efficiency: 45.84% (dense blocks: 1, sparse nodes: 10408), hit rate: 1.61%
2023-06-15 11:49:02 osm2pgsql took 22s overall.
2023-06-15 11:49:03 [INFO]: Data imported until 2023-06-15T09:47Z. Backlog remaining: 1 minute(s)
2023-06-15 11:49:03 [INFO]: It took <1 minute (23 sec) to apply 5 minute(s) (300 sec) of changes. this is a speed of ×12.9.