Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tempo 2.0: Config Cleanup #1978

Merged
merged 19 commits into from
Jan 11, 2023
Merged

Conversation

joe-elliott
Copy link
Member

@joe-elliott joe-elliott commented Jan 6, 2023

What this PR does:
This is a config cleanup pass to prepare for 2.0. The following changes were made to config:

Defaults Updated
Settings listed with new defaults. These defaults were changed to work better with the new Parquet backend.

query_frontend:
    max_oustanding_per_tenant: 2000
    search:
        concurrent_jobs: 1000
        target_bytes_per_job: 104857600
        max_duration: 168h
        query_ingesters_until: 30m
    trace_by_id:
        query_shards: 50
querier:
    max_concurrent_queries: 20
    search:
        prefer_self: 10
ingester:
    concurrent_flushes: 4
    max_block_duration: 30m
    max_block_bytes: 524288000
storage:
    trace:
        pool:
            max_workers: 400
            queue_depth: 20000
        search:
            read_buffer_count: 32
            read_buffer_size_bytes: 1048576

Renamed/Moved/Removed Config

query_frontend:
    query_shards: // removed. use trace_by_id.query_shards
querier:
    query_timeout: // removed. use trace_by_id.query_timeout
compactor:
    compaction:
        chunk_size_bytes:   // renamed to v2_in_buffer_bytes
        flush_size_bytes:     // renamed to v2_out_buffer_bytes
        iterator_buffer_size: // renamed to v2_prefetch_traces_count
ingester:
    use_flatbuffer_search:   // removed. automatically set based on block type
storage:
    wal:
        encoding:  // renamed to v2_encoding
        version:     // removed and pinned to block.version
    block:
        index_downsample_bytes:  // renamed to v2_index_downsample_bytes
        index_page_size_bytes:      // renamed to v2_index_page_size_bytes
        encoding:                  	    // renamed to v2_encoding
        row_group_size_bytes:       // renamed to parquet_row_group_size_bytes

As part of the removing useFlatBufferSearch and requiring the wal and block Version to be the same the tests in ./modules/ingester had to be heavily reworked. This is b/c they were written for flatbuffer/v2 wal and they have now been updated to use the current block type for the wal (vparquet). During this process the following bugs were found and corrected:

  • SearchTags/TagValues was not working on the parquet headblock/completing blocks
  • When replaying a parquet wal block we were double counting total objects

Finally, I would still like to remove search_enabled and metrics_generator_enabled but I intend to do that in a future PR as this has gotten quite large.

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: Joe Elliott <number101010@gmail.com>
tempodb/encoding/common/config.go Show resolved Hide resolved
Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: Joe Elliott <number101010@gmail.com>
tempodb/wal/wal.go Outdated Show resolved Hide resolved
Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott joe-elliott merged commit 1c2e522 into grafana:main Jan 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants