Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(executor): implement time slicing for shared executor scheduling across queries #14770

Merged
merged 7 commits into from
Feb 28, 2024
Merged

feat(executor): implement time slicing for shared executor scheduling across queries #14770

merged 7 commits into from
Feb 28, 2024

Conversation

dqhl76
Copy link
Collaborator

@dqhl76 dqhl76 commented Feb 28, 2024 •
edited
Loading

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

This PR introduces time slicing to enable different queries to share a single executor for scheduling.

Split two executors from #14594 and provide a config to switch between them.

This PR should not affect current exeuctor. Previous PipelineExecutor became QueryPipelineExecutor. New added one called QueriesPipelineExecutor. Introduce a new enum PipelineExecutor to switch them according to config.

Tests

  • Unit Test
  • Logic Test
  • Benchmark Test
  • No Test - Explain why

Type of change

  • Bug Fix (non-breaking change which fixes an issue)
  • New Feature (non-breaking change which adds functionality)
  • Breaking Change (fix or feature that could cause existing functionality not to work as expected)
  • Documentation Update
  • Refactoring
  • Performance Improvement
  • Other (please describe):

This change is Reviewable

@github-actions github-actions bot added the pr-feature this PR introduces a new feature to the codebase label Feb 28, 2024
@dqhl76 dqhl76 marked this pull request as ready for review February 28, 2024 03:18
@zhang2014 zhang2014 added the ci-benchmark Benchmark: run all test label Feb 28, 2024
@dqhl76 dqhl76 added ci-benchmark Benchmark: run all test and removed ci-benchmark Benchmark: run all test labels Feb 28, 2024
@github-actions GitHub Actions
Copy link
Contributor

Docker Image for PR

  • tag: pr-14770-0c9c688

note: this image tag is only available for internal use,
please check the internal doc for more details.

@github-actions GitHub Actions
Copy link
Contributor

ClickBench Report

@zhang2014 zhang2014 added this pull request to the merge queue Feb 28, 2024
Merged via the queue into databendlabs:main with commit ff1dcbb Feb 28, 2024
70 checks passed
@dqhl76 dqhl76 deleted the continue-split-14594 branch February 29, 2024 00:18
yufan022 pushed a commit to yufan022/databend that referenced this pull request Jun 18, 2024
@zhang2014 @yufan022
feat(executor): implement time slicing for shared executor scheduling…
286e80c 
… across queries (databendlabs#14770)
yufan022 pushed a commit to yufan022/databend that referenced this pull request Jun 18, 2024
@dqhl76 @yufan022
feat(executor): implement time slicing for shared executor scheduling…
e4770c8 
… across queries (databendlabs#14770)

* refactor: split two executors and queues

* chore: add license

* fix: missing header

* refactor: apply review suggestions from last PR

* fix: remove unused function

---------

Co-authored-by: Winter Zhang <coswde@gmail.com>

(cherry picked from commit ff1dcbb)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zhang2014 zhang2014 zhang2014 approved these changes

Assignees
No one assigned
Labels
ci-benchmark Benchmark: run all test pr-feature this PR introduces a new feature to the codebase
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@dqhl76 @zhang2014