Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(query): window partition by spill to disk #16441

Open
wants to merge 45 commits into
base: main
Choose a base branch
from
Open

feat(query): window partition by spill to disk #16441

wants to merge 45 commits into from
+1,550 −304

Conversation

forsaken628
Copy link
Collaborator

@forsaken628 forsaken628 commented Sep 11, 2024 •
edited by drmingdrmer
Loading

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

window partition by spill to disk

Tests

  • Unit Test
  • Logic Test
  • Benchmark Test
  • No Test - Explain why

Type of change

  • Bug Fix (non-breaking change which fixes an issue)
  • New Feature (non-breaking change which adds functionality)
  • Breaking Change (fix or feature that could cause existing functionality not to work as expected)
  • Documentation Update
  • Refactoring
  • Performance Improvement
  • Other (please describe):

This change is Reviewable

@forsaken628
dma write
70d66b9 
Signed-off-by: coldWater <forsaken628@gmail.com>
@forsaken628
partition spill to disk
5666858 
Signed-off-by: coldWater <forsaken628@gmail.com>
@github-actions github-actions bot added the pr-feature this PR introduces a new feature to the codebase label Sep 11, 2024
@forsaken628
Merge remote-tracking branch 'up/main' into spill-disk
d757e08
@forsaken628
fix
c3432de 
Signed-off-by: coldWater <forsaken628@gmail.com>
@forsaken628
refine
ebe64bc 
Signed-off-by: coldWater <forsaken628@gmail.com>
@forsaken628
temp file
0e0184d 
Signed-off-by: coldWater <forsaken628@gmail.com>
@forsaken628
Merge remote-tracking branch 'up/main' into spill-disk
43110ec 
Signed-off-by: coldWater <forsaken628@gmail.com>
@forsaken628
refactor spill
3ac783e 
Signed-off-by: coldWater <forsaken628@gmail.com>
@forsaken628
clear temp dir
f41a038 
Signed-off-by: coldWater <forsaken628@gmail.com>
@forsaken628
move dma
11f3f70 
Signed-off-by: coldWater <forsaken628@gmail.com>
@forsaken628
config
c77dbcd 
Signed-off-by: coldWater <forsaken628@gmail.com>
@forsaken628
fix
205a1de 
Signed-off-by: coldWater <forsaken628@gmail.com>
@forsaken628
bytes limit
cdf6d78 
Signed-off-by: coldWater <forsaken628@gmail.com>
@forsaken628 forsaken628 marked this pull request as ready for review September 14, 2024 04:44
@dosubot dosubot bot added size:XXL This PR changes 1000+ lines, ignoring generated files. A-query Area: databend query C-feature Category: feature labels Sep 14, 2024
@BohuTANG
Copy link
Member

Is there some benchmark numbers for this PR?

@forsaken628
Copy link
Collaborator Author

Is there some benchmark numbers for this PR?

@BohuTANG
Here just replace the storage, where can I go to find the right environment to do benchmark? If the test environment is not representative, benchmark has no meaning.

@BohuTANG
Copy link
Member

Is there some benchmark numbers for this PR?

@BohuTANG Here just replace the storage, where can I go to find the right environment to do benchmark? If the test environment is not representative, benchmark has no meaning.

I think @Dousir9 has some cases to do benchmark, because #16448 also need it.

@forsaken628
rustix
37a8be8 
Signed-off-by: coldWater <forsaken628@gmail.com>
@forsaken628
rustix
121f15f 
Signed-off-by: coldWater <forsaken628@gmail.com>
@forsaken628
temp
1a1ae2a 
Signed-off-by: coldWater <forsaken628@gmail.com>
@Dousir9 Dousir9 added ci-cloud Build docker image for cloud test and removed ci-cloud Build docker image for cloud test labels Sep 29, 2024
@github-actions GitHub Actions
Copy link
Contributor

Docker Image for PR

  • tag: pr-16441-c7090de-1727602679

note: this image tag is only available for internal use,
please check the internal doc for more details.

@forsaken628
fix
8520d80 
Signed-off-by: coldWater <forsaken628@gmail.com>
@Dousir9 Dousir9 added ci-cloud Build docker image for cloud test and removed ci-cloud Build docker image for cloud test labels Sep 29, 2024
@github-actions GitHub Actions
Copy link
Contributor

Docker Image for PR

  • tag: pr-16441-9860569-1727612777

note: this image tag is only available for internal use,
please check the internal doc for more details.

@forsaken628
refactor dma
1583f77 
Signed-off-by: coldWater <forsaken628@gmail.com>
@forsaken628
change config
657c1c2 
Signed-off-by: coldWater <forsaken628@gmail.com>
@forsaken628
Alignment
0e70745 
Signed-off-by: coldWater <forsaken628@gmail.com>
@forsaken628
Merge branch 'main' into spill-disk
4210209
@forsaken628 forsaken628 added this pull request to the merge queue Sep 30, 2024
@Dousir9 Dousir9 removed this pull request from the merge queue due to a manual request Sep 30, 2024
@forsaken628 forsaken628 added ci-cloud Build docker image for cloud test and removed ci-cloud Build docker image for cloud test labels Sep 30, 2024
@github-actions GitHub Actions
Copy link
Contributor

Docker Image for PR

  • tag: pr-16441-4e7f81f-1727698886

note: this image tag is only available for internal use,
please check the internal doc for more details.

@forsaken628
cloud test
b06b3d7 
Signed-off-by: coldWater <forsaken628@gmail.com>
@forsaken628
Merge remote-tracking branch 'origin/spill-disk' into spill-disk
3ed1366
@forsaken628
x
37a2ec6 
Signed-off-by: coldWater <forsaken628@gmail.com>
@forsaken628 forsaken628 added ci-cloud Build docker image for cloud test and removed ci-cloud Build docker image for cloud test labels Sep 30, 2024
@github-actions GitHub Actions
Copy link
Contributor

Docker Image for PR

  • tag: pr-16441-71c69b3-1727706321

note: this image tag is only available for internal use,
please check the internal doc for more details.

@forsaken628
clean
36162bf 
Signed-off-by: coldWater <forsaken628@gmail.com>
@forsaken628 forsaken628 added ci-cloud Build docker image for cloud test and removed ci-cloud Build docker image for cloud test labels Sep 30, 2024
@github-actions GitHub Actions
Copy link
Contributor

Docker Image for PR

  • tag: pr-16441-a41f29e-1727715020

note: this image tag is only available for internal use,
please check the internal doc for more details.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Dousir9 Dousir9 Dousir9 left review comments

@sundy-li sundy-li sundy-li approved these changes

Assignees
No one assigned
Labels
A-query Area: databend query C-feature Category: feature ci-cloud Build docker image for cloud test lgtm This PR has been approved by a maintainer pr-feature this PR introduces a new feature to the codebase size:XXL This PR changes 1000+ lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@forsaken628 @BohuTANG @Dousir9 @sundy-li