Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dal2: Fix SeekableRead not read correctly #4107

Merged
merged 2 commits into from
Feb 12, 2022
Merged

Conversation

Xuanwo
Copy link
Member

@Xuanwo Xuanwo commented Feb 11, 2022

Signed-off-by: Xuanwo github@xuanwo.io

I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/

Summary

Part of #3677

The following SQL doesn't works:

create table test as select * from numbers(1234567890);
select count(*) from test;

Meeting EOF errors like:

2022-02-11T12:22:55.395256Z ERROR databend_query::servers::mysql::writers::query_result_writer: OnQuery Error: Code: 1024, displayText = fail to read block _b/4e5ffe0fad65439990109f9a0d9bf5ff.parquet, Code: 1002, displayText = External format error: underlying IO error: unexpected end of file..

   0: common_exception::exception_code::<impl common_exception::exception::ErrorCode>::ParquetError
             at common/exception/src/exception_code.rs:36:66
   1: databend_query::storages::fuse::operations::read::<impl databend_query::storages::fuse::table::FuseTable>::do_read::{{closure}}::{{closure}}::{{closure}}::{{closure}}
             at query/src/storages/fuse/operations/read.rs:88:25
   2: core::result::Result<T,E>::map_err

This is caused by the incorrect SeekableReader implementation.

Changelog

  • New Feature

Test Plan

Unit Tests

Stateless Tests

Signed-off-by: Xuanwo <github@xuanwo.io>
@databend-bot
Copy link
Member

Thanks for the contribution!
I have applied any labels matching special text in your PR Changelog.

Please review the labels and make any necessary changes.

@databend-bot databend-bot added pr-feature this PR introduces a new feature to the codebase labels Feb 11, 2022
@databend-bot
Copy link
Member

Thanks for the contribution!
I have applied any labels matching special text in your PR Changelog.

Please review the labels and make any necessary changes.

@vercel
Copy link

vercel bot commented Feb 11, 2022

This pull request is being automatically deployed with Vercel (learn more).
To see the status of your deployment, click below or on the icon next to each commit.

🔍 Inspect: https://vercel.com/databend/databend/3WXALENzejEVq6uBHZrRX1ynnQHL
✅ Preview: Canceled

[Deployment for 9310603 canceled]

@BohuTANG
Copy link
Member

I remember we have a test about create table select from ..., why the test passed before?

@Xuanwo
Copy link
Member Author

Xuanwo commented Feb 11, 2022

I remember we have a test about create table select from ..., why the test passed before?

Maybe the dataset is too small that can be fit into buffer so that there's no extra read. (under 1MB)

I will add this case into stateless test too.

@BohuTANG
Copy link
Member

It seems we need more unit tests on dal2, like seekable.rs, to make our logic is right.

@Xuanwo
Copy link
Member Author

Xuanwo commented Feb 11, 2022

It seems we need more unit tests on dal2, like seekable.rs, to make our logic is right.

True, I will keep improving this part.

@codecov-commenter
Copy link

codecov-commenter commented Feb 11, 2022

Codecov Report

Merging #4107 (9310603) into main (85d443b) will increase coverage by 0%.
The diff coverage is 100%.

Impacted file tree graph

@@          Coverage Diff           @@
##            main   #4107    +/-   ##
======================================
  Coverage     57%     57%            
======================================
  Files        857     857            
  Lines      46462   46255   -207     
======================================
- Hits       26707   26670    -37     
+ Misses     19755   19585   -170     
Impacted Files Coverage Δ
common/dal2/src/readers/seekable.rs 97% <100%> (+<1%) ⬆️
common/datavalues/src/arrays/string/builder.rs 43% <0%> (-13%) ⬇️
query/src/common/mod.rs 76% <0%> (-8%) ⬇️
common/dal/src/context.rs 59% <0%> (-5%) ⬇️
common/datavalues/src/arrays/comparison.rs 6% <0%> (-5%) ⬇️
common/functions/src/scalars/function2_factory.rs 91% <0%> (-1%) ⬇️
common/datavalues/src/arrays/primitive/mod.rs 38% <0%> (-1%) ⬇️
common/functions/src/scalars/strings/string.rs 98% <0%> (ø)
metasrv/src/meta_service/raftmeta.rs 49% <0%> (+<1%) ⬆️
... and 8 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 85d443b...9310603. Read the comment docs.

Signed-off-by: Xuanwo <github@xuanwo.io>
@BohuTANG
Copy link
Member

/lgtm

Preferred add the unit tests in the later PR :D

@BohuTANG BohuTANG merged commit 0147804 into databendlabs:main Feb 12, 2022
@Xuanwo Xuanwo deleted the fix-dal2 branch February 12, 2022 05:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
need-review pr-feature this PR introduces a new feature to the codebase
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants