Tracking issue for Vision of Databend DAL

[Xuanwo]

• proposal: Vision of Databend DAL #3662
• Related Issues
  • Copy INTO <table> from external location  #3586
  • CREATE STAGE  #2976
  • Make parquet reader works with only one stream reader. #3225

Roadmap

• dal: Refactor to make it easier for contributors to add new storage type #3436
  • dal2: Add basic operations of read, write, stat, delete #3575
• Implement services for PoC
  • dal2: Implement fs support #3695
  • dal2: Implement s3 support #3774
    • We need to wait for dal: Add integration tests target on production services apache/opendal#8
    • Need enable_pod_iam_policy support
• Integrate dal2 into streams (replace dal, implement AsyncSeek) #3874
  • Sometimes we only need to read till end, don't need total size
    • dal2: Eliminate type parameters in DAL #4001 (comment)
• dal2: Implement LimitedReader #3937
• dal2: Implement DAL Layer support #4067
• dal2: Implement Context support #4069
• query: Replace dal with dal2, let's rock! #4081
  • Remove all block_on to avoid performance issues
  • eliminate all unwrap in non-testing code
  • create table test as select * from numbers(1234567890);select * from test; doesn't works: Fixed in dal2: Fix SeekableRead not read correctly #4107
  • Make sure streaming load works: https://databend.rs/user/data-loading/http-streaming-load#examples
• dal: Add integration tests target on production services apache/opendal#8

Longterm Ideas

• Rewrite parquet2 into cloud storage native style
  • Remove AsyncSeek trait requirement (use style like pread instead of open-seek-read)
  • parquet is designed by appending metadata in the file tail, this task is impossible.
  • Maybe we can find a more s3 native data format
• Split dal2 into a separate repo: moved to https://github.com/datafuselabs/opendal
  • more independent
  • easy to do integration tests

[Xuanwo]

All required work has been done. opendal will handle all new feature requests. Let's start a new journey!

Thank @dantengsky, for much valuable help!