Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[v22.3.x] Add an asychronous zstd stream compression class #7850

Merged
merged 11 commits into from
Jan 5, 2023
Merged

[v22.3.x] Add an asychronous zstd stream compression class #7850

merged 11 commits into from
Jan 5, 2023

Commits on Dec 19, 2022

  1. v/compress: add async zstd class 

    (cherry picked from commit 9716f66)
    @ballard26 @vbotbuildovich
    ballard26 authored and vbotbuildovich committed Dec 19, 2022
    Configuration menu
    Copy the full SHA
    244be3f View commit details
    Browse the repository at this point in the history

  2. rpc/simple_protocol: coroutinize send_reply 

    (cherry picked from commit e938dcb)
    @ballard26 @vbotbuildovich
    ballard26 authored and vbotbuildovich committed Dec 19, 2022
    Configuration menu
    Copy the full SHA
    f5e183f View commit details
    Browse the repository at this point in the history

  3. rpc/netbuf: refactored as_scattered to be asynchronous 

    (cherry picked from commit af1a079)
    @ballard26 @vbotbuildovich
    ballard26 authored and vbotbuildovich committed Dec 19, 2022
    Configuration menu
    Copy the full SHA
    ff0153a View commit details
    Browse the repository at this point in the history

  4. rpc/transport: move as_scattered call from dispatch_send to `do_s… 

    …end`

Adding additional scheduling points in `dispatch_send` to account for
the newly futurized `as_scattered` results in a race condition where
messages are sent out of order. As such the easiest solution is to call
`as_scattered` outside of `dispatch_send` where adding a scheduling point
won't result in this race condition. Then pass the result to `dispatch_send`.

(cherry picked from commit c6ca7fd)
    @ballard26 @vbotbuildovich
    ballard26 authored and vbotbuildovich committed Dec 19, 2022
    Configuration menu
    Copy the full SHA
    186cfb1 View commit details
    Browse the repository at this point in the history

  5. rpc/netbuf: switch as_scattered to use async_stream_zstd 

    (cherry picked from commit ae838bb)
    @ballard26 @vbotbuildovich
    ballard26 authored and vbotbuildovich committed Dec 19, 2022
    Configuration menu
    Copy the full SHA
    95d699f View commit details
    Browse the repository at this point in the history

  6. rpc/parse_utils: switch from stream_zstd to async_stream_zstd 

    (cherry picked from commit 1cee0bf)
    @ballard26 @vbotbuildovich
    ballard26 authored and vbotbuildovich committed Dec 19, 2022
    Configuration menu
    Copy the full SHA
    9ad130f View commit details
    Browse the repository at this point in the history

  7. redpanda/application: init workspace for async_stream_zstd 

    (cherry picked from commit 2a00125)
    @ballard26 @vbotbuildovich
    ballard26 authored and vbotbuildovich committed Dec 19, 2022
    Configuration menu
    Copy the full SHA
    51194c9 View commit details
    Browse the repository at this point in the history

  8. tests/zstd_stream_bench: add benchmarking tests for async_stream_zstd 

    Results of performance testing vs the existing `stream_zstd` class are as follows:

```
test                                      iterations      median         mad         min         max      allocs       tasks        inst
streaming_zstd_1mb.compress                     6385   130.290us     1.513us   126.861us   137.173us      16.000       0.000         0.0
streaming_zstd_1mb.uncompress                   4185    78.486us     2.100us    75.342us    80.586us     108.616       0.000         0.0
streaming_zstd_10mb.compress                     653     1.118ms    24.680us     1.041ms     1.142ms     942.830       0.000         0.0
streaming_zstd_10mb.uncompress                   429   844.631us    80.656us   760.340us   976.986us    2095.301       0.000         0.0
async_stream_zstd.1mb_compress                  7616   100.022us   262.374ns    99.547us   100.598us     148.278       2.082         0.0
async_stream_zstd.1mb_uncompress                4804    73.661us   108.821ns    73.542us    73.819us     312.587       6.156         0.0
async_stream_zstd.10mb_compress                  772   979.403us     1.092us   978.266us   985.029us    2176.762      56.239         0.0
async_stream_zstd.10mb_uncompress                490   728.405us   720.304ns   726.690us   731.632us    4089.282     109.756         0.0
```

(cherry picked from commit bb0e700)
    @ballard26 @vbotbuildovich
    ballard26 authored and vbotbuildovich committed Dec 19, 2022
    Configuration menu
    Copy the full SHA
    f96d220 View commit details
    Browse the repository at this point in the history

  9. rpc/transport: add comment to dispatch_send warning about race cond… 

    …itions

(cherry picked from commit 12349cc)
    @ballard26 @vbotbuildovich
    ballard26 authored and vbotbuildovich committed Dec 19, 2022
    Configuration menu
    Copy the full SHA
    fc45e22 View commit details
    Browse the repository at this point in the history

  10. v/utils: add object_pool class 

    (cherry picked from commit 2922bd5)
    @ballard26 @vbotbuildovich
    ballard26 authored and vbotbuildovich committed Dec 19, 2022
    Configuration menu
    Copy the full SHA
    f90115f View commit details
    Browse the repository at this point in the history

  11. compression/async_stream_zstd: switch to using object pool 

    (cherry picked from commit 13223b6)
    @ballard26 @vbotbuildovich
    ballard26 authored and vbotbuildovich committed Dec 19, 2022
    Configuration menu
    Copy the full SHA
    800fcaa View commit details
    Browse the repository at this point in the history