Skip to content

Commit

Permalink
Change native spill compression kind to zstd
Browse files Browse the repository at this point in the history
This helps reduce the spill bytes to speedup spill performance. For exmaple 20240608_210530_00001_iyjad
execution time reduced from 12mins to 9 mins. The spilled bytes has been reduced from 2.62TB to 960GB. The
total disk write time has been reduced from 12hrs to 4hrs. The max time of a single aggregation operator has
been reduced from 2.4mins to 40s. And there is not much change to the compression time. And the disk read
time has been reduced from 1.29 days to 14.46 hours. The max time of a single aggregation operator has been
reduced from 5 mins to 2 mins. This explains the overall e2e time reduction
  • Loading branch information
xiaoxmeng committed Jun 11, 2024
1 parent ed65051 commit 1137a1f
Showing 1 changed file with 1 addition and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -1617,7 +1617,7 @@ public SystemSessionProperties(
NATIVE_SPILL_COMPRESSION_CODEC,
"Native Execution only. The compression algorithm type to compress the spilled data.\n " +
"Supported compression codecs are: ZLIB, SNAPPY, LZO, ZSTD, LZ4 and GZIP. NONE means no compression.",
"snappy",
"zstd",
false),
longProperty(
NATIVE_SPILL_WRITE_BUFFER_SIZE,
Expand Down

0 comments on commit 1137a1f

Please sign in to comment.