From 1137a1f005e602054437cb68d7f96047b1e61111 Mon Sep 17 00:00:00 2001 From: xiaoxmeng Date: Sat, 8 Jun 2024 15:26:03 -0700 Subject: [PATCH] Change native spill compression kind to zstd This helps reduce the spill bytes to speedup spill performance. For exmaple 20240608_210530_00001_iyjad execution time reduced from 12mins to 9 mins. The spilled bytes has been reduced from 2.62TB to 960GB. The total disk write time has been reduced from 12hrs to 4hrs. The max time of a single aggregation operator has been reduced from 2.4mins to 40s. And there is not much change to the compression time. And the disk read time has been reduced from 1.29 days to 14.46 hours. The max time of a single aggregation operator has been reduced from 5 mins to 2 mins. This explains the overall e2e time reduction --- .../main/java/com/facebook/presto/SystemSessionProperties.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/presto-main/src/main/java/com/facebook/presto/SystemSessionProperties.java b/presto-main/src/main/java/com/facebook/presto/SystemSessionProperties.java index 5562ff6a9778..bdc3bdf7486a 100644 --- a/presto-main/src/main/java/com/facebook/presto/SystemSessionProperties.java +++ b/presto-main/src/main/java/com/facebook/presto/SystemSessionProperties.java @@ -1617,7 +1617,7 @@ public SystemSessionProperties( NATIVE_SPILL_COMPRESSION_CODEC, "Native Execution only. The compression algorithm type to compress the spilled data.\n " + "Supported compression codecs are: ZLIB, SNAPPY, LZO, ZSTD, LZ4 and GZIP. NONE means no compression.", - "snappy", + "zstd", false), longProperty( NATIVE_SPILL_WRITE_BUFFER_SIZE,