Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Change native spill compression kind to zstd
This helps reduce the spill bytes to speedup spill performance. For exmaple 20240608_210530_00001_iyjad execution time reduced from 12mins to 9 mins. The spilled bytes has been reduced from 2.62TB to 960GB. The total disk write time has been reduced from 12hrs to 4hrs. The max time of a single aggregation operator has been reduced from 2.4mins to 40s. And there is not much change to the compression time. And the disk read time has been reduced from 1.29 days to 14.46 hours. The max time of a single aggregation operator has been reduced from 5 mins to 2 mins. This explains the overall e2e time reduction
- Loading branch information