Skip Scan #3000
Merged
Skip Scan #3000
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
svenklemm
force-pushed
the
skipscan
branch
3 times, most recently
from
f6723a3
to
4a650aa
Compare
Codecov Report
@@ Coverage Diff @@
## master #3000 +/- ##
========================================
Coverage 90.28% 90.28%
========================================
Files 213 215 +2
Lines 34885 35198 +313
========================================
+ Hits 31495 31778 +283
- Misses 3390 3420 +30
Continue to review full report at Codecov.
|
svenklemm
force-pushed
the
skipscan
branch
2 times, most recently
from
304903a
to
60eb779
Compare
svenklemm
force-pushed
the
skipscan
branch
21 times, most recently
from
6003360
to
4b9b272
Compare
svenklemm
force-pushed
the
skipscan
branch
3 times, most recently
from
bddcbba
to
e5cb0db
Compare
mkindahl
approved these changes
Apr 8, 2021
There was a problem hiding this comment.
I do not see a check for a table with only null values in it, which could be a potentially bad case for the state machine. You have tests for tables that contain a mix of null and non-null values, but I do not see a test where the column has only nulls.
I'm approving anyway, since this is a minor thing, but please add a test for that.
svenklemm
force-pushed
the
skipscan
branch
3 times, most recently
from
05671a1
to
57fd98a
Compare
Add helper functions to check whether a CustomPath is a ChunkAppendPath and ConstraintAwareAppendPath.
svenklemm
force-pushed
the
skipscan
branch
4 times, most recently
from
9460222
to
3e3c119
Compare
This patch implements SkipScan; an optimization for SELECT DISTINCT ON. Usually for SELECT DISTINCT ON postgres will plan either a UNIQUE over a sorted path, or some form of aggregate. In either case, it needs to scan the entire table, even in cases where there are only a few unique values. A skip scan optimizes this case when we have an ordered index. Instead of scanning the entire table and deduplicating after, the scan remembers the last value returned, and searches the index for the next value after that one. This means that for a table with k keys, with u distinct values, a skip scan runs in time u * log(k) as opposed to scanning then deduplicating, which takes time k. We can write the number of unique values u as of function of k by dividing by the number of repeats r i.e. u = k/r this means that a skip scan will be faster if each key is repeated more than a logarithmic number of times, i.e. if r > log(k) then u * log(k) < k/log(k) * log(k) < k. Co-authored-by: Joshua Lockerman <josh@timescale.com>
svenklemm
force-pushed
the
skipscan
branch
2 times, most recently
from
2af251c
to
0b114ee
Compare
k-rus
added a commit
to k-rus/timescaledb
that referenced
this pull request
Apr 12, 2021
This release adds major new features since the 2.1.1 release. We deem it moderate priority for upgrading. This release adds the skip-scan optimization, which significantly improves performance of queries with DISTINCT ON. This optimization is not available for queries on distributed hypertables. For multinode, this release adds a function to create a distributed restore point, which allows the consistent restore of a multinode cluster from a backup. The release also includes improvements to distributed query performance and memory usage. The bug fixes in this release address issues with size and stats functions, high memory usage in distributed inserts, slow distributed ORDER BY queries, indexes involving INCLUDE, and single chunk query planning. **Major Features** * timescale#2843 Add distributed restore point functionality * timescale#3000 SkipScan to speed up SELECT DISTINCT **Bugfixes** * timescale#2989 Refactor and harden size and stats functions * timescale#3058 Reduce memory usage for distributed inserts * timescale#3067 Fix extremely slow multi-node order by queries * timescale#3082 Fix chunk index column name mapping * timescale#3083 Keep Append pathkeys in ChunkAppend **Thanks** * @BowenGG for reporting an issue with indexes with INCLUDE * @fvannee for reporting an issue with ChunkAppend pathkeys * @pedrokost and @RobAtticus for reporting an issue with size functions on empty hypertables * @phemmer and @ryanbooz for reporting issues with slow multi-node order by queries * @stephane-moreau for reporting an issue with high memory usage during single-transaction inserts on a distributed hypertable.
Merged
k-rus
added a commit
to k-rus/timescaledb
that referenced
this pull request
Apr 12, 2021
This release adds major new features since the 2.1.1 release. We deem it moderate priority for upgrading. This release adds the Skip Scan optimization, which significantly improves the performance of queries with DISTINCT ON. This optimization is not available for queries on distributed hypertables. This release also adds a function to create a distributed restore point, which allows performing a consistent restore of a multi-node cluster from a backup. The bug fixes in this release address issues with size and stats functions, high memory usage in distributed inserts, slow distributed ORDER BY queries, indexes involving INCLUDE, and single chunk query planning. **Major Features** * timescale#2843 Add distributed restore point functionality * timescale#3000 SkipScan to speed up SELECT DISTINCT **Bugfixes** * timescale#2989 Refactor and harden size and stats functions * timescale#3058 Reduce memory usage for distributed inserts * timescale#3067 Fix extremely slow multi-node order by queries * timescale#3082 Fix chunk index column name mapping * timescale#3083 Keep Append pathkeys in ChunkAppend **Thanks** * @BowenGG for reporting an issue with indexes with INCLUDE * @fvannee for reporting an issue with ChunkAppend pathkeys * @pedrokost and @RobAtticus for reporting an issue with size functions on empty hypertables * @phemmer and @ryanbooz for reporting issues with slow multi-node order by queries * @stephane-moreau for reporting an issue with high memory usage during single-transaction inserts on a distributed hypertable.
k-rus
added a commit
to k-rus/timescaledb
that referenced
this pull request
Apr 12, 2021
This release adds major new features since the 2.1.1 release. We deem it moderate priority for upgrading. This release adds the Skip Scan optimization, which significantly improves the performance of queries with DISTINCT ON. This optimization is not yet available for queries on distributed hypertables. This release also adds a function to create a distributed restore point, which allows performing a consistent restore of a multi-node cluster from a backup. The bug fixes in this release address issues with size and stats functions, high memory usage in distributed inserts, slow distributed ORDER BY queries, indexes involving INCLUDE, and single chunk query planning. **PostgreSQL 11 deprecation announcement** Timescale is working hard on our next exciting features. To make that possible, we require functionality that is unfortunately absent on PostgreSQL 11. For this reason, we will continue supporting PostgreSQL 11 until mid-June 2021. Sooner to that time, we will announce the specific version of TimescaleDB in which PostgreSQL 11 support will not be included going forward. **Major Features** * timescale#2843 Add distributed restore point functionality * timescale#3000 SkipScan to speed up SELECT DISTINCT **Bugfixes** * timescale#2989 Refactor and harden size and stats functions * timescale#3058 Reduce memory usage for distributed inserts * timescale#3067 Fix extremely slow multi-node order by queries * timescale#3082 Fix chunk index column name mapping * timescale#3083 Keep Append pathkeys in ChunkAppend **Thanks** * @BowenGG for reporting an issue with indexes with INCLUDE * @fvannee for reporting an issue with ChunkAppend pathkeys * @pedrokost and @RobAtticus for reporting an issue with size functions on empty hypertables * @phemmer and @ryanbooz for reporting issues with slow multi-node order by queries * @stephane-moreau for reporting an issue with high memory usage during single-transaction inserts on a distributed hypertable.
k-rus
added a commit
to k-rus/timescaledb
that referenced
this pull request
Apr 13, 2021
This release adds major new features since the 2.1.1 release. We deem it moderate priority for upgrading. This release adds the Skip Scan optimization, which significantly improves the performance of queries with DISTINCT ON. This optimization is not yet available for queries on distributed hypertables. This release also adds a function to create a distributed restore point, which allows performing a consistent restore of a multi-node cluster from a backup. The bug fixes in this release address issues with size and stats functions, high memory usage in distributed inserts, slow distributed ORDER BY queries, indexes involving INCLUDE, and single chunk query planning. **PostgreSQL 11 deprecation announcement** Timescale is working hard on our next exciting features. To make that possible, we require functionality that is unfortunately absent on PostgreSQL 11. For this reason, we will continue supporting PostgreSQL 11 until mid-June 2021. Sooner to that time, we will announce the specific version of TimescaleDB in which PostgreSQL 11 support will not be included going forward. **Major Features** * timescale#2843 Add distributed restore point functionality * timescale#3000 SkipScan to speed up SELECT DISTINCT **Bugfixes** * timescale#2989 Refactor and harden size and stats functions * timescale#3058 Reduce memory usage for distributed inserts * timescale#3067 Fix extremely slow multi-node order by queries * timescale#3082 Fix chunk index column name mapping * timescale#3083 Keep Append pathkeys in ChunkAppend **Thanks** * @BowenGG for reporting an issue with indexes with INCLUDE * @fvannee for reporting an issue with ChunkAppend pathkeys * @pedrokost and @RobAtticus for reporting an issue with size functions on empty hypertables * @phemmer and @ryanbooz for reporting issues with slow multi-node order by queries * @stephane-moreau for reporting an issue with high memory usage during single-transaction inserts on a distributed hypertable.
k-rus
added a commit
that referenced
this pull request
Apr 13, 2021
This release adds major new features since the 2.1.1 release. We deem it moderate priority for upgrading. This release adds the Skip Scan optimization, which significantly improves the performance of queries with DISTINCT ON. This optimization is not yet available for queries on distributed hypertables. This release also adds a function to create a distributed restore point, which allows performing a consistent restore of a multi-node cluster from a backup. The bug fixes in this release address issues with size and stats functions, high memory usage in distributed inserts, slow distributed ORDER BY queries, indexes involving INCLUDE, and single chunk query planning. **PostgreSQL 11 deprecation announcement** Timescale is working hard on our next exciting features. To make that possible, we require functionality that is unfortunately absent on PostgreSQL 11. For this reason, we will continue supporting PostgreSQL 11 until mid-June 2021. Sooner to that time, we will announce the specific version of TimescaleDB in which PostgreSQL 11 support will not be included going forward. **Major Features** * #2843 Add distributed restore point functionality * #3000 SkipScan to speed up SELECT DISTINCT **Bugfixes** * #2989 Refactor and harden size and stats functions * #3058 Reduce memory usage for distributed inserts * #3067 Fix extremely slow multi-node order by queries * #3082 Fix chunk index column name mapping * #3083 Keep Append pathkeys in ChunkAppend **Thanks** * @BowenGG for reporting an issue with indexes with INCLUDE * @fvannee for reporting an issue with ChunkAppend pathkeys * @pedrokost and @RobAtticus for reporting an issue with size functions on empty hypertables * @phemmer and @ryanbooz for reporting issues with slow multi-node order by queries * @stephane-moreau for reporting an issue with high memory usage during single-transaction inserts on a distributed hypertable.
k-rus
added a commit
to k-rus/timescaledb
that referenced
this pull request
Apr 13, 2021
This release adds major new features since the 2.1.1 release. We deem it moderate priority for upgrading. This release adds the Skip Scan optimization, which significantly improves the performance of queries with DISTINCT ON. This optimization is not yet available for queries on distributed hypertables. This release also adds a function to create a distributed restore point, which allows performing a consistent restore of a multi-node cluster from a backup. The bug fixes in this release address issues with size and stats functions, high memory usage in distributed inserts, slow distributed ORDER BY queries, indexes involving INCLUDE, and single chunk query planning. **PostgreSQL 11 deprecation announcement** Timescale is working hard on our next exciting features. To make that possible, we require functionality that is unfortunately absent on PostgreSQL 11. For this reason, we will continue supporting PostgreSQL 11 until mid-June 2021. Sooner to that time, we will announce the specific version of TimescaleDB in which PostgreSQL 11 support will not be included going forward. **Major Features** * timescale#2843 Add distributed restore point functionality * timescale#3000 SkipScan to speed up SELECT DISTINCT **Bugfixes** * timescale#2989 Refactor and harden size and stats functions * timescale#3058 Reduce memory usage for distributed inserts * timescale#3067 Fix extremely slow multi-node order by queries * timescale#3082 Fix chunk index column name mapping * timescale#3083 Keep Append pathkeys in ChunkAppend **Thanks** * @BowenGG for reporting an issue with indexes with INCLUDE * @fvannee for reporting an issue with ChunkAppend pathkeys * @pedrokost and @RobAtticus for reporting an issue with size functions on empty hypertables * @phemmer and @ryanbooz for reporting issues with slow multi-node order by queries * @stephane-moreau for reporting an issue with high memory usage during single-transaction inserts on a distributed hypertable.
Merged
k-rus
added a commit
that referenced
this pull request
Apr 13, 2021
This release adds major new features since the 2.1.1 release. We deem it moderate priority for upgrading. This release adds the Skip Scan optimization, which significantly improves the performance of queries with DISTINCT ON. This optimization is not yet available for queries on distributed hypertables. This release also adds a function to create a distributed restore point, which allows performing a consistent restore of a multi-node cluster from a backup. The bug fixes in this release address issues with size and stats functions, high memory usage in distributed inserts, slow distributed ORDER BY queries, indexes involving INCLUDE, and single chunk query planning. **PostgreSQL 11 deprecation announcement** Timescale is working hard on our next exciting features. To make that possible, we require functionality that is unfortunately absent on PostgreSQL 11. For this reason, we will continue supporting PostgreSQL 11 until mid-June 2021. Sooner to that time, we will announce the specific version of TimescaleDB in which PostgreSQL 11 support will not be included going forward. **Major Features** * #2843 Add distributed restore point functionality * #3000 SkipScan to speed up SELECT DISTINCT **Bugfixes** * #2989 Refactor and harden size and stats functions * #3058 Reduce memory usage for distributed inserts * #3067 Fix extremely slow multi-node order by queries * #3082 Fix chunk index column name mapping * #3083 Keep Append pathkeys in ChunkAppend **Thanks** * @BowenGG for reporting an issue with indexes with INCLUDE * @fvannee for reporting an issue with ChunkAppend pathkeys * @pedrokost and @RobAtticus for reporting an issue with size functions on empty hypertables * @phemmer and @ryanbooz for reporting issues with slow multi-node order by queries * @stephane-moreau for reporting an issue with high memory usage during single-transaction inserts on a distributed hypertable.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This patch implements a Skip Scan; an optimization for
SELECT DISTINCT ON.Usually for
SELECT DISTINCT ONpostgres will plan either aUNIQUEover asorted path, or some form of aggregate. In either case, it needs to scan the
entire table, even in cases where there are only a few unique values.
A skip-scan optimizes this case when we have an ordered index. Instead of
scanning the entire table and deduplicating after, the scan remembers the last
value returned, and searches the index for the next value after that one. This
means that for a table with
kkeys, withudistinct values, a skip-scan runsin time
u * log(k)as opposed to scanning then deduplicating, which takes timek. We can write the number of unique valuesuas of function ofkbydividing by the number of repeats
ri.e.u = k/rthis means that a skip-scanwill be faster if each key is repeated more than a logarithmic number of times,
i.e. if
r > log(k)thenu * log(k) < k/log(k) * log(k) < k.Disable-check: commit-count