Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add documentation for segment replication GA release #3461

Merged
merged 18 commits into from
Apr 18, 2023
Merged

Add documentation for segment replication GA release #3461

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
9da6776
Add documentation for segment replication GA release
ariamarble Mar 15, 2023
0d594e8
Merge branch 'main' of https://github.com/opensearch-project/document…
ariamarble Apr 5, 2023
4567d50
SegRep doc updates GA
ariamarble Apr 5, 2023
5796571
Merge branch 'main' of https://github.com/opensearch-project/document…
ariamarble Apr 10, 2023
52f11b0
content updates
ariamarble Apr 10, 2023
5446b9e
Apply suggestions from doc review
ariamarble Apr 10, 2023
9d8809e
additional doc review changes
ariamarble Apr 10, 2023
813ed17
small changes
ariamarble Apr 10, 2023
4523881
Apply suggestions from editorial review
ariamarble Apr 11, 2023
1289018
Update _tuning-your-cluster/availability-and-recovery/segment-replica…
ariamarble Apr 11, 2023
2f551ca
Merge branch 'main' of https://github.com/opensearch-project/document…
ariamarble Apr 11, 2023
dbc27dd
Add primary balance information
kolchfa-aws Apr 14, 2023
a6c2438
Merge branch 'issue12255' of https://github.com/opensearch-project/do…
ariamarble Apr 15, 2023
966585f
Merge branch 'main' of https://github.com/opensearch-project/document…
ariamarble Apr 15, 2023
d79ab3f
adding backpressure page
ariamarble Apr 17, 2023
8d51eeb
Implemented tech review comments
kolchfa-aws Apr 17, 2023
b697049
Merge branch 'main' of https://github.com/opensearch-project/document…
ariamarble Apr 18, 2023
343a954
final changes
ariamarble Apr 18, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
84 changes: 0 additions & 84 deletions ...ing-your-cluster/availability-and-recovery/segment-replication/configuration.md
View file
Open in desktop

This file was deleted.

188 changes: 181 additions & 7 deletions _tuning-your-cluster/availability-and-recovery/segment-replication/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,17 +11,191 @@ redirect_from:

# Segment replication

Segment replication is an experimental feature with OpenSearch 2.3. Therefore, we do not recommend the use of segment replication in a production environment. For updates on the progress of segment replication or if you want leave feedback that could help improve the feature, see the [Segment replication git issue](https://github.com/opensearch-project/OpenSearch/issues/2194).
{: .warning}

With segment replication, segment files are copied across shards instead of documents being indexed on each shard copy. This improves indexing throughput and lowers resource utilization at the expense of increased network utilization.

As an experimental feature, segment replication will be behind a feature flag and must be enabled on **each node** of a cluster and pass a new setting during index creation.
{: .note }
This is the first feature in a series of features designed to decouple reads and writes in order to lower compute costs.

### Potential use cases
## Use cases

- Users who have high write loads but do not have high search requirements and are comfortable with longer refresh times.
- Users with very high loads who want to add new nodes, as you do not need to index all nodes when adding a new node to the cluster.
ariamarble marked this conversation as resolved.
Show resolved Hide resolved
- OpenSearch cluster deployments with low replica counts, such as those used for log analytics.

## Segment replication configuration

To set segment replication as the replication strategy, create an index with replication.type set to `SEGMENT`:

````json
PUT /my-index1
{
"settings": {
"index": {
"replication.type": "SEGMENT"
}
}
}
````

## Segment vs document replication benchmarks
ariamarble marked this conversation as resolved.
Show resolved Hide resolved
ariamarble marked this conversation as resolved.
Show resolved Hide resolved

During initial benchmarks, segment replication users reported 40% higher throughput than when using document replication with the same cluster setup.

The following benchmarks were collected with [OpenSearch-benchmark](https://github.com/opensearch-project/opensearch-benchmark) using the [`stackoverflow`](https://www.kaggle.com/datasets/stackoverflow/stackoverflow) and [`nyc_taxi`](https://github.com/topics/nyc-taxi-dataset) datasets.

Both test runs were performed on a 10-node (m5.xlarge) cluster with 10 shards and 5 replicas. Each shard was about <!-- TODO: insert size --> xxMBs in size. The benchmarking results are listed in the following table.

<table>
<tr>
<td></td>
<td></td>
<td>Document Replication</td>
<td>Segment Replication</td>
<td>Percent difference</td>
</tr>
<tr>
<td>Test execution time (minutes)</td>
<td></td>
<td>40.00</td>
<td>22.00</td>
<td></td>
</tr>
<tr>
<td rowspan="3">Throughput (number of requests per second)</td>
<td>p0</td>
<td>17553.90</td>
<td>28584.30</td>
<td>63%</td>
</tr>
<tr>
<td>p50</td>
<td>20647.20</td>
<td>32790.20</td>
<td>59%</td>
</tr>
<tr>
<td>p100</td>
<td>23209.00</td>
<td>34286.00</td>
<td>48%</td>
</tr>
<tr>
<td rowspan="4">CPU (%)</td>
<td>p50</td>
<td>65.00</td>
<td>30.00</td>
<td>-54%</td>
</tr>
<tr>
<td>p90</td>
<td>79.00</td>
<td>35.00</td>
<td>-56%</td>
</tr>
<tr>
<td>p99</td>
<td>98.00</td>
<td>45.08</td>
<td>-54%</td>
</tr>
<tr>
<td>p100</td>
<td>98.00</td>
<td>59.00</td>
<td>-40%</td>
</tr>
<tr>
<td rowspan="4">Memory (%)</td>
<td>p50</td>
<td>48.20</td>
<td>39.00</td>
<td>-19%</td>
</tr>
<tr>
<td>p90</td>
<td>62.00</td>
<td>61.00</td>
<td>-2%</td>
</tr>
<tr>
<td>p99</td>
<td>66.21</td>
<td>68.00</td>
<td>3%</td>
</tr>
<tr>
<td>p100</td>
<td>71.00</td>
<td>69.00</td>
<td>-3%</td>
</tr>
<tr>
<td rowspan="4">IOPS</td>
<td>p50</td>
<td></td>
<td></td>
<td></td>
</tr>
<tr>
<td>p90</td>
<td></td>
<td></td>
<td></td>
</tr>
<tr>
<td>p99</td>
<td></td>
<td></td>
<td></td>
</tr>
<tr>
<td>p100</td>
<td></td>
<td></td>
<td></td>
</tr>
<tr>
<td rowspan="4">Latency</td>
<td>p50</td>
<td></td>
<td></td>
<td></td>
</tr>
<tr>
<td>p90</td>
<td></td>
<td></td>
<td></td>
</tr>
<tr>
<td>p99</td>
<td></td>
<td></td>
<td></td>
</tr>
<tr>
<td>p100</td>
<td></td>
<td></td>
<td></td>
</tr>
</table>

Your results may vary based on the cluster topology, hardware used, shard count, and merge settings.
{: .note }

## Other considerations
ariamarble marked this conversation as resolved.
Show resolved Hide resolved

When using segment replication, consider the following:

1. Enabling segment replication for an existing index requires [reindexing](https://github.com/opensearch-project/OpenSearch/issues/3685).
1. Rolling upgrades are not currently supported. Full cluster restarts are required when upgrading indexes using segment replication. See [Issue 3881](https://github.com/opensearch-project/OpenSearch/issues/3881).
1. [Cross-cluster replication](https://github.com/opensearch-project/OpenSearch/issues/4090) does not currently use segment replication to copy between clusters.
ariamarble marked this conversation as resolved.
Show resolved Hide resolved
1. Increased network congestion on primary shards. See [Issue - Optimize network bandwidth on primary shards](https://github.com/opensearch-project/OpenSearch/issues/4245).
1. Integration with remote-backed storage as the source of replication is [currently unsupported](https://github.com/opensearch-project/OpenSearch/issues/4448).
1. Read-after-write guarantees: The `wait_until` refresh policy is not compatible with segment replication. If you use the `wait_until` refresh policy while ingesting documents, you'll get a response only after the primary node has refreshed and made those documents searchable. Replica shards will respond only after having written to their local translog. We are exploring other mechanisms for providing read-after-write guarantees. For more information, see the corresponding [GitHub issue](https://github.com/opensearch-project/OpenSearch/issues/6046).
1. System indexes will continue to use document replication internally until read-after-write guarantees are available. In this case, document replication does not hinder the overall performance because there are few system indexes.

## Next steps

This is the first step in a series of features designed to decouple reads and writes in order to lower compute costs.
1. Track [future enhancements to segment replication](https://github.com/orgs/opensearch-project/projects/99).
1. Read [this blog post about segment replication](https://github.com/opensearch-project/project-website/pull/1504/files).