Skip to content

Fix up the 'Templates' section #2178

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 7 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions manage-data/data-store/index-basics.md
Original file line number Diff line number Diff line change
Expand Up @@ -289,7 +289,7 @@ You’re now ready to create new indices using your index template.
GET /my-index-000001,my-index-000002
```

### Manage component templates
### Manage component templates [index-management-manage-component-templates]

Component templates are a type of [template](/manage-data/data-store/templates.md) used as reusable building blocks within index templates to configure index settings, mappings, and aliases.

Expand All @@ -303,7 +303,7 @@ Create, edit, clone, and delete your component templates in the **Component Temp
* To show details and perform operations, click the template name.
* To create new component templates, use the **Create component template** wizard.

### Manage enrich policies
### Manage enrich policies [manage-enrich-policies]

An [enrich policy](/manage-data/ingest/transform-enrich/data-enrichment.md#enrich-policy) is a set of configuration options used to add data from your existing indices to incoming documents during ingest. An enrich policy contains:

Expand Down
123 changes: 68 additions & 55 deletions manage-data/data-store/templates.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,27 +8,80 @@ products:
- id: elasticsearch
---

# Templates [index-templates]
# Templates [elasticsearch-templates]

::::{note}
This topic describes the composable index templates introduced in {{es}} 7.8. For information about how index templates worked previously, see the [legacy template documentation](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-template).
::::
Templates are tools used to construct indices in {{es}}. They specify the properties that each created index will have.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel the word tools a bit vague in this context, and I'd like to treat templates in singular, as the mechanism. Let me know your thoughts about this proposal:

Suggested change
Templates are tools used to construct indices in {{es}}. They specify the properties that each created index will have.
Templates are the mechanism that {{es}} uses to apply settings, mappings, and other configurations when creating indices or data streams.

(I like including also data streams, as the way to create data streams is also through templates).

Another possibility:

Templates are the mechanism by which {{es}} applies settings, mappings, and other configurations when creating indices or data streams.


You configure templates prior to creating indices. This way, when each index is created, either manually or through the indexing of a document, the template settings are used as a basis for creating the index. When used with a [data stream](/manage-data/data-store/data-streams.md), a template configures the stream's backing indices as they are created.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
You configure templates prior to creating indices. This way, when each index is created, either manually or through the indexing of a document, the template settings are used as a basis for creating the index. When used with a [data stream](/manage-data/data-store/data-streams.md), a template configures the stream's backing indices as they are created.
You configure templates prior to creating indices or data streams. When an index is created, either manually or by indexing a document, the matching template determines the settings, mappings, and other configurations to apply. When used with a [data stream](/manage-data/data-store/data-streams.md), a template also defines how each backing index is configured as it is created.


$$$getting$$$
An index template is a way to tell {{es}} how to configure an index when it is created. For data streams, the index template configures the stream’s backing indices as they are created. Templates are configured **prior to index creation**. When an index is created - either manually or through indexing a document - the template settings are used as a basis for creating the index.
There are two types of template:

There are two types of templates: index templates and [component templates](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-component-template). Component templates are reusable building blocks that configure mappings, settings, and aliases. While you can use component templates to construct index templates, they aren’t directly applied to a set of indices. Index templates can contain a collection of component templates, as well as directly specify settings, mappings, and aliases.
* An [**index template**](#index-templates) is used to configure indices when they are created, by specifying the mappings, settings, and aliases that each index inherits.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* An [**index template**](#index-templates) is used to configure indices when they are created, by specifying the mappings, settings, and aliases that each index inherits.
* An [**index template**](#index-templates) is the main configuration object applied when creating an index or data stream. It matches index names using `index_patterns` and resolves conflicts using a `priority` value. An index template can optionally define settings, mappings, and aliases directly, and refer to a list of component templates that provide reusable configuration blocks. It can also indicate whether it should create a data stream or a regular index.

Not sure.... maybe too long, but in the intro to index templates I think there's something important to highlight:

  • is the component that has to match the index or data stream name
  • it's used to determine if the item to create is a regular index or a data stream (for this we can just link to data streams documentation).
  • can optionally include settings, mappings, and aliases, and refer to component templates.

The current description just puts the focus on mappings, settings and aliases, which is more aligned with the legacy templates and old way of working, without using composable templates.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe you prefer something much shorter here and describe the points I mention later in the section dedicated to index templates. Whatever you prefer :)


The following conditions apply to index templates:
* A [**component template**](#component-templates) is a reusable building block used to construct index templates.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* A [**component template**](#component-templates) is a reusable building block used to construct index templates.
* A [**component template**](#component-templates) is a reusable building block that defines settings, mappings, and aliases. Component templates are not applied directly; they must be referenced by index templates.


* Composable templates take precedence over legacy templates. If no composable template matches a given index, a legacy template may still match and be applied.
The following conditions apply to using templates:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding one sentence that can be the glue for what's coming next.

Suggested change
The following conditions apply to using templates:
Together, index templates and their referenced component templates form what is known as *composable templates*.
The following conditions apply to using templates:


* Composable index templates take precedence over legacy templates. If no composable template matches a given index, a legacy template may still match and be applied.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add a link in legacy templates? The concept of legacy templates doesn't appear in the doc, and might be difficult for a reader to understand what's about.

* If an index is created with explicit settings and also matches an index template, the settings from the [create index](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-create) request take precedence over settings specified in the index template and its component templates.
* Settings specified in the index template itself take precedence over the settings in its component templates.
* If a new data stream or index matches more than one index template, the index template with the highest priority is used.
* When you create an index template, be careful to avoid [naming pattern collisions](#avoid-index-pattern-collisions) with built-in {{es}} index templates.


## Index templates [index-templates]

An **index template** is used to configure an index when it is created. [Mappings](/manage-data/data-store/mapping.md), [settings](elasticsearch://reference/elasticsearch/index-settings/index.md), and [aliases](/manage-data/data-store/aliases.md) specified in the index template are inherited by each created index. These can be specified directly in the index template, and they can be specified in one or more component templates used to construct the index template.

You can create and manage index templates on the **Index management** page in {{kib}} or by using the [index template](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-index-template) API.

For the {{kib}} steps and a walk-through example, refer to [Manage index templates](/manage-data/data-store/index-basics.md#index-management-manage-index-templates).

Using the API, the following request creates an index template that is *composed of* the two component templates shown in the [component templates](#component-templates) example.

```console
PUT _index_template/template_1
{
"index_patterns": ["te*", "bar*"],
"template": {
"settings": {
"number_of_shards": 1
},
"mappings": {
"_source": {
"enabled": true
},
"properties": {
"host_name": {
"type": "keyword"
},
"created_at": {
"type": "date",
"format": "EEE MMM dd HH:mm:ss Z yyyy"
}
}
},
"aliases": {
"mydata": { }
}
},
"priority": 501,
"composed_of": ["component_template1", "runtime_component_template"],
"version": 3,
"_meta": {
"description": "my custom"
}
}
```

::::{admonition} Avoid index pattern collisions
:name: avoid-index-pattern-collisions
:::{tip}
The following features can be useful when you're setting up index templates:

* You can test the effect of an index template before putting it into use. Refer to [Simulate multi-component templates](/manage-data/data-store/templates/simulate-multi-component-templates.md) to learn more.
* You can create an index template for a component template that does not yet exist. When doing so, you can use the `ignore_missing_component_templates` configuration option in an index template so that the missing component template is ignored. Refer to [Ignore missing component templates](/manage-data/data-store/templates/ignore-missing-component-templates.md) to learn more.
:::

### Avoid index pattern collisions [avoid-index-pattern-collisions]

{{es}} has built-in index templates, each with a priority of `100`, for the following index patterns:

Expand All @@ -49,15 +102,13 @@ If you use {{fleet}} or {{agent}}, assign your index templates a priority lower
* To avoid naming collisions with built-in and Fleet-managed index templates, avoid using `@` as part of the name of your own index templates.
* Beginning in {{stack}} version 9.1, {{fleet}} uses indices named `fleet-synced-integrations*` for a feature. Avoid using this name to avoid collisions with built-in indices.

::::

## Component templates [component-templates]

A **component template** is a reusable building block, containing mappings, settings, and aliases, used to construct an index template. Component templates are used only to construct index templates and are not applied to indices directly.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Used to construct index templates sound a bit weird when we want to mean that index templates include component templates to reuse their contents.

Suggested change
A **component template** is a reusable building block, containing mappings, settings, and aliases, used to construct an index template. Component templates are used only to construct index templates and are not applied to indices directly.
A **component template** is a reusable building block that defines mappings, settings, and aliases. Component templates are not applied directly to indices, but referenced by index templates and used when determining the final configuration of an index.


## Create index template [create-index-templates]
You can create and manage component templates on the **Index management** page in {{kib}} or by using the [component template](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-component-template) API. For the {{kib}} steps, refer to [Manage component templates](/manage-data/data-store/index-basics.md#index-management-manage-component-templates).

Use the [put index template](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-index-template) and [put component template](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-component-template) APIs to create and update index templates. You can also [manage index templates](/manage-data/data-store/index-basics.md#index-management) from Stack Management in {{kib}}.

The following requests create two component templates.
Using the API, the following request creates the two component templates used in the previous index template example:

```console
PUT _component_template/component_template1
Expand Down Expand Up @@ -91,41 +142,3 @@ PUT _component_template/runtime_component_template
```

1. This component template adds a [runtime field](mapping/map-runtime-field.md) named `day_of_week` to the mappings when a new index matches the template.


The following request creates an index template that is *composed of* these component templates.

```console
PUT _index_template/template_1
{
"index_patterns": ["te*", "bar*"],
"template": {
"settings": {
"number_of_shards": 1
},
"mappings": {
"_source": {
"enabled": true
},
"properties": {
"host_name": {
"type": "keyword"
},
"created_at": {
"type": "date",
"format": "EEE MMM dd HH:mm:ss Z yyyy"
}
}
},
"aliases": {
"mydata": { }
}
},
"priority": 501,
"composed_of": ["component_template1", "runtime_component_template"],
"version": 3,
"_meta": {
"description": "my custom"
}
}
```
122 changes: 1 addition & 121 deletions manage-data/data-store/templates/index-template-management.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,124 +12,4 @@ products:

# Manage index templates [manage-index-templates]

Create, edit, clone, and delete your index templates in the **Index Templates** view. Changes made to an index template do not affect existing indices.

:::{image} /manage-data/images/elasticsearch-reference-management-index-templates.png
:alt: Index templates
:screenshot:
:::

In {{serverless-full}}, the default **logs** template uses the logsDB index mode to create a [logs data stream](../data-streams/logs-data-stream.md).

If you don’t have any templates, you can create one using the **Create template** wizard.

### Try it: Create an index template [_try_it_create_an_index_template]

In this tutorial, you’ll create an index template and use it to configure two new indices.

**Step 1. Add a name and index pattern**

1. In the **Index Templates** view, open the **Create template** wizard.

:::{image} /manage-data/images/elasticsearch-reference-management_index_create_wizard.png
:alt: Create wizard
:screenshot:
:::

2. In the **Name** field, enter `my-index-template`.
3. Set **Index pattern** to `my-index-*` so the template matches any index with that index pattern.
4. Leave **Data Stream**, **Priority**, **Version**, and **_meta field** blank or as-is.

**Step 2. Add settings, mappings, and aliases**

1. Add [component templates](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-component-template) to your index template.

Component templates are pre-configured sets of mappings, index settings, and aliases you can reuse across multiple index templates. Badges indicate whether a component template contains mappings (**M**), index settings (**S**), aliases (**A**), or a combination of the three.

Component templates are optional. For this tutorial, do not add any component templates.

:::{image} /manage-data/images/elasticsearch-reference-management_index_component_template.png
:alt: Component templates page
:screenshot:
:::

2. Define index settings. These are optional. For this tutorial, leave this section blank.
3. Define a mapping that contains an [object](elasticsearch://reference/elasticsearch/mapping-reference/object.md) field named `geo` with a child [`geo_point`](elasticsearch://reference/elasticsearch/mapping-reference/geo-point.md) field named `coordinates`:

:::{image} /manage-data/images/elasticsearch-reference-management-index-templates-mappings.png
:alt: Mapped fields page
:screenshot:
:::

Alternatively, you can click the **Load JSON** link and define the mapping as JSON:

```js
{
"properties": {
"geo": {
"properties": {
"coordinates": {
"type": "geo_point"
}
}
}
}
}
```

You can create additional mapping configurations in the **Dynamic templates** and **Advanced options** tabs. For this tutorial, do not create any additional mappings.

4. Define an alias named `my-index`:

```js
{
"my-index": {}
}
```

5. On the review page, check the summary. If everything looks right, click **Create template**.

**Step 3. Create new indices**

You’re now ready to create new indices using your index template.

1. Index the following documents to create two indices: `my-index-000001` and `my-index-000002`.

```console
POST /my-index-000001/_doc
{
"@timestamp": "2019-05-18T15:57:27.541Z",
"ip": "225.44.217.191",
"extension": "jpg",
"response": "200",
"geo": {
"coordinates": {
"lat": 38.53146222,
"lon": -121.7864906
}
},
"url": "<example-url-01>"
}

POST /my-index-000002/_doc
{
"@timestamp": "2019-05-20T03:44:20.844Z",
"ip": "198.247.165.49",
"extension": "php",
"response": "200",
"geo": {
"coordinates": {
"lat": 37.13189556,
"lon": -76.4929875
}
},
"memory": 241720,
"url": "<example-url-02>"
}
```

2. Use the [get index API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-get) to view the configurations for the new indices. The indices were configured using the index template you created earlier.

```console
GET /my-index-000001,my-index-000002
```
This page has moved. Refer to [Manage index templates](/manage-data/data-store/index-basics.md#index-management-manage-index-templates) for all of the steps to create and manage index templates in {{kib}}.
2 changes: 1 addition & 1 deletion manage-data/toc.yml
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ toc:
children:
- file: data-store/templates/simulate-multi-component-templates.md
- file: data-store/templates/ignore-missing-component-templates.md
- file: data-store/templates/index-template-management.md
- hidden: data-store/templates/index-template-management.md
- file: data-store/aliases.md
- file: data-store/manage-data-from-the-command-line.md
- file: ingest.md
Expand Down
5 changes: 4 additions & 1 deletion redirects.yml
Original file line number Diff line number Diff line change
Expand Up @@ -217,4 +217,7 @@ redirects:

# Related to https://github.com/elastic/docs-content/pull/2097
'explore-analyze/machine-learning/machine-learning-in-kibana/xpack-ml-anomalies.md': 'explore-analyze/machine-learning/anomaly-detection.md'
'explore-analyze/machine-learning/machine-learning-in-kibana/xpack-ml-dfa-analytics.md': 'explore-analyze/machine-learning/data-frame-analytics.md'
'explore-analyze/machine-learning/machine-learning-in-kibana/xpack-ml-dfa-analytics.md': 'explore-analyze/machine-learning/data-frame-analytics.md'

# Related to https://github.com/elastic/docs-content/pull/2010
'manage-data/data-store/templates/index-template-management.md': 'manage-data/data-store/index-basics.md'
Loading