[8.8](backport #35386) Add more detail about generated docs to the contributors guide

docs/devguide/contributing.asciidoc

@@ -193,17 +193,20 @@ Running the testsuite has the following requirements:
 * Docker >= {docker}
 * Docker-compose >= {docker-compose}
 
-For more details check the <<testing>> guide.
-
+For more details, refer to the <<testing>> guide.
 
 [float]
 [[documentation]]
 === Documentation
 
-The documentation for each Beat is located under `{beatname}/docs` and is based
-on asciidoc. After changing the docs, you should verify that the docs are still
-building to avoid breaking the automated docs build. To learn more about
-contributing docs, see the https://github.com/elastic/docs/blob/master/README.asciidoc[Docs HOWTO].
+The main documentation for each Beat is located under `<beatname>/docs` and is
+based on https://docs.asciidoctor.org/asciidoc/latest/[AsciiDoc]. The Beats
+documentation also makes extensive use of conditionals and content reuse to
+ensure consistency and accuracy. Before contributing to the documentation, read
+the following resources:
+
+* https://github.com/elastic/docs/blob/master/README.asciidoc[Docs HOWTO]
+* <<contributing-docs>>
 
 [float]
 [[dependencies]]
@@ -238,3 +241,4 @@ The following packages are required to run `go generate`:
 To keep up to date with changes to the official Beats for community developers,
 follow the developer changelog
 https://github.com/elastic/beats/blob/main/CHANGELOG-developer.next.asciidoc[here].
+

docs/devguide/documentation.asciidoc

@@ -0,0 +1,115 @@
+[[contributing-docs]]
+=== Contributing to the docs
+
+The Beats documentation follows the tagging guidelines described in the
+https://github.com/elastic/docs/blob/master/README.asciidoc[Docs HOWTO]. However
+it extends these capabilities in a couple ways:
+
+* The documentation makes extensive use of
+https://docs.asciidoctor.org/asciidoc/latest/directives/conditionals/[AsciiDoc conditionals]
+to provide content that is reused across multiple books. This means that there
+might not be a single source file for each published HTML page. Some files are
+shared across multiple books, either as complete pages or snippets. For more
+details, refer to <<where-to-find-files>>.
+
+* The documentation includes some files that are generated from YAML source or
+pieced together from content that lives in `_meta` directories under the code
+(for example, the module and exported fields documentation). For more details,
+refer to <<generated-docs>>.
+
+[float]
+[[where-to-find-files]]
+==== Where to find the Beats docs source
+
+Because the Beats documentation makes use of shared content, doc generation
+scripts, and componentization, the source files are located in several places:
+
+|===
+| Documentation | Location of source files
+
+| Main docs for the Beat, including index files
+| `<beatname>/docs`
+
+| Shared docs and Beats Platform Reference
+| `libbeat/docs`
+
+| Processor docs
+| `docs` folders under processors in `libbeat/processors/`,
+`x-pack/<beatname>/processors/`, and `x-pack/libbeat/processors/`
+
+| Output docs
+| `docs` folders under outputs in `libbeat/outputs/`
+
+| Module docs
+| `_meta` folders under modules and datasets in `libbeat/module/`,
+`<beatname>/module/`, and `x-pack/<beatname>/module/`
+|===
+
+The https://github.com/elastic/docs/blob/master/conf.yaml[conf.yaml] file in the
+`docs` repo shows all the resources used to build each book. This file is used
+to drive the classic docs build and is the source of truth for file locations.
+
+TIP: If you can't find the source for a page you want to update, go to the
+published page at www.elastic.co and click the Edit link to navigate to the
+source.
+
+The Beats documentation build also has dependencies on the following files in
+the https://github.com/elastic/docs[docs] repo:
+
+* `shared/versions/stack/<version>.asciidoc`
+* `shared/attributes.asciidoc`
+
+[float]
+[[generated-docs]]
+==== Generated docs
+
+After updating `docs.asciidoc` files in `_meta` directories, you must run the
+doc collector scripts to regenerate the docs.
+
+Make sure you
+<<setting-up-dev-environment,set up your Beats development environment>> and use
+the correct Go version. The Go version is listed in the `version.asciidoc` file
+for the branch you want to update. 
+
+To run the docs collector scripts, change to the beats directory and run:
+
+`make update`
+
+WARNING: The `make update` command overwrites files in the `docs` directories
+**without warning**. If you accidentally update a generated file and run
+`make update`, your changes will be overwritten.
+
+To format your files, you might also need to run this command:
+
+`make fmt`
+
+The make command calls the following scripts to generate the docs:
+
+https://github.com/elastic/beats/blob/main/auditbeat/scripts/docs_collector.py[auditbeat/scripts/docs_collector.py]
+generates:
+
+* `auditbeat/docs/modules_list.asciidoc`
+* `auditbeat/docs/modules/*.asciidoc`
+
+https://github.com/elastic/beats/blob/main/filebeat/scripts/docs_collector.py[filebeat/scripts/docs_collector.py]
+generates:
+
+* `filebeat/docs/modules_list.asciidoc`
+* `filebeat/docs/modules/*.asciidoc`
+
+https://github.com/elastic/beats/blob/main/metricbeat/scripts/mage/docs_collector.go[metricbeat/scripts/mage/docs_collector.go]
+generates:
+
+* `metricbeat/docs/modules_list.asciidoc`
+* `metricbeat/docs/modules/*.asciidoc`
+
+https://github.com/elastic/beats/blob/main/libbeat/scripts/generate_fields_docs.py[libbeat/scripts/generate_fields_docs.py]
+generates
+
+* `auditbeat/docs/fields.asciidoc`
+* `filebeat/docs/fields.asciidoc`
+* `functionbeat/docs/fields.asciidoc`
+* `heartbeat/docs/fields.asciidoc`
+* `metricbeat/docs/fields.asciidoc`
+* `packetbeat/docs/fields.asciidoc`
+* `winlogbeat/docs/fields.asciidoc`

docs/devguide/index.asciidoc

@@ -19,6 +19,8 @@ include::./pull-request-guidelines.asciidoc[]
 
 include::./contributing.asciidoc[]
 
+include::./documentation.asciidoc[]
+
 include::./testing.asciidoc[]
 
 include::{libbeat-dir}/communitybeats.asciidoc[]