contributing-new-log-fields.md

Adding new log fields

Although OpenSearch is a NoSQL database and as-such is "unstructured" and "schemaless," adding a new data source field requires it be defined in several places in order for it to show up and be usable throughout Malcolm. Minimally, this involves three files:

• [arkime/etc/config.ini]({{ site.github.repository_url }}/blob/{{ site.github.build_revision }}/arkime/etc/config.ini) - follow existing examples in the [custom-fields] and [custom-views] sections in order for Arkime to be aware of the new fields
• [arkime/wise/source.zeeklogs.js]({{ site.github.repository_url }}/blob/{{ site.github.build_revision }}/arkime/wise/source.zeeklogs.js) - add new fields to the allFields array for Malcolm to create Arkime value actions for the fields
• [dashboards/templates/composable/component/__(name)__.json]({{ site.github.repository_url }}/blob/{{ site.github.build_revision }}/dashboards/templates/composable/component/) - add new fields to a new composable index template file in this directory and add its name (prefixed with custom_) to the composed_of section of [dashboards/templates/malcolm_template.json]({{ site.github.repository_url }}/blob/{{ site.github.build_revision }}/dashboards/templates/malcolm_template.json) in order for it to be included as part of the arkime_sessions3-* index template used by Arkime and OpenSearch Dashboards in Malcolm

When possible, it is recommended to use (or at least take inspiration from) the Elastic Common Schema (ECS) Reference when deciding how to define new field names.

If new log fields are coming from a Zeek script where they are defined in a record as &log fields, the script [scripts/zeek_script_to_malcolm_boilerplate.py]({{ site.github.repository_url }}/blob/{{ site.github.build_revision }}/scripts/zeek_script_to_malcolm_boilerplate.py) may help by autogenerating the parts of the files mentioned above.