Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[entityanalytics_okta] Initial Release for the Entity Analytics Okta #6911

Merged
merged 4 commits into from
Jul 18, 2023

Conversation

brijesh-elastic
Copy link
Contributor

What does this PR do?

  • Generated the skeleton of the Entity Analytics Okta integration package.
  • Added a data stream.
  • Added a data collection logic for the data stream.
  • Added the ingest pipeline for the data stream.
  • Mapped fields according to the ECS schema and added Fields metadata in the appropriate yml files.
  • Added dashboard and visualizations.
  • Added a pipeline test for the data stream.
  • Added a system test cases for the data stream.

Integration release checklist

This checklist is intended for integrations maintainers to ensure consistency
when creating or updating a Package, Module or Dataset for an Integration.

All changes

  • Change follows the contributing guidelines
  • Supported versions of the monitoring target is documented
  • Supported operating systems are documented (if applicable)
  • Integration or System tests exist
  • Documentation exists
  • Fields follow ECS and naming conventions
  • At least a manual test with ES / Kibana / Agent has been performed.
  • Required Kibana version set to: ^8.9.0

New Package

  • Screenshot of the "Add Integration" page on Fleet added

Dashboards changes

  • Dashboards exists
  • Screenshots added or updated
  • Datastream filters added to visualizations

Log dataset changes

  • Pipeline tests exist (if applicable)
  • Generated output for at least 1 log file exists
  • Sample event (sample_event.json) exists

How to test this PR locally

  • Clone integrations repo.
  • Install elastic package locally.
  • Start elastic stack using elastic-package.
  • Move to integrations/packages/entityanalytics_okta directory.
  • Run the following command to run tests.

elastic-package test

Screenshots

Integration Page
Overview page

Automated Test

2023/07/07 16:49:35 DEBUG Enable verbose logging
Run pipeline tests for the package
--- Test results for package: entityanalytics_okta - START ---
╭──────────────────────┬─────────────┬───────────┬────────────────┬────────┬──────────────╮
│ PACKAGE              │ DATA STREAM │ TEST TYPE │ TEST NAME      │ RESULT │ TIME ELAPSED │
├──────────────────────┼─────────────┼───────────┼────────────────┼────────┼──────────────┤
│ entityanalytics_okta │ user        │ pipeline  │ test-user.json │ PASS   │  12.462813ms │
╰──────────────────────┴─────────────┴───────────┴────────────────┴────────┴──────────────╯
--- Test results for package: entityanalytics_okta - END   ---
Done

2023/07/07 16:48:02 DEBUG Enable verbose logging
Run system tests for the package
2023/07/07 16:48:02 DEBUG Running system tests for data stream
2023/07/07 16:48:02 DEBUG running test with configuration 'default'
2023/07/07 16:48:02 DEBUG setting up service...
2023/07/07 16:48:02 DEBUG setting up service using Docker Compose service deployer
2023/07/07 16:48:02 DEBUG running command: /usr/local/bin/docker-compose version --short
2023/07/07 16:48:03 DEBUG Determined Docker Compose version: 1.29.2, the tool will use Compose V1
2023/07/07 16:48:03 DEBUG output command: /usr/bin/docker network inspect elastic-package-stack_default
2023/07/07 16:48:03 DEBUG running command: /usr/local/bin/docker-compose -f /root/integrations/packages/entityanalytics_okta/_dev/deploy/docker/docker-compose.yml -p elastic-package-service up --build -d
Creating network "elastic-package-service_default" with the default driver
Creating elastic-package-service_entityanalytics_okta_1 ... done
2023/07/07 16:48:04 DEBUG running command: /usr/local/bin/docker-compose -f /root/integrations/packages/entityanalytics_okta/_dev/deploy/docker/docker-compose.yml -p elastic-package-service ps -q
2023/07/07 16:48:05 DEBUG Wait for healthy containers: 401e476f2726b3c1e686ad2b47622f4c811187c216ee182bb0048336fb801aed
2023/07/07 16:48:05 DEBUG output command: /usr/bin/docker inspect 401e476f2726b3c1e686ad2b47622f4c811187c216ee182bb0048336fb801aed
2023/07/07 16:48:05 DEBUG Container status: {"Config":{"Image":"docker.elastic.co/observability/stream:v0.10.0","Labels":{"BRANCH_NAME":"v0.10.0","GIT_SHA":"2a076c9b1acdf1c35b5f5c2f8c23904c7c2c441a","GO_VERSION":"1.19.5","TIMESTAMP":"2023-01-30_11:29","com.docker.compose.config-hash":"cdd887d3c90592da5f89631af4604b333c1035986647afaa25239c160a8311be","com.docker.compose.container-number":"1","com.docker.compose.oneoff":"False","com.docker.compose.project":"elastic-package-service","com.docker.compose.project.config_files":"/root/integrations/packages/entityanalytics_okta/_dev/deploy/docker/docker-compose.yml","com.docker.compose.project.working_dir":"/root/integrations/packages/entityanalytics_okta/_dev/deploy/docker","com.docker.compose.service":"entityanalytics_okta","com.docker.compose.version":"1.29.2"}},"ID":"401e476f2726b3c1e686ad2b47622f4c811187c216ee182bb0048336fb801aed","State":{"Status":"running","ExitCode":0,"Health":null}}
2023/07/07 16:48:05 DEBUG run command: /usr/bin/docker network connect elastic-package-stack_default elastic-package-service_entityanalytics_okta_1
2023/07/07 16:48:05 DEBUG adding service container elastic-package-service_entityanalytics_okta_1 internal ports to context
2023/07/07 16:48:05 DEBUG running command: /usr/local/bin/docker-compose -f /root/integrations/packages/entityanalytics_okta/_dev/deploy/docker/docker-compose.yml -p elastic-package-service config
2023/07/07 16:48:06 DEBUG Installing package...
2023/07/07 16:48:06 DEBUG GET https://127.0.0.1:5601/api/status
2023/07/07 16:48:06 DEBUG Build directory: /root/integrations/build/packages/entityanalytics_okta/0.1.0
2023/07/07 16:48:06 DEBUG Clear target directory (path: /root/integrations/build/packages/entityanalytics_okta/0.1.0)
2023/07/07 16:48:06 DEBUG Copy package content (source: /root/integrations/packages/entityanalytics_okta)
2023/07/07 16:48:06 DEBUG Copy license file if needed
2023/07/07 16:48:06  INFO License text found in "/root/integrations/LICENSE.txt" will be included in package
2023/07/07 16:48:06 DEBUG Encode dashboards
2023/07/07 16:48:06 DEBUG Resolve external fields
2023/07/07 16:48:06 DEBUG Package has external dependencies defined
2023/07/07 16:48:06 DEBUG data_stream/user/fields/base-fields.yml: source file hasn't been changed
2023/07/07 16:48:06 DEBUG data_stream/user/fields/beats.yml: source file hasn't been changed
2023/07/07 16:48:06 DEBUG data_stream/user/fields/fields.yml: source file hasn't been changed
2023/07/07 16:48:06  INFO Import ECS mappings into the built package (technical preview)
2023/07/07 16:48:06 DEBUG Build zipped package
2023/07/07 16:48:06 DEBUG Compress using archiver.Zip (destination: /root/integrations/build/packages/entityanalytics_okta-0.1.0.zip)
2023/07/07 16:48:06 DEBUG Create work directory for archiving: /tmp/elastic-package-1001619349/entityanalytics_okta-0.1.0
2023/07/07 16:48:06 DEBUG Skip validation of the built .zip package
2023/07/07 16:48:06 DEBUG POST https://127.0.0.1:5601/api/fleet/epm/packages
2023/07/07 16:48:08 DEBUG creating test policy...
2023/07/07 16:48:08 DEBUG POST https://127.0.0.1:5601/api/fleet/agent_policies
2023/07/07 16:48:12 DEBUG adding package data stream to test policy...
2023/07/07 16:48:12 DEBUG POST https://127.0.0.1:5601/api/fleet/package_policies
2023/07/07 16:48:15 DEBUG deleting old data in data stream...
2023/07/07 16:48:15 DEBUG found 0 hits in logs-entityanalytics_okta.user-ep data stream: index_not_found_exception: no such index [logs-entityanalytics_okta.user-ep] Status=404
2023/07/07 16:48:15 DEBUG GET https://127.0.0.1:5601/api/fleet/agents
2023/07/07 16:48:15 DEBUG filter agents using criteria: NamePrefix=docker-fleet-agent
2023/07/07 16:48:15 DEBUG found 1 enrolled agent(s)
2023/07/07 16:48:15 DEBUG GET https://127.0.0.1:5601/api/fleet/agent_policies/f38003b0-1cb7-11ee-9f7e-77b04ebefa97
2023/07/07 16:48:15 DEBUG assigning package data stream to agent...
2023/07/07 16:48:15 DEBUG PUT https://127.0.0.1:5601/api/fleet/agents/8e33d03e-a022-4cf9-babb-1a8b586334d4/reassign
2023/07/07 16:48:17 DEBUG GET https://127.0.0.1:5601/api/fleet/agents/8e33d03e-a022-4cf9-babb-1a8b586334d4
2023/07/07 16:48:17 DEBUG Agent data: {"id":"8e33d03e-a022-4cf9-babb-1a8b586334d4","policy_id":"f38003b0-1cb7-11ee-9f7e-77b04ebefa97","local_metadata":{"host":{"name":"docker-fleet-agent"}}}
2023/07/07 16:48:17 DEBUG Wait until the policy (ID: f38003b0-1cb7-11ee-9f7e-77b04ebefa97, revision: 2) is assigned to the agent (ID: 8e33d03e-a022-4cf9-babb-1a8b586334d4)...
2023/07/07 16:48:19 DEBUG GET https://127.0.0.1:5601/api/fleet/agents/8e33d03e-a022-4cf9-babb-1a8b586334d4
2023/07/07 16:48:19 DEBUG Agent data: {"id":"8e33d03e-a022-4cf9-babb-1a8b586334d4","policy_id":"f38003b0-1cb7-11ee-9f7e-77b04ebefa97","local_metadata":{"host":{"name":"docker-fleet-agent"}}}
2023/07/07 16:48:19 DEBUG Wait until the policy (ID: f38003b0-1cb7-11ee-9f7e-77b04ebefa97, revision: 2) is assigned to the agent (ID: 8e33d03e-a022-4cf9-babb-1a8b586334d4)...
2023/07/07 16:48:21 DEBUG GET https://127.0.0.1:5601/api/fleet/agents/8e33d03e-a022-4cf9-babb-1a8b586334d4
2023/07/07 16:48:21 DEBUG Agent data: {"id":"8e33d03e-a022-4cf9-babb-1a8b586334d4","policy_id":"f38003b0-1cb7-11ee-9f7e-77b04ebefa97","local_metadata":{"host":{"name":"docker-fleet-agent"}}}
2023/07/07 16:48:21 DEBUG Wait until the policy (ID: f38003b0-1cb7-11ee-9f7e-77b04ebefa97, revision: 2) is assigned to the agent (ID: 8e33d03e-a022-4cf9-babb-1a8b586334d4)...
2023/07/07 16:48:23 DEBUG GET https://127.0.0.1:5601/api/fleet/agents/8e33d03e-a022-4cf9-babb-1a8b586334d4
2023/07/07 16:48:23 DEBUG Agent data: {"id":"8e33d03e-a022-4cf9-babb-1a8b586334d4","policy_id":"f38003b0-1cb7-11ee-9f7e-77b04ebefa97","policy_revision":2,"local_metadata":{"host":{"name":"docker-fleet-agent"}}}
2023/07/07 16:48:23 DEBUG Policy revision assigned to the agent (ID: 8e33d03e-a022-4cf9-babb-1a8b586334d4)...
2023/07/07 16:48:23 DEBUG checking for expected data in data stream...
2023/07/07 16:48:23 DEBUG found 0 hits in logs-entityanalytics_okta.user-ep data stream: index_not_found_exception: no such index [logs-entityanalytics_okta.user-ep] Status=404
2023/07/07 16:48:24 DEBUG found 0 hits in logs-entityanalytics_okta.user-ep data stream: index_not_found_exception: no such index [logs-entityanalytics_okta.user-ep] Status=404
2023/07/07 16:48:25 DEBUG found 0 hits in logs-entityanalytics_okta.user-ep data stream: index_not_found_exception: no such index [logs-entityanalytics_okta.user-ep] Status=404
2023/07/07 16:48:26 DEBUG found 0 hits in logs-entityanalytics_okta.user-ep data stream
2023/07/07 16:48:27 DEBUG found 3 hits in logs-entityanalytics_okta.user-ep data stream
2023/07/07 16:48:27 DEBUG check whether or not synthetics is enabled (component template logs-entityanalytics_okta.user@package)...
2023/07/07 16:48:27 DEBUG data stream logs-entityanalytics_okta.user-ep has synthetics enabled: false
2023/07/07 16:48:28 DEBUG reassigning original policy back to agent...
2023/07/07 16:48:28 DEBUG PUT https://127.0.0.1:5601/api/fleet/agents/8e33d03e-a022-4cf9-babb-1a8b586334d4/reassign
2023/07/07 16:48:28 DEBUG GET https://127.0.0.1:5601/api/fleet/agents/8e33d03e-a022-4cf9-babb-1a8b586334d4
2023/07/07 16:48:28 DEBUG Agent data: {"id":"8e33d03e-a022-4cf9-babb-1a8b586334d4","policy_id":"elastic-agent-managed-ep","local_metadata":{"host":{"name":"docker-fleet-agent"}}}
2023/07/07 16:48:28 DEBUG Wait until the policy (ID: elastic-agent-managed-ep, revision: 3) is assigned to the agent (ID: 8e33d03e-a022-4cf9-babb-1a8b586334d4)...
2023/07/07 16:48:30 DEBUG GET https://127.0.0.1:5601/api/fleet/agents/8e33d03e-a022-4cf9-babb-1a8b586334d4
2023/07/07 16:48:30 DEBUG Agent data: {"id":"8e33d03e-a022-4cf9-babb-1a8b586334d4","policy_id":"elastic-agent-managed-ep","local_metadata":{"host":{"name":"docker-fleet-agent"}}}
2023/07/07 16:48:30 DEBUG Wait until the policy (ID: elastic-agent-managed-ep, revision: 3) is assigned to the agent (ID: 8e33d03e-a022-4cf9-babb-1a8b586334d4)...
2023/07/07 16:48:32 DEBUG GET https://127.0.0.1:5601/api/fleet/agents/8e33d03e-a022-4cf9-babb-1a8b586334d4
2023/07/07 16:48:32 DEBUG Agent data: {"id":"8e33d03e-a022-4cf9-babb-1a8b586334d4","policy_id":"elastic-agent-managed-ep","local_metadata":{"host":{"name":"docker-fleet-agent"}}}
2023/07/07 16:48:32 DEBUG Wait until the policy (ID: elastic-agent-managed-ep, revision: 3) is assigned to the agent (ID: 8e33d03e-a022-4cf9-babb-1a8b586334d4)...
2023/07/07 16:48:34 DEBUG GET https://127.0.0.1:5601/api/fleet/agents/8e33d03e-a022-4cf9-babb-1a8b586334d4
2023/07/07 16:48:34 DEBUG Agent data: {"id":"8e33d03e-a022-4cf9-babb-1a8b586334d4","policy_id":"elastic-agent-managed-ep","policy_revision":3,"local_metadata":{"host":{"name":"docker-fleet-agent"}}}
2023/07/07 16:48:34 DEBUG Policy revision assigned to the agent (ID: 8e33d03e-a022-4cf9-babb-1a8b586334d4)...
2023/07/07 16:48:34 DEBUG deleting test policy...
2023/07/07 16:48:34 DEBUG POST https://127.0.0.1:5601/api/fleet/agent_policies/delete
2023/07/07 16:48:37 DEBUG DELETE https://127.0.0.1:5601/api/fleet/epm/packages/entityanalytics_okta-0.1.0
2023/07/07 16:48:39 DEBUG tearing down service...
2023/07/07 16:48:39 DEBUG tearing down service using Docker Compose runner
2023/07/07 16:48:39 DEBUG running command: /usr/local/bin/docker-compose version --short
2023/07/07 16:48:40 DEBUG Determined Docker Compose version: 1.29.2, the tool will use Compose V1
2023/07/07 16:48:40 DEBUG running command: /usr/local/bin/docker-compose -f /root/integrations/packages/entityanalytics_okta/_dev/deploy/docker/docker-compose.yml -p elastic-package-service logs
2023/07/07 16:48:40  INFO Write container logs to file: /root/integrations/build/container-logs/entityanalytics_okta-1688728720906242006.log
2023/07/07 16:48:40 DEBUG running command: /usr/local/bin/docker-compose -f /root/integrations/packages/entityanalytics_okta/_dev/deploy/docker/docker-compose.yml -p elastic-package-service down --volumes
Stopping elastic-package-service_entityanalytics_okta_1 ... done
Removing elastic-package-service_entityanalytics_okta_1 ... done
Removing network elastic-package-service_default
2023/07/07 16:48:42 DEBUG deleting data in data stream...
2023/07/07 16:48:42 DEBUG Dump Elastic stack data
2023/07/07 16:48:42 DEBUG Dump stack logs (location: /tmp/test-system-3018837560)
2023/07/07 16:48:42 DEBUG Dump stack logs for elasticsearch
2023/07/07 16:48:42 DEBUG running command: /usr/local/bin/docker-compose version --short
2023/07/07 16:48:42 DEBUG Determined Docker Compose version: 1.29.2, the tool will use Compose V1
2023/07/07 16:48:42 DEBUG running command: /usr/local/bin/docker-compose -f /root/.elastic-package/profiles/default/stack/snapshot.yml -p elastic-package-stack logs elasticsearch
2023/07/07 16:48:43 DEBUG Dump stack logs for elastic-agent
2023/07/07 16:48:43 DEBUG running command: /usr/local/bin/docker-compose version --short
2023/07/07 16:48:44 DEBUG Determined Docker Compose version: 1.29.2, the tool will use Compose V1
2023/07/07 16:48:44 DEBUG running command: /usr/local/bin/docker-compose -f /root/.elastic-package/profiles/default/stack/snapshot.yml -p elastic-package-stack logs elastic-agent
2023/07/07 16:48:44 DEBUG running command: /usr/local/bin/docker-compose version --short
2023/07/07 16:48:45 DEBUG Determined Docker Compose version: 1.29.2, the tool will use Compose V1
2023/07/07 16:48:45 DEBUG run command: /usr/bin/docker cp elastic-package-stack_elastic-agent_1:/usr/share/elastic-agent/state/data/logs/ /tmp/test-system-3018837560/logs/elastic-agent-internal
2023/07/07 16:48:45 DEBUG Dump stack logs for fleet-server
2023/07/07 16:48:45 DEBUG running command: /usr/local/bin/docker-compose version --short
2023/07/07 16:48:46 DEBUG Determined Docker Compose version: 1.29.2, the tool will use Compose V1
2023/07/07 16:48:46 DEBUG running command: /usr/local/bin/docker-compose -f /root/.elastic-package/profiles/default/stack/snapshot.yml -p elastic-package-stack logs fleet-server
2023/07/07 16:48:47 DEBUG running command: /usr/local/bin/docker-compose version --short
2023/07/07 16:48:47 DEBUG Determined Docker Compose version: 1.29.2, the tool will use Compose V1
2023/07/07 16:48:47 DEBUG run command: /usr/bin/docker cp elastic-package-stack_fleet-server_1:/usr/share/elastic-agent/state/data/logs/ /tmp/test-system-3018837560/logs/fleet-server-internal
2023/07/07 16:48:47 DEBUG Dump stack logs for kibana
2023/07/07 16:48:47 DEBUG running command: /usr/local/bin/docker-compose version --short
2023/07/07 16:48:48 DEBUG Determined Docker Compose version: 1.29.2, the tool will use Compose V1
2023/07/07 16:48:48 DEBUG running command: /usr/local/bin/docker-compose -f /root/.elastic-package/profiles/default/stack/snapshot.yml -p elastic-package-stack logs kibana
2023/07/07 16:48:49 DEBUG Dump stack logs for package-registry
2023/07/07 16:48:49 DEBUG running command: /usr/local/bin/docker-compose version --short
2023/07/07 16:48:49 DEBUG Determined Docker Compose version: 1.29.2, the tool will use Compose V1
2023/07/07 16:48:49 DEBUG running command: /usr/local/bin/docker-compose -f /root/.elastic-package/profiles/default/stack/snapshot.yml -p elastic-package-stack logs package-registry
2023/07/07 16:48:50 DEBUG skipped malformed docker-compose log line: Attaching to elastic-package-stack_elastic-agent_1
--- Test results for package: entityanalytics_okta - START ---
╭──────────────────────┬─────────────┬───────────┬───────────┬────────┬───────────────╮
│ PACKAGE              │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │  TIME ELAPSED │
├──────────────────────┼─────────────┼───────────┼───────────┼────────┼───────────────┤
│ entityanalytics_okta │ user        │ system    │ default   │ PASS   │ 25.068587576s │
╰──────────────────────┴─────────────┴───────────┴───────────┴────────┴───────────────╯
--- Test results for package: entityanalytics_okta - END   ---
Done

2023/07/07 16:49:57 DEBUG Enable verbose logging
Run static tests for the package
--- Test results for package: entityanalytics_okta - START ---
╭──────────────────────┬─────────────┬───────────┬──────────────────────────┬────────┬──────────────╮
│ PACKAGE              │ DATA STREAM │ TEST TYPE │ TEST NAME                │ RESULT │ TIME ELAPSED │
├──────────────────────┼─────────────┼───────────┼──────────────────────────┼────────┼──────────────┤
│ entityanalytics_okta │ user        │ static    │ Verify sample_event.json │ PASS   │ 137.744969ms │
╰──────────────────────┴─────────────┴───────────┴──────────────────────────┴────────┴──────────────╯
--- Test results for package: entityanalytics_okta - END   ---
Done

2023/07/07 16:50:23 DEBUG Enable verbose logging
Run asset tests for the package
2023/07/07 16:50:23 DEBUG installing package...
2023/07/07 16:50:23 DEBUG GET https://127.0.0.1:5601/api/status
2023/07/07 16:50:24 DEBUG Build directory: /root/integrations/build/packages/entityanalytics_okta/0.1.0
2023/07/07 16:50:24 DEBUG Clear target directory (path: /root/integrations/build/packages/entityanalytics_okta/0.1.0)
2023/07/07 16:50:24 DEBUG Copy package content (source: /root/integrations/packages/entityanalytics_okta)
2023/07/07 16:50:24 DEBUG Copy license file if needed
2023/07/07 16:50:24  INFO License text found in "/root/integrations/LICENSE.txt" will be included in package
2023/07/07 16:50:24 DEBUG Encode dashboards
2023/07/07 16:50:24 DEBUG Resolve external fields
2023/07/07 16:50:24 DEBUG Package has external dependencies defined
2023/07/07 16:50:24 DEBUG data_stream/user/fields/base-fields.yml: source file hasn't been changed
2023/07/07 16:50:24 DEBUG data_stream/user/fields/beats.yml: source file hasn't been changed
2023/07/07 16:50:24 DEBUG data_stream/user/fields/fields.yml: source file hasn't been changed
2023/07/07 16:50:24  INFO Import ECS mappings into the built package (technical preview)
2023/07/07 16:50:24 DEBUG Build zipped package
2023/07/07 16:50:24 DEBUG Compress using archiver.Zip (destination: /root/integrations/build/packages/entityanalytics_okta-0.1.0.zip)
2023/07/07 16:50:24 DEBUG Create work directory for archiving: /tmp/elastic-package-2748993626/entityanalytics_okta-0.1.0
2023/07/07 16:50:24 DEBUG Skip validation of the built .zip package
2023/07/07 16:50:24 DEBUG POST https://127.0.0.1:5601/api/fleet/epm/packages
2023/07/07 16:50:25 DEBUG removing package...
2023/07/07 16:50:25 DEBUG DELETE https://127.0.0.1:5601/api/fleet/epm/packages/entityanalytics_okta-0.1.0
--- Test results for package: entityanalytics_okta - START ---
╭──────────────────────┬─────────────┬───────────┬───────────────────────────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE              │ DATA STREAM │ TEST TYPE │ TEST NAME                                                                     │ RESULT │ TIME ELAPSED │
├──────────────────────┼─────────────┼───────────┼───────────────────────────────────────────────────────────────────────────────┼────────┼──────────────┤
│ entityanalytics_okta │             │ asset     │ dashboard entityanalytics_okta-e5242a60-0f35-11ee-8319-1d33c4a0c7ae is loaded │ PASS   │      3.377µs │
│ entityanalytics_okta │             │ asset     │ search entityanalytics_okta-d4f05110-0f7a-11ee-8319-1d33c4a0c7ae is loaded    │ PASS   │        754ns │
│ entityanalytics_okta │ user        │ asset     │ index_template logs-entityanalytics_okta.user is loaded                       │ PASS   │        731ns │
│ entityanalytics_okta │ user        │ asset     │ ingest_pipeline logs-entityanalytics_okta.user-0.1.0 is loaded                │ PASS   │        943ns │
╰──────────────────────┴─────────────┴───────────┴───────────────────────────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: entityanalytics_okta - END   ---
Done

@elasticmachine
Copy link

elasticmachine commented Jul 11, 2023

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2023-07-18T12:27:03.673+0000

  • Duration: 14 min 23 sec

Test stats 🧪

Test Results
Failed 0
Passed 7
Skipped 0
Total 7

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

@elasticmachine
Copy link

elasticmachine commented Jul 11, 2023

🌐 Coverage report

Name Metrics % (covered/total) Diff
Packages 100.0% (1/1) 💚
Files 100.0% (1/1) 💚
Classes 100.0% (1/1) 💚
Methods 100.0% (9/9) 💚 33.333
Lines 90.925% (521/573) 👎 -9.075
Conditionals 100.0% (0/0) 💚

@jamiehynds jamiehynds requested a review from a team July 11, 2023 15:42
responses:
- status_code: 200
body: |
[{"id":"00ub0oNGTSWTBKOLGLNR","status":"ACTIVE","created":"2013-06-24T16:39:18.000Z","activated":"2013-06-24T16:39:19.000Z","statusChanged":"2013-06-24T16:39:19.000Z","lastLogin":"2013-06-24T17:39:19.000Z","lastUpdated":"2013-07-02T21:36:25.344Z","passwordChanged":"2013-07-02T21:36:25.344Z","profile":{"firstName":"Isaac","lastName":"Brock","email":"isaac.brock@example.com","login":"isaac.brock@example.com","mobilePhone":"555-415-1337"},"credentials":{"password":{},"recovery_question":{"question":"Who's a major player in the cowboy scene?"},"provider":{"type":"OKTA","name":"OKTA"}}}]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the password field, please add subfields, "value": "tlpWENT2m" and to recovery_question please add "answer": "Annie Oakley". These will be stripped by the provider so the change will test that this is the case and we are not retaining secrets.

Suggested change
[{"id":"00ub0oNGTSWTBKOLGLNR","status":"ACTIVE","created":"2013-06-24T16:39:18.000Z","activated":"2013-06-24T16:39:19.000Z","statusChanged":"2013-06-24T16:39:19.000Z","lastLogin":"2013-06-24T17:39:19.000Z","lastUpdated":"2013-07-02T21:36:25.344Z","passwordChanged":"2013-07-02T21:36:25.344Z","profile":{"firstName":"Isaac","lastName":"Brock","email":"isaac.brock@example.com","login":"isaac.brock@example.com","mobilePhone":"555-415-1337"},"credentials":{"password":{},"recovery_question":{"question":"Who's a major player in the cowboy scene?"},"provider":{"type":"OKTA","name":"OKTA"}}}]
[{"id":"00ub0oNGTSWTBKOLGLNR","status":"ACTIVE","created":"2013-06-24T16:39:18.000Z","activated":"2013-06-24T16:39:19.000Z","statusChanged":"2013-06-24T16:39:19.000Z","lastLogin":"2013-06-24T17:39:19.000Z","lastUpdated":"2013-07-02T21:36:25.344Z","passwordChanged":"2013-07-02T21:36:25.344Z","profile":{"firstName":"Isaac","lastName":"Brock","email":"isaac.brock@example.com","login":"isaac.brock@example.com","mobilePhone":"555-415-1337"},"credentials":{"password":{"value": "tlpWENT2m"},"recovery_question":{"question":"Who's a major player in the cowboy scene?","answer": "Annie Oakley"},"provider":{"type":"OKTA","name":"OKTA"}}}]

Comment on lines 39 to 55
"credentials": {
"password": {
"hash": {
"algorithm": "SHA-1",
"salt": "UEO3wsAsgzQ=",
"saltOrder": "POSTFIX",
"value": "xjrauE6J6kbjcvMjWSSc+PsBBls="
}
},
"recovery_question": {
"question": "Who's a major player in the cowboy scene?"
},
"provider": {
"type": "OKTA",
"name": "OKTA"
}
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The provider does not fill password or recovery_question. If you are finding that it does, this is a defect, so please don't have pipeline tests that include these fields (or if you do, ensure that the secrets are deleted from the document before ingest).

https://github.com/elastic/beats/blob/3de3d53e29437c80ea8a4d54060564ab44fd2360/x-pack/filebeat/input/entityanalytics/provider/okta/internal/okta/okta.go#L92-L96

Looking at sample_event.json, this appears to be true, happily.

- name: credentials
type: group
fields:
- name: password
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This group should never be present.

type: keyword
- name: type
type: keyword
- name: recovery_question
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This group should never be present.

- name: sync_interval
type: text
title: Sync Interval
description: How often full synchronizations should occur. Must be greater than Update Interval. Expected value is a duration string (15m, 1h, 1m30, etc), defaults to 24h. NOTE:- Supported units for this parameter are h/m/s.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
description: How often full synchronizations should occur. Must be greater than Update Interval. Expected value is a duration string (15m, 1h, 1m30, etc), defaults to 24h. NOTE:- Supported units for this parameter are h/m/s.
description: How often full synchronizations should occur. Must be greater than Update Interval. Expected value is a duration string (15m, 1h, 1m30, etc), defaults to 24h. Supported units for this parameter are h/m/s.

- name: update_interval
type: text
title: Update Interval
description: How often incremental updates should occur. Must be less than Sync Interval. Expected value is a duration string (15m, 1h, 1m30, etc), defaults to 15m. NOTE:- Supported units for this parameter are h/m/s.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
description: How often incremental updates should occur. Must be less than Sync Interval. Expected value is a duration string (15m, 1h, 1m30, etc), defaults to 15m. NOTE:- Supported units for this parameter are h/m/s.
description: How often incremental updates should occur. Must be less than Sync Interval. Expected value is a duration string (15m, 1h, 1m30, etc), defaults to 15m. Supported units for this parameter are h/m/s.

- name: http_client_timeout
type: text
title: HTTP Client Timeout
description: "Duration before declaring that the HTTP client connection has timed out. NOTE: Valid time units are ns, us, ms, s, m, h."
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
description: "Duration before declaring that the HTTP client connection has timed out. NOTE: Valid time units are ns, us, ms, s, m, h."
description: "Duration before declaring that the HTTP client connection has timed out. Valid time units are ns, us, ms, s, m, h."

Copy link
Contributor

@efd6 efd6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM after clarifications.

@@ -0,0 +1,234 @@
- name: asset
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the asset group formally approved?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We've added the asset and user group objects to the fields.yml because these fields are coming to the upcoming ECS Schema. (@jamiehynds told us to follow the PR and map this field to the upcoming proposed ECS fields.)

I've one concern: if in the future (let's say 8.9) this proposed field comes in ECS, then it may be a conflict that we've added that in fields.yml and also that it will be added by import mappings (from the build.yml). (Again, that scenario may arise when we have the same fields in agent.yml and ecs.yml)

Copy link
Contributor

@efd6 efd6 Jul 17, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. I had not seen that ECS RFC. When the fields are in we can clean up any issues that arise.

- name: type
type: object
description: user type that determines the schema for the user's profile.
- name: labels
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same query for labels.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For labels, it is a necessary field that I have to add in fields.yml in order to execute a successful system test. If we don't provide that, then it results in labels.identity_source is undefined error. even if the labels field is added by the import_mapping feature.

So, to resolve this issue, I've added labels.identity_source in fields.yml

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since labels is an ECS field, please add this to the ecs.yml, its totally fine to define ECS fields in ecs.yml even though dynamic mappings are enabled, especially if they are not covered by it. I will take a note to update the dynamic template.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@P1llus, should we add upcoming proposed ECS fields in ecs.yml too ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@brijesh-elastic sure that would be great!

@P1llus P1llus merged commit 64f5d35 into elastic:main Jul 18, 2023
1 check passed
@elasticmachine
Copy link

Package entityanalytics_okta - 0.1.0 containing this change is available at https://epr.elastic.co/search?package=entityanalytics_okta

gizas pushed a commit that referenced this pull request Sep 5, 2023
…6911)

* Initial Release for the Entity Analytics Okta

* Update the changelog entry

* Resolve the comments

* Resolve the comments
@andrewkroh andrewkroh added the Integration:entityanalytics_okta Okta Entity Analytics label Jul 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants