[Long-Running PR] Kubernetes Asset and Topology POC

[jasonrhodes]

Replaces #15, #18, and #21

This PR tracks all of the work merged for the asset inventory POC.

Other PRs for this work should be made against this branch and not against main.

Running this PR

• Checkout the asset-inventory branch of my kibana fork.
• Run yarn kbn bootstrap to get the needed dependencies installed
• Run Elasticsearch -- I typically use yarn es snapshot --license=trial --ssl
• Set the necessary environment variables (see below)
• Run Kibana, pointed at your ES cluster. I use a separate config file for this:

$ cd kibana
$ cat config/kibana.es-snapshot.yml
server.basePath: '/local'

# config for "yarn es snapshot --ssl"
elasticsearch.hosts: ['https://localhost:9200']
elasticsearch.username: kibana_system
elasticsearch.password: changeme
elasticsearch.ssl.verificationMode: none

• Then I run Kibana with this command:

$ yarn start --config=./config/kibana.es-snapshot.yml

Once running, Kibana makes the following available:

• A general asset exploration page at http://localhost:5601/local/app/asset-inventory
• A specific K8s cluster listing POC at http://localhost:5601/local/app/asset-inventory/k8s/clusters
• REST API endpoints, documented below

Required environment vars

These are the env vars that this Kibana branch looks for in order to (a) collect asset data, and (b) write that data to ES. I use a .env file (now added to Kibana's .gitignore) and the autoenv tool, but you can set these however you like so long as they are set in the context of the Kibana running process(es).

# Asset Datastream
ASSETS_DATASTREAM="assets"

ASSETS_ELASTICSEARCH_HOST="https://localhost:9200"
ASSETS_ELASTICSEARCH_USERNAME="elastic"
ASSETS_ELASTICSEARCH_PASSWORD="changeme"
ASSETS_ELASTICSEARCH_REJECT_UNAUTHORIZED_TLS=0

# For AWS collection (note, REGIONS not REGION)
ASSETS_AWS_REGIONS="us-east-1,us-east-2"

# For Azure collection (subscription ID, not name)
ASSETS_AZURE_SUBSCRIPTION_ID="your-azure-subscription-id"

REST API

All available endpoints can be seen in code in the plugin file: https://github.com/jasonrhodes/kibana/blob/asset-inventory/x-pack/plugins/asset_inventory/server/plugin.ts

Important Notes:

• Don't forget to include the base path for your instance, if you're using one.
• Kibana APIs require a kbn-xsrf header that can be any string you like.

GET /api/asset-inventory/ping

Verifies that the API is up and running.

GET /api/asset-inventory

Returns assets for the asset inventory explore page

GET /api/asset-inventory/k8s/clusters

Returns list of known K8s clusters

POST /api/asset-inventory/collect

With body (or "all" for all of the collections):

{
  "types": ["k8s", "aws-k8s", "azure-k8s"]
}