Integration tests for new platform

[kimjoar]

The new platform is currently only running unit tests. We need to look at getting a good setup for integration tests.

• Pull out proc_runner from x-pack-kibana into kibana Pull out proc_runner from x-pack-kibana into kibana #17120
• Use proc runner from @kbn/dev-utils in x-pack-kibana https://github.com/elastic/x-pack-kibana/pull/4824
• Better integration test runner Functional test setup with kbn-test package #18568
  • can chain multiple configs in succession
  • use @kbn/es to quickly start and stop servers
• Run Kibana Functional Tests on CI using kbn-test [Functional Tests] Use @kbn/test on Kibana CI #18967
• Http server integration tests [Tests] Add http integration test setup #19261

At this point we could work to merge up the new platform (because http server would be tested)

• Write integration tests for new platform
• Change kibana to have a single set of functional test scripts that by default executes OSS & x-pack tests. It should fall back to oss-only tests if a flag is passed.

Improvements to @kbn/test:

• Option to run kibana from build for CI #19125 Fix accidentally bumped feature--run kibana from build on x-pack CI

Future:

• Decide future of FTR--maybe the FTR top level just runs from kbn/test. Maybe kbn/test is renamed to kbn/functional_tests
• [Tests] Enhance functional_tests script to accept args to run just servers or just tests #17818 - Allow functional_tests to run servers + tests or just servers from single script
• Presets for starting es and Kibana with saml, with security, other presets

Historical:

(now outdated) Some ideas in a gist

[rhoboat]

The purpose is to establish confidence in the new platform http server changes. Before merging that new platform http server, create these tests, for each type of kibana setup:

• with and without base-path set
• with and without https

Just need one test case for each thing, like a smoke test.

[rhoboat]

- integration-tests
  - http
    - with-basepath
      - config
         - kibana/kibana.yml (plus other files)
         - elasticsearch/elasticsearch.yml (plus other files)
    - without-basepath
    - with-ssl

running:

node scripts/ftr --config=integration_tests/http/with-basepath

make the test runner able to start both kibana and es with what's inside config

[rhoboat]

Running:

node scripts/ftr --config=integration_tests/config.js  # config.js requires all top level index.js files

• each index.js references a test suite or bunch of test suites
• each test suite has a beforeAll that starts up kibana and elasticsearch with the ymls in the local directory

[kimjoar]

The approach we discussed for this was starting with extracting the proc_runner from x-pack into a package in Kibana (maybe ./packages/kbn-proc-runner?)