[ML] Integration with Kibana Spaces

[jgowdyelastic]

Meta issue to track the work needed to make ML completely space aware.
Target release 7.11

• Update kibana capabilities to include license checks on initialisation.
  • Populate ml_capabilities with kibana capabilities rather than es privileges. ([ML] Moving to kibana capabilities #64057) 7.8
  • Add license check to capabilities switcher in kibana setup. ([ML] Moving to kibana capabilities #64057) 7.8
  • Remove license check from ml_capabilities endpoint. ([ML] Moving to kibana capabilities #64057) 7.8
  • Add access checks to all endpoints using the privilege api tag list (7.8) ([ML] Adding endpoint capability checks #64662)
  • Add access checks to all functions shared via our setup contract. ([ML] Adding capabilities checks to shared functions #70069) (7.9)
• Change all integrated plugins so they are using our functions or endpoints
  • Add a new job/group ID parameter to the mlAnomalySearch ML plugin shared function. This will be used to check which jobs the user has access to in the Space. ([ML] Adding space aware jobs #77916) 7.11
  • All es queries to our indices should go through a shared function, e.g. mlAnomalySearch. (7.9)
    • APM ([APM] [ML] Use ML's mlAnomalySearch for all searches on the anomaly index #67860), ([APM] Replace direct anomaly searches with the mlAnomalySearch client API #69092)
    • Logs UI ([Logs UI] [ML] Use ML's mlAnomalySearch for all searches on the anomaly index  #67861)
• Space aware jobs
  • Create anomaly detector shared saved objects. (7.x)
  • Create data frame analytics shared saved objects. (7.x)
  • Create filter shared saved objects (not yet needed)
  • Create calendar shared saved objects (not yet needed)
  • ML plugin UI enhancements to use shared saved objects (7.x)
    • Add job id lookup to all endpoints - are the job ids passed to the endpoint available in this space? (7.x)
  • Management plugin enhancements to use shared saved objects. This page will need to allow management of all jobs across all spaces. (7.x)
  • Add initialisation checks for non-space aware jobs. On kibana start up, add spaceless jobs to the default space. (7.x)
• Rely on Kibana privileges to control access to ML functionality
  • All calls to ML endpoints and searches of ML hidden indices are made using the Kibana system user instead of the logged-in user (after checking Kibana privileges) (7.9) ([ML] Changing all calls to ML endpoints to use internal user #70487)
  • Credentials of the logged-in user are passed as secondary credentials on these calls (7.9) (([ML] Changing all calls to ML endpoints to use internal user #70487)
  • Security team to add support for users to be granted access to ML both via (legacy) machine_learning_admin and machine_learning_user roles, and via the (new) ML all / read feature privileges Feature Controls - support migrating away from reserved privileges #68379 (7.9)
  • Edit ML server/plugin.ts registerFeature block to switch from reserved privileges to all and read blocks in privileges block (7.9) ([ML] Changing all calls to ML endpoints to use internal user #70487)
• Supporting Elasticsearch changes
  • Use secondary auth credentials for datafeeds and data frame analytics searches of input data. ([ML] prefer secondary authorization header for data[feed|frame] authz elasticsearch#54121)
  • Give the kibana_system role access to all ML endpoints and ML hidden indices (Add ML admin permissions to the kibana_system role elasticsearch#58061)

[elasticmachine]

Pinging @elastic/ml-ui (:ml)

[weltenwort]

Let me know when you need the Logs UI to be migrated to new APIs.

[peteharverson]

Closing issue as work required to make ML jobs space aware has been completed for 7.11.

@jgowdyelastic the few remaining low priority follow-up tasks have been moved to #82415.