[Onboarding] AWS Service detection re-design for Firehose flow (#192860)

Closes #191989
Closes #190799
Closes #191731

This change implements the design changes done to improve the AWS
service discovery in the new Firehose flow.


[Figma](https://www.figma.com/design/CPhMyRNOgo0wsEiaIMZJ14/Onboarding-Quick-Starts?node-id=454-24601&t=Y7saMXwJfMinghMq-1)


https://github.com/user-attachments/assets/57e0bbb3-1ace-42df-ae6d-5e34d0fd9368

### How To Test

You going to need an AWS account. You can use a personal one or "Elastic
Observability" account which you can access through Okta (type "AWS" in
Okta's search and you should see "AWS - Elastic Observability").

In case you decide to use the shared "Elastic Observability" account,
make sure it does not already have
`Elastic-CloudwatchLogsAndMetricsToFirehose` CloudFormation stack left
from the previous tester. Feel free to delete it if it's there.

1. In AWS account, create a few entities that generate logs and put them
into a CloudWatch log group (see instructions below for a few services).
1. Generate some logs by accessing the entities that you've created and
make sure they appear in CloudWatch (mind that there is a ~1 minute
delay). **If you don't see anything in CloudWatch, there is no point in
proceeding further, make sure to fix your AWS setup before starting the
flow in Kibana.**
1. Go to the serverless Kibana instance deployed from this PR (see the
latest `[Deploy Serverless Kibana] ...` comment by ` kibanamachine`)
1. Add Data → Collect and analyze logs → View AWS Collection → Firehose
quickstart
1. Open the Firehose flow and create CloudFormation stack using one of
the two options.
1. Wait for the stack to finish creating.
1. Generate some some logs by accessing the AWS services you've created.
1. Go back to the Kibana screen, after a minute or so incoming logs
should be detected and corresponding AWS service will be appear.


### Example AWS Services Configs

**Before creating any resources, make sure you're in the same region
(top right corner in AWS Console) you've used while configuring AWS
CLI.**

#### API Gateway

1. [Create an IAM
role](https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-logging.html#set-up-access-logging-permissions)
to grant API Gateway permissions to write into a CloudWatch log groups
1. Copy the ARN of the created role
1. Open "CloudWatch" in AWS and select "Log groups" in the sidebar
1. Create a new log group with default parameters
1. Copy the ARN of the new group
1. Open **API Gateway** in AWS
1. Navigate to "Settings" in the sidebar
1. In the "Logging" section click "Edit" and paste the ARN of the IAM
role you created in step 1. Hit "Save changes"
1. Now go back to "APIs" in the sidebar and click "Create API"
1. In "REST API" click "Build"
1. Select "Example API" and click "Create API"
1. Click on "Deploy API"
1. For "Stage" dropdown select "New stage", give it any name and click
"Deploy"
1. You will now see "Invoke URL", you can use it later to access this
API and generate logs
1. Scroll to "Logs and tracing" section and click "Edit"
1. In the dropdown select "Full request and response logs"
1. Toggle "Custom access logging"
1. Paste the ARN of the CloudWatch log group you've created in step 4.
But make sure to not include ":*" symbols at the end.
1. In the log format input paste [this format from our
docs](https://www.elastic.co/docs/current/integrations/aws/apigateway#data-streams)
and click "Save"
```
{"requestId":"$context.requestId","ip":"$context.identity.sourceIp","caller":"$context.identity.caller","user":"$context.identity.user","requestTime":"$context.requestTime","httpMethod":"$context.httpMethod","resourcePath":"$context.resourcePath","status":"$context.status","protocol":"$context.protocol","responseLength":"$context.responseLength","apiId":"$context.apiId","domainName":"$context.domainName","stage":"$context.stage"}
```
1. Now when you access this API, you should see logs coming into the
CloudWatch group.

#### WAF

**This sets up WAF for an API Gateway, see above if you don't have one
already.**

1. Open WAF in AWS
3. Click "Web ACLs" in the sidebar
4. Click "Create web ACL"
5. Select the region where you've created your API Gateway and give ACL
any name
6. In the "Associated AWS resources" section click "Add AWS resources"
7. Select you API Gateway and click "Add"
8. Click "Next"
9. Create some basic rule, for example to block requests that have a
specific parameter in the URL
10. Click through the other configuration step leaving everything as is
and then finally click "Create web ACL"
11. Select the created ACL and click on the "Logging and metrics" tab
12. Click "Edit" in "Logging" section 
13. Click "Create new" in the "Amazon CloudWatch Logs log group" section
14. Create a new log group. **The log group name should start with
`aws-waf-logs-`**.
15. Select the new group in the dropdown and click "Save"
16. Now you should have logs generated and saved into the log group when
you access your API gateway

#### VPC

1. [Create an IAM
role](https://docs.aws.amazon.com/vpc/latest/tgw/flow-logs-cwl.html#flow-logs-iam)
to write flow logs to CloudWatch log groups.
3. Create and EC2 instance and configure there some HTTP server like
Nginx. Using Docker would probably be the fastest way.
4. Create a CloudWatch log group with default parameters
5. Open "VPC" in AWS and select the VPC where you've created the EC2
instance.
6. Click the "Flow logs" tab and click "Create flow logs"
7. In "Maximum aggregation interval" select 1 minute to see logs faster
8. In "Destination log group" select the log group you've created in
step 3
9. In "IAM role" select the role you've created in step 1
10. Click "Create flow log"
11. Now when you access your EC2 instance, you should see logs in the
CloudWatch log group

x-pack/plugins/observability_solution/observability_onboarding/public/application/pages/firehose.tsx

@@ -19,7 +19,7 @@ export const FirehosePage = () => (
         headlineCopy={i18n.translate(
           'xpack.observability_onboarding.experimentalOnboardingFlow.customHeader.firehose.text',
           {
-            defaultMessage: 'Setting up Amazon Data Firehose',
+            defaultMessage: 'Set up Amazon Data Firehose',
           }
         )}
         captionCopy={i18n.translate(
@@ -29,6 +29,7 @@ export const FirehosePage = () => (
               'This installation is tailored for setting up Firehose in your Observability project with minimal configuration.',
           }
         )}
+        isTechnicalPreview={true}
       />
     }
   >

x-pack/plugins/observability_solution/observability_onboarding/public/application/quickstart_flows/firehose/auto_refresh_callout.tsx

@@ -0,0 +1,54 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+import React from 'react';
+import { i18n } from '@kbn/i18n';
+import {
+  EuiFlexGroup,
+  EuiFlexItem,
+  EuiIcon,
+  EuiText,
+  useEuiTheme,
+  useGeneratedHtmlId,
+} from '@elastic/eui';
+import { css } from '@emotion/react';
+import { HAS_DATA_FETCH_INTERVAL } from './utils';
+
+export function AutoRefreshCallout() {
+  const { euiTheme } = useEuiTheme();
+  const messageId = useGeneratedHtmlId();
+
+  return (
+    <EuiFlexGroup>
+      <EuiFlexItem
+        role="status"
+        aria-labelledby={messageId}
+        grow={false}
+        css={css`
+          background-color: ${euiTheme.colors.lightestShade};
+          padding: ${euiTheme.size.m} ${euiTheme.size.base};
+          border-radius: ${euiTheme.border.radius.medium};
+        `}
+      >
+        <EuiFlexGroup gutterSize="s" alignItems="center">
+          <EuiIcon type="timeRefresh" size="m" />
+          <EuiText size="s">
+            <p id={messageId}>
+              {i18n.translate(
+                'xpack.observability_onboarding.firehosePanel.autorefreshCalloutLabel',
+                {
+                  defaultMessage: 'Auto-refreshing every {intervalSeconds} s',
+                  values: { intervalSeconds: Math.round(HAS_DATA_FETCH_INTERVAL / 1000) },
+                }
+              )}
+            </p>
+          </EuiText>
+        </EuiFlexGroup>
+      </EuiFlexItem>
+    </EuiFlexGroup>
+  );
+}

x-pack/plugins/observability_solution/observability_onboarding/public/application/quickstart_flows/firehose/create_stack_command_snippet.tsx

@@ -5,7 +5,6 @@
  * 2.0.
  */
 
-import React from 'react';
 import {
   EuiAccordion,
   EuiCodeBlock,
@@ -14,19 +13,20 @@ import {
   EuiText,
   useGeneratedHtmlId,
 } from '@elastic/eui';
-import { FormattedMessage } from '@kbn/i18n-react';
 import { i18n } from '@kbn/i18n';
+import { FormattedMessage } from '@kbn/i18n-react';
+import React from 'react';
 import {
   FIREHOSE_CLOUDFORMATION_STACK_NAME,
   FIREHOSE_LOGS_STREAM_NAME,
   FIREHOSE_METRICS_STREAM_NAME,
 } from '../../../../common/aws_firehose';
 import { CopyToClipboardButton } from '../shared/copy_to_clipboard_button';
+import { DownloadTemplateCallout } from './download_template_callout';
 import { buildCreateStackCommand, buildStackStatusCommand } from './utils';
 
 interface Props {
   encodedApiKey: string;
-  onboardingId: string;
   elasticsearchUrl: string;
   templateUrl: string;
   isCopyPrimaryAction: boolean;
@@ -57,7 +57,7 @@ export function CreateStackCommandSnippet({
         <p>
           <FormattedMessage
             id="xpack.observability_onboarding.firehosePanel.createFirehoseStreamDescription"
-            defaultMessage="Run the command bellow in your terminal where you have {awsCLIInstallGuideLink} configured. The command will create a CloudFormation stack that includes a Firehose delivery, backup S3 bucket, CloudWatch subscription filter and metrics stream along with required IAM roles."
+            defaultMessage="Run the command bellow in your terminal where you have {awsCLIInstallGuideLink} configured. The command will create a CloudFormation stack from our template that includes a Firehose delivery, backup S3 bucket, CloudWatch subscription filter and metrics stream along with required IAM roles."
             values={{
               awsCLIInstallGuideLink: (
                 <EuiLink
@@ -75,6 +75,10 @@ export function CreateStackCommandSnippet({
             }}
           />
         </p>
+
+        <p>
+          <DownloadTemplateCallout />
+        </p>
       </EuiText>
 
       <EuiSpacer />
@@ -94,7 +98,15 @@ export function CreateStackCommandSnippet({
 
       <EuiSpacer />
 
-      <EuiAccordion id={stackStatusAccordionId} buttonContent="Check stack status">
+      <EuiAccordion
+        id={stackStatusAccordionId}
+        buttonContent={i18n.translate(
+          'xpack.observability_onboarding.firehosePanel.stackStatusAccordionButtonLabel',
+          {
+            defaultMessage: 'Check status of the CloudFormation stack',
+          }
+        )}
+      >
         <EuiSpacer size="xs" />
         <EuiCodeBlock language="text" paddingSize="m" fontSize="m" isCopyable>
           {stackStatusCommand}

x-pack/plugins/observability_solution/observability_onboarding/public/application/quickstart_flows/firehose/create_stack_in_aws_console.tsx

@@ -0,0 +1,73 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+import { EuiButton, EuiSpacer, EuiText } from '@elastic/eui';
+import { i18n } from '@kbn/i18n';
+import { FormattedMessage } from '@kbn/i18n-react';
+import React from 'react';
+import {
+  FIREHOSE_CLOUDFORMATION_STACK_NAME,
+  FIREHOSE_LOGS_STREAM_NAME,
+  FIREHOSE_METRICS_STREAM_NAME,
+} from '../../../../common/aws_firehose';
+import { DownloadTemplateCallout } from './download_template_callout';
+import { buildCreateStackAWSConsoleURL } from './utils';
+
+interface Props {
+  encodedApiKey: string;
+  elasticsearchUrl: string;
+  templateUrl: string;
+  isPrimaryAction: boolean;
+}
+
+export function CreateStackInAWSConsole({
+  encodedApiKey,
+  elasticsearchUrl,
+  templateUrl,
+  isPrimaryAction,
+}: Props) {
+  const awsConsoleURL = buildCreateStackAWSConsoleURL({
+    templateUrl,
+    stackName: FIREHOSE_CLOUDFORMATION_STACK_NAME,
+    logsStreamName: FIREHOSE_LOGS_STREAM_NAME,
+    metricsStreamName: FIREHOSE_METRICS_STREAM_NAME,
+    elasticsearchUrl,
+    encodedApiKey,
+  });
+
+  return (
+    <>
+      <EuiText>
+        <p>
+          <FormattedMessage
+            id="xpack.observability_onboarding.firehosePanel.createFirehoseStreamInAWSConsoleDescription"
+            defaultMessage="Click the button below to create a CloudFormation stack from our template. The stack will include a Firehose delivery stream, backup S3 bucket, CloudWatch subscription filter, metrics stream, and necessary IAM roles. Keep this page open, and return once you've submitted the form in AWS Console"
+          />
+        </p>
+        <p>
+          <DownloadTemplateCallout />
+        </p>
+      </EuiText>
+
+      <EuiSpacer size="m" />
+
+      <EuiButton
+        data-test-subj="observabilityOnboardingCreateStackInAWSConsoleButton"
+        href={awsConsoleURL}
+        target="_blank"
+        iconSide="right"
+        iconType="popout"
+        fill={isPrimaryAction}
+      >
+        {i18n.translate(
+          'xpack.observability_onboarding.createStackInAWSConsole.createFirehoseStreamInAWSConsoleButtonLabel',
+          { defaultMessage: 'Create Firehose Stream in AWS' }
+        )}
+      </EuiButton>
+    </>
+  );
+}

x-pack/plugins/observability_solution/observability_onboarding/public/application/quickstart_flows/firehose/download_template_callout.tsx

@@ -0,0 +1,35 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+import React from 'react';
+import { EuiLink } from '@elastic/eui';
+import { i18n } from '@kbn/i18n';
+import { FormattedMessage } from '@kbn/i18n-react';
+import { FIREHOSE_CLOUDFORMATION_TEMPLATE_URL } from '../../../../common/aws_firehose';
+
+export function DownloadTemplateCallout() {
+  return (
+    <FormattedMessage
+      id="xpack.observability_onboarding.firehosePanel.downloadTemplateDescription"
+      defaultMessage="If needed, you can {downloadLink} to use it as part of an existing IaC setup."
+      values={{
+        downloadLink: (
+          <EuiLink
+            data-test-subj="observabilityOnboardingFirehosePanelDownloadCloudFormationTemplateLink"
+            href={FIREHOSE_CLOUDFORMATION_TEMPLATE_URL}
+            download={true}
+          >
+            {i18n.translate(
+              'xpack.observability_onboarding.firehosePanel.downloadCloudFormationTemplateButtonLabel',
+              { defaultMessage: 'download and modify the CloudFormation template' }
+            )}
+          </EuiLink>
+        ),
+      }}
+    />
+  );
+}