diff --git a/docs/_data/toc.yml b/docs/_data/toc.yml index ead74d72c8..f5fe7b43c3 100644 --- a/docs/_data/toc.yml +++ b/docs/_data/toc.yml @@ -1,4 +1,5 @@ Overview: +- Samples Tutorials: - Create a Virtual Assistant - Customize a Virtual Assistant @@ -9,7 +10,9 @@ Tutorials: How To: - Virtual Assistant - Skills +- Samples Reference: - Virtual Assistant - Skills -- Analytics \ No newline at end of file +- Analytics +- Samples \ No newline at end of file diff --git a/docs/_docs/howto/samples/enterprisenotifications.md b/docs/_docs/howto/samples/enterprisenotifications.md new file mode 100644 index 0000000000..1294712e73 --- /dev/null +++ b/docs/_docs/howto/samples/enterprisenotifications.md @@ -0,0 +1,129 @@ +--- +category: How To +subcategory: Samples +title: Set up Enterprise Notifications for a Virtual Assistant +description: Steps for configuring the Enterprise Notifications sample +order: 1 +--- + +# {{ page.title }} +{:.no_toc} + +## In this how-to +{:.no_toc} + +* +{:toc} + +## Prerequisites + +1. [Create a Virtual Assistant]({{ site.baseurl }}/tutorials/csharp/create-assistant/1_intro/) to setup your Virtual Assistant environment. + +1. Manually deploy the following Azure resources: + + - [Create](https://ms.portal.azure.com/#create/Microsoft.EventHub) a [Azure Event Hub](https://azure.microsoft.com/en-us/services/event-hubs/) resource + - [Create](https://ms.portal.azure.com/#create/Microsoft.FunctionApp) a [Azure Function](https://azure.microsoft.com/en-us/services/functions/) resource + - [Create](https://ms.portal.azure.com/#create/Microsoft.NotificationHub) a [Azure Notification Hub](https://azure.microsoft.com/en-us/services/notification-hubs/) resource + - [Create](https://ms.portal.azure.com/#create/Microsoft.DocumentDB) a [Azure Cosmos DB](https://azure.microsoft.com/en-us/services/cosmos-db/) resource + +1. Install the [Bot Framework Emulator](https://aka.ms/botframeworkemulator) to use in testing. + +## Event Producer + +This sample includes an example [Event Producer]({{site.repo}}/samples/EnterpriseNotification/EventProducer) console application that sends an Event to the Event Hub for processing simulating creation of a notification. + +- Update `appSettings.json` with the `EventHubName` and `EventHubConnectionString` which you can find by going to your EventHub resource, creating an instance and then a `Shared Access Policy` + +### Azure Function - Event Handler +This sample includes an example [EventHandler Azure Function]({{site.repo}}/Samples/EnterpriseNotification/EventHandler) which is triggered by Event delivery and handles Event processing. + + +1. Update [Function1.cs]({{site.repo}}/samples/EnterpriseNotification/EventHandler/Function1.cs) and change the `EventHubTrigger` to reflect your Event Hub name. + ```csharp + public static async Task Run([EventHubTrigger("YourEventHubName", Connection = "EventHubConnection")] EventData[] events, ILogger log)` + ``` +2. The Azure Functions blade in the Azure Portal provides a wide range of routes to deploy the provided code to your newly created Azure Function including Visual Studio and VSCode. Follow this to deploy the sample EventHandler project. +3. Once deployed, go to the Azure Function in Azure and choose Configuration. +4. Create a new `ConnectionString` called `EventHubConnection` property and provide the same EventHub connection string as in the previous section. +5. In the `Application Settings` section create the following settings which are used bvy the Event Handler. + - `DirectLineSecret` - Located within the Channels section of your Azure Bot Service registration. Required to communicate with your assistant and send events. + - `DocumentDbEndpointUrl` - Located within the CosmoDB Azure Portal blade. Required to access the User Preference store. + - `DocumentDbPrimaryKey`- Located within the CosmoDB Azure Portal blade. + +## Virtual Assistant + +### ProactiveState Middleware + +In order to be able to deliver messages to a conversation the end user must already have had an interaction with the assistant. As part of this interaction a `ConversationReference` needs to be persisted and used to resume the conversation. + +We provide a middleware component to perform this ConversationReference storage which can be found in the Bot.Builder.Solutions package. + +1. Add this line to your `Startup.cs` to register the proactive state. +```csharp + services.AddSingleton(); +``` +2. Within your `DefaultAdapter.cs` add this line to the constructor +```csharp + ProactiveState proactiveState +``` +3. Within your `DefaultAdapter.cs` add this line: +```csharp + Use(new ProactiveStateMiddleware(proactiveState)); +``` + +### Event Handling + +The following code handles the `BroadcastEvent` event type sent by the Azure function and is added to the Event Handling code. Within Virtual Assistant this is handled by `OnEventAsync` within MainDialog.cs. + +The `_proactiveStateAccessor` is the state that contains a mapping between UserId and previously persisted conversation. It retrieves the proactive state from a store previously saved by enabling the `ProactiveStateMiddleware`. + +Within `MainDialog.cs` add the following changes: + +1. Add this variable to your `MainDialog` class. + ```csharp + private IStatePropertyAccessor _proactiveStateAccessor; + ``` +2. Add this line to the constructor + ```csharp + ProactiveState proactiveState + ``` + and initialise the state in the constructor + ```csharp + _proactiveStateAccessor = proactiveState.CreateProperty(nameof(ProactiveModel)); + ``` +3. Add this event handler to your `OnEventAsync` handler to handle the `BroadcastEvent` + + ```csharp + case "BroadcastEvent": + var eventData = JsonConvert.DeserializeObject(dc.Context.Activity.Value.ToString()); + + var proactiveModel = await _proactiveStateAccessor.GetAsync(dc.Context, () => new ProactiveModel()); + + var conversationReference = proactiveModel[MD5Util.ComputeHash(eventData.UserId)].Conversation; + await dc.Context.Adapter.ContinueConversationAsync(_appCredentials.MicrosoftAppId, conversationReference, ContinueConversationCallback(dc.Context, eventData.Message), cancellationToken); + break; + ``` + +## Testing and Validation + +Now events can be sent to a user through your Virtual Assistant in an active conversation. + +### Bot Framework Emulator + +Event generation must generate Events with the same `UserId` as the Emulator is using so the existing conversation can be matched and notifications can be delivered. + +![UserId Settings]({{ site.baseurl }}/assets/images/emulator-userid.png) + +1. In the **Bot Framework Emulator**, navigate to **Settings** and provide a guid to represent a simulated user ID. This will ensure any conversations with your Assistant use the same user ID. + +1. Begin a conversation with your Assistant to create a proactive state record for future user. + +## Event Producer + +1. Copy the user ID used in the **Bot Framework Emulator** into the `SendMessagesToEventHub` method within `Program.cs` of the **Event Producer**. +This ensures any notifications sent are routed to your active conversation. + + +1. Run the **Event Producer** to generate a message and observe that the message is shown within your session. + +![Enterprise Notification Demo]({{ site.baseurl }}/assets/images/enterprisenotification-demo.png) \ No newline at end of file diff --git a/docs/_docs/howto/samples/vaclient_android.md b/docs/_docs/howto/samples/vaclient_android.md new file mode 100644 index 0000000000..f454eb01d4 --- /dev/null +++ b/docs/_docs/howto/samples/vaclient_android.md @@ -0,0 +1,99 @@ +--- +category: How To +subcategory: Samples +title: Use the Virtual Assistant Client (Android) +description: Steps for using the Virtual Assistant Client on Android +order: 1 +--- + +# {{ page.title }} +{:.no_toc} + +## In this how-to +{:.no_toc} + +* +{:toc} + +## Prerequisites + +1. Install [Android Studio](https://developer.android.com/studio/) on your PC. + +1. [Create a Virtual Assistant]({{ site.baseurl }}/tutorials/csharp/create-assistant/1_intro/) to setup your Virtual Assistant environment. + +1. [Enable speech]({{ site.baseurl }}/tutorials/enable-speech/1_intro) on your new Virtual Assistant, which enables you to retrieve a + - [Microsoft Speech Cognitive Service subscription key]({{ site.baseurl }}/tutorials/enable-speech/2_create_speech_instance/) + - [Add the Direct Line Speech channel to your Assistant]({{ site.baseurl }}/tutorials/enable-speech/3_add_speech_channel/) + +## Overview +![Virtual Assistant Client (Android) overview diagram]({{ site.baseurl }}/assets/images/virtualassistantclient-android-overview.png) + +A user can interact with their Assistant using the **Virtual Assistant Client app** via widgets on the home screen or the main UI of the app. +These bot responses can optionally be broadcast to an **Event Companion app** that does't need to implement the Speech SDK. + +## Building the project + +### Provide credentials to the application + +The following configuration values must be supplied to `DefaultConfiguration.java` to connect to the Assistant via the Direct Line Speech channel: +* `SPEECH_SERVICE_SUBSCRIPTION_KEY` +* `DIRECT_LINE_SPEECH_SECRET_KEY` +* `USER_FROM_ID` + +```java +public class DefaultConfiguration { + + // Replace below with your own subscription key + public static final String SPEECH_SERVICE_SUBSCRIPTION_KEY = "YOUR_KEY_HERE";//TODO + + public static final String DIRECT_LINE_SPEECH_SECRET_KEY = "YOUR_DIRECTLINE_SPEECH_KEY_HERE";//TODO + + public static final String SPEECH_SERVICE_SUBSCRIPTION_KEY_REGION = "westus2";//TODO + + public static final String USER_NAME = "User"; + public static final String USER_FROM_ID = "YOUR_USER_FROM_ID_HERE";//TODO + + public static final String LOCALE = "en-us"; + + public static final String KEYWORD = "computer"; + + // please note that the default colors are read from /res/values/colors.xml +} +``` +**Note:** that there are two versions, one for debug and one for release build flavors. + +The USER_FROM_ID is a unique identifier for all messages generated by the user, this is typically combined with [Linked Accounts]({{ site.baseurl }}/howto/virtual-assistant/linkedaccounts/). + +### Deploy +1. Select the desired build flavor (debug or release) and ensure credentials are set for the desired build flavor +2. Deploy to emulator or device + +## Using the Project +### Permissions + - **Record Audio** + - Required for the user to make voice requests to a bot. With this a user can only use the keyboard. + - **Fine Location** + - Allow your Assistant to receive the `VA.Location` event with GPS coordinates to utilize location-based skills like Point of Interest. + +## Interacting with your Assistant +### Conversation view +![Conversation view]({{ site.baseurl }}/assets/images/virtualassistantclient-android-fullconversationview.png) + +To demonstrate a chat app experience with an Assistant, the main screen shows user/bot interactions with a threaded conversation. +Trigger a conversation by selecting either: +* Mic button to speak to the bot +* Keyboard button to type to the bot + +### Native view +![Native view]({{ site.baseurl }}/assets/images/virtualassistantclient-android-widgetview.png) + +Using widgets, you can demonstrate an Assistant having a native chat experience on a device. + +### Menu +Swipe from the left to access the menu, providing the following functionality: +* Restart conversation +* Settings + +## Features + +Learn more about the [Virtual Assistant Client (Android) comprehensive feature set]({{ site.baseurl }}/reference/samples/vaclient_android/) \ No newline at end of file diff --git a/docs/_docs/overview/samples/enterprisenotifications.md b/docs/_docs/overview/samples/enterprisenotifications.md new file mode 100644 index 0000000000..aa4e58adad --- /dev/null +++ b/docs/_docs/overview/samples/enterprisenotifications.md @@ -0,0 +1,100 @@ +--- +category: Overview +subcategory: Samples +title: Enterprise Notifications +order: 4 +--- + +# {{ page.title }} +## Contents +{:.no_toc} + +* +{:toc} + +## Intro + +There are many scenarios where an enterprise-focused Assistant needs to push notifications or messages to employees. +These messages may need to be delivered on a variety of [channels](https://docs.microsoft.com/en-us/azure/bot-service/bot-service-manage-channels?view=azure-bot-service-4.0) and customized by each employees. +It's important to consider the range of channels you wish to offer to customers and whether they provide a persistent conversation over time and the channel itself supports proactive message delivery. Microsoft Teams is an example of a persistent channel enabling conversations to occur over a longer period of time and across a range of devices. This contrasts with WebChat which is only available for the life of the browser window. + +In addition to conversational canvases mobile devices are another key end user channel and these same notifications/messages should be delivered as appropriate to these devices. + +We provide this sample to demonstrate how to build a notification/broadcasting scenario using a Virtual Assistant and various Azure resources. +Each customer scenario will vary significantly, so this is an MVP (minimum viable product) to get started with. + +## Sample Capabilities + +This sample demonstrates the following capabilities: + +1. A console application that shows how a supporting system can create an event for a specific event and send for processing. +2. An Azure function that handles events and routes them to a User via a Bot (Virtual Assistant). In this same handler mobile application push notification can be added as additional custom steps. +3. A User Preference store that enables user preferences for notification delivery to be stored and is used by the Azure Function. +3. The extensions to a Bot required to display an event to a user and also store ConversationReference objects enabling proactive message delivery. + +## Sample Architecture + +![Enterprise Notifications sample architecture]({{ site.baseurl }}/assets/images/enterprisenotifications-architecture.png) + +### Event Producer + +Azure Functions are used to collect events from upstream systems and convert them into a canonical event schema before handing over to the Event Hub for centralized handling. In this sample, we simulate this event generation functionality for ease of testing by using the console application located [here](/samples/EnterpriseNotification/EventProducer). + +### Azure Event Hub + +In this sample, the Azure Event Hub is the centralized service that manages events gathered from different parts of the system and sent through the Azure Function aforementioned. For any event to reach an end user, it has to flow into the Azure Event Hub first. + +### Azure Functions - Event Handler + +After an event is posted to the Azure Event Hub, an Azure Function service is triggered to process them. The background to the use of Azure Functions is as follows: + +- Azure Functions natively support triggers against a variety of Azure services, Event Hub trigger is one of these. +- Azure Functions scales against Event Hub services by managing locks on partitions of the Azure Event Hub internally as part of the framework. + +#### Notification Handler (Trigger) + +The triggering element of the Azure function is handled as part of the [EventHandler](/samples/EnterpriseNotification/EventHandler). The `Run` method within [Function1.cs]({{site.repo}}/samples/EnterpriseNotification/EventHandler/Function1.cs) is automatically invoked once an event is available. + +#### Notification Handler (Run) + +Following the trigger the following steps are performed as part of the same [EventHandler]({{site.repo}}/samples/EnterpriseNotification/EventHandler) example: + +- Unpack the event +- Read from a UserPreference store to check user's profile settings +- If the user has 'SendNotificationToMobileDevice' flag to be true, then send a notification to user's mobile device with the event content. +- If the user has 'SendNotificationToConversation' flag to be true, then send a message to the bot with the event content. + +This sample uses CosmosDB as the UserPreference store but can be modified to reflect an existing store you may already have in place. The code will check if there's a record for the particular user. If not, it will then add a record with default settings of both destinations set to true. + +This sample doesn't include the implementation for sending a notification to mobile devices as this requires additional configuration. You can refer to [this documentation](https://docs.microsoft.com/en-us/azure/notification-hubs/notification-hubs-aspnet-backend-ios-apple-apns-notification) for more information. + +The message the Event Handler sends to the bot is an Activity of type `event`, with the name `BroadcastEvent` and value is set to the data received rom the Event Hub. + +### Notification Hub + +[Notification Hubs](https://azure.microsoft.com/en-us/services/notification-hubs) provide the capability to delivery notifications to end user devices. Please refer to [this documentation](https://docs.microsoft.com/en-us/azure/notification-hubs/notification-hubs-aspnet-backend-ios-apple-apns-notification) for additional steps to perform this integration as needed. + +### Virtual Assistant + +The assistant is responsible for surfacing the message received from the Event Handler back to the user. An example project is located [here]({{site.repo}}/samples/EnterpriseNotification/VirtualAssistant) which has a small number of extensions compared to a normal Virtual Assistant. + +### Adaptive Cards and Web Dashboards + +When Notification Handler handles events emitted from Azure Event Hub, it can persist the events into a user data store. + +This would enable user/system administrator to look at the events later on from a Web Dashboard where AdaptiveCards and other Web components can be used to render them to provide companion experiences to the assistant. This part is not included in the sample implementation at the time. + + +## Next Steps + +
+
+
+

Set up Enterprise Notifications for a Virtual Assistant

+

Steps for configuring the Enterprise Notifications sample.

+
+ +
+
diff --git a/docs/_docs/reference/samples/vaclient_android.md b/docs/_docs/reference/samples/vaclient_android.md new file mode 100644 index 0000000000..1770313480 --- /dev/null +++ b/docs/_docs/reference/samples/vaclient_android.md @@ -0,0 +1,74 @@ +--- +category: Reference +subcategory: Samples +title: Virtual Assistant Client (Android) comprehensive feature set +description: Detailed information on the features of the Virtual Assistant Client on Android +order: 1 +--- + + +# {{ page.title }} +{:.no_toc} + +## In this reference +{:.no_toc} + +* +{:toc} + +## Activity UI for testing and demonstrating a bot on the Direct Line Speech channel + +* Interaction + * User may send requests to bot via voice or text + * Responses from the bot can render text responses and Adaptive Card attachments (single and carousel) + * Bot responses are audible (controlled by 'Media' volume bar) + * Volume rocket adjusts 'Media' volume bar without having to select it + * Adaptive Cards amy be clickable to send a response to bot + * The listening mode adapts to a bot's input hints and user's input method + * User may restart a new conversation with the bot + * User may use "Show Assistant Settings" as a shortcut to the default assist app menu + +* Appearance + * Newest response is rendered at the bottom of the conversation history (scrolled automatically) + * User may elect to show a threaded conversation (by default only bot responses shown) + * User may adjust dark/light color scheme + +## Settings UI to configure a bot endpoint and experience + +* A user can configure: + * Speech Service subscription key + * Speech Service subscrition key region + * Direct Line Speech secret key + * User From.Id value + * Locale + * Chat History line-count + * Timezone + * Bot Speech Bubble Background Color + * User Speech Bubble Background Color + * Bot Text Color + * User Text Color + * GPS Sent on (read-only) + * Send GPS Location (resent `VA.Location` event activity with GPS coordinates) + + +## Widgets for demonstrating a native chat experience + + * Users may add resizable widgets to their homescreen by long-pressing and scrolling to the Virtual Assistant Client app + 1. Microphone + * Trigger the service to listen to user requests + 1. User utterance (request) + * Echo the user's request in text format + 1. Bot response (response) + * Show the text response from the bot - does not show Adaptive Cards + + +## Always-on background service + +* Stores state client side +* Interface between clients (widget and Activity UI) and service +* Interface with plug-in apps (via AIDL and broadcasts) +* Sends GPS location periodically +* Receives bot responses and plays audio without showing the Activity UI +* Open default apps (navigation and music) as necessary to fulfill user's request + * Navigation: attempts to open Waze first. If unavailable, opens via Google Maps. + * Music: opens Spotify. \ No newline at end of file diff --git a/docs/_docs/tutorials/enable-speech/2_create_speech_instance.md b/docs/_docs/tutorials/enable-speech/2_create_speech_instance.md index c65c675f24..6ef0467e0c 100644 --- a/docs/_docs/tutorials/enable-speech/2_create_speech_instance.md +++ b/docs/_docs/tutorials/enable-speech/2_create_speech_instance.md @@ -7,9 +7,9 @@ order: 2 # Tutorial: Enable Speech for your Assistant -## Create a Microsoft Speech nstance +## Create a Microsoft Speech Azure resource -The first step is to create a Microsoft Speech instance to perform the Speech-To-Text and Text-To-Speech capabilities for your assistant. +The first step is to create a Microsoft Speech Cognitive Services Azure resource to perform the Speech-To-Text and Text-To-Speech capabilities for your assistant. - Select an Azure region. Direct Line Speech Channel is a preview service limited to [these Azure regions](https://docs.microsoft.com/en-us/azure/cognitive-services/speech-service/regions#voice-first-virtual-assistants). For best performance (reduced round-trip time) deploy your Virtual Assistant bot and Direct Line Speech channel to the same Azure region, and one that is closest to you. To help you decide, look up exact [geographical location](https://azure.microsoft.com/en-us/global-infrastructure/locations/) for each Azure region. - Create a Microsoft Speech Cognitive Service instance in your Azure Subscription using the [Azure Portal](https://ms.portal.azure.com/#create/Microsoft.CognitiveServicesSpeechServices). In the *Location* field specify the selected Azure region based on the above. diff --git a/docs/_docs/tutorials/enable-speech/3_add_speech_channel.md b/docs/_docs/tutorials/enable-speech/3_add_speech_channel.md index bc9a5c67a2..2727185296 100644 --- a/docs/_docs/tutorials/enable-speech/3_add_speech_channel.md +++ b/docs/_docs/tutorials/enable-speech/3_add_speech_channel.md @@ -1,13 +1,13 @@ --- category: Tutorials subcategory: Enable Speech -title: Add the speech channel +title: Add the Direct Line Speech channel order: 3 --- # Tutorial: Enable Speech for your Assistant -## Add the Speech Channel to your Assistant +## Add the Direct Line Speech channel to your Assistant The first step is to add the Direct-Line Speech Channel to your deployed Assistant. diff --git a/docs/assets/images/android_project_overview.jpg b/docs/assets/images/android_project_overview.jpg deleted file mode 100644 index 98f02c0594..0000000000 Binary files a/docs/assets/images/android_project_overview.jpg and /dev/null differ diff --git a/docs/assets/images/sample-notification-system-architecture.png b/docs/assets/images/enterprisenotifications-architecture.png similarity index 100% rename from docs/assets/images/sample-notification-system-architecture.png rename to docs/assets/images/enterprisenotifications-architecture.png diff --git a/docs/assets/images/virtualassistantclient-android-fullconversationview.png b/docs/assets/images/virtualassistantclient-android-fullconversationview.png new file mode 100644 index 0000000000..1a63298099 Binary files /dev/null and b/docs/assets/images/virtualassistantclient-android-fullconversationview.png differ diff --git a/docs/assets/images/virtualassistantclient-android-overview.png b/docs/assets/images/virtualassistantclient-android-overview.png new file mode 100644 index 0000000000..c88909a3a9 Binary files /dev/null and b/docs/assets/images/virtualassistantclient-android-overview.png differ diff --git a/docs/assets/images/virtualassistantclient-android-widgetview.png b/docs/assets/images/virtualassistantclient-android-widgetview.png new file mode 100644 index 0000000000..d5172d21eb Binary files /dev/null and b/docs/assets/images/virtualassistantclient-android-widgetview.png differ