| page.title=Optimizing Content for the Assistant |
| page.metaDescription=Support contextually relevant actions through the Assist API. |
| page.tags=assist, accessibility, now, now on tap |
| meta.tags="assistant", "marshmallow", "now" |
| page.image=images/cards/card-assist_16-9_2x.png |
| |
| page.article=true |
| @jd:body |
| |
| <div id="tb-wrapper"> |
| <div id="tb"> |
| <h2>In this document</h2> |
| <ol> |
| <li><a href="#assist_api">Using the Assistant</a> |
| <ol> |
| <li><a href="#source_app">Source app</a></li> |
| <li><a href="#destination_app">Destination app</a></li> |
| </ol> |
| </li> |
| <li><a href="#implementing_your_own_assistant">Implementing Your |
| Own Assistant</a></li> |
| </ol> |
| </div> |
| </div> |
| |
| <p> |
| Android 6.0 Marshmallow introduces a new way for users to engage with apps |
| through the assistant. The assistant is a top-level window that users can view to obtain |
| contextually relevant actions for the current activity. These actions might include deep links |
| to other apps on the device.</p> |
| |
| <p> |
| Users activate the assistant with a long press on the Home button or by saying a |
| <a href="{@docRoot}reference/android/service/voice/AlwaysOnHotwordDetector.html">keyphrase</a>. |
| In response, the system opens a top-level window that displays contextually |
| relevant actions. |
| </p> |
| |
| <p> |
| Google App implements the assistant overlay window through a feature called |
| Now on Tap, which works with the Android platform-level functionality. The system allows |
| the user to select an assistant app, which obtains contextual information from your app |
| using Android’s Assist API. |
| </p> |
| <p> |
| This guide explains how Android apps use Android's Assist API to improve the assistant |
| user experience. |
| <p/> |
| </p> |
| |
| |
| <h2 id="assist_api">Using the Assistant</h2> |
| |
| <p> |
| Figure 1 illustrates a typical user interaction with the assistant. When the user long-presses |
| the Home button, the Assist API callbacks are invoked |
| in the <em>source</em> app (step 1). The assistant renders the overlay window (steps 2 and 3), |
| and then the user selects the action to perform. The assistant executes the selected action, |
| such as firing an intent with a deep link to the (<em>destination</em>) restaurant app (step 4). |
| </p> |
| |
| <div> |
| <img src="{@docRoot}images/training/assistant/image01.png"> |
| <p class="img-caption" style="text-align:center;"> |
| Figure 1. Assistant interaction example with the Now on Tap feature of |
| the Google App |
| </p> |
| </div> |
| |
| <p> |
| Users can configure the assistant by selecting <strong>Settings > Apps > Default Apps > |
| Assist & voice input</strong>. Users can change system options such as accessing |
| the screen contents as text and accessing a screenshot, as shown in Figure 2. |
| </p> |
| |
| <div id="assist-input-settings" style="float:right;margin:1em;max-width:300px"> |
| <img src="{@docRoot}images/training/assistant/image02.png"> |
| <p class="img-caption" style="text-align:center;"> |
| Figure 2. Assist & voice input settings |
| </p> |
| </div> |
| |
| <h3 id="source_app">Source app</h3> |
| |
| <p> |
| To ensure that your app works with the assistant as a source of information for the user, |
| you need only follow <a href= |
| "{@docRoot}guide/topics/ui/accessibility/apps.html">accessibility best |
| practices</a>. This section describes how to provide additional information |
| to help improve the assistant user experience as well as scenarios |
| that need special handling, such as custom Views. |
| </p> |
| <h4 id="share_additional_information_with_the_assistant">Share additional information |
| with the assistant</h4> |
| |
| <p> |
| In addition to the text and the screenshot, your app can share |
| other information with the assistant. For example, your music |
| app can choose to pass current album information so that the assistant can |
| suggest smarter actions tailored to the current activity. |
| </p> |
| |
| <p> |
| To provide additional information to the assistant, your app provides |
| <em>global application context</em> by registering an app listener and |
| supplies activity-specific information with activity callbacks as shown in |
| Figure 3: |
| </p> |
| |
| <div> |
| <img src="{@docRoot}images/training/assistant/image03.png"> |
| <p class="img-caption" style="text-align:center;"> |
| Figure 3. Assist API lifecycle sequence diagram |
| </p> |
| </div> |
| |
| <p> |
| To provide global application context, the app creates an implementation of |
| {@link android.app.Application.OnProvideAssistDataListener} and registers it |
| using {@link |
| android.app.Application#registerOnProvideAssistDataListener(android.app.Application.OnProvideAssistDataListener) registerOnProvideAssistDataListener()}. |
| To provide activity-specific contextual information, the activity |
| overrides {@link android.app.Activity#onProvideAssistData(android.os.Bundle) onProvideAssistData()} |
| and {@link |
| android.app.Activity#onProvideAssistContent(android.app.assist.AssistContent) onProvideAssistContent()}. |
| The two activity methods are called <em>after</em> the optional global |
| callback is invoked. Because the callbacks execute on the main thread, they should |
| complete <a href="{@docRoot}training/articles/perf-anr.html">promptly</a>. |
| The callbacks are invoked only when the activity is <a href= |
| "{@docRoot}reference/android/app/Activity.html#ActivityLifecycle">running</a>. |
| </p> |
| |
| <h5 id="providing_context">Providing context</h5> |
| |
| <p> |
| When the user activates the assistant, |
| {@link android.app.Activity#onProvideAssistData(android.os.Bundle) onProvideAssistData()} is called to build a full |
| {@link |
| android.content.Intent#ACTION_ASSIST} Intent with all of the context of the |
| current application represented as an instance of the {@link |
| android.app.assist.AssistStructure}. You can override this method to place |
| anything you like into the bundle to appear in the |
| {@link android.content.Intent#EXTRA_ASSIST_CONTEXT} part of the assist intent. |
| </p> |
| |
| <h5 id="describing_content">Describing content</h5> |
| |
| <p> |
| Your app can implement {@link |
| android.app.Activity#onProvideAssistContent(android.app.assist.AssistContent) onProvideAssistContent()} |
| to improve the assistant user experience by providing content-related references |
| related to the current activity. You can describe the app content using the |
| common vocabulary defined by <a href="https://schema.org" class="external-link">Schema.org</a> |
| through a JSON-LD object. In the example below, a music app provides |
| structured data to describe the music album that the user is currently |
| viewing: |
| </p> |
| |
| <pre class="prettyprint"> |
| @Override |
| public void onProvideAssistContent(AssistContent <strong>assistContent</strong>) { |
| super.onProvideAssistContent(<strong>assistContent</strong>); |
| |
| String structuredJson = <strong>new </strong>JSONObject() |
| .put(<strong>"@type"</strong>, <strong>"MusicRecording"</strong>) |
| .put(<strong>"@id"</strong>, <strong>"https://example.com/music/recording"</strong>) |
| .put(<strong>"name"</strong>, <strong>"Album Title"</strong>) |
| .toString(); |
| |
| <strong>assistContent</strong>.setStructuredData(structuredJson); |
| } |
| </pre> |
| |
| <p> |
| You can also improve the user experience with custom implementations of |
| {@link |
| android.app.Activity#onProvideAssistContent(android.app.assist.AssistContent) onProvideAssistContent()}, |
| which can provide the following benefits: |
| </p> |
| <ul> |
| <li><a href="{@docRoot}reference/android/app/assist/AssistContent.html#setIntent(android.content.Intent)"> |
| Adjusts the provided content |
| intent</a> to |
| better reflect the top-level context of the activity.</li> |
| <li><a href="{@docRoot}reference/android/app/assist/AssistContent.html#setWebUri(android.net.Uri)"> |
| Supplies the URI</a> |
| of the displayed content.</li> |
| <li>Fills in {@link |
| android.app.assist.AssistContent#setClipData(android.content.ClipData) setClipData()} with additional |
| content of interest that the user is currently viewing.</li> |
| </ul> |
| <p class="note"> |
| <strong>Note: </strong>Apps that use a custom text selection implementation likely need |
| to implement {@link |
| android.app.Activity#onProvideAssistContent(android.app.assist.AssistContent) onProvideAssistContent()} |
| and call {@link android.app.assist.AssistContent#setClipData(android.content.ClipData) setClipData()}. |
| </p> |
| |
| <h4 id="default_implementation">Default implementation</h4> |
| |
| <p> |
| If neither the {@link |
| android.app.Activity#onProvideAssistData(android.os.Bundle) onProvideAssistData()} nor the {@link |
| android.app.Activity#onProvideAssistContent(android.app.assist.AssistContent) onProvideAssistContent()} |
| callback is implemented, the system still proceeds and passes the |
| automatically collected information to the assistant unless the current |
| window is flagged as <a href="#excluding_views">secure</a>. |
| As shown in Figure 3, the system uses the default implementations of {@link |
| android.view.View#onProvideStructure(android.view.ViewStructure) onProvideStructure()} and {@link |
| android.view.View#onProvideVirtualStructure(android.view.ViewStructure) onProvideVirtualStructure()} to |
| collect text and view hierarchy information. If your view implements custom |
| text drawing, override {@link |
| android.view.View#onProvideStructure(android.view.ViewStructure) onProvideStructure()} to provide |
| the assistant with the text shown to the user by calling {@link |
| android.view.ViewStructure#setText(java.lang.CharSequence) setText(CharSequence)}. |
| </p> |
| |
| <p> |
| <em>In most cases, implementing accessibility support enables the |
| assistant to obtain the information it needs.</em> To implement accessibility support, |
| observe the best practices described in <a href= |
| "{@docRoot}guide/topics/ui/accessibility/apps.html">Making Applications |
| Accessible</a>, including the following:</p> |
| |
| <ul> |
| <li>Provide {@link android.R.attr#contentDescription |
| android:contentDescription} attributes.</li> |
| <li>Populate {@link |
| android.view.accessibility.AccessibilityNodeInfo} for custom views.</li> |
| <li>Make |
| sure that custom {@link android.view.ViewGroup ViewGroup} objects correctly |
| <a href="{@docRoot}reference/android/view/ViewGroup.html#getChildAt(int)">expose</a> |
| their children.</li> |
| </ul> |
| |
| <h4 id="excluding_views">Excluding views from the assistant</h4> |
| |
| <p> |
| To handle sensitive information, your app can exclude the current view from the assistant |
| by setting the {@link android.view.WindowManager.LayoutParams#FLAG_SECURE |
| FLAG_SECURE} layout parameter of the {@link android.view.WindowManager}. You must set {@link |
| android.view.WindowManager.LayoutParams#FLAG_SECURE |
| FLAG_SECURE} explicitly for |
| every window created by the activity, including dialogs. Your app can also use |
| {@link android.view.SurfaceView#setSecure(boolean) setSecure()} to exclude |
| a surface from the assistant. There is no |
| global (app-level) mechanism to exclude all views from the assistant. Note |
| that {@link android.view.WindowManager.LayoutParams#FLAG_SECURE |
| FLAG_SECURE} does not cause the Assist API callbacks to stop |
| firing. The activity that uses {@link android.view.WindowManager.LayoutParams#FLAG_SECURE |
| FLAG_SECURE} can still explicitly |
| provide information to the assistant using the callbacks described earlier |
| this guide. |
| </p> |
| |
| <p class="note"><strong>Note: </strong>For enterprise accounts (Android for Work), |
| the administrator can disable |
| the collection of assistant data for the work profile by using the {@link |
| android.app.admin.DevicePolicyManager#setScreenCaptureDisabled(android.content.ComponentName, boolean) |
| setScreenCaptureDisabled()} method of the {@link android.app.admin.DevicePolicyManager} API.</p> |
| |
| <h4 id="voice_interactions">Voice interactions</h4> |
| |
| <p> |
| Assist API callbacks are also invoked upon |
| <a href="{@docRoot}reference/android/service/voice/AlwaysOnHotwordDetector.html">keyphrase |
| detection</a>. For more information, see the |
| <a href="https://developers.google.com/voice-actions/" class="external-link">Voice |
| Actions</a> documentation. |
| </p> |
| |
| <h4 id="z-order_considerations">Z-order considerations</h4> |
| |
| <p> |
| The assistant uses a lightweight overlay window displayed on top of the |
| current activity. Because the user can activate the assistant at any time, |
| don't create permanent <a |
| href="{@docRoot}reference/android/Manifest.permission.html#SYSTEM_ALERT_WINDOW"> |
| system alert</a> windows that interfere with the overlay window, as shown in |
| Figure 4. |
| </p> |
| |
| <div style=""> |
| <img src="{@docRoot}images/training/assistant/image04.png"> |
| <p class="img-caption" style="text-align:center;"> |
| Figure 4. Assist layer Z-order |
| </p> |
| </div> |
| |
| <p> |
| If your app uses <a |
| href="{@docRoot}reference/android/Manifest.permission.html#SYSTEM_ALERT_WINDOW"> |
| system alert</a> windows, remove them promptly because leaving them on the |
| screen degrades the user experience. |
| </p> |
| |
| <h3 id="destination_app">Destination app</h3> |
| |
| <p> |
| The assistant typically takes advantage of deep linking to find destination apps. To make your |
| app a potential destination app, consider adding <a href= |
| "{@docRoot}training/app-indexing/deep-linking.html">deep linking</a> support. The matching |
| between the current user context and deep links or other potential actions displayed in the |
| overlay window (shown in step 3 in Figure 1) is specific to the assistant’s implementation. |
| For |
| example, the Google App uses deep linking and <a href= |
| "https://developers.google.com/app-indexing/" class="external-link">Firebase App Indexing</a> in order to |
| drive traffic to destination apps. |
| </p> |
| |
| <h2 id="implementing_your_own_assistant">Implementing Your Own Assistant </h2> |
| |
| <p> |
| You may wish to implement your own assistant. As shown in <a href="#assist-input-settings">Figure |
| 2</a>, the user can select the active assistant app. The |
| assistant app must provide an implementation of {@link |
| android.service.voice.VoiceInteractionSessionService} and {@link |
| android.service.voice.VoiceInteractionSession} as shown in <a href= |
| "https://android.googlesource.com/platform/frameworks/base/+/marshmallow-release/tests/VoiceInteraction/" class="external-link"> |
| this <code>VoiceInteraction</code> example</a>. It also requires the {@link |
| android.Manifest.permission#BIND_VOICE_INTERACTION} permission. The assistant can then |
| receive the text and view hierarchy represented as an instance of the {@link |
| android.app.assist.AssistStructure} in {@link |
| android.service.voice.VoiceInteractionSession#onHandleAssist(android.os.Bundle, |
| android.app.assist.AssistStructure,android.app.assist.AssistContent) onHandleAssist()}. |
| It receives the screenshot through {@link |
| android.service.voice.VoiceInteractionSession#onHandleScreenshot(android.graphics.Bitmap) |
| onHandleScreenshot()}. |
| </p> |