src/devices/graphics/implement-hwc.jd

page.title=Implementing the Hardware Composer HAL
@jd:body

<!--
    Copyright 2016 The Android Open Source Project

    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.
    You may obtain a copy of the License at

        http://www.apache.org/licenses/LICENSE-2.0

    Unless required by applicable law or agreed to in writing, software
    distributed under the License is distributed on an "AS IS" BASIS,
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
    limitations under the License.
-->

<div id="qv-wrapper">
  <div id="qv">
    <h2>In this document</h2>
    <ol id="auto-toc">
    </ol>
  </div>
</div>


<p>The Hardware Composer HAL (HWC) is used by SurfaceFlinger to composite
surfaces to the screen. The HWC abstracts objects such as overlays and 2D
blitters and helps offload some work that would normally be done with OpenGL.</p>

<p>Android 7.0 includes a new version of HWC (HWC2) used by SurfaceFlinger to
talk to specialized window composition hardware. SurfaceFlinger contains a
fallback path that uses the 3D graphics processor (GPU) to perform the task of
window composition, but this path is not ideal for a couple of reasons:</p>

<ul>
  <li>Typically, GPUs are not optimized for this use case and may use more power
  than necessary to perform composition.</li>
  <li>Any time SurfaceFlinger is using the GPU for composition is time that
  applications cannot use the processor for their own rendering, so it is
  preferable to use specialized hardware for composition instead of the GPU
  whenever possible.</li>
</ul>

<h2 id="guidance">General guidance</h2>

<p>As the physical display hardware behind the Hardware Composer abstraction
layer can vary from device to device, it's difficult to give recommendations on
specific features. In general, use the following guidance:</p>

<ul>
  <li>The HWC should support at least four overlays (status bar, system bar,
  application, and wallpaper/background).</li>
  <li>Layers can be bigger than the screen, so the HWC should be able to handle
  layers that are larger than the display (for example, a wallpaper).</li>
  <li>Pre-multiplied per-pixel alpha blending and per-plane alpha blending
  should be supported at the same time.</li>
  <li>The HWC should be able to consume the same buffers the GPU, camera, and
  video decoder are producing, so supporting some of the following
  properties is helpful:
  <ul>
    <li>RGBA packing order</li>
    <li>YUV formats</li>
    <li>Tiling, swizzling, and stride properties</li>
  </ul>
  <li>To support protected content, a hardware path for protected video playback
  must be present.</li>
  </ul>

<p>The general recommendation is to implement a non-operational HWC first; after
the structure is complete, implement a simple algorithm to delegate composition
to the HWC (for example, delegate only the first three or four surfaces to the
overlay hardware of the HWC).</p>

<p>Focus on optimization, such as intelligently selecting the surfaces to send
to the overlay hardware that maximizes the load taken off of the GPU. Another
optimization is to detect whether the screen is updating; if it isn't, delegate
composition to OpenGL instead of the HWC to save power. When the screen updates
again, continue to offload composition to the HWC.</p>

<p>Prepare for common use cases, such as:</p>

<ul>
  <li>Full-screen games in portrait and landscape mode</li>
  <li>Full-screen video with closed captioning and playback control</li>
  <li>The home screen (compositing the status bar, system bar, application
  window, and live wallpapers)</li>
  <li>Protected video playback</li>
  <li>Multiple display support</li>
</ul>

<p>These use cases should address regular, predictable uses rather than edge
cases that are rarely encountered (otherwise, optimizations will have little
benefit). Implementations must balance two competing goals: animation smoothness
and interaction latency.</p>


<h2 id="interface_activities">HWC2 interface activities</h2>

<p>HWC2 provides a few primitives (layer, display) to represent composition work
and its interaction with the display hardware.</p>
<p>A <em>layer</em> is the most important unit of composition; every layer has a
set of properties that define how it interacts with other layers. Property
categories include the following:</p>

<ul>
<li><strong>Positional</strong>. Defines where the layer appears on its display.
Includes information such as the positions of a layer's edges and its <em>Z
order</em> relative to other layers (whether it should be in front of or behind
other layers).</li>
<li><strong>Content</strong>. Defines how content displayed on the layer should
be presented within the bounds defined by the positional properties. Includes
information such as crop (to expand a portion of the content to fill the bounds
of the layer) and transform (to show rotated or flipped content).</li>
<li><strong>Composition</strong>. Defines how the layer should be composited
with other layers. Includes information such as blending mode and a layer-wide
alpha value for
<a href="https://en.wikipedia.org/wiki/Alpha_compositing#Alpha_blending">alpha
compositing</a>.</li>
<li><strong>Optimization</strong>. Provides information not strictly necessary
to correctly composite the layer, but which can be used by the HWC device to
optimize how it performs composition. Includes information such as the visible
region of the layer and which portion of the layer has been updated since the
previous frame.</li>
</ul>

<p>A <em>display</em> is another important unit of composition. Every layer can
be present only on one display. A system can have multiple displays, and
displays can be added or removed during normal system operations. This
addition/removal can come at the request of the HWC device (typically in
response to an external display being plugged into or removed from the device,
called <em>hotplugging</em>), or at the request of the client, which permits the
creation of <em>virtual displays</em> whose contents are rendered into an
off-screen buffer instead of to a physical display.</p>
<p>HWC2 provides functions to determine the properties of a given display, to
switch between different configurations (e.g., 4k or 1080p resolution) and color
modes (e.g., native color or true sRGB), and to turn the display on, off, or
into a low-power mode if supported.</p>
<p>In addition to layers and displays, HWC2 also provides control over the
hardware vertical sync (VSYNC) signal along with a callback into the client to
notify it of when a vsync event has occurred.</p>

<h3 id="func_pointers">Function pointers</h3>
<p>In this section and in HWC2 header comments, HWC interface functions are
referred to by lowerCamelCase names that do not actually exist in the interface
as named fields. Instead, almost every function is loaded by requesting a
function pointer using <code>getFunction</code> provided by
<code>hwc2_device_t</code>. For example, the function <code>createLayer</code>
is a function pointer of type <code>HWC2_PFN_CREATE_LAYER</code>, which is
returned when the enumerated value <code>HWC2_FUNCTION_CREATE_LAYER</code> is
passed into <code>getFunction</code>.</p>
<p>For detailed documentation on functions (including functions required for
every HWC2 implementation), refer to the HWC2 header.</p>

<h3 id="layer_display_handles">Layer and display handles</h3>
<p>Layers and displays are manipulated by opaque handles.</p>
<p>When SurfaceFlinger wants to create a new layer, it calls the
<code>createLayer</code> function, which then returns an opaque handle of type
<code>hwc2_layer_t</code>. From that point on, any time SurfaceFlinger wants to
modify a property of that layer, it passes that <code>hwc2_layer_t</code> value
into the appropriate modification function, along with any other information
needed to make the modification. The <code>hwc2_layer_t</code> type was made
large enough to be able to hold either a pointer or an index, and it will be
treated as opaque by SurfaceFlinger to provide HWC implementers maximum
flexibility.</p>
<p>Most of the above also applies to display handles, though handles are created
differently depending on whether they are hotplugged (where the handle is passed
through the hotplug callback) or requested by the client as a virtual display
(where the handle is returned from <code>createVirtualDisplay</code>).</p>

<h2 id="display_comp_ops">Display composition operations</h2>
<p>Once per hardware vsync, SurfaceFlinger wakes if it has new content to
composite. This new content could be new image buffers from applications or just
a change in the properties of one or more layers. When it wakes, it performs the
following steps:</p>

<ol>
<li>Apply transactions, if present. Includes changes in the properties of layers
specified by the window manager but not changes in the contents of layers (i.e.,
graphic buffers from applications).</li>
<li>Latch new graphic buffers (acquire their handles from their respective
applications), if present.</li>
<li>If step 1 or 2 resulted in a change to the display contents, perform a new
composition (described below).</li>
</ol>

<p>Steps 1 and 2 have some nuances (such as deferred transactions and
presentation timestamps) that are outside the scope of this section. However,
step 3 involves the HWC interface and is detailed below.</p>
<p>At the beginning of the composition process, SurfaceFlinger will create and
destroy layers or modify layer state as applicable. It will also update the
layers with their current contents, using calls such as
<code>setLayerBuffer</code> or <code>setLayerColor</code>. After all layers have
been updated, it will call <code>validateDisplay</code>, which tells the device
to examine the state of the various layers and determine how composition will
proceed. By default, SurfaceFlinger usually attempts to configure every layer
such that it will be composited by the device, though there may be some
circumstances where it will mandate that it be composited by the client.</p>
<p>After the call to <code>validateDisplay</code>, SurfaceFlinger will follow up
with a call to <code>getChangedCompositionTypes</code> to see if the device
wants any of the layers' composition types changed before performing the actual
composition. SurfaceFlinger may choose to:</p>

<ul>
<li>Change some of the layer composition types and re-validate the display.</li>
</ul>

<blockquote><strong><em>OR</strong></em></blockquote>

<ul>
<li>Call <code>acceptDisplayChanges</code>, which has the same effect as
changing the composition types as requested by the device and re-validating
without actually having to call <code>validateDisplay</code> again.</li>
</ul>

<p>In practice, SurfaceFlinger always takes the latter path (calling
<code>acceptDisplayChanges</code>) though this behavior may change in the
future.</p>
<p>At this point, the behavior differs depending on whether any of the layers
have been marked for client composition. If any (or all) layers have been marked
for client composition, SurfaceFlinger will now composite all of those layers
into the client target buffer. This buffer will be provided to the device using
the <code>setClientTarget</code> call so that it may be either displayed
directly on the screen or further composited with layers that have not been
marked for client composition. If no layers have been marked for client
composition, then the client composition step is bypassed.</p>
<p>Finally, after all of the state has been validated and client composition has
been performed if needed, SurfaceFlinger will call <code>presentDisplay</code>.
This is the HWC device's cue to complete the composition process and display the
final result.</p>

<h2 id="multiple_displays">Multiple displays in Android N</h2>
<p>While the HWC2 interface is quite flexible when it comes to the number of
displays in the system, the rest of the Android framework is not yet as
flexible. When designing a HWC2 implementation intended for use on Android N,
there are some additional restrictions not present in the HWC definition itself:
</p>

<ul>
<li>It is assumed that there is exactly one <em>primary</em> display; that is,
that there is one physical display that will be hotplugged immediately during
the initialization of the device (specifically after the hotplug callback is
registered).</li>
<li>In addition to the primary display, exactly one <em>external</em> display
may be hotplugged during normal operation of the device.</li>
</ul>

<p>While the SurfaceFlinger operations described above are performed per-display
(eventual goal is to be able to composite displays independently of each other),
they are currently performed sequentially for all active displays, even if only
the contents of one display are updated.</p>
<p>For example, if only the external display is updated, the sequence is:</p>

<pre>
// Update state for internal display
// Update state for external display
validateDisplay(&lt;internal display&gt;)
validateDisplay(&lt;external display&gt;)
presentDisplay(&lt;internal display&gt;)
presentDisplay(&lt;external display&gt;)
</pre>


<h2 id="sync_fences">Synchronization fences</h2>
<p>Synchronization (sync) fences are a crucial aspect of the Android graphics
system. Fences allow CPU work to proceed independently from concurrent GPU work,
blocking only when there is a true dependency.</p>
<p>For example, when an application submits a buffer that is being produced on
the GPU, it will also submit a fence object; this fence signals only when the
GPU has finished writing into the buffer. Since the only part of the system that
truly needs the GPU write to have finished is the display hardware (the hardware
abstracted by the HWC HAL), the graphics pipeline is able to pass this fence
along with the buffer through SurfaceFlinger to the HWC device. Only immediately
before that buffer would be displayed does the device need to actually check
that the fence has signaled.</p>
<p>Sync fences are integrated tightly into HWC2 and organized in the following
categories:</p>

<ol>
<li>Acquire fences are passed along with input buffers to the
<code>setLayerBuffer</code> and <code>setClientTarget</code> calls. These
represent a pending write into the buffer and must signal before the HWC client
or device attempts to read from the associated buffer to perform composition.
</li>
<li>Release fences are retrieved after the call to <code>presentDisplay</code>
using the <code>getReleaseFences</code> call and are passed back to the
application along with buffers that will be replaced during the next
composition. These represent a pending read from the buffer, and must signal
before the application attempts to write new contents into the buffer.</li>
<li>Retire fences are returned, one per frame, as part of the call to
<code>presentDisplay</code> and represent when the composition of this frame
has completed, or alternately, when the composition result of the prior frame is
no longer needed. For physical displays, this is when the current frame appears
on the screen and can also be interpreted as the time after which it is safe to
write to the client target buffer again (if applicable). For virtual displays,
this is the time when it is safe to read from the output buffer.</li>
</ol>

<h3 id="hwc2_changes">Changes in HWC2</h3>
<p>The meaning of sync fences in HWC 2.0 has changed significantly relative to
previous versions of the HAL.</p>
<p>In HWC v1.x, the release and retire fences were speculative. A release fence
for a buffer or a retire fence for the display retrieved in frame N would not
signal any sooner than frame N + 1. In other words, the meaning of the fence
was "the content of the buffer you provided for frame N is no longer needed."
This is speculative because in theory SurfaceFlinger may not run again after
frame N for an indeterminate period of time, which would leave those fences
unsignaled for the same period.</p>
<p>In HWC 2.0, release and retire fences are non-speculative. A release or
retire fence retrieved in frame N will signal as soon as the content of the
associated buffers replaces the contents of the buffers from frame N - 1, or in
other words, the meaning of the fence is "the content of the buffer you provided
for frame N has now replaced the previous content." This is non-speculative,
since this fence should signal shortly after <code>presentDisplay</code> is
called as soon as the hardware presents this frame's content.</p>
<p>For implementation details, refer to the HWC2 header.</p>