<?xml version="1.0" encoding="utf-8"?>
<!-- Copyright (c) 2007-2010 Nokia Corporation and/or its subsidiary(-ies) All rights reserved. -->
<!-- This component and the accompanying materials are made available under the terms of the License 
"Eclipse Public License v1.0" which accompanies this distribution, 
and is available at the URL "http://www.eclipse.org/legal/epl-v10.html". -->
<!-- Initial Contributors:
    Nokia Corporation - initial contribution.
Contributors: 
-->
<!DOCTYPE concept
  PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
<concept id="GUID-859CAA08-59C9-5FD3-98DE-6BDD0D6ED50B" xml:lang="en"><title>Graphics
Composition</title><prolog><metadata><keywords/></metadata></prolog><conbody>
<p>Composition is the process of putting together the output elements from
various different sources to create the screen display that the end user sees
on the device. </p>
<p> <b>Variant</b>: ScreenPlay. </p>
<p>In a multi-tasking device many of the activities taking place simultaneously
generate output for display on the screen. Their output can include words,
pictures, video, games and the screen furniture (scroll bars, buttons, icons,
borders, tabs, menus, title bars) familiar to every computer user. </p>
<p>Many of these output elements can appear at the same time, either next
to each other or overlapping each other. They can be opaque, such that they
obscure anything behind, or semi-transparent such that the elements underneath
are partially visible. </p>
<p>The diagram below illustrates how the display that the viewer sees (looking
down from the top) is a two-dimensional representation composed from a series
of layers. </p>
<fig id="GUID-19D41F69-1264-5726-9CDD-C4DD0F231BE3">
<title>           The display is an orthogonal view of a series of layers.
         </title>
<image href="GUID-7F3F89C0-999A-552E-90BB-17D720C53DE6_d0e184682_href.png" placement="inline"/>
</fig>
<p>Composition requires: </p>
<ul>
<li id="GUID-7C593A87-268F-59E1-B295-9DE2A0DF3092"><p>Calculations based on
the size, position, visibility, transparency and ordering of the layers to
determine what will be displayed. This is a <b>logic</b> exercise referred
to as UI Composition. </p> </li>
<li id="GUID-B10758D4-8C30-5446-958D-0B81530B69BF"><p>Handling of image content,
which is a <b>data processing</b> exercise referred to as Image Composition. </p> </li>
</ul>
<p>For this reason the two are handled separately. </p>
<p><b>UI Composition </b> </p>
<p>UI Composition is performed by the Window Server. Each application has
its own window group containing all of its child windows. The Window Server
keeps track of the windows' positions, sizes, visibilities, transparencies
and z-order and is able to establish which windows, and which bits of each
window, are visible. It ensures that each visible bit of window is kept up-to-date
by calling its application when necessary. </p>
<p><b>Image composition </b> </p>
<p>Prior to the introduction of ScreenPlay the Window Server did rudimentary
composition in its main thread and rendered composited output to the screen
buffer using the GDI (Graphics Device Interface). To achieve high frame rates,
applications typically bypassed the Window Server using Direct Screen Access
(DSA) and animation DLLs. </p>
<p>In <xref href="GUID-D93978BE-11A3-5CE3-B110-1DEAA5AD566C.dita">ScreenPlay</xref> sources
that generate complex graphical output render directly to graphics surfaces.
The Window Server delegates the composition of surfaces to a composition engine
that device creators can adapt to take advantage of graphics processing hardware. </p>
<p>Although the composition engine runs in the Window Server's process, it
is largely transparent to application developers and works quite differently.
Instead of using windows, it uses surfaces to store pixel data and elements
to manipulate size, position, z-order, visibility and transparency. </p>
<p>While an element is a simple lightweight object, and easy to manipulate,
a surface stores a large amount of data and its handling requires more consideration. </p>
<p>The diagram below is a simplistic representation of how applications create
output which is rendered, composited and displayed. </p>
<fig id="GUID-9DF4C86A-C06D-5C5A-9AB6-E9991CC1937A">
<title>           Graphics Composition          </title>
<image href="GUID-F31EC49A-FE01-58B2-9CB5-4A3BBCCB7DA7_d0e184746_href.png" placement="inline"/>
</fig>
<p>In a device with graphics acceleration hardware (a Graphics Processing
Unit or GPU) there might, in addition to the memory managed by the CPU, be
additional memory managed by the GPU. Image data may therefore be considered
to have been <b>software rendered</b> (onto a surface in CPU memory) or <b>hardware
rendered</b> (onto a surface in hardware accelerated GPU memory). The following
diagram shows applications and other graphical data sources rendering to surfaces
in software and hardware. </p>
<fig id="GUID-2717F861-0045-5598-A3EC-7CF678BEFA70">
<title>           Hardware rendered graphics - the wrong way!          </title>
<image href="GUID-EADC4EA6-4492-5A00-A29E-6F7747FCAAC9_d0e184763_href.png" placement="inline"/>
</fig>
<p>The diagram, however, represents a system with several problem areas that
would render it unsuitable for any practical implementation </p>
<ul>
<li id="GUID-AF13B0A1-3DC0-576D-81B1-6DF66AA9FA77"><p>In practice it is likely
that once data has been rendered to hardware-managed memory it is, to all
intents and purposes, unavailable to software: the CPU is unable to access
it sufficiently quickly. The dotted paths on the diagram above must therefore
be avoided. </p> </li>
<li id="GUID-0B109999-94FE-5A5C-B0D0-D3F2157B5CC6"><p>Furthermore, GDI rendered
data (the UI) is typically 'stored' as redrawing instructions rather than
bitmapped pixel data—so rendering it to bitmaps before composition is likely
to use a lot of memory unnecessarily. </p> </li>
</ul>
<p>In ScreenPlay, the UI is therefore composed and rendered onto a single
surface before being composited with any other surfaces. This surface is termed
the <b>UI surface</b> and is displayed on a layer placed in front of all of
the others. In fact, the UI surface is created for the Window Server during
system start up and is then passed to the composition components as a 'special
case' surface for composition. </p>
<section id="GUID-6E274D1A-E4CB-4980-B351-B396FEC48DB8"><title>Combining Window Server and Composition </title> <p>The compositing
of surfaces according to their origin means that the physical composition
process behaves differently from the logical composition process that is based
on what the user and the UI are doing. Logically the windows in the UI and
memory rendered surfaces may be on layers that are interleaved yet the memory
rendered surfaces are physically composed behind. </p> <p>ScreenPlay addresses
this issue by associating 'external' surfaces with windows in the Window Server
using the <xref href="GUID-1460DD8F-9AA1-3B99-8FFD-F309959CCA34.dita#GUID-1460DD8F-9AA1-3B99-8FFD-F309959CCA34/GUID-4EBFFA14-418A-3A2A-B147-39BFC96CE45F"><apiname>RWindowBase::SetBackgroundSurface()</apiname></xref> method. This
means that the Window Server is able to include them in its logical composition
and make provision for them during data composition. A window with its background
set to a surface in this way becomes transparent in the UI rendered UI surface. </p> <p>In
most cases this is not apparent to the viewer. Surfaces that are physically
composed behind the UI appear in the correct position on the two-dimensional
display. The diagram below illustrates how Window Server-rendered UI content
and external surfaces are composited using the UI surface. </p> <fig id="GUID-3448FD6B-9A39-58CD-8819-39B0B6CC4E13">
<title>              Hardware composition and the Flattened UI            </title>
<image href="GUID-B0797210-4EE3-557B-A5A6-D215D030BA0E_d0e184810_href.png" placement="inline"/>
</fig> <p>Although this method of composition is flexible and powerful, it
does have some limitations, particularly with respect to semi-transparent
hardware-accelerated surfaces. </p> <p>It is not possible, for example, to
include a semi-transparent surface in front of the UI surface. In such cases
the composition engine determines how the display is composed. </p> <p>Here
is a second version of the diagram at the top of the page showing how the
same composition might be achieved in practice. The UI menus, windows and
dialogs are composited by the Window Server onto a single surface. The light
green layer displays a hardware rendered surface so it is actually behind
the layer on which it appears. </p> <fig id="GUID-1FB1194D-FCAE-534C-94DC-AF4CF00E3A1C">
<title>              The UI surface            </title>
<image href="GUID-79009102-0490-5C61-9722-C5EE49A1AF2B_d0e184827_href.png" placement="inline"/>
</fig> </section>
<section id="GUID-41E9CA54-759F-5651-95F8-9F39808BE740"><title>Composition
Examples</title> <p>This illustrations below illustrate the use of hardware
accelerated surfaces and the UI surface </p> <fig id="GUID-8178DBA8-E10C-56F3-A828-80746CE6A993">
<title>              Video rendered to a hardware accelerated surface mapped
to a layer              behind the UI surface            </title>
<image href="GUID-FE3C8D39-CE17-5AC7-AB6A-4D6664D52196_d0e184843_href.png" placement="inline"/>
</fig> <fig id="GUID-51411B10-6DC3-5B74-BCEF-11EC1D0FBCA8">
<title>              As above with a semi-transparent dialog on the UI surface
           </title>
<image href="GUID-4616CCC9-7BD3-5D91-873A-6027167329ED_d0e184851_href.png" placement="inline"/>
</fig> </section>
</conbody><related-links>
<link href="GUID-D93978BE-11A3-5CE3-B110-1DEAA5AD566C.dita"><linktext>The ScreenPlay
Architecture</linktext></link>
<link href="GUID-EF62BF88-3687-505D-8BD7-EEDF36246E56.dita"><linktext>Hardware
Acceleration</linktext></link>

</related-links></concept>