|
1 <?xml version="1.0" encoding="utf-8"?> |
|
2 <!-- Copyright (c) 2007-2010 Nokia Corporation and/or its subsidiary(-ies) All rights reserved. --> |
|
3 <!-- This component and the accompanying materials are made available under the terms of the License |
|
4 "Eclipse Public License v1.0" which accompanies this distribution, |
|
5 and is available at the URL "http://www.eclipse.org/legal/epl-v10.html". --> |
|
6 <!-- Initial Contributors: |
|
7 Nokia Corporation - initial contribution. |
|
8 Contributors: |
|
9 --> |
|
10 <!DOCTYPE concept |
|
11 PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd"> |
|
12 <concept id="GUID-859CAA08-59C9-5FD3-98DE-6BDD0D6ED50B" xml:lang="en"><title>Graphics |
|
13 Composition</title><prolog><metadata><keywords/></metadata></prolog><conbody> |
|
14 <p>Composition is the process of putting together the output elements from |
|
15 various different sources to create the screen display that the end user sees |
|
16 on the device. </p> |
|
17 <p> <b>Variant</b>: ScreenPlay. </p> |
|
18 <p>In a multi-tasking device many of the activities taking place simultaneously |
|
19 generate output for display on the screen. Their output can include words, |
|
20 pictures, video, games and the screen furniture (scroll bars, buttons, icons, |
|
21 borders, tabs, menus, title bars) familiar to every computer user. </p> |
|
22 <p>Many of these output elements can appear at the same time, either next |
|
23 to each other or overlapping each other. They can be opaque, such that they |
|
24 obscure anything behind, or semi-transparent such that the elements underneath |
|
25 are partially visible. </p> |
|
26 <p>The diagram below illustrates how the display that the viewer sees (looking |
|
27 down from the top) is a two-dimensional representation composed from a series |
|
28 of layers. </p> |
|
29 <fig id="GUID-19D41F69-1264-5726-9CDD-C4DD0F231BE3"> |
|
30 <title> The display is an orthogonal view of a series of layers. |
|
31 </title> |
|
32 <image href="GUID-7F3F89C0-999A-552E-90BB-17D720C53DE6_d0e191278_href.png" placement="inline"/> |
|
33 </fig> |
|
34 <p>Composition requires: </p> |
|
35 <ul> |
|
36 <li id="GUID-7C593A87-268F-59E1-B295-9DE2A0DF3092"><p>Calculations based on |
|
37 the size, position, visibility, transparency and ordering of the layers to |
|
38 determine what will be displayed. This is a <b>logic</b> exercise referred |
|
39 to as UI Composition. </p> </li> |
|
40 <li id="GUID-B10758D4-8C30-5446-958D-0B81530B69BF"><p>Handling of image content, |
|
41 which is a <b>data processing</b> exercise referred to as Image Composition. </p> </li> |
|
42 </ul> |
|
43 <p>For this reason the two are handled separately. </p> |
|
44 <p><b>UI Composition </b> </p> |
|
45 <p>UI Composition is performed by the Window Server. Each application has |
|
46 its own window group containing all of its child windows. The Window Server |
|
47 keeps track of the windows' positions, sizes, visibilities, transparencies |
|
48 and z-order and is able to establish which windows, and which bits of each |
|
49 window, are visible. It ensures that each visible bit of window is kept up-to-date |
|
50 by calling its application when necessary. </p> |
|
51 <p><b>Image composition </b> </p> |
|
52 <p>Prior to the introduction of ScreenPlay the Window Server did rudimentary |
|
53 composition in its main thread and rendered composited output to the screen |
|
54 buffer using the GDI (Graphics Device Interface). To achieve high frame rates, |
|
55 applications typically bypassed the Window Server using Direct Screen Access |
|
56 (DSA) and animation DLLs. </p> |
|
57 <p>In <xref href="GUID-D93978BE-11A3-5CE3-B110-1DEAA5AD566C.dita">ScreenPlay</xref> sources |
|
58 that generate complex graphical output render directly to graphics surfaces. |
|
59 The Window Server delegates the composition of surfaces to a composition engine |
|
60 that device creators can adapt to take advantage of graphics processing hardware. </p> |
|
61 <p>Although the composition engine runs in the Window Server's process, it |
|
62 is largely transparent to application developers and works quite differently. |
|
63 Instead of using windows, it uses surfaces to store pixel data and elements |
|
64 to manipulate size, position, z-order, visibility and transparency. </p> |
|
65 <p>While an element is a simple lightweight object, and easy to manipulate, |
|
66 a surface stores a large amount of data and its handling requires more consideration. </p> |
|
67 <p>The diagram below is a simplistic representation of how applications create |
|
68 output which is rendered, composited and displayed. </p> |
|
69 <fig id="GUID-9DF4C86A-C06D-5C5A-9AB6-E9991CC1937A"> |
|
70 <title> Graphics Composition </title> |
|
71 <image href="GUID-F31EC49A-FE01-58B2-9CB5-4A3BBCCB7DA7_d0e191342_href.png" placement="inline"/> |
|
72 </fig> |
|
73 <p>In a device with graphics acceleration hardware (a Graphics Processing |
|
74 Unit or GPU) there might, in addition to the memory managed by the CPU, be |
|
75 additional memory managed by the GPU. Image data may therefore be considered |
|
76 to have been <b>software rendered</b> (onto a surface in CPU memory) or <b>hardware |
|
77 rendered</b> (onto a surface in hardware accelerated GPU memory). The following |
|
78 diagram shows applications and other graphical data sources rendering to surfaces |
|
79 in software and hardware. </p> |
|
80 <fig id="GUID-2717F861-0045-5598-A3EC-7CF678BEFA70"> |
|
81 <title> Hardware rendered graphics - the wrong way! </title> |
|
82 <image href="GUID-EADC4EA6-4492-5A00-A29E-6F7747FCAAC9_d0e191359_href.png" placement="inline"/> |
|
83 </fig> |
|
84 <p>The diagram, however, represents a system with several problem areas that |
|
85 would render it unsuitable for any practical implementation </p> |
|
86 <ul> |
|
87 <li id="GUID-AF13B0A1-3DC0-576D-81B1-6DF66AA9FA77"><p>In practice it is likely |
|
88 that once data has been rendered to hardware-managed memory it is, to all |
|
89 intents and purposes, unavailable to software: the CPU is unable to access |
|
90 it sufficiently quickly. The dotted paths on the diagram above must therefore |
|
91 be avoided. </p> </li> |
|
92 <li id="GUID-0B109999-94FE-5A5C-B0D0-D3F2157B5CC6"><p>Furthermore, GDI rendered |
|
93 data (the UI) is typically 'stored' as redrawing instructions rather than |
|
94 bitmapped pixel data—so rendering it to bitmaps before composition is likely |
|
95 to use a lot of memory unnecessarily. </p> </li> |
|
96 </ul> |
|
97 <p>In ScreenPlay, the UI is therefore composed and rendered onto a single |
|
98 surface before being composited with any other surfaces. This surface is termed |
|
99 the <b>UI surface</b> and is displayed on a layer placed in front of all of |
|
100 the others. In fact, the UI surface is created for the Window Server during |
|
101 system start up and is then passed to the composition components as a 'special |
|
102 case' surface for composition. </p> |
|
103 <section id="GUID-6E274D1A-E4CB-4980-B351-B396FEC48DB8"><title>Combining Window Server and Composition </title> <p>The compositing |
|
104 of surfaces according to their origin means that the physical composition |
|
105 process behaves differently from the logical composition process that is based |
|
106 on what the user and the UI are doing. Logically the windows in the UI and |
|
107 memory rendered surfaces may be on layers that are interleaved yet the memory |
|
108 rendered surfaces are physically composed behind. </p> <p>ScreenPlay addresses |
|
109 this issue by associating 'external' surfaces with windows in the Window Server |
|
110 using the <xref href="GUID-1460DD8F-9AA1-3B99-8FFD-F309959CCA34.dita#GUID-1460DD8F-9AA1-3B99-8FFD-F309959CCA34/GUID-4EBFFA14-418A-3A2A-B147-39BFC96CE45F"><apiname>RWindowBase::SetBackgroundSurface()</apiname></xref> method. This |
|
111 means that the Window Server is able to include them in its logical composition |
|
112 and make provision for them during data composition. A window with its background |
|
113 set to a surface in this way becomes transparent in the UI rendered UI surface. </p> <p>In |
|
114 most cases this is not apparent to the viewer. Surfaces that are physically |
|
115 composed behind the UI appear in the correct position on the two-dimensional |
|
116 display. The diagram below illustrates how Window Server-rendered UI content |
|
117 and external surfaces are composited using the UI surface. </p> <fig id="GUID-3448FD6B-9A39-58CD-8819-39B0B6CC4E13"> |
|
118 <title> Hardware composition and the Flattened UI </title> |
|
119 <image href="GUID-B0797210-4EE3-557B-A5A6-D215D030BA0E_d0e191406_href.png" placement="inline"/> |
|
120 </fig> <p>Although this method of composition is flexible and powerful, it |
|
121 does have some limitations, particularly with respect to semi-transparent |
|
122 hardware-accelerated surfaces. </p> <p>It is not possible, for example, to |
|
123 include a semi-transparent surface in front of the UI surface. In such cases |
|
124 the composition engine determines how the display is composed. </p> <p>Here |
|
125 is a second version of the diagram at the top of the page showing how the |
|
126 same composition might be achieved in practice. The UI menus, windows and |
|
127 dialogs are composited by the Window Server onto a single surface. The light |
|
128 green layer displays a hardware rendered surface so it is actually behind |
|
129 the layer on which it appears. </p> <fig id="GUID-1FB1194D-FCAE-534C-94DC-AF4CF00E3A1C"> |
|
130 <title> The UI surface </title> |
|
131 <image href="GUID-79009102-0490-5C61-9722-C5EE49A1AF2B_d0e191423_href.png" placement="inline"/> |
|
132 </fig> </section> |
|
133 <section id="GUID-41E9CA54-759F-5651-95F8-9F39808BE740"><title>Composition |
|
134 Examples</title> <p>This illustrations below illustrate the use of hardware |
|
135 accelerated surfaces and the UI surface </p> <fig id="GUID-8178DBA8-E10C-56F3-A828-80746CE6A993"> |
|
136 <title> Video rendered to a hardware accelerated surface mapped |
|
137 to a layer behind the UI surface </title> |
|
138 <image href="GUID-FE3C8D39-CE17-5AC7-AB6A-4D6664D52196_d0e191439_href.png" placement="inline"/> |
|
139 </fig> <fig id="GUID-51411B10-6DC3-5B74-BCEF-11EC1D0FBCA8"> |
|
140 <title> As above with a semi-transparent dialog on the UI surface |
|
141 </title> |
|
142 <image href="GUID-4616CCC9-7BD3-5D91-873A-6027167329ED_d0e191447_href.png" placement="inline"/> |
|
143 </fig> </section> |
|
144 </conbody><related-links> |
|
145 <link href="GUID-D93978BE-11A3-5CE3-B110-1DEAA5AD566C.dita"><linktext>The ScreenPlay |
|
146 Architecture</linktext></link> |
|
147 <link href="GUID-EF62BF88-3687-505D-8BD7-EEDF36246E56.dita"><linktext>Hardware |
|
148 Acceleration</linktext></link> |
|
149 |
|
150 </related-links></concept> |