Framing the Canvas (September-October 2014)

[Presented at Droidcon Paris and Droidcon London 2014]

This talk will guide you through a topic that is often ignored: the Canvas API. Even if there's only scarce documentation for it, the Canvas API is at the heart of everything that shows a UI on Android. You'll need to tame this powerful and mysterious creature if you want to create great custom views and brilliant, lightweight UIs for your apps.

What is a Canvas? How do you Paint? Can you draw Paths? What is Skia, and who invited it anyway? What actually happens during a drawing pass? What are Shaders? What are Filters? Is it true that text is basically impossible to measure correctly? What shape does a Shape have if noone's painting it? These and other questions will get an answer during this session. Well, some of them will not, probably.

Connect the dots in the spotty documentation the Android team has assembled, and become a true pixel pusher!

Transcript

1. Framing the Canvas Get your Paint together, follow the Path and master Shaders
2. Sebastiano Poggi Insultez / Stalkez moi sur: @seebrock3r +SebastianoPoggi rock3r Android Software Craftsman à Londres I don’t know about you, but when I work with the Canvas API, I spend most of my time looking like this…
3. CC-BY-NC Joe Benjamin on Flickr Yes, let’s say that is my confused, or WTF, face. And again, yes, I had another guy looking puzzled at my side the whole time. But that’s a different story. ! The point is, if you want to do amazing UI on Android you have to master the Canvas API, with all its quirks and secrets. Most of the time you have to look at code to understand what’s going on, but that only gives you a narrow view. What you miss is the big picture, and I hope you’ll be leaving this room with a better understanding of what does what when we talk about drawing stuff on screen.
4. Paint on the Canvas CC-BY-NC-ND anna gutermuth on Flickr So, let’s start from the basics. How many of you have actually ever created custom Views or crafted some custom drawing logic? The rest of you probably have never used the Canvas API.
5. • Base for any Android UI • Drawing on a buffer • Hardware or Software • API: Canvas, Paint, … Canvas CC-BY Cara St.Hilaire on Flickr The Canvas API is used in Android to draw all the UI. The Canvas is the interface between your code and the underlying drawing buffer, that is then shown on screen. That buffer is either backed by HW or SW. ! The name is really helpful to understand what the role of the Canvas is. But as in real life, a Canvas alone isn’t enough to paint. You also need other tools to draw colour, shapes and text onto your Canvas. These tools take the name of Canvas APIs.
6. WELL GOOGLE, IF WE COULD HAVE SOME DOCUMENTATION EVERY NOW AND THEN YEAH, THAT’D BE GREAT Of course, it’s vastly under documented at best. Hey, it’s a Google thing.
7. We should love Skia • Underneath Canvas • Native code • 2D graphics rendering toolkit • Basic painting • Advanced features walcor on DeviantArt Canvas is basically a Java wrapper around Skia, a 2D rendering toolkit also used in Chrome and Chrome OS. Skia is implemented in native code and accessed by the wrapper through JNI. ! Skia is very powerful. It can handle all the usual drawing operations, including blitting bitmaps, drawing primitive shapes, rendering text (using the FreeText engine). It can also do complex operations, such as change the painting behaviour using Shaders, different transfer modes, matricial manipulation, etc.
8. • Text is a pain • Weird quirks • E.g., no proper arcs • No documentation …but Skia kinda sucks Eugenio Marletti on Medium Not everything that glitters is gold. Skia falls short in some areas, especially when it comes to text (you can’t change the kerning; there’s no reliable way to exactly measure text). Some functions are simply badly implemented. For example, try drawing an arc using the drawArc method. On top of it, there’s almost no documentation available for both the Java wrapper and the C++ implementation, besides some really generic and high level infos. Good luck with that!
9. Google • Draw using HWUI • Subset of Skia • Honeycomb and later • With limitations (of course) • Revert to software buffer OK, Canvas is still cool On the plus side, Canvas APIs can be used to draw on HW accelerated surfaces since Honeycomb — using HWUI. ! HWUI is a subset of Skia that is accelerated on hardware using OpenGL ES 2. Trivia? Trivia! Given that OpenGL ES 2 cannot be emulated in SW on Android, that is the reason why devices running Android 3+ are required to have an OpenGL ES 2. ! From now on we’re going to just talk about Skia, as HWUI for what we care about is basically the same thing. ! Since life is never that easy, of course HWUI comes with quite a list of caveats. Some things are simply not available on HW accelerated layers. MaskFilters, for example. Or some XferModes. Or drawing hinted/subpixel text. You’ll need to explicitly set your Canvas to use software rendering if you want to use those! But you then lose HW speed improvements on that Canvas.
10. Unsupported ops on HW Google As you can see, there are quite a few operations that are not supported under HW acceleration. If you try to use them in an HW-accelerated Canvas, they will map out to no-ops. These tables come from the Android Developers website; trust me, they’re not exhaustive. Things are really slowly improving, but there are still quite a few use cases not covered under HW acceleration.
11. Good read Hardware Acceleration on Android Developers website http://goo.gl/HgV8D
12. Using the Canvas • Transforms using matrices and Cameras • Views must use it on the UI thread • Or go with a SurfaceView • Canvas is stateful Canvas can be transformed in a variety of ways. You can scale, translate, rotate your canvases using the Canvas methods. All the transformations resolve into matrix transforms (yay, linear algebra!). This means you should brush off your maths skills if you don’t want to shoot yourself in the foot here; concatenating matrix transformations can quickly lead to grief, suffering and pain if you’re not careful. ! Views must only perform operations on the Canvas on the UI thread — but you shouldn’t be doing anything UI-related anywhere else anyway. If for some reason you want to draw on a Canvas off of the UI thread, use a SurfaceView, which provides asynchronous drawing callbacks. ! Lastly, remember: each Canvas has a state. It can be your best ally in avoid random and random-like drawing behaviours.
13. State of the art 1.Call save() — or saveToCount() 2.Do your stuff 3.Restore original state: restore() — or restoreToCount() So, how to deal with transformations on a Canvas? In your onDraw() method, you just have to remember to do everything this way: 1. Save the Canvas state before doing anything by calling save(). Use saveToCount() if you want to do more than one save/restore, or if you don’t trust subclasses to balance off their save/restores. 2. Do everything you need: transform, draw, set paints, push pixels. 3. Before leaving onDraw remember to call restore(), or random stuff might — or rather, will — happen. Which you probably don’t want.
14. Seb’s top tip Here’s a Seb top tip: how to draw a bunch of stuff on a canvas with a non fully opaque alpha. It’s not as trivial as you might think to achieve in some cases. But then it is if you use this trick.
15. Seb’s top tip Alpha composite overlaps, the framework way 1.int state = saveLayerAlpha(…); 2.Do your stuff 3.Blit back: restoreToCount(state); If you need to paint overlapping objects, you can’t use the Paint alpha property, because they’ll add up and you’ll end up with less transparent areas. Instead what you want to do is the same thing the framework does when you set a View’s alpha. 1. Save the current Canvas state using saveLayerAlpha(). This method is like save() but takes an alpha value and it also allocates an offscreen buffer. All Canvas calls from now on will happen on that buffer. 2. Paint whatever you want to paint, without caring for the alpha. 3. When you do a balancing restore() call, the contents of the buffer get blitted back on the Canvas with the alpha you set.
16. Surfacing ideas CC-BY-NC-SA arbyreed on Flickr This is all nice and well, but how does what you paint get on the screen?
17. Surfaces and flingers • Draw on surfaces • Buffers • SurfaceFlinger • SW or HW backing CC-BY-NC-SA Nicolas Hoizey on Flickr When you talk about drawing on a Canvas, you’re actually drawing on a Surface. A surface is basically a buffer, as we already called them when talking about Skia. ! Surfaces are composited together to be shown on screen by the SurfaceFlinger, which manages all the offscreen buffers and coordinates them to be shown on screen. ! Surfaces can be backed in hardware, in which case they are also said to be HW composed, or in software. HW composed surfaces are managed by the instance of SurfaceFlinger that is backed by an HardwareComposer. HWComposer is a HAL for a memory buffer hosted outside of system memory, usually on a 2D blitter, that can also be the GPU. ! When the HWComp cannot handle a surface (because it might not have enough memory available, because it’s been asked not to, etc) then a surface can be backed in SW. That means it resides on system memory, and is managed by an instance of SurfaceFlinger that is backed by OpenGL ES 1.
18. Good read AOSP Graphics on AOSP Source website http://goo.gl/i5iVwH The graphics subsystem in Android is incredibly complex and a slide is barely enough to name the main components. I strongly recommend you to read the fairly recent documentation on the AOSP source website. It’s long and can be quite hard for a Java dev to understand it from time to time, but it’s definitely worth it if you want to understand what’s going on under the hood. ! Besides, it’s one of the very few examples of good documentation in AOSP, so might as well use what we get.
19. Layers • Offscreen buffers • Every Canvas has one • Can be blitted with a Paint • Different types of layer CC-BY-NC-SA Doug88888 on Flickr A Layer is basically an offscreen buffer that you can paint onto. Does this remind you of anything? Yep, they’re basically mapped to surfaces. ! Every Canvas has a backing layer, and you can create more layers as needed, to use as buffers, as we’ve just seen in the previous slide. ! Views can have an offscreen layer, and use it to achieve some special effects when compositing back the result. In particular, it’s interesting that you can use any Paint object to have the buffer blitted back on the Canvas. You can use this Paint to apply Shaders, for example to tweak the brightness/hue/…, or some other color transformation. ! There are different layer types, which we’ll now see.
20. View Layer types (3.0+) • NONE: no buffer • SOFTWARE: bitmap buffer • HARDWARE: HW-backed texture • Can improve performances (e.g., animations) On Android 3.0+ (API 11+) Google has introduced the setLayerType() method and thus different types of layers. All Canvases before then were backed by software layers. ! This allows you to specify the backing layer type for a View’s Canvas: - The NONE flag indicates that the Canvas doesn’t have any backing layer (buffer), either SW or HW - The SOFTWARE flag indicates that the Canvas is backed by an in-memory drawing buffer. This is useful to use features not available with HW acceleration - The HARDWARE flag indicates that the Canvas is backed by an hardware texture buffer on the GPU (where available) ! Using offscreen buffers can help improve performance. In the case of HW textures, they’re kept around until the invalidate() method is called, so the Views can be redrawn without calling onDraw again until needed. In the case of SW bitmaps, they are used as a cache by the View class, to achieve a similar result. ! Besides that, using offscreen HW buffers can also help saving time when blitting with some kind of effects. For example, if you use setAlpha() on a View that has a backing layer, the framework will not have to call onDraw again to change the alpha; it will simply blit with a different Alpha level. This is
21. CC-BY Brandon Morse on Flickr Pushing pixels What happens under the hood in Skia then, from the moment you issue a draw command to the point where you see the results on screen? Let’s take a look at the Skia pipeline.
22. Canvas API Paint Shape Path Matrix Color Drawable Bitmap Camera Movie NinePatch Picture Interpolator SurfaceTexture Typeface The Canvas API has got a lot of things in it. This is just a small part of all the classes you will have to deal with. And yet, we barely have the time to cover one, which is arguably the most important one:
23. Drawable Canvas API Shape Path Color Bitmap Camera Movie Picture Interpolator SurfaceTexture Typeface Matrix NinePatch Paint Paint.
24. Just deal with it. ! Also, hello Tim! This gif’s for ya.
25. Paint it (black) • Knows how to colour the pixels • All drawing calls need a Paint • Handles text as well • TextPaint subclass CC-BY-NC-ND Mark Chadwick on Flickr Paint is the object that holds the informations about how to paint something on a Canvas. Actually, all drawing operations require that you pass a Paint along. It’s really the intuitive idea of the paint that you use to draw something in real life: if you have a canvas and a brush but no paint, there’s no way you’re ever going to be able to draw anything. ! Paint holds informations about the usual basic properties about how to draw: the fill color, the stroke (color, width, caps, etc), the alpha level, etc. It also holds informations about the antialiasing to apply, the bitmap filtering (off by default) and the text hinting. Of course not all of Paint’s properties are useful for all kinds of drawing operations. ! Paint knows also how to measure and draw text. It holds informations on the Typeface to use, the text size, style, scaling ratio, etc. It offers methods to measure text size (all of which do a pretty bad job at it, and each of them has a nuanced different meaning from the others, but that’s really confused and difficult to understand), and there’s even a text-specific subclass, TextPaint, that holds more informations about the text should be measured and painted. ! The Paint object you pass to a drawing call is used throughout the painting pipeline.
26. Skia pipeline Adapted from Laurence Gonsalves’ post on Xenomachina The rendering pipeline for drawing calls in Skia is composed of four main stages: path generation, rasterization, shading and transfer. This is a simplified view of the general pipeline; it’s worth noting that there are several shortcuts and optimisations that can change the way this works. ! The first stage, path generation, computes a Path that contains the region that will be painted. This path is then fed into the Rasterization step, which can happen in two different ways. ! If the user has specified a Rasterizer, then that’s going to take the whole step onto itself and generate the final alpha mask. If not, the Path is trivially rasterised using a scan conversion (including semi-transparent pixels on anti-aliased edges). ! The next step (again, this might even happen in parallel to the previous steps) is Shading, which determines the colour that the pixel will be painted with. ! This outputs an image that is then used in conjunction with the mask in the last step, Transfer. In this step the image and the mask are blended and painted on the destination Canvas.
27. Good read Android's 2D Canvas Rendering Pipeline by Laurence Gonsalves on Xenomachina http://goo.gl/4W5R0Z If you want to read in more detail how the Skia pipeline works, here’s an excellent article that I only found out recently. Of course I found it after I had spent an inordinate amount of time on doing my own research, because OF COURSE. So frustrating! But no, seriously, read it. It’s full of super interesting stuff.
28. Effective painting • Steps have two phases each • “Two-pass” strategy • Effects modify steps output • Second passes default to identity CC-BY-NC-SA ClintJCL on Flickr Each step output, as you could see in the flowchart, can be modified using the so-called Effects. Each step has generally two effects that you can apply, in a sort of “two-pass” strategy. Think of it as a first pass to achieve an intermediate result, and a second pass to refine it. ! There is an exception, which is…? Yes, the Rasterisation pass. If you assign a Rasterizer to your Paint, it will bypass the built-in rasterisation logic and also any MaskEffect you might have set. Fun. Even better, undocumented fun! Luckily enough you’ll probably never need to use a Rasterizer anyway, especially considering there only is one available, LayerRasterizer, which implements a matrioska of shaders. ! In the pipeline, if you don’t specify any custom effect, Skia will use its defaults. Usually, “second passes” effects default to the identity function, which means, they output what they receive as input, unaltered.
29. Mask filters • Rasterization phase • Affect alpha mask • Not HW accelerated • Blur and Emboss P S T R Mask filters allow you to affect the alpha mask of the paint, before it is colorized and drawn. The alpha mask is, in fact, an ALPHA_8 bitmap that is generated by rasterising the Path resulting from the first step. Remember that the MaskFilters will be bypassed if you use a Rasterizer! ! In case for some reason you didn’t know what an Alpha channel is, but you still ended up in this session, it’s the channel that contains per-pixel information about the transparency. A value of 0 (conventionally depicted as black) means fully transparent, and a value of 255 (white as per conventions) means fully opaque. ! The image shows how the Blur filter affects the alpha mask for a circle. The top half is drawn on a software layer, the bottom part on a HW layer. ! If you were wondering, YES, this means MaskFilters aren’t available on HW layers. You can set the layer to use a SOFTWARE buffer to get access to the MaskFilters, but remember that applying them can be pretty expensive, performance-wise. ! There’s another built-in Mask filter: Emboss. Nobody ever cared about embossing stuff for, like, ever.
30. Good read Blurring Images (series) by Mark Allison on Styling Android http://goo.gl/zZVs2V There are other ways to blur something. RenderScript is surely the best one, as it offers a Blur intrinsic. Remember that using RenderScript involves marshalling data through JNI, as it would if you were using native code, and that can be a bottleneck. Also, since we’re talking about alpha masks here, probably using a mask drawable is more efficient. ! In case you’re interested on how to use Renderscript to blur out stuff, you can find an in-depth series of articles on Mark Allison’s Styling Android blog. And then get him a beer. He deserves it.
31. Shaders • Shading phase • Similar to OpenGL shaders • Non programmable • TileMode CC-BY-NC-SA Andreas Köberle on Flickr P S T R Shaders are used in the Shading phase (duh!) to determine the colour of each pixel that will be drawn. ! Conceptually, a Skia Shader is kinda similar to an OpenGL shader (but not really). There’s a huge difference, which is, in Skia shaders aren’t programmable — not on Java, anyway. This means they’re basically immutable, which means that to alter a Paint’s Shader you must create a new one. Yay GC! ! Good news is, this is supposed to change in future versions of Android.
32. minSdkLevel ‘over9000’ Bad news is, see you in a few years’ time before we can actually use them. Go, minSdkLevel ‘over9000’!
33. Shaders • Shading phase • Similar to OpenGL shaders • Non programmable • TileMode CC-BY-NC-SA Andreas Köberle on Flickr P S T R Shaders aren’t affected by the Paint mask. If drawing outside of the shader bounds, you can use Shader.TileMode to specify if you want to CLAMP, MIRROR or REPEAT the shader contents outside of the clipping mask. ! CLAMP is usually the most efficient; if your bitmaps have power-of-two sizes, then…
34. …everything is awesome. Pretty much.
35. A Shady bunch Shader BitmapShader ComposeShader *Gradient Use a Bitmap as texture when painting Combine two different shaders and mix them with a Xfermode Paint using a Linear, Radial or Sweep gradient P S T R There are three main types of Shaders: - The BitmapShader is a Shader that loads a bitmap as a texture and uses it to paint the pixels - The Gradient shaders (LinearGradient, RadialGradient and SweepGradient) paint the pixels using the specified gradient - The ComposeShader allows you to combine two shaders together when painting, specifying the XferMode to use to mix them. ! Note that in HW accelerated canvases there is a very limited support for ComposeShaders (can’t nest them, can’t use two shaders of the same type).
36. Color Filters • Adjust colours after Shaders • Uniform transformation for all pixels ColorFilter ColorMatrixColorFilter LightingColorFilter PorterDuffColorFilter Apply a 4x5 colour matrix transform Multiply SRC colours by a colour, and then add a second colour to them Apply a colour to the SRC colour using a Porter-Duff mode P S T R Color filters are used to modify the paint colours after the Shaders have set them. They are the second phase of the Shading step. ! ColorFilters aren’t function of the pixel coordinates, which means, they apply the same transformation to all pixels. ! There are three types of ColorFilters: - The ColorMatrixColorFilter transforms the colors in each pixel by using a 4x5 ColorMatrix. This can be used for changing the Brightness/Saturation/Hue, to convert between RGB and YUV color spaces, etc. You can also concatenate matrices to achieve composite transforms. - The LightingColorFilter allows you to premultiply all pixels’ colors by a specified color, and then offset the result by a second color. All values are limited within the [0, 255] range - The PorterDuffColorFilter applies a single color to every pixel, mixing it with the source color using the specified Porter-Duff mode
37. Transfer modes • Second step of Transfer phase • Blend SRC image onto DST through mask Xfermode AvoidXfermode PixelXorXfermode PorterDuffXfermode Draw (or don’t draw) pixels based on the “distance” from a reference colour Exclusive-OR source and destination pixels. Drops the alpha channel! Blends the source and destination colours using a Porter-Duff mode P S T R Transfer modes (Xfermodes for the friends) are the second phase of the Transfer step, and the last of the pipeline. ! Transfer modes are used to determine the way the shaded source image (SRC) is blended with the destination (DST), through the mask. ! There are three Xfermodes in Skia: - The AvoidXfermode draws, or doesn’t draw, pixels that are within a certain colorimetric distance (tolerance) from the specified key color. - The PixelXorXfermode XORs all the pixels of the source and destination. Since this Xfermode doesn’t follow premultiplied conventions, the resulting pixels will always hava a solid alpha (255). - The PorterDuffXfermode mixing the destination with the source pixel colors using the specified Porter-Duff mode.
38. Q&A pipeline outline code? onDraw jdoc — canvas — views
39. We’re hiring! join@novoda.com If you like what you’ve seen and you’d want to be working on awesome Android stuff then drop us an email! We don’t generally bite.