View To Pixel 2.0 (Droidcon SF 17')

On what's going on in the Android system when it created pixels on the screen, out of the Views that we write in code.

Also here: https://docs.google.com/presentation/d/1__ezgkz5DzFWxHGy4p0wx5NLr96aML5_asskM6OsEJM/present#slide=id.g299bece29b_1_927

Presented in Droidcon SF 2017.

Transcript

1. View To Pixel 2.0
   Britt Barak
   DroidconSF 2017
   

   View Slide

2. View Slide

3. Android Lead
   Android Academy TLV
   Women Techmakers IL
   Britt Barak
   

   View Slide

4. Today’s story
   

   View Slide

5. CPU
   new
   View()
   

   View Slide

6. Rasterization
   From a high level object
   into pixels on screen (or in texture).
   

   View Slide

7. GPU -
   For The Rescue !
   

   View Slide

8. CPU
   new
   View()
   GPU
   

   View Slide

9. CPU
   new
   View()
   GPU
   Polygons
   Textures
   

   View Slide

10. CPU
    new
    View()
    GPU
    Polygons
    Textures
    DisplayList
    

    View Slide

11. Refresh Rate
    60
    Hz
    

    View Slide

12. CPU
    new
    View()
    GPU
    Polygons
    Textures
    DisplayList
    Frame Rate
    60
    fps
    

    View Slide

13. What
    happens
    if...
    

    View Slide

14. CPU GPU
    

    View Slide

15. CPU GPU
    

    View Slide

16. CPU GPU
    

    View Slide

17. View Slide

18. What
    do we do?
    

    View Slide

19. Double Buffer
    

    View Slide

20. Double Triple Buffer
    

    View Slide

21. CPU
    new
    View()
    GPU
    Polygons
    Textures
    DisplayList
    Back
    Buffer
    Frame
    Buffer
    

    View Slide

22. CPU
    new
    View()
    GPU
    Polygons
    Textures
    DisplayList
    Back
    Buffer
    Frame
    Buffer
    

    View Slide

23. CPU
    new
    View()
    GPU
    Polygons
    Textures
    DisplayList
    Back
    Buffer
    Frame
    Buffer
    

    View Slide

24. CPU
    new
    View()
    GPU
    Polygons
    Textures
    DisplayList
    Back
    Buffer
    Frame
    Buffer
    

    View Slide

25. CPU
    new
    View()
    GPU
    Polygons
    Textures
    DisplayList
    Back
    Buffer
    Frame
    Buffer
    

    View Slide

26. VSync
    

    View Slide

27. In some more details...
    

    View Slide

28. CPU GPU
    Vsync
    

    View Slide

29. CPU
    new
    View()
    GPU
    Measure
    

    View Slide

30. CPU GPU
    Layout
    

    View Slide

31. CPU GPU
    new
    View()
    DisplayList
    Draw
    

    View Slide

32. CPU
    new
    View()
    GPU
    Polygons
    Textures
    DisplayList
    Sync & Upload
    

    View Slide

33. CPU
    new
    View()
    GPU
    Polygons
    Textures
    DisplayList
    Execute
    Back Buffer
    

    View Slide

34. CPU
    new
    View()
    GPU
    Polygons
    Textures
    DisplayList
    Swap buffers
    Back Buffer
    

    View Slide

35. Measure/ Layout
    Draw
    Sync & Upload
    Execute
    Swap Buffers
    VSync/Misc
    

    View Slide

36. actually...
    

    View Slide

37. Input
    

    View Slide

38. Animation
    

    View Slide

39. VSync/Misc
    Input
    Animation
    Measure/ Layout
    Draw
    Sync & Upload
    Command Issue
    Swap Buffers
    

    View Slide

40. Let’s go a bit deeper...
    

    View Slide

41. Window
    Manager
    

    View Slide

42. Window
    Manager
    lifecycles, input events, focus events, screen orientation,
    transitions, animations, position, transforms, z-order...
    

    View Slide

43. ● Knows all surfaces
    ● Only service that can modify display’s content
    SurfaceFlinger
    

    View Slide

44. Status bar
    App UI
    Navigation bar
    

    View Slide

45. “Listen, I have
    another ‘woower’...”
    Window
    Manager
    SurfaceFlinger
    

    View Slide

46. “K”
    Window
    Manager
    SurfaceFlinger
    

    View Slide

47. Window
    Manager
    SurfaceFlinger
    Surface
    BufferQueue
    

    View Slide

48. Window
    Manager
    SurfaceFlinger
    Surface
    BufferQueue
    

    View Slide

49. Surface
    BufferQueue
    Buffer Pool
    Queue
    Gralloc
    (HAL interface)
    

    View Slide

50. App
    Window
    Manager
    “Here’s a Binder
    object!”
    

    View Slide

51. App
    “I have something to
    display!”
    Binder
    Buffer
    Queue
    

    View Slide

52. App
    Binder
    Buffer
    Queue
    “Here’s a free
    buffer”
    

    View Slide

53. App
    Binder
    Buffer
    Queue
    

    View Slide

54. App
    Binder
    Buffer
    Queue
    “Done!”
    

    View Slide

55. App
    Binder
    Buffer
    Queue
    

    View Slide

56. Binder
    Buffer
    Queue
    “I’m ready!”
    SurfaceFlinger Display
    VSYNC
    “K”
    

    View Slide

57. Binder
    Buffer
    Queue SurfaceFlinger Display
    Binder
    Buffer
    Queue
    Binder
    Buffer
    Queue
    

    View Slide

58. SurfaceFlinger Display
    Hardware
    Composer
    HAL
    Binder
    Buffer
    Queue
    Binder
    Buffer
    Queue
    Binder
    Buffer
    Queue
    

    View Slide

59. SurfaceFlinger Display
    Hardware
    Composer
    HAL
    Binder
    Buffer
    Queue
    Binder
    Buffer
    Queue
    Binder
    Buffer
    Queue
    

    View Slide

60. Let’s talk some more about
    creating the frame...
    

    View Slide

61. VSync/Misc
    Input
    Animation
    Measure/ Layout
    Draw
    Sync & Upload
    Command Issue
    Swap Buffers
    CPU
    

    View Slide

62. 1. Order is guaranteed
    2. A step is called up to once
    

    View Slide

63. Choreographer
    

    View Slide

64. I/Choreographer(1378): Skipped 12 frames!
    The application may be doing too much work on its
    main thread.
    

    View Slide

65. Process
    Main (/UI) Thread
    

    View Slide

66. Main Thread (UI Thread)
    Process
    System Event Input Event Service Application UI Drawing
    

    View Slide

67. Java Thread
    run()
    start()
    

    View Slide

68. Main Thread (UI Thread)
    Looper
    Message queue
    Process
    

    View Slide

69. Main Thread (UI Thread)
    Looper
    Message queue
    Handler
    Handle
    MSG
    Send
    MSG
    Process
    

    View Slide

70. Main Thread (UI Thread)
    Looper
    Message queue
    Handler1
    Handle
    MSG
    Send
    MSG
    Process
    Handler2
    Handle
    MSG
    Send
    MSG
    

    View Slide

71. private Choreographer
    (Looper looper, int vsyncSource) ;
    

    View Slide

72. Choreographer.java
    private final class FrameHandler extends Handler {
    @Override
    public void handleMessage(Message msg) {
    switch (msg.what) {
    case MSG_DO_FRAME:
    doFrame(System.nanoTime(), 0);
    break;
    case MSG_DO_SCHEDULE_VSYNC:
    doScheduleVsync();
    break;
    case MSG_DO_SCHEDULE_CALLBACK:
    doScheduleCallback(msg.arg1);
    break;
    }
    }
    }
    

    View Slide

73. 1. Order is guaranteed.
    2. Steps happen up to once
    Choreographer.java
    void doFrame(long frameTimeNanos, int frame) {
    //...
    doCallbacks(Choreographer.CALLBACK_INPUT, frameTimeNanos);
    doCallbacks(Choreographer.CALLBACK_ANIMATION, frameTimeNanos);
    doCallbacks(Choreographer.CALLBACK_TRAVERSAL, frameTimeNanos);
    doCallbacks(Choreographer.CALLBACK_COMMIT, frameTimeNanos);
    //…
    

    View Slide

74. Choreographer.java
    void doCallbacks(int callbackType, long frameTimeNanos) {
    callbacks = mCallbackQueues[callbackType]
    .extractDueCallbacksLocked(now / TimeUtils.NANOS_PER_MS);
    for (CallbackRecord c = callbacks; c != null; c = c.next) {
    //…
    c.run(frameTimeNanos);
    

    View Slide

75. Measure Layout Draw!
    

    View Slide

76. What can we do?
    ● Flatten hierarchy
    ● on
    ● Avoid when near the hierarchy top
    ● Avoid when many layouts
    ● Minimize requestLayout()
    ● ConstrainLayout
    

    View Slide

77. Draw()
    

    View Slide

78. View.java
    public void draw(Canvas canvas)
    

    View Slide

79. //Step 1: draw background
    drawBackground(canvas);
    // Step 2, save the canvas' layers (prepare Fading, optional)
    canvas.saveLayer(…)
    //Step 3, draw view's content
    onDraw(canvas);
    public void draw(Canvas canvas) {
    

    View Slide

80. // Step 4, draw children
    dispatchDraw(canvas);
    // Step 5, draw the fade effect and restore layers (optional)
    final Shader fade = scrollabilityCache.shader;
    paint.setShader(fade);
    canvas.drawRect(left, top, right, top , paint);
    public void draw(Canvas canvas) {
    

    View Slide

81. // Step 6, draw decorations (foreground, scrollbars)
    onDrawForeground(canvas);
    // Step 7, draw the default focus highlight
    drawDefaultFocusHighlight(canvas);
    public void draw(Canvas canvas) {
    

    View Slide

82. background.draw(canvas)
    scrollBar.draw(canvas)
    foreground.draw(canvas)
    

    View Slide

83. GPU Overdraw Tool
    https://developer.android.com/studio/profile/dev-options-overdraw.html
    

    View Slide

84. A bit more about Canvas
    

    View Slide

85. 1. Primitives
    Canvas.
    drawPath(), drawOval(), drawRect()
    

    View Slide

86. 2. PathEffect
    paint.setPathEffect()
    ComposePathEffect, CornerPathEffect, DashPathEffect,
    DiscretePathEffect, PathDashPathEffect, SumPathEffect
    

    View Slide

87. 3. Alpha
    Decide alpha channel for each pixel
    paint.setAlpha()
    

    View Slide

88. 4. Shading
    determine each pixel’s color (ARGB).
    paint.setColor(), paint.setColorFilter(), paint.setShader()
    Chiu-Ki Chan: Fun With Android Shaders And Filters
    

    View Slide

89. public void draw(Canvas canvas) {
    //...
    // Step 4, draw children
    dispatchDraw(canvas);
    // Step 5, draw the fade effect and restore layers (optional)
    final Shader fade = scrollabilityCache.shader;
    paint.setShader(fade);
    canvas.drawRect(left, top, right, top , paint);
    

    View Slide

90. Shader
    paint.setShader(shader)
    LinearGradient (on fade), RadialGradient,
    SweepGradient, BitmapShader, ComposeShader
    https://developer.android.com/reference/android/graphics/Shader.html
    

    View Slide

91. 5. Transfer
    How to transfer color to destination Bitmap
    paint.setXfermode()
    Source Image Destination Image Transfer Mode
    Defaults to :
    PorterDuffXferMode
    (SRC_OVER)
    https://developer.android.com/reference/android/graphics/PorterDuffXfermode.html
    

    View Slide

92. View Slide

93. View Slide

94. View Slide

95. Binder
    Buffer
    Queue
    surface
    canvas
    openGL
    openGLES
    SurfaceFlinger
    

    View Slide

96. VSync/Misc
    Input
    Animation
    Measure/ Layout
    Draw
    Sync & Upload
    Command Issue
    Swap Buffers
    

    View Slide

97. GPU Profiling Tool
    ● A graph per visible app
    ● A bar per frame
    ● 16ms benchmark
    ● Crossing = skipping frame
    https://developer.android.com/studio/profile/dev-options-rendering.html
    

    View Slide

98. Frame Metrics
    On Android N+
    - Per-Frame data:
    - Window.OnFrameMetricsAvailableListener()
    - Aggregated data:
    - FrameMetricsAggregator (Support lib v26)
    https://developer.android.com/reference/android/view/FrameMetrics.html
    

    View Slide

99. View Slide

100. Thank you!
     Keep in touch!
     Britt Barak
     @brittBarak
     

     View Slide