Fluid User Interface with Hardware Acceleration

Talk at W3Conf 2013, San Francisco.

Transcript

1. Fluid User Interface with
   Hardware Acceleration
   Ariya Hidayat
   San Francisco
   Feb 22, 2013
   @ariyahidayat
   1
   

   View Slide

2. View Slide

3. Magical Advice
   Use translate3d for
   hardware acceleration
   3
   

   View Slide

4. whoami
   4
   

   View Slide

5. CSS3
   5
   

   View Slide

6. Web Browser from 10,000 ft
   6
   

   View Slide

7. Browser: High Level
   User Interface
   Browser Engine
   Graphics
   Stack
   Data Persistence
   Render Engine
   JavaScript
   Engine
   Networking
   I/O
   7
   

   View Slide

8. From Contents to Pixels
   HTML
   Parser
   DOM
   CSS
   Parser
   HTML
   Style
   Sheets
   DOM
   Tree
   Style
   Rules
   Render
   Tree
   Render
   Layout
   Paint
   http://www.html5rocks.com/en/tutorials/internals/howbrowserswork
   8
   

   View Slide

9. DOM Tree
   Hello
   HTMLDocument
   HTMLBodyElement
   HTMLParagraphElement attributes, children, contents
   9
   

   View Slide

10. DOM Tree vs Render Tree
    HTMLDocument
    HTMLBodyElement
    HTMLParagraphElement
    RenderRoot
    RenderBody
    RenderText
    tree structure/navigation
    metrics (box model)
    hit testing
    RenderStyle
    computed style
    many:1
    10
    

    View Slide

11. Style Recalc vs Layout
    document.getElementById('box').style.top = '100px';
    Aggregated “dirty” area
    document.getElementById('box').style.backgroundColor = 'red';
    No layout necessary,
    metrics are not flagged as “changed”
    11
    

    View Slide

12. Minimizing Layout
    http://gent.ilcore.com/2011/03/how-not-to-trigger-layout-in-webkit.html
    http://code.google.com/p/chromium/source/search?q=%22-%3EupdateLayoutIgnorePendingStylesheets()%22
    clientHeight
    clientLeft
    clientTop
    clientWidth
    offsetHeight
    offsetLeft
    offsetParent
    offsetTop
    offsetWidth
    scrollLeft
    scrollTop
    scrollWidth
    innerText
    outerText
    Element
    focus()
    getBoundingClientRect()
    getClientRects()
    scrollByLines()
    scrollByPages()
    scrollHeight
    scrollIntoView()
    scrollIntoViewIfNeeded()
    12
    

    View Slide

13. CSS Painting: 10 Stages
    http://www.w3.org/TR/CSS2/zindex.html
    Appendix E. Elaborate description of Stacking Contexts
    •
    Background
    •
    Floats
    •
    Foreground
    •
    Selection
    •
    Outline
    Done by the render objects
    (in the render tree)
    13
    

    View Slide

14. Graphics Stack: Crash Course
    14
    

    View Slide

15. Path is Everything
    15
    

    View Slide

16. Stroke = Outline
    Solid Dashed Dotted Textured
    16
    

    View Slide

17. Brush = Fill
    Solid
    None Gradient Textured
    17
    

    View Slide

18. Path Approximation
    Curves
    Polygon
    18
    

    View Slide

19. Antialiasing
    19
    

    View Slide

20. Transformation
    Scaling
    Rotation
    Perspective
    20
    

    View Slide

21. Graphics Processing Unit
    “Fixed” geometry
    Transformation
    Textured triangles
    Parallelism
    21
    

    View Slide

22. Optimized for Games
    http://epicgames.com/technology/epic-citadel http://www.rage.com/
    22
    

    View Slide

23. GPU Workflow
    Vertices Rendered Textured
    Matrix
    23
    

    View Slide

24. Web Page Rendering
    24
    

    View Slide

25. Game vs Web Page
    Predictable contents (assets) ✔
    Mostly text
    ✔
    Mostly images
    ✔
    immediate
    Initial response
    25
    

    View Slide

26. GPU Physical Limitations
    • Memory: Can’t store too much stuff
    • Speed: Quad-core CPU can do certain operations better
    • Bandwidth: Data transfer can be the bottleneck
    • Power: Electrons always need energy
    26
    

    View Slide

27. Fluid Animation
    At the beginning, push as many resources
    as possible to the GPU
    During the animation, minimize
    the interaction between CPU-GPU
    1
    2
    27
    

    View Slide

28. Immediate vs Retained
    draw the shape at (x, y)
    x = x + 10
    blit the buffer at (x, y)
    x = x + 10
    For every animation tick...
    At the beginning...
    draw the shape onto
    a buffer
    Off main thread
    28
    

    View Slide

29. Animation Mechanics
    Initialization
    Animation step
    “Hey, this is good stuff. Cache it as texture #42.”
    “Apply [operation] to texture #42.”
    Animation loop
    29
    

    View Slide

30. Element = Layer
    http://www.webkit.org/blog-files/leaves/
    30
    

    View Slide

31. Compositing: “Logical” 3-D
    Compositor
    Tweening
    Actual operation
    31
    

    View Slide

32. By Force
    transform: translateZ(0)
    Not needed for CSS animation!
    .someid {
    animation-name: somekeyframeanimation;
    animation-duration: 7s;
    transform: translateZ(0);
    }
    Don’t do that
    32
    

    View Slide

33. Tools & Examples
    33
    

    View Slide

34. Debugging in Safari
    defaults write com.apple.Safari IncludeInternalDebugMenu 1
    https://developer.apple.com/videos/wwdc/2012/?id=601
    34
    

    View Slide

35. Compositing Indicator
    Composited layer
    Container layer
    No associated texture
    Overflow
    Texture (number = upload)
    35
    

    View Slide

36. Texture Upload
    No reupload Upload
    Opacity
    Position
    Size
    Filter
    Everything else!
    “Hardware accelerated CSS”
    36
    

    View Slide

37. Constant Upload = Bad
    http://codepen.io/ariya/full/xuwgy
    translate3d ???
    37
    

    View Slide

38. Transformation
    http://ariya.github.com/css/spinningcube/ http://dev.sencha.com/animator/demos/cogs/
    38
    

    View Slide

39. Fade In/Out
    http://dev.sencha.com/animator/demos/ions/
    http://codepen.io/ariya/full/spzad
    39
    

    View Slide

40. three.js Periodic Table
    http://mrdoob.github.com/three.js/examples/css3d_periodictable.html
    40
    

    View Slide

41. Montage MovieShow
    http://montagejs.github.com/montage/samples/popcorn/
    41
    

    View Slide

42. Photon CSS Lighting
    http://photon.attasi.com
    42
    

    View Slide

43. FPS View
    http://www.keithclark.co.uk/labs/css3-fps/
    43
    

    View Slide

44. Filter
    http://html.adobe.com/webstandards/csscustomfilters/cssfilterlab/
    44
    

    View Slide

45. Hacks & Tricks
    45
    

    View Slide

46. Traffic Congestion
    The speed of the car
    vs the traffic condition
    46
    

    View Slide

47. Avoid Overcapacity
    ....
    ....
    Texture upload
    Think of the GPU memory like a cache.
    47
    

    View Slide

48. Prepare and Reuse
    Hide the poster (=layer) offscreen
    Viewport
    48
    

    View Slide

49. Color Transition
    @keyframes box {
    0% { transform: background-color: blue; }
    100% { transform: background-color: green; }
    }
    Need a new texture uploaded to
    the GPU for every in-between color
    49
    

    View Slide

50. Color Transition Trick
    Blended with
    http://codepen.io/ariya/full/ofDIh
    50
    

    View Slide

51. Timer Latency
    Depending on
    user reaction
    Animation end triggers the JavaScript callback
    JavaScript code kicks the next animation
    Prepare both
    animation and hide the
    “wrong” one
    51
    

    View Slide

52. More Stuff
    Jank Busters: http://www.youtube.com/watch?v=hAzhayTnhEI
    Continuous painting:
    http://updates.html5rocks.com/2013/02/Profiling-Long-Paint-Times-with-
    DevTools-Continuous-Painting-Mode
    Chrome GPU rendering benchmark (Telemetry):
    http://www.chromium.org/developers/design-documents/rendering-benchmarks
    Skia debugger:
    http://wesleyhales.com/blog/2013/02/18/Adventures-With-the-Skia-Debugger/
    52
    

    View Slide

53. Final Words
    53
    

    View Slide

54. http://www.trollquotes.org/619-super-spider-bat-troll-quote/
    54
    

    View Slide

55. Use CSS3 animation & transition
    Be advised of texture uploads
    Never assume, always measure
    55
    

    View Slide

56. There is No Silver Bullet
    Traditional graphics
    programming has been
    always full of tricks.
    It will always be.
    56
    

    View Slide

57. Thank You
    ariya.hidayat@gmail.com
    @AriyaHidayat
    ariya.ofilabs.com
    Background artwork from http://kde-look.org/content/show.php/Sin+Flow?content=139751 CC BY-NC-SA
    speakerdeck.com/ariya
    57
    

    View Slide