The Challenges Behind a Better Sticker Input Experience

Jiaheng Huang
LINE Fukuoka Development H Team Software Engineer
Yu Yang
LINE Fukuoka Development H Team Android Engineer
https://linedevday.linecorp.com/2020/ja/sessions/7642
https://linedevday.linecorp.com/2020/en/sessions/7642

Transcript

2. Agenda › An introduction of the new LINE sticker keyboard › A deep dive on the rendering performance issue on Android › Create smooth expand & collapse animations on iOS
3. What’s new in the sticker keyboard?
4. What’s new in the sticker keyboard?
5. What’s new in the sticker keyboard?
6. What’s new in the sticker keyboard?
7. What’s new in the sticker keyboard?
8. Yang Yu A deep dive on the rendering performance issue on Android
10. Our initial solution… <androidx.recyclerview.widget.RecyclerView 
 android:id="@+id/sticker_grid_view" 
 android:layout_width="match_parent" 
 android:layout_height="wrap_content"/>
11. Our initial solution… <androidx.recyclerview.widget.RecyclerView 
 android:id="@+id/sticker_grid_view" 
 android:layout_width="match_parent" 
 android:layout_height="wrap_content"/> 
 
 <androidx.recyclerview.widget.RecyclerView 
 android:id=“@+id/author_latest_stickers_view" 
 android:layout_width="match_parent" 
 android:layout_height="wrap_content" />
12. Our initial solution… <androidx.core.widget.NestedScrollView 
 android:id="@+id/nested_scroll_view" 
 android:layout_width="match_parent" 
 android:layout_height="match_parent"> 
 
 <LinearLayout 
 android:layout_width="match_parent" 
 android:layout_height="wrap_content" 
 android:orientation="vertical"> 
 
 <androidx.recyclerview.widget.RecyclerView 
 android:id="@+id/sticker_grid_view" 
 android:layout_width="match_parent" 
 android:layout_height="wrap_content"/> 
 
 <androidx.recyclerview.widget.RecyclerView 
 android:id="@+id/author_latest_stickers_view" 
 android:layout_width="match_parent" 
 android:layout_height="wrap_content" /> 
 
 </LinearLayout> 
 
 </androidx.core.widget.NestedScrollView>
13. But, then we found…
14. But, then we found…
15. How Android draws views? ViewGroup View View DecorView ViewGroup
16. How Android draws views? ViewGroup View View DecorView ViewGroup Pre-order traversal of the layout tree
17. How Android draws views? ViewRootImpl.java 
 private void performTraversals() {
 }
18. How Android draws views? ViewRootImpl.java 
 private void performTraversals() {
 ... 
 int childWidthMeasureSpec = getRootMeasureSpec(mWidth, lp.width); 
 int childHeightMeasureSpec = getRootMeasureSpec(mHeight, lp.height); 
 ... 
 performMeasure(childWidthMeasureSpec, childHeightMeasureSpec); 
 ... 
 performLayout(lp, mWidth, mHeight); 
 ... 
 performDraw(); 
 }
19. How Android draws views? ViewRootImpl.java 
 private void performTraversals() {
 ... 
 int childWidthMeasureSpec = getRootMeasureSpec(mWidth, lp.width); 
 int childHeightMeasureSpec = getRootMeasureSpec(mHeight, lp.height); 
 ... 
 performMeasure(childWidthMeasureSpec, childHeightMeasureSpec); 
 ... 
 performLayout(lp, mWidth, mHeight); 
 ... 
 performDraw(); 
 }
20. How Android draws views? ViewRootImpl.java 
 private void performTraversals() {
 ... 
 int childWidthMeasureSpec = getRootMeasureSpec(mWidth, lp.width); 
 int childHeightMeasureSpec = getRootMeasureSpec(mHeight, lp.height); 
 ... 
 performMeasure(childWidthMeasureSpec, childHeightMeasureSpec); 
 ... 
 performLayout(lp, mWidth, mHeight); 
 ... 
 performDraw(); 
 }
21. How Android draws views? ViewRootImpl.java 
 private void performTraversals() {
 ... 
 int childWidthMeasureSpec = getRootMeasureSpec(mWidth, lp.width); 
 int childHeightMeasureSpec = getRootMeasureSpec(mHeight, lp.height); 
 ... 
 performMeasure(childWidthMeasureSpec, childHeightMeasureSpec); 
 ... 
 performLayout(lp, mWidth, mHeight); 
 ... 
 performDraw(); 
 }
22. How Android draws views? Measure Layout Draw performTraversals
23. How Android draws views? Measure Layout Draw performTraversals For a child view to be measured, two specifications are required:
24. How Android draws views? Measure Layout Draw performTraversals › MATCH_PARENT › WRAP_CONTENT › an exact number (e.g.,100dp) ViewGroup.LayoutParams For a child view to be measured, two specifications are required:
25. How Android draws views? Measure Layout Draw performTraversals › MATCH_PARENT › WRAP_CONTENT › an exact number (e.g.,100dp) ViewGroup.LayoutParams MeasureSpec › UNSPECIFIED › EXACTLY › AT MOST For a child view to be measured, two specifications are required:
26. How a RecycleView inside a NestedScrollView is measured?
27. How a RecycleView inside a NestedScrollView is measured? NestedScrollView.java
 protected void measureChild(..., int parentWidthMeasureSpec, int parentHeightMeasureSpec) {
 }
28. How a RecycleView inside a NestedScrollView is measured? NestedScrollView.java
 protected void measureChild(..., int parentWidthMeasureSpec, int parentHeightMeasureSpec) {
 ...
 childWidthMeasureSpec = getChildMeasureSpec(...); 
 childHeightMeasureSpec = MeasureSpec.makeMeasureSpec(0, MeasureSpec.UNSPECIFIED);
 child.measure(childWidthMeasureSpec, childHeightMeasureSpec); 
 }
29. How a RecycleView inside a NestedScrollView is measured? NestedScrollView.java
 protected void measureChild(..., int parentWidthMeasureSpec, int parentHeightMeasureSpec) {
 ...
 childWidthMeasureSpec = getChildMeasureSpec(...); 
 childHeightMeasureSpec = MeasureSpec.makeMeasureSpec(0, MeasureSpec.UNSPECIFIED);
 child.measure(childWidthMeasureSpec, childHeightMeasureSpec); 
 }
30. How a RecycleView inside a NestedScrollView is measured? NestedScrollView.java
 protected void measureChild(..., int parentWidthMeasureSpec, int parentHeightMeasureSpec) {
 ...
 childWidthMeasureSpec = getChildMeasureSpec(...); 
 childHeightMeasureSpec = MeasureSpec.makeMeasureSpec(0, MeasureSpec.UNSPECIFIED);
 child.measure(childWidthMeasureSpec, childHeightMeasureSpec); 
 }
31. How a RecycleView inside a NestedScrollView is measured? RecyclerView.java 
 protected void onMeasure(int widthSpec, int heightSpec) { 
 ... 
 final int widthMode = MeasureSpec.getMode(widthSpec); 
 final int heightMode = MeasureSpec.getMode(heightSpec); 
 mLayout.onMeasure(mRecycler, mState, widthSpec, heightSpec); 
 final boolean measureSpecModeIsExactly = widthMode == MeasureSpec.EXACTLY && heightMode == MeasureSpec.EXACTLY; 
 if (measureSpecModeIsExactly || mAdapter == null) return; 
 ... 
 mLayout.setMeasuredDimensionFromChildren(widthSpec, heightSpec); 
 ... 
 }
32. How a RecycleView inside a NestedScrollView is measured? RecyclerView.java 
 protected void onMeasure(int widthSpec, int heightSpec) { 
 ... 
 final int widthMode = MeasureSpec.getMode(widthSpec); 
 final int heightMode = MeasureSpec.getMode(heightSpec); 
 mLayout.onMeasure(mRecycler, mState, widthSpec, heightSpec); 
 final boolean measureSpecModeIsExactly = widthMode == MeasureSpec.EXACTLY && heightMode == MeasureSpec.EXACTLY; 
 if (measureSpecModeIsExactly || mAdapter == null) return; 
 ... 
 mLayout.setMeasuredDimensionFromChildren(widthSpec, heightSpec); 
 ... 
 }
33. How a RecycleView inside a NestedScrollView is measured? RecyclerView.java 
 void setMeasuredDimensionFromChildren(int widthSpec, int heightSpec) { 
 ... 
 for (int i = 0; i < count; i++) { 
 View child = getChildAt(i); 
 final Rect bounds = mRecyclerView.mTempRect; 
 getDecoratedBoundsWithMargins(child, bounds); 
 if (bounds.left < minX) minX = bounds.left; 
 if (bounds.right > maxX) maxX = bounds.right; 
 if (bounds.top < minY) minY = bounds.top; 
 if (bounds.bottom > maxY) maxY = bounds.bottom; 
 } 
 mRecyclerView.mTempRect.set(minX, minY, maxX, maxY); 
 setMeasuredDimension(mRecyclerView.mTempRect, widthSpec, heightSpec); 
 }
34. How a RecycleView inside a NestedScrollView is measured? RecyclerView.java 
 void setMeasuredDimensionFromChildren(int widthSpec, int heightSpec) { 
 ... 
 for (int i = 0; i < count; i++) { 
 View child = getChildAt(i); 
 final Rect bounds = mRecyclerView.mTempRect; 
 getDecoratedBoundsWithMargins(child, bounds); 
 if (bounds.left < minX) minX = bounds.left; 
 if (bounds.right > maxX) maxX = bounds.right; 
 if (bounds.top < minY) minY = bounds.top; 
 if (bounds.bottom > maxY) maxY = bounds.bottom; 
 } 
 mRecyclerView.mTempRect.set(minX, minY, maxX, maxY); 
 setMeasuredDimension(mRecyclerView.mTempRect, widthSpec, heightSpec); 
 }
35. How a RecycleView inside a NestedScrollView is measured? RecyclerView.java 
 public void setMeasuredDimension(Rect childrenBounds, int wSpec, int hSpec) { 
 int usedWidth = childrenBounds.width() + getPaddingLeft() + getPaddingRight(); 
 int usedHeight = childrenBounds.height() + getPaddingTop() + getPaddingBottom(); 
 int width = chooseSize(wSpec, usedWidth, getMinimumWidth()); 
 int height = chooseSize(hSpec, usedHeight, getMinimumHeight()); 
 setMeasuredDimension(width, height); 
 }
36. How a RecycleView inside a NestedScrollView is measured? RecyclerView.java 
 public void setMeasuredDimension(Rect childrenBounds, int wSpec, int hSpec) { 
 int usedWidth = childrenBounds.width() + getPaddingLeft() + getPaddingRight(); 
 int usedHeight = childrenBounds.height() + getPaddingTop() + getPaddingBottom(); 
 int width = chooseSize(wSpec, usedWidth, getMinimumWidth()); 
 int height = chooseSize(hSpec, usedHeight, getMinimumHeight()); 
 setMeasuredDimension(width, height); 
 }
37. How a RecycleView inside a NestedScrollView is measured? RecyclerView.java 
 public void setMeasuredDimension(Rect childrenBounds, int wSpec, int hSpec) { 
 int usedWidth = childrenBounds.width() + getPaddingLeft() + getPaddingRight(); 
 int usedHeight = childrenBounds.height() + getPaddingTop() + getPaddingBottom(); 
 int width = chooseSize(wSpec, usedWidth, getMinimumWidth()); 
 int height = chooseSize(hSpec, usedHeight, getMinimumHeight()); 
 setMeasuredDimension(width, height); 
 }
38. How a RecycleView inside a NestedScrollView is measured? RecyclerView.java 
 public static int chooseSize(int spec, int desired, int min) { 
 final int mode = View.MeasureSpec.getMode(spec); 
 final int size = View.MeasureSpec.getSize(spec); 
 switch (mode) { 
 case View.MeasureSpec.EXACTLY: 
 return size; 
 case View.MeasureSpec.AT_MOST: 
 return Math.min(size, Math.max(desired, min)); 
 case View.MeasureSpec.UNSPECIFIED: 
 default: 
 return Math.max(desired, min); 
 } 
 }
39. How a RecycleView inside a NestedScrollView is measured? RecyclerView.java 
 public static int chooseSize(int spec, int desired, int min) { 
 final int mode = View.MeasureSpec.getMode(spec); 
 final int size = View.MeasureSpec.getSize(spec); 
 switch (mode) { 
 case View.MeasureSpec.EXACTLY: 
 return size; 
 case View.MeasureSpec.AT_MOST: 
 return Math.min(size, Math.max(desired, min)); 
 case View.MeasureSpec.UNSPECIFIED: 
 default: 
 return Math.max(desired, min); 
 } 
 }
40. How a RecycleView inside a NestedScrollView is measured? RecyclerView.java 
 public static int chooseSize(int spec, int desired, int min) { 
 final int mode = View.MeasureSpec.getMode(spec); 
 final int size = View.MeasureSpec.getSize(spec); 
 switch (mode) { 
 case View.MeasureSpec.EXACTLY: 
 return size; 
 case View.MeasureSpec.AT_MOST: 
 return Math.min(size, Math.max(desired, min)); 
 case View.MeasureSpec.UNSPECIFIED: 
 default: 
 return Math.max(desired, min); 
 } 
 }
41. Our initial solution…
42. Our initial solution… So, the height of the RecyclerView would not be restricted, and its height would be the total height of all child view items and the paddings (if any). In other words, all child view items will be rendered at once.
43. The combination of LayoutParams and MeasureSpec EXACTLY AT_MOST UNSPECIFIED DP EXACTLY childSize EXACTLY childSize EXACTLY childSize MATCH_PARENT EXACTLY parentSize AT_MOST parentSize UNSPECIFIED 0 WRAP_CONTENT AT_MOST parentSize AT_MOST parentSize UNSPECIFIED 0
44. The combination of LayoutParams and MeasureSpec EXACTLY AT_MOST UNSPECIFIED DP EXACTLY childSize EXACTLY childSize EXACTLY childSize MATCH_PARENT EXACTLY parentSize AT_MOST parentSize UNSPECIFIED 0 WRAP_CONTENT AT_MOST parentSize AT_MOST parentSize UNSPECIFIED 0
45. The combination of LayoutParams and MeasureSpec EXACTLY AT_MOST UNSPECIFIED DP EXACTLY childSize EXACTLY childSize EXACTLY childSize MATCH_PARENT EXACTLY parentSize AT_MOST parentSize UNSPECIFIED 0 WRAP_CONTENT AT_MOST parentSize AT_MOST parentSize UNSPECIFIED 0
46. How this problem could be solved?
47. How this problem could be solved? › Use an exact height value instead of WRAP_CONTENT for the RecyclerView.
48. How this problem could be solved? › DO NOT wrap a RecyclerView inside a NestedScrollView. Instead, use a single RecyclerView with multiple view types to achieve complex UI. › Use an exact height value instead of WRAP_CONTENT for the RecyclerView.
49. How this problem could be solved? › DO NOT wrap a RecyclerView inside a NestedScrollView. Instead, use a single RecyclerView with multiple view types to achieve complex UI. › Luckily LINE client has its own customized adapter called RecyclerViewModelAdapter to easily support multiple view types in a RecyclerView. › Use an exact height value instead of WRAP_CONTENT for the RecyclerView.
50. Finally, ! " #
51. Recap
52. Recap › Discussed how a RecyclerView inside a NestedScrollView is measured.
53. Recap › When MATCH_PARENT or WRAP_CONTENT is specified for a RecyclerView’s height inside a NestedScrollView, all list view items would rendered at once. › Discussed how a RecyclerView inside a NestedScrollView is measured.
54. Recap › When MATCH_PARENT or WRAP_CONTENT is specified for a RecyclerView’s height inside a NestedScrollView, all list view items would rendered at once. › Discussed how a RecyclerView inside a NestedScrollView is measured. › DO NOT put a RecyclerView inside a NestedScrollView unless the height is known in advance.
55. Johnny Huang Create Smooth Expand & collapse animation on iOS
56. Sticker Keyboard Category Tab
57. Challenge We don’t want to change keyboard height during expanding animation, otherwise the input bar will go up Keyboard is not full screen height, the expand animation will go beyond keyboard area
58. How to implement this? Calculate layout frame by frame › Too much verbose calculation
59. How to implement this? UIPropertyAnimator? Calculate layout frame by frame › Too much verbose calculation
60. UIPropertyAnimator › Introduced in iOS 10 › Can modify animation dynamically › eg. Scrub through a animation by modifying the fractionComplete property
61. How to implement this? › Need to handle hitTest(_:with:) through out the view hierarchy UIPropertyAnimator Calculate layout frame by frame
62. How to implement this? › Need to handle hitTest(_:with:) through out the view hierarchy › Not a good place to handle background dimming logic UIPropertyAnimator Calculate layout frame by frame
63. How to implement this? How about presenting a view controller here?
64. How to implement this? How about presenting a view controller here? › Present view controller works on keyboard
65. How to implement this? How about presenting a view controller here? › Present view controller works on keyboard › Still can take advantage of the powerful UIPropertyAnimator inside UIViewControllerTransitioningDelegate
66. How to implement this? How about presenting a view controller here? › Present view controller works on keyboard › Still can take advantage of the powerful UIPropertyAnimator inside UIViewControllerTransitioningDelegate › No need to handle hitTest(_:with:)
67. How to implement this? How about presenting a view controller here? › Present view controller works on keyboard › Still can take advantage of the powerful UIPropertyAnimator inside UIViewControllerTransitioningDelegate › No need to handle hitTest(_:with:) › Can handle background dimming logic and device rotation easily in UIPresentationController
68. Implementation
69. Implementation In View Controller Define two states enum State { case expanded, collapsed }
70. Implementation In View Controller Layout UI during view controller appear and disappear using transition coordinator Define two states enum State { case expanded, collapsed } transitionCoordinator?.animate(alongsideTransition: { [weak self] (_) in self?.configUI(.expanded) }, completion: nil)
71. TransitionCoordinator Transition Coordinator Presented View Controller Presenting View Controller Animator
72. Demo
73. Demo › Applied iOS 13 default presenting animation
75. › Applied iOS 13 default presenting animation › Has interactive dismiss but no interactive present › Incorrect initial frame of presentation Demo
76. UIViewControllerTransitioningDelegate UIViewController TransitioningDelegate Presented View Controller Presenting View Controller Animator Interactive Animator Presentation Controller
77. Implementation In UIViewControllerTransitioningDelegate Create a custom Presentation Controller func presentationController(forPresented presented: UIViewController, presenting: UIViewController?, source: UIViewController) -> UIPresentationController? › Define final layout of presented view controller › Add an extra view to do dimming background
78. Presentation Controller Define final layout class PresentationController: UIPresentationController { private var topSpacing: CGFloat override var frameOfPresentedViewInContainerView: CGRect { let containerSize = containerView!.bounds.size var frame: CGRect = .zero frame.size = size( forChildContentContainer: presentedViewController, withParentContainerSize: containerSize ) return frame } override func size( forChildContentContainer container: UIContentContainer, withParentContainerSize parentSize: CGSize ) -> CGSize { let size = CGSize(width: parentSize.width, height: parentSize.height - topSpacing) return size } }
79. Presentation Controller Add dimming view class PresentationController: UIPresentationController { private let dimmingView: UIView override func presentationTransitionWillBegin() { containerView?.insertSubview(dimmingView, at: 0) ... presentedViewController.transitionCoordinator?.animate(alongsideTransition: { _ in self.dimmingView.alpha = 0.7 }) } override func presentationTransitionDidEnd(_ completed: Bool) { if !completed { dimmingView.removeFromSuperview() } } override func dismissalTransitionWillBegin() { presentedViewController.transitionCoordinator?.animate(alongsideTransition: { _ in self.dimmingView.alpha = 0.0 }) } override func dismissalTransitionDidEnd(_ completed: Bool) { if completed { dimmingView.removeFromSuperview() } } }
80. Implementation In UIViewControllerTransitioningDelegate Create an Animator Object Create a custom Presentation Controller func animationController(forDismissed dismissed: UIViewController) -> UIViewControllerAnimatedTransitioning? func animationController(forPresented presented: UIViewController, presenting: UIViewController, source: UIViewController) -> UIViewControllerAnimatedTransitioning? { › Set correct initial frame for presented view controller in Animator › Use UIPropertyAnimator to define a interruptible animation
81. Animator class TagSearchResultTransitionAnimator: NSObject, UIViewControllerAnimatedTransitioning { let isPresenting: Bool private func transitionAnimator(using transitionContext: UIViewControllerContextTransitioning) -> UIViewImplicitlyAnimating { // Set Start position if isPresenting { container.addSubview(presentedView) presentedView.frame = startFrame } let springTiming = UISpringTimingParameters(…) let animator = UIViewPropertyAnimator(duration: duration, timingParameters: springTiming) animator.addAnimations { presentedView.frame = self.isPresenting ? finalFrame : startFrame } animator.addCompletion { position in let didComplete = !transitionContext.transitionWasCancelled // After a failed presentation or successful dismissal, remove the view. if self.isPresenting && !didComplete || !self.isPresenting && didComplete { presentedView.removeFromSuperview() } transitionContext.completeTransition(didComplete) } return animator } }
82. Implementation In UIViewControllerTransitioningDelegate Create an Animator Object Create a custom Presentation Controller Create an Interactive Animator › The easiest way to create an interactive animator is to subclass UIPercentDrivenInteractiveTransition class and add gesture recognizer event-handling code to your subclass func interactionControllerForPresentation(using animator: UIViewControllerAnimatedTransitioning) -> UIViewControllerInteractiveTransitioning? func interactionControllerForDismissal(using animator: UIViewControllerAnimatedTransitioning) -> UIViewControllerInteractiveTransitioning?
83. Interactive Animator class PresentInteractor: UIPercentDrivenInteractiveTransition { init(gestureView: UIView) { super.init() } }
84. Interactive Animator class PresentInteractor: UIPercentDrivenInteractiveTransition { init(gestureView: UIView) { super.init() panGestureRecognizer = UIPanGestureRecognizer(target: self, action: #selector(handlePan(_:))) gestureView.addGestureRecognizer(panGestureRecognizer) } @objc private func handlePan(_ panGestureRecognizer: UIPanGestureRecognizer) { switch panGestureRecognizer.state { case .possible: break case .began: presentViewController() case .changed: shouldCompleteTransition = percentage > 0.1 update(percentage) case .ended: if shouldCompleteTransition { finish() } else { cancel() } case .cancelled, .failed: cancel() @unknown default: break } } }
85. › After adopting UIViewControllerTransitioningDelegate Demo
86. Implementation One Last Step Don’t let user notice there’re two view controllers › Sync status when start expanding and end collapsing struct ViewState { let contentOffset: CGPoint } protocol ViewStateSyncDelegate { var initialViewStateForExpandedViewController: ViewState { get } func expandedViewDidFinishCollapsing(_ viewState: ViewState) }
87. Recap › ViewController presenting works on Keyboard › Use transition coordinator to define appear and disappear animation › Use UIViewControllerTransitioningDelegate to create custom interactive transition