Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The Challenges Behind a Better Sticker Input Experience

The Challenges Behind a Better Sticker Input Experience

Jiaheng Huang
LINE Fukuoka Development H Team Software Engineer
Yu Yang
LINE Fukuoka Development H Team Android Engineer
https://linedevday.linecorp.com/2020/ja/sessions/7642
https://linedevday.linecorp.com/2020/en/sessions/7642

LINE DevDay 2020

November 26, 2020
Tweet

More Decks by LINE DevDay 2020

Other Decks in Technology

Transcript

  1. Agenda › An introduction of the new LINE sticker keyboard

    › A deep dive on the rendering performance issue on Android › Create smooth expand & collapse animations on iOS
  2. Our initial solution… <androidx.recyclerview.widget.RecyclerView 
 android:id="@+id/sticker_grid_view" 
 android:layout_width="match_parent" 
 android:layout_height="wrap_content"/>

    
 
 <androidx.recyclerview.widget.RecyclerView 
 android:id=“@+id/author_latest_stickers_view" 
 android:layout_width="match_parent" 
 android:layout_height="wrap_content" />
  3. Our initial solution… <androidx.core.widget.NestedScrollView 
 android:id="@+id/nested_scroll_view" 
 android:layout_width="match_parent" 
 android:layout_height="match_parent">

    
 
 <LinearLayout 
 android:layout_width="match_parent" 
 android:layout_height="wrap_content" 
 android:orientation="vertical"> 
 
 <androidx.recyclerview.widget.RecyclerView 
 android:id="@+id/sticker_grid_view" 
 android:layout_width="match_parent" 
 android:layout_height="wrap_content"/> 
 
 <androidx.recyclerview.widget.RecyclerView 
 android:id="@+id/author_latest_stickers_view" 
 android:layout_width="match_parent" 
 android:layout_height="wrap_content" /> 
 
 </LinearLayout> 
 
 </androidx.core.widget.NestedScrollView>
  4. How Android draws views? ViewRootImpl.java 
 private void performTraversals() {


    ... 
 int childWidthMeasureSpec = getRootMeasureSpec(mWidth, lp.width); 
 int childHeightMeasureSpec = getRootMeasureSpec(mHeight, lp.height); 
 ... 
 performMeasure(childWidthMeasureSpec, childHeightMeasureSpec); 
 ... 
 performLayout(lp, mWidth, mHeight); 
 ... 
 performDraw(); 
 }
  5. How Android draws views? ViewRootImpl.java 
 private void performTraversals() {


    ... 
 int childWidthMeasureSpec = getRootMeasureSpec(mWidth, lp.width); 
 int childHeightMeasureSpec = getRootMeasureSpec(mHeight, lp.height); 
 ... 
 performMeasure(childWidthMeasureSpec, childHeightMeasureSpec); 
 ... 
 performLayout(lp, mWidth, mHeight); 
 ... 
 performDraw(); 
 }
  6. How Android draws views? ViewRootImpl.java 
 private void performTraversals() {


    ... 
 int childWidthMeasureSpec = getRootMeasureSpec(mWidth, lp.width); 
 int childHeightMeasureSpec = getRootMeasureSpec(mHeight, lp.height); 
 ... 
 performMeasure(childWidthMeasureSpec, childHeightMeasureSpec); 
 ... 
 performLayout(lp, mWidth, mHeight); 
 ... 
 performDraw(); 
 }
  7. How Android draws views? ViewRootImpl.java 
 private void performTraversals() {


    ... 
 int childWidthMeasureSpec = getRootMeasureSpec(mWidth, lp.width); 
 int childHeightMeasureSpec = getRootMeasureSpec(mHeight, lp.height); 
 ... 
 performMeasure(childWidthMeasureSpec, childHeightMeasureSpec); 
 ... 
 performLayout(lp, mWidth, mHeight); 
 ... 
 performDraw(); 
 }
  8. How Android draws views? Measure Layout Draw performTraversals For a

    child view to be measured, two specifications are required:
  9. How Android draws views? Measure Layout Draw performTraversals › MATCH_PARENT

    › WRAP_CONTENT › an exact number (e.g.,100dp) ViewGroup.LayoutParams For a child view to be measured, two specifications are required:
  10. How Android draws views? Measure Layout Draw performTraversals › MATCH_PARENT

    › WRAP_CONTENT › an exact number (e.g.,100dp) ViewGroup.LayoutParams MeasureSpec › UNSPECIFIED › EXACTLY › AT MOST For a child view to be measured, two specifications are required:
  11. How a RecycleView inside a NestedScrollView is measured? NestedScrollView.java
 protected

    void measureChild(..., int parentWidthMeasureSpec, int parentHeightMeasureSpec) {
 }
  12. How a RecycleView inside a NestedScrollView is measured? NestedScrollView.java
 protected

    void measureChild(..., int parentWidthMeasureSpec, int parentHeightMeasureSpec) {
 ...
 childWidthMeasureSpec = getChildMeasureSpec(...); 
 childHeightMeasureSpec = MeasureSpec.makeMeasureSpec(0, MeasureSpec.UNSPECIFIED);
 child.measure(childWidthMeasureSpec, childHeightMeasureSpec); 
 }
  13. How a RecycleView inside a NestedScrollView is measured? NestedScrollView.java
 protected

    void measureChild(..., int parentWidthMeasureSpec, int parentHeightMeasureSpec) {
 ...
 childWidthMeasureSpec = getChildMeasureSpec(...); 
 childHeightMeasureSpec = MeasureSpec.makeMeasureSpec(0, MeasureSpec.UNSPECIFIED);
 child.measure(childWidthMeasureSpec, childHeightMeasureSpec); 
 }
  14. How a RecycleView inside a NestedScrollView is measured? NestedScrollView.java
 protected

    void measureChild(..., int parentWidthMeasureSpec, int parentHeightMeasureSpec) {
 ...
 childWidthMeasureSpec = getChildMeasureSpec(...); 
 childHeightMeasureSpec = MeasureSpec.makeMeasureSpec(0, MeasureSpec.UNSPECIFIED);
 child.measure(childWidthMeasureSpec, childHeightMeasureSpec); 
 }
  15. How a RecycleView inside a NestedScrollView is measured? RecyclerView.java 


    protected void onMeasure(int widthSpec, int heightSpec) { 
 ... 
 final int widthMode = MeasureSpec.getMode(widthSpec); 
 final int heightMode = MeasureSpec.getMode(heightSpec); 
 mLayout.onMeasure(mRecycler, mState, widthSpec, heightSpec); 
 final boolean measureSpecModeIsExactly = widthMode == MeasureSpec.EXACTLY && heightMode == MeasureSpec.EXACTLY; 
 if (measureSpecModeIsExactly || mAdapter == null) return; 
 ... 
 mLayout.setMeasuredDimensionFromChildren(widthSpec, heightSpec); 
 ... 
 }
  16. How a RecycleView inside a NestedScrollView is measured? RecyclerView.java 


    protected void onMeasure(int widthSpec, int heightSpec) { 
 ... 
 final int widthMode = MeasureSpec.getMode(widthSpec); 
 final int heightMode = MeasureSpec.getMode(heightSpec); 
 mLayout.onMeasure(mRecycler, mState, widthSpec, heightSpec); 
 final boolean measureSpecModeIsExactly = widthMode == MeasureSpec.EXACTLY && heightMode == MeasureSpec.EXACTLY; 
 if (measureSpecModeIsExactly || mAdapter == null) return; 
 ... 
 mLayout.setMeasuredDimensionFromChildren(widthSpec, heightSpec); 
 ... 
 }
  17. How a RecycleView inside a NestedScrollView is measured? RecyclerView.java 


    void setMeasuredDimensionFromChildren(int widthSpec, int heightSpec) { 
 ... 
 for (int i = 0; i < count; i++) { 
 View child = getChildAt(i); 
 final Rect bounds = mRecyclerView.mTempRect; 
 getDecoratedBoundsWithMargins(child, bounds); 
 if (bounds.left < minX) minX = bounds.left; 
 if (bounds.right > maxX) maxX = bounds.right; 
 if (bounds.top < minY) minY = bounds.top; 
 if (bounds.bottom > maxY) maxY = bounds.bottom; 
 } 
 mRecyclerView.mTempRect.set(minX, minY, maxX, maxY); 
 setMeasuredDimension(mRecyclerView.mTempRect, widthSpec, heightSpec); 
 }
  18. How a RecycleView inside a NestedScrollView is measured? RecyclerView.java 


    void setMeasuredDimensionFromChildren(int widthSpec, int heightSpec) { 
 ... 
 for (int i = 0; i < count; i++) { 
 View child = getChildAt(i); 
 final Rect bounds = mRecyclerView.mTempRect; 
 getDecoratedBoundsWithMargins(child, bounds); 
 if (bounds.left < minX) minX = bounds.left; 
 if (bounds.right > maxX) maxX = bounds.right; 
 if (bounds.top < minY) minY = bounds.top; 
 if (bounds.bottom > maxY) maxY = bounds.bottom; 
 } 
 mRecyclerView.mTempRect.set(minX, minY, maxX, maxY); 
 setMeasuredDimension(mRecyclerView.mTempRect, widthSpec, heightSpec); 
 }
  19. How a RecycleView inside a NestedScrollView is measured? RecyclerView.java 


    public void setMeasuredDimension(Rect childrenBounds, int wSpec, int hSpec) { 
 int usedWidth = childrenBounds.width() + getPaddingLeft() + getPaddingRight(); 
 int usedHeight = childrenBounds.height() + getPaddingTop() + getPaddingBottom(); 
 int width = chooseSize(wSpec, usedWidth, getMinimumWidth()); 
 int height = chooseSize(hSpec, usedHeight, getMinimumHeight()); 
 setMeasuredDimension(width, height); 
 }
  20. How a RecycleView inside a NestedScrollView is measured? RecyclerView.java 


    public void setMeasuredDimension(Rect childrenBounds, int wSpec, int hSpec) { 
 int usedWidth = childrenBounds.width() + getPaddingLeft() + getPaddingRight(); 
 int usedHeight = childrenBounds.height() + getPaddingTop() + getPaddingBottom(); 
 int width = chooseSize(wSpec, usedWidth, getMinimumWidth()); 
 int height = chooseSize(hSpec, usedHeight, getMinimumHeight()); 
 setMeasuredDimension(width, height); 
 }
  21. How a RecycleView inside a NestedScrollView is measured? RecyclerView.java 


    public void setMeasuredDimension(Rect childrenBounds, int wSpec, int hSpec) { 
 int usedWidth = childrenBounds.width() + getPaddingLeft() + getPaddingRight(); 
 int usedHeight = childrenBounds.height() + getPaddingTop() + getPaddingBottom(); 
 int width = chooseSize(wSpec, usedWidth, getMinimumWidth()); 
 int height = chooseSize(hSpec, usedHeight, getMinimumHeight()); 
 setMeasuredDimension(width, height); 
 }
  22. How a RecycleView inside a NestedScrollView is measured? RecyclerView.java 


    public static int chooseSize(int spec, int desired, int min) { 
 final int mode = View.MeasureSpec.getMode(spec); 
 final int size = View.MeasureSpec.getSize(spec); 
 switch (mode) { 
 case View.MeasureSpec.EXACTLY: 
 return size; 
 case View.MeasureSpec.AT_MOST: 
 return Math.min(size, Math.max(desired, min)); 
 case View.MeasureSpec.UNSPECIFIED: 
 default: 
 return Math.max(desired, min); 
 } 
 }
  23. How a RecycleView inside a NestedScrollView is measured? RecyclerView.java 


    public static int chooseSize(int spec, int desired, int min) { 
 final int mode = View.MeasureSpec.getMode(spec); 
 final int size = View.MeasureSpec.getSize(spec); 
 switch (mode) { 
 case View.MeasureSpec.EXACTLY: 
 return size; 
 case View.MeasureSpec.AT_MOST: 
 return Math.min(size, Math.max(desired, min)); 
 case View.MeasureSpec.UNSPECIFIED: 
 default: 
 return Math.max(desired, min); 
 } 
 }
  24. How a RecycleView inside a NestedScrollView is measured? RecyclerView.java 


    public static int chooseSize(int spec, int desired, int min) { 
 final int mode = View.MeasureSpec.getMode(spec); 
 final int size = View.MeasureSpec.getSize(spec); 
 switch (mode) { 
 case View.MeasureSpec.EXACTLY: 
 return size; 
 case View.MeasureSpec.AT_MOST: 
 return Math.min(size, Math.max(desired, min)); 
 case View.MeasureSpec.UNSPECIFIED: 
 default: 
 return Math.max(desired, min); 
 } 
 }
  25. Our initial solution… So, the height of the RecyclerView would

    not be restricted, and its height would be the total height of all child view items and the paddings (if any). In other words, all child view items will be rendered at once.
  26. The combination of LayoutParams and MeasureSpec EXACTLY AT_MOST UNSPECIFIED DP

    EXACTLY childSize EXACTLY childSize EXACTLY childSize MATCH_PARENT EXACTLY parentSize AT_MOST parentSize UNSPECIFIED 0 WRAP_CONTENT AT_MOST parentSize AT_MOST parentSize UNSPECIFIED 0
  27. The combination of LayoutParams and MeasureSpec EXACTLY AT_MOST UNSPECIFIED DP

    EXACTLY childSize EXACTLY childSize EXACTLY childSize MATCH_PARENT EXACTLY parentSize AT_MOST parentSize UNSPECIFIED 0 WRAP_CONTENT AT_MOST parentSize AT_MOST parentSize UNSPECIFIED 0
  28. The combination of LayoutParams and MeasureSpec EXACTLY AT_MOST UNSPECIFIED DP

    EXACTLY childSize EXACTLY childSize EXACTLY childSize MATCH_PARENT EXACTLY parentSize AT_MOST parentSize UNSPECIFIED 0 WRAP_CONTENT AT_MOST parentSize AT_MOST parentSize UNSPECIFIED 0
  29. How this problem could be solved? › Use an exact

    height value instead of WRAP_CONTENT for the RecyclerView.
  30. How this problem could be solved? › DO NOT wrap

    a RecyclerView inside a NestedScrollView. Instead, use a single RecyclerView with multiple view types to achieve complex UI. › Use an exact height value instead of WRAP_CONTENT for the RecyclerView.
  31. How this problem could be solved? › DO NOT wrap

    a RecyclerView inside a NestedScrollView. Instead, use a single RecyclerView with multiple view types to achieve complex UI. › Luckily LINE client has its own customized adapter called RecyclerViewModelAdapter to easily support multiple view types in a RecyclerView. › Use an exact height value instead of WRAP_CONTENT for the RecyclerView.
  32. Recap › When MATCH_PARENT or WRAP_CONTENT is specified for a

    RecyclerView’s height inside a NestedScrollView, all list view items would rendered at once. › Discussed how a RecyclerView inside a NestedScrollView is measured.
  33. Recap › When MATCH_PARENT or WRAP_CONTENT is specified for a

    RecyclerView’s height inside a NestedScrollView, all list view items would rendered at once. › Discussed how a RecyclerView inside a NestedScrollView is measured. › DO NOT put a RecyclerView inside a NestedScrollView unless the height is known in advance.
  34. Challenge We don’t want to change keyboard height during expanding

    animation, otherwise the input bar will go up Keyboard is not full screen height, the expand animation will go beyond keyboard area
  35. UIPropertyAnimator › Introduced in iOS 10 › Can modify animation

    dynamically › eg. Scrub through a animation by modifying the fractionComplete property
  36. How to implement this? › Need to handle hitTest(_:with:) through

    out the view hierarchy UIPropertyAnimator Calculate layout frame by frame
  37. How to implement this? › Need to handle hitTest(_:with:) through

    out the view hierarchy › Not a good place to handle background dimming logic UIPropertyAnimator Calculate layout frame by frame
  38. How to implement this? How about presenting a view controller

    here? › Present view controller works on keyboard
  39. How to implement this? How about presenting a view controller

    here? › Present view controller works on keyboard › Still can take advantage of the powerful UIPropertyAnimator inside UIViewControllerTransitioningDelegate
  40. How to implement this? How about presenting a view controller

    here? › Present view controller works on keyboard › Still can take advantage of the powerful UIPropertyAnimator inside UIViewControllerTransitioningDelegate › No need to handle hitTest(_:with:)
  41. How to implement this? How about presenting a view controller

    here? › Present view controller works on keyboard › Still can take advantage of the powerful UIPropertyAnimator inside UIViewControllerTransitioningDelegate › No need to handle hitTest(_:with:) › Can handle background dimming logic and device rotation easily in UIPresentationController
  42. Implementation In View Controller Layout UI during view controller appear

    and disappear using transition coordinator Define two states enum State { case expanded, collapsed } transitionCoordinator?.animate(alongsideTransition: { [weak self] (_) in self?.configUI(.expanded) }, completion: nil)
  43. › Applied iOS 13 default presenting animation › Has interactive

    dismiss but no interactive present › Incorrect initial frame of presentation Demo
  44. Implementation In UIViewControllerTransitioningDelegate Create a custom Presentation Controller func presentationController(forPresented

    presented: UIViewController, presenting: UIViewController?, source: UIViewController) -> UIPresentationController? › Define final layout of presented view controller › Add an extra view to do dimming background
  45. Presentation Controller Define final layout class PresentationController: UIPresentationController { private

    var topSpacing: CGFloat override var frameOfPresentedViewInContainerView: CGRect { let containerSize = containerView!.bounds.size var frame: CGRect = .zero frame.size = size( forChildContentContainer: presentedViewController, withParentContainerSize: containerSize ) return frame } override func size( forChildContentContainer container: UIContentContainer, withParentContainerSize parentSize: CGSize ) -> CGSize { let size = CGSize(width: parentSize.width, height: parentSize.height - topSpacing) return size } }
  46. Presentation Controller Add dimming view class PresentationController: UIPresentationController { private

    let dimmingView: UIView override func presentationTransitionWillBegin() { containerView?.insertSubview(dimmingView, at: 0) ... presentedViewController.transitionCoordinator?.animate(alongsideTransition: { _ in self.dimmingView.alpha = 0.7 }) } override func presentationTransitionDidEnd(_ completed: Bool) { if !completed { dimmingView.removeFromSuperview() } } override func dismissalTransitionWillBegin() { presentedViewController.transitionCoordinator?.animate(alongsideTransition: { _ in self.dimmingView.alpha = 0.0 }) } override func dismissalTransitionDidEnd(_ completed: Bool) { if completed { dimmingView.removeFromSuperview() } } }
  47. Implementation In UIViewControllerTransitioningDelegate Create an Animator Object Create a custom

    Presentation Controller func animationController(forDismissed dismissed: UIViewController) -> UIViewControllerAnimatedTransitioning? func animationController(forPresented presented: UIViewController, presenting: UIViewController, source: UIViewController) -> UIViewControllerAnimatedTransitioning? { › Set correct initial frame for presented view controller in Animator › Use UIPropertyAnimator to define a interruptible animation
  48. Animator class TagSearchResultTransitionAnimator: NSObject, UIViewControllerAnimatedTransitioning { let isPresenting: Bool private

    func transitionAnimator(using transitionContext: UIViewControllerContextTransitioning) -> UIViewImplicitlyAnimating { // Set Start position if isPresenting { container.addSubview(presentedView) presentedView.frame = startFrame } let springTiming = UISpringTimingParameters(…) let animator = UIViewPropertyAnimator(duration: duration, timingParameters: springTiming) animator.addAnimations { presentedView.frame = self.isPresenting ? finalFrame : startFrame } animator.addCompletion { position in let didComplete = !transitionContext.transitionWasCancelled // After a failed presentation or successful dismissal, remove the view. if self.isPresenting && !didComplete || !self.isPresenting && didComplete { presentedView.removeFromSuperview() } transitionContext.completeTransition(didComplete) } return animator } }
  49. Implementation In UIViewControllerTransitioningDelegate Create an Animator Object Create a custom

    Presentation Controller Create an Interactive Animator › The easiest way to create an interactive animator is to subclass UIPercentDrivenInteractiveTransition class and add gesture recognizer event-handling code to your subclass func interactionControllerForPresentation(using animator: UIViewControllerAnimatedTransitioning) -> UIViewControllerInteractiveTransitioning? func interactionControllerForDismissal(using animator: UIViewControllerAnimatedTransitioning) -> UIViewControllerInteractiveTransitioning?
  50. Interactive Animator class PresentInteractor: UIPercentDrivenInteractiveTransition { init(gestureView: UIView) { super.init()

    panGestureRecognizer = UIPanGestureRecognizer(target: self, action: #selector(handlePan(_:))) gestureView.addGestureRecognizer(panGestureRecognizer) } @objc private func handlePan(_ panGestureRecognizer: UIPanGestureRecognizer) { switch panGestureRecognizer.state { case .possible: break case .began: presentViewController() case .changed: shouldCompleteTransition = percentage > 0.1 update(percentage) case .ended: if shouldCompleteTransition { finish() } else { cancel() } case .cancelled, .failed: cancel() @unknown default: break } } }
  51. Implementation One Last Step Don’t let user notice there’re two

    view controllers › Sync status when start expanding and end collapsing struct ViewState { let contentOffset: CGPoint } protocol ViewStateSyncDelegate { var initialViewStateForExpandedViewController: ViewState { get } func expandedViewDidFinishCollapsing(_ viewState: ViewState) }
  52. Recap › ViewController presenting works on Keyboard › Use transition

    coordinator to define appear and disappear animation › Use UIViewControllerTransitioningDelegate to create custom interactive transition