Testing Apps at Facebook

Testing Apps at Facebook

How do Facebook engineers ship features to a billion people every two weeks? Learn how we continuously test mobile applications. We will also talk about stress testing and how to measure the impact of code changes on performance.

C0e29150e7b848e783d3195b6bdced49?s=128

Martin Konicek

October 31, 2014
Tweet

Transcript

  1. None
  2. Martin Konicek Testing Android Applications

  3. 100s engineers 100Ms people × 1,000s changes / week ×

    =
  4. Release every 2 weeks

  5. Automated cut every 2 weeks 2 weeks of development master

    release branch 2 weeks of stability 2 weeks of development Release
  6. Ship to beta every night 2 weeks of development Ship

    to alpha every night
  7. None
  8. None
  9. Android has slow rollouts

  10. In practice

  11. None
  12. Unit tests As a first line of defense ! !

    ! ! ! $ buck test javatests/com/facebook/timeline/ PASS 10 Passed 0 Failed TimelineDbCacheTest PASS 16 Passed 0 Failed TimelineRamCacheTest
 PASS 4 Passed 0 Failed FetchTimelineHeaderMethodTest
 ...
 TESTS PASSED

  13. Unit tests As a first line of defense ! !

    ! Mocked: ▪ Backend ▪ The OS ! $ buck test javatests/com/facebook/timeline/ PASS 10 Passed 0 Failed TimelineDbCacheTest PASS 16 Passed 0 Failed TimelineRamCacheTest
 PASS 4 Passed 0 Failed FetchTimelineHeaderMethodTest
 ...
 TESTS PASSED

  14. End-to-end testing

  15. None
  16. Instrumentation ! ▪ Toolkits: ▪ Robotium ▪ Espresso ▪ Test

    code runs on the device

  17. Selendroid

  18. ! ! 
 public class AudiencePickerTest { ! private DroidDriver

    driver; private Wait<DroidDriver> wait; private NewsFeed newsFeed; ! @Before public void login() { driver = DroidDriver.startApp(Apps.FB4A); wait = new DroidWait(mDriver, Duration.seconds(20)); ! TestUser user = TestUser.create(); newsFeed = new Login(driver).as(user); }
 ! ! !
  19. ! @Test public void changeAudience() { AudiencePicker audiencePicker = newsFeed

    .goToStatusComposer() .openAudiencePicker() .selectAudience(Audience.FRIENDS); .type("Droidcon London is the best!”) .post(); ! // Check the story in the backend FeedStory story = wait.until(GraphApi.visibilityOfStory(user)); assertEquals( "Story audience should respect selection", Audience.FRIENDS, story.getAudience()); }

  20. None
  21. Fast feedback Before pushing to master

  22. Test Console Quick overview of test status

  23. Performance testing

  24. None
  25. None
  26. Discovering hard-to-find regressions

  27. What we measure ▪ Time to perform an action ▪

    Allocations ▪ FPS ▪ Memory, network, CPU usage ▪ Anything (extensible framework)
  28. Measuring time

  29. None
  30. Measuring time Challenges
 
 

 The app provides timing information


    
 $ adb logcat | grep CommentsLoad ! D/fb4a:PerformanceLogger: Name: CommentsLoad; Timestamp: 9318686 D/fb4a:PerformanceLogger: Name: CommentsLoad; Timestamp: 9318722; Elapsed: 36
 ...
  31. Measuring time Challenges $ adb logcat | grep CommentsLoad |

    grep Elapsed | awk -F';' '{print $3}'
 ! Elapsed: 46 Elapsed: 66 Elapsed: 76 Elapsed: 36 Elapsed: 40 Elapsed: 83 Elapsed: 34 Elapsed: 36 Elapsed: 33
  32. Measuring time Challenges ▪ Garbage collection ▪ Thread scheduling
 


    ! ! ▪ Network ▪ Overhead of measuring code
 

  33. Measuring time Challenges ▪ Garbage collection ▪ Thread scheduling
 


    ! ! ▪ Network ▪ Overhead of measuring code
 

  34. None
  35. @RunWith(PerformanceTestRunner.class) @TestRepetitions(50) public class CommentsLoadTest { ! ... ! @BeforeRepetition

    public void loginUser() { ...
 driver.disableNewsFeedAutoRefresh(); newsFeed = new Login(driver).as(user); } ! @Test public void openComments() { Button b = newsFeed.getFirstStory().getFeedbackButton(); ! driver.gc(); ! b.click(); waitForMarkers("CommentsLoad", 5, TimeUnit.SECONDS); } ! @After public void goBackToNewsFeed() { driver.sendKeys(SelendroidKeys.BACK); } }
  36. What we measure ▪ Time to perform an action ▪

    Allocations

  37. Garbage Collection in Dalvik

  38. Garbage Collection in Dalvik ▪ Non-compacting


  39. Garbage Collection in Dalvik $ adb logcat
 
 ...
 dalvikvm(6316):

    GC_CONCURRENT freed 431K, 7% free 12898K/13831K, paused 1ms+2ms dalvikvm(6316): GC_CONCURRENT freed 381K, 7% free 12928K/13831K, paused 1ms+2ms dalvikvm(6316): GC_CONCURRENT freed 433K, 7% free 12936K/13831K, paused 1ms+3ms dalvikvm(6316): GC_CONCURRENT freed 121K, 5% free 13211K/13831K, paused 1ms+2ms ▪ Non-compacting

  40. Garbage Collection in Dalvik $ adb logcat
 
 ...
 dalvikvm(6316):

    GC_CONCURRENT freed 431K, 7% free 12898K/13831K, paused 1ms+2ms dalvikvm(6316): GC_CONCURRENT freed 381K, 7% free 12928K/13831K, paused 1ms+2ms dalvikvm(6316): GC_CONCURRENT freed 433K, 7% free 12936K/13831K, paused 1ms+3ms dalvikvm(6316): GC_CONCURRENT freed 121K, 5% free 13211K/13831K, paused 1ms+2ms ▪ Non-compacting ▪ Non-generational

  41. Garbage Collection in Dalvik $ adb logcat
 
 ...
 dalvikvm(6316):

    GC_CONCURRENT freed 431K, 7% free 12898K/13831K, paused 1ms+2ms dalvikvm(6316): GC_CONCURRENT freed 381K, 7% free 12928K/13831K, paused 1ms+2ms dalvikvm(6316): GC_CONCURRENT freed 433K, 7% free 12936K/13831K, paused 1ms+3ms dalvikvm(6316): GC_CONCURRENT freed 121K, 5% free 13211K/13831K, paused 1ms+2ms
 
 ...
 dalvikvm(6316): GC_FOR_ALLOC freed 762K, 13% free 4116K/4688K, 
 paused 181ms, total 182ms ▪ Non-compacting ▪ Non-generational

  42. Debugging Allocations Android Device Monitor


  43. Debugging OOM Errors Eclipse Memory Analyzer


  44. Experiments

  45. Experiments

  46. How we built this

  47. None
  48. None
  49. None
  50. None
  51. Questions We’re hiring!