Automated layout failure detection for responsive web pages without an explicit oracle

Automated layout failure detection for responsive web pages without an explicit oracle

Interested in learning more about this topic? Visit this web site to read the paper: https://www.gregorykapfhammer.com/research/papers/Walsh2017/

4ae30d49c8cc07e42d5a871efb9bcfba?s=128

Gregory Kapfhammer

July 10, 2017
Tweet

Transcript

  1. AUTOMATED LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN

    EXPLICIT ORACLE THOMAS WALSH1, PHIL MCMINN1, GREGORY KAPFHAMMER2 1UNIVERSITY OF SHEFFIELD, 2ALLEGHENY COLLEGE 1
  2. ISSTA 2017 A LONG TIME AGO… 2 THOMAS WALSH TWALSH1@SHEFFIELD.AC.UK

    AUTOMATED LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE
  3. ISSTA 2017 THE WEB TODAY 3 THOMAS WALSH TWALSH1@SHEFFIELD.AC.UK AUTOMATED

    LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE https://blog.pivotal.io/labs/labs/an-introduction-to-qa-at-xtreme-labs
  4. ISSTA 2017 WHY SUPPORT ALL DEVICES? 4 AUTOMATED LAYOUT FAILURE

    DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE THOMAS WALSH TWALSH1@SHEFFIELD.AC.UK
  5. ISSTA 2017 HOW CAN DEVELOPERS MAKE WEBSITES SUPPORT ALL DEVICES?

    USE RESPONSIVE WEB DESIGN. 5 AUTOMATED LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE THOMAS WALSH TWALSH1@SHEFFIELD.AC.UK
  6. ISSTA 2017 THOMAS WALSH TWALSH1@SHEFFIELD.AC.UK AUTOMATED LAYOUT FAILURE DETECTION FOR

    RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE RESPONSIVE WEB DESIGN 6 http://getbootstrap.com
  7. ISSTA 2017 RESPONSIVE LAYOUT FAILURES 7 THOMAS WALSH TWALSH1@SHEFFIELD.AC.UK AUTOMATED

    LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE
  8. ISSTA 2017 RESPONSIVE LAYOUT FAILURES 8 THOMAS WALSH TWALSH1@SHEFFIELD.AC.UK AUTOMATED

    LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE
  9. ISSTA 2017 RESPONSIVE LAYOUT FAILURES 9 THOMAS WALSH TWALSH1@SHEFFIELD.AC.UK AUTOMATED

    LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE
  10. ISSTA 2017 RESPONSIVE LAYOUT FAILURES 10 THOMAS WALSH TWALSH1@SHEFFIELD.AC.UK AUTOMATED

    LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE
  11. ISSTA 2017 RESPONSIVE LAYOUT FAILURES 11 THOMAS WALSH TWALSH1@SHEFFIELD.AC.UK AUTOMATED

    LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE
  12. ISSTA 2017 RESPONSIVE LAYOUT FAILURES 12 THOMAS WALSH TWALSH1@SHEFFIELD.AC.UK AUTOMATED

    LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE
  13. ISSTA 2017 HOW DO DEVELOPERS TEST RESPONSIVE WEBSITES? THEY “SPOT-CHECK”.

    WHAT IS THE ORACLE FOR RESPONSIVE WEBSITES? THE HUMAN TESTER. 13 AUTOMATED LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE THOMAS WALSH TWALSH1@SHEFFIELD.AC.UK
  14. ISSTA 2017 A SPOTCHECKING EXAMPLE 14 AUTOMATED LAYOUT FAILURE DETECTION

    FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE THOMAS WALSH TWALSH1@SHEFFIELD.AC.UK 320 pixels 768 pixels 1024 pixels 1200 pixels 980 pixels
  15. ISSTA 2017 SUMMARY OF APPROACH 15 AUTOMATED LAYOUT FAILURE DETECTION

    FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE THOMAS WALSH TWALSH1@SHEFFIELD.AC.UK RESPONSIVE LAYOUT GRAPH (RLG) COMMON RESPONSIVE LAYOUT FAILURE DETECTION
  16. ISSTA 2017 THE RESPONSIVE LAYOUT GRAPH 16 THOMAS WALSH TWALSH1@SHEFFIELD.AC.UK

    AUTOMATED LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE DIV[1] DIV[2] DIV[3] DIV[1] DIV[2] DIV[3] IMG 320PX-1400PX BODY 768PX-1400PX IMG 320PX-1400PX DIV[3] 320PX-1400PX DIV[2] 320PX-1400PX DIV[1] 320px-767px, above 768px-1400px, left-of 320px-767px , centre-justified 768px-1400px, left-justified 320 pixels 768 pixels
  17. ISSTA 2017 RLF 1 - ELEMENT COLLISION 17 THOMAS WALSH

    TWALSH1@SHEFFIELD.AC.UK AUTOMATED LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE
  18. 18

  19. 19

  20. ISSTA 2017 RLF 1 - ELEMENT COLLISION 1. Check whether

    any pair of elements are overlapping at any point. 2. If two elements are overlapping, check the layout at the wider viewport width. 3. If elements no longer overlapping, report an element collision failure. 20 THOMAS WALSH TWALSH1@SHEFFIELD.AC.UK AUTOMATED LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE
  21. ISSTA 2017 RLF 1 - ELEMENT COLLISION 21 THOMAS WALSH

    TWALSH1@SHEFFIELD.AC.UK AUTOMATED LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE 320PX-1400PX FORM 320PX-1400PX DIV 320px-767px, above 768px-1000px, overlapping 1001px-1400px, left-of
  22. ISSTA 2017 RLF 1 - ELEMENT COLLISION 1. Check whether

    any pair of elements are overlapping at any point. 2. If two elements are overlapping, check the layout at the wider viewport width. 3. If elements no longer overlapping, report an element collision failure. 22 THOMAS WALSH TWALSH1@SHEFFIELD.AC.UK AUTOMATED LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE
  23. ISSTA 2017 RLF 1 - ELEMENT COLLISION 23 THOMAS WALSH

    TWALSH1@SHEFFIELD.AC.UK AUTOMATED LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE 320PX-1400PX FORM 320PX-1400PX DIV 320px-767px, above 768px-1000px, overlapping 1001px-1400px, left-of
  24. ISSTA 2017 RLF 1 - ELEMENT COLLISION 1. Check whether

    any pair of elements are overlapping at any point. 2. If two elements are overlapping, check the layout at the wider viewport width. 3. If elements no longer overlapping, report an element collision failure. 24 THOMAS WALSH TWALSH1@SHEFFIELD.AC.UK AUTOMATED LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE
  25. ISSTA 2017 RLF 1 - ELEMENT COLLISION 25 THOMAS WALSH

    TWALSH1@SHEFFIELD.AC.UK AUTOMATED LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE Element collision between 768px and 1000px. 320PX-1400PX FORM 320PX-1400PX DIV 320px-767px, above 768px-1000px, overlapping 1001px-1400px, left-of
  26. ISSTA 2017 RLF 2 - ELEMENT PROTRUSION 26 THOMAS WALSH

    TWALSH1@SHEFFIELD.AC.UK AUTOMATED LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE
  27. 27

  28. 28

  29. ISSTA 2017 RLF 2 - ELEMENT PROTRUSION 1. Check whether

    any pair of elements are overlapping at any point. 2. If two elements are overlapping, check the layout at the wider viewport width. 3. If one element now contains the other, report an element protrusion failure. 29 THOMAS WALSH TWALSH1@SHEFFIELD.AC.UK AUTOMATED LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE
  30. ISSTA 2017 RLF 2 - ELEMENT PROTRUSION 30 THOMAS WALSH

    TWALSH1@SHEFFIELD.AC.UK AUTOMATED LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE 320PX-1400PX HEADER 320PX-1400PX FORM 320px-640px, overlapping 641px-1400px, contains
  31. ISSTA 2017 RLF 2 - ELEMENT PROTRUSION 1. Check whether

    any pair of elements are overlapping at any point. 2. If two elements are overlapping, check the layout at the wider viewport width. 3. If one element now contains the other, report an element protrusion failure. 31 THOMAS WALSH TWALSH1@SHEFFIELD.AC.UK AUTOMATED LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE
  32. ISSTA 2017 RLF 2 - ELEMENT PROTRUSION 32 THOMAS WALSH

    TWALSH1@SHEFFIELD.AC.UK AUTOMATED LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE Element protrusion between 320px and 640px. 320px-640px, overlapping 641px-1400px, contains 320PX-1400PX HEADER 320PX-1400PX FORM
  33. ISSTA 2017 RLF 3 - VIEWPORT PROTRUSION 33 THOMAS WALSH

    TWALSH1@SHEFFIELD.AC.UK AUTOMATED LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE
  34. 34

  35. 35

  36. ISSTA 2017 RLF 3 - VIEWPORT PROTRUSION 1. Look through

    each element. 2. Check that for every viewport width the element is visible, it is contained within some other element. 3. If, at any point, the element is not contained at all, report a viewport protrusion failure. 36 THOMAS WALSH TWALSH1@SHEFFIELD.AC.UK AUTOMATED LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE
  37. ISSTA 2017 RLF 3 - VIEWPORT PROTRUSION 37 THOMAS WALSH

    TWALSH1@SHEFFIELD.AC.UK AUTOMATED LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE 981PX-1400PX DIV 1050px-1400px, contains 320PX-1400PX BODY
  38. ISSTA 2017 RLF 3 - VIEWPORT PROTRUSION 1. Look through

    each element. 2. Check that for every viewport width the element is visible, it is contained within some other element. 3. If, at any point, the element is not contained at all, report a viewport protrusion failure. 38 THOMAS WALSH TWALSH1@SHEFFIELD.AC.UK AUTOMATED LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE
  39. ISSTA 2017 RLF 3 - VIEWPORT PROTRUSION 39 THOMAS WALSH

    TWALSH1@SHEFFIELD.AC.UK AUTOMATED LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE 1050px-1400px, contains 981PX-1400PX DIV 320PX-1400PX BODY
  40. ISSTA 2017 RLF 3 - VIEWPORT PROTRUSION 1. Look through

    each element. 2. Check that for every viewport width the element is visible, it is contained within some other element. 3. If, at any point, the element is not contained at all, report a viewport protrusion failure. 40 THOMAS WALSH TWALSH1@SHEFFIELD.AC.UK AUTOMATED LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE
  41. ISSTA 2017 RLF 3 - VIEWPORT PROTRUSION 41 THOMAS WALSH

    TWALSH1@SHEFFIELD.AC.UK AUTOMATED LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE 1050px-1400px, contains Viewport protrusion between 981px and 1049px. 981PX-1400PX DIV 320PX-1400PX BODY
  42. ISSTA 2017 RLF 4 - SMALL-RANGE LAYOUTS 42 THOMAS WALSH

    TWALSH1@SHEFFIELD.AC.UK AUTOMATED LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE
  43. 43

  44. 44

  45. ISSTA 2017 1. Are any layouts only applied for a

    small number of viewport widths (e.g., 5)? 2. If so, check if similar layouts exist both before and after the small-range layout. 3. If so, report the small-range layout failure. 45 THOMAS WALSH TWALSH1@SHEFFIELD.AC.UK AUTOMATED LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE RLF 4 - SMALL-RANGE LAYOUTS
  46. ISSTA 2017 46 THOMAS WALSH TWALSH1@SHEFFIELD.AC.UK AUTOMATED LAYOUT FAILURE DETECTION

    FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE 320PX-1400PX HEADER 320PX-1400PX BANNER 320px-979px, above 980px-980px, overlapping 981px-1400px, above RLF 4 - SMALL-RANGE LAYOUTS
  47. ISSTA 2017 1. Are any layouts only applied for a

    small number of viewport widths (e.g., 5)? 2. If so, check if similar layouts exist both before and after the small-range layout. 3. If so, report the small-range layout failure. 47 THOMAS WALSH TWALSH1@SHEFFIELD.AC.UK AUTOMATED LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE RLF 4 - SMALL-RANGE LAYOUTS
  48. ISSTA 2017 48 THOMAS WALSH TWALSH1@SHEFFIELD.AC.UK AUTOMATED LAYOUT FAILURE DETECTION

    FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE 320PX-1400PX HEADER 320PX-1400PX BANNER 320px-979px, above 980px-980px, overlapping 981px-1400px, above RLF 4 - SMALL-RANGE LAYOUTS
  49. ISSTA 2017 1. Are any layouts only applied for a

    small number of viewport widths (e.g., 5)? 2. If so, check if similar layouts exist both before and after the small-range layout. 3. If so, report the small-range layout failure. 49 THOMAS WALSH TWALSH1@SHEFFIELD.AC.UK AUTOMATED LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE RLF 4 - SMALL-RANGE LAYOUTS
  50. ISSTA 2017 50 THOMAS WALSH TWALSH1@SHEFFIELD.AC.UK AUTOMATED LAYOUT FAILURE DETECTION

    FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE Small-range layout at 980 pixels. 320PX-1400PX HEADER 320PX-1400PX BANNER 320px-979px, above 980px-980px, overlapping 981px-1400px, above RLF 4 - SMALL-RANGE LAYOUTS
  51. ISSTA 2017 RLF 5 - ELEMENT WRAPPING 51 THOMAS WALSH

    TWALSH1@SHEFFIELD.AC.UK AUTOMATED LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE
  52. 52

  53. 53

  54. ISSTA 2017 RLF 5 - ELEMENT WRAPPING 1. Analyse layout

    information to determine which elements are laid out in rows. 2. Are there any rows which one element is not in? 3. If this element is in the row at a wider viewport, report an element wrapping failure. 54 THOMAS WALSH TWALSH1@SHEFFIELD.AC.UK AUTOMATED LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE
  55. ISSTA 2017 RLF 5 - ELEMENT WRAPPING 55 THOMAS WALSH

    TWALSH1@SHEFFIELD.AC.UK AUTOMATED LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE LI[3] LI[1] LI[2] 768px-1400px, left-of 768px-800px, above 801px-1400px, left-of 768px-800px, above 801px-1400px, left-of
  56. ISSTA 2017 RLF 5 - ELEMENT WRAPPING 1. Analyse layout

    information to determine which elements are laid out in rows. 2. Are there any rows which one element is not in? 3. If this element is in the row at a wider viewport, report an element wrapping failure. 56 THOMAS WALSH TWALSH1@SHEFFIELD.AC.UK AUTOMATED LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE
  57. ISSTA 2017 RLF 5 - ELEMENT WRAPPING 57 THOMAS WALSH

    TWALSH1@SHEFFIELD.AC.UK AUTOMATED LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE LI[3] LI[1] LI[2] 768px-1400px, left- 768px-800px, above 801px-1400px, left-of 768px-800px, above 801px-1400px, left-of
  58. ISSTA 2017 RLF 5 - ELEMENT WRAPPING 1. Analyse layout

    information to determine which elements are laid out in rows. 2. Are there any rows which one element is not in? 3. If this element is in the row at a wider viewport, report an element wrapping failure. 58 THOMAS WALSH TWALSH1@SHEFFIELD.AC.UK AUTOMATED LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE
  59. ISSTA 2017 RLF 5 - ELEMENT WRAPPING 59 THOMAS WALSH

    TWALSH1@SHEFFIELD.AC.UK AUTOMATED LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE Element wrapping between 768px and 800px. LI[3] LI[1] LI[2] 768px-1400px, left- 768px-800px, above 801px-1400px, left-of 768px-800px, above 801px-1400px, left-of
  60. ISSTA 2017 EMPIRICAL EVALUATION 1. Can the approach detect responsive

    layout failures in real-world responsive web pages? 2. How effective are manual “spotchecking” approaches in comparison to our approach? 3. How long do our techniques take to run? ReDeCheck - Responsive Design Checker 60 AUTOMATED LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE THOMAS WALSH TWALSH1@SHEFFIELD.AC.UK
  61. ISSTA 2017 61 AUTOMATED LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB

    PAGES WITHOUT AN EXPLICIT ORACLE THOMAS WALSH TWALSH1@SHEFFIELD.AC.UK
  62. ISSTA 2017 RQ1 CAN THE APPROACH DETECT RESPONSIVE LAYOUT FAILURES

    IN REAL-WORLD RESPONSIVE WEB PAGES? 62 AUTOMATED LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE THOMAS WALSH TWALSH1@SHEFFIELD.AC.UK
  63. ISSTA 2017 63 AUTOMATED LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB

    PAGES WITHOUT AN EXPLICIT ORACLE THOMAS WALSH TWALSH1@SHEFFIELD.AC.UK
  64. ISSTA 2017 64 AUTOMATED LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB

    PAGES WITHOUT AN EXPLICIT ORACLE THOMAS WALSH TWALSH1@SHEFFIELD.AC.UK
  65. ISSTA 2017 RESEARCH QUESTION ONE ▸ Responsive layout failures found

    in 16 of the 26 web pages. ▸ 196 true positive results conflated to 33 distinct RLFs. ▸ False positives mainly caused by “coincidental” attributes being assigned to constraints. ▸ Many NOIs showed significant structural issues, which could easily manifest as TPs in the future. ▸ On average, developers need to check 4 distinct viewport ranges to observe each TP failure. 65 Failure Type TP FP NOI RLFs Element Collision 8 0 24 8 Element Protrusion 3 0 36 3 Viewport Protrusion 24 0 23 11 Small-Range 152 43 0 1 Wrapping 10 5 0 10 Total 196 48 83 33 THOMAS WALSH TWALSH1@SHEFFIELD.AC.UK AUTOMATED LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE
  66. ISSTA 2017 RQ2 HOW EFFECTIVE ARE MANUAL “SPOTCHECKING” APPROACHES IN

    COMPARISON TO OUR APPROACH? 66 AUTOMATED LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE THOMAS WALSH TWALSH1@SHEFFIELD.AC.UK
  67. ISSTA 2017 RESEARCH QUESTION TWO 67 THOMAS WALSH TWALSH1@SHEFFIELD.AC.UK AUTOMATED

    LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE 0 6 12 18 24 30 36 KERSLEY RESPONSINATOR RD CHECKER V-RESIZER FIREFOX RWD REDECHECK 33 27 26 23 22 22
  68. ISSTA 2017 RQ3 HOW LONG DO OUR TECHNIQUES TAKE TO

    RUN? 68 AUTOMATED LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE THOMAS WALSH TWALSH1@SHEFFIELD.AC.UK
  69. ISSTA 2017 RESEARCH QUESTION THREE 69 THOMAS WALSH TWALSH1@SHEFFIELD.AC.UK AUTOMATED

    LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE 0 60 120 180 240 300 3-MINUTE-JOURNAL ACCOUNTKILLER AIRBNB BUGMENOT CLOUDCONVERT CONSUMER-REPORTS COVERED-CALENDAR DAYS-OLD DICTATION DUOLINGO HONEY HOTELWIFITEST MAILINATOR MIDWAYMEETUP NINITE PDF-ESCAPE PEPFEED POCKET RAINYMOOD RUNPEE STUMBLEUPON TOPDOCUMENTARY USERSEARCH WSIRN WILLMYPHONEWORK ZERODOLLARMOVIES
  70. ISSTA 2017 THE REDECHECK TOOL 70 THOMAS WALSH TWALSH1@SHEFFIELD.AC.UK AUTOMATED

    LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE https://github.com/redecheck/redecheck http://redecheck.org https://github.com/redecheck/example-webpages
  71. ISSTA 2017 71 THOMAS WALSH TWALSH1@SHEFFIELD.AC.UK AUTOMATED LAYOUT FAILURE DETECTION

    FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE
  72. THANK YOU TWALSH1@SHEFFIELD.AC.UK THOMASWALSH.CO.UK REDECHECK.ORG 72