Prediction is very difficult, especially about the future

Presented at 2013 IRE CAR conference in Louisville, Kentucky.

Transcript

1. Prediction is very difficult,
   especially about the future
   Andy Cox
   The Weather Company
   February 28, 2013
   @InfoVizard
   

   View Slide

2. Who am I?
   

   View Slide

3. View Slide

4. View Slide

5. View Slide

6. View Slide

7. View Slide

8. View Slide

9. View Slide

10. View Slide

11. View Slide

12. ?
    

    View Slide

13. View Slide

14. Prediction and forecasting
    Probability, uncertainty and confidence
    Predictive modeling
    Communicating uncertainty
    Weather forecasting models
    Goals
    

    View Slide

15. Goals
    Not going to teach you how to do linear
    regression or interpret a p-value
    

    View Slide

16. View Slide

17. What is predictive modeling?
    Working definition:
    Using data to build an approximation of some
    aspect of the real world that we can use to
    better understand it or predict how it will
    behave to previously unseen input.
    

    View Slide

18. So what?
    Why should I care?
    

    View Slide

19. Probability
    Don't be scared!
    

    View Slide

20. Basic probability
    

    View Slide

21. View Slide

22. View Slide

23. View Slide

24. View Slide

25. View Slide

26. View Slide

27. (Sample) size matters
    

    View Slide

28. View Slide

29. 10 percent ≠ impossible
    90 percent ≠ certain
    

    View Slide

30. Uncertainty is inherent
    Ignore at your own peril
    

    View Slide

31. Essentially, all models are
    wrong, but some are useful.
    George E.P. Box
    

    View Slide

32. View Slide

33. Uncertainty is inherent...
    But people often want forecaster to take a stand
    

    View Slide

34. View Slide

35. Uncertainty is inherent...
    What should be communicated in a forecast?
    

    View Slide

36. Source: Coursera
    Linear Regression
    

    View Slide

37. Predictive modeling dangers
    Overfitting and overconfidence
    

    View Slide

38. View Slide

39. Occam's Razor
    

    View Slide

40. Predictive modeling dangers
    Overcoming with human + model approach
    

    View Slide

41. DICast Digit Topps
    Integrated
    first guess
    Human-in-
    the-loop
    Public-oriented
    aggregation
    TWC Forecast System
    

    View Slide

42. Predictive modeling dangers
    Data dredging
    

    View Slide

43. Meaningful? Or data dredging?
    

    View Slide

44. Correlation vs. causation
    

    View Slide

45. View Slide

46. View Slide

47. View Slide

48. Source: kenpom.com
    

    View Slide

49. Forecast vs. prediction
    Let's roll the dice
    

    View Slide

50. Forecast vs. prediction
    Result of a single event vs. probability
    distribution over the long term
    

    View Slide

51. View Slide

52. Communicating
    predictions
    

    View Slide

53. Communicating
    predictions
    National Hurricane Center
    

    View Slide

54. View Slide

55. View Slide

56. View Slide

57. Communicating
    predictions
    National Weather Service
    Storm Prediction Center
    

    View Slide

58. View Slide

59. View Slide

60. View Slide

61. View Slide

62. Source: NWS Storm Prediction Center
    

    View Slide

63. View Slide

64. REMEMBER...A TORNADO WATCH MEANS CONDITIONS ARE FAVORABLE FOR
    TORNADOES AND SEVERE THUNDERSTORMS IN AND CLOSE TO THE WATCH
    AREA. PERSONS IN THESE AREAS SHOULD BE ON THE LOOKOUT FOR
    THREATENING WEATHER CONDITIONS AND LISTEN FOR LATER STATEMENTS
    AND POSSIBLE WARNINGS.
    Tornado watch
    

    View Slide

65. View Slide

66. PRECAUTIONARY/PREPAREDNESS ACTIONS...
    THE SAFEST PLACE TO BE DURING A TORNADO IS UNDER A WORKBENCH OR
    OTHER PIECE OF STURDY FURNITURE. SEEK SHELTER ON THE LOWEST
    FLOOR OF THE BUILDING IN AN INTERIOR HALLWAY OR ROOM SUCH AS A
    CLOSET. USE BLANKETS OR PILLOWS TO COVER YOUR BODY AND ALWAYS
    STAY AWAY FROM WINDOWS.
    IF IN MOBILE HOMES OR VEHICLES...EVACUATE THEM AND GET INSIDE A
    SUBSTANTIAL SHELTER. IF NO SHELTER IS AVAILABLE...LIE FLAT IN
    THE NEAREST DITCH OR OTHER LOW SPOT AND COVER YOUR HEAD WITH
    YOUR HANDS.
    Tornado warning
    

    View Slide

67. AT 6:57 PM CDT...A LARGE TORNADO WAS MOVING ALONG INTERSTATE 44
    WEST OF NEWCASTLE. ON ITS PRESENT PATH...THIS LARGE DAMAGING
    TORNADO WILL ENTER SOUTHWEST SECTIONS OF THE OKLAHOMA CITY
    METRO AREA BETWEEN 7:15 AND 7:30 PM. PERSONS IN MOORE AND
    SOUTH OKLAHOMA CITY SHOULD TAKE IMMEDIATE TORNADO PRECAUTIONS!
    THIS IS AN EXTREMELY DANGEROUS AND LIFE THREATENING SITUATION.
    IF YOU ARE IN THE PATH OF THIS LARGE AND DESTRUCTIVE TORNADO...
    TAKE COVER IMMEDIATELY.
    Tornado warning
    

    View Slide

68. Numerical Weather
    Prediction
    

    View Slide

69. Source: National Oceanic and Atmospheric Administration
    

    View Slide

70. Source: NWS Aviation Weather Center
    

    View Slide

71. Source: National Center for Atmospheric Research
    

    View Slide

72. Source: National Center for Atmospheric Research
    

    View Slide

73. View Slide

74. View Slide

75. Ensemble forecast plume
    Source: http://eyewall.met.psu.edu/rich/gefs/Plumes.html
    

    View Slide

76. View Slide

77. http://imgs.xkcd.com/comics/extrapolating.png
    

    View Slide

78. Thank you
    @InfoVizard
    

    View Slide