Upgrade to Pro — share decks privately, control downloads, hide ads and more …

HunchWorks: Combining Human Expertise and Big Data

Dane Petersen
September 23, 2011

HunchWorks: Combining Human Expertise and Big Data

O'Reilly Strata Conference
New York City, NY
September 23, 2011

Dane Petersen

September 23, 2011
Tweet

More Decks by Dane Petersen

Other Decks in Design

Transcript

  1. HunchWorks:
    Combining human expertise and big data
    September 23rd, 2011
    www.unglobalpulse.org
    @unglobalpulse
    #hunchworks

    View Slide

  2. Chris van der Walt,
    Project Lead, HunchWorks, Global Pulse
    @cvanderwalt
    Dane Petersen,
    Experience Designer, Adaptive Path
    @thegreatsunra
    Sara Farmer,
    Chief Platform Architect, Global Pulse
    @bodaceacat

    View Slide

  3. Harnessing innovation
    to protect the vulnerable

    View Slide

  4. Vulnerability:
    The risk of households falling in or remaining in
    poverty due to idiosyncratic or aggregate hazards.
    Source: Professor Wim Naudé, United Nations University

    View Slide

  5. 1.373
    billion people
    living on less than $1.25 per day
    and billions more at risk
    of falling into extreme poverty
    Source: The World Bank

    View Slide

  6. Information Gap
    HUNGER
    AND POVERTY
    1 YEAR 2 YEARS 3 YEARS 4 YEARS
    EDUCATION
    GENDER
    EQUALITY
    CHILD
    MORTALITY
    MATERNAL
    HEALTH
    HIV AND
    MALARIA
    Source: Millennium Development Goals Report 2011

    View Slide

  7. Ok, so now what?

    View Slide


  8. millions of networked sensors are being embedded in the
    physical world in devices such as mobile phones and
    automobiles, sensing, creating, and communicating data
    OPPORTUNITY 1:
    Proliferation of sensors

    View Slide

  9. Source: McKinsey Global Institute (2011) Big data: The next frontier for innovation, competition, and productivity

    30 billion pieces of content shared on Facebook every month.

    40% projected growth in global data generated per year
    vs 5% in global IT spending.

    increasingly massive amount of information out there, problem is separating
    signal from noise
    OPPORTUNITY 2:
    Big data

    View Slide


  10. we can engage human beings all over the world and leverage their situational
    awareness, expertise and capacity.
    OPPORTUNITY 3:
    Social & collaboration technologies

    View Slide


  11. human capacity to sift through data to detect and understand meaningful
    connections can be vastly magnified

    combine human understanding and computational reasoning capacity to take
    advantage of the strengths of both
    OPPORTUNITY 4:
    Assistive reasoning

    View Slide


  12. 5 billion mobile phones

    the international community has sta on the ground in many developing
    countries

    the reach of the organization can be increased through engaging others outside
    of the organization via social technologies

    challenge is communication
    OPPORTUNITY 5:
    People

    View Slide

  13. Weak signals:
    Advanced, noisy and socially situated
    indicators of change in trends and systems
    that constitute raw informational material
    for enabling anticipatory action.
    Wikipedia (2011) Future studies. Available at http://en.wikipedia.org/wiki/Future_studies

    View Slide

  14. View Slide

  15. View Slide

  16. HunchWorks Overview

    View Slide


  17. a hypothesis management system to detect weak signals

    both humans and data mining tools (wrapped as intelligent
    agents) can post ‘hunches’ about anomalous or interesting data
    and observations that might indicate that a social crisis is
    starting to unfold.

    a tool to engage and connect the global network of UN and
    international development field sta with the greater institution
    to rapidly collect information about emerging situations
    What is HunchWorks?
    17

    View Slide

  18. 1. Detect an interesting “weak signal”
    2. Hypothesize (make a hunch) about the nature of the signal
    3. Share the hunch with the network
    4. Hunch acts as an attractor for related signals and other evidence
    5. Engage communities of practice to support rapid verification
    through the social graph
    6. Use confirmed hunch as a basis for further action and to refine
    trust within the system
    The hunch mechanism
    18

    View Slide

  19. HunchWorks: a use case

    View Slide

  20. Kissa: analyst
    Kissa works in Pulse Lab, Uganda.
    She notices an increase in population
    movements from Kenya and suspects that
    it is in response to the unfolding drought
    in the Horn of Africa but lacks concrete
    evidence.
    She decides to post to HunchWorks to
    gather further evidence.
    20

    View Slide

  21. 21

    View Slide

  22. 22

    View Slide

  23. 23

    View Slide

  24. 24

    View Slide

  25. 25

    View Slide

  26. 26

    View Slide

  27. 27

    View Slide

  28. 28

    View Slide

  29. 29

    View Slide

  30. 30

    View Slide

  31. 31

    View Slide

  32. 32

    View Slide

  33. Resolving a hunch 33
    Once Kissa has determined that enough credible evidence is
    amassed, she may opt to change the status of the hunch to
    reflect the hunch being:
    • proven
    • disproven
    • closed
    We are also considering allowing contributors to vote on hunches
    where the hunch owner allows it or for whatever reason is no

    View Slide

  34. Designing the HunchWorks Experience

    View Slide

  35. View Slide

  36. View Slide

  37. View Slide

  38. View Slide

  39. View Slide

  40. View Slide

  41. View Slide

  42. View Slide

  43. View Slide

  44. View Slide

  45. View Slide

  46. View Slide

  47. View Slide

  48. View Slide

  49. View Slide

  50. View Slide

  51. View Slide

  52. View Slide

  53. View Slide

  54. View Slide

  55. View Slide

  56. View Slide

  57. People are complicated.

    View Slide

  58. Saving the world is complicated.

    View Slide

  59. Trying to save the world with people?

    View Slide

  60. photo credit: flickr.com/photos/johnmcnab/4248698746

    View Slide

  61. photo credit: flickr.com/photos/johnmcnab/4248698746

    View Slide

  62. 7 billion people.

    View Slide

  63. 7 billion people.
    5 billion mobile phones.

    View Slide

  64. People are more than tailpipes.

    View Slide

  65. People are more than tailpipes.
    7 billion HunchWorks users?

    View Slide

  66. From 7 billion to 2,000.

    View Slide

  67. HunchWorks Users

    View Slide

  68. View Slide

  69. View Slide

  70. View Slide

  71. View Slide

  72. View Slide

  73. View Slide

  74. How?

    View Slide

  75. How?

    View Slide

  76. How?
    Why?

    View Slide

  77. Tools alone will not be enough
    to sustain HunchWorks.

    View Slide

  78. Tools alone will not be enough
    to sustain HunchWorks.
    We must build a strong community
    around these tools.

    View Slide

  79. Leverage existing connections…

    View Slide

  80. Leverage existing connections…
    and suggest new ones.

    View Slide

  81. View Slide

  82. View Slide

  83. View Slide

  84. View Slide

  85. View Slide

  86. Connections depend on trust.

    View Slide

  87. View Slide

  88. View Slide

  89. View Slide

  90. View Slide

  91. A picture of a hunch.

    View Slide

  92. A picture of a hunch.

    View Slide

  93. A hunch is the discussion surrounding it.

    View Slide

  94. View Slide

  95. View Slide

  96. View Slide

  97. View Slide

  98. HunchWorks is an action engine.

    View Slide

  99. HunchWorks is an action engine.
    “Is this hunch true enough to
    do something about it?”

    View Slide

  100. Making that action happen.

    View Slide

  101. Open Source and the UN

    View Slide

  102. Open Source and the UN
    Code Jams
    Blue Hacks
    Open Innovation
    Git
    Repositories
    Hackathons
    Licensing
    Models
    Facilitation

    View Slide

  103. Integration

    View Slide

  104. Algorithms

    Similarity and complementarity metrics

    Relevance

    Credibility

    Hunch splitting, merging and clustering

    Mixing human insight with big data results

    View Slide

  105. System Issues

    Security

    Localisation

    Federation

    View Slide

  106. Federation
    ?
    HUNCHWORKS
    DATA FEED
    DATA SITES
    (OPEN DATA,
    MAPS, GAZETTEERS ETC)
    ?
    HUNCHWORKS
    ?
    HUNCHWORKS
    ?
    HUNCHWORKS
    ?
    HUNCHWORKS
    ?
    HUNCHWORKS
    Intermittent
    connection
    No connection/sneakerware

    View Slide

  107. Timeline

    View Slide

  108. Roadmap

    to November 2011 (prototype delivery)
    Hunchworks is in the Proof of Concept stage.

    From November 2011 til January 2012 (first release)
    Hunchworks is in seed stage (early development), looking for
    angel-style funding through UN Foundation.

    From Feb 2012 till Jan 2013 (first full-featured release)
    HunchWorks is in early stage development

    View Slide

  109. We need your help!

    Funding

    Beta Communities

    Development partners

    Did I mention funding...?

    View Slide

  110. Thank you
    Chris van der Walt
    @cvanderwalt
    Dane Petersen
    @thegreatsunra
    Sara Farmer
    @bodaceacat
    www.unglobalpulse.org
    @unglobalpulse
    [email protected]

    View Slide