$30 off During Our Annual Pro Sale. View Details »

HunchWorks: Combining Human Expertise and Big Data

Dane Petersen
September 23, 2011

HunchWorks: Combining Human Expertise and Big Data

O'Reilly Strata Conference
New York City, NY
September 23, 2011

Dane Petersen

September 23, 2011

More Decks by Dane Petersen

Other Decks in Design


  1. HunchWorks: Combining human expertise and big data September 23rd, 2011

    www.unglobalpulse.org @unglobalpulse #hunchworks
  2. Chris van der Walt, Project Lead, HunchWorks, Global Pulse @cvanderwalt

    Dane Petersen, Experience Designer, Adaptive Path @thegreatsunra Sara Farmer, Chief Platform Architect, Global Pulse @bodaceacat
  3. Harnessing innovation to protect the vulnerable

  4. Vulnerability: The risk of households falling in or remaining in

    poverty due to idiosyncratic or aggregate hazards. Source: Professor Wim Naudé, United Nations University
  5. 1.373 billion people living on less than $1.25 per day

    and billions more at risk of falling into extreme poverty Source: The World Bank
  6. Information Gap HUNGER AND POVERTY 1 YEAR 2 YEARS 3

  7. Ok, so now what?

  8. ‣ millions of networked sensors are being embedded in the

    physical world in devices such as mobile phones and automobiles, sensing, creating, and communicating data OPPORTUNITY 1: Proliferation of sensors
  9. Source: McKinsey Global Institute (2011) Big data: The next frontier

    for innovation, competition, and productivity ‣ 30 billion pieces of content shared on Facebook every month. ‣ 40% projected growth in global data generated per year vs 5% in global IT spending. ‣ increasingly massive amount of information out there, problem is separating signal from noise OPPORTUNITY 2: Big data
  10. ‣ we can engage human beings all over the world

    and leverage their situational awareness, expertise and capacity. OPPORTUNITY 3: Social & collaboration technologies
  11. ‣ human capacity to sift through data to detect and

    understand meaningful connections can be vastly magnified ‣ combine human understanding and computational reasoning capacity to take advantage of the strengths of both OPPORTUNITY 4: Assistive reasoning
  12. ‣ 5 billion mobile phones ‣ the international community has

    sta on the ground in many developing countries ‣ the reach of the organization can be increased through engaging others outside of the organization via social technologies ‣ challenge is communication OPPORTUNITY 5: People
  13. Weak signals: Advanced, noisy and socially situated indicators of change

    in trends and systems that constitute raw informational material for enabling anticipatory action. Wikipedia (2011) Future studies. Available at http://en.wikipedia.org/wiki/Future_studies
  14. None
  15. None
  16. HunchWorks Overview

  17. ‣ a hypothesis management system to detect weak signals ‣

    both humans and data mining tools (wrapped as intelligent agents) can post ‘hunches’ about anomalous or interesting data and observations that might indicate that a social crisis is starting to unfold. ‣ a tool to engage and connect the global network of UN and international development field sta with the greater institution to rapidly collect information about emerging situations What is HunchWorks? 17
  18. 1. Detect an interesting “weak signal” 2. Hypothesize (make a

    hunch) about the nature of the signal 3. Share the hunch with the network 4. Hunch acts as an attractor for related signals and other evidence 5. Engage communities of practice to support rapid verification through the social graph 6. Use confirmed hunch as a basis for further action and to refine trust within the system The hunch mechanism 18
  19. HunchWorks: a use case

  20. Kissa: analyst Kissa works in Pulse Lab, Uganda. She notices

    an increase in population movements from Kenya and suspects that it is in response to the unfolding drought in the Horn of Africa but lacks concrete evidence. She decides to post to HunchWorks to gather further evidence. 20
  21. 21

  22. 22

  23. 23

  24. 24

  25. 25

  26. 26

  27. 27

  28. 28

  29. 29

  30. 30

  31. 31

  32. 32

  33. Resolving a hunch 33 Once Kissa has determined that enough

    credible evidence is amassed, she may opt to change the status of the hunch to reflect the hunch being: • proven • disproven • closed We are also considering allowing contributors to vote on hunches where the hunch owner allows it or for whatever reason is no
  34. Designing the HunchWorks Experience

  35. None
  36. None
  37. None
  38. None
  39. None
  40. None
  41. None
  42. None
  43. None
  44. None
  45. None
  46. None
  47. None
  48. None
  49. None
  50. None
  51. None
  52. None
  53. None
  54. None
  55. None
  56. None
  57. People are complicated.

  58. Saving the world is complicated.

  59. Trying to save the world with people?

  60. photo credit: flickr.com/photos/johnmcnab/4248698746

  61. photo credit: flickr.com/photos/johnmcnab/4248698746

  62. 7 billion people.

  63. 7 billion people. 5 billion mobile phones.

  64. People are more than tailpipes.

  65. People are more than tailpipes. 7 billion HunchWorks users?

  66. From 7 billion to 2,000.

  67. HunchWorks Users

  68. None
  69. None
  70. None
  71. None
  72. None
  73. None
  74. How?

  75. How?

  76. How? Why?

  77. Tools alone will not be enough to sustain HunchWorks.

  78. Tools alone will not be enough to sustain HunchWorks. We

    must build a strong community around these tools.
  79. Leverage existing connections…

  80. Leverage existing connections… and suggest new ones.

  81. None
  82. None
  83. None
  84. None
  85. None
  86. Connections depend on trust.

  87. None
  88. None
  89. None
  90. None
  91. A picture of a hunch.

  92. A picture of a hunch.

  93. A hunch is the discussion surrounding it.

  94. None
  95. None
  96. None
  97. None
  98. HunchWorks is an action engine.

  99. HunchWorks is an action engine. “Is this hunch true enough

    to do something about it?”
  100. Making that action happen.

  101. Open Source and the UN

  102. Open Source and the UN Code Jams Blue Hacks Open

    Innovation Git Repositories Hackathons Licensing Models Facilitation
  103. Integration

  104. Algorithms ‣ Similarity and complementarity metrics ‣ Relevance ‣ Credibility

    ‣ Hunch splitting, merging and clustering ‣ Mixing human insight with big data results
  105. System Issues ‣ Security ‣ Localisation ‣ Federation


    GAZETTEERS ETC) ? HUNCHWORKS ? HUNCHWORKS ? HUNCHWORKS ? HUNCHWORKS ? HUNCHWORKS Intermittent connection No connection/sneakerware
  107. Timeline

  108. Roadmap ‣ to November 2011 (prototype delivery) Hunchworks is in

    the Proof of Concept stage. ‣ From November 2011 til January 2012 (first release) Hunchworks is in seed stage (early development), looking for angel-style funding through UN Foundation. ‣ From Feb 2012 till Jan 2013 (first full-featured release) HunchWorks is in early stage development
  109. We need your help! ‣ Funding ‣ Beta Communities ‣

    Development partners ‣ Did I mention funding...?
  110. Thank you Chris van der Walt @cvanderwalt Dane Petersen @thegreatsunra

    Sara Farmer @bodaceacat www.unglobalpulse.org @unglobalpulse info@unglobalpulse.org