Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Are Safer Looking Neighborhoods More Lively? A multimodal Investigation into Urban Life

Are Safer Looking Neighborhoods More Lively? A multimodal Investigation into Urban Life

Presentation for the Responsive Environments: City eMotion course of Harvard University 2017.
Study from Google street view images in order to understand safety perception in urban enviroment with Deep Learning techniques. Original paper: https://arxiv.org/abs/1608.00462

Marco De Nadai

March 30, 2017
Tweet

More Decks by Marco De Nadai

Other Decks in Research

Transcript

  1. Marco De Nadai
    Are Safer Looking
    Neighborhoods More Lively?
    A multimodal Investigation
    into Urban Life

    View Slide

  2. Who am I?
    2
    • Ph.D. student in Computer science
    • Reality: I’m in Computational Social Science
    • University of Trento (Italy) - FBK

    View Slide

  3. Outline
    3
    • Motivation to study cities
    • How we do it
    • Security perception
    • Methods
    • Results
    • Next

    View Slide

  4. Cities, very difficult to explain
    4
    COMPLEX
    Not only agglomeration of
    residents, factories, shops
    • Millions of individuals
    • Continuously evolving
    A small change generates a
    cascading throughout

    View Slide

  5. The role of data mining and machine learning
    5
    DATA MINING
    • Inexpensive way to understand mechanisms;
    • New stimulus to social research;
    MACHINE LEARNING
    • New tools to expand the notion of what is predictable;
    Shmueli, Galit. "To explain or to predict?." Statistical science 25, no. 3 (2010): 289-310.

    View Slide

  6. Predict deprivation and per-capita
    income solely relying on mobility
    diversity and social diversity
    6
    Pappalardo, L., Vanhoof, M., Gabrielli, L., Smoreda, Z.,
    Pedreschi, D., & Giannotti, F. (2016). An analytical framework to
    nowcast well-being using mobile phone data. International
    Journal of Data Science and Analytics

    View Slide

  7. Predict poverty from satellite
    imagery
    (75% variation of economic outcomes)
    7
    Jean, N., Burke, M., Xie, M., Davis, W. M., Lobell, D. B., & Ermon, S.
    (2016). Combining satellite imagery and machine learning to
    predict poverty. Science

    View Slide

  8. Predict crime rates from POIs,
    mobility, demographics
    8
    Wang, H., Kifer, D., Graif, C., & Li, Z. (2016). Crime rate inference
    with big data. In Proceedings of the 22nd ACM SIGKDD.

    View Slide

  9. Understand underlying mechanisms of a city
    9
    Arbesman, Samuel. Overcomplicated: Technology at the Limits of Comprehension. Penguin, 2016.

    View Slide

  10. Understand underlying mechanisms of a city
    10
    Arbesman, Samuel. Overcomplicated: Technology at the Limits of Comprehension. Penguin, 2016.
    Describe

    View Slide

  11. Understand underlying mechanisms of a city
    11
    Arbesman, Samuel. Overcomplicated: Technology at the Limits of Comprehension. Penguin, 2016.
    Describe Predict

    View Slide

  12. Understand underlying mechanisms of a city
    12
    Arbesman, Samuel. Overcomplicated: Technology at the Limits of Comprehension. Penguin, 2016.
    Describe Predict Generate

    View Slide

  13. Understand underlying mechanisms of a city
    13
    Arbesman, Samuel. Overcomplicated: Technology at the Limits of Comprehension. Penguin, 2016.
    Describe Predict Generate

    View Slide

  14. Multi-modal understanding
    14

    View Slide

  15. A multitude of dimensions and aspects!
    15
    DESCRIBE
    &
    PREDICT
    PREDICT
    &
    GENERATE
    Crime
    3
    Urban vitality
    1
    Security perception
    2
    Structural design
    4

    View Slide

  16. Security perception
    SUBJECTIVE CHARACTERISTICS
    Q: how can new sources of data and deep learning models help to
    link urban visual perception and the behavior of people?

    View Slide

  17. Broken windows theory
    • City mismanagement
    • Dirty places
    • Poor infrastructure
    Lead to misbehavior => Crime
    17
    Wilson, James Q., and George L. Kelling. "Broken windows."
    Critical issues in policing: Contemporary readings (1982): 395-
    407.
    2
    1 3
    SECURITY PERCEPTION 4

    View Slide

  18. Jane Jacobs and Newman
    Two of the most influential books in
    city planning
    • Lit streets
    • Street-facing windows
    • Physical demarcation private-public
    18
    Klemek, C. (2011) ‘Dead or Alive at Fifty? Reading Jane Jacobs
    on her Golden Anniversary’ Dissent, Vol. 58, No. 2, 75–79.
    2
    1 3
    URBAN VITALITY 4

    View Slide

  19. Are people avoiding places
    where they feel unsafe?
    QUESTION

    View Slide

  20. Answer by surveys!
    • High costs
    • Not scalable
    • # people
    • # cities
    • How should we design the survey?
    20

    View Slide

  21. Visual perception and liveliness
    AUTOMATICALLY LINKED

    View Slide

  22. The multi-modal approach
    22
    SAFETY PERCEPTION LIVELINESS
    1 2
    MULTI- MODAL APPROACH

    View Slide

  23. Safety perception: MIT Place Pulse
    23
    1 2
    MULTI- MODAL APPROACH

    View Slide

  24. Place Pulse 2.0
    24
    Salesses, P., Schechtner, K., & Hidalgo, C. A. (2013). The collaborative image of the city: mapping the
    inequality of urban perception. PloS one
    1 2
    MULTI- MODAL APPROACH

    View Slide

  25. Safety perception: MIT Place Pulse
    25

    1
    10
    1 2
    MULTI- MODAL APPROACH
    Place Pulse
    • New York
    • Boston
    • Linz
    • Salzburg
    Place Pulse 2
    • Rome
    • Milan
    PROBLEM:
    • Just thousands of examples
    • Sparse in space and # votes

    View Slide

  26. Safety perception: fix sparse images
    • From 6.7 to 100 images per km^2
    • 360 degrees of images
    • Learning safety perception,
    predict in Rome and Milan
    • Aggregation per district
    26
    1 2
    MULTI- MODAL APPROACH

    View Slide

  27. Deep Learning Networks (DNNs)
    • How do you extract features?
    • Handcraft features
    • Implicitly (Kernels)
    • Self-taught learning (learn representation)
    • Black boxes
    • Hard to train
    27
    1 2
    MULTI- MODAL APPROACH

    View Slide

  28. DNNs: Representation
    28
    1 2
    MULTI- MODAL APPROACH
    * From Paolo Frasconi slides – Deep Learning course – University of Trento

    View Slide

  29. DNNs: Representation
    29
    1 2
    MULTI- MODAL APPROACH

    View Slide

  30. DNNs: Representation
    30
    1 2
    MULTI- MODAL APPROACH

    View Slide

  31. DNNs: pre-trained Imagenet/Places
    31
    1 2
    MULTI- MODAL APPROACH
    • Trained on 1.2 millions of images
    Hard to train?

    View Slide

  32. Deep Neural Networks
    32
    1 2
    MULTI- MODAL APPROACH
    x1
    x2
    x3
    Layer 1 Layer 2
    Layer 4
    Layer 3
    x4
    Score

    View Slide

  33. Safety perception: fix sparse votes
    • Learning safety perception,
    predict in Rome and Milan
    • Standard architecture AlexNet CNN
    33
    * B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva. “Learning Deep Features for Scene Recognition using
    Places Database.” NIPS, 2014.
    1 2
    MULTI- MODAL APPROACH

    View Slide

  34. Safety perception: fix sparse votes
    • Learning safety perception,
    predict in Rome and Milan
    • Standard architecture AlexNet CNN
    34
    * B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva. “Learning Deep Features for Scene Recognition using
    Places Database.” NIPS, 2014.
    Model type
    NY - NY
    NY - Boston
    Boston - Boston
    Boston - NY
    1 2
    MULTI- MODAL APPROACH

    View Slide

  35. Safety perception: fix sparse votes
    • Learning safety perception,
    predict in Rome and Milan
    • Standard architecture AlexNet CNN
    35
    * B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva. “Learning Deep Features for Scene Recognition using
    Places Database.” NIPS, 2014.
    ** Ordonez, Vicente, and Tamara L. Berg. "Learning high-level judgments of urban perception.” ECCV, 2014.
    Model type State of art [**] Our model
    NY - NY 0.687 0.718
    NY - Boston 0.701 0.734
    Boston - Boston 0.718 0.744
    Boston - NY 0.636 0.693
    1 2
    MULTI- MODAL APPROACH

    View Slide

  36. Safety perception: aggregation
    36
    1 2
    MULTI- MODAL APPROACH
    3.5 3.7 3.9 4.2 4.4 4.6 4.8 5.0
    Safety score
    DUOMO
    SAN SIRO
    QUARTO
    OGGIARO
    CITTA'
    STUDI
    BICOCCA
    TRASTEVERE
    TIBURTINO
    OSTIENSE
    PRIMAVALLE
    LESS SAFE SAFER
    ROME MILAN

    View Slide

  37. Liveliness

    View Slide

  38. 1 2
    Liveliness: mobile phone data
    • Anonymized mobile phone calls
    activity as a proxy for urban
    liveliness
    • Broken down by gender, age
    • 3 months time span (in 2015)
    • Rome and Milan
    38
    MULTI- MODAL APPROACH

    View Slide

  39. Liveliness: metrics
    • People around in district
    |',)*+
    |
    |'
    |
    • Fraction females
    |',)*+
    |
    |',)*+
    |
    • People below 30 (and above 50)
    |(< 30)',)*+
    |
    |',)*+
    |
    39
    0.2
    0.0
    0.2
    0.4
    0.6
    0.8
    1.0
    Land Use Mix
    0.8 1.2 1.8 2.7 4.1 6.1 9.3 14.0 21.1 31.9
    Activity density ⇥ 103
    ROME
    MILAN
    0.8 1.2 1.8 2.7 4.1 6.1 9.3 14.0 21.1 31.9
    Activity density 103
    MILAN
    MULTI- MODAL APPROACH 1 2

    View Slide

  40. 1
    Link: regression
    40
    4.2 4.4 4.6 4.8 5.0
    afety score
    DUOMO
    SAN SIRO
    QUARTO
    OGGIARO
    CITTA'
    STUDI
    BICOCCA
    MULTI- MODAL APPROACH 2
    0.2
    0.0
    0.2
    0.4
    0.6
    0.8
    1.0
    1.2
    Land Use Mix
    0.8 1.2 1.8 2.7 4.1 6.1 9.3 14.0 21.1 3
    Activity density ⇥ 103
    MILAN
    SAFETY PERCEPTION LIVELINESS

    View Slide

  41. = :
    :
    + )
    )
    + ⋯ + >
    >
    +
    The Linear Regression
    41
    Liveliness metric
    1
    MULTI- MODAL APPROACH 2

    View Slide

  42. The Linear Regression
    42
    Liveliness metric
    Perceived safety
    = :
    :
    + )
    )
    + ⋯ + >
    >
    +
    1
    MULTI- MODAL APPROACH 2

    View Slide

  43. The Linear Regression
    43
    Liveliness metric
    Perceived safety
    Population density
    = :
    :
    + )
    )
    + ⋯ + >
    >
    +
    1
    MULTI- MODAL APPROACH 2

    View Slide

  44. Spatial auto-correlation
    44
    Getis, Arthur, and Daniel A. Griffith. "Comparative spatial filtering in
    regression analysis." Geographical analysis 34.2 (2002): 130-140.
    “everything is related to everything else,
    but near things are more related than
    distant things.”
    Tobler's first law of geography
    • We control for spatial auto-correlation
    1 2
    MULTI- MODAL APPROACH

    View Slide

  45. The multi-modal approach
    45
    (SPATIAL) OLS
    REGRESSION
    SAFETY PERCEPTION LIVELINESS
    1
    MULTI- MODAL APPROACH 2

    View Slide

  46. Safety perception and liveliness
    46
    1
    Presence
    of people
    Women, Young people
    around
    Visual elements
    for safety
    2 3

    View Slide

  47. Presence of people
    1st

    View Slide

  48. 2
    1
    Safety perception <-> presence of people
    48
    3
    TESTS
    Urban metric Beta coefficient P-value
    Population density
    Employees density
    Deprivation
    Distance from the center
    Safety appearance
    adj − R) 0.91

    View Slide

  49. 2
    1
    Safety perception <-> presence of people
    49
    3
    TESTS
    Urban metric Beta coefficient P-value
    Population density 0.155 **
    Employees density 0.328 **
    Deprivation -0.022
    Distance from the center -0.257 **
    Safety appearance 0.105 **
    adj − R) 0.91
    ** p-value < 0.001; * p-value < 0.01;

    View Slide

  50. Women and young people around
    2nd

    View Slide

  51. 1 2
    51
    3
    TESTS
    ** p-value < 0.001; * p-value < 0.01;
    Safety perception <-> women around
    Urban metric Beta coefficient P-value
    Percentage of women (census)
    Deprivation
    Distance from the center
    Safety appearance
    adj − R) 0.65

    View Slide

  52. 1 2
    52
    3
    TESTS
    ** p-value < 0.001; * p-value < 0.01;
    Safety perception <-> women around
    Urban metric Beta coefficient P-value
    Percentage of women (census) 0.001
    Deprivation -0.005
    Distance from the center -0.003
    Safety appearance 0.020 **
    adj − R) 0.65

    View Slide

  53. 1 2
    53
    3
    TESTS
    ** p-value < 0.001; * p-value < 0.01;
    Safety perception <-> young people around
    Urban metric Beta coefficient P-value
    Percentage of people < 30 years (census)
    Deprivation
    Distance from the center
    Safety appearance
    adj − R) 0.66

    View Slide

  54. 1 2
    54
    3
    TESTS
    ** p-value < 0.001; * p-value < 0.01;
    Safety perception <-> young people around
    Urban metric Beta coefficient P-value
    Percentage of people < 30 years (census) -0.001
    Deprivation 0.032 **
    Distance from the center -0.150 **
    Safety appearance -0.048 **
    adj − R) 0.66

    View Slide

  55. Visual elements for safety
    3rd

    View Slide

  56. Deep Neural Networks
    56
    x1
    x2
    x3
    Layer 1 Layer 2
    Layer 4
    Layer 3
    x4
    Score
    1 2 3
    TESTS

    View Slide

  57. 1 2 3
    Visual elements for safety perception
    57
    TESTS
    HIGH SAFETY PERCEPTION
    RANDOMLY OBSCURE
    PART OF THE IMAGE
    AND PREDICT

    View Slide

  58. 1 2 3
    Visual elements for safety perception
    58
    TESTS
    HIGH SAFETY PERCEPTION
    RANDOMLY OBSCURE
    PART OF THE IMAGE
    AND PREDICT

    View Slide

  59. 1 2 3
    Visual elements for safety perception
    59
    TESTS
    LOW SAFETY PERCEPTION
    RANDOMLY OBSCURE
    PART OF THE IMAGE
    AND PREDICT

    View Slide

  60. Street facing windows and greenery
    contribute positively to the appearance
    of safety
    1 2 3
    TESTS

    View Slide

  61. Why does it matter?

    View Slide

  62. “What this [paper] does is put the facts on the table,
    and that’s a big step”

    “It will bring up a lot of other research, in which, I
    don’t have any doubt, this will be put up as a
    seminal step”
    Luis Valenzuela, Urban Planner
    Harvard University
    Source: http://news.mit.edu/2016/quantifying-urban-revitalization-1024

    View Slide

  63. Useful!
    • Evaluate the districts vitality through visual appearance
    • Plan regulatory interventions
    • Mobile phone data as valid proxy to census
    63
    Barlacchi, Gianni, et al. "A multi-source dataset of urban life in the city of Milan and the Province of
    Trentino." Scientific data 2 (2015).

    View Slide

  64. Practical applications
    • Real estate evaluation (land and buildings)
    • Know in advance the best places for retails (age/gender)
    • Analyze economic shocks in a region and respond to them
    64

    View Slide

  65. In the (next) future
    • Temporal dimension (aesthetic change, people’s behavior)
    • How urban perception influences our mobility?
    65
    A B

    View Slide

  66. 66
    1
    Presence
    of people
    Women, Young people
    around
    Visual elements
    for safety
    2 3
    Safety perception and liveliness
    a multi-modal approach

    View Slide

  67. Thanks
    @denadai2

    View Slide