Slide 1
Slide 1 text
Marco De Nadai
Are Safer Looking
Neighborhoods More Lively?
A multimodal Investigation
into Urban Life
Slide 2
Slide 2 text
Who am I?
2
• Ph.D. student in Computer science
• Reality: I’m in Computational Social Science
• University of Trento (Italy) - FBK
Slide 3
Slide 3 text
Outline
3
• Motivation to study cities
• How we do it
• Security perception
• Methods
• Results
• Next
Slide 4
Slide 4 text
Cities, very difficult to explain
4
COMPLEX
Not only agglomeration of
residents, factories, shops
• Millions of individuals
• Continuously evolving
A small change generates a
cascading throughout
Slide 5
Slide 5 text
The role of data mining and machine learning
5
DATA MINING
• Inexpensive way to understand mechanisms;
• New stimulus to social research;
MACHINE LEARNING
• New tools to expand the notion of what is predictable;
Shmueli, Galit. "To explain or to predict?." Statistical science 25, no. 3 (2010): 289-310.
Slide 6
Slide 6 text
Predict deprivation and per-capita
income solely relying on mobility
diversity and social diversity
6
Pappalardo, L., Vanhoof, M., Gabrielli, L., Smoreda, Z.,
Pedreschi, D., & Giannotti, F. (2016). An analytical framework to
nowcast well-being using mobile phone data. International
Journal of Data Science and Analytics
Slide 7
Slide 7 text
Predict poverty from satellite
imagery
(75% variation of economic outcomes)
7
Jean, N., Burke, M., Xie, M., Davis, W. M., Lobell, D. B., & Ermon, S.
(2016). Combining satellite imagery and machine learning to
predict poverty. Science
Slide 8
Slide 8 text
Predict crime rates from POIs,
mobility, demographics
8
Wang, H., Kifer, D., Graif, C., & Li, Z. (2016). Crime rate inference
with big data. In Proceedings of the 22nd ACM SIGKDD.
Slide 9
Slide 9 text
Understand underlying mechanisms of a city
9
Arbesman, Samuel. Overcomplicated: Technology at the Limits of Comprehension. Penguin, 2016.
Slide 10
Slide 10 text
Understand underlying mechanisms of a city
10
Arbesman, Samuel. Overcomplicated: Technology at the Limits of Comprehension. Penguin, 2016.
Describe
Slide 11
Slide 11 text
Understand underlying mechanisms of a city
11
Arbesman, Samuel. Overcomplicated: Technology at the Limits of Comprehension. Penguin, 2016.
Describe Predict
Slide 12
Slide 12 text
Understand underlying mechanisms of a city
12
Arbesman, Samuel. Overcomplicated: Technology at the Limits of Comprehension. Penguin, 2016.
Describe Predict Generate
Slide 13
Slide 13 text
Understand underlying mechanisms of a city
13
Arbesman, Samuel. Overcomplicated: Technology at the Limits of Comprehension. Penguin, 2016.
Describe Predict Generate
Slide 14
Slide 14 text
Multi-modal understanding
14
Slide 15
Slide 15 text
A multitude of dimensions and aspects!
15
DESCRIBE
&
PREDICT
PREDICT
&
GENERATE
Crime
3
Urban vitality
1
Security perception
2
Structural design
4
Slide 16
Slide 16 text
Security perception
SUBJECTIVE CHARACTERISTICS
Q: how can new sources of data and deep learning models help to
link urban visual perception and the behavior of people?
Slide 17
Slide 17 text
Broken windows theory
• City mismanagement
• Dirty places
• Poor infrastructure
Lead to misbehavior => Crime
17
Wilson, James Q., and George L. Kelling. "Broken windows."
Critical issues in policing: Contemporary readings (1982): 395-
407.
2
1 3
SECURITY PERCEPTION 4
Slide 18
Slide 18 text
Jane Jacobs and Newman
Two of the most influential books in
city planning
• Lit streets
• Street-facing windows
• Physical demarcation private-public
18
Klemek, C. (2011) ‘Dead or Alive at Fifty? Reading Jane Jacobs
on her Golden Anniversary’ Dissent, Vol. 58, No. 2, 75–79.
2
1 3
URBAN VITALITY 4
Slide 19
Slide 19 text
Are people avoiding places
where they feel unsafe?
QUESTION
Slide 20
Slide 20 text
Answer by surveys!
• High costs
• Not scalable
• # people
• # cities
• How should we design the survey?
20
Slide 21
Slide 21 text
Visual perception and liveliness
AUTOMATICALLY LINKED
Slide 22
Slide 22 text
The multi-modal approach
22
SAFETY PERCEPTION LIVELINESS
1 2
MULTI- MODAL APPROACH
Slide 23
Slide 23 text
Safety perception: MIT Place Pulse
23
1 2
MULTI- MODAL APPROACH
Slide 24
Slide 24 text
Place Pulse 2.0
24
Salesses, P., Schechtner, K., & Hidalgo, C. A. (2013). The collaborative image of the city: mapping the
inequality of urban perception. PloS one
1 2
MULTI- MODAL APPROACH
Slide 25
Slide 25 text
Safety perception: MIT Place Pulse
25
…
1
10
1 2
MULTI- MODAL APPROACH
Place Pulse
• New York
• Boston
• Linz
• Salzburg
Place Pulse 2
• Rome
• Milan
PROBLEM:
• Just thousands of examples
• Sparse in space and # votes
Slide 26
Slide 26 text
Safety perception: fix sparse images
• From 6.7 to 100 images per km^2
• 360 degrees of images
• Learning safety perception,
predict in Rome and Milan
• Aggregation per district
26
1 2
MULTI- MODAL APPROACH
Slide 27
Slide 27 text
Deep Learning Networks (DNNs)
• How do you extract features?
• Handcraft features
• Implicitly (Kernels)
• Self-taught learning (learn representation)
• Black boxes
• Hard to train
27
1 2
MULTI- MODAL APPROACH
Slide 28
Slide 28 text
DNNs: Representation
28
1 2
MULTI- MODAL APPROACH
* From Paolo Frasconi slides – Deep Learning course – University of Trento
Slide 29
Slide 29 text
DNNs: Representation
29
1 2
MULTI- MODAL APPROACH
Slide 30
Slide 30 text
DNNs: Representation
30
1 2
MULTI- MODAL APPROACH
Slide 31
Slide 31 text
DNNs: pre-trained Imagenet/Places
31
1 2
MULTI- MODAL APPROACH
• Trained on 1.2 millions of images
Hard to train?
Slide 32
Slide 32 text
Deep Neural Networks
32
1 2
MULTI- MODAL APPROACH
x1
x2
x3
Layer 1 Layer 2
Layer 4
Layer 3
x4
Score
Slide 33
Slide 33 text
Safety perception: fix sparse votes
• Learning safety perception,
predict in Rome and Milan
• Standard architecture AlexNet CNN
33
* B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva. “Learning Deep Features for Scene Recognition using
Places Database.” NIPS, 2014.
1 2
MULTI- MODAL APPROACH
Slide 34
Slide 34 text
Safety perception: fix sparse votes
• Learning safety perception,
predict in Rome and Milan
• Standard architecture AlexNet CNN
34
* B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva. “Learning Deep Features for Scene Recognition using
Places Database.” NIPS, 2014.
Model type
NY - NY
NY - Boston
Boston - Boston
Boston - NY
1 2
MULTI- MODAL APPROACH
Slide 35
Slide 35 text
Safety perception: fix sparse votes
• Learning safety perception,
predict in Rome and Milan
• Standard architecture AlexNet CNN
35
* B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva. “Learning Deep Features for Scene Recognition using
Places Database.” NIPS, 2014.
** Ordonez, Vicente, and Tamara L. Berg. "Learning high-level judgments of urban perception.” ECCV, 2014.
Model type State of art [**] Our model
NY - NY 0.687 0.718
NY - Boston 0.701 0.734
Boston - Boston 0.718 0.744
Boston - NY 0.636 0.693
1 2
MULTI- MODAL APPROACH
Slide 36
Slide 36 text
Safety perception: aggregation
36
1 2
MULTI- MODAL APPROACH
3.5 3.7 3.9 4.2 4.4 4.6 4.8 5.0
Safety score
DUOMO
SAN SIRO
QUARTO
OGGIARO
CITTA'
STUDI
BICOCCA
TRASTEVERE
TIBURTINO
OSTIENSE
PRIMAVALLE
LESS SAFE SAFER
ROME MILAN
Slide 37
Slide 37 text
Liveliness
Slide 38
Slide 38 text
1 2
Liveliness: mobile phone data
• Anonymized mobile phone calls
activity as a proxy for urban
liveliness
• Broken down by gender, age
• 3 months time span (in 2015)
• Rome and Milan
38
MULTI- MODAL APPROACH
Slide 39
Slide 39 text
Liveliness: metrics
• People around in district
|',)*+
|
|'
|
• Fraction females
|',)*+
|
|',)*+
|
• People below 30 (and above 50)
|(< 30)',)*+
|
|',)*+
|
39
0.2
0.0
0.2
0.4
0.6
0.8
1.0
Land Use Mix
0.8 1.2 1.8 2.7 4.1 6.1 9.3 14.0 21.1 31.9
Activity density ⇥ 103
ROME
MILAN
0.8 1.2 1.8 2.7 4.1 6.1 9.3 14.0 21.1 31.9
Activity density 103
MILAN
MULTI- MODAL APPROACH 1 2
Slide 40
Slide 40 text
1
Link: regression
40
4.2 4.4 4.6 4.8 5.0
afety score
DUOMO
SAN SIRO
QUARTO
OGGIARO
CITTA'
STUDI
BICOCCA
MULTI- MODAL APPROACH 2
0.2
0.0
0.2
0.4
0.6
0.8
1.0
1.2
Land Use Mix
0.8 1.2 1.8 2.7 4.1 6.1 9.3 14.0 21.1 3
Activity density ⇥ 103
MILAN
SAFETY PERCEPTION LIVELINESS
Slide 41
Slide 41 text
= :
:
+ )
)
+ ⋯ + >
>
+
The Linear Regression
41
Liveliness metric
1
MULTI- MODAL APPROACH 2
Slide 42
Slide 42 text
The Linear Regression
42
Liveliness metric
Perceived safety
= :
:
+ )
)
+ ⋯ + >
>
+
1
MULTI- MODAL APPROACH 2
Slide 43
Slide 43 text
The Linear Regression
43
Liveliness metric
Perceived safety
Population density
= :
:
+ )
)
+ ⋯ + >
>
+
1
MULTI- MODAL APPROACH 2
Slide 44
Slide 44 text
Spatial auto-correlation
44
Getis, Arthur, and Daniel A. Griffith. "Comparative spatial filtering in
regression analysis." Geographical analysis 34.2 (2002): 130-140.
“everything is related to everything else,
but near things are more related than
distant things.”
Tobler's first law of geography
• We control for spatial auto-correlation
1 2
MULTI- MODAL APPROACH
Slide 45
Slide 45 text
The multi-modal approach
45
(SPATIAL) OLS
REGRESSION
SAFETY PERCEPTION LIVELINESS
1
MULTI- MODAL APPROACH 2
Slide 46
Slide 46 text
Safety perception and liveliness
46
1
Presence
of people
Women, Young people
around
Visual elements
for safety
2 3
Slide 47
Slide 47 text
Presence of people
1st
Slide 48
Slide 48 text
2
1
Safety perception <-> presence of people
48
3
TESTS
Urban metric Beta coefficient P-value
Population density
Employees density
Deprivation
Distance from the center
Safety appearance
adj − R) 0.91
Slide 49
Slide 49 text
2
1
Safety perception <-> presence of people
49
3
TESTS
Urban metric Beta coefficient P-value
Population density 0.155 **
Employees density 0.328 **
Deprivation -0.022
Distance from the center -0.257 **
Safety appearance 0.105 **
adj − R) 0.91
** p-value < 0.001; * p-value < 0.01;
Slide 50
Slide 50 text
Women and young people around
2nd
Slide 51
Slide 51 text
1 2
51
3
TESTS
** p-value < 0.001; * p-value < 0.01;
Safety perception <-> women around
Urban metric Beta coefficient P-value
Percentage of women (census)
Deprivation
Distance from the center
Safety appearance
adj − R) 0.65
Slide 52
Slide 52 text
1 2
52
3
TESTS
** p-value < 0.001; * p-value < 0.01;
Safety perception <-> women around
Urban metric Beta coefficient P-value
Percentage of women (census) 0.001
Deprivation -0.005
Distance from the center -0.003
Safety appearance 0.020 **
adj − R) 0.65
Slide 53
Slide 53 text
1 2
53
3
TESTS
** p-value < 0.001; * p-value < 0.01;
Safety perception <-> young people around
Urban metric Beta coefficient P-value
Percentage of people < 30 years (census)
Deprivation
Distance from the center
Safety appearance
adj − R) 0.66
Slide 54
Slide 54 text
1 2
54
3
TESTS
** p-value < 0.001; * p-value < 0.01;
Safety perception <-> young people around
Urban metric Beta coefficient P-value
Percentage of people < 30 years (census) -0.001
Deprivation 0.032 **
Distance from the center -0.150 **
Safety appearance -0.048 **
adj − R) 0.66
Slide 55
Slide 55 text
Visual elements for safety
3rd
Slide 56
Slide 56 text
Deep Neural Networks
56
x1
x2
x3
Layer 1 Layer 2
Layer 4
Layer 3
x4
Score
1 2 3
TESTS
Slide 57
Slide 57 text
1 2 3
Visual elements for safety perception
57
TESTS
HIGH SAFETY PERCEPTION
RANDOMLY OBSCURE
PART OF THE IMAGE
AND PREDICT
Slide 58
Slide 58 text
1 2 3
Visual elements for safety perception
58
TESTS
HIGH SAFETY PERCEPTION
RANDOMLY OBSCURE
PART OF THE IMAGE
AND PREDICT
Slide 59
Slide 59 text
1 2 3
Visual elements for safety perception
59
TESTS
LOW SAFETY PERCEPTION
RANDOMLY OBSCURE
PART OF THE IMAGE
AND PREDICT
Slide 60
Slide 60 text
Street facing windows and greenery
contribute positively to the appearance
of safety
1 2 3
TESTS
Slide 61
Slide 61 text
Why does it matter?
Slide 62
Slide 62 text
“What this [paper] does is put the facts on the table,
and that’s a big step”
…
“It will bring up a lot of other research, in which, I
don’t have any doubt, this will be put up as a
seminal step”
Luis Valenzuela, Urban Planner
Harvard University
Source: http://news.mit.edu/2016/quantifying-urban-revitalization-1024
Slide 63
Slide 63 text
Useful!
• Evaluate the districts vitality through visual appearance
• Plan regulatory interventions
• Mobile phone data as valid proxy to census
63
Barlacchi, Gianni, et al. "A multi-source dataset of urban life in the city of Milan and the Province of
Trentino." Scientific data 2 (2015).
Slide 64
Slide 64 text
Practical applications
• Real estate evaluation (land and buildings)
• Know in advance the best places for retails (age/gender)
• Analyze economic shocks in a region and respond to them
64
Slide 65
Slide 65 text
In the (next) future
• Temporal dimension (aesthetic change, people’s behavior)
• How urban perception influences our mobility?
65
A B
Slide 66
Slide 66 text
66
1
Presence
of people
Women, Young people
around
Visual elements
for safety
2 3
Safety perception and liveliness
a multi-modal approach
Slide 67
Slide 67 text
Thanks
@denadai2