Learning Engineer () - lived and worked in Rotterdam () and London () Currently Data Science Manager at Bumble Inc for Trust & Safety @massibelloni /massibelloni [email protected]
give easy (and free!) access to hundreds of millions of people around the world. Everyone is on them. 2 People can be reached on their personal devices, often at home and potentially in vulnerable situations. One-to-one conversations are powerful (and risky) tools 3 Technology advancements don’t just benefit good actors, but also bad ones. Better software, artificial intelligence, etc. can massively improve the reach and quality of scam attacks. 4 Spam and scam on social media aren’t just noisy and annoying, but can lead to real crimes and destroy people lives.
content and users even before their account is active or their posts are published. 0 2 Be fast Constantly get signals from users and content to be as fast as possible to take action and reduce spread. 0 3 Learn from mistakes Continuously investigate bad content and patterns to improve the systems, especially when they fail.
are they based? Which country they say they are based? • Which email address (and domain!) they used to register? • Weird username? Weird characters? • Phone verification? ID verification? Which proof do we have about who they claim they are? • Device footprint? User agent and OS? • Are they using a VPN? • ….
currencies ($£€) mentioned? • Emojis? Digits? • Any easy to spot regular expression? • Is the whole paragraph known to us? • Does the text trigger any machine learning model?
is suspicious? 2 Any user logging in from country xyz has malicious intent? 5 How many times a user can mention about money in the same message? 6 Is it ok to use too many emojis in the username? 3 Is it ok to use the platform from a country that’s different from the one selected during registration? 7 Are words like “sell”, “success”, “crypto”, “buy” related to scams more generally? …….
used to manage data, especially in a relational database management system. It is particularly useful in handling structured data, i.e., data incorporating relations among entities and variables. Wikipedia Insert data Analyse data Structure data format
456 massibelloni ES ES 789 ilovecats95 UK JP Reporter Post ID 123 ABC 456 DEF Post ID Author Text ABC 456 Such a sunny day DEF 123 Invest in ETH to make $$$ GHI 123 Paid a coffee 10$ JKL 789 Slept 12 hours ytd night
REPORTS r WHERE r.POST_ID = p.POST_ID) AND (text LIKE “%$%” or text LIKE “%€%”) SELECT count(*) FROM USERS u WHERE exists ( SELECT * FROM REPORTS r JOIN POSTS p ON r.POST_ID = p.POST_ID WHERE p.AUTHOR = u.USER_ID ) AND country != country_ip
REPORTS r WHERE r.POST_ID = p.POST_ID) AND (text LIKE “%$%” or text LIKE “%€%”) SELECT count(*) FROM USERS u WHERE exists ( SELECT * FROM REPORTS r JOIN POSTS p ON r.POST_ID = p.POST_ID WHERE p.AUTHOR = u.USER_ID ) AND country != country_ip Number of reported posts where $ or € is mentioned at least once.
REPORTS r WHERE r.POST_ID = p.POST_ID) AND (text LIKE “%$%” or text LIKE “%€%”) SELECT count(*) FROM USERS u WHERE exists ( SELECT * FROM REPORTS r JOIN POSTS p ON r.POST_ID = p.POST_ID WHERE p.AUTHOR = u.USER_ID ) AND country != country_ip Number reported users (authors of at least one reported post!) that logged in from a country different from the registered one
x while their registered country is y 53% Users that use the app connected through a VPN eventually reported 95% Users connecting from country X while their registered country is not X mentioning $$$ more than three times get eventually banned NOTE: all this statistics are completely random and for educational purposes only!
computer system in which domain-specific knowledge is represented in the form of rules and general-purpose reasoning is used to solve problems in the domain. Rule-based systems emerged within the field of artificial intelligence in the 1970s and [...] use if-then rules to derive actions from conditions Wikipedia (ish) A B C =2 != 2 Action Action Action >5 <= 5
using a VPN is suspicious? 2 Any user logging in from country xyz has malicious intent? 5 How many times a user can mention about money in the same message? 6 Is it ok to use too many emojis in the username? 3 Is it ok to use the platform from a country that’s different from the one selected during registration? 7 Are words like “sell”, “success”, “crypto”, “buy” related to scams more generally? …….
using a VPN is suspicious? 2 Any user logging in from country xyz has malicious intent? 5 How many times a user can mention about money in the same message? 6 Is it ok to use too many emojis in the username? 3 Is it ok to use the platform from a country that’s different from the one selected during registration? 7 Are words like “sell”, “success”, “crypto”, “buy” related to scams more generally?
X country X Leveraging the insights collected from data exploration and all the information we collect during profile registration and post submission we can derive rules as deep and complex as we wish to approve or reject posts and potentially block users! != registration ✅ All good! yes no ✅ All good! ❌ Block
X country X != registration ✅ All good! yes no ✅ All good! ❌ Block We can trigger one check (or multiple checks in parallel!) when a new post is submitted (prior to publishing) or every time a new user is registered or logs in! Deciding when to trigger a rule and collecting relevant information (feature engineering) is 90% of the task
c t e d Good ✅ Bad ❌ Actual Good ✅ Bad ❌ False Positive FP True Positive TP False Negative FN True Negative TN How reliable the system is when it says that something is bad? How much does the system alone solve for the issue?
is not X mentioning $$$ more than three times get eventually banned 3% Amount of banned users that are described from the pattern above Precision Recall
All good! ✅ All good! ❌ Block ✅ All good! ✅ All good! ✅ All good! ll good! ✅ All good! ✅ All good! ✅ All good! ✅ All good! ✅ All good! ✅ All good! ✅ All good! ~80% ~70% Precision Recall NOTE: all this statistics are completely random and for educational purposes only!
understand Direct actioning of data insights Require human expertise and knowledge to improve Complexity explodes quite quickly with the amount of rules Good performance if the data available is rich and accurate enough Their output doesn’t include confidence - binary by design
study in artificial intelligence concerned with the development and study of statistical algorithms that can learn from data and generalize to unseen data, and thus perform tasks without explicit instructions. Wikipedia ML History Reality
model is trained on a history of bad content and good content and tries to mimic the behavior of human reviewers/moderators in order to autonomously take decisions on this task.
It’s an iterative trial and error approach where we experiment with all possible lines and we try to research the one that describes our points better (minimum error!) b m
= 3.123 x + 7.765 3.123 and 7.765 are our model’s weights and contain all the knowledge that our data have. They can be used to predict the future (in 2D!)
model is trained on a history of bad content and good content and tries to mimic the behavior of human reviewers/moderators in order to autonomously take decisions on this task.
- allows flexibility Requires lots of good data to train on High expertise to design and maintain Easy to maintain and improve - centralised Not reactive to changes in behaviors
All good! ✅ All good! ❌ Block ✅ All good! ✅ All good! ✅ All good! ll good! ✅ All good! ✅ All good! ✅ All good! ✅ All good! ✅ All good! ✅ All good! ✅ All good! ML
able to automatically block 95% of accounts that were identified as spam or scam accounts. Within the first two months of testing the tool, Bumble saw user reports of spam, scams and fake accounts reduced by 45%. Deception Detector operates alongside Bumble’s human moderation team.
massibelloni
iotcomjpadmin
itohiro73
hrsued
abenben
lamodatech
kzkmaeda
kotauchisunsun
tiltmax3
shift_evolve
nagisa53
rocio
chrisshort
shpigford
smashingmag
chriscoyier
csswizardry
ufuk
scottboms
dougneiner
jcasabona
lara
