Keeping It Professional: Relevance, Recommendations, and Reputation at LinkedIn

The CMU Artificial Intelligence Seminar Series presentation discusses three key areas - relevance, recommendations, and reputation - that LinkedIn is focusing on to improve the user experience. Relevance refers to improving search and discovery of profiles by combining different types of signals. Recommendations match members to jobs, groups, and other connections. Reputation involves identifying member expertise in skills and who the experts are in particular skills. Open problems remain around exploratory search, balancing exploration and exploitation of the data, and incentivizing the development of online reputation.

Transcript

1. 1 Recruiting Solutions Recruiting Solutions Recruiting Solutions Keeping It Professional: Relevance, Recommendations, and Reputation. Daniel Tunkelang Principal Data Scientist at LinkedIn Daniel
2. 2 Overview  What is LinkedIn?  Hard problems we’re tackling in:  Relevance  Recommendations  Reputation  Open problems
3. 3 Identity Connect, find and be found LinkedIn Profile, Address Book, Search Insights Be great at what you do Homepage, LinkedIn Today, Groups Work wherever our members work Everywhere Mobile, APIs, Plug-Ins Desktop Rolodex, Resume, Business Card Newspapers, Trade Magazines, Events What is LinkedIn?
4. 4 Identity: Profile of Record
5. 5 Identity: Connect with Others
6. 6 Identity: Join the Conversation
7. 7 Insights: Power of Aggregation Before employees worked at Yahoo! (169) Google (96) Oracle (78) Microsoft (72) IBM (43) Before employees worked at Google(475) Microsoft (448) LinkedIn (169) Apple, Inc. (154) ebay (133)
8. 8 Insights: Market Research
9. 9 Insights: Data Stories
10. 10 Everywhere
11. 11 Hard Problems: Examples  Relevance  People Search  Recommendations  Job Matching  Reputation  Skills
12. 12 People Search
13. 13  120M+ members  2B searches in 2010  Based on (cf. http://sna-projects.com/) People Search: Scale
14. 14 People Search: Faceted Search
15. 15 People Search: Network Facet
16. 16 People Search: Type-Ahead
17. 17  Query-Independent Signals  Network Rank, Profile Quality  Query-Dependent Signals  Field-Based Relevance  Personalized Signals  Network Distance People Search: Relevance
18. 18 People Search: Query-Independent Signals
19. 19 People Search: Network Rank
20. 20 People Search: Profile Quality
21. 21 People Search: SEO
22. 22 People Search: Query-Dependent Signals
23. 23 People Search: Inferring Structure
24. 24 vs. vs. People Search: Ambiguity
25. 25 for i in [1..n] s  w 1 w 2 … w i if P c (s) > 0 a  new Segment() a.segs  {s} a.prob  P c (s) B[i]  {a} for j in [1..i-1] for b in B[j] s  w j w j+1 … w i if P c (s) > 0 a  new Segment() a.segs  b.segs U {s} a.prob  b.prob * P c (s) B[i]  B[i] U {a} sort B[i] by prob truncate B[i] to size k People Search: HMM + Segmentation
26. 26 People Search: Personalized Signals
27. 27  QCon 2010 presentation by John Wang on “LinkedIn Search: Searching the Social Graph in Real Time” http://www.infoq.com/presentations/LinkedIn-Search  SIGIR 2011 Workshop on Entity-Oriented Search http://research.microsoft.com/en-us/um/beijing/events/eos2011/  HCIR 2011 paper by Jonathan Koren on “Faceted Search Query Log Analysis” (forthcoming) http://hcir.info/hcir-2011/ People Search: Further Reading
28. 28 Job Matching
29. 29  Job Features  Job Description, Location, Similar Jobs, …  Candidate Features  Profile Data, Network, Activity, …  Standardization  Companies, Job Titles, Education, … Job Matching: Overview
30. 30 Corpus Stats Job User Base Filtered title geo company industry description functional area … Candidate General expertise specialties education headline geo experience Current Position title summary tenure length industry functional area … Similarity (candidate expertise, job description) 0.56 Similarity (candidate specialties, job description) 0.2 Transition probability (candidate industry, job industry) 0.43 Title Similarity 0.8 Similarity (headline, title) 0.7 . . . derived Matching Binary Exact matches: geo, industry, … Soft transition probabilities, similarity, … Text Job Matching: Algorithm Transition probabilities Connectivity yrs of experience to reach title education needed for this title …
31. 31  Most people aren't looking for jobs.  Complicates evaluation, training.  Important not to offend users.  e.g., by offering Peter Norvig a postdoc.  You can’t always get what you want  Every employer wants the hottest candidates. Job Matching: Challenges
32. 32  KDD 2011 paper by Bekkerman & Gavish on “High- Precision Phrase-based Document Classification” http://www.stanford.edu/~gavish/documents/phrase_based.pdf  SIGIR 2011 paper by Cetintas et al. on “Identifying Similar People in Professional Social Networks” http://dl.acm.org/citation.cfm?id=2010123  Blog post on LinkedIn’s recommendation engine http://blog.linkedin.com/2011/03/02/linkedin-products-you-may-like/ Job Matching: Further Reading
33. 33 Skills
34. 34 Skills: What are Skills?
35. 35 Skills: Identifying Skills
36. 36 Skills: Cluster and Disambiguate angel
37. 37 Skills: Assigning Skills to People
38. 38 Skills: Who are the Experts?
39. 39 • Relevance • Combine query-independent, query-dependent, and personalized features. • Recommendations • Match people to jobs, groups, news, … • Reputation • Expertise relative to professional skills. Summary: The 3 Rs
40. 40 ¿ Open Problems ?
41. 41 Exploratory Search Fact retrieval Known item search Navigation Transition Verification Question answering Knowledge acquisition Comprehension/Interpretation Comparison Aggregation/Integration Socialize Accretion Analysis Exclusion/Negation Synthesis Evaluation Discovery Planning/Forecasting Transformation Lookup Investigate Learn Exploratory Search
42. 42 Explore / Exploit
43. 43 Incentives for Online Reputation
44. 44 Questions? Contact: dtunkelang@linkedin.com We’re Hiring! http://engineering.linkedin.com/ Thank You!