Engineering the New LinkedIn Profile

Overview of the new (2012) frontend architecture used for the Profile at LinkedIn.

Blog version of this slidedeck: https://engineering.linkedin.com/profile/engineering-new-linkedin-profile

1. Engineering the New Profile Josh Clemm Profile Tech Lead March 2013 | LinkedIn David Fleming Senior Product Manager at Zoomjax San Francisco Bay Area | Software Previous Education Golden Phase, FixDex Silicon Valley Business Academy
2. Let's compare...
3. Really a brand new product Refreshed Look & Feel Simplified Design Surfaced New & Interactive Modules New Data Insights
4. New Features Improved in-line editing experience Modules with In-line searching & pagination
5. The New Profile - Goals • Represent your entire professional identity ◦ Not just your resume ◦ Activity, Groups, Following, Connections, Insights about you and your network
6. The New Profile - Goals • Represent your entire professional identity ◦ Not just your resume ◦ Activity, Groups, Following, Connections, Insights about you and your network • Needs to be more interactive ◦ Keep users engaged on the page ◦ Inline pagination, editing, searching
7. The New Profile - Goals • Represent your entire professional identity ◦ Not just your resume ◦ Activity, Groups, Following, Connections, Insights about you and your network • Needs to be more interactive ◦ Keep users engaged on the page ◦ Inline pagination, editing, searching • Needs to be fluid, flexible, fast ◦ Progressive rendering ◦ Maintain high performance
8. So how did we achieve all that?
9. Using a lot of Technologies
10. And in particular...
11. Let's look at our High Level Frontend Architecture
12. Groups Content Service Connections Content Service Profile Content Service Client/Browser CDN Load Balancer SCDS Fizzy Profile Web App Profile Content Service Profile Web App Connections Content Service Groups Content Service Dust/JS/CSS Server Retrieve Data Models from Mid-tier /profile/view?id=32 top_card.tl, background.tl /profile/topcard /profile/background
13. Groups Content Service Connections Content Service Profile Content Service Client/Browser CDN Load Balancer SCDS Fizzy Profile Web App Profile Content Service Profile Web App Connections Content Service Groups Content Service Dust/JS/CSS Server Retrieve Data Models /profile/view?id=32 top_card.tl, background.tl /profile/topcard /profile/background Our new architecture uses new tech at all layers
14. Groups Content Service Connections Content Service Profile Content Service Client/Browser CDN Load Balancer SCDS Fizzy Profile Web App Profile Content Service Profile Web App Connections Content Service Groups Content Service Dust/JS/CSS Server Retrieve Data Models /profile/view?id=32 top_card.tl, background.tl /profile/topcard /profile/background Let's start at the bottom with Mappers
15. Mappers - JSON endpoints • Convert data models from mid-tier services into JSON • Each have an unique endpoint URL ( /profile/positions?id=42 ) Positions Mapper JSON "positions": [{ "position": {"id:{}"}, ]} Biz Profile Model Profile Model Rich Media Rest Model References Model Profile Flex Model
16. Mappers - an example public class PictureMapper extends ProfileParametersAwareMapper { private PictureContentMap pictureContentMap; private static final int ZOOMABLE_DIMENSION = 225; @Override public void doService() { PictureContentModel picCM = getContent(PictureContentModel.class); //declare content needs picCM.criteria().setId(getVieweeId()); //supply any input params assemble(); //invoke framework to retrieve declared content if (isResolvedWithoutErrors(picCM)) { //all went well, create new content map to hold output (JavaBean-like objects) pictureContentMap = ContentMap.proxyNew(PictureContentMap.class); Integer pictureWidth = picCM.getPictureWidth(); pictureContentMap.setIsZoomable(pictureWidth >= ZOOMABLE_DIMENSION); if(pictureWidth != null && pictureWidth > 0) { pictureContentMap.setWidth(pictureWidth); pictureContentMap.setHeight(picCM.getPictureHeight()); } pictureContentMap.setPictureID(picCM.getPictureID()); } } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29
17. Mappers - an example public class PictureMapper extends ProfileParametersAwareMapper { private PictureContentMap pictureContentMap; private static final int ZOOMABLE_DIMENSION = 225; @Override public void doService() { PictureContentModel picCM = getContent(PictureContentModel.class); //declare content needs picCM.criteria().setId(getVieweeId()); //supply any input params assemble(); //invoke framework to retrieve declared content if (isResolvedWithoutErrors(picCM)) { //all went well, create new content map to hold output (JavaBean-like objects) pictureContentMap = ContentMap.proxyNew(PictureContentMap.class); Integer pictureWidth = picCM.getPictureWidth(); pictureContentMap.setIsZoomable(pictureWidth >= ZOOMABLE_DIMENSION); if(pictureWidth != null && pictureWidth > 0) { pictureContentMap.setWidth(pictureWidth); pictureContentMap.setHeight(picCM.getPictureHeight()); } pictureContentMap.setPictureID(picCM.getPictureID()); } } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 Declare the data you need Set the data you want coming back as JSON
18. Mappers - features • Modular ◦ A single mapper can retrieve data for a section ▪ Positions, Educations, Groups, etc. • Reusable & Combinable ◦ Mappers can be used more than once ▪ Positions Mapper is used for Positions section and the positions part of Top Card ◦ You can Aggregate Mappers under a common root element and a new URL endpoint
19. Profile's Many Mappers • Each section on Profile has either a single Mapper or Aggregated Mapper (like Top Card) for its data
20. Profile's Many Mappers • Each section on Profile has either a single Mapper or Aggregated Mapper (like Top Card) for its data Profile Web App Summary Mapper Positions Mapper Educations Mapper Picture Mapper Top Card Mapper "Summary": { "summary": "I'm an experienced..."} JSON JSON "TopCard": { "positions": {}, "educations":{}, "picture":{}} Groups Mapper Connections Mapper Patents Mapper "Connections": { "profiles": [ ...]} JSON
21. So we have these Mappers that return JSON for each section. Who calls each one?
22. Groups Content Service Connections Content Service Profile Content Service Client/Browser CDN Load Balancer SCDS Fizzy Profile Web App Profile Content Service Profile Web App Connections Content Service Groups Content Service Dust/JS/CSS Server Retrieve Data Models /profile/view?id=32 top_card.tl, background.tl /profile/topcard /profile/background Fizzy - the UI aggregator
23. Fizzy • Fizzy is an UI aggregator in 2 parts: ◦ Fizzy Server fetches the content you want ◦ Fizzy Client renders it when ready • Your base page defines its structure and which UI components it needs (called "embeds") *Fizzy Server is an Apache Traffic Server Plugin, Fizzy Client is a JS library
24. Profile's Embeds Top Card Embed Activity Embed Embed Embed Embed Embed Yet another embed
25. Profile's Embeds in code <html> <body> ... <div id="wrapper"> <div id=“profile"> <script type=“embed" fs-id=“topcard" fs-uri=“/profile/topcard"/> <script type=“embed" fs-id=“background" fs-uri=“/profile/background"/> ... <script type=“embed" fs-id=“connections" fs-uri=“/profile/connections"/> </div> <div id=“insights"> <script type=“embed" fs-id=“people_you_may_know" fs-uri=“/profile/pymk"/> <script type=“embed" fs-id=“strength_meter" fs-uri=“/profile/strength"/> ... <script type=“embed" fs-id=“in_common" fs-uri=“/profile/incommon"/> </div> </div> </body> </html> 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
26. Profile's Embeds in code <html> <body> ... <div id=“wrapper"> <div id=“profile"> <script type=“embed" fs-id=“topcard" fs-uri=“/profile/topcard"/> <script type=“embed" fs-id=“background" fs-uri=“/profile/background"/> ... <script type=“embed" fs-id=“connections" fs-uri=“/profile/connections"/> </div> <div id=“insights"> <script type=“embed" fs-id=“people_you_may_know" fs-uri=“/profile/pymk"/> <script type=“embed" fs-id=“strength_meter" fs-uri=“/profile/strength"/> ... <script type=“embed" fs-id=“in_common" fs-uri=“/profile/incommon"/> </div> </div> </body> </html> 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 Each embed specifies a Mapper endpoint that Fizzy will fetch
27. Fizzy fetches data and sends it to the browser. What's rendering the actual markup?
28. Groups Content Service Connections Content Service Profile Content Service Client/Browser CDN Load Balancer SCDS Fizzy Profile Web App Profile Content Service Profile Web App Connections Content Service Groups Content Service Dust/JS/CSS Server Retrieve Data Models /profile/view?id=32 top_card.tl, background.tl /profile/topcard /profile/background You guessed it... Dust client templates
29. {Dust} client templates • LinkedIn's latest and greatest rendering layer • Logic-less client template language • Profile page made up of many* templates • Templates + JSON = full markup! *over 400 actually
30. {Dust} - what it looks like <div id="top_card"> <h4>{name}</h4> <h5>{headline}</h5> {#info} <h6>{location} | {industry}</h6> {/info} </div> { "name": "Frank Stallone", "headline": "Actor and Less Famous Brother", "info":{ "location": "Hollywood", "industry": "Entertainment" } }
31. {Dust} - why it's cool for profile • Cached markup in CDNs and browsers ◦ Members browse many profiles in a row • Very DRY and reusable ◦ Improved development speed ◦ Templates within templates within templates... ▪ date range, degree badge, formatted summary field • Super easy to refresh a section on the page ◦ Re-render and replace ◦ Useful in pagination, inline searching, and inline editing
32. {Dust} on profile <hgroup> {>"tl/apps/profile/v2/embed/company_logo"/} <h4> <a href="{link_title_pivot}" name='title'>{title_highlight|s}</a> {?selfView} <span class="edit-tools"> <a class="edit-section">{i18n_Edit}</a> </span> {/selfView} </h4> <h5> {?companyName} {>"tl/apps/profile/v2/embed/company_link" track_param="prof-exp"/} {/companyName} </h5> </hgroup> <span class="experience-date-locale"> {>"tl/apps/profile/v2/partial/daterange"/} {! Location !} {@pre.fmt key="fmt_location" type="geo.region" value="{location}" render="false"/} {?fmt_location}<span class="locality">{fmt_location}</span> {:else} {?locationName} <span class="locality">{locationName}</span> {/locationName} {/fmt_location} </span> {>"tl/apps/profile/v2/partial/summary_field" _summary=summary/} {>"tl/apps/profile/v2/partial/associated_content" trkCodePrefix="exp"/}
33. {Dust} on profile <hgroup> {>"tl/apps/profile/v2/embed/company_logo"/} <h4> <a href="{link_title_pivot}" name='title'>{title_highlight|s}</a> {?selfView} <span class="edit-tools"> <a class="edit-section">{i18n_Edit}</a> </span> {/selfView} </h4> <h5> {?companyName} {>"tl/apps/profile/v2/embed/company_link" track_param="prof-exp"/} {/companyName} </h5> </hgroup> <span class="experience-date-locale"> {>"tl/apps/profile/v2/partial/daterange"/} {! Location !} {@pre.fmt key="fmt_location" type="geo.region" value="{location}" render="false"/} {?fmt_location}<span class="locality">{fmt_location}</span> {:else} {?locationName} <span class="locality">{locationName}</span> {/locationName} {/fmt_location} </span> {>"tl/apps/profile/v2/partial/summary_field" _summary=summary/} {>"tl/apps/profile/v2/partial/associated_content" trkCodePrefix="exp"/} Partial templates like this are used on almost all the background sections
34. We have our Mappers, Fizzy, and Dust templates... Let's bring it all together.
35. Feature: Inline Editing • Dust + Mappers make this easy • Dust templates for view and edit • Mappers return just the data we need to refresh "View" template "Edit" template Summary Section
36. Feature: Inline Editing • Dust + Mappers make this easy • Dust templates for view and edit • Mappers return just the data we need to refresh "View" template "Edit" template Summary Section When edit link is clicked, render "edit" template
37. Feature: Inline Editing • Dust + Mappers make this easy • Dust templates for view and edit • Mappers return just the data we need to refresh "View" template "Edit" template Summary Section On submit, our endpoint sends back JSON, and we re-render
38. Inline editing - Example Either the "view" or "edit" template is shown at a time
39. Inline editing sounds easy... • Issue #1: Profile data is highly coupled with different sections. ◦ Adding/editing a position needs to be reflected on Top Card... ◦ Deleting a project needs to also be removed from any associated position... • Solution: since we have mappers for each section, we know which to re-fetch upon a save and refresh the respective templates
40. What about pagination, search? • Same idea but easier (not coupled with other sections) • Mappers can take URL offset params Switch out a partial template containing just list of profiles
41. Let's talk performance
42. High Performance Page • Profile is the most trafficked page on LinkedIn • Profile fetches 48+ different types of content on each page load ◦ This content results in 250+ total downstream calls
43. High Performance Page • Profile is the most trafficked page on LinkedIn • Profile fetches 48+ different types of content on each page load ◦ This content results in 250+ total downstream calls Bottom Line: We need to be fast, but need to consider downstream fanout
44. High Performance - Parallel Requests Fizzy Profile App Profile App Profile App Profile App Profile App Profile App Profile App Profile App Profile App Profile App In the beginning, Profile had 15 embeds with 15 different endpoints which is great for speed...
45. High Performance - Parallel Requests Fizzy Profile App Profile App Profile App Profile App Profile App Profile App Profile App Profile App Profile App Profile App Content Service Content Service Content Service Content Service Content Service Content Service Content Service Content Service Content Service Content Service That's a lot of downstream calls, often requesting the same data too. Content Service Content Service
46. High Performance - Parallel Requests Fizzy Profile App Profile App Profile App Profile App Profile App Profile App Profile App Profile App Profile App Profile App Content Service Content Service Content Service Content Service Content Service Content Service Content Service Content Service Content Service Content Service Content Service Content Service and those calls call more and more... until...
47. High Performance - Parallel Requests Fizzy Profile App Profile App Profile App Profile App Profile App Profile App Profile App Profile App Profile App Profile App Content Service Content Service Content Service Content Service Content Service Content Service Content Service Content Service Content Service Content Service Content Service Content Service aaaaand the site's down
48. High Performance - Parallel Requests • Tradeoff: Speed vs Scalability • 15 parallel calls will be fast but with Profile's load will take down site
49. High Performance - Parallel Requests • Tradeoff: Speed vs Scalability • 15 parallel calls will be fast but with Profile's load will take down site Batched Endpoints to the rescue
50. Batching Calls to Mappers Originally, we had separate endpoints (URIs) for each embed 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 <html> <body> ... <div id=“wrapper"> <div id=“profile"> <script type=“embed" fs-id=“topcard" fs-uri=“/profile/topcard"/> <script type=“embed" fs-id=“background" fs-uri=“/profile/background"/> ... <script type=“embed" fs-id=“connections" fs-uri=“/profile/connections"/> </div> <div id=“insights"> <script type=“embed" fs-id=“peeps_you_may_know" fs-uri=“/profile/pymk"/> <script type=“embed" fs-id=“strength_meter" fs-uri=“/profile/strength"/> ... <script type=“embed" fs-id=“in_common" fs-uri=“/profile/incommon"/> </div> </div> </body> </html>
51. Batching Calls to Mappers We now tell framework to batch these two endpoints (Fizzy knows how to deliver the right data to the right embed) <html> <body> ... <div id=“wrapper"> <div id=“profile"> <script type=“embed" fs-id=“topcard" fs-uri=“/profile/mappers?a=topcard,background"/> <script type=“embed" fs-id=“background" fs-uri=“/profile/mappers?a=topcard,background"/> ... <script type=“embed" fs-id=“connections" fs-uri=“/profile/connections"/> </div> <div id=“insights"> <script type=“embed" fs-id=“peeps_you_may_know" fs-uri=“/profile/pymk"/> <script type=“embed" fs-id=“strength_meter" fs-uri=“/profile/strength"/> ... <script type=“embed" fs-id=“in_common" fs-uri=“/profile/incommon"/> </div> </div> </body> </html> 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
52. Batching Calls in Profile • Today, we use between 3-5 parallel requests to Profile • It's a good balance of speed vs. scalability • Batching requests that need the same data has the added benefit of less downstream calls
53. Progressive Rendering With parallel fetch, Profile modules render when ready for improved perceived performance
54. Progressive Rendering With parallel fetch, Profile modules render when ready for improved perceived performance
55. How else to improve page load times?
56. Profiles can be long... Embed Embed Embed Embed Embed Embed Embed Profile Page
57. Optimize for Above the Fold Embed Embed Embed Embed Embed Embed Embed Profile Page The fold No need to render No need to render or fetch
58. Optimize for Above the Fold • The New Profile renders above the fold as fast as we can ◦ Deferring everything at the bottom ◦ Reduces the static content we need to initially download
59. Optimize for Above the Fold • The New Profile renders above the fold as fast as we can ◦ Deferring everything at the bottom ◦ Reduces the static content we need to initially download Sorry guys, we’ll get to you later
60. Optimize for Above the Fold • The New Profile defers fetching the modules at the bottom of the page ◦ Improves server side assembly times ◦ Lowers payload and improves network time ◦ Improves client rendering times
61. Optimize for Above the Fold • The New Profile defers fetching the modules at the bottom of the page ◦ Improves server side assembly times ◦ Lowers payload and improves network time ◦ Improves client rendering times Go fetch!
62. So we covered the product, technologies, and some features... let's revisit our goals.
63. Revisiting the New Profile Goals • Needs to surface your entire professional identity ◦ Easily expose new data endpoints • Needs to be more interactive ◦ Inline editing, searching • Needs to be ◦ Fluid ◦ Flexible ◦ Fast
64. Revisiting the New Profile Goals • Needs to surface your entire professional identity ◦ Easily expose new data endpoints (Mappers) • Needs to be more interactive ◦ Inline editing, searching (Dust + Mappers) • Needs to be ◦ Fluid (Fizzy progressive rendering) ◦ Flexible (Fizzy deferred rendering) ◦ Fast (Fizzy parallel calls + defer fetch, batching Mappers, Dust template caching)
65. Takeaways • Mappers for each module makes sense ◦ You can combine and reuse with ease ◦ A must if refreshing part of page • When structuring your page, start with many embeds ◦ You can control number of requests, when embeds are rendered, and when to fetch data • Many partial templates are a good thing ◦ Allows you finer control over what to re-render ◦ Improves developer speed
66. Takeaways cont. • Make these technologies work for you ◦ Our Mappers are structured to guarantee minimal downstream calls ◦ Take advantage of decoupling rendering layer with server side endpoints ▪ Engineers can build endpoints ▪ Web devs can start the templates with mocked JSON data
67. Thanks! Josh Clemm linkedin.com/in/joshclemm