How to separate frontend from a highload python project with no problems - PyCon Ukraine 2018

Everybody knows that it is hard to scale old highload monolithic projects that use pythonic templates for frontend. I am gonna tell how we transformed our product using trending and proper technologies like GraphQL, Apollo, Node.js with limited developer resources in a short period of time.

And how migration to SSR and GraphGL improved bounce rate and other business metrics.

Transcript

1. How to separate frontend from a highload python project with no problems Oleksandr Tarasenko EVO.company / prom.ua
2. Some prom.ua numbers • RPS 2000-3000 • users over 2.5 million per day • paid sites ~ 50 000 • total sites over 300 000 • products ~ 100 million • pages ~ 300 million 2
3. What we had • monolithic WSGI app (10 years) • mako templates • monolithic database • monolithic Nginx • poor REST API • slow Delivery and Deployment • new features were hard to develop 3
4. 4 1. Resolve url 2. Collect base context 3. Call render_mako 4. Collect template context 5. Return html
5. Our goals • SEO • Independent frontend • Good user experience • Documentation • Data validation • Microservices way • Independent CI/CD 5
6. Tools and expertise • ReactJS / Redux • GraphQL server (hiku) • Container system (vagga/lithos) • Metrics (grafana) • Logs (kibana) 6
7. GraphQL + React + Apollo + Node.js = success story 7
8. 8 GraphQL
9. http://graphql.org/users/ 9
10. Why we choose GraphQL • good for readonly requests and data • mutations to REST • every site has unique user settings • different templates 10
11. Some prom.ua sites code facts • a lot of views and url endpoints • almost no documentation • poor versioning • a lot of template context • duplicating queries 11
12. We built two-level graph • low-level graph for database mapping (data loading) • high-level graph for business logic • auto documentation with graphiql tool • data validation 12
13. product_query = sa.FieldsQuery('db.session', Product.__table__) low_level_graph = Graph([ Node('Product', [ Field('id', Integer, product_query), Field('name', String, product_query), Field('price', None, product_query), ]) 13
14. """ SELECT id, name, price FROM product WHERE id IN (:id1, :id2, ..., :idN) """ product(ids: [1234, 1235]) { id name price } 14
15. low_level_graph = Graph([ Node('Product', [ Field('id', Integer, product_query), ... Link( 'discount', Optional[TypeRef['Discount']], product_to_discount_query, requires='id', ), ]), Node('Discount', [ Field('id', Integer, discount_query), Field('amount', Integer, discount_query), ]) ]) 15
16. 16 discount_query = sa.FieldsQuery('db.session', Discount.__table__) """ SELECT id, name, price, price_currency_id FROM product WHERE id IN (:id1, :id2, ..., :idN) """ product_to_discount_query = sa.LinkQuery( 'db.session', from_column=Discount.product_id, to_column=Discount.product_id, ) """ SELECT product_id FROM discount WHERE product_id IN (:id1, :id2, ..., :idN) """
17. 17 product_sg = SubGraph(low_level_graph, 'Product') high_level_graph = Graph([ Node('Product', [ Field('id', String, product_sg), Field('name', String, product_sg.compile(S.this.name)), Field('priceText', String, product_sg.compile( get_price_text(S.this) )), Field('hasDiscount', Boolean, product_sg.compile( is_discount_available(S.this, S.this.discount) )), Field('discountedPriceText', String, product_sg.compile( get_discounted_price_text(S.this, S.this.discount) )), ]), ])
18. 18 @define(Record[{ 'price': Float, 'price_currency_id': Integer, 'currency_settings': Any }]) def get_price_text(product): product_price_text = format_currency_data( product_price, product['price_currency_id'], ) return product_price_text.formatted_number
19. Some facts of the new graph • number of fields: ~ 500 • number of links: ~ 125 • number of nodes: ~ 75 • single entry point /graphql • refactoring and new vision for code • better monitoring • easy to test API with graphiql 19 Hiku
20. 20 Frontend
21. Stateless ReactJS • React Apollo • React Router • React Helmet • No Redux/Flux/State management lib 21
22. Apollo client benefits • great GraphQL implementation • no sugar in React components • query batching • state cache • community • combine | join graphs (Apollo server) 22
23. 23 export const Article = ({ data }) => { if (data.loading) return <Spinner />; if (!data.article.id) return <NotFound />; return ( <BasePage> <ArticleViewMeta id={data.article.id} /> <BaseViewComponent {...data.article} /> <Footer /> </BasePage> ); };
24. 24 const articleViewQuery = gql` query getArticleViewData($id: String) { article(id: $id){ id title text } }`; export const ArticleWithData = graphql(articleViewQuery, { options: ({ id }) => { return { variables: { id } }; }, })(Article);
25. Other javascript implementations • Relay modern / Relay classic • Apollo client / Apollo server / Apollo engine • Lokka • Prisma • GraphQL.js • express-graphql 25
26. 26 Node.js
27. Node.js implementation • Only for the first request (SSR) • Good for React prerender • Routing • Two-step implementation 27
31. 31 Nginx
34. Success product metrics 34 from 4.8 to 7.1
35. Success product metrics 35 from 57.5% to 49%
36. Some numbers of the new scheme • node.js workers 8 • React render 5 ms • prom.ua workers over 750 • number of requests split to node/python 36
37. 37 Problems
38. 38 Node.js leaks
39. 39 Node.js leaks :)
40. 40 Node.js leaks ;(
41. Sometimes it’s hard to analyze js errors from graph and Apollo 41
42. 42 Hard to split routing in monolithic app
43. 43 Some resources and logic need duplication
44. Plans • GraphQL everywhere • Backend as API • Progressive web apps • More frontends 44
45. 45 GraphQL REST API Questions/Answers