Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Опыт разработки геопространственной социальной сети

Опыт разработки геопространственной социальной сети

Антон Кавалеров (Siemens, R&D Engineer) @ Moscow Python №41

"Как мы практически с нуля создали полностью функционирующую социальную сеть для мобильных устройств. Какую мы создали инфраструктуру во время разработки, как мы взаимодействовали между собой, тестировали и т.п. Расскажу про архитектуру".

Видео: http://www.moscowpython.ru/meetup/41/opyt-razrabotki-geoprostranstvennoj-socialnoj-seti/

Moscow Python Meetup

December 22, 2016
Tweet

More Decks by Moscow Python Meetup

Other Decks in Programming

Transcript

  1. 2 Idea SPONT helps Bob, Eve and Mike find each

    other before travel Filters on the map Period of time Gender Age Hobbies I wanna meet someone for wandering around this place. I’ll be here from 23 August till 15 September. Bob’s Eve’s Mike’s Hobbies
  2. 3 Server side architecture Ubuntu (Amazon EC2) Queue (Celery, RabbitMQ)

    E-mail sending (Amazon SES, boto3 (http based interaction)) Push notifications (APNS) NoSQL Database (Redis) - sessions - blacklist for messenger - parallelisation web- sockets in Tornado (in process) Proxy & HTTP server (nginx, SSL, static web pages (landing, documentation), load balancer for web-socket (in process)) Asynchronous web server (web-socket, REST-like, Tornado) Documentation (Sphinx, custom templates) WSGI server (load balancer, Gunicorn) Synchronous web server (HTTP, REST, Django) Landing page (static + some requests to sync server) Demonization (supervisord) - Gunicorn - Tornado - Redis - Celery Git (fetching project source from production branch) Testing (Python tests) Testing (SoapUI, Advance Rest Client, Postman) Clients DNS (Amazon Rout53) SQL Database (PostgreSQL, Amazon RDS)
  3. 4 Testing and developing • SoapUI, Advanced Rest Client •

    Auto tests using requests and websocket-client • Beta testig • Bug tracking • Some sort of CI. Deployment: setup.sh • Google analytics, Fabric DEV server PROD server DEV app PROD app DEV DB PROD DB Git
  4. 5 Messenger • WebSockets (Tornado) • NoSQL requests • Push

    notifications (if application is open) • Features: text, photo, location, event, …
  5. 6 Messenger id text datetime_post sender_id thread_id Message id last_message_id

    Thread id datetime_unsubscribe thread_id user_id Thread Participants Bottlenecks: • Real-time checking: black list, threads • Get the last message in a thread (when user update list of conversations). Use DB Triggers • Send message id message_id thread_id user_id is_read Is Read sessions, black list, threads
  6. 7 Geo data Events People • Geospatial SQL requests •

    Dynamic server side clustering (grid based clustering) • Push notifications (when someone you like post an event near you) • Available filters: - Age - Gender - Period of event - Interests
  7. 8 Geo data: Figure of the Earth The Earth is

    not a sphere (even ellipsoid)
  8. 9 Geo data: Figure of the Earth 1. World ocean

    2. Earth ellipsoid 3. Plumb lines 4. True figure of the Earth 5. Geoid
  9. 10 Geo data: Projections Moscow region offset (elliptical vs. spherical

    Mercator) is about 19 km. http://habrahabr.ru/post/239251/ Yandex - elliptical Mercator (default). Google - spherical Mercator (default). Both use WGS84 datum. https://developers.google.com/maps/documentation/javascri pt/maptypes?#WorldCoordinates https://tech.yandex.ru/maps/doc/theory/concepts/coordinate s-docpage/
  10. 12 Geo data: Display problem • Dynamic objects (post, delete,

    filters, ..) • The map always in motion • Scaling
  11. 13 Geo data: Clustering http://qingkaikong.blogspot.ru/2016/08/clustering-with-dbscan.html Weaknesses: DBSCAN does not work

    to well when we are dealing with clusters of varying densities or with high dimensional data http://scikit-learn.org/stable/modules/clustering.html
  12. 16 Geo data: Clustering Client side clustering: Download all events,

    then clustering Server side clustering: Clustering, then download what user sees Initial
  13. 17 Geo data: Quaternary Triangular Mesh "Quaternary Triangular Mesh" or

    QTM that was developed by Geoffrey Dutton in the early 1980s. http://www.spatial-effects.com/SE-papers1.html
  14. 18 To be continued… • Parallel tornado • Access control

    list vs Many-to-Many • End-to-end encryption (tor?) • Android, Web • Auto managing (spam filtration, moderation) • Store libraries in repository (Independence on libraries)
  15. 19 Above all else • Idea (Analyse) • Team (compromise)

    • Pursuit of ambition • Self-examination • Keep calm and carry on [email protected] Anton Kavalerov https://spont.io