Опыт разработки геопространственной социальной сети

Антон Кавалеров (Siemens, R&D Engineer) @ Moscow Python №41

"Как мы практически с нуля создали полностью функционирующую социальную сеть для мобильных устройств. Какую мы создали инфраструктуру во время разработки, как мы взаимодействовали между собой, тестировали и т.п. Расскажу про архитектуру".

Видео: http://www.moscowpython.ru/meetup/41/opyt-razrabotki-geoprostranstvennoj-socialnoj-seti/

Transcript

1. An experience of developing
   geospatial social network
   Anton Kavalerov
   Igor Shmakov
   https://spont.io
   

   View Slide

2. 2
   Idea
   SPONT helps Bob, Eve and Mike
   find each other before travel
   Filters on the map
   Period of time Gender
   Age
   Hobbies
   I wanna meet
   someone for
   wandering
   around this
   place. I’ll be
   here from 23
   August till 15
   September.
   Bob’s
   Eve’s
   Mike’s
   Hobbies
   

   View Slide

3. 3
   Server side architecture
   Ubuntu (Amazon EC2)
   Queue
   (Celery, RabbitMQ)
   E-mail sending
   (Amazon SES, boto3
   (http based
   interaction))
   Push notifications
   (APNS)
   NoSQL Database
   (Redis)
   - sessions
   - blacklist for
   messenger
   - parallelisation web-
   sockets in Tornado
   (in process)
   Proxy & HTTP server
   (nginx, SSL, static web pages (landing,
   documentation), load balancer for
   web-socket (in process))
   Asynchronous web server
   (web-socket, REST-like, Tornado)
   Documentation
   (Sphinx, custom templates)
   WSGI server
   (load balancer, Gunicorn)
   Synchronous web
   server
   (HTTP, REST, Django)
   Landing page
   (static + some requests to sync
   server)
   Demonization
   (supervisord)
   - Gunicorn
   - Tornado
   - Redis
   - Celery
   Git
   (fetching project source from
   production branch)
   Testing
   (Python tests)
   Testing
   (SoapUI, Advance
   Rest Client,
   Postman)
   Clients
   DNS (Amazon Rout53)
   SQL Database
   (PostgreSQL, Amazon RDS)
   

   View Slide

4. 4
   Testing and developing
   • SoapUI, Advanced Rest Client
   • Auto tests using requests and websocket-client
   • Beta testig
   • Bug tracking
   • Some sort of CI. Deployment: setup.sh
   • Google analytics, Fabric
   DEV server PROD server
   DEV app PROD app
   DEV DB PROD DB
   Git
   

   View Slide

5. 5
   Messenger
   • WebSockets (Tornado)
   • NoSQL requests
   • Push notifications (if application is open)
   • Features: text, photo, location, event, …
   

   View Slide

6. 6
   Messenger
   id text datetime_post sender_id thread_id
   Message
   id last_message_id
   Thread
   id datetime_unsubscribe thread_id user_id
   Thread Participants
   Bottlenecks:
   • Real-time checking: black list, threads
   • Get the last message in a thread
   (when user update list of conversations).
   Use DB Triggers
   • Send message
   id message_id thread_id user_id is_read
   Is Read
   sessions, black list, threads
   

   View Slide

7. 7
   Geo data
   Events People
   • Geospatial SQL requests
   • Dynamic server side clustering (grid based clustering)
   • Push notifications (when someone you like post an event near you)
   • Available filters:
   - Age
   - Gender
   - Period of event
   - Interests
   

   View Slide

8. 8
   Geo data: Figure of the Earth
   The Earth is not a sphere
   (even ellipsoid)
   

   View Slide

9. 9
   Geo data: Figure of the Earth
   1. World ocean
   2. Earth ellipsoid
   3. Plumb lines
   4. True figure of the Earth
   5. Geoid
   

   View Slide

10. 10
    Geo data: Projections
    Moscow region offset (elliptical vs. spherical Mercator) is about 19 km.
    http://habrahabr.ru/post/239251/
    Yandex - elliptical Mercator (default).
    Google - spherical Mercator (default).
    Both use WGS84 datum.
    https://developers.google.com/maps/documentation/javascri
    pt/maptypes?#WorldCoordinates
    https://tech.yandex.ru/maps/doc/theory/concepts/coordinate
    s-docpage/
    

    View Slide

11. 11
    Geo data: Projections
    http://www.gis.osu.edu/misc/map-projections/
    https://www.jasondavies.com/maps/transition/
    

    View Slide

12. 12
    Geo data: Display problem
    • Dynamic objects (post, delete, filters, ..)
    • The map always in motion
    • Scaling
    

    View Slide

13. 13
    Geo data: Clustering
    http://qingkaikong.blogspot.ru/2016/08/clustering-with-dbscan.html
    Weaknesses:
    DBSCAN does not work to well when we are dealing with clusters of varying densities or with high dimensional data
    http://scikit-learn.org/stable/modules/clustering.html
    

    View Slide

14. 14
    Geo data: Clustering
    R*-tree Quadtree
    M-tree
    

    View Slide

15. 15
    Geo data: Clustering
    https://msdn.microsoft.com/en-us/library/bb259689.aspx
    Правильная организация клиент-карточного взаимодействия —
    Антон Корзунов
    http://www.slideshare.net/yandex/ss-39316041
    

    View Slide

16. 16
    Geo data: Clustering
    Client side clustering:
    Download all events, then clustering
    Server side clustering:
    Clustering, then download what user sees
    Initial
    

    View Slide

17. 17
    Geo data: Quaternary Triangular Mesh
    "Quaternary Triangular Mesh" or QTM that was
    developed by Geoffrey Dutton in the early 1980s.
    http://www.spatial-effects.com/SE-papers1.html
    

    View Slide

18. 18
    To be continued…
    • Parallel tornado
    • Access control list vs Many-to-Many
    • End-to-end encryption (tor?)
    • Android, Web
    • Auto managing (spam filtration, moderation)
    • Store libraries in repository
    (Independence on libraries)
    

    View Slide

19. 19
    Above all else
    • Idea (Analyse)
    • Team (compromise)
    • Pursuit of ambition
    • Self-examination
    • Keep calm and carry on
    antonkavalerov@spont.io
    Anton Kavalerov
    https://spont.io
    

    View Slide