Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Опыт разработки геопространственной социальной сети

Опыт разработки геопространственной социальной сети

Антон Кавалеров (Siemens, R&D Engineer) @ Moscow Python №41

"Как мы практически с нуля создали полностью функционирующую социальную сеть для мобильных устройств. Какую мы создали инфраструктуру во время разработки, как мы взаимодействовали между собой, тестировали и т.п. Расскажу про архитектуру".

Видео: http://www.moscowpython.ru/meetup/41/opyt-razrabotki-geoprostranstvennoj-socialnoj-seti/

Moscow Python Meetup

December 22, 2016
Tweet

More Decks by Moscow Python Meetup

Other Decks in Programming

Transcript

  1. An experience of developing
    geospatial social network
    Anton Kavalerov
    Igor Shmakov
    https://spont.io

    View full-size slide

  2. 2
    Idea
    SPONT helps Bob, Eve and Mike
    find each other before travel
    Filters on the map
    Period of time Gender
    Age
    Hobbies
    I wanna meet
    someone for
    wandering
    around this
    place. I’ll be
    here from 23
    August till 15
    September.
    Bob’s
    Eve’s
    Mike’s
    Hobbies

    View full-size slide

  3. 3
    Server side architecture
    Ubuntu (Amazon EC2)
    Queue
    (Celery, RabbitMQ)
    E-mail sending
    (Amazon SES, boto3
    (http based
    interaction))
    Push notifications
    (APNS)
    NoSQL Database
    (Redis)
    - sessions
    - blacklist for
    messenger
    - parallelisation web-
    sockets in Tornado
    (in process)
    Proxy & HTTP server
    (nginx, SSL, static web pages (landing,
    documentation), load balancer for
    web-socket (in process))
    Asynchronous web server
    (web-socket, REST-like, Tornado)
    Documentation
    (Sphinx, custom templates)
    WSGI server
    (load balancer, Gunicorn)
    Synchronous web
    server
    (HTTP, REST, Django)
    Landing page
    (static + some requests to sync
    server)
    Demonization
    (supervisord)
    - Gunicorn
    - Tornado
    - Redis
    - Celery
    Git
    (fetching project source from
    production branch)
    Testing
    (Python tests)
    Testing
    (SoapUI, Advance
    Rest Client,
    Postman)
    Clients
    DNS (Amazon Rout53)
    SQL Database
    (PostgreSQL, Amazon RDS)

    View full-size slide

  4. 4
    Testing and developing
    • SoapUI, Advanced Rest Client
    • Auto tests using requests and websocket-client
    • Beta testig
    • Bug tracking
    • Some sort of CI. Deployment: setup.sh
    • Google analytics, Fabric
    DEV server PROD server
    DEV app PROD app
    DEV DB PROD DB
    Git

    View full-size slide

  5. 5
    Messenger
    • WebSockets (Tornado)
    • NoSQL requests
    • Push notifications (if application is open)
    • Features: text, photo, location, event, …

    View full-size slide

  6. 6
    Messenger
    id text datetime_post sender_id thread_id
    Message
    id last_message_id
    Thread
    id datetime_unsubscribe thread_id user_id
    Thread Participants
    Bottlenecks:
    • Real-time checking: black list, threads
    • Get the last message in a thread
    (when user update list of conversations).
    Use DB Triggers
    • Send message
    id message_id thread_id user_id is_read
    Is Read
    sessions, black list, threads

    View full-size slide

  7. 7
    Geo data
    Events People
    • Geospatial SQL requests
    • Dynamic server side clustering (grid based clustering)
    • Push notifications (when someone you like post an event near you)
    • Available filters:
    - Age
    - Gender
    - Period of event
    - Interests

    View full-size slide

  8. 8
    Geo data: Figure of the Earth
    The Earth is not a sphere
    (even ellipsoid)

    View full-size slide

  9. 9
    Geo data: Figure of the Earth
    1. World ocean
    2. Earth ellipsoid
    3. Plumb lines
    4. True figure of the Earth
    5. Geoid

    View full-size slide

  10. 10
    Geo data: Projections
    Moscow region offset (elliptical vs. spherical Mercator) is about 19 km.
    http://habrahabr.ru/post/239251/
    Yandex - elliptical Mercator (default).
    Google - spherical Mercator (default).
    Both use WGS84 datum.
    https://developers.google.com/maps/documentation/javascri
    pt/maptypes?#WorldCoordinates
    https://tech.yandex.ru/maps/doc/theory/concepts/coordinate
    s-docpage/

    View full-size slide

  11. 11
    Geo data: Projections
    http://www.gis.osu.edu/misc/map-projections/
    https://www.jasondavies.com/maps/transition/

    View full-size slide

  12. 12
    Geo data: Display problem
    • Dynamic objects (post, delete, filters, ..)
    • The map always in motion
    • Scaling

    View full-size slide

  13. 13
    Geo data: Clustering
    http://qingkaikong.blogspot.ru/2016/08/clustering-with-dbscan.html
    Weaknesses:
    DBSCAN does not work to well when we are dealing with clusters of varying densities or with high dimensional data
    http://scikit-learn.org/stable/modules/clustering.html

    View full-size slide

  14. 14
    Geo data: Clustering
    R*-tree Quadtree
    M-tree

    View full-size slide

  15. 15
    Geo data: Clustering
    https://msdn.microsoft.com/en-us/library/bb259689.aspx
    Правильная организация клиент-карточного взаимодействия —
    Антон Корзунов
    http://www.slideshare.net/yandex/ss-39316041

    View full-size slide

  16. 16
    Geo data: Clustering
    Client side clustering:
    Download all events, then clustering
    Server side clustering:
    Clustering, then download what user sees
    Initial

    View full-size slide

  17. 17
    Geo data: Quaternary Triangular Mesh
    "Quaternary Triangular Mesh" or QTM that was
    developed by Geoffrey Dutton in the early 1980s.
    http://www.spatial-effects.com/SE-papers1.html

    View full-size slide

  18. 18
    To be continued…
    • Parallel tornado
    • Access control list vs Many-to-Many
    • End-to-end encryption (tor?)
    • Android, Web
    • Auto managing (spam filtration, moderation)
    • Store libraries in repository
    (Independence on libraries)

    View full-size slide

  19. 19
    Above all else
    • Idea (Analyse)
    • Team (compromise)
    • Pursuit of ambition
    • Self-examination
    • Keep calm and carry on
    [email protected]
    Anton Kavalerov
    https://spont.io

    View full-size slide