Voting - Trends for language, practices - Talk with Speakers - People have similar problems - Networking and social event - Unknown unknowns become known unknowns
really getting popular - Deep Learning, neural networks everywhere! - MicroPython is becoming very popular for gadgets - SocketIO is pretty nice and ready for prod - How to better configure web servers - Fancy new command line apps - Hear war stories - Lightning Talks!
troubleshooting - Asyncio first - use mypy - use pypy - Use poetry in disfavor of pipenv - Try out cython or rust bindings - Play with SocketIO - Play with PyTorch for fun and profit
(Kiwi) ▪ parallel execution of discrete tasks with no blocking ▪ parts of task queues: Task, Queue, Producer, Consumer ▪ used for decoupling large tasks, that may run periodically, get more throughput ▪ in library terms: don't use something lightweight like rq you will end up reimplementing most of the stuff in celery in the long run ▪ celery has lots of included goodies, like a repeater ▪ connexion is really nice for OpenAPIv3 (ex Swagger) and auth ▪ never use the Database for Queueing, use Redis or AMQP ▪ pass simple objects/data to talks (i.e. PKs vs data that may change) ▪ set retry limits before failure ( celery: max_retries) set only for certail errors (autoretry_for(Exception)) use retry_jitter to avoid scheduling at the same exact time; ▪ have a separate queues for slow/fast taks to avoid congestion for demanding tasks ▪ aim for atomicity and idempotency as tasks fail and have retries ▪ downside for celery: you'll share the same codebase between the consumer and producer ▪ the celery documentation is really worth reading, there is much knowledge there
▪ the python standard library and numpy has lots of C/Forrtan backend code ▪ python C API has many downsides that you have to do manual reference counting on objects: Py_INCREF, Py_DECREF ▪ variable bindings have ownership ▪ PyO3 is a rust crate to bind rust with python ▪ PyResult is to interface the rust errors/exception with Python also the return values ▪ FFI (foreign function interfaces): you can expose C functions to various languages like (JavaScript, Ruby, Python) ▪ before diving into rust as a backend always see if Cython is viable option as its closer to Python
of little known command line tools: exa, fd, rq, tokkei, httpie, bat, hyperfine, mdp, genact ▪ CLI is very efficient, provides much productivity with no GUI ▪ exa: a nice combination between ls and tree with colors and metadata ▪ fd: an improved find command, ignores files mentioned in gitignore and has color ▪ rq: a nice colored grep alternative ▪ tokei: much faster, easier to use tool than CLOC ▪ httpie: very nice colored HTTP cli tool prints headers, pretty json for requests ▪ bat: cat but with colors and syntax highliting defaults to default pager ▪ hyperfine: microbenchmarking similar to %timeit in ipython. does multiple runs and shows average runtime with standard deviation ▪ mdp: CLI slide presenter; like powerpoint but with %1 of the features ▪ genact: shows random "work" on your screen, as if you are working. Can be a nice screensaver
str: - Vita Smid ▪ warning mypy is still under active development ▪ use a stricter config than the defaults as they are way too permissive ▪ aim for full coverage, restrict dynamic typing when possible ▪ do not disable warnings, handle them ▪ to begin with try doing a gradual coverage, a nice way to increase coverage is by doing internal hackathons ▪ do not use the skip imports option ▪ as for tests let mypy be more lenient ▪ use number parametrization and algebraic types ▪ read and understand the protocols: nominal vs structural types(traits, interfaces) ▪ use custom types ▪ there are many plug-ins for mypy for edge cases: dataclasses, django models generally for code with meta-classes and magic
Yenny Cheung (Yelp) ▪ benefits: it will be easier for you to do internal presentation for your coleagues ▪ life is much more that coding, presenting is an an important skill as well ▪ if you are an introvert, it helps you network with people; they will come to you ▪ you need to come to term and control your nervousness while presenting; it is natural ▪ find your "harbour" and use a power pose while presenting; humour may help you if you are a natural joker ▪ expect mistakes to happen and have a contingency plan for them; rehearse handling mistakes ▪ time yourself when presenting, you don't need to fill the timeslot, leave time for questions (25m for a 30m slot) ▪ have talk variants for different time lengths (45m 15m 10m) ▪ prepare and rehearse for stage failure: projector failure, missing adaptors, no wifi/internet, don't overly rely on speaker notes ▪ from time to time drink water to help you think ▪ when asked a question don't be afraid to said "I don't know" it's an art: you can rephrase it with a spin ▪ take note of the questions asked during QA, it can help you improve your talk ▪ your audience is supportive generally, nobody wants you to fail: don't be overcome by the imposter syndrome ▪ pick a topic that you are genuinely interested in. Preparing for a talk is hard and you will question why you are doing it ▪ don't be annoyed if your talk is rejected, there are lots of reasons it happens ▪ repeat yourself doing the presentation, re-watch it to figure out if you have verbal tics such as "Ummm" ▪ expectations: your first several talks will suck, get over it and do lots of them to improve quickly ▪ books: Present! , Crucial Conversations, Mindset
core-dev) ▪ references to many people talks: Brendan D. Gregg(articles, books), Victor Stinner(benchmarking), Dmitry Levin(strace), Michal Sekletar(tools for system engineers) ▪ definitions: Tracing is observing and monitoring behavior, Profiling is gathering and analyzing metrics ▪ in benchmarks the most important parameters are: average, median, standard deviation, and percentile ▪ ptrace (~1985): process trace syscall, user-space tracing, used by debuggers and code analysis tools(gdb, strace, ltrace) ▪ ltrace: library call tracer, a bit slow ▪ strace: trace system calls and signals ▪ trace-cmd: low level kernel tracing ▪ perf, ftrace, dtrace: advanced tracing tools; generate call graphs ▪ ext4slower: trace slow ext4 file operations with per-event details ▪ bpftrace: toolkit for creating efficient kernel tracing and manipulation programs. ▪ stap (2005): system tap, kernel module for scripting language for dynamic instrumentation
- Max Bélanger (Dropbox) ▪ they provide client side desktop apps written in Python ▪ most of the work is done via integrations with the host platform: OS APIs ▪ most of the integrations is done via bindings and are exposed via C APIs ▪ ctypes: are a way to solve the problem bit are brittle, consider cFFI insteead or Cython ▪ for C++ bindings: pybind11 and pyobj for MacOS and pywin32 for Windows and COM ▪ for windows 10 use winRT via xlang ▪ there tradeoff for GUIs are between convenience (Qt, 3rd party) and control (Cocoa, native) ▪ always compile and rebuild libraries yourself, never trust binaries ▪ make sure you audit licenses and pin versions and verify hashes ▪ freezers: basically bundler for desktop apps; PyInstaller and Py2App(Mac) ▪ automatic updates: Squirell and Sparkle ▪ make sure you redirect logging to a file and not to standard output (problem on win) ▪ crash reporting: automate it; don't rely on users; CrashPad, PLCrashReporter (mac), Sentry ▪ performance: move CPU bound work outside python ▪ pydevd: debugger used in PyDev, PyCharm and VSCode Python
dev) ▪ many interpreters: CPython (1989), Jython (1997), IronPython(2006), PyPy (2007), uPython (2014) ▪ aproaches to make python faster: fork or scrach implementation ▪ CPython: GIL limitation though it has 30 active core devs ▪ Dropbox switched to GO for perfomance hotspots ▪ PyPy: though pypy is aound 5x faster with a tracing GC, it has a considerable memory footprint and startup-time, with limited support for C extensions ▪ GIL limitations can be overcome by using the multiprocessing module: sharing memory, better pickle (3.8) for large objects ▪ Cython is pretty nice and has a very similar language to python; translates to native c code ▪ speed.benchmark.org: telco bechmark for visualizing the speed improvements in CPython over time/commits ▪ PyHandle: new. correct C API that fixes old issues; old C API won't be deprecated ▪ gilectomy: replace unique global lock with one lock per mutable object; ref counting does not scale with number of threads ▪ future work: implementing a tracing GC in CPython ▪ PEP 554: implement multiple interpreter per stdlib ▪ reference: https://faster-cpython.readthedocs.io/
servers around 10% internet traffic ▪ Bryan Cantrill: production is truly a war ▪ asyncio relies on an event loop ▪ PEP-3156: tulip period around Dev 2012 ▪ history of how async landed in Py 3.6 ▪ watch our for synchronous async code gotchas (is your code truly async) ▪ use asyncio.ensure_future ▪ always await awaitables and check async task status ▪ old packages: momoko async wrapper for psycopg2 DEAD ▪ use async_test and uvloop(caution with packages) ▪ asynch_with statement ▪ async_bomb: limit tasks created and run solved by aiojobs scheduler ▪ rule: you are not a uService if you depend on another uService(especialy a threaded slow one) ▪ use a message queue (AMQP) for a group of uServices ▪ make sure your DB (threaded) has enough workers
▪ average package on PyPI has around 2-3 dependencies ▪ make sure you have reproducible builds ▪ Platypus: analogy of the packaging ecosystem, celebrate the diversity ▪ PEP-518: pyproject.toml replaces setup.py,cfg ▪ steady consistent releases are a sign of health ▪ pipenv: very opinionated and slow with config hell; vendors lots of packages, venv names unpredictable ▪ poetry: much more friendly, customizable, adheres to PEP-518; currently only for pure python packages; has plug-in support ▪ continuous releases: avoid error prone manual steps, use autopublish after request merges via a "release file" ▪ example workflow: https://hypothesis.works/articles/continuous-releases/ and https://github.com/strawberry-graphql/strawberry ▪ AutoPub: enables project maintainers to release new package versions to PyPI by merging pull requests.
Peter Sperl (Bloomberg) ▪ uwsgi is from 2008, a long long time ago, when assumptions about deployment were much different ▪ should be fixed in release v2.1 but as of July 2019 not published ▪ strongly recommendations to look at read this first ▪ enable strict mode: don't continue when unknown config option has been encountered ▪ enable master: master process is necessary to gracefully re-spawn and pre-fork workers ▪ enable vacuum: to remove files, sockets after restart ▪ enable thread: disabled by default and you probably will use threads ▪ enable single-interpreter: uwsgi can run in tandem js, ruby etc ▪ enable die-on-term: will catch the SIGTERM and shutdown, the default ignores it ▪ enable need-app: prevents uWSGI from starting if it is unable to find or load your application module ▪ enable disable-loggign: default logging way too verbose ▪ enable log-4xx and log-5xx ▪ worker-recycling: set options for max-requests 1000, max-worker-lifetime 3600 etc ▪ dynamic worker scalling set to "busyness": always have spare workers available, which is useful when anticipating unexpected traffic surges. ▪ harakiri: kills workers on timeouts ▪ enable py-call-osafterfork: workers can receive signals ▪ process labelling: procname-prefix = "myservice " ▪ caching: uwsgi also exposes caching a-la memcache/redis
van der Geer • live demo for classifying cities based on their skyline in PyTorch • most commonly used Image Model for classification is VGG16, a large neural net, 16 layer with 144 million weights • ImageNet has 14 million images with 1000 annotated classes • of all the ML frameworks out there PyTorch is the most pythonic, flexible, easy to use; slower than others and not recommended for multi GPU and production; internals are accesible • SqueeezeNet is VGG light and is much easier to use • training classifiers is not that hard • keras is more faster • clean your data before using them for training • classification accuracy peaks at just under 90%
backend engineer with machine learning hat • natural language processing is hard; there is ambiguity and data is compositional • has been adopted in automated translation and summary generation • learning word vector for sentiment analysis • bag of words is the simplest representation • many references to ML mumbo jumbo with NLP • progress from simple architectures to complex ones; • go to book: Neural Network Methods in Natural Language Processing
demo for multiplayer chess program with socket.io • servers can invoke client endpoints as well • can invoke one/some/all endpoints • packages: python-socketio, uvicorn, watchgod(file changes)
- Andrey Vlasovskikh (PyCharm) • 3.8 beta is out, try out new fatures • PEP-572: assignment expression (walrus), assignment has value and can be used in comprehensions as well • PEP-570: positional only parameters. This introduces / as a marker to indicate that the arguments to its left are positional only • new types: Protocol (structural typing), Literal, Final, TypedDict • mypyc: compiles mypy-annotated, statically typed Python modules into CPython C extensions. • to be released on 2019-10-21
professor at university teaches python to students at finance masters (full -time 3 months program) • uses ipython notebook to enable quick development • implements DSL for modelling of financial instruments (options) • teaches Strategy pattern for multiple pricing engines: Monte Carlo, Black-Scholes, Binomail • data source is Bloomberg API • implementation of a delta hedging market maker in the on github named Probo
zoliszeredi
mhrtech
tetsuya28
takahashim
gionn
yazm
jozono
glaforge
rfdnxbro
prog893
.png){kind=avatar}
yamad
occhi
sansantech
jnunemaker
codingconduct
afnizarnur
morganepeng
aleyda
soudai
bryan
letsgokoyo
thirion
tamaranovitovic
