Python Developer's Daily Routine

Slide 1

Slide 1 text

Python Developer's Daily Routine Maxim Avanov (github.com/avanov)

Slide 2

Slide 2 text

Project structure

Slide 3

Slide 3 text

Project structure . / c h e b i t | - - c h e b i t | ` - - _ _ i n i t _ _ . p y ` - - s e t u p . p y 3

Slide 4

Slide 4 text

setup.py f r o m s e t u p t o o l s i m p o r t s e t u p , f i n d _ p a c k a g e s s e t u p ( n a m e = ' C h e b i t ' , v e r s i o n = ' 0 . 0 . 1 ' , p a c k a g e s = f i n d _ p a c k a g e s ( ) , t e s t _ s u i t e = ' t e s t s ' , t e s t s _ r e q u i r e = [ ' p y t e s t ' , ' c o v e r a g e ' ] , i n s t a l l _ r e q u i r e s = [ ' P y r a m i d = = 1 . 5 a 2 ' ] ) 4

Slide 5

Slide 5 text

Bootstrapping $ python setup.py install $ python setup.py develop $ python setup.py test $ python setup.py regtister sdist [bdist_egg] upload 5

Slide 6

Slide 6 text

Package management

Slide 7

Slide 7 text

pip commands $ p i p i n s t a l l P y r a m i d $ p i p i n s t a l l P y r a m i d - - u p g r a d e $ p i p i n s t a l l P y r a m i d = = 1 . 5 a 2 - - u p g r a d e $ p i p u n i n s t a l l P y r a m i d 7

Slide 8

Slide 8 text

pip commands $ p i p f r e e z e B a b e l = = 1 . 3 C h e b i t = = 0 . 0 . 1 C o f f e e S c r i p t = = 1 . 0 . 8 J i n j a 2 = = 2 . 7 . 1 M a k o = = 0 . 9 . 0 . . . 8

Slide 9

Slide 9 text

pip-tools $ p i p - r e v i e w r e q u e s t s = = 0 . 1 3 . 4 a v a i l a b l e ( y o u h a v e 0 . 1 3 . 2 ) $ p i p - r e v i e w - - a u t o . . . < p i p i n s t a l l o u t p u t > $ p i p - r e v i e w - - i n t e r a c t i v e r e q u e s t s = = 0 . 1 4 . 0 a v a i l a b l e ( y o u h a v e 0 . 1 3 . 2 ) U p g r a d e n o w ? [ Y ] e s , [ N ] o , [ A ] l l , [ Q ] u i t y . . . 9

Slide 10

Slide 10 text

Cookiecutter • Generates initial project files from defined templates; • File names are also templates; • Transparently works with remote VCS. 10

Slide 11

Slide 11 text

Cookiecutter template . / c h e b i t - t e m p l a t e | - - c o o k i e c u t t e r . j s o n ` - - { { c o o k i e c u t t e r . p r o j e c t _ n a m e } } | - - { { c o o k i e c u t t e r . p r o j e c t _ n a m e } } | ` - - _ _ i n i t _ _ . p y ` - - s e t u p . p y 11

Slide 12

Slide 12 text

cookiecutter.json { " p r o j e c t _ n a m e " : " m y p r o j e c t " , " p r o j e c t _ v e r s i o n " : " 0 . 0 . 1 " } 12

Slide 13

Slide 13 text

Cookiecutter in action $ c o o k i e c u t t e r { p r o j e c t _ t e m p l a t e _ p a t h } p r o j e c t _ n a m e ( d e f a u l t i s " m y p r o j e c t " ) ? m y n e w p r o j e c t _ v e r s i o n ( d e f a u l t i s " 0 . 0 . 1 " ) ? 0 . 1 . 1 $ t r e e . / m y n e w . / m y n e w | - - m y n e w | ` - - _ _ i n i t _ _ . p y ` - - s e t u p . p y 13

Slide 14

Slide 14 text

We'd like to isolate project environments

Slide 15

Slide 15 text

virtualenv $ v i r t u a l e n v ~ / v e n v / { p r o j e c t _ n a m e } $ s o u r c e ~ / v e n v / { p r o j e c t _ n a m e } / b i n / a c t i v a t e What if we need another Python version? $ v i r t u a l e n v ~ / v e n v / { p r o j e c t _ n a m e } - p p y t h o n 3 15

Slide 16

Slide 16 text

Still not happy enough!

Slide 17

Slide 17 text

Introducing pyenv

Slide 18

Slide 18 text

pyenv - 5 steps to happiness $ g i t c l o n e g i t : / / g i t h u b . c o m / y y u u / p y e n v . g i t ~ / p y e n v $ e c h o ' e x p o r t P Y E N V _ R O O T = " $ H O M E / p y e n v " ' > > ~ / . b a s h r c $ e c h o ' e x p o r t P A T H = " $ P Y E N V _ R O O T / b i n : $ P A T H " ' > > ~ / . b a s h r c $ e c h o ' e v a l " $ ( p y e n v i n i t - ) " ' > > ~ / . b a s h r c $ e x e c $ S H E L L 18

Slide 19

Slide 19 text

pyenv - Rise and Shine! $ p y e n v i n s t a l l - l $ p y e n v i n s t a l l 2 . 7 . 6 $ p y e n v i n s t a l l 3 . 3 . 3 $ p y e n v i n s t a l l p y p y - 2 . 2 . 1 $ p y e n v r e h a s h 19

Slide 20

Slide 20 text

The Happiness $ pyenv versions $ pyenv version $ pyenv local 2.7.6 3.3.3 $ pyenv global 3.3.3 $ pyenv shell pypy-2.2.1 $ pyenv whence 2to3 20

Slide 21

Slide 21 text

... $ w h i c h p y t h o n / h o m e / g h o s t w r i t e r / p y e n v / s h i m s / p y t h o n $ p y t h o n P y t h o n 2 . 7 . 3 ( 8 7 a a 9 d e 1 0 f 9 c , N o v 2 4 2 0 1 3 , 1 8 : 4 8 : 1 3 ) [ P y P y 2 . 2 . 1 w i t h G C C 4 . 6 . 3 ] o n l i n u x 2 > > > > 21

Slide 22

Slide 22 text

virtualenv meets pyenv

Slide 23

Slide 23 text

pyenv-virtualenv $ g i t c l o n e g i t : / / g i t h u b . c o m / y y u u / p y e n v - v i r t u a l e n v . g i t ~ / p y e n v / p l u g i n s / p y e n v - v i r t u a l e n v $ p y e n v v i r t u a l e n v p y p y - 2 . 2 . 1 ~ / v e n v / { p r o j e c t _ n a m e } 23

Slide 24

Slide 24 text

pyenv-virtualenv in action $ p y e n v v i r t u a l e n v s { p r o j e c t _ n a m e } ( c r e a t e d f r o m / h o m e / g h o s t w r i t e r / p y e n v / v e r s i o n s / p y p y - 2 . 2 . 1 ) $ s o u r c e ~ / p y e n v / v e r s i o n s / { p r o j e c t _ n a m e } / b i n / a c t i v a t e 24

Slide 25

Slide 25 text

Well done!

Slide 26

Slide 26 text

Now, what about other components?

Slide 27

Slide 27 text

Virtualization

Slide 28

Slide 28 text

Vagrant • Written in Ruby • Relies on VirtualBox • Runs on Linux / OS X / Win 28

Slide 29

Slide 29 text

Juju • Written in Python / Go • Relies on LXC • Linux only 29

Slide 30

Slide 30 text

Docker • Written in Go • Relies on LXC and AUFS • Linux only 30

Slide 31

Slide 31 text

Vagrant's Success Story... 1. Put a Vagrantfile into the project root 2. $ vagrant up 3. $ vagrant ssh 4. Profit! 31

Slide 32

Slide 32 text

Vagrantfile V a g r a n t . c o n f i g u r e ( " 2 " ) d o | c o n f i g | c o n f i g . v m . b o x = " p r o j e c t _ n a m e " c o n f i g . v m . b o x _ u r l = " h t t p : / / f i l e s . v a g r a n t u p . c o m / p r e c i s e 6 4 . b o x " c o n f i g . v m . n e t w o r k : p r i v a t e _ n e t w o r k , i p : " 1 9 2 . 1 6 8 . 3 3 . 1 0 0 " e n d 32

Slide 33

Slide 33 text

...Vagrant's Success Story 1. Put a Vagrantfile into the project root 2. $ vagrant up 3. $ vagrant ssh 4. Profit! ..??? 5. Deployment 33

Slide 34

Slide 34 text

Deployment tools Chef (Ruby) Puppet (Ruby) Ansible (Python) Salt (Python) 34

Slide 35

Slide 35 text

Ansible Success Story 1. Specify an inventory file 2. Specify a playbook 3. $ ansible-playbook ./playbook.yml -i ./hosts 35

Slide 36

Slide 36 text

Ansible inventory [ d e v e l o p - v m ] 1 9 2 . 1 6 8 . 3 3 . 1 0 0 36

Slide 37

Slide 37 text

Ansible playbook - - - - h o s t s : d e v e l o p - v m s u d o : y e s t a s k s : - n a m e : e n s u r e a p t c a c h e i s u p t o d a t e a c t i o n : a p t u p d a t e _ c a c h e = y e s http://www.ansibleworks.com/docs/modules.html 37

Slide 38

Slide 38 text

Ansible meets Vagrant $ a n s i b l e - p l a y b o o k . / p l a y b o o k . y m l - i . / h o s t s - v - u v a g r a n t - c p a r a m i k o - - p r i v a t e - k e y = $ H O M E / . v a g r a n t . d / i n s e c u r e _ p r i v a t e _ k e y or $ v a g r a n t p r o v i s i o n 38

Slide 39

Slide 39 text

Ansible - Vagrant Integration # V a g r a n t f i l e V a g r a n t . c o n f i g u r e ( " 2 " ) d o | c o n f i g | # . . . c o n f i g . v m . p r o v i s i o n : a n s i b l e d o | a n s i b l e | a n s i b l e . p l a y b o o k = " . / p l a y b o o k . y m l " a n s i b l e . i n v e n t o r y _ p a t h = " . / h o s t s " a n s i b l e . v e r b o s e = " v " e n d e n d 39

Slide 40

Slide 40 text

Python & Web

Slide 41

Slide 41 text

Web Server Gateway Interface Standardized by PEP 333, PEP 3333 d e f a p p l i c a t i o n ( e n v i r o n , s t a r t _ r e s p o n s e ) : s t a r t _ r e s p o n s e ( ' 2 0 0 O K ' , [ ( ' C o n t e n t - T y p e ' , ' t e x t / p l a i n ' ) ] ) y i e l d ' H e l l o W o r l d \ n ' 41

Slide 42

Slide 42 text

WSGI Servers • CherryPy WSGI Server • gunicorn • tornado.wsgi • uWSGI • Waitress http://nichol.as/benchmark-of-python-web-servers 42

Slide 43

Slide 43 text

Frameworks

Slide 44

Slide 44 text

Heavyweights • Django • Twisted • web2py • Zope 44

Slide 45

Slide 45 text

Middleweights • CherryPy • Pyramid • Tornado • TurboGears 45

Slide 46

Slide 46 text

Flyweights • Bottle • Flask • Webpy 46

Slide 47

Slide 47 text

Database Access

Slide 48

Slide 48 text

ORM • SQLAlchemy ORM • Django ORM (Django only) • Pony ORM • Peewee 48

Slide 49

Slide 49 text

Something else • Native DB-API 2.0 (standardized with PEP 249) • SQLAlchemy's raw queries • mosql 49

Slide 50

Slide 50 text

SQL DDL Versioning & Data Migration • South (Django only) • Alembic (SQLAlchemy) • Pyrseas 50

Slide 51

Slide 51 text

Background task processing

Slide 52

Slide 52 text

Celery • Out of the box solution for complex cases; • Multiple brokers - RabbitMQ, Redis, RDBMS, SQS, MongoDB etc; • Tasks scheduler; • Results backends. 52

Slide 53

Slide 53 text

RQ (Redis Queue) • Designed to have a low barrier to enter; • Uses Redis' Pub/Sub feature. 53

Slide 54

Slide 54 text

Process managers

Slide 55

Slide 55 text

Process managers • Supervisor • Circus.io 55

Slide 56

Slide 56 text

Sentry

Slide 57

Slide 57 text

No content

Slide 58

Slide 58 text

No content

Slide 59

Slide 59 text

No content

Slide 60

Slide 60 text

Testing tools • unittest • py.test • nose • WebTest • coverage • tox 60

Slide 61

Slide 61 text

Thank you!