Python Developer's Daily Routine

My Chebit'13 presentation

Transcript

1. Python Developer's Daily Routine Maxim Avanov (github.com/avanov)
2. Project structure
3. Project structure . / c h e b i t | - - c h e b i t | ` - - _ _ i n i t _ _ . p y ` - - s e t u p . p y 3
4. setup.py f r o m s e t u p t o o l s i m p o r t s e t u p , f i n d _ p a c k a g e s s e t u p ( n a m e = ' C h e b i t ' , v e r s i o n = ' 0 . 0 . 1 ' , p a c k a g e s = f i n d _ p a c k a g e s ( ) , t e s t _ s u i t e = ' t e s t s ' , t e s t s _ r e q u i r e = [ ' p y t e s t ' , ' c o v e r a g e ' ] , i n s t a l l _ r e q u i r e s = [ ' P y r a m i d = = 1 . 5 a 2 ' ] ) 4
5. Bootstrapping $ python setup.py install $ python setup.py develop $ python setup.py test $ python setup.py regtister sdist [bdist_egg] upload 5
6. Package management
7. pip commands $ p i p i n s t a l l P y r a m i d $ p i p i n s t a l l P y r a m i d - - u p g r a d e $ p i p i n s t a l l P y r a m i d = = 1 . 5 a 2 - - u p g r a d e $ p i p u n i n s t a l l P y r a m i d 7
8. pip commands $ p i p f r e e z e B a b e l = = 1 . 3 C h e b i t = = 0 . 0 . 1 C o f f e e S c r i p t = = 1 . 0 . 8 J i n j a 2 = = 2 . 7 . 1 M a k o = = 0 . 9 . 0 . . . 8
9. pip-tools $ p i p - r e v i e w r e q u e s t s = = 0 . 1 3 . 4 a v a i l a b l e ( y o u h a v e 0 . 1 3 . 2 ) $ p i p - r e v i e w - - a u t o . . . < p i p i n s t a l l o u t p u t > $ p i p - r e v i e w - - i n t e r a c t i v e r e q u e s t s = = 0 . 1 4 . 0 a v a i l a b l e ( y o u h a v e 0 . 1 3 . 2 ) U p g r a d e n o w ? [ Y ] e s , [ N ] o , [ A ] l l , [ Q ] u i t y . . . 9
10. Cookiecutter • Generates initial project files from defined templates; • File names are also templates; • Transparently works with remote VCS. 10
11. Cookiecutter template . / c h e b i t - t e m p l a t e | - - c o o k i e c u t t e r . j s o n ` - - { { c o o k i e c u t t e r . p r o j e c t _ n a m e } } | - - { { c o o k i e c u t t e r . p r o j e c t _ n a m e } } | ` - - _ _ i n i t _ _ . p y ` - - s e t u p . p y 11
12. cookiecutter.json { " p r o j e c t _ n a m e " : " m y p r o j e c t " , " p r o j e c t _ v e r s i o n " : " 0 . 0 . 1 " } 12
13. Cookiecutter in action $ c o o k i e c u t t e r { p r o j e c t _ t e m p l a t e _ p a t h } p r o j e c t _ n a m e ( d e f a u l t i s " m y p r o j e c t " ) ? m y n e w p r o j e c t _ v e r s i o n ( d e f a u l t i s " 0 . 0 . 1 " ) ? 0 . 1 . 1 $ t r e e . / m y n e w . / m y n e w | - - m y n e w | ` - - _ _ i n i t _ _ . p y ` - - s e t u p . p y 13
14. We'd like to isolate project environments
15. virtualenv $ v i r t u a l e n v ~ / v e n v / { p r o j e c t _ n a m e } $ s o u r c e ~ / v e n v / { p r o j e c t _ n a m e } / b i n / a c t i v a t e What if we need another Python version? $ v i r t u a l e n v ~ / v e n v / { p r o j e c t _ n a m e } - p p y t h o n 3 15
16. Still not happy enough!
17. Introducing pyenv
18. pyenv - 5 steps to happiness $ g i t c l o n e g i t : / / g i t h u b . c o m / y y u u / p y e n v . g i t ~ / p y e n v $ e c h o ' e x p o r t P Y E N V _ R O O T = " $ H O M E / p y e n v " ' > > ~ / . b a s h r c $ e c h o ' e x p o r t P A T H = " $ P Y E N V _ R O O T / b i n : $ P A T H " ' > > ~ / . b a s h r c $ e c h o ' e v a l " $ ( p y e n v i n i t - ) " ' > > ~ / . b a s h r c $ e x e c $ S H E L L 18
19. pyenv - Rise and Shine! $ p y e n v i n s t a l l - l $ p y e n v i n s t a l l 2 . 7 . 6 $ p y e n v i n s t a l l 3 . 3 . 3 $ p y e n v i n s t a l l p y p y - 2 . 2 . 1 $ p y e n v r e h a s h 19
20. The Happiness $ pyenv versions $ pyenv version $ pyenv local 2.7.6 3.3.3 $ pyenv global 3.3.3 $ pyenv shell pypy-2.2.1 $ pyenv whence 2to3 20
21. ... $ w h i c h p y t h o n / h o m e / g h o s t w r i t e r / p y e n v / s h i m s / p y t h o n $ p y t h o n P y t h o n 2 . 7 . 3 ( 8 7 a a 9 d e 1 0 f 9 c , N o v 2 4 2 0 1 3 , 1 8 : 4 8 : 1 3 ) [ P y P y 2 . 2 . 1 w i t h G C C 4 . 6 . 3 ] o n l i n u x 2 > > > > 21
22. virtualenv meets pyenv
23. pyenv-virtualenv $ g i t c l o n e g i t : / / g i t h u b . c o m / y y u u / p y e n v - v i r t u a l e n v . g i t ~ / p y e n v / p l u g i n s / p y e n v - v i r t u a l e n v $ p y e n v v i r t u a l e n v p y p y - 2 . 2 . 1 ~ / v e n v / { p r o j e c t _ n a m e } 23
24. pyenv-virtualenv in action $ p y e n v v i r t u a l e n v s { p r o j e c t _ n a m e } ( c r e a t e d f r o m / h o m e / g h o s t w r i t e r / p y e n v / v e r s i o n s / p y p y - 2 . 2 . 1 ) $ s o u r c e ~ / p y e n v / v e r s i o n s / { p r o j e c t _ n a m e } / b i n / a c t i v a t e 24
25. Well done!
26. Now, what about other components?
27. Virtualization
28. Vagrant • Written in Ruby • Relies on VirtualBox • Runs on Linux / OS X / Win 28
29. Juju • Written in Python / Go • Relies on LXC • Linux only 29
30. Docker • Written in Go • Relies on LXC and AUFS • Linux only 30
31. Vagrant's Success Story... 1. Put a Vagrantfile into the project root 2. $ vagrant up 3. $ vagrant ssh 4. Profit! 31
32. Vagrantfile V a g r a n t . c o n f i g u r e ( " 2 " ) d o | c o n f i g | c o n f i g . v m . b o x = " p r o j e c t _ n a m e " c o n f i g . v m . b o x _ u r l = " h t t p : / / f i l e s . v a g r a n t u p . c o m / p r e c i s e 6 4 . b o x " c o n f i g . v m . n e t w o r k : p r i v a t e _ n e t w o r k , i p : " 1 9 2 . 1 6 8 . 3 3 . 1 0 0 " e n d 32
33. ...Vagrant's Success Story 1. Put a Vagrantfile into the project root 2. $ vagrant up 3. $ vagrant ssh 4. Profit! ..??? 5. Deployment 33
34. Deployment tools Chef (Ruby) Puppet (Ruby) Ansible (Python) Salt (Python) 34
35. Ansible Success Story 1. Specify an inventory file 2. Specify a playbook 3. $ ansible-playbook ./playbook.yml -i ./hosts 35
36. Ansible inventory [ d e v e l o p - v m ] 1 9 2 . 1 6 8 . 3 3 . 1 0 0 36
37. Ansible playbook - - - - h o s t s : d e v e l o p - v m s u d o : y e s t a s k s : - n a m e : e n s u r e a p t c a c h e i s u p t o d a t e a c t i o n : a p t u p d a t e _ c a c h e = y e s http://www.ansibleworks.com/docs/modules.html 37
38. Ansible meets Vagrant $ a n s i b l e - p l a y b o o k . / p l a y b o o k . y m l - i . / h o s t s - v - u v a g r a n t - c p a r a m i k o - - p r i v a t e - k e y = $ H O M E / . v a g r a n t . d / i n s e c u r e _ p r i v a t e _ k e y or $ v a g r a n t p r o v i s i o n 38
39. Ansible - Vagrant Integration # V a g r a n t f i l e V a g r a n t . c o n f i g u r e ( " 2 " ) d o | c o n f i g | # . . . c o n f i g . v m . p r o v i s i o n : a n s i b l e d o | a n s i b l e | a n s i b l e . p l a y b o o k = " . / p l a y b o o k . y m l " a n s i b l e . i n v e n t o r y _ p a t h = " . / h o s t s " a n s i b l e . v e r b o s e = " v " e n d e n d 39
40. Python & Web
41. Web Server Gateway Interface Standardized by PEP 333, PEP 3333 d e f a p p l i c a t i o n ( e n v i r o n , s t a r t _ r e s p o n s e ) : s t a r t _ r e s p o n s e ( ' 2 0 0 O K ' , [ ( ' C o n t e n t - T y p e ' , ' t e x t / p l a i n ' ) ] ) y i e l d ' H e l l o W o r l d \ n ' 41
42. WSGI Servers • CherryPy WSGI Server • gunicorn • tornado.wsgi • uWSGI • Waitress http://nichol.as/benchmark-of-python-web-servers 42
43. Frameworks
44. Heavyweights • Django • Twisted • web2py • Zope 44
45. Middleweights • CherryPy • Pyramid • Tornado • TurboGears 45
46. Flyweights • Bottle • Flask • Webpy 46
47. Database Access
48. ORM • SQLAlchemy ORM • Django ORM (Django only) • Pony ORM • Peewee 48
49. Something else • Native DB-API 2.0 (standardized with PEP 249) • SQLAlchemy's raw queries • mosql 49
50. SQL DDL Versioning & Data Migration • South (Django only) • Alembic (SQLAlchemy) • Pyrseas 50
51. Background task processing
52. Celery • Out of the box solution for complex cases; • Multiple brokers - RabbitMQ, Redis, RDBMS, SQS, MongoDB etc; • Tasks scheduler; • Results backends. 52
53. RQ (Redis Queue) • Designed to have a low barrier to enter; • Uses Redis' Pub/Sub feature. 53
54. Process managers
55. Process managers • Supervisor • Circus.io 55
56. Sentry
60. Testing tools • unittest • py.test • nose • WebTest • coverage • tox 60
61. Thank you!