2016 - Moshe Zadka - Pants, Building Python for Fun and Profit

Description
For integrated services, it makes sense to keep several logical Python projects in a single repository -- a common library, a web front end and a back end service. For such repositories, Pants (build in Python for Python, Java, C++ and more) helps maintain dependencies and build (mostly) stand-alone executables which simplify deployment.
​
Abstract
Pants is a modern build system written in Python. It can build Python, Java, C++, Go and more. Twitter, Square and FourSquare use it internally, and contribute to it.

Bio
Moshe is a Twisted contributor, and has contributed to core Python. He loves infrastructure for building, monitoring and making services highly available.

https://youtu.be/f5nNzI8riNg

Transcript

1. Pants Building Python for Fun and Profit Moshe Zadka https://github.com/moshez
2. Motivation
3. Deploy via pip install On source machine: $ python setup.py sdist On target machine: $ pip install -r requirements.txt $ tar xvzf *.tar.gz $ python setup.py install
4. Deploy via virtualenv On source machine: $ virtualenv /myenv $ /myenv/bin/pip install -r requirements.txt $ /myenv/bin/python setup.py install $ tar cvzf deploy.tar.gz /myenv On target machine: $ cd / $ tar xvzf deploy.tar.gz
5. Dream On source machine: $ magic-command -o one-file On target machine: $ # relax
6. Using Pex On source machine: $ virtualenv /myenv $ /myenv/bin/pip install pex $ /myenv/bin/pex -o one-file -r requirements.txt . On target machine: $ # relax
7. How Pex works Pt.1 $ python file.zip Add to s y s . p a t h Run _ _ m a i n _ _ . p y
8. How Pex works Pt.2 Zip files can have arbitrary content prepended # ! / u s r / b i n / e n v p y t h o n
9. Pex Limitations Core Python Shared libraries
10. Running Pants $ ./pants goals Root where p a n t s is!
11. Pants Test $ ./pants test hello:hello-test ... hello/test_hello.py F ==================== FAILURES ==================== _________________ test_something _________________ def test_something(): > assert 1 == 2 E assert 1 == 2 hello/test_hello.py:2: AssertionError ============ 1 failed in 0.01 seconds ============ FAILURE ...
12. Pants Run $ ./pants run hello ... Hello world ...
13. Pants Binary $ ./pants binary hello ... $ ./dist/hello.pex Hello world
14. History Loosely based on Blaze Written at Twitter Adopted by Square, Medium, FourSquare
15. Look for Root discovery p a n t s
16. Install virtualenv curl shell script pex
17. Disclaimer Reproducible install
18. Bootstrap: Reproducible install $ PIP_URL="https://pypi.python.org/[.....]/pip-8.1.2.tar.gz" $ mkdir vendor/ $ curl -O PIP_URL $ mv pip-* vendor/ $ git add vendor && git commit -a -m 'Added pip ball'
19. Pants: Reproducible install # ...imports... HERE = os.path.dirname(os.path.abspath(sys.argv[0])) PANTS = os.path.join(HERE, 'build/pants.pex') PENV = os.path.join(HERE, 'build/penv') VENDOR = os.path.join(HERE, 'vendor') cc = subprocess.check_call if not os.path.exists(PANTS): if os.path.exists(PENV): shutil.rmtree(PENV) cc(['virtualenv', PENV]) cc([os.path.join(PENV, 'bin/pip'), 'install', '--upgrade', 'pip', 'pantsbuild.pants', '--no-index', '--find-links', VENDOR]) cc([os.path.join(PENV, 'bin/pex'), '--no-wheel', '--repo', VENDOR, '--no-index', '--output-file', PANTS, '-m', 'pants.bin.pants_exe', 'pantsbuild.pants']) os.execv(PANTS, [PANTS] + sys.argv[1:])
20. Revendor pt.1: Reproducible install # ...imports... HERE = os.path.dirname(os.path.abspath(sys.argv[0])) BUILD = os.path.join(HERE, 'build') PENV = os.path.join(BUILD, 'revenvdor') VENDOR = os.path.join(HERE, 'vendor') NEW_VENDOR = os.path.join(BUILD, 'new-vendor') if os.path.exists(PENV): shutil.rmtree(PENV) cc = subprocess.check_call cc(['virtualenv', PENV]) cc([os.path.join(PENV, 'bin/pip'), 'install', '--upgrade', 'pip', '--no-index', '--find-links', VENDOR]) if os.path.exists(NEW_VENDOR): shutil.rmtree(NEW_VENDOR) os.makedirs(NEW_VENDOR) cc([os.path.join(PENV, 'bin/pip'), 'download', '--no-binary', ':all:', 'pantsbuild.pants', 'pip'], cwd=NEW_VENDOR)
21. Revendor pt.2: Reproducible install old_files = set(os.listdir(VENDOR)) new_files = set(os.listdir(NEW_VENDOR)) removed = old_files - new_files if removed: cc(['git', 'rm'] + [os.path.join(VENDOR, fname) for fname in removed]) added = new_files - old_files for fname in new_files - old_files: shutil.copy(os.path.join(NEW_VENDOR, fname), os.path.join(VENDOR, fname)) cc(['git', 'add', VENDOR])
22. Server $ ./pants server INFO] Launched server with pid 16959 at http://localhost:42516
23. pants.ini Has to be there Command line option mapping
24. pants.ini overrides $ PANTS_CONFIG_OVERRIDE="['pants-other.ini']" ./pants ...
25. pants-ci.ini [GLOBAL] enable_pantsd: False
26. BUILD Defines sources, dependencies Build hints
27. python_binary $ cat BUILD python_binary( name = 'hello', source = 'hello.py', ) $ cat hello.py print("Hello world")
28. entry_point $ cat BUILD python_binary( name = 'hello2', source = 'hello2.py', entry_point = 'hello2:main', ) $ cat hello2.py def main(): print("Hello world")
29. python_library Does not produce output Good dependency target
30. python_library Example libhello/BUILD python_binary(name='hellobin', source='main.py', dependencies=[":libhello"]) python_library(name="libhello", sources=['hello.py']) libhello/hello.py def greet(): return "Hello world" libhello/main.py from hello import greet print(greet()) $ ./pants binary libhello:hellobin ... $ ./dist/hellobin.pex Hello world
31. python_test Runs tests Uses py.test
32. python_test Example $ cat testhello/BUILD python_tests( name = 'hello-test', sources = ['test_hello.py'], ) $ cat testhello/test_hello.py def test_something(): assert 1 == 2
33. Backends [DEFAULT] backend_packages: [ "pants.contrib.python.checks", ]
34. 3rd Party Pattern Concentrate all requirements in one place
35. 3rd Party BUILD $ cat 3rdparty/python python_requirement_library( name='ansicolors', requirements=[ python_requirement(name='ansicolors', requirement='ansicolors==1.0.2'), ])
36. Local code BUILD $ cat greet/BUILD python_library(name='greet', dependencies=[ '3rdparty/python:ansicolors', ], sources=globs('*.py'), )
37. Using requirements.txt $ cat 3rdparty/python/requirements.txt ansicolors==1.0.2 $ cat 3rdparty/python/BUILD python_requirements()
38. Alternative repositories $ cat pants.ini [python-repos] indexes: [URL]
39. Plugin types Loose Built
40. Q&A