Building conda Packages (Trent Nelson)

Building conda packages
Trent Nelson @trentnelson
as presented to NYC Python

Transcript

1. Building conda packages PyData NYC, AppNexus, 6th February 2014 Trent Nelson trent.nelson@continuum.io, @trentnelson
2. In 15 minutes time you should know… • What a conda recipe is • How to create a recipe for a new package • How to build a recipe on OS X, Linux and Windows • What binstar is, and why conda keeps asking if you want to upload to it
3. Pre-requisites • http://continuum.io/downloads • Either Anaconda or Miniconda oAnaconda installation • Everything oMiniconda installation • Just Python + conda • (Pro-tip: you can install Miniconda then `conda install anaconda` to bring everything else down)
4. Step 1: Install conda-build • conda install conda-build
5. Conda Recipes • meta.yaml o Simple description of the package o Name, version o Where to download • build.sh & bld.bat o conda runs this in order to build a package • Docs: http://docs.continuum.io/conda/build.html
6. Step 2: Clone conda-recipes • git clone https://github.com/conda/conda-recipes
7. Step 3: Try build something • Let’s try pyfaker
8. How I go about creating new packages… • Copy an existing recipe that is a similar package type • Edit meta.yaml and fill in according to new package • Double-check build.sh o $PYTHON setup.py install? o ./configure --PREFIX=$PREFIX • Or bld.bat if you’re doing a Windows build • Keep typing `conda build .` ‘til it works :-) o Simple packages are usually easy o Hard packages are possible • Binstar upload when done!
9. Dependencies • Patchelf example o Needed automake o Which needed autoconf o Which needed m4 o Ended up creating four packages: o https:// github.com/conda/conda-recipes/commit/57c008afc2dedd1d34fe7be4c421aede 347dd1bd • Divide and conquer!
10. Common Issues/Mistakes
11. chrpath (Linux) (trent@centos5x64:ttypts/3) (Thu/17:27) .. (~s/conda-recipes/pyfaker) % conda build . Error: Did not find 'chrpath' in: /home/trent/anaconda/envs/_build/bin:/home/trent/anaconda/bin:/home/trent/anaconda/bin: /usr/local/bin:/bin:/usr/bin:.:/home/trent/.zsh/bin 'chrpath' is necessary for building conda packages on Linux with relocatable ELF libraries. You can install chrpath using apt-get, yum or conda. (trent@centos5x64:ttypts/3) (Thu/17:32) .. (~s/conda-recipes/pyfaker) % conda install chrpath Package plan for installation in environment /home/trent/anaconda: The following packages will be downloaded: package | build ---------------------------|----------------- chrpath-0.13 | 0 13 KB The following packages will be linked: package | build ---------------------------|----------------- chrpath-0.13 | 0 hard-link Proceed ([y]/n)? y Fetching packages ... chrpath-0.13-0.tar.bz2 100% |########################| Time: 0:00:00 219.53 kB/s Extracting packages ... [ COMPLETE ] |##################################################| 100% Linking packages ... [ COMPLETE ] |##################################################| 100% (trent@centos5x64:ttypts/3) (Thu/17:33) .. (~s/conda-recipes/pyfaker) % which chrpath
12. Forgetting ‘- distribute’ build requirement (trent@centos5x64:ttypts/3) (Thu/17:43) .. (~s/conda-recipes/pyfak % conda build . <snip> source tree in: /home/trent/anaconda/conda-bld/work + /home/trent/anaconda/envs/_build/bin/python setup.py install Traceback (most recent call last): File "setup.py", line 4, in <module> from setuptools import setup, find_packages ImportError: No module named setuptools Command failed: /bin/bash -x -e /mnt/hgfs/Trent/src/conda-recipes/pyfaker/bu
13. Forgetting ‘- distribute’ build requirement
14. You’ve built a package… now what?
15. After a successful build…
16. Binstar!
17. Your own package distribution channel
18. Conda & Binstar
19. Conda & Binstar
20. Recent Package Rampages • autoconf, automake, m4, apr • mysql, postgresql • postgis (proj4, geos, json-c) • Windows specific stuff: o rktools – Windows Server 2003 Resource Kit o sysinternals – Entire Sysinternals Suite (procexp etc) o notepad2-mod – Handy way to get a less-suckier version of notepad.exe • (These are all in https://github.com/conda/conda-recipes.git and accessible by adding ‘trent’ conda channel)
21. Conda and R • Add the r channel: o conda config --add channels r • Install just R: o conda install r • Install R + rpy2 R/Python bridge: o conda install rpy2 • Install a CRAN task view: o conda install r-view-bayesian-core o conda install r-view-machinelearning-core
22. Conda and R
23. OEM Support: DataNitro • Continuum and DataNitro are friends • I wanted to evaluate DataNitro at a client’s site but didn’t have local admin privs • One Saturday later…
24. Anaconda Server
25. Anaconda Server • Instead of this: o Install Anaconda manually on developers’ workstations. o ….and manually on the Linux servers o Then remember to `conda install` all the packages you use o And keep them at the same versions o ….and make sure people set their http_proxy and https_proxy env vars o ….and make sure people add the right conda channels o etc • You get this: o Centralized package management (within your firewall) o Private package repositories (think internal binstar) o Customized installers o Leverage conda packages internally: • `conda install ourcompany-devkit` o Commercial offerings: NumbaPro, IOPro