Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Building conda Packages (Trent Nelson)

NYC Python
February 06, 2014

Building conda Packages (Trent Nelson)

Building conda packages
Trent Nelson @trentnelson
as presented to NYC Python

NYC Python

February 06, 2014
Tweet

More Decks by NYC Python

Other Decks in Technology

Transcript

  1. In 15 minutes time you should know… • What a

    conda recipe is • How to create a recipe for a new package • How to build a recipe on OS X, Linux and Windows • What binstar is, and why conda keeps asking if you want to upload to it
  2. Pre-requisites • http://continuum.io/downloads • Either Anaconda or Miniconda oAnaconda installation

    • Everything oMiniconda installation • Just Python + conda • (Pro-tip: you can install Miniconda then `conda install anaconda` to bring everything else down)
  3. Conda Recipes • meta.yaml o Simple description of the package

    o Name, version o Where to download • build.sh & bld.bat o conda runs this in order to build a package • Docs: http://docs.continuum.io/conda/build.html
  4. How I go about creating new packages… • Copy an

    existing recipe that is a similar package type • Edit meta.yaml and fill in according to new package • Double-check build.sh o $PYTHON setup.py install? o ./configure --PREFIX=$PREFIX • Or bld.bat if you’re doing a Windows build • Keep typing `conda build .` ‘til it works :-) o Simple packages are usually easy o Hard packages are possible • Binstar upload when done!
  5. Dependencies • Patchelf example o Needed automake o Which needed

    autoconf o Which needed m4 o Ended up creating four packages: o https:// github.com/conda/conda-recipes/commit/57c008afc2dedd1d34fe7be4c421aede 347dd1bd • Divide and conquer!
  6. chrpath (Linux) (trent@centos5x64:ttypts/3) (Thu/17:27) .. (~s/conda-recipes/pyfaker) % conda build .

    Error: Did not find 'chrpath' in: /home/trent/anaconda/envs/_build/bin:/home/trent/anaconda/bin:/home/trent/anaconda/bin: /usr/local/bin:/bin:/usr/bin:.:/home/trent/.zsh/bin 'chrpath' is necessary for building conda packages on Linux with relocatable ELF libraries. You can install chrpath using apt-get, yum or conda. (trent@centos5x64:ttypts/3) (Thu/17:32) .. (~s/conda-recipes/pyfaker) % conda install chrpath Package plan for installation in environment /home/trent/anaconda: The following packages will be downloaded: package | build ---------------------------|----------------- chrpath-0.13 | 0 13 KB The following packages will be linked: package | build ---------------------------|----------------- chrpath-0.13 | 0 hard-link Proceed ([y]/n)? y Fetching packages ... chrpath-0.13-0.tar.bz2 100% |########################| Time: 0:00:00 219.53 kB/s Extracting packages ... [ COMPLETE ] |##################################################| 100% Linking packages ... [ COMPLETE ] |##################################################| 100% (trent@centos5x64:ttypts/3) (Thu/17:33) .. (~s/conda-recipes/pyfaker) % which chrpath
  7. Forgetting ‘- distribute’ build requirement (trent@centos5x64:ttypts/3) (Thu/17:43) .. (~s/conda-recipes/pyfak %

    conda build . <snip> source tree in: /home/trent/anaconda/conda-bld/work + /home/trent/anaconda/envs/_build/bin/python setup.py install Traceback (most recent call last): File "setup.py", line 4, in <module> from setuptools import setup, find_packages ImportError: No module named setuptools Command failed: /bin/bash -x -e /mnt/hgfs/Trent/src/conda-recipes/pyfaker/bu
  8. Recent Package Rampages • autoconf, automake, m4, apr • mysql,

    postgresql • postgis (proj4, geos, json-c) • Windows specific stuff: o rktools – Windows Server 2003 Resource Kit o sysinternals – Entire Sysinternals Suite (procexp etc) o notepad2-mod – Handy way to get a less-suckier version of notepad.exe • (These are all in https://github.com/conda/conda-recipes.git and accessible by adding ‘trent’ conda channel)
  9. Conda and R • Add the r channel: o conda

    config --add channels r • Install just R: o conda install r • Install R + rpy2 R/Python bridge: o conda install rpy2 • Install a CRAN task view: o conda install r-view-bayesian-core o conda install r-view-machinelearning-core
  10. OEM Support: DataNitro • Continuum and DataNitro are friends •

    I wanted to evaluate DataNitro at a client’s site but didn’t have local admin privs • One Saturday later…
  11. Anaconda Server • Instead of this: o Install Anaconda manually

    on developers’ workstations. o ….and manually on the Linux servers o Then remember to `conda install` all the packages you use o And keep them at the same versions o ….and make sure people set their http_proxy and https_proxy env vars o ….and make sure people add the right conda channels o etc • You get this: o Centralized package management (within your firewall) o Private package repositories (think internal binstar) o Customized installers o Leverage conda packages internally: • `conda install ourcompany-devkit` o Commercial offerings: NumbaPro, IOPro