Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
Writing a Python C extension in 2018
Jean-Baptiste Aviat
May 16, 2018
Programming
1
860
Writing a Python C extension in 2018
Jean-Baptiste Aviat
May 16, 2018
Tweet
Share
More Decks by Jean-Baptiste Aviat
See All by Jean-Baptiste Aviat
Content Security Policy in your Rails apps in 30s
aviat
0
180
App Security is just a Git flow (with built-in conflicts)
aviat
0
960
Tune your App Perf (and get fit for summer)
aviat
2
370
Other Decks in Programming
See All in Programming
Azure Functionsをサクッと開発、サクッとデプロイ/vscodeconf2023-baba
nina01
1
340
jq at the Shortcuts
cockscomb
1
420
花き業界のサプライチェーンを繋げるプロダクト開発の進め方
userlike1
0
170
Remote SSHで行うVS Codeリモートホスト開発とトラブルシューティング
smt7174
1
460
フロントエンドで学んだことをデータ分析で使ってみた話
daichi_igarashi
0
180
xarray-Datatree: Hierarchical Data Structures for Multi-Model Science
tomnicholas
0
210
ITエンジニア特化型Q&Aサイトteratailを 言語、DB、クラウドなど フルリプレイスした話
leveragestech
0
410
Rによる大規模データの処理
s_uryu
2
630
TokyoR#103_DataProcessing
kilometer
0
530
PHPアプリケーションにおけるアーキテクチャメトリクスについて / Architecture Metrics in PHP Applications
isanasan
1
240
WordPress(再)入門 - 基礎知識・環境編
oleindesign
1
130
NGK2023S - OCaml最高! スマホ開発にも使えちゃう?!
haochenxie
0
120
Featured
See All Featured
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
227
16k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
500
130k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
217
21k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
29
7.9k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
2
400
A designer walks into a library…
pauljervisheath
199
16k
What the flash - Photography Introduction
edds
64
10k
Making Projects Easy
brettharned
102
4.8k
Web Components: a chance to create the future
zenorocha
304
40k
Typedesign – Prime Four
hannesfritz
34
1.5k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
182
15k
The Power of CSS Pseudo Elements
geoffreycrofte
52
4.3k
Transcript
Writing a C Python extension in 2018 Paris.py 2018
Jean-Baptiste Aviat CTO & Co-founder of sqreen.io Former Hacker at
Apple (Red Team) PyMiniRacer author
[email protected]
@jbaviat Who Am I?
Someday… we needed to use V8 from Python. What we
ship: • is public • is widely used • need to have frictionless install $ pip install sqreen —> it just works™!
None
PyMiniRacer: cool JS binding github.com/sqreen/PyMiniRacer
The problem • V8 is C++ • How do you
run C++ in Python? …We need some kind of binding between these 2 worlds.
Many popular packages rely on binary cryptography numpy pymongo psycopg
simplejson lxml sqlalchemy…
People do it! Let’s do it too.
What are our goals? • minimize maintenance • make setup
easy • make testing easy • have great performance • have low memory fingerprint • dev time is a constraint We want to: And (obviously)…
built-in pythonic Python version independant open to other languages high
throughput capable CPython ✔ ✔ ✔ ctypes ✔ ✔ ✔ cffi ✔ ✔ ✔ ✔ Cython ✔ ✔ SWIG ✔ ✔
ctypes Built into Python Binary is Python independant: • can
be used on any Python version • can be used in other languages! No tight integration to Python • not high throughput capable • less Pythonic Complex syntax (C types wrapped in Python…) Not for C++
$ python >>> path = "./hello.so" >>> import ctypes >>>
lib = ctypes.cdll.LoadLibrary(path) >>> lib.hello_world() Hello world! C file Python interface binary object
Overview V8 (C++ interface) C interface to V8 Python interface
3rd party binaries import ctypes class PyMiniRacer(object): … #include <v8.h> int miniracer_init(); … V8 library (libv8.a) V8 headers (v8.h) linking ctypes C/C++ code Python library
How to put this together? $ cat setup.py from distutils.core
import setup, Extension extension = Extension('hello', ['hello.c']) setup(name=‘hello', version='1.0', ext_modules=[extension]) $ python setup.py build running build running build_ext building 'hello' extension clang […] -c hello.c -o hello.o creating build/lib.macosx-10.6-intel-2.7 clang -bundle […] hello.o -o hello.so
None
None
Ship it
Ever had trouble installing packages?
None
(at Sqreen, we are dog friendly) Come get your dog
sticker!
Python packaging history sdist (source distribution) eggs wheels —> manylinux
wheels (built distribution) 2004 2012 2016 Python 2.4 Python 3.3 Python 3.6 ❤
manylinux wheels Python standard: PEP513 / PEP571 Compatible on most
(real world) Linux Only in pip >= 8.1 Need build for all Python versions Binaries need to be built on CentOS 5
CentOS 5?? PEP571 → bumped requirement to CentOS 6!
Wheels vs compilation iso builds (crash can be reproduced) many
packages need maintenance one build per user only one package harder to install…
Best of 2 worlds We can have: ctypes: use a
Python independant binary wheels: ship compiled binaries By combining these… ctypes + wheels = ship only one binary
Thanks! Questions? We’re hiring! https://sqreen.io/jobs