Upgrade to Pro — share decks privately, control downloads, hide ads and more …

PyConZA 2012: "Our hybrid programming journey w...

Pycon ZA
October 05, 2012

PyConZA 2012: "Our hybrid programming journey with Python and C++" by James Saunders

We run a fairly complicated stack that includes a C++ computation library, we serve this all from a Django based web server. The process of wrapping a complex C++ library for easy use in Python turned out to be both intricate and interesting. We not only use C++ code from Python but also use Python libraries from C++. What I really want to talk about in the basics of how we did it, from how we started trying to do it (which was a monolithic nightmare) to our current system which is fully automated and uses our own Python DSL on top of Py++ (A C++ wrapping code generator) on top of boost/Python on top of our code. The talk should be accessible for people without a great understanding of either Python or C++, but will have enough interesting subtleties for more advanced users to also learn something. In the end we will also briefly talk about alternative solutions and how we might have done things differently if we started again.

Python is a great language, but sometimes you need to use a lower-level library, such as a C++ library. One of the great things about Python is that you can. However if you are trying to wrap (expose to Python) a fairly complex library it can be a lot more tricky. We faced exactly this problem and will share our solution as well as all the little tricks and gotchas along the way. We will focus on the broad principles of hybrid coding (using multiple languages), the technologies we used to communicate between the languages, the systems we built to automate the process, as well as some of the broad lessons and reflections on what we could have done differently and how its made us think differently about development. By the end of the talk the audience should have a reasonably practical understanding of how to wrap and use a complex C++ library in Python and vice versa.

Pycon ZA

October 05, 2012
Tweet

More Decks by Pycon ZA

Other Decks in Programming

Transcript

  1. 2YHUYLHZ Ɣ 7DONVKRXOGEH ż 1RWVXSHUWHFKQLFDO ż 6RPH&NQRZOHGJHUHTXLUHG ż 1RDGYDQFHGXQGHUVWDQGLQJRIS\WKRQUHTXLUHG Ɣ

    :RQ WGLYHWRRGHHSO\LQWRVSHFLILFFRGH Ɣ :LOOWU\WREHSUDJPDWLF Ɣ 1RWH[KDXVWLYH Ɣ 6KRXOGJLYHDQLGHDRIKRZWRH[SRVHD FRPSOH[&OLEUDU\WRS\WKRQDQGYLFH YHUVD
  2. 8VLQJ&LQS\WKRQ H[WHQGLQJ FODVV)RUWXQH7HOOHU^ SXEOLF )RUWXQH7HOOHU LQWOXFNLQHVV  YHFWRULQW! JHWBORWWHU\ 

    ` IURPP\FSSOLELPSRUW )RUWXQH7HOOHU REM )RUWXQH7HOOHU  OXFN\BQXPEHUV REM JHWBORWWHU\ IRUQXPLQOXFN\BQXPEHUV SULQWQXP
  3. 7KH)XQGDPHQWDOV Ɣ 3\WKRQH[WHQVLRQPRGXOHVDUHVKDUHG OLEUDULHV ż VRLQOLQX[ ż GOOLQZLQGRZV ż ,NQRZQRWKLQJDERXW0DFV

    Ɣ F3\WKRQLVZULWWHQLQ&DQGKDVQDWLYH VXSSRUWIRUEHLQJH[WHQGHGLQ& Ɣ $FFHVVHGWKURXJK3\WKRQK
  4. 3\WKRQK VWDWLF3\2EMHFW P\BFDOOEDFN 18// VWDWLF3\2EMHFW P\BVHWBFDOOEDFN 3\2EMHFW GXPP\3\2EMHFW DUJV ^

    3\2EMHFW UHVXOW 18// 3\2EMHFW WHPS LI 3\$UJB3DUVH7XSOH DUJV2VHWBFDOOEDFN WHPS ^ LI 3\&DOODEOHB&KHFN WHPS ^ 3\(UUB6HW6WULQJ 3\([FB7\SH(UURUSDUDPHWHUPXVWEHFDOODEOH  UHWXUQ18// ` 3\B;,1&5() WHPS  $GGDUHIHUHQFHWRQHZFDOOEDFN  3\B;'(&5() P\BFDOOEDFN  'LVSRVHRISUHYLRXVFDOOEDFN  P\BFDOOEDFN WHPS 5HPHPEHUQHZFDOOEDFN   %RLOHUSODWHWRUHWXUQ1RQH  3\B,1&5() 3\B1RQH  UHVXOW 3\B1RQH ` UHWXUQUHVXOW ` %/(+* Ɣ /RZ/HYHO Ɣ &EDVHG Ɣ /RWVRI ERLOHUSODWH
  5. 3\DQGDXWRPDWHGZUDSSHU JHQHUDWLRQ Ɣ 3\WKRQSDFNDJH Ɣ 7DNHVLQ&KHDGHUILOHV Ɣ *HQHUDWHVERRVWS\WKRQFRGH & WRZUDS

    WKHJLYHQKHDGHUILOHV Ɣ 3UHWW\FRPSUHKHQVLYH Ɣ 5HDVRQDEO\VWDEOH Y Ɣ 1RWYHU\DFWLYH 1HZPDLQWDLQHUDQ\RQH" Ɣ 6WDFNRYHUIORZLV\RXUIULHQG
  6. %DVLF6WUDWHJ\ & KHDGHU ILOHV KSS 3\ VFULSW & ZUDSSLQJ 6RXUFH

    ILOHV FSS &RPSLOH 3\WKRQ ([WHQVLRQ PRGXOH VR
  7. 2XUILUVWDWWHPSW 7KHKRUURU Ɣ OLQHPRQROLWKLF&PRGXOH Ɣ 7RRNIRUHYHUWRFRPSLOH Ɣ +DGWREHKDQGWZHDNHGWRJHWLWWRFRPSLOH Ɣ &KDQJHVLQYROYHGJHQHUDWLQJDQHZYHUVLRQ

    DQGFRS\LQJRYHUVHFWLRQVRIFRGHIURPWKH ROGYHUVLRQ Ɣ 7HUULI\LQJWRPDNHFKDQJHVWRWKH& Ɣ 0DVVLYHO\VORZHGGRZQGHYHORSPHQW :RUVWWKLQJHYHU
  8. :ULWHDOLWWOH VRUWRI '6/ Ɣ 'HFODUDWLYH Ɣ $EVWUDFWWKHORZHUOHYHOS\PHWKRGV Ɣ ([SOLFLWDQGFOHDU Ɣ

    %DVLFDOO\DFRXSOHRIS\WKRQIXQFWLRQV ż GHIOLPLWBFODVV FODVVQDPHIXQFWLRQBQDPHV   ż GHIH[FOXGHBFODVV FODVVQDPH   Ɣ &OHDUSURFHVVWRDGGDQGFKDQJHZUDSSLQJV Ɣ /HDYHIXOOFDSDELOLWLHVRI3\ZKHQQHHGHG
  9. ([SRVHRQO\ZKDW\RXQHHG Ɣ %\GHIDXOWS\ZLOOH[SRVHHYHU\FODVVDQG DOOSXEOLFSURWHFWHGPHWKRGV Ɣ 7KLVPHDQVHYHQVPDOOFKDQJHVWRWKH& FDQPHVVZLWKWKHZUDSSLQJV Ɣ ([SOLFLWO\FKRRVHZKDWWRH[SRVHWRS\WKRQ ż

    $SXEOLFLQWHUIDFH Ɣ 0DNHVFKDQJHVHDVLHUWRUHDVRQDERXW Ɣ /LPLWVXQH[SHFWHGFKDQJHVLQWKHS\WKRQ LQWHUIDFH XVHRIZKLFKLVQRWVWDWLFDOO\W\SH FKHFNHG
  10. &RQYHUWZKHUH\RXFDQ Ɣ 6RPHWLPHVLWVHDVLHUWRDXWRPDWLFDOO\FRQYHUW EHWZHHQ&W\SHVDQG3\WKRQW\SHV ż 6RPHW\SHVDUHMXVWWRRGLIILFXOWWRZUDS ż 2IWHQW\SHVKDYHQDWXUDO3\WKRQDQDORJV Ɣ 'RQHIRUPDQ\EXLOWLQW\SHVHJVWULQJV

    Ɣ &DQVHWDXWRPDWLFFRQYHUWHUV ż )URP&UHWXUQYDOXHVWRS\WKRQYDOXHV ż )URP3\WKRQDUJXPHQWVWR&DUJXPHQWV Ɣ &RQVLGHUSHUIRUPDQFH Ɣ 6HHKWWSPLVVSHQWZRUGSUHVV FRPKRZWRZULWHERRVWS\WKRQ FRQYHUWHUV
  11. &RQYHUVLRQ([DPSOHV :RUNHGZHOO Ɣ S\WKRQGDWHWLPHWRERRVWSRVL[BWLPH Ɣ /RWVRIXWLOLWLHVLQS\WKRQIRUGDWHWLPH Ɣ ERRVWSRVL[BWLPHLVVXSHUKDUGWRZUDS )DLOHG Ɣ

    3\WKRQVHWWR&VHW Ɣ 3\WKRQVHWVDUHKDVKVHWV&VHWVDUH WUHHV Ɣ 'LIIHUHQWVHPDQWLFV RUGHULQJ VXEWOHHUURUV Ɣ ([SHQVLYHWRFRQYHUW
  12. 8VHWKHSUHSURFHVVRU Ɣ 7KH& SUHSURFHVVRULVIDVW Ɣ JFF[PO ZKLFK SRZHUVS\ LV VORZ

    Ɣ 8VHLWWRDJJUHJDWH DOOWKHKHDGHUV\RX QHHGLQWRRQH KHDGHUILOH DOOKSS Ɣ 0DNHVWKLQJVZD\ IDVWHU :KDWLVWKH&SUHSURFHVVRU" ,WWKHWKLQJWKDWLQWHUSUHWVVWDWHPHQWV OLNHWKHVH LQFOXGHILOH$KSS RU LIQGHI),/($ GHI),/($  HQGLI ,WLVUXQRQ\RXU&VRXUFHILOHV EHIRUHFRPSLODWLRQ
  13. &XVWRPZUDSSHUIXQFWLRQV Ɣ 6RPHWLPHVIXQFWLRQVMXVWGRQ WZUDSQLFHO\ ż HJZKHQWKH\WDNHDYHFWRU6RPHWKLQJ! DQG\RXZDQWWRSDVVDUHJXODUS\WKRQOLVWWR WKHP Ɣ :ULWHVRPHFXVWRPFRGHWKDWGRHVWKH

    PDUVKDOOLQJEHWZHHQWKHW\SHV\RXZDQWWRZRUN ZLWKLQS\WKRQDQGWKHW\SHVLQ& Ɣ ,QMHFWWKLVLQWRWKHZUDSSHUV S\DOORZV\RXWR GRWKLV Ɣ 'RQ WGRWKLVE\KDQGRQWKHJHQHUDWHGILOHV Ɣ &DQPDNHZUDSSHGREMHFWVPRUHS\WKRQLF
  14. &XVWRPZUDSSHUH[DPSOH 7RZUDSDPHWKRG LQWP\BVXPBPHWKRG YHFWRULQW!QXPEHUV WRWDNHDS\WKRQOLVW LQWP\BVXPBPHWKRG ESOLVW S\OLVWBQXPEHUV ^ VWGYHFWRULQW!YHFWRUBQXPEHUV

    IRU LQWL LOHQ S\OLVWBQXPEHUV L ^ LQWQXPEHU ESH[WUDFWLQW! S\OLVWBQXPEHUV>L@  YHFWRUBQXPEHUVSXVKBEDFN QXPEHU  ` UHWXUQP\BVXPBPHWKRG YHFWRUBQXPEHU  ` 3\WKRQOLVWDVDSDUDPHWHU 6DPHQDPHDV XQGHUO\LQJIXQFWLRQXVHV RYHUORDGLQJ ([WUDFWFRQWHQWVRIS\WKRQ OLVWDQGSODFHLWLQYHFWRU &DOORULJLQDOPHWKRG
  15. &DOOSROLFLHV Ɣ 0HWKRGVFDQUHWXUQREMHFWVWKDWKDYHWREHWUHDWHGLQ GLIIHUHQWZD\V ż 2EMHFWVE\YDOXH ż 5DZSRLQWHUV ż 5HIHUHQFHV

    Ɣ 6RPHWLPHV3\FDQILJXUHRXWZKDWWRGRVRPHWLPHV \RXQHHGWRKHOSLW Ɣ <RXFDQVHWWKHFDOOSROLF\IRUDPHWKRGLQS\HJ P\2EMPHPEHUBIXQFWLRQV JHWBFKLOG FDOOBSROLFLHV  FDOOBSROLFLHVUHWXUQBLQWHUQDOBUHIHUHQFH
  16. 5HOHDVLQJWKH*,/ Ɣ <RXFDQUHOHDVH WKH*,/WRDOORZ RWKHUWKUHDGVWR UXQ Ɣ %XWWKHQ\RXKDYH WRDTXLUHLWZKHQ \RXUPHWKRGHQGV

    Ɣ 'RQ WVFUHZWKLVXS ż 7KLQNDERXW H[FHSWLRQV Ɣ 8VHIXO0DFURV <RXU&FRGH 3\B%(*,1B$//2:B7+5($'6 3\B(1'B$//2:B7+5($'6 3\WKRQFRGH 3\WKRQFRGH <RXUORQJUXQQLQJ&PHWKRG
  17. :ULWH\RXUFRGHZLWKZUDSSLQJLQ PLQG Ɣ 6RPHWLPHV\RXKDYHWRFKDQJHWKHZD\\RX ZULWHFRGH Ɣ 6KRXOGWU\WRDYRLGWKLVEXW%(35$*0$7,& Ɣ 6RPHFRQVWUXFWVGRQRWWUDQVODWHHDVLO\ Ɣ

    'RQ WXVHH[RWLFVWUXFWXUHV XQLRQVZLHUG PHPRU\PDSVHWF Ɣ 5HWXUQW\SHVWKDWDUHHDVLO\ZUDSSHGRU FRQYHUWHG KDYHQDWXUDODQDORJV Ɣ .HHS\RXUFRGHVLPSOH
  18. $XWRPDWHHYHU\WKLQJ Ɣ &XVWRPLVDWLRQZUDSSHUJHQHUDWLRQDQG FRPSLODWLRQVKRXOGDOOEHDXWRPDWHG Ɣ 8VHDGHFHQWEXLOGV\VWHP 6FRQVWUXFW &PDNH Ɣ 3\ZRQ

    WUHJHQHUDWHILOHVWKDWKDYHQ W FKDQJHGZRUNVZHOOZLWK0$.( Ɣ 'RQ WFKHFNJHQHUDWHGFRGHLQWR\RXUVRXUFH FRQWUROV\VWHP JLWE]UKJVYQ ż 0DNHWKHJHQHUDWLRQHDVLO\UHSURGXFLEOH Ɣ 'RQ WOHWDQ\WKLQJVORZ\RXUWHDPGRZQ
  19. 7KHILQDOV\VWHP & KHDGHU ILOHV KSS 3\ & ZUDSSLQJ 6RXUFH ILOHV

    FSS &RPSLOH 3\WKRQ ([WHQVLRQ PRGXOH VR 6LQJOH KHDGHU ILOH DOOKSS :UDSSLQJ '6/ VFULSW 0DNH)LOHV &0DNH %XLOG 6\VWHP
  20. ERRVWS\WKRQDJDLQ Ɣ 0DNHVLWHDV\WR XVHS\WKRQREMHFWV LQ&NLQGDIHHOV OLNHS\WKRQ Ɣ +DVKLJKOHYHO REMHFWVWKDWPLPLF S\WKRQREMHFWV

    ż ESREMHFWESOLVW HWF Ɣ 1RQHHGWRWRXFK 3\2EMHFW 3\WKRQ GHII [\  LI \  IRR  [>@  EDU HOVH [LWHPV  UHWXUQ[ &XVLQJERRVWS\WKRQ REMHFWI REMHFW[REMHFW\ ^ LI \ IRR [VOLFH   EDU HOVH [DWWU LWHPV   UHWXUQ[ `
  21. &DOOLQJPHWKRGV 6LPSOHZD\VWRFDOOPHWKRGVDQGJHW&W\SHV EDFN VWULQJLGBQXPEHU  REMHFW\ [DWWU JHWBQDPH LGBQXPEHU 

    VWULQJQDPH H[WUDFWVWULQJ! \  RUDXWRPDWLFDOO\GRWKHW\SHFRQYHUVLRQ ESFDOOBPHWKRGVWULQJ! [JHWBQDPHLGBQXPEHU  3UHWW\VLPSOHKH\"
  22. 7KHVOLGLQJVFDOHEHWZHHQ3\WKRQ DQG& 3\WKRQ & Ɣ 3HUIRUPDQFH Ɣ +XJHDPRXQWRI H[LVWLQJFRGH Ɣ

    6SHHGRI GHYHORSPHQW Ɣ (OHJDQFH Ɣ MRLHGHYLYUH $FWXDOO\SUHWW\IDVW 1RWWKDWEDGWRXVH :KDW VULJKWIRU\RX"