embedPy, the newest interface between kdb+ and Python

By Aimi Forgan, kdb+ Engineer
Description: "embedPy, the newest interface between kdb+ and Python” followed by a live demo

Where: Dogpatch Labs, CHQ, IFSC, D1

https://www.meetup.com/PyLadiesDublin/events/dclgvlyxjbzb/

Transcript

1. 1 embedPy, the newest interface between kdb+ and Python PyLadies Dublin Meetup Louise Totten & Aimi McConnell 19 June 2018
2. 2 • written for non-kdb+ programmers to use • well documented, with understandable and useful examples • maintained and supported by Kx on a best-efforts basis, at no cost to customers • released under the Apache 2 license • free for all use cases, including 64-bit and commercial use • embedPy is part of the Fusion for kdb+ interface collection Fusion interfaces to kdb+
3. 3 • Allows the kdb+ interpreter to manipulate Python objects and call Python functions. • Python functions can be called from q, and q functions can be called from Python. embedPy programmers q/kdb+ programmers embedPy embedPy gives direct access to data in kdb+ embedPy offers easy access to the numerous computational and visualization libraries
4. 4 embedPy Requirements kdb+ ≥ v3.5 64-bit Anaconda Python 3.x macOS or Linux
5. 5 • Loads Python into kdb+, allowing access to a rich ecosystem of libraries such as: • Python variables and objects become q variables – and either language can act upon them • Python code and files can be embedded within q code • Python functions can be called as q functions embedPy
6. 6 Execute Python code directly in a q console or from a script, by prefixing the Python code with p) q)p)print(1+2) 3 You cannot run multi-line Python code directly in the console, but you can add to a q script using the syntax below. p) and subsequent lines indented according to usual Python indentation rules, for example: $cat testScript.q a:1 p)def add1(arg1): return arg1+1 embedPy
7. 7 To run a q script with embedded python code, simply load the script into your q session: q)\l testScript.q q)p)print(add1(12)) 13 To run a Python script, rename the file from file.py to file.p and load in the same way as above: $cat helloq.p print(“Hello q!”) q)\l helloq.p Hello q! embedPy
8. 8 Jupyter kernel for kdb+. Features include • Syntax Highlighting for q • Code completion for q keywords, .z/.h/.Q/.j namespace functions, and user defined variables • Code help for q keywords and basic help (display and type information) for user defined objects • script like execution of code (mulit-line input) • Inline display of charts created using embedPy and matplotlib • console stdout/stderr capture and display in notebooks • Inline loading and saving of scripts into and from notebook cells jupyterq
9. 9 jupyterq Requirements kdb+ ≥ v3.5 64-bit Anaconda Python 3.x embedPy
10. 10 Machine-learning capabilities are at the heart of future technology development at Kx. Libraries are released under the Apache 2 license, and are free for all use cases, including 64-bit and commercial use. EmbedPy forms the basis of machine learning initiative at Kx Machine Learning
11. 11 Kx have added kdb, embedPy and jupyterq to the Anaconda Python distribution platform at https://anaconda.org/kx Setup kdb, embedPy and juypterq
12. 12 There are three packages that you can install from https://anaconda.org/kx • kdb (non-commercial personal edition, requires internet connection to work) • embedPy • jupyterq Setup kdb, embedPy and juypterq TIP: installing jupyterq will automatically install the embedPy and kdb packages
13. 13 For q 64 bit version you need to request a licence from: https://ondemand.kx.com/ Setup q, embedPy and juypterq
14. 14 conda install -c kx jupyterq mcclulou@DESKTOP-4KSDJ7D:~$ conda install -c kx jupyterq Solving environment: done ## Package Plan ## environment location: /home/mcclulou/anaconda3 added / updated specs: - jupyterq The following packages will be downloaded: package | build ---------------------------|----------------- certifi-2018.4.16 | py36_0 142 KB jupyterq-1.0.1 | py36_3 43 KB kx kdb-3.6 | 2 310 KB kx embedpy-1.1.2 | py36h14c3975_0 51 KB kx ------------------------------------------------------------ Total: 547 KB The following NEW packages will be INSTALLED: embedpy: 1.1.2-py36h14c3975_0 kx jupyterq: 1.0.1-py36_3 kx kdb: 3.6-2 kx
15. 15 mcclulou@DESKTOP-4KSDJ7D:~$ git clone https://github.com/KxSystems/jupyterq.git Cloning into 'jupyterq'... remote: Counting objects: 193, done. remote: Compressing objects: 100% (38/38), done. remote: Total 193 (delta 30), reused 54 (delta 24), pack-reused 124 Receiving objects: 100% (193/193), 691.92 KiB | 0 bytes/s, done. Resolving deltas: 100% (72/72), done. Checking connectivity... done. mcclulou@DESKTOP-4KSDJ7D:~$ ls anaconda3 jupyterq notebooks q tmp mcclulou@DESKTOP-4KSDJ7D:~$ cd jupyterq mcclulou@DESKTOP-4KSDJ7D:~/jupyterq$ ls appveyor.yml install.sh kdb+Notebooks.ipynb q_widgets.ipynb compile.bat jupyterq_b64.q kernelspec README.md configure.q jupyterq_help.q kxpy requirements.txt doc jupyterq_htmlgen.q lib src docker jupyterq_kernel.q LICENSE importmatplotlib.q jupyterq_pyzmq.q makefile.in install.bat jupyterq_server.q matplotlibexample.p git clone https://github.com/KxSystems/jupyterq.git
16. 16 jupyter notebook kdb+Notebooks.ipynb mcclulou@DESKTOP-4KSDJ7D:~/jupyterq$ jupyter notebook kdb+Notebooks.ipynb [I 13:17:44.242 NotebookApp] JupyterLab beta preview extension loaded from /home/mcclulou/ anaconda3/lib/python3.6/site-packages/jupyterlab [I 13:17:44.244 NotebookApp] JupyterLab application directory is /home/mcclulou/anaconda3/ share/jupyter/lab [I 13:17:44.306 NotebookApp] Serving notebooks from local directory: /home/mcclulou/jupyterq [I 13:17:44.308 NotebookApp] 0 active kernels [I 13:17:44.313 NotebookApp] The Jupyter Notebook is running at: [I 13:17:44.314 NotebookApp] http://localhost:8888/? token=5d2095f6a5dff27020ad9a3fddc88cec5335dd00ac02ab8b [I 13:17:44.315 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). [W 13:17:44.323 NotebookApp] No web browser found: could not locate runnable browser. [C 13:17:44.323 NotebookApp] Copy/paste this URL into your browser when you connect for the first time, to login with a token: http://localhost:8888/?token=5d2095f6a5dff27020ad9a3fddc88cec5335dd00ac02ab8b
17. 17 kdb+Notebooks.ipynb
18. 18 mcclulou@DESKTOP-4KSDJ7D:~$ git clone https://github.com/awilson-kx/ notebooks.git Cloning into 'notebooks'... remote: Counting objects: 11050, done. remote: Compressing objects: 100% (10076/10076), done. remote: Total 11050 (delta 987), reused 11027 (delta 971), pack-reused 0 Receiving objects: 100% (11050/11050), 22.26 MiB | 5.51 MiB/s, done. Resolving deltas: 100% (987/987), done. Checking connectivity... done. Checking out files: 100% (10939/10939), done. mcclulou@DESKTOP-4KSDJ7D:~$ cd notebooks/ mcclulou@DESKTOP-4KSDJ7D:~/notebooks$ ls data images notebooks README.md requirements.txt utils mcclulou@DESKTOP-4KSDJ7D:~/notebooks$ ls notebooks/ ML01 Neural Networks.ipynb ML05 Decision Trees.ipynb ML02 Dimensionality Reduction.ipynb ML06 Random Forests.ipynb ML03 K-Nearest Neighbours.ipynb ML07 Natural Language Processing.ipynb ML04 Feature Engineering.ipynb git clone https://github.com/awilson-kx/notebooks.git
19. 19 jupyter notebook notebooks mcclulou@DESKTOP-4KSDJ7D:~/notebooks$ jupyter notebook notebooks [I 13:34:52.026 NotebookApp] JupyterLab beta preview extension loaded from /home/mcclulou/ anaconda3/lib/python3.6/site-packages/jupyterlab [I 13:34:52.028 NotebookApp] JupyterLab application directory is /home/mcclulou/anaconda3/ share/jupyter/lab [I 13:34:52.109 NotebookApp] Serving notebooks from local directory: /home/mcclulou/notebooks/ notebooks [I 13:34:52.111 NotebookApp] 0 active kernels [I 13:34:52.116 NotebookApp] The Jupyter Notebook is running at: [I 13:34:52.116 NotebookApp] http://localhost:8888/? token=ed0be711b11472d2db5cb6ba52f6f9d23ce6aeb0afb780d7 [I 13:34:52.117 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). [W 13:34:52.195 NotebookApp] No web browser found: could not locate runnable browser. [C 13:34:52.196 NotebookApp] Copy/paste this URL into your browser when you connect for the first time, to login with a token: http://localhost:8888/?token=ed0be711b11472d2db5cb6ba52f6f9d23ce6aeb0afb780d7
20. 20 jupyter notebook notebooks DEMO
21. 21 Kx TECHNOLOGY FUND Int. Office Access £10 Mil Funding Kx Tech Access Sales & Marketing Support Logistical Support Learn more at www.kx.com/kx-technology-fund/
22. 22 KX TECHNOLOGY FUND
23. 23 STREAM CONTROL ANALYST DASHBOARDS Dashboards for Kx provides rich visualization of real-time streaming, intraday and historical business data. Analyst For Kx provides a complete real- time data transformation, exploration and discovery workflow. Control for Kx allows organizations to manage their application landscape, instantly determining what processes run, who owns them, their parameters and functionality. Stream for Kx is a proven data management solution that can be deployed locally, in the cloud, or as part of a wider Big Data architecture. Enterprise Tools
24. 24 • Opportunities to travel and live internationally • Paid for city accommodation • Competitive salary and expenses package • Health and travel insurance programme • Generous retirement savings programme Recruitment Interested in joining us or know someone who might be? Talk to me at the end or visit www.firstderivatives.com/careers
25. 25 Louise McCluskey (Totten) Kdb+ Developer lmccluskey@kx.com https://kx.com/ https://kx.com/download/ http://code.kx.com/qref/ Contact Aimi McConnell (Forgan) Python Developer aforgan@firstderivatives.com