Upgrade to Pro — share decks privately, control downloads, hide ads and more …

embedPy, the newest interface between kdb+ and Python

embedPy, the newest interface between kdb+ and Python

By Aimi Forgan, kdb+ Engineer
Description: "embedPy, the newest interface between kdb+ and Python” followed by a live demo

Where: Dogpatch Labs, CHQ, IFSC, D1

https://www.meetup.com/PyLadiesDublin/events/dclgvlyxjbzb/

PyLadies Dublin

June 19, 2018
Tweet

More Decks by PyLadies Dublin

Other Decks in Technology

Transcript

  1. 1
    embedPy, the newest interface between kdb+ and Python
    PyLadies Dublin Meetup
    Louise Totten & Aimi McConnell
    19 June 2018

    View Slide

  2. 2
    • written for non-kdb+ programmers to use
    • well documented, with understandable and useful examples
    • maintained and supported by Kx on a best-efforts basis, at no cost to customers
    • released under the Apache 2 license
    • free for all use cases, including 64-bit and commercial use
    • embedPy is part of the Fusion for kdb+ interface collection
    Fusion interfaces to kdb+

    View Slide

  3. 3
    • Allows the kdb+ interpreter to manipulate Python objects and call Python functions.
    • Python functions can be called from q, and q functions can be called from Python.
    embedPy
    programmers
    q/kdb+ programmers
    embedPy
    embedPy gives direct access to data
    in kdb+
    embedPy offers easy access to the
    numerous computational and
    visualization libraries

    View Slide

  4. 4
    embedPy
    Requirements
    kdb+ ≥ v3.5 64-bit
    Anaconda Python 3.x
    macOS or Linux

    View Slide

  5. 5
    • Loads Python into kdb+, allowing access to a rich ecosystem of libraries such as:
    • Python variables and objects become q variables – and either language can act upon them
    • Python code and files can be embedded within q code
    • Python functions can be called as q functions
    embedPy

    View Slide

  6. 6
    Execute Python code directly in a q console or from a script, by prefixing the Python code
    with p)
    q)p)print(1+2)
    3
    You cannot run multi-line Python code directly in the console, but you can add to a q script
    using the syntax below.
    p) and subsequent lines indented according to usual Python indentation rules, for example:
    $cat testScript.q
    a:1
    p)def add1(arg1):
    return arg1+1
    embedPy

    View Slide

  7. 7
    To run a q script with embedded python code, simply load the script into your q session:
    q)\l testScript.q
    q)p)print(add1(12))
    13
    To run a Python script, rename the file from file.py to file.p and load in the same way as
    above:
    $cat helloq.p
    print(“Hello q!”)
    q)\l helloq.p
    Hello q!
    embedPy

    View Slide

  8. 8
    Jupyter kernel for kdb+. Features include
    • Syntax Highlighting for q
    • Code completion for q keywords, .z/.h/.Q/.j namespace functions, and user defined
    variables
    • Code help for q keywords and basic help (display and type information) for user defined
    objects
    • script like execution of code (mulit-line input)
    • Inline display of charts created using embedPy and matplotlib
    • console stdout/stderr capture and display in notebooks
    • Inline loading and saving of scripts into and from notebook cells
    jupyterq

    View Slide

  9. 9
    jupyterq
    Requirements
    kdb+ ≥ v3.5 64-bit
    Anaconda Python 3.x
    embedPy

    View Slide

  10. 10
    Machine-learning capabilities are at the heart of future
    technology development at Kx.
    Libraries are released under the Apache 2 license, and are
    free for all use cases, including 64-bit and commercial use.
    EmbedPy forms the basis of machine learning initiative at
    Kx
    Machine Learning

    View Slide

  11. 11
    Kx have added kdb, embedPy and jupyterq to the Anaconda Python distribution platform at
    https://anaconda.org/kx
    Setup kdb, embedPy and juypterq

    View Slide

  12. 12
    There are three packages that you can install from https://anaconda.org/kx
    • kdb (non-commercial personal edition, requires internet connection to work)
    • embedPy
    • jupyterq
    Setup kdb, embedPy and juypterq
    TIP: installing jupyterq will
    automatically install the embedPy and
    kdb packages

    View Slide

  13. 13
    For q 64 bit version you need to request a licence from: https://ondemand.kx.com/
    Setup q, embedPy and juypterq

    View Slide

  14. 14
    conda install -c kx jupyterq
    [email protected]:~$ conda install -c kx jupyterq
    Solving environment: done
    ## Package Plan ##
    environment location: /home/mcclulou/anaconda3
    added / updated specs:
    - jupyterq
    The following packages will be downloaded:
    package | build
    ---------------------------|-----------------
    certifi-2018.4.16 | py36_0 142 KB
    jupyterq-1.0.1 | py36_3 43 KB kx
    kdb-3.6 | 2 310 KB kx
    embedpy-1.1.2 | py36h14c3975_0 51 KB kx
    ------------------------------------------------------------
    Total: 547 KB
    The following NEW packages will be INSTALLED:
    embedpy: 1.1.2-py36h14c3975_0 kx
    jupyterq: 1.0.1-py36_3 kx
    kdb: 3.6-2 kx

    View Slide

  15. 15
    [email protected]:~$ git clone https://github.com/KxSystems/jupyterq.git
    Cloning into 'jupyterq'...
    remote: Counting objects: 193, done.
    remote: Compressing objects: 100% (38/38), done.
    remote: Total 193 (delta 30), reused 54 (delta 24), pack-reused 124
    Receiving objects: 100% (193/193), 691.92 KiB | 0 bytes/s, done.
    Resolving deltas: 100% (72/72), done.
    Checking connectivity... done.
    [email protected]:~$ ls
    anaconda3 jupyterq notebooks q tmp
    [email protected]:~$ cd jupyterq
    [email protected]:~/jupyterq$ ls
    appveyor.yml install.sh kdb+Notebooks.ipynb q_widgets.ipynb
    compile.bat jupyterq_b64.q kernelspec README.md
    configure.q jupyterq_help.q kxpy requirements.txt
    doc jupyterq_htmlgen.q lib src
    docker jupyterq_kernel.q LICENSE
    importmatplotlib.q jupyterq_pyzmq.q makefile.in
    install.bat jupyterq_server.q matplotlibexample.p
    git clone https://github.com/KxSystems/jupyterq.git

    View Slide

  16. 16
    jupyter notebook kdb+Notebooks.ipynb
    [email protected]:~/jupyterq$ jupyter notebook kdb+Notebooks.ipynb
    [I 13:17:44.242 NotebookApp] JupyterLab beta preview extension loaded from /home/mcclulou/
    anaconda3/lib/python3.6/site-packages/jupyterlab
    [I 13:17:44.244 NotebookApp] JupyterLab application directory is /home/mcclulou/anaconda3/
    share/jupyter/lab
    [I 13:17:44.306 NotebookApp] Serving notebooks from local directory: /home/mcclulou/jupyterq
    [I 13:17:44.308 NotebookApp] 0 active kernels
    [I 13:17:44.313 NotebookApp] The Jupyter Notebook is running at:
    [I 13:17:44.314 NotebookApp] http://localhost:8888/?
    token=5d2095f6a5dff27020ad9a3fddc88cec5335dd00ac02ab8b
    [I 13:17:44.315 NotebookApp] Use Control-C to stop this server and shut down all kernels
    (twice to skip confirmation).
    [W 13:17:44.323 NotebookApp] No web browser found: could not locate runnable browser.
    [C 13:17:44.323 NotebookApp]
    Copy/paste this URL into your browser when you connect for the first time,
    to login with a token:
    http://localhost:8888/?token=5d2095f6a5dff27020ad9a3fddc88cec5335dd00ac02ab8b

    View Slide

  17. 17
    kdb+Notebooks.ipynb

    View Slide

  18. 18
    [email protected]:~$ git clone https://github.com/awilson-kx/
    notebooks.git
    Cloning into 'notebooks'...
    remote: Counting objects: 11050, done.
    remote: Compressing objects: 100% (10076/10076), done.
    remote: Total 11050 (delta 987), reused 11027 (delta 971), pack-reused 0
    Receiving objects: 100% (11050/11050), 22.26 MiB | 5.51 MiB/s, done.
    Resolving deltas: 100% (987/987), done.
    Checking connectivity... done.
    Checking out files: 100% (10939/10939), done.
    [email protected]:~$ cd notebooks/
    [email protected]:~/notebooks$ ls
    data images notebooks README.md requirements.txt utils
    [email protected]:~/notebooks$ ls notebooks/
    ML01 Neural Networks.ipynb ML05 Decision Trees.ipynb
    ML02 Dimensionality Reduction.ipynb ML06 Random Forests.ipynb
    ML03 K-Nearest Neighbours.ipynb ML07 Natural Language Processing.ipynb
    ML04 Feature Engineering.ipynb
    git clone https://github.com/awilson-kx/notebooks.git

    View Slide

  19. 19
    jupyter notebook notebooks
    [email protected]:~/notebooks$ jupyter notebook notebooks
    [I 13:34:52.026 NotebookApp] JupyterLab beta preview extension loaded from /home/mcclulou/
    anaconda3/lib/python3.6/site-packages/jupyterlab
    [I 13:34:52.028 NotebookApp] JupyterLab application directory is /home/mcclulou/anaconda3/
    share/jupyter/lab
    [I 13:34:52.109 NotebookApp] Serving notebooks from local directory: /home/mcclulou/notebooks/
    notebooks
    [I 13:34:52.111 NotebookApp] 0 active kernels
    [I 13:34:52.116 NotebookApp] The Jupyter Notebook is running at:
    [I 13:34:52.116 NotebookApp] http://localhost:8888/?
    token=ed0be711b11472d2db5cb6ba52f6f9d23ce6aeb0afb780d7
    [I 13:34:52.117 NotebookApp] Use Control-C to stop this server and shut down all kernels
    (twice to skip confirmation).
    [W 13:34:52.195 NotebookApp] No web browser found: could not locate runnable browser.
    [C 13:34:52.196 NotebookApp]
    Copy/paste this URL into your browser when you connect for the first time,
    to login with a token:
    http://localhost:8888/?token=ed0be711b11472d2db5cb6ba52f6f9d23ce6aeb0afb780d7

    View Slide

  20. 20
    jupyter notebook notebooks
    DEMO

    View Slide

  21. 21
    Kx TECHNOLOGY FUND
    Int. Office
    Access
    £10 Mil
    Funding
    Kx Tech
    Access
    Sales &
    Marketing
    Support
    Logistical
    Support
    Learn more at www.kx.com/kx-technology-fund/

    View Slide

  22. 22
    KX TECHNOLOGY FUND

    View Slide

  23. 23
    STREAM
    CONTROL
    ANALYST
    DASHBOARDS
    Dashboards for Kx
    provides rich
    visualization of
    real-time
    streaming,
    intraday and
    historical business
    data.
    Analyst For Kx
    provides a
    complete real-
    time data
    transformation,
    exploration and
    discovery
    workflow.
    Control for Kx allows
    organizations to
    manage their
    application
    landscape, instantly
    determining what
    processes run, who
    owns them, their
    parameters and
    functionality.
    Stream for Kx is a
    proven data
    management
    solution that can be
    deployed locally, in
    the cloud, or as
    part of a wider Big
    Data architecture.
    Enterprise Tools

    View Slide

  24. 24
    • Opportunities to travel and live internationally
    • Paid for city accommodation
    • Competitive salary and expenses package
    • Health and travel insurance programme
    • Generous retirement savings programme
    Recruitment
    Interested in joining us or know someone who might be?
    Talk to me at the end or visit
    www.firstderivatives.com/careers

    View Slide

  25. 25
    Louise McCluskey (Totten)
    Kdb+ Developer
    [email protected]
    https://kx.com/
    https://kx.com/download/
    http://code.kx.com/qref/
    Contact
    Aimi McConnell (Forgan)
    Python Developer
    [email protected]

    View Slide