Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Integrating Jython with Java by Jim Baker and Shashank Bharadwaj

Integrating Jython with Java by Jim Baker and Shashank Bharadwaj

PyCon 2013

March 17, 2013
Tweet

More Decks by PyCon 2013

Other Decks in Programming

Transcript

  1. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    Integrating Jython with Java
    Jim Baker Shashank Bharadwaj
    March 16, 2013

    View Slide

  2. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    Jython status
    What’s new in Jython
    Jython 2.7 beta 1 just released!
    We will be sprinting on Jython 2.7 during the sprints

    View Slide

  3. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    Why Jython?
    Very compliant implementation of Python:
    Jython 2.5 tracked CPython 2.5
    Jython 2.7 tracks CPython 2.7
    Java ecosystem is huge and growing
    This size is even beneficial
    Pick the domain, there’s likely to be a library for it
    Some of them are quite good

    View Slide

  4. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    Why not Jython?
    No C extension API, ctypes, Cython, etc
    Certainly possible however

    View Slide

  5. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    Other possibilities
    But first, other integrations are possible
    JEPP - embed CPython within the JVM using JNI
    JPype - embed JVM within CPython using JNI
    Bridging between CPython and Java using interprocess
    communication
    Or using IPC between CPython and Jython

    View Slide

  6. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    Example: execnet
    Bridges CPython and Jython efficiently:
    import execnet
    gw = execnet.makegateway("popen//python=jython")
    channel = gw.remote_exec("""
    from java.util import Vector
    v = Vector()
    v.add(’aaa’)
    v.add(’bbb’)
    for val in v:
    channel.send(val)
    """)
    for item in channel:
    print (item)

    View Slide

  7. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    Issues with alternatives
    But the integrations are very shallow:
    Callbacks
    Extending Java classes in Python
    etc etc

    View Slide

  8. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    Jython compatibility
    High degree of compatibility, with the regrtest to prove it
    Keep in mind: different GC model which impacts
    finalization/bad code
    No GIL of course
    And threading is currently efficient!
    Sequential consistency (* almost)

    View Slide

  9. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    Talk examples
    Metacircular approach: use as examples snippets from
    Jython tests and libraries
    Models CPython - tests and libraries use Python too
    Tests cover lots of corner cases
    Testing Jython with Jython itself is a very pleasant way to
    test
    Occasional disadvantages, such as requiring a substantial
    amount of the language and runtime be availableh
    Aside: Jython could see more unit tests of the runtime

    View Slide

  10. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    Java from Jython
    Python code can directly import Java classes, as if they
    were Python classes
    Construct Java objects from Java classes, as if they were
    Python classes
    Work with Java objects. Your programs can call Java
    methods, or have Java call back into Python.
    And this “magic”/integration works because we have built
    a proxy on the fly

    View Slide

  11. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    Apache POI: Read and write MS Office docs
    from contextlib import closing
    from org.apache.poi.hssf.usermodel import *
    from java.io import FileInputStream
    with closing(FileInputStream("data.xls")) as fis, \
    closing(HSSFWorkbook(fis)) as wb:
    sheet = wb.getSheetAt(0)
    num_rows = sheet.getPhysicalNumberOfRows()
    for i in xrange(num_rows):
    row = sheet.getRow(i)
    if row:
    num_cols = sheet.getRow(i).\
    getPhysicalNumberOfCells()
    for j in xrange(num_cols):
    cell = row.getCell(j)
    # then do something interesting!!!

    View Slide

  12. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    Python and Java type equivalence
    Java type Python type
    char str
    boolean bool
    byte, short, int, long int
    java.lang.String unicode
    byte[], char[] str
    java.lang.Class JavaClass
    x[] array.array
    org.python.core.PyObject unchanged
    Foo PyJavaType proxying Foo

    View Slide

  13. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    Achieving equivalence
    Jython uses the following methods
    tojava
    java2py
    Usually used transparently
    Sometimes significant overhead because of
    boxing/unboxing
    Not always because boxing/unboxing is a common idiom
    in Java, so JVMs can often optimize this away (*
    significant but it depends!)

    View Slide

  14. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    Example: datetime.py
    if _sys.platform.startswith(’java’):
    def __tojava__(self, java_class):
    if java_class not in (Calendar, Date, Object):
    return Py.NoConversion
    calendar = Calendar.getInstance()
    calendar.clear()
    calendar.set(
    self.year, self.month - 1, self.day)
    if java_class == Calendar:
    return calendar
    else:
    return Date(calendar.getTimeInMillis())

    View Slide

  15. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    Use of overloaded methods
    What happens when you call overloaded methods? How
    does Jython pick the one you want?
    Sometimes the runtime picks wrong
    Force it by using constructors from java.lang - Boolean,
    Integer, Double, etc

    View Slide

  16. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    JSR-223: Scripting for the Java platform
    Part of the Jython release since 2.5.1
    JSR 223 is intentionally shallow - common operations we
    expect to see in scripting scenarios
    Can still be incredibly effective in driving Jython at the top
    level
    Just like we would expect in a good script

    View Slide

  17. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    Example code
    public void testEvalString()
    throws ScriptException {
    ScriptEngineManager manager =
    new ScriptEngineManager();
    ScriptEngine pythonEngine =
    manager.getEngineByName("python");
    assertNull(pythonEngine.eval("x = 5"));
    assertEquals(Integer.valueOf(5),
    pythonEngine.eval("x"));
    }
    Source:
    /tests/java/org/python/jsr223/ScriptEngineTest.java

    View Slide

  18. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    Jython, testing Jython
    You can even test JSR 223 support from Jython itself:
    def test_factory(self):
    engine = ScriptEngineManager().\
    getEngineByName("python")
    f = engine.factory
    language_version = ".".join(str(comp)
    for comp in sys.version_info[0:2])
    impl_version = ".".join(str(comp)
    for comp in sys.version_info[0:3])
    self.assertNotEqual(f.scriptEngine, engine)
    self.assertEqual(f.engineName, "jython")
    self.assertEqual(f.engineVersion, impl_version)
    self.assertEqual(set(f.extensions), set([’py’]))
    self.assertEqual(f.languageName, "python")

    View Slide

  19. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    And more such tests
    # variants
    self.assertEqual(
    f.getParameter(ScriptEngine.ENGINE),
    "jython")
    self.assertEqual(
    f.getParameter(ScriptEngine.ENGINE_VERSION),
    impl_version)
    self.assertEqual(
    f.getParameter(ScriptEngine.LANGUAGE_VERSION),
    language_version)
    self.assertEqual(
    f.getOutputStatement("abc"),
    "print u’abc’")
    self.assertEqual(
    f.getProgram("x = 42", "y = ’abc’"),
    "x = 42\ny = ’abc’\n")

    View Slide

  20. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    Or perhaps not
    But not without issues, as we found in our own usage:
    # XXX Earlier version of this test also tested
    # put, get, eval on the engine, however this
    # introduced action at a distance where aspects
    # of the sys state changed
    # (notably sys.stdin.newlines), which then
    # impacted test_univnewlines later in the regrtest.
    #
    # For now, there may be limits in how much
    # we can test Jython from itself, no matter
    # how attractive from an ouroboros perspective that
    # may be :). Certainly worth revisiting in 2.6.
    (Or later as the case may be.)

    View Slide

  21. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    Polyglot programming, from Groovy
    engine = mgr.getEngineByName("jython")
    engine.eval(’’’
    def factorial(n):
    if i == 0:
    return 1
    else:
    return n * factorial(n - 1)
    result = factorial(4)
    ’’’)
    println ’jython: ’ + engine.result
    Slightly modified example from
    http://groovy.codehaus.org/JSR-
    223+access+to+other+JVM+languages

    View Slide

  22. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    Callbacks
    Callbacks are commonly used in Java
    Generally requires a fair amount of boilerplate code, even
    with anoymous classes
    And generally you need to pass in some state

    View Slide

  23. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    Using closures instead
    Java methods that take objects of single method interfaces
    Useful for working with Callable or Runnable interfaces
    Can pass in a lexical closure, that is a function that closes
    over variables in its lexical scope

    View Slide

  24. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    How does that actually work?
    We need to support Python -> Java -> Python
    For lexically closed variables, you are actually using a
    closure
    For globally scoped variables, Python can look up the right
    state referenced by a thread local, transparent to the using
    Python code

    View Slide

  25. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    Using Python code from Java: Object factories
    Java code can use the Jython runtime to construct Python
    objects
    Object factories use Jython-specific APIs to efficiently
    build Python objects

    View Slide

  26. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    Example: Use the runtime
    Use the Python runtime to get a PyObject for the type of
    interest
    public class SomeClassFactory {
    private PyObject someClass;
    public SomeClassFactory() {
    PythonInterpreter interpreter = new PythonInte
    interpreter.exec("from something import SomeCl
    someClass = interpreter.get("SomeClass");
    }

    View Slide

  27. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    Construct away!
    public SomeClassType create (String x, String y) {
    PyObject someObject =
    someClass.__call__(
    new PyString(x), new PyString(y))
    return (SomeClassType)someObject.__tojava__(
    someClassType.class);
    }
    }

    View Slide

  28. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    Gradual Typing
    A new type system proposed by Jeremy Siek & Walid
    Taha; extended by others
    Seamless integration between statically typed parts and
    dynamically typed parts of code in the language
    Guarantees that statically typed part (after type checking)
    will not raise type error at runtime.

    View Slide

  29. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    Gradual Typing for Python
    gradual module for Python 3.x
    Provides type checking based on user defined type
    annotations
    Uses function annotations and decoraters

    View Slide

  30. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    Gradual Typing for Python - Example
    from gradual import typed
    @typed
    class YourAwesomeApi(object):
    def answerToLife(self, a: int, b: int) -> int:
    return a + b
    def use_api(x):
    print(x.answerToLife(10, 32))
    print(x.answerToLife(’hello’, ’there’))
    # TypeError: Expected int, got:
    # ’hello’ of type str
    use_api(YourAwesomeApi())

    View Slide

  31. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    Gradual Typing for Python: Work in Progress
    Prototype - in experimental mode!
    pip install gradual to get it
    Repo at: http://bitbucket.org/gradual/py-gradual
    Working on accurate representation of objects
    Working on Blame Tracking to produce accurate error
    messages

    View Slide

  32. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    Future: Integrate Gradual Typing into Jython
    Use Gradual Typing to enable typed-Jython code
    Enable better Java integration
    type information can be used to remove wrappers in Java
    code
    reduce the proxies involved
    Enable optimizations
    Drive type information to generate idiomatic Java code
    (where applicable)

    View Slide

  33. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    Questions
    Thank You!
    Jim Baker Shashank Bharadwaj
    [email protected] [email protected]
    @jimbaker @ shanka

    View Slide