Integrating Jython with Java by Jim Baker and Shashank Bharadwaj

Transcript

1. Integrating
   Jython with
   Java
   Jim Baker,
   Shashank
   Bharadwaj
   Integrating Jython with Java
   Jim Baker Shashank Bharadwaj
   March 16, 2013
   

   View Slide

2. Integrating
   Jython with
   Java
   Jim Baker,
   Shashank
   Bharadwaj
   Jython status
   What’s new in Jython
   Jython 2.7 beta 1 just released!
   We will be sprinting on Jython 2.7 during the sprints
   

   View Slide

3. Integrating
   Jython with
   Java
   Jim Baker,
   Shashank
   Bharadwaj
   Why Jython?
   Very compliant implementation of Python:
   Jython 2.5 tracked CPython 2.5
   Jython 2.7 tracks CPython 2.7
   Java ecosystem is huge and growing
   This size is even beneficial
   Pick the domain, there’s likely to be a library for it
   Some of them are quite good
   

   View Slide

4. Integrating
   Jython with
   Java
   Jim Baker,
   Shashank
   Bharadwaj
   Why not Jython?
   No C extension API, ctypes, Cython, etc
   Certainly possible however
   

   View Slide

5. Integrating
   Jython with
   Java
   Jim Baker,
   Shashank
   Bharadwaj
   Other possibilities
   But first, other integrations are possible
   JEPP - embed CPython within the JVM using JNI
   JPype - embed JVM within CPython using JNI
   Bridging between CPython and Java using interprocess
   communication
   Or using IPC between CPython and Jython
   

   View Slide

6. Integrating
   Jython with
   Java
   Jim Baker,
   Shashank
   Bharadwaj
   Example: execnet
   Bridges CPython and Jython efficiently:
   import execnet
   gw = execnet.makegateway("popen//python=jython")
   channel = gw.remote_exec("""
   from java.util import Vector
   v = Vector()
   v.add(’aaa’)
   v.add(’bbb’)
   for val in v:
   channel.send(val)
   """)
   for item in channel:
   print (item)
   

   View Slide

7. Integrating
   Jython with
   Java
   Jim Baker,
   Shashank
   Bharadwaj
   Issues with alternatives
   But the integrations are very shallow:
   Callbacks
   Extending Java classes in Python
   etc etc
   

   View Slide

8. Integrating
   Jython with
   Java
   Jim Baker,
   Shashank
   Bharadwaj
   Jython compatibility
   High degree of compatibility, with the regrtest to prove it
   Keep in mind: different GC model which impacts
   finalization/bad code
   No GIL of course
   And threading is currently efficient!
   Sequential consistency (* almost)
   

   View Slide

9. Integrating
   Jython with
   Java
   Jim Baker,
   Shashank
   Bharadwaj
   Talk examples
   Metacircular approach: use as examples snippets from
   Jython tests and libraries
   Models CPython - tests and libraries use Python too
   Tests cover lots of corner cases
   Testing Jython with Jython itself is a very pleasant way to
   test
   Occasional disadvantages, such as requiring a substantial
   amount of the language and runtime be availableh
   Aside: Jython could see more unit tests of the runtime
   

   View Slide

10. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    Java from Jython
    Python code can directly import Java classes, as if they
    were Python classes
    Construct Java objects from Java classes, as if they were
    Python classes
    Work with Java objects. Your programs can call Java
    methods, or have Java call back into Python.
    And this “magic”/integration works because we have built
    a proxy on the fly
    

    View Slide

11. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    Apache POI: Read and write MS Office docs
    from contextlib import closing
    from org.apache.poi.hssf.usermodel import *
    from java.io import FileInputStream
    with closing(FileInputStream("data.xls")) as fis, \
    closing(HSSFWorkbook(fis)) as wb:
    sheet = wb.getSheetAt(0)
    num_rows = sheet.getPhysicalNumberOfRows()
    for i in xrange(num_rows):
    row = sheet.getRow(i)
    if row:
    num_cols = sheet.getRow(i).\
    getPhysicalNumberOfCells()
    for j in xrange(num_cols):
    cell = row.getCell(j)
    # then do something interesting!!!
    

    View Slide

12. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    Python and Java type equivalence
    Java type Python type
    char str
    boolean bool
    byte, short, int, long int
    java.lang.String unicode
    byte[], char[] str
    java.lang.Class JavaClass
    x[] array.array
    org.python.core.PyObject unchanged
    Foo PyJavaType proxying Foo
    

    View Slide

13. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    Achieving equivalence
    Jython uses the following methods
    tojava
    java2py
    Usually used transparently
    Sometimes significant overhead because of
    boxing/unboxing
    Not always because boxing/unboxing is a common idiom
    in Java, so JVMs can often optimize this away (*
    significant but it depends!)
    

    View Slide

14. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    Example: datetime.py
    if _sys.platform.startswith(’java’):
    def __tojava__(self, java_class):
    if java_class not in (Calendar, Date, Object):
    return Py.NoConversion
    calendar = Calendar.getInstance()
    calendar.clear()
    calendar.set(
    self.year, self.month - 1, self.day)
    if java_class == Calendar:
    return calendar
    else:
    return Date(calendar.getTimeInMillis())
    

    View Slide

15. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    Use of overloaded methods
    What happens when you call overloaded methods? How
    does Jython pick the one you want?
    Sometimes the runtime picks wrong
    Force it by using constructors from java.lang - Boolean,
    Integer, Double, etc
    

    View Slide

16. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    JSR-223: Scripting for the Java platform
    Part of the Jython release since 2.5.1
    JSR 223 is intentionally shallow - common operations we
    expect to see in scripting scenarios
    Can still be incredibly effective in driving Jython at the top
    level
    Just like we would expect in a good script
    

    View Slide

17. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    Example code
    public void testEvalString()
    throws ScriptException {
    ScriptEngineManager manager =
    new ScriptEngineManager();
    ScriptEngine pythonEngine =
    manager.getEngineByName("python");
    assertNull(pythonEngine.eval("x = 5"));
    assertEquals(Integer.valueOf(5),
    pythonEngine.eval("x"));
    }
    Source:
    /tests/java/org/python/jsr223/ScriptEngineTest.java
    

    View Slide

18. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    Jython, testing Jython
    You can even test JSR 223 support from Jython itself:
    def test_factory(self):
    engine = ScriptEngineManager().\
    getEngineByName("python")
    f = engine.factory
    language_version = ".".join(str(comp)
    for comp in sys.version_info[0:2])
    impl_version = ".".join(str(comp)
    for comp in sys.version_info[0:3])
    self.assertNotEqual(f.scriptEngine, engine)
    self.assertEqual(f.engineName, "jython")
    self.assertEqual(f.engineVersion, impl_version)
    self.assertEqual(set(f.extensions), set([’py’]))
    self.assertEqual(f.languageName, "python")
    

    View Slide

19. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    And more such tests
    # variants
    self.assertEqual(
    f.getParameter(ScriptEngine.ENGINE),
    "jython")
    self.assertEqual(
    f.getParameter(ScriptEngine.ENGINE_VERSION),
    impl_version)
    self.assertEqual(
    f.getParameter(ScriptEngine.LANGUAGE_VERSION),
    language_version)
    self.assertEqual(
    f.getOutputStatement("abc"),
    "print u’abc’")
    self.assertEqual(
    f.getProgram("x = 42", "y = ’abc’"),
    "x = 42\ny = ’abc’\n")
    

    View Slide

20. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    Or perhaps not
    But not without issues, as we found in our own usage:
    # XXX Earlier version of this test also tested
    # put, get, eval on the engine, however this
    # introduced action at a distance where aspects
    # of the sys state changed
    # (notably sys.stdin.newlines), which then
    # impacted test_univnewlines later in the regrtest.
    #
    # For now, there may be limits in how much
    # we can test Jython from itself, no matter
    # how attractive from an ouroboros perspective that
    # may be :). Certainly worth revisiting in 2.6.
    (Or later as the case may be.)
    

    View Slide

21. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    Polyglot programming, from Groovy
    engine = mgr.getEngineByName("jython")
    engine.eval(’’’
    def factorial(n):
    if i == 0:
    return 1
    else:
    return n * factorial(n - 1)
    result = factorial(4)
    ’’’)
    println ’jython: ’ + engine.result
    Slightly modified example from
    http://groovy.codehaus.org/JSR-
    223+access+to+other+JVM+languages
    

    View Slide

22. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    Callbacks
    Callbacks are commonly used in Java
    Generally requires a fair amount of boilerplate code, even
    with anoymous classes
    And generally you need to pass in some state
    

    View Slide

23. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    Using closures instead
    Java methods that take objects of single method interfaces
    Useful for working with Callable or Runnable interfaces
    Can pass in a lexical closure, that is a function that closes
    over variables in its lexical scope
    

    View Slide

24. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    How does that actually work?
    We need to support Python -> Java -> Python
    For lexically closed variables, you are actually using a
    closure
    For globally scoped variables, Python can look up the right
    state referenced by a thread local, transparent to the using
    Python code
    

    View Slide

25. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    Using Python code from Java: Object factories
    Java code can use the Jython runtime to construct Python
    objects
    Object factories use Jython-specific APIs to efficiently
    build Python objects
    

    View Slide

26. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    Example: Use the runtime
    Use the Python runtime to get a PyObject for the type of
    interest
    public class SomeClassFactory {
    private PyObject someClass;
    public SomeClassFactory() {
    PythonInterpreter interpreter = new PythonInte
    interpreter.exec("from something import SomeCl
    someClass = interpreter.get("SomeClass");
    }
    

    View Slide

27. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    Construct away!
    public SomeClassType create (String x, String y) {
    PyObject someObject =
    someClass.__call__(
    new PyString(x), new PyString(y))
    return (SomeClassType)someObject.__tojava__(
    someClassType.class);
    }
    }
    

    View Slide

28. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    Gradual Typing
    A new type system proposed by Jeremy Siek & Walid
    Taha; extended by others
    Seamless integration between statically typed parts and
    dynamically typed parts of code in the language
    Guarantees that statically typed part (after type checking)
    will not raise type error at runtime.
    

    View Slide

29. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    Gradual Typing for Python
    gradual module for Python 3.x
    Provides type checking based on user defined type
    annotations
    Uses function annotations and decoraters
    

    View Slide

30. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    Gradual Typing for Python - Example
    from gradual import typed
    @typed
    class YourAwesomeApi(object):
    def answerToLife(self, a: int, b: int) -> int:
    return a + b
    def use_api(x):
    print(x.answerToLife(10, 32))
    print(x.answerToLife(’hello’, ’there’))
    # TypeError: Expected int, got:
    # ’hello’ of type str
    use_api(YourAwesomeApi())
    

    View Slide

31. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    Gradual Typing for Python: Work in Progress
    Prototype - in experimental mode!
    pip install gradual to get it
    Repo at: http://bitbucket.org/gradual/py-gradual
    Working on accurate representation of objects
    Working on Blame Tracking to produce accurate error
    messages
    

    View Slide

32. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    Future: Integrate Gradual Typing into Jython
    Use Gradual Typing to enable typed-Jython code
    Enable better Java integration
    type information can be used to remove wrappers in Java
    code
    reduce the proxies involved
    Enable optimizations
    Drive type information to generate idiomatic Java code
    (where applicable)
    

    View Slide

33. Integrating
    Jython with
    Java
    Jim Baker,
    Shashank
    Bharadwaj
    Questions
    Thank You!
    Jim Baker Shashank Bharadwaj
    jbaker@zyasoft.com shashank.bharadwaj@colorado.edu
    @jimbaker @ shanka
    

    View Slide