Slide 1

Slide 1 text

Integrating Jython with Java Jim Baker, Shashank Bharadwaj Integrating Jython with Java Jim Baker Shashank Bharadwaj March 16, 2013

Slide 2

Slide 2 text

Integrating Jython with Java Jim Baker, Shashank Bharadwaj Jython status What’s new in Jython Jython 2.7 beta 1 just released! We will be sprinting on Jython 2.7 during the sprints

Slide 3

Slide 3 text

Integrating Jython with Java Jim Baker, Shashank Bharadwaj Why Jython? Very compliant implementation of Python: Jython 2.5 tracked CPython 2.5 Jython 2.7 tracks CPython 2.7 Java ecosystem is huge and growing This size is even beneficial Pick the domain, there’s likely to be a library for it Some of them are quite good

Slide 4

Slide 4 text

Integrating Jython with Java Jim Baker, Shashank Bharadwaj Why not Jython? No C extension API, ctypes, Cython, etc Certainly possible however

Slide 5

Slide 5 text

Integrating Jython with Java Jim Baker, Shashank Bharadwaj Other possibilities But first, other integrations are possible JEPP - embed CPython within the JVM using JNI JPype - embed JVM within CPython using JNI Bridging between CPython and Java using interprocess communication Or using IPC between CPython and Jython

Slide 6

Slide 6 text

Integrating Jython with Java Jim Baker, Shashank Bharadwaj Example: execnet Bridges CPython and Jython efficiently: import execnet gw = execnet.makegateway("popen//python=jython") channel = gw.remote_exec(""" from java.util import Vector v = Vector() v.add(’aaa’) v.add(’bbb’) for val in v: channel.send(val) """) for item in channel: print (item)

Slide 7

Slide 7 text

Integrating Jython with Java Jim Baker, Shashank Bharadwaj Issues with alternatives But the integrations are very shallow: Callbacks Extending Java classes in Python etc etc

Slide 8

Slide 8 text

Integrating Jython with Java Jim Baker, Shashank Bharadwaj Jython compatibility High degree of compatibility, with the regrtest to prove it Keep in mind: different GC model which impacts finalization/bad code No GIL of course And threading is currently efficient! Sequential consistency (* almost)

Slide 9

Slide 9 text

Integrating Jython with Java Jim Baker, Shashank Bharadwaj Talk examples Metacircular approach: use as examples snippets from Jython tests and libraries Models CPython - tests and libraries use Python too Tests cover lots of corner cases Testing Jython with Jython itself is a very pleasant way to test Occasional disadvantages, such as requiring a substantial amount of the language and runtime be availableh Aside: Jython could see more unit tests of the runtime

Slide 10

Slide 10 text

Integrating Jython with Java Jim Baker, Shashank Bharadwaj Java from Jython Python code can directly import Java classes, as if they were Python classes Construct Java objects from Java classes, as if they were Python classes Work with Java objects. Your programs can call Java methods, or have Java call back into Python. And this “magic”/integration works because we have built a proxy on the fly

Slide 11

Slide 11 text

Integrating Jython with Java Jim Baker, Shashank Bharadwaj Apache POI: Read and write MS Office docs from contextlib import closing from org.apache.poi.hssf.usermodel import * from java.io import FileInputStream with closing(FileInputStream("data.xls")) as fis, \ closing(HSSFWorkbook(fis)) as wb: sheet = wb.getSheetAt(0) num_rows = sheet.getPhysicalNumberOfRows() for i in xrange(num_rows): row = sheet.getRow(i) if row: num_cols = sheet.getRow(i).\ getPhysicalNumberOfCells() for j in xrange(num_cols): cell = row.getCell(j) # then do something interesting!!!

Slide 12

Slide 12 text

Integrating Jython with Java Jim Baker, Shashank Bharadwaj Python and Java type equivalence Java type Python type char str boolean bool byte, short, int, long int java.lang.String unicode byte[], char[] str java.lang.Class JavaClass x[] array.array org.python.core.PyObject unchanged Foo PyJavaType proxying Foo

Slide 13

Slide 13 text

Integrating Jython with Java Jim Baker, Shashank Bharadwaj Achieving equivalence Jython uses the following methods tojava java2py Usually used transparently Sometimes significant overhead because of boxing/unboxing Not always because boxing/unboxing is a common idiom in Java, so JVMs can often optimize this away (* significant but it depends!)

Slide 14

Slide 14 text

Integrating Jython with Java Jim Baker, Shashank Bharadwaj Example: datetime.py if _sys.platform.startswith(’java’): def __tojava__(self, java_class): if java_class not in (Calendar, Date, Object): return Py.NoConversion calendar = Calendar.getInstance() calendar.clear() calendar.set( self.year, self.month - 1, self.day) if java_class == Calendar: return calendar else: return Date(calendar.getTimeInMillis())

Slide 15

Slide 15 text

Integrating Jython with Java Jim Baker, Shashank Bharadwaj Use of overloaded methods What happens when you call overloaded methods? How does Jython pick the one you want? Sometimes the runtime picks wrong Force it by using constructors from java.lang - Boolean, Integer, Double, etc

Slide 16

Slide 16 text

Integrating Jython with Java Jim Baker, Shashank Bharadwaj JSR-223: Scripting for the Java platform Part of the Jython release since 2.5.1 JSR 223 is intentionally shallow - common operations we expect to see in scripting scenarios Can still be incredibly effective in driving Jython at the top level Just like we would expect in a good script

Slide 17

Slide 17 text

Integrating Jython with Java Jim Baker, Shashank Bharadwaj Example code public void testEvalString() throws ScriptException { ScriptEngineManager manager = new ScriptEngineManager(); ScriptEngine pythonEngine = manager.getEngineByName("python"); assertNull(pythonEngine.eval("x = 5")); assertEquals(Integer.valueOf(5), pythonEngine.eval("x")); } Source: /tests/java/org/python/jsr223/ScriptEngineTest.java

Slide 18

Slide 18 text

Integrating Jython with Java Jim Baker, Shashank Bharadwaj Jython, testing Jython You can even test JSR 223 support from Jython itself: def test_factory(self): engine = ScriptEngineManager().\ getEngineByName("python") f = engine.factory language_version = ".".join(str(comp) for comp in sys.version_info[0:2]) impl_version = ".".join(str(comp) for comp in sys.version_info[0:3]) self.assertNotEqual(f.scriptEngine, engine) self.assertEqual(f.engineName, "jython") self.assertEqual(f.engineVersion, impl_version) self.assertEqual(set(f.extensions), set([’py’])) self.assertEqual(f.languageName, "python")

Slide 19

Slide 19 text

Integrating Jython with Java Jim Baker, Shashank Bharadwaj And more such tests # variants self.assertEqual( f.getParameter(ScriptEngine.ENGINE), "jython") self.assertEqual( f.getParameter(ScriptEngine.ENGINE_VERSION), impl_version) self.assertEqual( f.getParameter(ScriptEngine.LANGUAGE_VERSION), language_version) self.assertEqual( f.getOutputStatement("abc"), "print u’abc’") self.assertEqual( f.getProgram("x = 42", "y = ’abc’"), "x = 42\ny = ’abc’\n")

Slide 20

Slide 20 text

Integrating Jython with Java Jim Baker, Shashank Bharadwaj Or perhaps not But not without issues, as we found in our own usage: # XXX Earlier version of this test also tested # put, get, eval on the engine, however this # introduced action at a distance where aspects # of the sys state changed # (notably sys.stdin.newlines), which then # impacted test_univnewlines later in the regrtest. # # For now, there may be limits in how much # we can test Jython from itself, no matter # how attractive from an ouroboros perspective that # may be :). Certainly worth revisiting in 2.6. (Or later as the case may be.)

Slide 21

Slide 21 text

Integrating Jython with Java Jim Baker, Shashank Bharadwaj Polyglot programming, from Groovy engine = mgr.getEngineByName("jython") engine.eval(’’’ def factorial(n): if i == 0: return 1 else: return n * factorial(n - 1) result = factorial(4) ’’’) println ’jython: ’ + engine.result Slightly modified example from http://groovy.codehaus.org/JSR- 223+access+to+other+JVM+languages

Slide 22

Slide 22 text

Integrating Jython with Java Jim Baker, Shashank Bharadwaj Callbacks Callbacks are commonly used in Java Generally requires a fair amount of boilerplate code, even with anoymous classes And generally you need to pass in some state

Slide 23

Slide 23 text

Integrating Jython with Java Jim Baker, Shashank Bharadwaj Using closures instead Java methods that take objects of single method interfaces Useful for working with Callable or Runnable interfaces Can pass in a lexical closure, that is a function that closes over variables in its lexical scope

Slide 24

Slide 24 text

Integrating Jython with Java Jim Baker, Shashank Bharadwaj How does that actually work? We need to support Python -> Java -> Python For lexically closed variables, you are actually using a closure For globally scoped variables, Python can look up the right state referenced by a thread local, transparent to the using Python code

Slide 25

Slide 25 text

Integrating Jython with Java Jim Baker, Shashank Bharadwaj Using Python code from Java: Object factories Java code can use the Jython runtime to construct Python objects Object factories use Jython-specific APIs to efficiently build Python objects

Slide 26

Slide 26 text

Integrating Jython with Java Jim Baker, Shashank Bharadwaj Example: Use the runtime Use the Python runtime to get a PyObject for the type of interest public class SomeClassFactory { private PyObject someClass; public SomeClassFactory() { PythonInterpreter interpreter = new PythonInte interpreter.exec("from something import SomeCl someClass = interpreter.get("SomeClass"); }

Slide 27

Slide 27 text

Integrating Jython with Java Jim Baker, Shashank Bharadwaj Construct away! public SomeClassType create (String x, String y) { PyObject someObject = someClass.__call__( new PyString(x), new PyString(y)) return (SomeClassType)someObject.__tojava__( someClassType.class); } }

Slide 28

Slide 28 text

Integrating Jython with Java Jim Baker, Shashank Bharadwaj Gradual Typing A new type system proposed by Jeremy Siek & Walid Taha; extended by others Seamless integration between statically typed parts and dynamically typed parts of code in the language Guarantees that statically typed part (after type checking) will not raise type error at runtime.

Slide 29

Slide 29 text

Integrating Jython with Java Jim Baker, Shashank Bharadwaj Gradual Typing for Python gradual module for Python 3.x Provides type checking based on user defined type annotations Uses function annotations and decoraters

Slide 30

Slide 30 text

Integrating Jython with Java Jim Baker, Shashank Bharadwaj Gradual Typing for Python - Example from gradual import typed @typed class YourAwesomeApi(object): def answerToLife(self, a: int, b: int) -> int: return a + b def use_api(x): print(x.answerToLife(10, 32)) print(x.answerToLife(’hello’, ’there’)) # TypeError: Expected int, got: # ’hello’ of type str use_api(YourAwesomeApi())

Slide 31

Slide 31 text

Integrating Jython with Java Jim Baker, Shashank Bharadwaj Gradual Typing for Python: Work in Progress Prototype - in experimental mode! pip install gradual to get it Repo at: http://bitbucket.org/gradual/py-gradual Working on accurate representation of objects Working on Blame Tracking to produce accurate error messages

Slide 32

Slide 32 text

Integrating Jython with Java Jim Baker, Shashank Bharadwaj Future: Integrate Gradual Typing into Jython Use Gradual Typing to enable typed-Jython code Enable better Java integration type information can be used to remove wrappers in Java code reduce the proxies involved Enable optimizations Drive type information to generate idiomatic Java code (where applicable)

Slide 33

Slide 33 text

Integrating Jython with Java Jim Baker, Shashank Bharadwaj Questions Thank You! Jim Baker Shashank Bharadwaj [email protected] [email protected] @jimbaker @ shanka