Integrating Jython with Java by Jim Baker and Shashank Bharadwaj

Transcript

1. Integrating Jython with Java Jim Baker, Shashank Bharadwaj Integrating Jython with Java Jim Baker Shashank Bharadwaj March 16, 2013
2. Integrating Jython with Java Jim Baker, Shashank Bharadwaj Jython status What’s new in Jython Jython 2.7 beta 1 just released! We will be sprinting on Jython 2.7 during the sprints
3. Integrating Jython with Java Jim Baker, Shashank Bharadwaj Why Jython? Very compliant implementation of Python: Jython 2.5 tracked CPython 2.5 Jython 2.7 tracks CPython 2.7 Java ecosystem is huge and growing This size is even beneficial Pick the domain, there’s likely to be a library for it Some of them are quite good
4. Integrating Jython with Java Jim Baker, Shashank Bharadwaj Why not Jython? No C extension API, ctypes, Cython, etc Certainly possible however
5. Integrating Jython with Java Jim Baker, Shashank Bharadwaj Other possibilities But first, other integrations are possible JEPP - embed CPython within the JVM using JNI JPype - embed JVM within CPython using JNI Bridging between CPython and Java using interprocess communication Or using IPC between CPython and Jython
6. Integrating Jython with Java Jim Baker, Shashank Bharadwaj Example: execnet Bridges CPython and Jython efficiently: import execnet gw = execnet.makegateway("popen//python=jython") channel = gw.remote_exec(""" from java.util import Vector v = Vector() v.add(’aaa’) v.add(’bbb’) for val in v: channel.send(val) """) for item in channel: print (item)
7. Integrating Jython with Java Jim Baker, Shashank Bharadwaj Issues with alternatives But the integrations are very shallow: Callbacks Extending Java classes in Python etc etc
8. Integrating Jython with Java Jim Baker, Shashank Bharadwaj Jython compatibility High degree of compatibility, with the regrtest to prove it Keep in mind: different GC model which impacts finalization/bad code No GIL of course And threading is currently efficient! Sequential consistency (* almost)
9. Integrating Jython with Java Jim Baker, Shashank Bharadwaj Talk examples Metacircular approach: use as examples snippets from Jython tests and libraries Models CPython - tests and libraries use Python too Tests cover lots of corner cases Testing Jython with Jython itself is a very pleasant way to test Occasional disadvantages, such as requiring a substantial amount of the language and runtime be availableh Aside: Jython could see more unit tests of the runtime
10. Integrating Jython with Java Jim Baker, Shashank Bharadwaj Java from Jython Python code can directly import Java classes, as if they were Python classes Construct Java objects from Java classes, as if they were Python classes Work with Java objects. Your programs can call Java methods, or have Java call back into Python. And this “magic”/integration works because we have built a proxy on the fly
11. Integrating Jython with Java Jim Baker, Shashank Bharadwaj Apache POI: Read and write MS Office docs from contextlib import closing from org.apache.poi.hssf.usermodel import * from java.io import FileInputStream with closing(FileInputStream("data.xls")) as fis, \ closing(HSSFWorkbook(fis)) as wb: sheet = wb.getSheetAt(0) num_rows = sheet.getPhysicalNumberOfRows() for i in xrange(num_rows): row = sheet.getRow(i) if row: num_cols = sheet.getRow(i).\ getPhysicalNumberOfCells() for j in xrange(num_cols): cell = row.getCell(j) # then do something interesting!!!
12. Integrating Jython with Java Jim Baker, Shashank Bharadwaj Python and Java type equivalence Java type Python type char str boolean bool byte, short, int, long int java.lang.String unicode byte[], char[] str java.lang.Class JavaClass x[] array.array org.python.core.PyObject unchanged Foo PyJavaType proxying Foo
13. Integrating Jython with Java Jim Baker, Shashank Bharadwaj Achieving equivalence Jython uses the following methods tojava java2py Usually used transparently Sometimes significant overhead because of boxing/unboxing Not always because boxing/unboxing is a common idiom in Java, so JVMs can often optimize this away (* significant but it depends!)
14. Integrating Jython with Java Jim Baker, Shashank Bharadwaj Example: datetime.py if _sys.platform.startswith(’java’): def __tojava__(self, java_class): if java_class not in (Calendar, Date, Object): return Py.NoConversion calendar = Calendar.getInstance() calendar.clear() calendar.set( self.year, self.month - 1, self.day) if java_class == Calendar: return calendar else: return Date(calendar.getTimeInMillis())
15. Integrating Jython with Java Jim Baker, Shashank Bharadwaj Use of overloaded methods What happens when you call overloaded methods? How does Jython pick the one you want? Sometimes the runtime picks wrong Force it by using constructors from java.lang - Boolean, Integer, Double, etc
16. Integrating Jython with Java Jim Baker, Shashank Bharadwaj JSR-223: Scripting for the Java platform Part of the Jython release since 2.5.1 JSR 223 is intentionally shallow - common operations we expect to see in scripting scenarios Can still be incredibly effective in driving Jython at the top level Just like we would expect in a good script
17. Integrating Jython with Java Jim Baker, Shashank Bharadwaj Example code public void testEvalString() throws ScriptException { ScriptEngineManager manager = new ScriptEngineManager(); ScriptEngine pythonEngine = manager.getEngineByName("python"); assertNull(pythonEngine.eval("x = 5")); assertEquals(Integer.valueOf(5), pythonEngine.eval("x")); } Source: /tests/java/org/python/jsr223/ScriptEngineTest.java
18. Integrating Jython with Java Jim Baker, Shashank Bharadwaj Jython, testing Jython You can even test JSR 223 support from Jython itself: def test_factory(self): engine = ScriptEngineManager().\ getEngineByName("python") f = engine.factory language_version = ".".join(str(comp) for comp in sys.version_info[0:2]) impl_version = ".".join(str(comp) for comp in sys.version_info[0:3]) self.assertNotEqual(f.scriptEngine, engine) self.assertEqual(f.engineName, "jython") self.assertEqual(f.engineVersion, impl_version) self.assertEqual(set(f.extensions), set([’py’])) self.assertEqual(f.languageName, "python")
19. Integrating Jython with Java Jim Baker, Shashank Bharadwaj And more such tests # variants self.assertEqual( f.getParameter(ScriptEngine.ENGINE), "jython") self.assertEqual( f.getParameter(ScriptEngine.ENGINE_VERSION), impl_version) self.assertEqual( f.getParameter(ScriptEngine.LANGUAGE_VERSION), language_version) self.assertEqual( f.getOutputStatement("abc"), "print u’abc’") self.assertEqual( f.getProgram("x = 42", "y = ’abc’"), "x = 42\ny = ’abc’\n")
20. Integrating Jython with Java Jim Baker, Shashank Bharadwaj Or perhaps not But not without issues, as we found in our own usage: # XXX Earlier version of this test also tested # put, get, eval on the engine, however this # introduced action at a distance where aspects # of the sys state changed # (notably sys.stdin.newlines), which then # impacted test_univnewlines later in the regrtest. # # For now, there may be limits in how much # we can test Jython from itself, no matter # how attractive from an ouroboros perspective that # may be :). Certainly worth revisiting in 2.6. (Or later as the case may be.)
21. Integrating Jython with Java Jim Baker, Shashank Bharadwaj Polyglot programming, from Groovy engine = mgr.getEngineByName("jython") engine.eval(’’’ def factorial(n): if i == 0: return 1 else: return n * factorial(n - 1) result = factorial(4) ’’’) println ’jython: ’ + engine.result Slightly modified example from http://groovy.codehaus.org/JSR- 223+access+to+other+JVM+languages
22. Integrating Jython with Java Jim Baker, Shashank Bharadwaj Callbacks Callbacks are commonly used in Java Generally requires a fair amount of boilerplate code, even with anoymous classes And generally you need to pass in some state
23. Integrating Jython with Java Jim Baker, Shashank Bharadwaj Using closures instead Java methods that take objects of single method interfaces Useful for working with Callable or Runnable interfaces Can pass in a lexical closure, that is a function that closes over variables in its lexical scope
24. Integrating Jython with Java Jim Baker, Shashank Bharadwaj How does that actually work? We need to support Python -> Java -> Python For lexically closed variables, you are actually using a closure For globally scoped variables, Python can look up the right state referenced by a thread local, transparent to the using Python code
25. Integrating Jython with Java Jim Baker, Shashank Bharadwaj Using Python code from Java: Object factories Java code can use the Jython runtime to construct Python objects Object factories use Jython-specific APIs to efficiently build Python objects
26. Integrating Jython with Java Jim Baker, Shashank Bharadwaj Example: Use the runtime Use the Python runtime to get a PyObject for the type of interest public class SomeClassFactory { private PyObject someClass; public SomeClassFactory() { PythonInterpreter interpreter = new PythonInte interpreter.exec("from something import SomeCl someClass = interpreter.get("SomeClass"); }
27. Integrating Jython with Java Jim Baker, Shashank Bharadwaj Construct away! public SomeClassType create (String x, String y) { PyObject someObject = someClass.__call__( new PyString(x), new PyString(y)) return (SomeClassType)someObject.__tojava__( someClassType.class); } }
28. Integrating Jython with Java Jim Baker, Shashank Bharadwaj Gradual Typing A new type system proposed by Jeremy Siek & Walid Taha; extended by others Seamless integration between statically typed parts and dynamically typed parts of code in the language Guarantees that statically typed part (after type checking) will not raise type error at runtime.
29. Integrating Jython with Java Jim Baker, Shashank Bharadwaj Gradual Typing for Python gradual module for Python 3.x Provides type checking based on user defined type annotations Uses function annotations and decoraters
30. Integrating Jython with Java Jim Baker, Shashank Bharadwaj Gradual Typing for Python - Example from gradual import typed @typed class YourAwesomeApi(object): def answerToLife(self, a: int, b: int) -> int: return a + b def use_api(x): print(x.answerToLife(10, 32)) print(x.answerToLife(’hello’, ’there’)) # TypeError: Expected int, got: # ’hello’ of type str use_api(YourAwesomeApi())
31. Integrating Jython with Java Jim Baker, Shashank Bharadwaj Gradual Typing for Python: Work in Progress Prototype - in experimental mode! pip install gradual to get it Repo at: http://bitbucket.org/gradual/py-gradual Working on accurate representation of objects Working on Blame Tracking to produce accurate error messages
32. Integrating Jython with Java Jim Baker, Shashank Bharadwaj Future: Integrate Gradual Typing into Jython Use Gradual Typing to enable typed-Jython code Enable better Java integration type information can be used to remove wrappers in Java code reduce the proxies involved Enable optimizations Drive type information to generate idiomatic Java code (where applicable)
33. Integrating Jython with Java Jim Baker, Shashank Bharadwaj Questions Thank You! Jim Baker Shashank Bharadwaj jbaker@zyasoft.com shashank.bharadwaj@colorado.edu @jimbaker @ shanka