Java for Python Developers Christopher Neugebauer @chrisjrn [email protected] chris.neugebauer.id.au

Before I begin, I’d like to warn you about the contents of this talk. Yes, we’re at a PyCon, but we’re looking at interoperating with real code written by people who use other languages in the real world. What you will see here is probably disturbing to you.

public class AVeryShortProgramImpl { public static void main(String[] args) { System.out.println(“Hello, World!”); } } There are people in this world who think that it is perfectly reasonable that there are languages where this is the shortest possible Hello World program. And I know what you’re thinking. There’s so many words here. What does “public static void main” even mean, and why should I care? There’s no way in the world that this could be a short program!

public class AVeryShortProgramImpl { public static void main(String[] args) { System.out.println(“Hello, World!”); } } wat SO MANY WORDS huh so many namespaces There are people in this world who think that it is perfectly reasonable that there are languages where this is the shortest possible Hello World program. And I know what you’re thinking. There’s so many words here. What does “public static void main” even mean, and why should I care? There’s no way in the world that this could be a short program!

public class AVeryShortProgramImpl { public static void main(String[] args) { System.out.println(“Hello, World!”); } } wat LIES! SO MANY WORDS huh so many namespaces There are people in this world who think that it is perfectly reasonable that there are languages where this is the shortest possible Hello World program. And I know what you’re thinking. There’s so many words here. What does “public static void main” even mean, and why should I care? There’s no way in the world that this could be a short program!

print “Hello, World!” We’re here at PyCon because we know that there’s a better way, and that better way is 119 characters shorter

print (“Hello, World!”) (Or an even better way)

Unfortunately, the people who happen to *like* Java tend to be people who make important decisions. Java’s quite popular out there in the business world, and often, Java’s too entrenched to have any hope of swapping over to a more reasonable Python.

And one Python-loving company in particular likes Java a lot more than we’d like. We’ll get back to that later.

#!/usr/bin/python System.out.println(“Hello, World!”) Today’s talk is about tools you can use when you have no choice but to enter the world of Java, but retain your sanity as a Python developer!

Java: What Is It? Image: ©1998 Merbabu, CC-BY-SA So, let’s start with a quick primer. You may have heard of this Java thing. It’s indeed the topic of this entire talk, so we need to start by explaining exactly what it is.

Three Parts Java Virtual Machine “Java” refers to two things. There’s the Java Virtual Machine (or JVM) that sits at the core of a Java application. There’s the Java Programming Language, which you write code in And there’s the Class Library. If you’ve heard of Enterprise Java, or (sadly) Mobile Java, the difference between these and Standard Java is what you have available to you in the class Library.

Three Parts Java Virtual Machine (JVM) “Java” refers to two things. There’s the Java Virtual Machine (or JVM) that sits at the core of a Java application. There’s the Java Programming Language, which you write code in And there’s the Class Library. If you’ve heard of Enterprise Java, or (sadly) Mobile Java, the difference between these and Standard Java is what you have available to you in the class Library.

Three Parts Java Virtual Machine Java Programming Language (JVM) “Java” refers to two things. There’s the Java Virtual Machine (or JVM) that sits at the core of a Java application. There’s the Java Programming Language, which you write code in And there’s the Class Library. If you’ve heard of Enterprise Java, or (sadly) Mobile Java, the difference between these and Standard Java is what you have available to you in the class Library.

Three Parts Java Virtual Machine Java Programming Language Java Class Library (JVM) “Java” refers to two things. There’s the Java Virtual Machine (or JVM) that sits at the core of a Java application. There’s the Java Programming Language, which you write code in And there’s the Class Library. If you’ve heard of Enterprise Java, or (sadly) Mobile Java, the difference between these and Standard Java is what you have available to you in the class Library.

Java Applications .java Java Source The life of a Java application looks like this, usually: You write your code in Java, and compile it into JVM bytecode. This bytecode gets run on the virtual machine. You’ll need to access code from another source, such as the Class Library, or a third-party library.

Java Applications .java Java Source .class JVM Bytecode Compiler The life of a Java application looks like this, usually: You write your code in Java, and compile it into JVM bytecode. This bytecode gets run on the virtual machine. You’ll need to access code from another source, such as the Class Library, or a third-party library.

Java Applications Java Virtual Machine .java Java Source .class JVM Bytecode Compiler The life of a Java application looks like this, usually: You write your code in Java, and compile it into JVM bytecode. This bytecode gets run on the virtual machine. You’ll need to access code from another source, such as the Class Library, or a third-party library.

Java Applications Java Virtual Machine .java Java Source .class JVM Bytecode Compiler .class .class .class .class .cla Library code from other sources The life of a Java application looks like this, usually: You write your code in Java, and compile it into JVM bytecode. This bytecode gets run on the virtual machine. You’ll need to access code from another source, such as the Class Library, or a third-party library.

Java Applications Java Virtual Machine .java Java Source .class JVM Bytecode Compiler .class .class .class .class .cla Library code from other sources The life of a Java application looks like this, usually: You write your code in Java, and compile it into JVM bytecode. This bytecode gets run on the virtual machine. You’ll need to access code from another source, such as the Class Library, or a third-party library.

Python on Java What are the options for running Python code on Java-the-platform?

Java Applications Java Virtual Machine .java .class Java Source JVM Bytecode .class .class .class .class .cla Compiler Library code from other sources So Java takes source code, translates it into bytecode, and runs that bytecode on a virtual machine...

Python Applications Cpython Virtual Machine .py .pyc Python Source Cpython Bytecode .pyc .pyc .pyc .pyc .py import Library code from other sources In the classic Cpython environment, things are basically the same. There’s a virtual machine that executes bytecode; library code is lazily loaded off disk. The main difference is that there’s no separate build phase: Bytecode tends to get created when you first import a file -- the Python VM compiles the module, and spits out a .pyc file.

Jython Not Just A Typo Anymore™ Jython’s an alternative version of Python, written in Java, which runs on the JVM. It aims towards compatibility with Cpython wherever possible: a specific goal of theirs is able to run any pure-python library that Cpython can run.

Python Applications Cpython Virtual Machine .py .pyc Python Source Cpython Bytecode .pyc .pyc .pyc .pyc .py import Library code from other sources The lifecycle of Jython is remarkably similar to Cpython’s. But instead of producing .pyc files and executing on a custom VM...

Jython Applications Java Virtual Machine .py .class Python Source JVM Bytecode import Library code from other sources .class .class .class .class .cla ... Jython produces actual JVM bytecode when you import a Python file. It can dynamically load Java classes as needs be. It can interoperate with Java classes whenever you need it to, including subclassing and implementing interfaces.

Jython • Python written in Java • Runs on the JVM • Compiles Python to Java Classes • setuptools – makes standalone .jars

Jython 2.7b3 (default:e81256215fb0, Aug 4 2014, 02:39:51) [Java HotSpot(TM) 64-Bit Server VM (Oracle Corporation)] on java1.8.0_11 Type "help", "copyright", "credits" or "license" for more information. >>> from java.lang import System >>> System.out.println("Hello, World!") Hello, World! It has a repl that you can use, just like CPython, but as you can see, you’ve got full access to the Java classes available on your System JVM. This is hello world using standard Java calls from Jython

Jython 2.7b3 (default:e81256215fb0, Aug 4 2014, 02:39:51) [Java HotSpot(TM) 64-Bit Server VM (Oracle Corporation)] on java1.8.0_11 Type "help", "copyright", "credits" or "license" for more information. >>> from java.lang import System >>> System.out.println("Hello, World!") Hello, World! >>> >>> print "Hello, World!" Hello, World! But most of the standard trimmings of Python are there as well. Indeed, if you want to maintain compatibility with CPython as much as possible, using Python standard calls makes a lot of sense.

Jython • Stable: 2.5 (August 2012) • Preview: 2.7b3 (August 2014) Jython hasn’t had a lot of love over the years, but development’s starting to pick back up. Python 2.7 support is nearly there, and the latest Beta is good enough to run Requests, for instance.

Android Python on Java is in a pretty good state now. Pretty much anything you’d want to do in Java-the-language on the JVM you can do in Python. Most things you’d want to do in CPython 2.7 you can do on the JVM. Android’s an important and interesting use case for Java.

Gartner, 14 May 2013 Android’s important because it holds 75% of the worldwide smartphone market...

Gartner, 10 April 2013; 14 May 2013 0 50 100 150 200 Full-size computers Android devices Millions of Units, Q1 2013 And it’s important because last year, Android devices *alone* outsold full-size computers almost twofold. Android’s basically the biggest user base for Mobile computing out there.

Programmed with “Java” And you guessed it, it’s programmed with Java...

Programmed with “Java” :( ... but what do we mean by that?

Programmed with “Java” Well, Android’s environment is not, strictly speaking, Java. As a developer, for the most part, you don’t care that it’s not Java -- you write your code in Java, you use the Java compiler, and you get access to the Java library.

Programmed with “Java” Well, Android’s environment is not, strictly speaking, Java. As a developer, for the most part, you don’t care that it’s not Java -- you write your code in Java, you use the Java compiler, and you get access to the Java library.

Android Applications Dalvik Virtual Machine .java Java Source ????? But what differs is the runtime. Android uses what I’ll call the “Dalvik Model”. This involves using a special virtual machine called Dalvik, which is almost, but not quite entirely unlike the JVM. You may have heard of a thing called ART -- this is a drop-in replacement for Dalvik, but the distinction really isn’t important, so I won’t talk about it much.

Android Applications Dalvik Virtual Machine .java Java Source ????? Android Runtime (ART) But what differs is the runtime. Android uses what I’ll call the “Dalvik Model”. This involves using a special virtual machine called Dalvik, which is almost, but not quite entirely unlike the JVM. You may have heard of a thing called ART -- this is a drop-in replacement for Dalvik, but the distinction really isn’t important, so I won’t talk about it much.

Java Applications Java Virtual Machine .java .class Java Source JVM Bytecode .class .class .class .class .cla Compiler Library code from other sources So to recap -- the JVM model executes JVM bytecode which you’ve compiled from Java source. Or perhaps Jython source. External libraries are loaded from class files as needs be.

Android Applications Dalvik Virtual Machine .java .class Java Source JVM Bytecode .dex Dalvik Executable The Dalvik model introduces an extra step. Once you have your JVM bytecode, it gets translated into a thing called a Dalvik Executable, which then runs on the Dalvik virtual machine. But you may have noticed something missing. What about third-party code?

Dalvik Executables .dex Dalvik Executable .class .class Let’s look inside a Dalvik executable: inside it is your own compiled Java code. But the executable bundle *also* contains all of the third-party code you need. It gets merged down into a single file, and there’s no dynamic loading of external classes. So, on Android, you have whatever you had at build time, or nothing.

Dalvik Executables .dex Dalvik Executable .class .class .class .class .class .class .class .class .class Let’s look inside a Dalvik executable: inside it is your own compiled Java code. But the executable bundle *also* contains all of the third-party code you need. It gets merged down into a single file, and there’s no dynamic loading of external classes. So, on Android, you have whatever you had at build time, or nothing.

Jython So this means that Jython, which is basically predicated on creating Java class files to load as they’re needed, fundamentally doesn’t work under the Dalvik model.

What else could work? So with Dalvik’s almost-but-not-quite Java model for doing things, we need to find an alternative solution.

CPython has a C API! Let’s fall back to something we already know -- Cpython! Cpython’s always had a fully-fledged C API.

C Extensions! .py Cpython VM .so .so .so .so The idea is that you can write a C interface, which can talk to other C libraries on your system in order to get access to their functionality.

Java has a C API! Java also has a C API...

Java has a C API! and Android kept it! ... which has been implemented wholesale for Android!

Java Native Interface Java’s C interface is called the Java Native Interface, or JNI for short. So we C extension to call the bits of Java code that we need in our Python apps.

C Extensions! .py Cpython VM .so JNI Java VM .java Bridging code So the idea is that you write a C extension for CPython to that can call into the JNI, which in turn gives you access to things that are stuck behind a Java interface.

C Extensions! .py Cpython VM .so JNI Dalvik VM .java ... and this model works completely under Dalvik on Android as well.

extern "C" JNIEXPORT void JNICALL Java_ClassName_MethodName (JNIEnv *env, jobject obj, jstring javaString) { //Get the native string from javaString const char *nativeString = env->GetStringUTFChars(javaString, 0); std::cout << nativeString << std::endl; //DON'T FORGET THIS LINE!!! env->ReleaseStringUTFChars(javaString, nativeString); } (Wikipedia) The problem is that the JNI is quite complicated to write -- this function takes a string from Java, and prints it out from C++. For an echo function, this is stupidly complicated. You have to get this method name exactly right depending on what class and method you’re implementing. Getting variables out of Java-land has a really complex API And you have to manually manage your Java objects within C. For a codebase of any degree of complexity, it’s basically impossible to get this right and maintain it. In particular, if you do any refactoring of Java code, your C method names might be wrong.

extern "C" JNIEXPORT void JNICALL Java_ClassName_MethodName (JNIEnv *env, jobject obj, jstring javaString) { //Get the native string from javaString const char *nativeString = env->GetStringUTFChars(javaString, 0); std::cout << nativeString << std::endl; //DON'T FORGET THIS LINE!!! env->ReleaseStringUTFChars(javaString, nativeString); } (Wikipedia) 1. Precise naming semantics The problem is that the JNI is quite complicated to write -- this function takes a string from Java, and prints it out from C++. For an echo function, this is stupidly complicated. You have to get this method name exactly right depending on what class and method you’re implementing. Getting variables out of Java-land has a really complex API And you have to manually manage your Java objects within C. For a codebase of any degree of complexity, it’s basically impossible to get this right and maintain it. In particular, if you do any refactoring of Java code, your C method names might be wrong.

extern "C" JNIEXPORT void JNICALL Java_ClassName_MethodName (JNIEnv *env, jobject obj, jstring javaString) { //Get the native string from javaString const char *nativeString = env->GetStringUTFChars(javaString, 0); std::cout << nativeString << std::endl; //DON'T FORGET THIS LINE!!! env->ReleaseStringUTFChars(javaString, nativeString); } (Wikipedia) 1. Precise naming semantics 2. Complicated accessor API The problem is that the JNI is quite complicated to write -- this function takes a string from Java, and prints it out from C++. For an echo function, this is stupidly complicated. You have to get this method name exactly right depending on what class and method you’re implementing. Getting variables out of Java-land has a really complex API And you have to manually manage your Java objects within C. For a codebase of any degree of complexity, it’s basically impossible to get this right and maintain it. In particular, if you do any refactoring of Java code, your C method names might be wrong.

extern "C" JNIEXPORT void JNICALL Java_ClassName_MethodName (JNIEnv *env, jobject obj, jstring javaString) { //Get the native string from javaString const char *nativeString = env->GetStringUTFChars(javaString, 0); std::cout << nativeString << std::endl; //DON'T FORGET THIS LINE!!! env->ReleaseStringUTFChars(javaString, nativeString); } (Wikipedia) 1. Precise naming semantics 2. Complicated accessor API 3. Manual memory management The problem is that the JNI is quite complicated to write -- this function takes a string from Java, and prints it out from C++. For an echo function, this is stupidly complicated. You have to get this method name exactly right depending on what class and method you’re implementing. Getting variables out of Java-land has a really complex API And you have to manually manage your Java objects within C. For a codebase of any degree of complexity, it’s basically impossible to get this right and maintain it. In particular, if you do any refactoring of Java code, your C method names might be wrong.

Java Native Interface The Simple Expressiveness of Java The Reliable Memory Management of C++ ©2011 Adam Levine, CC BY-SA Basically, the JNI is a horrible mess to actually use. You lose the benefits of both Java and C. Basically you were seriously considering hand-writing interface code with it, you’re certifiably insane.

Auto-Wrappers! The solution is to get a tool that does all the heavy lifting for you, to bridge the two languages.

Kivy www.kivy.org There’s a project that’s been working on producing a Python touch UI API, focusing specifically on mobile devices. That project is called Kivy. Kivy’s been known for making complex touch-oriented mobile apps in Python. This screenshot’s from ProcessCraft, which I believe is a mandatory inclusion for talks about Kivy.

Kivy www.kivy.org obligatory ProcessCraft screenshot There’s a project that’s been working on producing a Python touch UI API, focusing specifically on mobile devices. That project is called Kivy. Kivy’s been known for making complex touch-oriented mobile apps in Python. This screenshot’s from ProcessCraft, which I believe is a mandatory inclusion for talks about Kivy.

Python-For-Android ... But a big part of their project is to get Python running on mobile platforms -- both iOS and Android. Their Python for Android project is called Python-For-Android!

Python-For-Android • Cpython API for Android • Deployment tools • PyJnius Python for Android consists of three key parts: - a CPython-based framework for Android, so you can run Python code on your device - tools for packaging and deployment - and a third thing -- PyJnius.

PyJnius ©2007 dherrera_96 CC-BY PyJnius is a library that handles the complicated bridging between Java and CPython.

System = autoclass('java.lang.System') System.out.println('Hello, World!') Here’s Hello World in PyJnius. You can see that we’re calling Java’s println method...

#!/usr/bin/python from jnius import autoclass System = autoclass('java.lang.System') System.out.println('Hello, World!') ... and in this case, all we’ve needed to do is import ‘autoclass’ from the Jnius library. Autoclass wraps any available Java class into a Python class. You can call Java methods and access Java attributes just as though they’re native Python calls. It’s pretty cool.

>>> System = autoclass('java.lang.System') >>> System >>> System.out >>> System.out.println And to show you what’s going on under the hood -- once you’ve called autoclass, jnius takes care of wrapping all of the Java attributes and methods into Python objects for you.

But Wait, There’s More! So many cases in Kivy -- where simply gaining access to Java APIs is enough -- what you’ve seen is pretty good. But what if you want to completely drive a Java library completely from Python -- there are things like callbacks, or implementing interfaces that you need to do. PyJnius can help with that too!

public class PyRunner { public void runThis(Runnable r) { r.run(); } } Here’s a really quite boring piece of Java. You pass in a runnable... and it runs it. There’s nothing terribly interesting about it. I’m just trying to make a point here.

Class HelloRunnable implements Runnable { public void run() { System.out.println("Hi from Java!"); } }; public static void main(String[] args) { PyRunner runner = new PyRunner(); Runnable runnable = new HelloRunnable(); runner.runThis(runnable); // prints “Hi from Java!” } So Here’s a simple Java program -- we’ve got a Runnable class that prints a message, and some code that uses our runner from the previous slide to run it. It does exactly what you’d expect!

class PythonRunnable(PythonJavaClass): __javainterfaces__ = ["java/lang/Runnable"] @java_method("()V") def run(self): print "Hello from Python!" Here’s a Python class. It’s got a method called “run”, and a bunch of other decorations and annotations, hopefully that gives you a clue of some sort

class PythonRunnable(PythonJavaClass): __javainterfaces__ = ["java/lang/Runnable"] @java_method("()V") def run(self): print "Hello from Python!" So our class derives from PythonJavaClass, it has a slot called javainterfaces, and the method is decorated with @java_method...

class PythonRunnable(PythonJavaClass): __javainterfaces__ = ["java/lang/Runnable"] @java_method("()V") def run(self): print "Hello from Python!" def main(): PyRunner = autoclass("PyRunner") runnable = PythonRunnable() runner = PyRunner() runner.runThis(runnable) And we can wrap our PyRunner class from earlier, and pass in an instance of our PythonRunnable class -- from Python -- and our Java class runs it. PyJnius takes care of all of wrapping our *Python* class for Java, so that Java can use it easily.

• Use CPython as glue for Java libraries • Implement native Java objects in Python PyJnius!!! In summary -- PyJnius lets you script Java objects from CPython. It’s close to feature- complete if you’re using Java objects in a consumption-only fashion. It’s also good for creating callbacks from Java code into Python code.

PyJnius runs on Android! Best of all, because CPython runs on Android, and the JNI’s properly implemented on Android, PyJnius works on Android as you’d expect. You can consume Android API objects as you need to, and where they provide listener interfaces, you can implement them in Python. This means that things like accelerometer and other hardware access that needs callbacks can be done entirely in Python.

Not Quite There Yet Unfortunately, while PyJnius is a good bridge, it’s not complete yet...

def main(): Thread = autoclass("java.lang.Thread") runnable = PythonRunnable() thread = Thread(runnable) thread.start() The first example I tried to write without making any Java calls at all. I tried to do it by creating Java threads, and that broke in various places...

@java_method("()V") There’s also usability bugs -- if you remember the @java_method decorator I showed you earlier, we have to provide this weird string here. This is called a JNI method signature -- the JNI uses this string to figure out which method to call from your C code. This one is for a method that takes no arguments, and has a void return type. This is literally as simple as it gets.

public Collection spam( String bacon, long egg, boolean sausage ) { ... } The problem is that if you have a method that’s even slightly complex, these strings become unmanageable. This method signature here...

public Collection spam( String bacon, long egg, boolean sausage ) { ... } @java_method( "(Ljava/lang/String;JZ)Ljava/util/Collection;" ) ... has that method signature. There’s no good reason why a Python programmer should have to learn how to write JNI signatures, and these are documentation liabilities for future programmers -- you have to read these signatures to understand them. A Python API for writing signatures is close to essential.

public class Foo extends Bar { The next problem is that PyJnius doesn’t do subclassing of Java objects. Python doesn’t really care about the distinction between implementing an interface and overriding a subclass, but Java does. They’re completely different mechanisms.

• Interfaces • Abstract Classes • Concrete Classes extends vs implements Java has three different types of type that you can implement in Java-the-language -- there are Interfaces, Abstract Classes (which are classes with some methods left unimplemented), and Concrete Classes (which are classes where all the methods are in place). and currently PyJnius only lets you implement Interfaces. Unfortunately, most Java frameworks (Android included) require you to override concrete or abstract classes to fully use them. That means that you can’t currently implement an entire Android application in Python.

• Interfaces • Abstract Classes • Concrete Classes extends vs implements Java has three different types of type that you can implement in Java-the-language -- there are Interfaces, Abstract Classes (which are classes with some methods left unimplemented), and Concrete Classes (which are classes where all the methods are in place). and currently PyJnius only lets you implement Interfaces. Unfortunately, most Java frameworks (Android included) require you to override concrete or abstract classes to fully use them. That means that you can’t currently implement an entire Android application in Python.

The best shot at a Python API for Native Android. ... the upside is that most of these bugs can be fixed, and when they are, it’s easily the best avenue towards making a complete Android API for Python.

Which should you choose? So I’ve shown you two tools for using Java and Python in the same application -- Jython and Jnius. This begs the question -- which should you use?

If Java’s more important, use Jython! If you need to add Python scripting to a Java platform, or your requirements require extensive JVM use -- basically, if Python’s peripheral to your greater goal, use Jython. It gives you complete JVM access and all your code runs on the JVM. The downside is that you only get experimental Python 2.7 support, and not all C extensions will work out of the box.

If Python’s more important, use PyJnius! If your use of the JVM is incidental compared with your Python code... or if you need to run on Android, Jnius is your tool. You get all the goodness of Cpython, at the tradeoff of having to use reflection and proxy objects to cross the JNI bridge. It’s also still got a few bugs to be ironed out.

The End • Java is a language and a platform • Jython’s getting better • Android’s not Java (platform), but PyJnius helps! So that’s the end. In summary: - When we talk about Java, we’re taking about the combination of a language, a runtime platform, and an API. Unfortunately, the distinction is important. - Jython’s finally getting some love again, with 2.7 support almost finished. - Android is not Java enough to match our understanding of Java-the-platform, and it doesn’t support enough of the JVM’s features to support Jython. PyJnius is doing a really good job of bridging the gap here.

Java for Python Developers Christopher Neugebauer @chrisjrn – [email protected] Questions?