Introduction to LuaJIT & How to bind CPP code base using LuaJIT FFI

This was presented at osdc.tw/2012, if you find anything wrong, please contact me!

igdshare@igdshare.org or
arch.jslin@gmail.com

1. Introduction to LuaJIT & How to bind C++ code base using LuaJIT FFI 2012.04.14 OSDC.tw
2. How to bind complex C++ code base using LuaJIT FFI
3. About me Johnson Lin
4. About me Johnson Lin Programmer & Indie Game
5. About me Johnson Lin Programmer & Indie Game igdshare.org ( 獨立遊戲開發者分享會 )
6. About me Johnson Lin Programmer & Indie Game igdshare.org & Game Jam! (4/28 ~ 4/29) ( 獨立遊戲開發者分享會 )
7. About me Johnson Lin Programmer & Indie Game igdshare.org & Game Jam! (4/28 ~ 4/29) ( 獨立遊戲開發者分享會 ) archilife.org ( 祐生研究基金會 )
8. About me Johnson Lin Programmer & Indie Game igdshare.org & Game Jam! (4/28 ~ 4/29) ( 獨立遊戲開發者分享會 ) archilife.org & Indie game contests scholarship ( 祐生研究基金會 )
9. Overview Introduction to Lua Introduction to LuaJIT How does Lua talk to C/C++ LuaJIT FFI (Foreign Function Interface)
10. Overview Introduction to Lua Introduction to LuaJIT How does Lua talk to C/C++ LuaJIT FFI (Foreign Function Interface)
11. Introduction to LUA Introduction to LUA
12. Introduction to LUA Introduction to LUA LUA is...
13. Introduction to LUA Introduction to LUA LUA is... Lua
14. Introduction to LUA Introduction to LUA LUA is... Lua Uppercase
15. Introduction to LUA Introduction to LUA LUA is... Lua Uppercase Accident This pun came from a clever guy Joseph Manning: http://lua-users.org/lists/lua-l/2011-06/msg01112.html
16. Introduction to Lua Lua = "Moon" in Portuguese Introduction to Lua
17. Introduction to Lua Lua = "Moon" in Portuguese Introduction to Lua
18. Introduction to Lua Lua = "Moon" in Portuguese Use Cases (Game Dev): Introduction to Lua
19. Introduction to Lua Lua = "Moon" in Portuguese Use Cases (Game Dev): Grim Fandango, Introduction to Lua
20. Introduction to Lua Lua = "Moon" in Portuguese Use Cases (Game Dev): Grim Fandango, World of Warcraft UI, Introduction to Lua
21. Introduction to Lua Lua = "Moon" in Portuguese Use Cases (Game Dev): Grim Fandango, World of Warcraft UI, Angry Birds Introduction to Lua
22. Introduction to Lua By Tecgraf / PUC-Rio (1993): Roberto Ierusalimschy, Luiz Henrique de Figueiredo, Waldemar Celes Introduction to Lua
23. Introduction to Lua By Tecgraf / PUC-Rio (1993): Roberto Ierusalimschy, Luiz Henrique de Figueiredo, Waldemar Celes Current version: 5.2.0 / 5.1.5 MIT License since 5.0 (BSD License before then) Introduction to Lua
24. Lua Features Introduction to Lua
25. Lua Features Fast Fastest among other well-known interpreted language Introduction to Lua
26. Lua Features Fast Fastest among other well-known interpreted language Small ~150kB binary, ~20kB mem overhead per interpreter Introduction to Lua
27. Lua Features Fast Fastest among other well-known interpreted language Small ~150kB binary, ~20kB mem overhead per interpreter Simple 22 keywords, 27 tokens Introduction to Lua
28. Lua Features Multithreading-Friendly multiple interpreters in multiple threads concurrently without GIL Introduction to Lua
29. Lua Features Multithreading-Friendly multiple interpreters in multiple threads concurrently without GIL Easy to Embed & Extend concise Lua/C API Introduction to Lua
30. Lua Features Multithreading-Friendly multiple interpreters in multiple threads concurrently without GIL Easy to Embed & Extend concise Lua/C API But, without "battery" although Luarocks is in place Introduction to Lua
31. Lua Features Introduction to Lua Multi-paradigm:
32. Lua Features Introduction to Lua Multi-paradigm: procedural functional prototype-based OO
33. Lua Features Introduction to Lua Multi-paradigm: procedural functional prototype-based OO Think Lua as JavaScript
34. Lua Features Introduction to Lua Multi-paradigm: procedural functional prototype-based OO Think Lua as JavaScript .....without the Bad Parts
35. Z Combinator in Lua Introduction to Lua local Y = function(f) return (function(x) return f (function(y) return (x(x))(y) end) end) (function(x) return f (function(y) return (x(x))(y) end) end) end
36. Overview Introduction to Lua Introduction to LuaJIT How does Lua talk to C/C++ LuaJIT FFI (Foreign Function Interface)
37. Introduction to LuaJIT Introduction to LuaJIT
38. Introduction to LuaJIT Just-In-Time compiler for Lua Introduction to LuaJIT
39. Introduction to LuaJIT Just-In-Time compiler for Lua Mike Pall Introduction to LuaJIT
40. Introduction to LuaJIT Just-In-Time compiler for Lua Mike Pall LuaJIT1 (2005~, current 1.1.7) LuaJIT2 (2009~, current 2.0-beta9) Introduction to LuaJIT
41. LuaJIT2 Features Interpreter written in assembler (DynASM) Introduction to LuaJIT
42. LuaJIT2 Features Interpreter written in assembler (DynASM) Trace compiler Introduction to LuaJIT
43. LuaJIT2 Features Interpreter written in assembler (DynASM) Trace compiler x86/64 (Windows, OSX, Linux, etc) SSE2 instructions required Introduction to LuaJIT
44. LuaJIT2 Features Interpreter written in assembler (DynASM) Trace compiler x86/64 (Windows, OSX, Linux, etc) SSE2 instructions required ARM (Android, iOS, etc), PPC, MIPS only the interpreter part on iOS works now, blame Apple Introduction to LuaJIT
45. LuaJIT2 Features API & ABI compatible with Lua 5.1 Bytecode NOT compatible Introduction to LuaJIT
46. LuaJIT2 Features API & ABI compatible with Lua 5.1 Bytecode NOT compatible Some Lua 5.2 functionality But not going to fully catch up anytime soon Introduction to LuaJIT
47. LuaJIT2 Features API & ABI compatible with Lua 5.1 Bytecode NOT compatible Some Lua 5.2 functionality But not going to fully catch up anytime soon Foreign Function Interface with C parser and a few extensions Introduction to LuaJIT
48. LuaJIT - Behind the Scene Introduction to LuaJIT
49. LuaJIT - Behind the Scene Introduction to LuaJIT Too hard for me! Sorry!
50. Some Use Cases of LuaJIT AI Implementation in The Settlers 7 (2010) GSL Shell https://github.com/malkia/ufo Includes OpenGL, OpenCL, SDL, ZeroMQ bindings.. OpenResty web framework based on nginx Introduction to LuaJIT
51. Some benchmarks Game of Life C++ v.s. LuaJIT (using FFI) Introduction to LuaJIT
52. Some benchmarks Game of Life C++ v.s. LuaJIT (using FFI) Please keep in mind this benchmark is pretty crude and favors LuaJIT, it just shows one extreme. Introduction to LuaJIT
53. Some benchmarks Game of Life Introduction to LuaJIT
54. Some benchmarks Introduction to LuaJIT Pueudo Code: while (true) { game.update(); game.render(); end //both update & render has a big 2d loop //doing state update and drawing cells respectively https://github.com/arch-jslin/mysandbox/tree/master/cpp/lifegame_sdl
55. Some benchmarks Introduction to LuaJIT Pueudo Code: while true do game:update() -- JIT kick in game:render() -- JIT kick in end --both update & render has a big 2d loop --doing state update and drawing cells respectively https://github.com/arch-jslin/mysandbox/blob/master/lua/lifegame_ffi_sdl_gl.lua
56. Some benchmarks Introduction to LuaJIT Pueudo Code: while true do game:update() -- JIT back off game:render() -- JIT kick in end --both update & render has a big 2d loop --doing state update and drawing cells respectively
57. Some benchmarks Introduction to LuaJIT Pueudo Code: while true do game:update() -- JIT back off game:render() -- JIT back off end --both update & render has a big 2d loop --doing state update and drawing cells respectively
58. LuaJIT Roadmap Introduction to LuaJIT
59. LuaJIT Roadmap LuaJIT 2.0 almost feature freeze ARM hard-fp support looking for sponsor Introduction to LuaJIT
60. LuaJIT Roadmap LuaJIT 2.0 almost feature freeze ARM hard-fp support looking for sponsor LuaJIT 2.1 garbage collector redesign Introduction to LuaJIT
61. LuaJIT Roadmap LuaJIT 2.0 almost feature freeze ARM hard-fp support looking for sponsor LuaJIT 2.1 garbage collector redesign Other unconfirmed: FFI C preprocessor? Introduction to LuaJIT
62. Overview Introduction to Lua Introduction to LuaJIT How does Lua talk to C/C++ LuaJIT FFI (Foreign Function Interface)
63. How does Lua talk to C/C++ How does Lua talk to C/C++ Manipulate lua_State (Lua Stack) Using Lua/C API core library: lua_xxx(); aux library: luaL_xxx();
64. Lua/C API -- Lua local a, b = func_from_c(2,3) How does Lua talk to C/C++
65. Lua/C API -- Lua local a, b = func_from_c(2,3) //C static int func_from_c(lua_State* L) { int x = luaL_checkint(L, 1); int y = luaL_checkint(L, -1); lua_pushinteger(L, x+y); lua_pushinteger(L, x-y); return 2; } How does Lua talk to C/C++
66. Lua/C API -- Lua local a, b = func_from_c(2,3) //C static int func_from_c(lua_State* L) { int x = luaL_checkint(L, 1); int y = luaL_checkint(L, -1); lua_pushinteger(L, x+y); lua_pushinteger(L, x-y); return 2; } How does Lua talk to C/C++
67. What about C++ object? How does Lua talk to C/C++ -- Lua local o = Simple(1) -- C++ obj
68. What about C++ object? How does Lua talk to C/C++ -- Lua local o = Simple(1) -- C++ obj //C //Let's see some actual code...
69. What about C++ object? How does Lua talk to C/C++ So that's a bit scary..
70. Binding Libaries How does Lua talk to C/C++
71. Binding Libaries Luabind - boost & template magic How does Lua talk to C/C++
72. Binding Libaries Luabind - boost & template magic OOLua - Macro & template(?) magic How does Lua talk to C/C++
73. Binding Libaries Luabind - boost & template magic OOLua - Macro & template(?) magic Simple Lua Binder - no boost, but still templates How does Lua talk to C/C++
74. Binding Libaries Luabind - boost & template magic OOLua - Macro & template(?) magic Simple Lua Binder - no boost, but still templates ...etc (post talk addition: I am not saying these are bad or what, just our considerations. If you find these tools "just work" for you, then you probably have no reason to abandon them.) How does Lua talk to C/C++
75. Binding Generators tolua & tolua++ How does Lua talk to C/C++
76. Binding Generators tolua & tolua++ SWIG (post talk addition: I am not saying these are bad or what, just our considerations. If you find these tools "just work" for you, then you probably have no reason to abandon them.) How does Lua talk to C/C++
77. Problem with those How does Lua talk to C/C++
78. Problem with those Missing feature How does Lua talk to C/C++
79. Problem with those Missing feature Code bloat How does Lua talk to C/C++
80. Problem with those Missing feature Code bloat Runtime overhead How does Lua talk to C/C++
81. Problem with those Missing feature Code bloat Runtime overhead Yet another glue language How does Lua talk to C/C++
82. Problem with those Missing feature Code bloat Runtime overhead Yet another glue language Not sure what to do in case of bugs How does Lua talk to C/C++
83. LuaJIT Foreign Function Interface LuaJIT FFI
84. Can be JIT compiled Unlike libffi based modules or other binding methods LuaJIT Foreign Function Interface LuaJIT FFI
85. Can be JIT compiled Unlike libffi based modules or other binding methods Has a C parser built-in LuaJIT Foreign Function Interface LuaJIT FFI
86. Can be JIT compiled Unlike libffi based modules or other binding methods Has a C parser built-in Pure Lua Syntax So there's got LuaFFI => https://github.com/jmckaskill/luaffi LuaJIT Foreign Function Interface LuaJIT FFI
87. -- Using C type local ffi = require 'ffi' LuaJIT Foreign Function Interface Examples of LuaJIT FFI
88. -- Using C type local ffi = require 'ffi' local a = ffi.new("int") LuaJIT Foreign Function Interface Examples of LuaJIT FFI
89. -- Using C type local ffi = require 'ffi' local a = ffi.new("int") local b = ffi.new("char const*") LuaJIT Foreign Function Interface Examples of LuaJIT FFI
90. -- Using C type local ffi = require 'ffi' local a = ffi.new("int") local b = ffi.new("char const*") local c = ffi.new("double[10]") LuaJIT Foreign Function Interface Examples of LuaJIT FFI
91. -- Using C type local ffi = require 'ffi' local a = ffi.new("int") local b = ffi.new("char const*") local c = ffi.new("double[10]") local d = ffi.new("double[?]", 20) LuaJIT Foreign Function Interface Examples of LuaJIT FFI
92. -- Calling external C functions local ffi = require 'ffi' LuaJIT Foreign Function Interface Examples of LuaJIT FFI
93. -- Calling external C functions local ffi = require 'ffi' local C = ffi.C LuaJIT Foreign Function Interface Examples of LuaJIT FFI
94. -- Calling external C functions local ffi = require 'ffi' local C = ffi.C ffi.cdef[[ int printf(const char*, ...); ]] LuaJIT Foreign Function Interface Examples of LuaJIT FFI
95. -- Calling external C functions local ffi = require 'ffi' local C = ffi.C ffi.cdef[[ int printf(const char*, ...); ]] C.printf('Hello %s!\n', 'world') LuaJIT Foreign Function Interface Examples of LuaJIT FFI
96. -- Calling external C functions local ffi = require 'ffi' local C = ffi.C ffi.cdef[[ int printf(const char*, ...); ]] C.printf('Hello %s!\n', 'world') C.printf('%d', ffi.cast('int', 3)) LuaJIT Foreign Function Interface Examples of LuaJIT FFI
97. -- Calling external C functions local ffi = require 'ffi' LuaJIT Foreign Function Interface Examples of LuaJIT FFI
98. -- Calling external C functions local ffi = require 'ffi' ffi.cdef[[ typedef struct { int x,y,z; } Point; void do_stuff(Point*); ]] LuaJIT Foreign Function Interface Examples of LuaJIT FFI
99. -- Calling external C functions local ffi = require 'ffi' ffi.cdef[[ typedef struct { int x,y,z; } Point; void do_stuff(Point*); ]] LuaJIT Foreign Function Interface Examples of LuaJIT FFI
100. -- Calling external C functions local ffi = require 'ffi' ffi.cdef[[ typedef struct { int x,y,z; } Point; void do_stuff(Point*); ]] LuaJIT Foreign Function Interface Examples of LuaJIT FFI
101. -- Calling external C functions local ffi = require 'ffi' ffi.cdef[[ typedef struct { int x,y,z; } Point; void do_stuff(Point*); ]] local Lib = ffi.load('path/to/your_c_lib') LuaJIT Foreign Function Interface Examples of LuaJIT FFI
102. -- Calling external C functions local ffi = require 'ffi' ffi.cdef[[ typedef struct { int x,y,z; } Point; void do_stuff(Point*); ]] local Lib = ffi.load('path/to/your_c_lib') local a = ffi.new('Point', 1, 2, 3) LuaJIT Foreign Function Interface Examples of LuaJIT FFI
103. -- Calling external C functions local ffi = require 'ffi' ffi.cdef[[ typedef struct { int x,y,z; } Point; void do_stuff(Point*); ]] local Lib = ffi.load('path/to/your_c_lib') local a = ffi.new('Point', 1, 2, 3) Lib.do_stuff(a) LuaJIT Foreign Function Interface Examples of LuaJIT FFI
104. -- Using C header directly (almost) local ffi = require 'ffi' LuaJIT Foreign Function Interface Examples of LuaJIT FFI
105. -- Using C header directly (almost) local ffi = require 'ffi' ffi.cdef(io.open('header.h', 'r'):read('*a')) LuaJIT Foreign Function Interface Examples of LuaJIT FFI
106. -- Using C header directly (almost) local ffi = require 'ffi' ffi.cdef(io.open('header.h', 'r'):read('*a')) -- This header.h has to be preprocessed LuaJIT Foreign Function Interface Examples of LuaJIT FFI
107. -- Using C header directly (almost) local ffi = require 'ffi' ffi.cdef(io.open('header.h', 'r'):read('*a')) -- This header.h has to be preprocessed -- gcc -E stub.c | grep -v '^#' LuaJIT Foreign Function Interface Examples of LuaJIT FFI
108. -- Using C header directly (almost) local ffi = require 'ffi' ffi.cdef(io.open('header.h', 'r'):read('*a')) -- This header.h has to be preprocessed -- gcc -E stub.c | grep -v '^#' -- stub.c only contain #include <header.h> LuaJIT Foreign Function Interface Examples of LuaJIT FFI
109. //C++ side class A { public: void method(); }; LuaJIT Foreign Function Interface But, in case of C++
110. //C++ side class A { public: void method(); }; extern "C" { A* A_new() { return new A; } void A_method(A* self) { self->method(); } void A_gc(A* self) { delete self; } } LuaJIT Foreign Function Interface But, in case of C++
111. //C++ side class A { public: void method(); }; extern "C" { A* A_new() { return new A; } void A_method(A* self) { self->method(); } void A_gc(A* self) { delete self; } } LuaJIT Foreign Function Interface But, in case of C++
112. //C++ side class A { public: void method(); }; extern "C" { A* A_new() { return new A; } void A_method(A* self) { self->method(); } void A_gc(A* self) { delete self; } } LuaJIT Foreign Function Interface But, in case of C++
113. //C++ side class A { public: void method(); }; extern "C" { //as before, export these A* A_new() { return new A; } void A_method(A* self) { self->method(); } void A_gc(A* self) { delete self; } } LuaJIT Foreign Function Interface But, in case of C++
114. ffi.cdef[[ typedef struct A A; //and then declare identical functions ]] LuaJIT Foreign Function Interface But, in case of C++
115. ffi.cdef[[ typedef struct A A; //and then declare identical functions ]] LuaJIT Foreign Function Interface But, in case of C++
116. ffi.cdef[[ typedef struct A A; //and then declare identical functions ]] local lib = ffi.load('/path/to/lib') LuaJIT Foreign Function Interface But, in case of C++
117. ffi.cdef[[ typedef struct A A; //and then declare identical functions ]] local lib = ffi.load('/path/to/lib') local mt = {} LuaJIT Foreign Function Interface But, in case of C++
118. ffi.cdef[[ typedef struct A A; //and then declare identical functions ]] local lib = ffi.load('/path/to/lib') local mt = {} mt.__index = mt LuaJIT Foreign Function Interface But, in case of C++
119. ffi.cdef[[ typedef struct A A; //and then declare identical functions ]] local lib = ffi.load('/path/to/lib') local mt = {} mt.__index = mt mt.method = lib.A_method LuaJIT Foreign Function Interface But, in case of C++
120. ffi.cdef[[ typedef struct A A; //and then declare identical functions ]] local lib = ffi.load('/path/to/lib') local mt = {} mt.__index = mt mt.method = lib.A_method ffi.metatype('A', mt) LuaJIT Foreign Function Interface But, in case of C++
121. ffi.cdef[[ typedef struct A A; //and then declare identical functions ]] local lib = ffi.load('/path/to/lib') local mt = {} mt.__index = mt mt.method = lib.A_method ffi.metatype('A', mt) local a = ffi.gc(lib.A_new(), lib.A_gc) LuaJIT Foreign Function Interface But, in case of C++
122. How to reuse header.h in case of namespace? LuaJIT Foreign Function Interface Our problems & solutions
123. How to reuse header.h in case of namespace? //after including binding target classes using namespace ns; extern "C" { #include "header.h" } LuaJIT Foreign Function Interface Our problems & solutions
124. How to reuse header.h in case of namespace? //after including binding target classes using namespace ns; extern "C" { #include "header.h" } A* A_new() { ... } void A_method(A* self) { ... } void A_gc(A* self) { ... } LuaJIT Foreign Function Interface Our problems & solutions
125. shared_ptrs are all over the place! LuaJIT Foreign Function Interface Our problems & solutions
126. shared_ptrs are all over the place! //No problem typedef shared_ptr<A> pA; just use pA in place of A, it's like a boxed pointer LuaJIT Foreign Function Interface Our problems & solutions
127. C++ is the main body, Lua is not LuaJIT Foreign Function Interface Our problems & solutions
128. C++ is the main body, Lua is not Use Lua/C API to push a handle to Lua side at the very beginning. And everything goes. LuaJIT Foreign Function Interface Our problems & solutions
129. C++ is the main body, Lua is not Use Lua/C API to push a handle to Lua side at the very beginning. And everything goes. (have to export a set of methods of this handle to Lua, too) LuaJIT Foreign Function Interface Our problems & solutions
130. Some of our binding codes and usages: 1. https://github.com/godfat/cubeat/blob/master/cubeat-core/include/script/ai/bindings.h 2. https://github.com/godfat/cubeat/blob/master/cubeat-core/src/script/ai/bindings.cpp 3. https://github.com/godfat/cubeat/blob/master/cubeat-core/rc/script/ai/ai.lua 4. https://github.com/godfat/cubeat/blob/master/cubeat-core/rc/script/ai/normal.lua LuaJIT Foreign Function Interface Our problems & solutions
131. Lua is cool and LuaJIT is fast LuaJIT FFI is a powerful tool for bindings Binding automation can still be done (and a lot easier), Conclusion
132. Lua is cool and LuaJIT is fast LuaJIT FFI is a powerful tool for bindings Binding automation can still be done (and a lot easier), IF NECESSARY Conclusion
133. www.lua.org luajit.org lua-users.org Lua mailing list http://en.wikipedia.org/wiki/Lua_%28programming_language%29 github.com/godfat/cubeat References
134. Thank You! Q & A arch.jslin@gmail.com gplus.to/jslin www.plurk.com/johnson_lin