Introduction to LuaJIT & How to bind CPP code base using LuaJIT FFI

0723b829421d651f25991ffb12ff0b55?s=47 IGDSHARE
April 15, 2012

Introduction to LuaJIT & How to bind CPP code base using LuaJIT FFI

This was presented at osdc.tw/2012, if you find anything wrong, please contact me!

igdshare@igdshare.org or
arch.jslin@gmail.com

0723b829421d651f25991ffb12ff0b55?s=128

IGDSHARE

April 15, 2012
Tweet

Transcript

  1. Introduction to LuaJIT & How to bind C++ code base

    using LuaJIT FFI 2012.04.14 OSDC.tw
  2. How to bind complex C++ code base using LuaJIT FFI

  3. About me Johnson Lin

  4. About me Johnson Lin Programmer & Indie Game

  5. About me Johnson Lin Programmer & Indie Game igdshare.org (

    獨立遊戲開發者分享會 )
  6. About me Johnson Lin Programmer & Indie Game igdshare.org &

    Game Jam! (4/28 ~ 4/29) ( 獨立遊戲開發者分享會 )
  7. About me Johnson Lin Programmer & Indie Game igdshare.org &

    Game Jam! (4/28 ~ 4/29) ( 獨立遊戲開發者分享會 ) archilife.org ( 祐生研究基金會 )
  8. About me Johnson Lin Programmer & Indie Game igdshare.org &

    Game Jam! (4/28 ~ 4/29) ( 獨立遊戲開發者分享會 ) archilife.org & Indie game contests scholarship ( 祐生研究基金會 )
  9. Overview Introduction to Lua Introduction to LuaJIT How does Lua

    talk to C/C++ LuaJIT FFI (Foreign Function Interface)
  10. Overview Introduction to Lua Introduction to LuaJIT How does Lua

    talk to C/C++ LuaJIT FFI (Foreign Function Interface)
  11. Introduction to LUA Introduction to LUA

  12. Introduction to LUA Introduction to LUA LUA is...

  13. Introduction to LUA Introduction to LUA LUA is... Lua

  14. Introduction to LUA Introduction to LUA LUA is... Lua Uppercase

  15. Introduction to LUA Introduction to LUA LUA is... Lua Uppercase

    Accident This pun came from a clever guy Joseph Manning: http://lua-users.org/lists/lua-l/2011-06/msg01112.html
  16. Introduction to Lua Lua = "Moon" in Portuguese Introduction to

    Lua
  17. Introduction to Lua Lua = "Moon" in Portuguese Introduction to

    Lua
  18. Introduction to Lua Lua = "Moon" in Portuguese Use Cases

    (Game Dev): Introduction to Lua
  19. Introduction to Lua Lua = "Moon" in Portuguese Use Cases

    (Game Dev): Grim Fandango, Introduction to Lua
  20. Introduction to Lua Lua = "Moon" in Portuguese Use Cases

    (Game Dev): Grim Fandango, World of Warcraft UI, Introduction to Lua
  21. Introduction to Lua Lua = "Moon" in Portuguese Use Cases

    (Game Dev): Grim Fandango, World of Warcraft UI, Angry Birds Introduction to Lua
  22. Introduction to Lua By Tecgraf / PUC-Rio (1993): Roberto Ierusalimschy,

    Luiz Henrique de Figueiredo, Waldemar Celes Introduction to Lua
  23. Introduction to Lua By Tecgraf / PUC-Rio (1993): Roberto Ierusalimschy,

    Luiz Henrique de Figueiredo, Waldemar Celes Current version: 5.2.0 / 5.1.5 MIT License since 5.0 (BSD License before then) Introduction to Lua
  24. Lua Features Introduction to Lua

  25. Lua Features Fast Fastest among other well-known interpreted language Introduction

    to Lua
  26. Lua Features Fast Fastest among other well-known interpreted language Small

    ~150kB binary, ~20kB mem overhead per interpreter Introduction to Lua
  27. Lua Features Fast Fastest among other well-known interpreted language Small

    ~150kB binary, ~20kB mem overhead per interpreter Simple 22 keywords, 27 tokens Introduction to Lua
  28. Lua Features Multithreading-Friendly multiple interpreters in multiple threads concurrently without

    GIL Introduction to Lua
  29. Lua Features Multithreading-Friendly multiple interpreters in multiple threads concurrently without

    GIL Easy to Embed & Extend concise Lua/C API Introduction to Lua
  30. Lua Features Multithreading-Friendly multiple interpreters in multiple threads concurrently without

    GIL Easy to Embed & Extend concise Lua/C API But, without "battery" although Luarocks is in place Introduction to Lua
  31. Lua Features Introduction to Lua Multi-paradigm:

  32. Lua Features Introduction to Lua Multi-paradigm: procedural functional prototype-based OO

  33. Lua Features Introduction to Lua Multi-paradigm: procedural functional prototype-based OO

    Think Lua as JavaScript
  34. Lua Features Introduction to Lua Multi-paradigm: procedural functional prototype-based OO

    Think Lua as JavaScript .....without the Bad Parts
  35. Z Combinator in Lua Introduction to Lua local Y =

    function(f) return (function(x) return f (function(y) return (x(x))(y) end) end) (function(x) return f (function(y) return (x(x))(y) end) end) end
  36. Overview Introduction to Lua Introduction to LuaJIT How does Lua

    talk to C/C++ LuaJIT FFI (Foreign Function Interface)
  37. Introduction to LuaJIT Introduction to LuaJIT

  38. Introduction to LuaJIT Just-In-Time compiler for Lua Introduction to LuaJIT

  39. Introduction to LuaJIT Just-In-Time compiler for Lua Mike Pall Introduction

    to LuaJIT
  40. Introduction to LuaJIT Just-In-Time compiler for Lua Mike Pall LuaJIT1

    (2005~, current 1.1.7) LuaJIT2 (2009~, current 2.0-beta9) Introduction to LuaJIT
  41. LuaJIT2 Features Interpreter written in assembler (DynASM) Introduction to LuaJIT

  42. LuaJIT2 Features Interpreter written in assembler (DynASM) Trace compiler Introduction

    to LuaJIT
  43. LuaJIT2 Features Interpreter written in assembler (DynASM) Trace compiler x86/64

    (Windows, OSX, Linux, etc) SSE2 instructions required Introduction to LuaJIT
  44. LuaJIT2 Features Interpreter written in assembler (DynASM) Trace compiler x86/64

    (Windows, OSX, Linux, etc) SSE2 instructions required ARM (Android, iOS, etc), PPC, MIPS only the interpreter part on iOS works now, blame Apple Introduction to LuaJIT
  45. LuaJIT2 Features API & ABI compatible with Lua 5.1 Bytecode

    NOT compatible Introduction to LuaJIT
  46. LuaJIT2 Features API & ABI compatible with Lua 5.1 Bytecode

    NOT compatible Some Lua 5.2 functionality But not going to fully catch up anytime soon Introduction to LuaJIT
  47. LuaJIT2 Features API & ABI compatible with Lua 5.1 Bytecode

    NOT compatible Some Lua 5.2 functionality But not going to fully catch up anytime soon Foreign Function Interface with C parser and a few extensions Introduction to LuaJIT
  48. LuaJIT - Behind the Scene Introduction to LuaJIT

  49. LuaJIT - Behind the Scene Introduction to LuaJIT Too hard

    for me! Sorry!
  50. Some Use Cases of LuaJIT AI Implementation in The Settlers

    7 (2010) GSL Shell https://github.com/malkia/ufo Includes OpenGL, OpenCL, SDL, ZeroMQ bindings.. OpenResty web framework based on nginx Introduction to LuaJIT
  51. Some benchmarks Game of Life C++ v.s. LuaJIT (using FFI)

    Introduction to LuaJIT
  52. Some benchmarks Game of Life C++ v.s. LuaJIT (using FFI)

    Please keep in mind this benchmark is pretty crude and favors LuaJIT, it just shows one extreme. Introduction to LuaJIT
  53. Some benchmarks Game of Life Introduction to LuaJIT

  54. Some benchmarks Introduction to LuaJIT Pueudo Code: while (true) {

    game.update(); game.render(); end //both update & render has a big 2d loop //doing state update and drawing cells respectively https://github.com/arch-jslin/mysandbox/tree/master/cpp/lifegame_sdl
  55. Some benchmarks Introduction to LuaJIT Pueudo Code: while true do

    game:update() -- JIT kick in game:render() -- JIT kick in end --both update & render has a big 2d loop --doing state update and drawing cells respectively https://github.com/arch-jslin/mysandbox/blob/master/lua/lifegame_ffi_sdl_gl.lua
  56. Some benchmarks Introduction to LuaJIT Pueudo Code: while true do

    game:update() -- JIT back off game:render() -- JIT kick in end --both update & render has a big 2d loop --doing state update and drawing cells respectively
  57. Some benchmarks Introduction to LuaJIT Pueudo Code: while true do

    game:update() -- JIT back off game:render() -- JIT back off end --both update & render has a big 2d loop --doing state update and drawing cells respectively
  58. LuaJIT Roadmap Introduction to LuaJIT

  59. LuaJIT Roadmap LuaJIT 2.0 almost feature freeze ARM hard-fp support

    looking for sponsor Introduction to LuaJIT
  60. LuaJIT Roadmap LuaJIT 2.0 almost feature freeze ARM hard-fp support

    looking for sponsor LuaJIT 2.1 garbage collector redesign Introduction to LuaJIT
  61. LuaJIT Roadmap LuaJIT 2.0 almost feature freeze ARM hard-fp support

    looking for sponsor LuaJIT 2.1 garbage collector redesign Other unconfirmed: FFI C preprocessor? Introduction to LuaJIT
  62. Overview Introduction to Lua Introduction to LuaJIT How does Lua

    talk to C/C++ LuaJIT FFI (Foreign Function Interface)
  63. How does Lua talk to C/C++ How does Lua talk

    to C/C++ Manipulate lua_State (Lua Stack) Using Lua/C API core library: lua_xxx(); aux library: luaL_xxx();
  64. Lua/C API -- Lua local a, b = func_from_c(2,3) How

    does Lua talk to C/C++
  65. Lua/C API -- Lua local a, b = func_from_c(2,3) //C

    static int func_from_c(lua_State* L) { int x = luaL_checkint(L, 1); int y = luaL_checkint(L, -1); lua_pushinteger(L, x+y); lua_pushinteger(L, x-y); return 2; } How does Lua talk to C/C++
  66. Lua/C API -- Lua local a, b = func_from_c(2,3) //C

    static int func_from_c(lua_State* L) { int x = luaL_checkint(L, 1); int y = luaL_checkint(L, -1); lua_pushinteger(L, x+y); lua_pushinteger(L, x-y); return 2; } How does Lua talk to C/C++
  67. What about C++ object? How does Lua talk to C/C++

    -- Lua local o = Simple(1) -- C++ obj
  68. What about C++ object? How does Lua talk to C/C++

    -- Lua local o = Simple(1) -- C++ obj //C //Let's see some actual code...
  69. What about C++ object? How does Lua talk to C/C++

    So that's a bit scary..
  70. Binding Libaries How does Lua talk to C/C++

  71. Binding Libaries Luabind - boost & template magic How does

    Lua talk to C/C++
  72. Binding Libaries Luabind - boost & template magic OOLua -

    Macro & template(?) magic How does Lua talk to C/C++
  73. Binding Libaries Luabind - boost & template magic OOLua -

    Macro & template(?) magic Simple Lua Binder - no boost, but still templates How does Lua talk to C/C++
  74. Binding Libaries Luabind - boost & template magic OOLua -

    Macro & template(?) magic Simple Lua Binder - no boost, but still templates ...etc (post talk addition: I am not saying these are bad or what, just our considerations. If you find these tools "just work" for you, then you probably have no reason to abandon them.) How does Lua talk to C/C++
  75. Binding Generators tolua & tolua++ How does Lua talk to

    C/C++
  76. Binding Generators tolua & tolua++ SWIG (post talk addition: I

    am not saying these are bad or what, just our considerations. If you find these tools "just work" for you, then you probably have no reason to abandon them.) How does Lua talk to C/C++
  77. Problem with those How does Lua talk to C/C++

  78. Problem with those Missing feature How does Lua talk to

    C/C++
  79. Problem with those Missing feature Code bloat How does Lua

    talk to C/C++
  80. Problem with those Missing feature Code bloat Runtime overhead How

    does Lua talk to C/C++
  81. Problem with those Missing feature Code bloat Runtime overhead Yet

    another glue language How does Lua talk to C/C++
  82. Problem with those Missing feature Code bloat Runtime overhead Yet

    another glue language Not sure what to do in case of bugs How does Lua talk to C/C++
  83. LuaJIT Foreign Function Interface LuaJIT FFI

  84. Can be JIT compiled Unlike libffi based modules or other

    binding methods LuaJIT Foreign Function Interface LuaJIT FFI
  85. Can be JIT compiled Unlike libffi based modules or other

    binding methods Has a C parser built-in LuaJIT Foreign Function Interface LuaJIT FFI
  86. Can be JIT compiled Unlike libffi based modules or other

    binding methods Has a C parser built-in Pure Lua Syntax So there's got LuaFFI => https://github.com/jmckaskill/luaffi LuaJIT Foreign Function Interface LuaJIT FFI
  87. -- Using C type local ffi = require 'ffi' LuaJIT

    Foreign Function Interface Examples of LuaJIT FFI
  88. -- Using C type local ffi = require 'ffi' local

    a = ffi.new("int") LuaJIT Foreign Function Interface Examples of LuaJIT FFI
  89. -- Using C type local ffi = require 'ffi' local

    a = ffi.new("int") local b = ffi.new("char const*") LuaJIT Foreign Function Interface Examples of LuaJIT FFI
  90. -- Using C type local ffi = require 'ffi' local

    a = ffi.new("int") local b = ffi.new("char const*") local c = ffi.new("double[10]") LuaJIT Foreign Function Interface Examples of LuaJIT FFI
  91. -- Using C type local ffi = require 'ffi' local

    a = ffi.new("int") local b = ffi.new("char const*") local c = ffi.new("double[10]") local d = ffi.new("double[?]", 20) LuaJIT Foreign Function Interface Examples of LuaJIT FFI
  92. -- Calling external C functions local ffi = require 'ffi'

    LuaJIT Foreign Function Interface Examples of LuaJIT FFI
  93. -- Calling external C functions local ffi = require 'ffi'

    local C = ffi.C LuaJIT Foreign Function Interface Examples of LuaJIT FFI
  94. -- Calling external C functions local ffi = require 'ffi'

    local C = ffi.C ffi.cdef[[ int printf(const char*, ...); ]] LuaJIT Foreign Function Interface Examples of LuaJIT FFI
  95. -- Calling external C functions local ffi = require 'ffi'

    local C = ffi.C ffi.cdef[[ int printf(const char*, ...); ]] C.printf('Hello %s!\n', 'world') LuaJIT Foreign Function Interface Examples of LuaJIT FFI
  96. -- Calling external C functions local ffi = require 'ffi'

    local C = ffi.C ffi.cdef[[ int printf(const char*, ...); ]] C.printf('Hello %s!\n', 'world') C.printf('%d', ffi.cast('int', 3)) LuaJIT Foreign Function Interface Examples of LuaJIT FFI
  97. -- Calling external C functions local ffi = require 'ffi'

    LuaJIT Foreign Function Interface Examples of LuaJIT FFI
  98. -- Calling external C functions local ffi = require 'ffi'

    ffi.cdef[[ typedef struct { int x,y,z; } Point; void do_stuff(Point*); ]] LuaJIT Foreign Function Interface Examples of LuaJIT FFI
  99. -- Calling external C functions local ffi = require 'ffi'

    ffi.cdef[[ typedef struct { int x,y,z; } Point; void do_stuff(Point*); ]] LuaJIT Foreign Function Interface Examples of LuaJIT FFI
  100. -- Calling external C functions local ffi = require 'ffi'

    ffi.cdef[[ typedef struct { int x,y,z; } Point; void do_stuff(Point*); ]] LuaJIT Foreign Function Interface Examples of LuaJIT FFI
  101. -- Calling external C functions local ffi = require 'ffi'

    ffi.cdef[[ typedef struct { int x,y,z; } Point; void do_stuff(Point*); ]] local Lib = ffi.load('path/to/your_c_lib') LuaJIT Foreign Function Interface Examples of LuaJIT FFI
  102. -- Calling external C functions local ffi = require 'ffi'

    ffi.cdef[[ typedef struct { int x,y,z; } Point; void do_stuff(Point*); ]] local Lib = ffi.load('path/to/your_c_lib') local a = ffi.new('Point', 1, 2, 3) LuaJIT Foreign Function Interface Examples of LuaJIT FFI
  103. -- Calling external C functions local ffi = require 'ffi'

    ffi.cdef[[ typedef struct { int x,y,z; } Point; void do_stuff(Point*); ]] local Lib = ffi.load('path/to/your_c_lib') local a = ffi.new('Point', 1, 2, 3) Lib.do_stuff(a) LuaJIT Foreign Function Interface Examples of LuaJIT FFI
  104. -- Using C header directly (almost) local ffi = require

    'ffi' LuaJIT Foreign Function Interface Examples of LuaJIT FFI
  105. -- Using C header directly (almost) local ffi = require

    'ffi' ffi.cdef(io.open('header.h', 'r'):read('*a')) LuaJIT Foreign Function Interface Examples of LuaJIT FFI
  106. -- Using C header directly (almost) local ffi = require

    'ffi' ffi.cdef(io.open('header.h', 'r'):read('*a')) -- This header.h has to be preprocessed LuaJIT Foreign Function Interface Examples of LuaJIT FFI
  107. -- Using C header directly (almost) local ffi = require

    'ffi' ffi.cdef(io.open('header.h', 'r'):read('*a')) -- This header.h has to be preprocessed -- gcc -E stub.c | grep -v '^#' LuaJIT Foreign Function Interface Examples of LuaJIT FFI
  108. -- Using C header directly (almost) local ffi = require

    'ffi' ffi.cdef(io.open('header.h', 'r'):read('*a')) -- This header.h has to be preprocessed -- gcc -E stub.c | grep -v '^#' -- stub.c only contain #include <header.h> LuaJIT Foreign Function Interface Examples of LuaJIT FFI
  109. //C++ side class A { public: void method(); }; LuaJIT

    Foreign Function Interface But, in case of C++
  110. //C++ side class A { public: void method(); }; extern

    "C" { A* A_new() { return new A; } void A_method(A* self) { self->method(); } void A_gc(A* self) { delete self; } } LuaJIT Foreign Function Interface But, in case of C++
  111. //C++ side class A { public: void method(); }; extern

    "C" { A* A_new() { return new A; } void A_method(A* self) { self->method(); } void A_gc(A* self) { delete self; } } LuaJIT Foreign Function Interface But, in case of C++
  112. //C++ side class A { public: void method(); }; extern

    "C" { A* A_new() { return new A; } void A_method(A* self) { self->method(); } void A_gc(A* self) { delete self; } } LuaJIT Foreign Function Interface But, in case of C++
  113. //C++ side class A { public: void method(); }; extern

    "C" { //as before, export these A* A_new() { return new A; } void A_method(A* self) { self->method(); } void A_gc(A* self) { delete self; } } LuaJIT Foreign Function Interface But, in case of C++
  114. ffi.cdef[[ typedef struct A A; //and then declare identical functions

    ]] LuaJIT Foreign Function Interface But, in case of C++
  115. ffi.cdef[[ typedef struct A A; //and then declare identical functions

    ]] LuaJIT Foreign Function Interface But, in case of C++
  116. ffi.cdef[[ typedef struct A A; //and then declare identical functions

    ]] local lib = ffi.load('/path/to/lib') LuaJIT Foreign Function Interface But, in case of C++
  117. ffi.cdef[[ typedef struct A A; //and then declare identical functions

    ]] local lib = ffi.load('/path/to/lib') local mt = {} LuaJIT Foreign Function Interface But, in case of C++
  118. ffi.cdef[[ typedef struct A A; //and then declare identical functions

    ]] local lib = ffi.load('/path/to/lib') local mt = {} mt.__index = mt LuaJIT Foreign Function Interface But, in case of C++
  119. ffi.cdef[[ typedef struct A A; //and then declare identical functions

    ]] local lib = ffi.load('/path/to/lib') local mt = {} mt.__index = mt mt.method = lib.A_method LuaJIT Foreign Function Interface But, in case of C++
  120. ffi.cdef[[ typedef struct A A; //and then declare identical functions

    ]] local lib = ffi.load('/path/to/lib') local mt = {} mt.__index = mt mt.method = lib.A_method ffi.metatype('A', mt) LuaJIT Foreign Function Interface But, in case of C++
  121. ffi.cdef[[ typedef struct A A; //and then declare identical functions

    ]] local lib = ffi.load('/path/to/lib') local mt = {} mt.__index = mt mt.method = lib.A_method ffi.metatype('A', mt) local a = ffi.gc(lib.A_new(), lib.A_gc) LuaJIT Foreign Function Interface But, in case of C++
  122. How to reuse header.h in case of namespace? LuaJIT Foreign

    Function Interface Our problems & solutions
  123. How to reuse header.h in case of namespace? //after including

    binding target classes using namespace ns; extern "C" { #include "header.h" } LuaJIT Foreign Function Interface Our problems & solutions
  124. How to reuse header.h in case of namespace? //after including

    binding target classes using namespace ns; extern "C" { #include "header.h" } A* A_new() { ... } void A_method(A* self) { ... } void A_gc(A* self) { ... } LuaJIT Foreign Function Interface Our problems & solutions
  125. shared_ptrs are all over the place! LuaJIT Foreign Function Interface

    Our problems & solutions
  126. shared_ptrs are all over the place! //No problem typedef shared_ptr<A>

    pA; just use pA in place of A, it's like a boxed pointer LuaJIT Foreign Function Interface Our problems & solutions
  127. C++ is the main body, Lua is not LuaJIT Foreign

    Function Interface Our problems & solutions
  128. C++ is the main body, Lua is not Use Lua/C

    API to push a handle to Lua side at the very beginning. And everything goes. LuaJIT Foreign Function Interface Our problems & solutions
  129. C++ is the main body, Lua is not Use Lua/C

    API to push a handle to Lua side at the very beginning. And everything goes. (have to export a set of methods of this handle to Lua, too) LuaJIT Foreign Function Interface Our problems & solutions
  130. Some of our binding codes and usages: 1. https://github.com/godfat/cubeat/blob/master/cubeat-core/include/script/ai/bindings.h 2.

    https://github.com/godfat/cubeat/blob/master/cubeat-core/src/script/ai/bindings.cpp 3. https://github.com/godfat/cubeat/blob/master/cubeat-core/rc/script/ai/ai.lua 4. https://github.com/godfat/cubeat/blob/master/cubeat-core/rc/script/ai/normal.lua LuaJIT Foreign Function Interface Our problems & solutions
  131. Lua is cool and LuaJIT is fast LuaJIT FFI is

    a powerful tool for bindings Binding automation can still be done (and a lot easier), Conclusion
  132. Lua is cool and LuaJIT is fast LuaJIT FFI is

    a powerful tool for bindings Binding automation can still be done (and a lot easier), IF NECESSARY Conclusion
  133. www.lua.org luajit.org lua-users.org Lua mailing list http://en.wikipedia.org/wiki/Lua_%28programming_language%29 github.com/godfat/cubeat References

  134. Thank You! Q & A arch.jslin@gmail.com gplus.to/jslin www.plurk.com/johnson_lin