Gr8conf EU 2013 Speed up your development: GroovyServ and Grails Improx Plugin

Transcript

1. Speed Up Your Development: GroovyServ & Grails Improx Plugin @nobeans Yasuharu NAKANO
2. GroovyServ
3. In case of Developing a Groovy Script
4. Write Run Common Strategy: “Trial and Error”
5. $  time  groovy  -­‐e  “println  ‘Hello,  Groovy’” Hello,  Groovy real        0m0.823s user        0m1.016s sys          0m0.096s But, It takes 1 sec!
6. N times Trial and Error takes N sec.
7. “Many a little makes a mickle”
8. Waiting time makes me Forget next action
9. Why so slow? • JVM initialization • Loading Many Many Classes • .... from Disk http://en.wikipedia.org/wiki/Java_performance#Startup_time “ It seems that much of the startup time is due to IO- bound operations rather than JVM initialization or class loading (the rt.jar class data file alone is 40 MB and the JVM must seek a lot of data in this huge file).
10. Doom of JVM Languages?
11. Any solutions?
12. $  groovyConsole
13. Demo groovyConsole
14. groovyConsole • Handy for trying Java/Groovy APIs, etc. • But features as editor aren’t so rich: • No code completion • No syntax checking • No auto import • etc. • I want to use my favorite IDE/editor!
15. Other solutions?
16. GroovyServ “GroovyServ makes Groovy’s startup time much faster, by pre-invoking Groovy as a server.” http://kobo.github.io/groovyserv/
17. Demo Groovy and GroovyServ
18. $  time  groovyclient  -­‐e  “println  ‘Hello,  GroovyServ’” Hello,  GroovyServ real        0m0.031s user        0m0.001s sys          0m0.002s Normal  Groovy: real        0m0.823s user        0m1.016s sys          0m0.096s
19. Normal Groovy User Groovy Script Shell Environment Java VM File System C LASSPATH cm d line args stdin stdout stderr Ctrl-C other EN Vs exit status System .in System .out System .err groovy User Groovy Script
20. Shell Environment Java VM File System User Groovy Script C LASSPATH cm d line args stdin stdout stderr Ctrl-C other EN Vs ≪file≫ Authtoken exit status TCP/IP System .in System .out System .err groovyserver groovyclient GroovyServ
21. In case of Developing a Groovy Script
22. GroovyServ Way • Write code with your favorite IDE/editor • Run the code by using GroovyServ as External Tool • You can also see the result on a console of IDE
23. Demo GroovyServ way
24. Available Editors/IDEs • IntelliJ IDEA • Eclipse/STS/GGTS • Sublime Text 2 • Vim + QuickRun.vim • and more...
25. GroovyServ “GroovyServ makes Groovy’s startup time much faster, by pre-invoking Groovy as a server.” http://kobo.github.io/groovyserv/
26. How to Install
27. How to Install • Downloadable Binaries for • Mac OS X (built at Mac OS X 10.8 (x86_64)) • Windows • bundled in Groovy Windows Installer • (built at WindowsXP (32bit) (x86)) • (built at Windows7 (64bit) (AMD64)) • Linux (i386) (linked with glibc 2.9) • Self-build (required Gradle) • Homebrew (only Mac) • $  brew  install  groovyserv • http://kobo.github.io/groovyserv/howtoinstall.html
28. No setting required • If there is a binary set, you can use it: • $  /xxx/groovyserv/bin/groovyclient • Adding to PATH, you can easily use it: • $  groovyclient
29. How to Use
30. Commands • groovyclient • groovyserver
31. groovyclient • Invokes a groovy file or one-liner code, by sending a request to server • Also invokes groovyserver process automatically if not exists
32. groovyserver • Invokes a JVM process as a server. • Should be used only when you want to specify special options. • e.g. • -­‐v                        verbose  option • -­‐-­‐allow-­‐from    for  remote  feature
33. Available for Other JVM Languages http://nobeans-en.blogspot.jp/2011/07/high-speed-start-up- jythonclojure-by.html
34. Jython $  alias  gythonserver="env  CLASSPATH=/your/jython.jar  groovyserver" $  alias  gython="groovyclient  -­‐e  'import  org.python.util.jython;   jython.main(args)'  -­‐-­‐" $  gythonserver  -­‐r ... $  time  gython  -­‐c  "print('Hello')" Hello real        0m0.057s user        0m0.001s sys          0m0.004s
35. Clojure $  alias  glojureserver="env  CLASSPATH=/your/clojure.jar  groovyserver" $  alias  glojure="groovyclient  -­‐e  'import  clojure.main;main.main(args)'  -­‐-­‐" $  glojureserver  -­‐r ... $  time  glojure  -­‐e  "(println  'Hello)" Hello real        0m0.053s user        0m0.001s sys          0m0.004s main  method  of closure.main  class
36. • A script to get schedule data from groupware • As editor’s macro • As mail filter • etc. Using at Runtime
37. Other Features • Access Control • Propagation of CLASSPATH • Propagation of Environment Variables • Handling System#exit() • Dynamic CWD
38. Improx Plugin codenamed:GrailsServ
39. In case of Developing Grails Application
40. Grails’ Test supports is Great.
41. How to run tests • All tests • grails  test-­‐app
42. While I’m writing a test, I want to run only it.
43. I want to run the test as fast as possible.
44. How to run a specified test fast • Run only a unit test • test-­‐app  unit:  <FQCN  of  TestClass> • Run only a integration test • test-­‐app  integration:  <FQCN  of  TestClass> • Run only a functional test • test-­‐app  functional:  <FQCN  of  TestClass>
45. Why does it need a test phase?
46. If test phase not specified when run unit test, integration phase is also run. $  cat  test/unit/test/app/sample/FooTests.groovy .... class  FooTests  {        void  testSomething()  {                //  do  nothing        } } $  time  grails  test-­‐app  FooTests |  Packaging  Grails  application..... |  Tests  PASSED  -­‐  view  reports  in  /xxx/test-­‐reports real        0m15.375s user        0m35.119s sys          0m1.858s Oops!
    Integration
    phase
    running!?
47. $  time  grails  test-­‐app  unit:  FooTests |  Completed  1  unit  test,  0  failed  in  546ms |  Tests  PASSED  -­‐  view  reports  in  /xxx/test-­‐reports real        0m7.303s user        0m18.068s sys          0m0.909s Less
    than
    a
    half
    of
    previous
    sample! Specifying a phase can avoid running an extra phase Without  a  phase: real        0m15.375s user        0m35.119s sys          0m1.858s
48. Required to run test-app • FQCN of target test class • Test phase of target test class
49. By the way
50. How do you run a test while writing it? • (A) By IDE support (GGTS, IntelliJ, etc.) • (B) By “grails test-app” as standalone • (C) By “grails test-app” via Interactive Mode • (D) Others
51. Interactive Mode • Very very useful • Makes command execution faster because the JVM doesn't have to be restarted for each command • $  grails • TAB completion is available • Since Grails 2.0
52. DEMO InteractiveMode/Pros
53. But still there is trouble • It needs frequent switching of windows for use while writing code in IDE/editor • To run a test, you must prepare: • FQCN of target test class • Test phase of target test class
54. DEMO InteractiveMode/Cons
55. I’m too tired from such Switching, Copy & Paste, etc...
56. Improx Plugin Provides the way of using interactive mode from other process via TCP. http://grails.org/plugin/improx
57. DEMO Improx Plugin
58. Improx Plugin • You can run a test • Faster than general IDE supports because using interactive mode • Directly from IDE/editor without complicated operations
59. Available Editors/IDEs • IntelliJ IDEA • Eclipse/STS/GGTS • Sublime Text 2 • Vim + QuickRun.vim • and more...
60. Interactive Mode PROXy
61. Grails Improx Shell Environment ≪script≫ improxClient TCP/IP ≪script≫ improxSmartInvoker ≪class≫ InteractiveMode ≪class≫ GrailsConsole ≪command≫ improx-start ≪command≫ improx-stop ≪command≫ improx-install-resources other clients ≪class≫ GrailsScriptRunner echo back delegate run script start stop grails command expand client scripts use
62. • As same as interactive mode • install-­‐plugin • uninstall-­‐plugin • Special built-in commands • create-­‐app • quit • stop-­‐app • exit • !(shell  command) • Using threads complexly • run-­‐app Unsupported Commands
63. How to Install
64. BuildConfig.groovy: plugins  { //...        compile  ":improx:0.2" //... } Install Plugin
65. Install Client Scripts $  grails  improx-­‐install-­‐resources |  Improx  resources  installed  successfully:  /xxx/improx-­‐resources $  chmod  +x  improx-­‐resources/scripts/*.sh $  mv  improx-­‐resources  $HOME/.improx-­‐resources If
    once
    you
    store
    them
    to
    global
     path,
    
    you
    don't
    need
    the
    preparation
     for
    other
    projects.
66. How to Use
67. Grails Commands • improx-­‐start • improx-­‐stop • improx-­‐install-­‐resources
68. improx-­‐start  /  improx-­‐stop • improx-­‐start • Starts an interactive mode proxy server • improx-­‐stop • Stops the interactive mode proxy server which is running
69. improx-­‐install-­‐resources • Installs “improx-resources” directory which has some client scripts into your application project improx-­‐resources/  !""  scripts            #""  improxClient.groovy            #""  improxClient.sh            #""  improxSmartInvoker.groovy            !""  improxSmartInvoker.sh
70. Client Scripts • improxClient.sh • improxClient.groovy • improxSmartInvoker.sh • improxSmartInvoker.groovy
71. improxClient.(sh|groovy)  <Command> • Runs a command in interactive mode via TCP port of improx server • Very simple
72. $  improxClient.sh  help Executing  'help'  via  interactive  mode  proxy... ...... Usage  (optionals  marked  with  *): grails  [environment]*  [options]*  [target]   [arguments]* ...(snipped)... test-­‐app tomcat uninstall-­‐plugin upgrade war wrapper $
73. improxSmartInvoker.(sh|groovy)  FILE_PATH • Invokes any *.groovy file in an appropriate way • Very useful for IDE/editor
74. How SMART is it?
75. grails>  test-­‐app  unit:  sample.SampleUnitTests $  improxSmartInvoker.sh  /path/to/yourApp/ test/unit/sample/SampleUnitTests.groovy Runs the following command on the interactive mode:
76. grails>  test-­‐app  integration:  sample.SampleIntegTests $  improxSmartInvoker.sh  /path/to/yourApp/test /integration/sample/SampleIntegTests.groovy Runs the following command on the interactive mode:
77. $  improxSmartInvoker.sh  /path/to/yourApp/test /functional/sample/SampleFuncTests.groovy grails  test-­‐app  functional:  sample.SampleFuncTests Because
    functional
    test
    doesn’t
    work
     well
    on
    interactive
    mode
    at
    present. Runs the following command as new standalone Grails process:
78. $  improxSmartInvoker.sh   /path/to/scriptDir/myTribialScript.groovy groovy  /path/to/scriptDir/myTribialScript.groovy If
    you've
    installed
    GroovyServ,
    the
    groovyclient
    is
     automatically
    used
    instead
    of
    groovy
    command. Runs the following command as a normal Groovy script:
79. All you have to do is to set improxSmartInvoker as External Tool of IDE/editor http://kobo.github.io/grails-improx/guide/integrationWithEditors.html
80. Simple Protocol
81. TCP • Request: COMMAND • Whole response is result of the command • You can also use as client • e.g. telnet, nc
82. $  telnet  localhost  8096 Trying  ::1... Connected  to  localhost. Escape  character  is  '^]'. help ...... Usage  (optionals  marked  with  *): grails  [environment]*  [options]*  [target]   [arguments]* ...(snipped)... uninstall-­‐plugin upgrade war wrapper $
83. HTTP • Request GET  /COMMAND  HTTP/[0-­‐9.]+ • Whole response is result of the command • You can also use as client • e.g. Web browser, wget, curl
85. • GroovyServ • http://kobo.github.io/groovyserv/index.html • https://github.com/kobo/groovyserv • Improx - Interactive Mode Proxy • http://kobo.github.io/grails-improx/guide/ • http://grails.org/plugin/improx • https://github.com/kobo/grails-improx • Demo code • https://github.com/nobeans/gr8confeu2013-demo
86. Appendix
87. GroovyServ Under the hood
88. Under the hood • Client-Server Communication • Access Control • Propagation of CLASSPATH • Propagation of Environment Variables • Handling System#exit() • Dynamic CWD
89. Client-Server Communication
91. Transfer of Standard In/Out/Err Stream
93. $  groovyclient  -­‐e  \ "println  ‘Hello,  GroovyServ’"
94. groovyclient ClientConnection OutputStream ≪script≫ println “Hello, GroovyServ!” ≪System.out≫ DynamicDelegatedPrintStream writes data converted based on a protocol into output stream of socket Hello, GroovyServ! if Channel header == ”err” To STDERR if Channel header == ”out” To STDOUT Client console
96. $  groovyclient  -­‐e  \ "System.in.eachLine{  println  it*5  }"
97. ClientConnection PipedOutputStream PipedInputStream groovyclient ≪thread≫ StreamRequestHandler ≪script≫ System.in.eachLine { ... } Connected rawData = Socket.inputStream.read() bodyData = convert(rawData) pipedOutputStream.write(bodyData) ≪System.in≫ DynamicDelegatedInputStream
98. Protocol over TCP
99. InvocationRequest 'Cwd:' <cwd> LF 'Arg:' <arg1> LF 'Arg:' <arg2> LF : 'Env:' <env1>=<value1> LF 'Env:' <env2>=<value2> LF : 'Cp:' <classpath> LF 'AuthToken:' <authToken> LF LF where: <cwd> is current working directory. <arg1>,<arg2>.. are commandline arguments which must be encoded by Base64. (optional) <env1>,<env2>.. are environment variable names which sent to the server. (optional) <value1>,<valeu2>.. are environment variable values which sent to the server. (optional) <classpath> is the value of environment variable CLASSPATH. (optional) <authToken> is authentication value which certify client is the user who invoked the server. LF is line feed (0x0a, '\n').
100. StreamRequest 'Size:' <size> LF LF <body from STDIN> where: <size> is the size of body to send to server. <size>==-1 means client exited. <body from STDIN> is byte sequence from standard input.
101. InvocationResponse 'Status:' <status> LF where: <status> is exit status of invoked groovy script.
102. StreamResponse 'Channel:' <id> LF 'Size:' <size> LF LF <body for STDERR/STDOUT> where: <id> is 'out' or 'err', where 'out' means standard output of the program. 'err' means standard error of the program. <size> is the size of chunk. <body from STDERR/STDOUT> is byte sequence from standard output/error.
103. Access Control
104. Against Evil Request From Evil Client
105. Client Address Checking / Authtoken • Request only from loopback address is allowed. • Authtoken is generated and stored into a file when server process starts. • $HOME/.groovy/groovyserver/authtoken-­‐<PORT> • Authtoken always is used for checking that the request is from valid owner user.
106. -­‐-­‐allow-­‐from • By default, A request only from loop-back address is available. • You can allow specified addresses. $  groovyserver  -­‐-­‐allow-­‐from  192.168.1.1 SECURITY RISK: Be careful when using this option
107. -­‐-­‐authtoken • By default, authtoken is generated automatically. • You can specify any string. SECURITY RISK: Be careful when using this option $  groovyserver  -­‐-­‐authtoken  MY_AUTHTOKEN
108. At your own risk
109. Propagation of CLASSPATH
110. Client and Server are Different Process
111. ≪process≫ groovyserver ≪process≫ groovyclient CLASSPATH= /my/classpath/xxx.jar CLASSPATH= /my/classpath/zzz.jar Individual Env for each
112. If GroovyServ were too simple, ...
113. ≪process≫ groovyserver ≪process≫ groovyclient CLASSPATH= /my/classpath/foo.jar
114. ≪process≫ groovyserver ≪process≫ groovyclient Please execute this! new foo.Foo().execute() CLASSPATH= /my/classpath/foo.jar
115. ≪process≫ groovyserver ≪process≫ groovyclient CLASSPATH= /my/classpath/foo.jar I don’t know its class! Please execute this! new foo.Foo().execute()
116. ≪process≫ groovyserver ≪process≫ groovyclient CLASSPATH= /my/classpath/foo.jar I don’t know its class! org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed: script_from_command_line: 1: unable to resolve class foo.Foo @ line 1, column 1. new foo.Foo().execute() ^ 1 error Please execute this! new foo.Foo().execute()
117. But GroovyServ is enough smart
118. ≪process≫ groovyserver ≪process≫ groovyclient Please execute this! new foo.Foo().execute() CLASSPATH= /my/classpath/foo.jar
119. ≪process≫ groovyserver ≪process≫ groovyclient CLASSPATH= /my/classpath/foo.jar Please execute this! new foo.Foo().execute() CLASSPATH= /my/classpath/foo.jar new foo.Foo().execute()
120. ≪process≫ groovyserver ≪process≫ groovyclient CLASSPATH= /my/classpath/foo.jar You got it! Please execute this! new foo.Foo().execute() CLASSPATH= /my/classpath/foo.jar
121. CLASSPATH which is propagated from client is Volatile
122. ≪process≫ groovyserver
123. ≪process≫ groovyserver ≪process≫ groovyclient CLASSPATH= /my/classpath/foo.jar OK Please execute this! new foo.Foo().execute() CLASSPATH= /my/classpath/foo.jar
124. ≪process≫ groovyserver Done
125. ≪process≫ groovyserver ≪process≫ groovyclient CLASSPATH= /my/classpath/bar.jar Please execute this! new bar.Bar().execute() CLASSPATH= /my/classpath/bar.jar OK
126. ≪process≫ groovyserver Done
127. How to specify CLASSPATH from Client
128. -classpath / -cp / --classpath $  groovyclient          -­‐cp  /my/classpath/foo.jar            -­‐e  “new  foo.Foo().execute()”
129. CLASSPATH env $  export  CLASSPATH=/my/classpath/foo.jar $  groovyclient  -­‐e  “new  foo.Foo().execute()”
130. Default CLASSPATH on Server
131. CLASSPATH env $  export  CLASSPATH=/my/classpath/foo.jar $  groovyserver
132. ≪process≫ groovyserver I feel some power... CLASSPATH= /my/classpath/foo.jar
133. ≪process≫ groovyserver ≪process≫ groovyclient CLASSPATH= /my/classpath/foo.jar Please execute this! new foo.Foo().execute() CLASSPATH= /my/classpath/foo.jar I already know foo.jar!
134. ≪process≫ groovyserver Done CLASSPATH= /my/classpath/foo.jar
135. ≪process≫ groovyserver ≪process≫ groovyclient CLASSPATH= /my/classpath/bar.jar Please execute this! new bar.Bar().execute() CLASSPATH= /my/classpath/bar.jar OK CLASSPATH= /my/classpath/foo.jar
136. ≪process≫ groovyserver Done CLASSPATH= /my/classpath/foo.jar
137. Propagation of Environment Variables
138. Handy with IDE/editor
139. ≪process≫ groovyserver ≪process≫ groovyclient CLASSPATH= /my/classpath/xxx.jar How to propagate env KEY_A_1=VALUE_A_1 KEY_A_2=VALUE_A_2 KEY_B=VALUE_B KEY_A_1=VALUE_A_1 KEY_B=VALUE_B KEY_A_2=VALUE_A_2
140. groovyclient options • -­‐Cenv  <substr> • Passes environment variables of which a name includes specified substr • -­‐Cenv-­‐all • pass all environment variables • -­‐Cenv-­‐exclude  <substr> • don't pass environment variables of which a name includes specified substr
141. ≪process≫ groovyserver ≪process≫ groovyclient CLASSPATH= /my/classpath/xxx.jar -­‐Cenv  _A_ KEY_A_1=VALUE_A_1 KEY_A_2=VALUE_A_2 KEY_B=VALUE_B KEY_A_1=VALUE_A_1 KEY_B=VALUE_B KEY_A_2=VALUE_A_2
142. ≪process≫ groovyserver ≪process≫ groovyclient CLASSPATH= /my/classpath/xxx.jar -­‐Cenv-­‐all KEY_A_1=VALUE_A_1 KEY_A_2=VALUE_A_2 KEY_B=VALUE_B KEY_A_1=VALUE_A_1 KEY_B=VALUE_B KEY_A_2=VALUE_A_2
143. ≪process≫ groovyserver ≪process≫ groovyclient CLASSPATH= /my/classpath/xxx.jar -­‐Cenv-­‐all  -­‐Cenv-­‐exclude  _2 KEY_A_1=VALUE_A_1 KEY_A_2=VALUE_A_2 KEY_B=VALUE_B KEY_A_1=VALUE_A_1 KEY_B=VALUE_B KEY_A_2=VALUE_A_2
144. Caution: ENV on server isn’t cleared after each invocation
145. ≪process≫ groovyserver ≪process≫ groovyclient CLASSPATH= /my/classpath/xxx.jar -­‐Cenv  KEY_A_1 KEY_A_1=VALUE_A_1 KEY_A_2=VALUE_A_2 KEY_B=VALUE_B KEY_A_1=VALUE_A_1
146. ≪process≫ groovyserver ≪process≫ groovyclient CLASSPATH= /my/classpath/xxx.jar -­‐Cenv  KEY_A_2 KEY_A_1=VALUE_A_1 KEY_A_2=VALUE_A_2 KEY_B=VALUE_B KEY_A_1=VALUE_A_1 KEY_A_2=VALUE_A_2
147. ≪process≫ groovyserver ≪process≫ groovyclient CLASSPATH= /my/classpath/xxx.jar -­‐Cenv  KEY_B KEY_A_1=VALUE_A_1 KEY_A_2=VALUE_A_2 KEY_B=VALUE_B KEY_A_1=VALUE_A_1 KEY_A_2=VALUE_A_2 KEY_B=VALUE_B
148. Handling System#exit()
149. Server process must not be terminated, even if a script executes System.exit()
150. SecurityManager public  class  NoExitSecurityManager2  extends  NoExitSecurityManager  {        public  void  checkExit(final  int  code)  {                throw  new  SystemExitException(code,                                            "called  System.exit("  +  code  +  ")");        } }
151. GroovyMain private  boolean  run()  {        try  {                //...                processArgs(...)                //...        }  catch  (SystemExitException  e)                throw  e;        }  catch  (Throwable  e)  {                //...        } } static  void  processArgs(...)  {        //...        if  (!process(cmd,  classpath))  {                //  Disabled  because  this  causes  a  secondary  disaster                //System.exit(1);        }        //... }
152. Dynamic CWD
153. CWD = Current Working Directory
154. $ cd /tmp $ groovyserver -r $ cd /home/nobeans $ cat test.txt Can you read me? $ groovyclient -e ‘println(new File(“test.txt”).text)’ Caught: java.io.FileNotFoundException: test.txt (No such file or directory) ...SNIP... ≪process≫ groovyserver CWD=/tmp I wanna use “test.txt” of current directory. Is it “/tmp/test.txt” ? Hmm, such a file not found.
155. CWD of client should be set to server for each invocation. But, ...
156. In Java, CWD is thoroughly hidden
157. Solution • Setting to “user.dir” system property • It affects File#getAbsolutePath() • Changing CWD of native JVM process • Many classes don’t use “user.dir” to complete an absolute path. • But, JNI is troublesome...
158. JNA is easy way to use native API UnixLibC.groovy: interface  UnixLibC  extends  com.sun.jna.Library.Library  {        int  chdir(String  dir)        int  setenv(String  envVarName,  String  envVarValue,  int  overwrite) } pom.xml:        <dependency>            <groupId>net.java.dev.jna</groupId>            <artifactId>jna</artifactId>            <version>3.2.2</version>        </dependency> Usage: def  libc  =  com.sun.jna.Native.loadLibrary("c",  UnixLibC.class) libc.chdir(“/tmp/foo/bar”)
159. $ cd /tmp $ groovyserver -r $ cd /home/nobeans $ cat test.txt Can you read me? $ groovyclient -e ‘println(new File(“test.txt”).text)’ Can you read me? ≪process≫ groovyserver CWD=/tmp I wanna use “test.txt” of current directory. ≪process≫ groovyserver CWD=/home/nobeans Is it “/home/nobeans/test.txt” ? OK, I got it. Yes!