Testing and developing with Google AppEngine

This talk is about issues that might appear while using Google AppEngine and presented several solutions have to fix those and how to test them also.

Transcript

1. HEISENBUG TESTING AND DEVELOPING WITH GOOGLE CONFERENCE MOSCOW 2017
2. VSEVOLOD BREKELOV Software developer @ @brekelov @volekerb
3. WHOM IS THIS TALK FOR? 3 QA DEVELOPER
4. WHAT IS THIS ABOUT? TESTING AND BUILDING APP WITH GOOGLE TOOLS AND GOOLE TECHNOLOGIES ▸ Google cloud environment types ▸ Limits within environment ▸ Issues which might appear due to limits and workarounds ▸ Cloud Memcache service ▸ Datastore (database) ▸ Bazel (build system) 4
5. WON’T BE COVERED IT'S NOT MARKETING TALK 5 ▸ Comparison with Amazon, Azure, Heroku ▸ Firebase ▸ Docker ▸ Angular or any UI things ▸ CI/CD
6. ENVIRONMENT TYPES
7. GOOGLE CLOUD STACK OVERFLOW: STANDARD VS FLEXIBLE 7
8. GOOGLE CLOUD Documentation link 8 ENVIRONMENT TYPES Flexible Standard Run within Docker containers Run in a sandbox Standard has faster auto scaling Any programming language Limited number of supported languages Not free Free or very low cost • Python 2.7 • Java 7, Java 8 • PHP 5.5 • Go 1.6
9. ПС WEB APPLICATION 9 BACK END FRONT END DB Google Cloud
10. LET THE STORY BEGIN
11. FIRST ISSUE HARD TIME LIMIT com.google.apphosting.api.DeadlineExceededException: 
 This request (0000015e4fbf804e) started at 2017/09/05 01:55:03.886 UTC and was still executing at 2017/09/05 01:56:03.821 UTC. 11
12. 30/60 Seconds API response limit
13. STANDARD ENVIRONMENT LIMITATIONS RESPONSE FROM BE TO FE SHOULD BE WITHIN 30/60 SECONDS com.google.apphosting.api.
 DeadlineExceededException while(getTweets) { try { QueryResult result = twitter.search(query); if (result.getTweets() == null || result.getTweets().isEmpty()) { getTweets = false; } else { int forCount = 0; for (Status status: result.getTweets()) { if (whileCount == 0 && forCount == 0) { sinceId = status.getId(); } if (forCount == result.getTweets().size()-1) { maxId = status.getId(); } forCount++; } 13
14. HOW TO FIX?
16. STANDARD ENVIRONMENT LIMITATIONS Send tasks Distribute tasks WORKERS PATTERN 16 CLIENT BROKER WORKER WORKER WORKER
17. https://cloud.google.com/appengine/docs/standard/java/taskqueue/
18. STANDARD ENVIRONMENT LIMITATIONS Code with external API call
 Code within for(;;) cycle
 
 
 
 
 TASK Yet another code QUEUE 18 twitter.search(query) for (Status status: result.getTweets()) {
 if (whileCount == 0 && forCount == 0) {
 sinceId = status.getId();
 }
 if (forCount == result.getTweets().size()-1) {
 maxId = status.getId();
 ….
 } TASK TASK
19. WHAT TASKS CODE LOOKS LIKE Queue queue = QueueFactory.getQueue("twitterSearch"); queue.add(TaskOptions.Builder.withUrl("/task/searchTwitter")
 .param(“query", query)); @RestController
 @RequestMapping("/task/searchTwitter")
 public final class TwitterWorker { private final TwitterClient twitterClient; @Autowired
 public TwitterWorker(TwitterClient twitterClient) {
 this.twitterClient = twitterClient;
 } @RequestMapping(method = RequestMethod.POST)
 public void processTask(String query) throws TwitterException {
 QueryResult result = twitterClient.search(query); } queue.xml
 <queue>
 <name>twitterSearch</name>
 <rate>20/s</rate>
 <bucket-size>40</bucket-size>
 </queue> 19
20. HOW TO TEST?
21. HOW TO FIND LOAD TESTS FOR EACH API ENDPOINT /GET ALL USERS /GET COLLECTION OF SMTH /DO BIG QUERY REQUEST Simulate load Backend 21
22. HOW TO WRITE UNIT TESTS private final LocalServiceTestHelper helper = new LocalServiceTestHelper(new LocalTaskQueueTestConfig()); @Before public void setUp() { helper.setUp(); } 22
23. HOW TO WRITE UNIT TESTS @Test
 public void simpleTest(){
 
 QueueFactory.getDefaultQueue()
 .add(TaskOptions.Builder
 .withParam(“twitterSearch", "Heisenbug")
 .taskName("task")); LocalTaskQueue ltq = LocalTaskQueueTestConfig.getLocalTaskQueue(); QueueStateInfo qsi = ltq.getQueueStateInfo().get(QueueFactory.getDefaultQueue().getQueueName()); assertEquals("task", qsi.getTaskInfo().get(0).getTaskName()); assertEquals("twitterQuery=Heisenbug", qsi.getTaskInfo().get(0).getBody()); assertEquals(1, qsi.getTaskInfo().size()); } 23
24. COM.GOOGLE.APPHOSTING.API.DEADLINEEXCEEDEDEXCEPTION AGAIN?!
25. STANDARD ENVIRONMENT LIMITATIONS Code with external API call Code within for(;;) cycle TASK Yet another code TASK TASK QUEUE 60 seconds limit 25
26. SOLUTION BEFORE ACTION 1 ACTION 2 ACTION 3 Task ACTION 1 Task 1 AFTER ACTION 2 Task 2 ACTION 3 Task 3 26
27. TASK CHAINING EXAMPLE @RestController @RequestMapping("/task/searchTwitter") public class TwitterWorker { private final TwitterClient twitterClient; private final TaskCreator taskCreator; @Autowired public TwitterWorker(TwitterClient twitterClient, TaskCreator taskCreator) { this.twitterClient = twitterClient; this.taskCreator = taskCreator; } @RequestMapping(method = RequestMethod.POST) public void processTask(String query) throws TwitterException { QueryResult result = twitterClient.search(query); List<Status> tweets = result.getTweets(); for (Status tweet : tweets) { String userName = tweet.getUser().getName(); taskCreator.createInstagramTask(userName); } } } 27 @RestController
 @RequestMapping(“/task/searchInstagram")
 public class InstagramWorker { … } public void createInstagramTask(String userName) { Queue queue = QueueFactory.getQueue("instagramSearch");
 queue.add(TaskOptions.Builder
 .withUrl("/task/searchInstagram")
 .param(“userName", userName)); } queue.xml
 <queue>
 <name>instagramSearch</name>
 <rate>20/s</rate>
 <bucket-size>40</bucket-size>
 </queue>
28. TASK CHAINING IS SIMPLE
29. APP WASN’T ABLE TO START IN 60 SECONDS SURPRISE!
30. STANDARD ENVIRONMENT LIMITATIONS APPLICATION SHOULD START WITHIN 60 SECONDS https://cloud.google.com/appengine/docs/standard/java/how-instances-are-managed https://medium.com/google-cloud/understanding-and-profiling-app-engine-cold-boot-time-908431aa971d Read Documentation Profiling 30
31. STANDARD ENVIRONMENT LIMITATIONS OPTIMIZATION https://stackoverflow.com/questions/30134708/google-app-engine-spring-slow-startup 31
32. STANDARD ENVIRONMENT LIMITATIONS SPRING VS GUICE FOR SIMPLE APP Spring*
 (Boot) Guice ~9sec ~3sec 32 https://github.com/google/guice
33. STANDARD ENVIRONMENT LIMITATIONS TAKEAWAYS ▸ Read documentation ▸ Response time limit is a cool feature! ▸ Use such framework which fits your needs ▸ Write unit tests ▸ Perform load tests 33
34. SPEED UP WITH CACHING
35. USING MEMCACHE WHAT IS MEMCACHE? KEY-VALUE PAIR DATASTORE https://cloud.google.com/appengine/articles/best-practices-for-app-engine-memcache 35 KEY VALUE ‘UserName’ {‘FirstItem’, ‘SecondItem’,..}
36. USING MEMCACHE WHAT IS MEMCACHE? SHARED SERVICE https://cloud.google.com/appengine/articles/best-practices-for-app-engine-memcache 36 APP INSTANCE APP INSTANCE APP INSTANCE MemcacheService
38. MEMCACHE USAGE @RequestMapping(method = RequestMethod.POST)
 public void processTask(String query) throws TwitterException { if (memcacheService.get(query) == null) {
 QueryResult result = twitterClient.search(query);
 List<Status> tweets = result.getTweets();
 memcacheService.put(query, tweets);
 } else {
 List<Status> tweets = (List<Status>) memcacheService.get(query);
 for (Status tweet : tweets) {
 String userName = tweet.getUser().getName();
 taskCreator.createInstagramTask(userName);
 }
 }
 } @RestController
 @RequestMapping("/task/searchTwitter")
 public class TwitterWorker {
 private final TwitterClient twitterClient;
 private final TaskCreator taskCreator;
 private MemcacheService memcacheService = MemcacheServiceFactory.getMemcacheService(); 38
39. MEMCACHE LIMIT 1MB LIMIT Exception: Values may not be more than 1048503 bytes in length; received 1071339 bytes 39
40. MEMCACHE LIMIT List<Status> tweets = result.getTweets();
 memcacheService.put(query, tweets); 40
41. HOW TO WRITE UNIT TESTS 41 private final LocalServiceTestHelper helper =
 new LocalServiceTestHelper(new LocalMemcacheServiceTestConfig()); @Before
 public void setUp() {
 helper.setUp();
 } @Test
 public void simpleUnitTest() {
 MemcacheService memcacheService = MemcacheServiceFactory.getMemcacheService();
 List<String> list = Arrays.asList("1","2","3");
 memcacheService.put("query", list);
 assertEquals(memcacheService.get("query"), list);
 }
42. IS IT POSSIBLE TO CATCH? MemcacheService memcacheService = 
 MemcacheServiceFactory.getMemcacheService(); 
 List<String> list = Collections.singletonList(
 new String(ByteBuffer.allocate(1048503).array()));
 memcacheService.put("query", list); 2097113 bytes =) Locally works fine Doesn’t work on cloud 42
43. MEMCACHE LIMIT SOLUTION. SPLITTING AND USING ‘PUTALL’ METHOD KEY COLLECTION OF OBJECTS KEY SUBCOLLECTION OF OBJECTS SUBCOLLECTION OF OBJECTS KEY KEY >1 MB <1 MB <1 MB 43 https://cloud.google.com/appengine/docs/standard/java/javadoc/com/google/appengine/api/memcache/MemcacheService
44. MEMCACHE NAMESPACES QueryResult result = twitterClient.search(query); List<Status> tweets = result.getTweets(); memcacheService.put(query, tweets); List<String> users = new ArrayList<>(); for (Status tweet : tweets) { String userName = tweet.getUser().getName(); users.add(userName); memcacheService.put(query, users); } 44
45. MEMCACHE NAMESPACES memcacheService.get(query); ? Tweets Users 45
46. MEMCACHE NAMESPACES DON’T USE DEFAULT NAMESPACE NUMBER OF USERS java.com.ololo.dosmth COLLECTION OF TWITS java.com.ololo.another.dosmth.else QUERY Key 46
47. MEMCACHE NAMESPACES USE DIFFERENT NAMESPACES 47 
 private MemcacheService twitsMemcacheService = 
 MemcacheServiceFactory.getMemcacheService(“twits”); 
 private MemcacheService usersMemcacheService = 
 MemcacheServiceFactory.getMemcacheService(“users”); twitsMemcacheService.put(query, tweets); usersMemcacheService.put(query, users);
48. HOW TO TEST?
49. MEMCACHE NAMESPACES HOW TO TEST? ▸ Black-box: is that real to find? =) ▸ Unit: same as default
 49
50. MEMCACHE ISSUES TAKEAWAYS ▸ Read documentation ▸ What is memcache and namespaces ▸ Write unit tests ▸ Hard to find via black-box 50
51. DATABASE
52. DATASTORE WHAT IS DATASTORE “Google Cloud Datastore is a NoSQL document database built for automatic scaling, high performance, and ease of application development.” https://cloud.google.com/appengine/docs/standard/java/datastore/ 52
54. DATASTORE DATASTORE + OBJECTIFY https://cloud.google.com/appengine/docs/standard/java/gettingstarted/using-datastore-objectify https://github.com/objectify/objectify @Entity class TwitterAccount { @Id String id; // Can be Long, long, or String String name; } ofy().save().entity(new TwitterAccount("1", “Ololo”)).now(); TwitterAccount acc = 
 ofy().load().type(TwitterAccount).id("1").now(); ofy().delete().entity(acc); 54
55. BUT WHAT ABOUT CHANGES IN DB?
56. DATASTORE MIGRATION: DATASTORE + OBJECTIFY @Entity class TwitterAccount { @Id String id; String name; } @Entity class TwitterAccount { @Id String superUniqueId; String superName; } ? 56
57. DATASTORE 57 @Entity class TwitterAccount { @AlsoLoad(“id”) @Id String superUniqueId; @AlsoLoad(“name”) String superName; } @GetMapping(“/cron/migrate")
 public void migrate() { Dao dao = new Dao();
 List<TwitterAccount> twitterAccounts = dao.loadAll();
 for (TwitterAccount twitterAccount : twitterAccounts) {
 dao.saveAccount(twitterAccount);
 } }
58. DATASTORE MIGRATION: CRON JOB https://cloud.google.com/appengine/docs/standard/java/config/cronref#schedule_format 58 <cronentries>
 <cron>
 <url>/migrate</url>
 <description>Migration </description>
 <schedule>1 of jan 00:00</schedule>
 </cron>
 </cronentries> cron.xml
59. CRON JOB CONFIGURATION 59
60. DATASTORE FLASHBACK MIGRATION: CRON JOB Q: How to schedule to run it manually? 
 A: Schedule for some time in future and don’t forget to remove it =)
 
 Q: How to test it locally? 
 A: Call API endpoint somehow. No manual way =) 60
61. BACK TO APPLICATION 61 FE BE JSON PROTO
62. PROTOCOL BUFFERS WHAT IS PROTO? “Protocol Buffers (a.k.a., protobuf) are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data” https://github.com/google/protobuf 62
63. DATASTORE DATASTORE + OBJECTIFY + PROTO FE BE DATASTORE BE PROTO ENTITY ENTITY FE PROTO 63
64. DATASTORE DATASTORE + OBJECTIFY + PROTO ▸ You need to convert Proto to Entity and otherwise by yourself ▸ BUT Datastore saves all Entities inside as proto files… 64
66. PROTOCOL BUFFERS BUT PROTO AT LEAST IS NICE FOR TESTS https://github.com/google/protobuf message TwitterAccount { required string name = 1; required int32 id = 2; optional string description = 3; repeated Twit twits = 4; } 66
67. PROTOCOL BUFFERS BUT PROTO AT LEAST IS NICE FOR TESTS PROTO MESSAGE JAVASCRIPT UI TESTS BE TESTS JAVA
68. DATASTORE + PROTO + OBJECTIFY TAKEAWAYS ▸ Using objectify with protobuf is weird ▸ Protobuf could be nice for testing ▸ Read documentation about schema migration ▸ Cron job is hard to test 68
70. BAZEL WHAT IS BAZEL? https://bazel.build/ “Bazel is a build tool which coordinates builds and runs tests” 70 {FAST, CORRECT} - CHOOSE TWO
71. BAZEL ONE BUILD FILE Front End (JavaScript) Back End (Java/C++) Tests (Java/Python/JavaScript) 71
72. BAZEL WHO USES? ▸ SpaceX ▸ Uber ▸ Pinterest ▸ Dropbox ▸ Huawei ▸ … 72 https://github.com/bazelbuild/bazel/wiki/Bazel-Users
73. BAZEL WORKSPACE BUILD SRC BUILD … 73
74. BAZEL java_binary( name = "ProjectRunner", srcs = glob(["src/main/java/com/example/*.java"]), ) BUILD https://docs.bazel.build/versions/master/tutorial/java.html $ bazel build //:ProjectRunner 74
75. BAZEL TESTS https://docs.bazel.build/versions/master/test-encyclopedia.html 75
76. BAZEL TESTS java_test( name = "hello", srcs = ["TestHello.java"], test_class = "com.example.myproject.TestHello", deps = [ "//examples/java-native/src/main/java/com/example/myproject:hello-lib", "//third_party:junit4", ], ) https://github.com/bazelbuild/bazel/blob/master/examples/java-native/src/test/java/com/example/myproject/BUILD $ bazel test //:ProjectRunner:all 76
77. FINAL TAKEAWAYS ▸ Reading documentation is very important ▸ Potential issues with Google AppEngine ▸ How to test task queues, memcache, datastore/objectify ▸ How to use proto in tests ▸ What is Bazel 77
78. THANK YOU! Q&A
79. LINKS ‣ Google I/O Tasks queues
 https://www.youtube.com/watch?v=AM0ZPO7-lcE 
 ‣ Google: What is memcache (Intro)
 https://www.youtube.com/watch?v=TGl81wr8lz8 
 ‣ Google: Datastore intro
 https://www.youtube.com/watch?v=fQazhzcC-rg 
 ‣ Google: App Engine architecture and services 
 https://www.youtube.com/watch?v=QJp6hmASstQ
 ‣ Bazel and Angular outside Google
 https://medium.com/@Jakeherringbone/what-angular-is-doing-with-bazel-and-closure-21f526f64a34 
 ‣ Build & Run Scalable Web Applications on Google's Infrastructure By Dan Sanderson
 http://shop.oreilly.com/product/0636920017547.do ‣ Appengine + Bazel
 https://github.com/bazelbuild/rules_appengine ‣ CodeLab from Google to run simple Spring Boot app with app engine
 https://codelabs.developers.google.com/codelabs/cloud-app-engine-springboot/index.html 79