Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Untangle your app

Yohan Hartanto
June 03, 2022
29

Untangle your app

Out of memory can be caused by incorrect usage of threads. This talk deep dives into how threading works under the hood in Android and how to avoid it causing a crash in your app.

Presented in Droidcon San Francisco, 2018 by Utkarsh and Yohan

Yohan Hartanto

June 03, 2022
Tweet

Transcript

  1. • Global app with LOTS of features • 100 Developers

    contributing to it • Very long sessions (2-10 hours) Uber Case Study: The Driver App Rewrite
  2. Uber Case Study: The Driver App Rewrite Old App •

    Multiple threading strategies ◦ new Thread() ◦ ExecutorService ◦ AsyncTask ◦ ReactiveX/RxJava
  3. Uber Case Study: The Driver App Rewrite Old App •

    Multiple threading strategies ◦ new Thread() ◦ ExecutorService ◦ AsyncTask ◦ ReactiveX/RxJava • Lots and lots of Out of Memory Crashes • ~40% of which were threading related
  4. Uber Case Study: The Driver App Rewrite Old App •

    Multiple threading strategies ◦ new Thread() ◦ ExecutorService ◦ AsyncTask ◦ ReactiveX/RxJava New App • Unify threading into RxJava • Lots and lots of Out of Memory Crashes • ~40% of which were threading related
  5. Uber Case Study: The Driver App Rewrite Old App •

    Multiple threading strategies ◦ new Thread() ◦ ExecutorService ◦ AsyncTask ◦ ReactiveX/RxJava New App • Unify threading into RxJava • Lots and lots of Out of Memory Crashes • ~40% of which were threading related • Still lots of Out of Memory Crashes • ~50% of which are threading related
  6. • Android and the Main Thread • Multithreading Primer •

    OOMs and Threading in Android • Multithreading with RxJava • Closing thoughts Agenda
  7. Timer App private void updateTimer() { while (true) { long

    newValue = timerValue.getValue() + 1; updateTimerValue(newValue); Thread.sleep(1000); timerView.setText(String.valueOf(newValue)); } }
  8. Timer App private void updateTimer() { while (true) { long

    newValue = timerValue.getValue() + 1; updateTimerValue(newValue); Thread.sleep(1000); timerView.setText(String.valueOf(newValue)); } }
  9. Timer App Class Looper() { void loop() { for(;;) {

    Message msg = messageQueue.next(); dispatchMessage(); } } } Class MessageQueue { Message next() { for(;;) { if (a message is ready to be delivered) return message; else pause; } } } Class Handler { void dispatchMessage(Message msg) { msg.callback.run(); } }
  10. Timer App Class Looper() { void loop() { for(;;) {

    Message msg = messageQueue.next(); dispatchMessage(); } } } Class MessageQueue { Message next() { for(;;) { if (a message is ready to be delivered) return message; else pause; } } } Class Handler { void dispatchMessage(Message msg) { msg.callback.run(); } }
  11. Timer App Class Looper() { void loop() { for(;;) {

    Message msg = messageQueue.next(); dispatchMessage(); } } } Class MessageQueue { Message next() { for(;;) { if (a message is ready to be delivered) return message; else pause; } } } Class Handler { void dispatchMessage(Message msg) { msg.callback.run(); } }
  12. Timer App Class Looper() { void loop() { for(;;) {

    Message msg = messageQueue.next(); dispatchMessage(); } } } Class MessageQueue { Message next() { for(;;) { if (a message is ready to be delivered) return message; else pause; } } } Class Handler { void dispatchMessage(Message msg) { msg.callback.run(); } }
  13. Timer App Class Looper() { void loop() { for(;;) {

    Message msg = messageQueue.next(); while (true) { long newValue = timerValue.getValue() + 1; updateTimerValue(newValue); Thread.sleep(1000); timerView.setText(String.valueOf(newValue)); } } } } Class MessageQueue { Message next() { for(;;) { if (a message is ready to be delivered) return message; else pause; } } }
  14. Timer App Class Looper() { void loop() { for(;;) {

    Message msg = messageQueue.next(); while (true) { long newValue = timerValue.getValue() + 1; updateTimerValue(newValue); Thread.sleep(1000); timerView.setText(String.valueOf(newValue)); -> handler.post(setText(String.valueOf(newValue)); } } } } Class MessageQueue { Message next() { for(;;) { if (a message is ready to be delivered) return message; else pause; } } }
  15. Timer App private void updateTimer() { mainHandler.postDelayed(() -> { long

    newValue = timerValue.getValue() + 1; updateTimerValue(newValue); updateTimer(); }, 1000); }
  16. Timer App private void updateTimer() { mainHandler.postDelayed(() -> { long

    newValue = timerValue.getValue() + 1; updateTimerValue(newValue); diskCache.writeLong(newValue); timerView.setText(String.valueOf(newValue)); updateTimer(); }, 1000); }
  17. Timer App public class TimerActivity extends AppCompatActivity { ... private

    ExecutorService executor = Executors.newFixedThreadPool(1); ... private void updateTimer() { mainHandler.postDelayed(() -> { long newValue = timerValue.getValue() + 1; updateTimerValue(newValue); timerView.setText(String.valueOf(newValue)); executor.submit(() -> diskCache.writeLong(value)); updateTimer(); }, 1000); }
  18. Timer App public class TimerActivity extends AppCompatActivity { ... private

    ExecutorService executor = Executors.newFixedThreadPool(1); ... private void updateTimer() { mainHandler.postDelayed(() -> { long newValue = timerValue.getValue() + 1; updateTimerValue(newValue); timerView.setText(String.valueOf(newValue)); executor.submit(() -> diskCache.writeLong(value)); updateTimer(); }, 1000); }
  19. Single threaded executor • Long running task can cause throttle

    others • Small bounded thread pool is exposed to the same problem
  20. Thread starvation deadlock private void updateTimer() { mainHandler.postDelayed(() -> {

    if (updateTimer) { long newValue = timerValue.getValue() + 1; updateTimerValue(newValue); timerView.setText(String.valueOf(newValue)); executor.submit(() -> { String authToken = diskCache.getStringAsync("authToken").get(); timerApi.saveTimerValue(authToken, newValue); }); updateTimer(); } }, 1000); } public class DiskCache { ... public Future<String> getStringAsync(String key) { return executor.submit(() -> getString(key)); } }
  21. Thread starvation deadlock private void updateTimer() { mainHandler.postDelayed(() -> {

    if (updateTimer) { long newValue = timerValue.getValue() + 1; updateTimerValue(newValue); timerView.setText(String.valueOf(newValue)); executor.submit(() -> { String authToken = diskCache.getStringAsync("authToken").get(); timerApi.saveTimerValue(authToken, newValue); }); updateTimer(); } }, 1000); } public class DiskCache { ... public Future<String> getStringAsync(String key) { return executor.submit(() -> getString(key)); } }
  22. Thread starvation deadlock • Interdependent tasks running on the same

    executor • A task is waiting for another task to complete on the same executor
  23. Timer App public class TimerActivity extends AppCompatActivity { ... private

    ExecutorService executorService = Executors.newFixedThreadPool(1); ... private void updateTimer() { mainHandler.postDelayed(() -> { if (updateTimer) { long newValue = timerValue.getValue() + 1; updateTimerValue(newValue); timerView.setText(String.valueOf(newValue)); executorService.submit(() -> diskCache.writeLong(value)); updateTimer(); } }, 1000); }
  24. Timer App public class TimerActivity extends AppCompatActivity { ... private

    ExecutorService executorService = Executors.newCachedThreadPool(); ... private void updateTimer() { mainHandler.postDelayed(() -> { if (updateTimer) { long newValue = timerValue.getValue() + 1; updateTimerValue(newValue); timerView.setText(String.valueOf(newValue)); executorService.submit(() -> diskCache.writeLong(value)); updateTimer(); } }, 1000); }
  25. Race Condition • More than one thread accessing the same

    resource • Sequence is Non-deterministic • Can also occur on small pool (more than 1 thread) • Can be solved with synchronization
  26. Deadlock • Long running task can cause resource contention •

    Two threads accessing resources can lead to deadlock • Always use a timeout • Synchronization is a major study
  27. “Unlimited resources?” Viola! OOM. java.lang.OutOfMemoryError: pthread_create (1040KB stack) failed: Try

    again at java.lang.Thread.nativeCreate(Native Method) at java.lang.Thread.start(Thread.java:1078) at com.ubercab.threadlimits.MainActivity$5.run(MainActivity.java:136) at android.os.Handler.handleCallback(Handler.java:751) at android.os.Handler.dispatchMessage(Handler.java:95) at android.os.Looper.loop(Looper.java:154) at android.app.ActivityThread.main(ActivityThread.java:6121) at java.lang.reflect.Method.invoke(Native Method) at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:889) at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:779)
  28. Viola! OOM. java.lang.OutOfMemoryError: pthread_create (1040KB stack) failed: Try again at

    java.lang.Thread.nativeCreate(Native Method) at java.lang.Thread.start(Thread.java:1078) at com.ubercab.threadlimits.MainActivity$5.run(MainActivity.java:136) at android.os.Handler.handleCallback(Handler.java:751) at android.os.Handler.dispatchMessage(Handler.java:95) at android.os.Looper.loop(Looper.java:154) at android.app.ActivityThread.main(ActivityThread.java:6121) at java.lang.reflect.Method.invoke(Native Method) at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:889) at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:779) “Unlimited resources?”
  29. Viola! OOM. java.lang.OutOfMemoryError: pthread_create (1040KB stack) failed: Try again at

    java.lang.Thread.nativeCreate(Native Method) at java.lang.Thread.start(Thread.java:1078) at com.ubercab.threadlimits.MainActivity$5.run(MainActivity.java:136) at android.os.Handler.handleCallback(Handler.java:751) at android.os.Handler.dispatchMessage(Handler.java:95) at android.os.Looper.loop(Looper.java:154) at android.app.ActivityThread.main(ActivityThread.java:6121) at java.lang.reflect.Method.invoke(Native Method) at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:889) at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:779) “Unlimited resources?”
  30. Viola! OOM. java.lang.OutOfMemoryError: pthread_create (1040KB stack) failed: Try again at

    java.lang.Thread.nativeCreate(Native Method) at java.lang.Thread.start(Thread.java:1078) at com.ubercab.threadlimits.MainActivity$5.run(MainActivity.java:136) at android.os.Handler.handleCallback(Handler.java:751) at android.os.Handler.dispatchMessage(Handler.java:95) at android.os.Looper.loop(Looper.java:154) at android.app.ActivityThread.main(ActivityThread.java:6121) at java.lang.reflect.Method.invoke(Native Method) at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:889) at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:779) “Unlimited resources?”
  31. Viola! OOM. java.lang.OutOfMemoryError: pthread_create (1040KB stack) failed: Try again at

    java.lang.Thread.nativeCreate(Native Method) at java.lang.Thread.start(Thread.java:1078) at com.ubercab.threadlimits.MainActivity$5.run(MainActivity.java:136) at android.os.Handler.handleCallback(Handler.java:751) at android.os.Handler.dispatchMessage(Handler.java:95) at android.os.Looper.loop(Looper.java:154) at android.app.ActivityThread.main(ActivityThread.java:6121) at java.lang.reflect.Method.invoke(Native Method) at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:889) at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:779) “Unlimited resources?”
  32. Viola! OOM. java.lang.OutOfMemoryError: pthread_create (1040KB stack) failed: Try again at

    java.lang.Thread.nativeCreate(Native Method) at java.lang.Thread.start(Thread.java:1078) at com.ubercab.threadlimits.MainActivity$5.run(MainActivity.java:136) at android.os.Handler.handleCallback(Handler.java:751) at android.os.Handler.dispatchMessage(Handler.java:95) at android.os.Looper.loop(Looper.java:154) at android.app.ActivityThread.main(ActivityThread.java:6121) at java.lang.reflect.Method.invoke(Native Method) at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:889) at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:779) “Unlimited resources?”
  33. private void init(ThreadGroup g, Runnable target, String name, long stackSize)

    { Thread parent = currentThread(); if (g == null) { g = parent.getThreadGroup(); } g.addUnstarted(); this.group = g; this.target = target; this.priority = parent.getPriority(); this.daemon = parent.isDaemon(); setName(name); init2(parent); /* Stash the specified stack size in case the VM cares */ this.stackSize = stackSize; tid = nextThreadID(); } Thread.java
  34. Viola! OOM. java.lang.OutOfMemoryError: pthread_create (1040KB stack) failed: Try again at

    java.lang.Thread.nativeCreate(Native Method) at java.lang.Thread.start(Thread.java:1078) at com.ubercab.threadlimits.MainActivity$5.run(MainActivity.java:136) at android.os.Handler.handleCallback(Handler.java:751) at android.os.Handler.dispatchMessage(Handler.java:95) at android.os.Looper.loop(Looper.java:154) at android.app.ActivityThread.main(ActivityThread.java:6121) at java.lang.reflect.Method.invoke(Native Method) at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:889) at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:779) “Unlimited resources?”
  35. public synchronized void start() { /* Notify the group that

    this thread is about to be started * so that it can be added to the group's list of threads * and the group's unstarted count can be decremented. */ group.add(this); started = false; try { nativeCreate(this, stackSize, daemon); started = true; } finally { <Error Handling> } } private native static void nativeCreate(Thread t, long stackSize, boolean daemon); Thread.java
  36. static void Thread_nativeCreate(JNIEnv* env, jclass, jobject java_thread, jlong stack_size, jboolean

    daemon) { // There are sections in the zygote that forbid thread creation. Runtime* runtime = Runtime::Current(); if (runtime->IsZygote() && runtime->IsZygoteNoThreadSection()) { jclass internal_error = env->FindClass("java/lang/InternalError"); CHECK(internal_error != nullptr); env->ThrowNew(internal_error, "Cannot create threads in zygote"); return; } Thread::CreateNativeThread(env, java_thread, stack_size, daemon == JNI_TRUE); } java_lang_Thread.cc
  37. void Thread::CreateNativeThread(JNIEnv* env, jobject java_peer, size_t stack_size, bool is_daemon) {

    ... <HouseKeeping and Some Checks> ... Thread* child_thread = new Thread(is_daemon); stack_size = FixStackSize(stack_size); child_jni_env_ext(JNIEnvExt::Create(child_thread, Runtime::Current()->GetJavaVM(), &error_msg)); int pthread_create_result = 0; if (child_jni_env_ext.get() != nullptr) { pthread_t new_pthread; pthread_attr_t attr; CHECK_PTHREAD_CALL(pthread_attr_init, (&attr), "new thread"); CHECK_PTHREAD_CALL(pthread_attr_setdetachstate, (&attr, PTHREAD_CREATE_DETACHED), "DETACHED"); CHECK_PTHREAD_CALL(pthread_attr_setstacksize, (&attr, stack_size), stack_size); pthread_create_result = pthread_create(&new_pthread, &attr, Thread::CreateCallback, child_thread); if (pthread_create_result == 0) { return; } } … <Clean up and Error Handling> ... } thread.cc
  38. void Thread::CreateNativeThread(JNIEnv* env, jobject java_peer, size_t stack_size, bool is_daemon) {

    ... <HouseKeeping and Some Checks> ... Thread* child_thread = new Thread(is_daemon); stack_size = FixStackSize(stack_size); child_jni_env_ext(JNIEnvExt::Create(child_thread, Runtime::Current()->GetJavaVM(), &error_msg)); int pthread_create_result = 0; if (child_jni_env_ext.get() != nullptr) { pthread_t new_pthread; pthread_attr_t attr; CHECK_PTHREAD_CALL(pthread_attr_init, (&attr), "new thread"); CHECK_PTHREAD_CALL(pthread_attr_setdetachstate, (&attr, PTHREAD_CREATE_DETACHED), "DETACHED"); CHECK_PTHREAD_CALL(pthread_attr_setstacksize, (&attr, stack_size), stack_size); pthread_create_result = pthread_create(&new_pthread, &attr, Thread::CreateCallback, child_thread); if (pthread_create_result == 0) { return; } } … <Clean up and Error Handling> ... } thread.cc
  39. void Thread::CreateNativeThread(JNIEnv* env, jobject java_peer, size_t stack_size, bool is_daemon) {

    ... <HouseKeeping and Some Checks> ... Thread* child_thread = new Thread(is_daemon); stack_size = FixStackSize(stack_size); child_jni_env_ext(JNIEnvExt::Create(child_thread, Runtime::Current()->GetJavaVM(), &error_msg)); int pthread_create_result = 0; if (child_jni_env_ext.get() != nullptr) { pthread_t new_pthread; pthread_attr_t attr; CHECK_PTHREAD_CALL(pthread_attr_init, (&attr), "new thread"); CHECK_PTHREAD_CALL(pthread_attr_setdetachstate, (&attr, PTHREAD_CREATE_DETACHED), "DETACHED"); CHECK_PTHREAD_CALL(pthread_attr_setstacksize, (&attr, stack_size), stack_size); pthread_create_result = pthread_create(&new_pthread, &attr, Thread::CreateCallback, child_thread); if (pthread_create_result == 0) { return; } } … <Clean up and Error Handling> ... } thread.cc
  40. void Thread::CreateNativeThread(JNIEnv* env, jobject java_peer, size_t stack_size, bool is_daemon) {

    ... <HouseKeeping and Some Checks> ... Thread* child_thread = new Thread(is_daemon); stack_size = FixStackSize(stack_size); child_jni_env_ext(JNIEnvExt::Create(child_thread, Runtime::Current()->GetJavaVM(), &error_msg)); int pthread_create_result = 0; if (child_jni_env_ext.get() != nullptr) { pthread_t new_pthread; pthread_attr_t attr; CHECK_PTHREAD_CALL(pthread_attr_init, (&attr), "new thread"); CHECK_PTHREAD_CALL(pthread_attr_setdetachstate, (&attr, PTHREAD_CREATE_DETACHED), "DETACHED"); CHECK_PTHREAD_CALL(pthread_attr_setstacksize, (&attr, stack_size), stack_size); pthread_create_result = pthread_create(&new_pthread, &attr, Thread::CreateCallback, child_thread); if (pthread_create_result == 0) { return; } } … <Clean up and Error Handling> ... } thread.cc
  41. void Thread::CreateNativeThread(JNIEnv* env, jobject java_peer, size_t stack_size, bool is_daemon) {

    ... <HouseKeeping and Some Checks> ... Thread* child_thread = new Thread(is_daemon); stack_size = FixStackSize(stack_size); child_jni_env_ext(JNIEnvExt::Create(child_thread, Runtime::Current()->GetJavaVM(), &error_msg)); int pthread_create_result = 0; if (child_jni_env_ext.get() != nullptr) { pthread_t new_pthread; pthread_attr_t attr; CHECK_PTHREAD_CALL(pthread_attr_init, (&attr), "new thread"); CHECK_PTHREAD_CALL(pthread_attr_setdetachstate, (&attr, PTHREAD_CREATE_DETACHED), "DETACHED"); CHECK_PTHREAD_CALL(pthread_attr_setstacksize, (&attr, stack_size), stack_size); pthread_create_result = pthread_create(&new_pthread, &attr, Thread::CreateCallback, child_thread); if (pthread_create_result == 0) { return; } } … <Clean up and Error Handling> ... } thread.cc
  42. void Thread::CreateNativeThread(JNIEnv* env, jobject java_peer, size_t stack_size, bool is_daemon) {

    ... <HouseKeeping and Some Checks> ... Thread* child_thread = new Thread(is_daemon); stack_size = FixStackSize(stack_size); child_jni_env_ext(JNIEnvExt::Create(child_thread, Runtime::Current()->GetJavaVM(), &error_msg)); int pthread_create_result = 0; if (child_jni_env_ext.get() != nullptr) { pthread_t new_pthread; pthread_attr_t attr; CHECK_PTHREAD_CALL(pthread_attr_init, (&attr), "new thread"); CHECK_PTHREAD_CALL(pthread_attr_setdetachstate, (&attr, PTHREAD_CREATE_DETACHED), "DETACHED"); CHECK_PTHREAD_CALL(pthread_attr_setstacksize, (&attr, stack_size), stack_size); pthread_create_result = pthread_create(&new_pthread, &attr, Thread::CreateCallback, child_thread); if (pthread_create_result == 0) { return; } } … <Clean up and Error Handling> ... } thread.cc
  43. void Thread::CreateNativeThread(JNIEnv* env, jobject java_peer, size_t stack_size, bool is_daemon) {

    ... <Stuff From Before> … pthread_create_result = pthread_create(&new_pthread, &attr, Thread::CreateCallback, child_thread); if (pthread_create_result == 0) { return; } { std::string msg(child_jni_env_ext.get() == nullptr ? StringPrintf("Could not allocate JNI Env: %s", error_msg.c_str()) : StringPrintf("pthread_create (%s stack) failed: %s", PrettySize(stack_size).c_str(), strerror(pthread_create_result))); ScopedObjectAccess soa(env); soa.Self()->ThrowOutOfMemoryError(msg.c_str()); } } thread.cc
  44. void Thread::CreateNativeThread(JNIEnv* env, jobject java_peer, size_t stack_size, bool is_daemon) {

    ... <Stuff From Before> … pthread_create_result = pthread_create(&new_pthread, &attr, Thread::CreateCallback, child_thread); if (pthread_create_result == 0) { return; } { std::string msg(child_jni_env_ext.get() == nullptr ? StringPrintf("Could not allocate JNI Env: %s", error_msg.c_str()) : StringPrintf("pthread_create (%s stack) failed: %s", PrettySize(stack_size).c_str(), strerror(pthread_create_result))); ScopedObjectAccess soa(env); soa.Self()->ThrowOutOfMemoryError(msg.c_str()); } } thread.cc
  45. void Thread::CreateNativeThread(JNIEnv* env, jobject java_peer, size_t stack_size, bool is_daemon) {

    ... <Stuff From Before> … pthread_create_result = pthread_create(&new_pthread, &attr, Thread::CreateCallback, child_thread); if (pthread_create_result == 0) { return; } { std::string msg(child_jni_env_ext.get() == nullptr ? StringPrintf("Could not allocate JNI Env: %s", error_msg.c_str()) : StringPrintf("pthread_create (%s stack) failed: %s", PrettySize(stack_size).c_str(), strerror(pthread_create_result))); ScopedObjectAccess soa(env); soa.Self()->ThrowOutOfMemoryError(msg.c_str()); } } thread.cc
  46. void Thread::CreateNativeThread(JNIEnv* env, jobject java_peer, size_t stack_size, bool is_daemon) {

    ... <Stuff From Before> … pthread_create_result = pthread_create(&new_pthread, &attr, Thread::CreateCallback, child_thread); if (pthread_create_result == 0) { return; } { std::string msg(child_jni_env_ext.get() == nullptr ? StringPrintf("Could not allocate JNI Env: %s", error_msg.c_str()) : StringPrintf("pthread_create (%s stack) failed: %s", PrettySize(stack_size).c_str(), strerror(pthread_create_result))); ScopedObjectAccess soa(env); soa.Self()->ThrowOutOfMemoryError(msg.c_str()); } } thread.cc
  47. Viola! OOM. java.lang.OutOfMemoryError: pthread_create (1040KB stack) failed: Try again at

    java.lang.Thread.nativeCreate(Native Method) at java.lang.Thread.start(Thread.java:1078) at com.ubercab.threadlimits.MainActivity$5.run(MainActivity.java:136) at android.os.Handler.handleCallback(Handler.java:751) at android.os.Handler.dispatchMessage(Handler.java:95) at android.os.Looper.loop(Looper.java:154) at android.app.ActivityThread.main(ActivityThread.java:6121) at java.lang.reflect.Method.invoke(Native Method) at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:889) at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:779) “Unlimited resources?”
  48. Viola! OOM. java.lang.OutOfMemoryError: pthread_create (1040KB stack) failed: Try again at

    java.lang.Thread.nativeCreate(Native Method) at java.lang.Thread.start(Thread.java:1078) at com.ubercab.threadlimits.MainActivity$5.run(MainActivity.java:136) at android.os.Handler.handleCallback(Handler.java:751) at android.os.Handler.dispatchMessage(Handler.java:95) at android.os.Looper.loop(Looper.java:154) at android.app.ActivityThread.main(ActivityThread.java:6121) at java.lang.reflect.Method.invoke(Native Method) at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:889) at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:779) “Unlimited resources?”
  49. stack=java.lang.StackOverflowError: stack size 8MB at com.ubercab.threadlimits.MainActivity.overflow(MainActivity.java:81) at com.ubercab.threadlimits.MainActivity.overflow(MainActivity.java:81) at com.ubercab.threadlimits.MainActivity.overflow(MainActivity.java:81)

    at com.ubercab.threadlimits.MainActivity.overflow(MainActivity.java:81) . . . stack=java.lang.StackOverflowError: stack size 1040KB at com.ubercab.threadlimits.MainActivity$2.run(MainActivity.java:146) at com.ubercab.threadlimits.MainActivity$2.run(MainActivity.java:146) at com.ubercab.threadlimits.MainActivity$2.run(MainActivity.java:146) at com.ubercab.threadlimits.MainActivity$2.run(MainActivity.java:146) . . .
  50. new Thread(() -> doSomething()); threadFactory.newThread(() -> doSomething()); private void init(ThreadGroup

    g, Runnable target, String name, long stackSize) { ... } public Thread(Runnable target) { init(null, target, "Thread-" + nextThreadNum(), 0); } Thread.java
  51. static size_t FixStackSize(size_t stack_size) { // Dalvik used the bionic

    pthread default stack size for native threads, // so include that here to support apps that expect large native stacks. stack_size += 1 * MB; // It's not possible to request a stack smaller than the system-defined PTHREAD_STACK_MIN. if (stack_size < PTHREAD_STACK_MIN) { stack_size = PTHREAD_STACK_MIN; } // add protected and stack overflow region space to stack_size ... // Some systems require the stack size to be a multiple of the system page size, so round up. stack_size = RoundUp(stack_size, kPageSize); return stack_size; } thread.cc
  52. static size_t FixStackSize(size_t stack_size) { // Dalvik used the bionic

    pthread default stack size for native threads, // so include that here to support apps that expect large native stacks. stack_size += 1 * MB; // It's not possible to request a stack smaller than the system-defined PTHREAD_STACK_MIN. if (stack_size < PTHREAD_STACK_MIN) { stack_size = PTHREAD_STACK_MIN; } // add protected and stack overflow region space to stack_size ... // Some systems require the stack size to be a multiple of the system page size, so round up. stack_size = RoundUp(stack_size, kPageSize); return stack_size; } thread.cc
  53. static size_t FixStackSize(size_t stack_size) { // Dalvik used the bionic

    pthread default stack size for native threads, // so include that here to support apps that expect large native stacks. stack_size += 1 * MB; // It's not possible to request a stack smaller than the system-defined PTHREAD_STACK_MIN. if (stack_size < PTHREAD_STACK_MIN) { stack_size = PTHREAD_STACK_MIN; } // add protected and stack overflow region space to stack_size ... // Some systems require the stack size to be a multiple of the system page size, so round up. stack_size = RoundUp(stack_size, kPageSize); return stack_size; } thread.cc
  54. static size_t FixStackSize(size_t stack_size) { // Dalvik used the bionic

    pthread default stack size for native threads, // so include that here to support apps that expect large native stacks. stack_size += 1 * MB; // It's not possible to request a stack smaller than the system-defined PTHREAD_STACK_MIN. if (stack_size < PTHREAD_STACK_MIN) { stack_size = PTHREAD_STACK_MIN; } // add protected and stack overflow region space to stack_size ... // Some systems require the stack size to be a multiple of the system page size, so round up. stack_size = RoundUp(stack_size, kPageSize); return stack_size; } thread.cc
  55. • Main Stack Size - 8 MB [Fixed] • Others

    - 1 MB+ [Configurable] • Buffer to catch StackOverflowErrors • Memory mapped on start, not init Stack Size Takeaways
  56. java.lang.OutOfMemoryError: pthread_create (1040KB stack) failed: Try again at java.lang.Thread.nativeCreate(Native Method)

    at java.lang.Thread.start(Thread.java:1078) at com.ubercab.threadlimits.MainActivity$5.run(MainActivity.java:136) at android.os.Handler.handleCallback(Handler.java:751) at android.os.Handler.dispatchMessage(Handler.java:95) at android.os.Looper.loop(Looper.java:154) at android.app.ActivityThread.main(ActivityThread.java:6121) at java.lang.reflect.Method.invoke(Native Method) at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:889) at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:779)
  57. W libc : pthread_create failed: couldn't allocate 315641856-bytes mapped space:

    Out of memory W art : Throwing OutOfMemoryError "pthread_create (301MB stack) failed: Try again" W libc : pthread_create failed: clone failed: Try again W art : Throwing OutOfMemoryError "pthread_create (1040KB stack) failed: Try again" java.lang.OutOfMemoryError: pthread_create (1040KB stack) failed: Try again at java.lang.Thread.nativeCreate(Native Method) at java.lang.Thread.start(Thread.java:1078) at com.ubercab.threadlimits.MainActivity$5.run(MainActivity.java:136) at android.os.Handler.handleCallback(Handler.java:751) at android.os.Handler.dispatchMessage(Handler.java:95) at android.os.Looper.loop(Looper.java:154) at android.app.ActivityThread.main(ActivityThread.java:6121) at java.lang.reflect.Method.invoke(Native Method) at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:889) at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:779)
  58. int pthread_create(pthread_t* thread_out, pthread_attr_t const* attr, void* (*start_routine)(void*), void* arg)

    { pthread_internal_t* thread = NULL; void* child_stack = NULL; int result = __allocate_thread(&thread_attr, &thread, &child_stack); if (result != 0) { return result; } int flags = CLONE_VM | CLONE_FS | CLONE_FILES | CLONE_SIGHAND | CLONE_THREAD | CLONE_SYSVSEM | CLONE_SETTLS | CLONE_PARENT_SETTID | CLONE_CHILD_CLEARTID; int rc = clone(__pthread_start, child_stack, flags, thread, &(thread->tid), tls, &(thread->tid)); if (rc == -1) { int clone_errno = errno; async_safe_format_log(ANDROID_LOG_WARN, "libc", "pthread_create failed: clone failed: %s", strerror(clone_errno)); return clone_errno; } return 0; } pthread_create.cpp
  59. int pthread_create(pthread_t* thread_out, pthread_attr_t const* attr, void* (*start_routine)(void*), void* arg)

    { pthread_internal_t* thread = NULL; void* child_stack = NULL; int result = __allocate_thread(&thread_attr, &thread, &child_stack); if (result != 0) { return result; } int flags = CLONE_VM | CLONE_FS | CLONE_FILES | CLONE_SIGHAND | CLONE_THREAD | CLONE_SYSVSEM | CLONE_SETTLS | CLONE_PARENT_SETTID | CLONE_CHILD_CLEARTID; int rc = clone(__pthread_start, child_stack, flags, thread, &(thread->tid), tls, &(thread->tid)); if (rc == -1) { int clone_errno = errno; async_safe_format_log(ANDROID_LOG_WARN, "libc", "pthread_create failed: clone failed: %s", strerror(clone_errno)); return clone_errno; } return 0; } pthread_create.cpp
  60. int pthread_create(pthread_t* thread_out, pthread_attr_t const* attr, void* (*start_routine)(void*), void* arg)

    { pthread_internal_t* thread = NULL; void* child_stack = NULL; int result = __allocate_thread(&thread_attr, &thread, &child_stack); if (result != 0) { return result; } int flags = CLONE_VM | CLONE_FS | CLONE_FILES | CLONE_SIGHAND | CLONE_THREAD | CLONE_SYSVSEM | CLONE_SETTLS | CLONE_PARENT_SETTID | CLONE_CHILD_CLEARTID; int rc = clone(__pthread_start, child_stack, flags, thread, &(thread->tid), tls, &(thread->tid)); if (rc == -1) { int clone_errno = errno; async_safe_format_log(ANDROID_LOG_WARN, "libc", "pthread_create failed: clone failed: %s", strerror(clone_errno)); return clone_errno; } return 0; } pthread_create.cpp
  61. void Thread::CreateNativeThread(JNIEnv* env, jobject java_peer, size_t stack_size, bool is_daemon) {

    ... <HouseKeeping and Some Checks> ... Thread* child_thread = new Thread(is_daemon); stack_size = FixStackSize(stack_size); child_jni_env_ext(JNIEnvExt::Create(child_thread, Runtime::Current()->GetJavaVM(), &error_msg)); int pthread_create_result = 0; if (child_jni_env_ext.get() != nullptr) { pthread_t new_pthread; pthread_attr_t attr; CHECK_PTHREAD_CALL(pthread_attr_init, (&attr), "new thread"); CHECK_PTHREAD_CALL(pthread_attr_setdetachstate, (&attr, PTHREAD_CREATE_DETACHED), "DETACHED"); CHECK_PTHREAD_CALL(pthread_attr_setstacksize, (&attr, stack_size), stack_size); pthread_create_result = pthread_create(&new_pthread, &attr, Thread::CreateCallback, child_thread); if (pthread_create_result == 0) { return; } } … <Clean up and Error Handling> ... } thread.cc
  62. int pthread_create(pthread_t* thread_out, pthread_attr_t const* attr, void* (*start_routine)(void*), void* arg)

    { pthread_internal_t* thread = NULL; void* child_stack = NULL; int result = __allocate_thread(&thread_attr, &thread, &child_stack); if (result != 0) { return result; } int flags = CLONE_VM | CLONE_FS | CLONE_FILES | CLONE_SIGHAND | CLONE_THREAD | CLONE_SYSVSEM | CLONE_SETTLS | CLONE_PARENT_SETTID | CLONE_CHILD_CLEARTID; int rc = clone(__pthread_start, child_stack, flags, thread, &(thread->tid), tls, &(thread->tid)); if (rc == -1) { int clone_errno = errno; async_safe_format_log(ANDROID_LOG_WARN, "libc", "pthread_create failed: clone failed: %s", strerror(clone_errno)); return clone_errno; } return 0; } pthread_create.cpp
  63. int pthread_create(pthread_t* thread_out, pthread_attr_t const* attr, void* (*start_routine)(void*), void* arg)

    { pthread_internal_t* thread = NULL; void* child_stack = NULL; int result = __allocate_thread(&thread_attr, &thread, &child_stack); if (result != 0) { return result; } int flags = CLONE_VM | CLONE_FS | CLONE_FILES | CLONE_SIGHAND | CLONE_THREAD | CLONE_SYSVSEM | CLONE_SETTLS | CLONE_PARENT_SETTID | CLONE_CHILD_CLEARTID; int rc = clone(__pthread_start, child_stack, flags, thread, &(thread->tid), tls, &(thread->tid)); if (rc == -1) { int clone_errno = errno; async_safe_format_log(ANDROID_LOG_WARN, "libc", "pthread_create failed: clone failed: %s", strerror(clone_errno)); return clone_errno; } return 0; } pthread_create.cpp
  64. int pthread_create(pthread_t* thread_out, pthread_attr_t const* attr, void* (*start_routine)(void*), void* arg)

    { pthread_internal_t* thread = NULL; void* child_stack = NULL; int result = __allocate_thread(&thread_attr, &thread, &child_stack); if (result != 0) { return result; } int flags = CLONE_VM | CLONE_FS | CLONE_FILES | CLONE_SIGHAND | CLONE_THREAD | CLONE_SYSVSEM | CLONE_SETTLS | CLONE_PARENT_SETTID | CLONE_CHILD_CLEARTID; int rc = clone(__pthread_start, child_stack, flags, thread, &(thread->tid), tls, &(thread->tid)); if (rc == -1) { int clone_errno = errno; async_safe_format_log(ANDROID_LOG_WARN, "libc", "pthread_create failed: clone failed: %s", strerror(clone_errno)); return clone_errno; } return 0; } pthread_create.cpp
  65. int pthread_create(pthread_t* thread_out, pthread_attr_t const* attr, void* (*start_routine)(void*), void* arg)

    { pthread_internal_t* thread = NULL; void* child_stack = NULL; int result = __allocate_thread(&thread_attr, &thread, &child_stack); if (result != 0) { return result; } int flags = CLONE_VM | CLONE_FS | CLONE_FILES | CLONE_SIGHAND | CLONE_THREAD | CLONE_SYSVSEM | CLONE_SETTLS | CLONE_PARENT_SETTID | CLONE_CHILD_CLEARTID; int rc = clone(__pthread_start, child_stack, flags, thread, &(thread->tid), tls, &(thread->tid)); if (rc == -1) { int clone_errno = errno; async_safe_format_log(ANDROID_LOG_WARN, "libc", "pthread_create failed: clone failed: %s", strerror(clone_errno)); return clone_errno; } return 0; } pthread_create.cpp
  66. int pthread_create(pthread_t* thread_out, pthread_attr_t const* attr, void* (*start_routine)(void*), void* arg)

    { pthread_internal_t* thread = NULL; void* child_stack = NULL; int result = __allocate_thread(&thread_attr, &thread, &child_stack); if (result != 0) { return result; } int flags = CLONE_VM | CLONE_FS | CLONE_FILES | CLONE_SIGHAND | CLONE_THREAD | CLONE_SYSVSEM | CLONE_SETTLS | CLONE_PARENT_SETTID | CLONE_CHILD_CLEARTID; int rc = clone(__pthread_start, child_stack, flags, thread, &(thread->tid), tls, &(thread->tid)); if (rc == -1) { int clone_errno = errno; async_safe_format_log(ANDROID_LOG_WARN, "libc", "pthread_create failed: clone failed: %s", strerror(clone_errno)); return clone_errno; } return 0; } pthread_create.cpp
  67. int pthread_create(pthread_t* thread_out, pthread_attr_t const* attr, void* (*start_routine)(void*), void* arg)

    { pthread_internal_t* thread = NULL; void* child_stack = NULL; int result = __allocate_thread(&thread_attr, &thread, &child_stack); if (result != 0) { return result; } int flags = CLONE_VM | CLONE_FS | CLONE_FILES | CLONE_SIGHAND | CLONE_THREAD | CLONE_SYSVSEM | CLONE_SETTLS | CLONE_PARENT_SETTID | CLONE_CHILD_CLEARTID; int rc = clone(__pthread_start, child_stack, flags, thread, &(thread->tid), tls, &(thread->tid)); if (rc == -1) { int clone_errno = errno; async_safe_format_log(ANDROID_LOG_WARN, "libc", "pthread_create failed: clone failed: %s", strerror(clone_errno)); return clone_errno; } return 0; } pthread_create.cpp
  68. Shutting down VM FATAL EXCEPTION: main Process: com.ubercab.threadlimits, PID: 3142

    java.lang.OutOfMemoryError: pthread_create (1040KB stack) failed: Try again at com.ubercab.threadlimits.MainActivity$5.run(MainActivity.java:136) at android.os.Handler.handleCallback(Handler.java:751) at android.os.Handler.dispatchMessage(Handler.java:95) at android.os.Looper.loop(Looper.java:154) at android.app.ActivityThread.main(ActivityThread.java:6121) at java.lang.reflect.Method.invoke(Native Method) at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:889) at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:779) Force finishing activity com.ubercab.threadlimits/.MainActivity W libc : pthread_create failed: couldn't allocate 315641856-bytes mapped space: Out of memory W art : Throwing OutOfMemoryError "pthread_create (301MB stack) failed: Try again" W libc : pthread_create failed: clone failed: Try again W art : Throwing OutOfMemoryError "pthread_create (1040KB stack) failed: Try again"
  69. int clone(int (*fn)(void*), void* child_stack, int flags, void* arg, ...)

    { ... yada yada yada ... // Actually do the clone. int clone_result; clone_result = __bionic_clone(flags, child_stack, parent_tid, new_tls, child_tid, fn, arg); ... more things ... return clone_result; } clone.cpp
  70. ENTRY_PRIVATE(__bionic_clone) mov ip, sp stmfd sp!, {r4, r5, r6, r7}

    ldmfd ip, {r4, r5, r6} stmdb r1!, {r5, r6} # Make the system call. ldr r7, =__NR_clone swi #0 movs r0, r0 beq .L_bc_child ldmfd sp!, {r4, r5, r6, r7} cmn r0, #(MAX_ERRNO + 1) bxls lr neg r0, r0 b __set_errno_internal .L_bc_child: mov lr, #0 # Call __start_thread with the 'fn' and 'arg' we stored on the child stack. pop {r0, r1} b __start_thread END(__bionic_clone) __bionic_clone.S
  71. Limit Soft Limit Hard Limit Units Max cpu time unlimited

    unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 9447 9447 processes Max open files 1024 4096 files Max locked memory 67108864 67108864 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 9447 9447 signals Max msgqueue size 819200 819200 bytes Max nice priority 40 40 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us ulimit -a
  72. int pthread_create(pthread_t* thread_out, pthread_attr_t const* attr, void* (*start_routine)(void*), void* arg)

    { pthread_internal_t* thread = NULL; void* child_stack = NULL; int result = __allocate_thread(&thread_attr, &thread, &child_stack); if (result != 0) { return result; } int flags = CLONE_VM | CLONE_FS | CLONE_FILES | CLONE_SIGHAND | CLONE_THREAD | CLONE_SYSVSEM | CLONE_SETTLS | CLONE_PARENT_SETTID | CLONE_CHILD_CLEARTID; int rc = clone(__pthread_start, child_stack, flags, thread, &(thread->tid), tls, &(thread->tid)); if (rc == -1) { int clone_errno = errno; async_safe_format_log(ANDROID_LOG_WARN, "libc", "pthread_create failed: clone failed: %s", strerror(clone_errno)); return clone_errno; } return 0; } pthread_create.cpp
  73. static int __allocate_thread(pthread_attr_t* attr, pthread_internal_t** threadp, void** child_stack) { size_t

    mmap_size; uint8_t* stack_top; // The caller didn't provide a stack, so allocate one. // Make sure the stack size and guard size are multiples of PAGE_SIZE. if (__builtin_add_overflow(attr->stack_size, attr->guard_size, &mmap_size)) return EAGAIN; if (__builtin_add_overflow(mmap_size, sizeof(pthread_internal_t), &mmap_size)) return EAGAIN; mmap_size = __BIONIC_ALIGN(mmap_size, PAGE_SIZE); attr->guard_size = __BIONIC_ALIGN(attr->guard_size, PAGE_SIZE); attr->stack_base = __create_thread_mapped_space(mmap_size, attr->guard_size); if (attr->stack_base == NULL) { return EAGAIN; } return 0; } pthread_create.cpp
  74. static void* __create_thread_mapped_space(size_t mmap_size, size_t stack_guard_size) { // Create a

    new private anonymous map. int prot = PROT_READ | PROT_WRITE; int flags = MAP_PRIVATE | MAP_ANONYMOUS | MAP_NORESERVE; void* space = mmap(NULL, mmap_size, prot, flags, -1, 0); if (space == MAP_FAILED) { async_safe_format_log(ANDROID_LOG_WARN, "libc", "pthread_create failed: couldn't allocate %zu-bytes mapped space: %s", mmap_size, strerror(errno)); return NULL; } return space; } pthread_create.cpp
  75. static void* __create_thread_mapped_space(size_t mmap_size, size_t stack_guard_size) { // Create a

    new private anonymous map. int prot = PROT_READ | PROT_WRITE; int flags = MAP_PRIVATE | MAP_ANONYMOUS | MAP_NORESERVE; void* space = mmap(NULL, mmap_size, prot, flags, -1, 0); if (space == MAP_FAILED) { async_safe_format_log(ANDROID_LOG_WARN, "libc", "pthread_create failed: couldn't allocate %zu-bytes mapped space: %s", mmap_size, strerror(errno)); return NULL; } return space; } pthread_create.cpp
  76. static void* __create_thread_mapped_space(size_t mmap_size, size_t stack_guard_size) { // Create a

    new private anonymous map. int prot = PROT_READ | PROT_WRITE; int flags = MAP_PRIVATE | MAP_ANONYMOUS | MAP_NORESERVE; void* space = mmap(NULL, mmap_size, prot, flags, -1, 0); if (space == MAP_FAILED) { async_safe_format_log(ANDROID_LOG_WARN, "libc", "pthread_create failed: couldn't allocate %zu-bytes mapped space: %s", mmap_size, strerror(errno)); return NULL; } return space; } pthread_create.cpp
  77. Shutting down VM FATAL EXCEPTION: main Process: com.ubercab.threadlimits, PID: 3142

    java.lang.OutOfMemoryError: pthread_create (1040KB stack) failed: Try again at java.lang.Thread.nativeCreate(Native Method) at java.lang.Thread.start(Thread.java:1078) at com.ubercab.threadlimits.MainActivity$5.run(MainActivity.java:136) at android.os.Handler.handleCallback(Handler.java:751) at android.os.Handler.dispatchMessage(Handler.java:95) at android.os.Looper.loop(Looper.java:154) at android.app.ActivityThread.main(ActivityThread.java:6121) at java.lang.reflect.Method.invoke(Native Method) at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:889) at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:779) Force finishing activity com.ubercab.threadlimits/.MainActivity W libc : pthread_create failed: couldn't allocate 315641856-bytes mapped space: Out of memory W art : Throwing OutOfMemoryError "pthread_create (301MB stack) failed: Try again" W libc : pthread_create failed: clone failed: Try again W art : Throwing OutOfMemoryError "pthread_create (1040KB stack) failed: Try again"
  78. void* mmap64(void* addr, size_t size, int prot, int flags, int

    fd, off64_t offset) { // prevent allocations large enough for `end - start` to overflow size_t rounded = __BIONIC_ALIGN(size, PAGE_SIZE); if (rounded < size || rounded > PTRDIFF_MAX) { errno = ENOMEM; return MAP_FAILED; } bool is_private_anonymous = (flags & (MAP_PRIVATE | MAP_ANONYMOUS)) == (MAP_PRIVATE | MAP_ANONYMOUS); void* result = __mmap2(addr, size, prot, flags, fd, offset >> MMAP2_SHIFT); return result; } void* mmap(void* addr, size_t size, int prot, int flags, int fd, off_t offset) { return mmap64(addr, size, prot, flags, fd, static_cast<off64_t>(offset)); } mmap.cpp
  79. void* mmap64(void* addr, size_t size, int prot, int flags, int

    fd, off64_t offset) { // prevent allocations large enough for `end - start` to overflow size_t rounded = __BIONIC_ALIGN(size, PAGE_SIZE); if (rounded < size || rounded > PTRDIFF_MAX) { errno = ENOMEM; return MAP_FAILED; } bool is_private_anonymous = (flags & (MAP_PRIVATE | MAP_ANONYMOUS)) == (MAP_PRIVATE | MAP_ANONYMOUS); void* result = __mmap2(addr, size, prot, flags, fd, offset >> MMAP2_SHIFT); return result; } void* mmap(void* addr, size_t size, int prot, int flags, int fd, off_t offset) { return mmap64(addr, size, prot, flags, fd, static_cast<off64_t>(offset)); } mmap.cpp
  80. ENTRY(__mmap2) mov ip, sp stmfd sp!, {r4, r5, r6, r7}

    .cfi_def_cfa_offset 16 .cfi_rel_offset r4, 0 .cfi_rel_offset r5, 4 .cfi_rel_offset r6, 8 .cfi_rel_offset r7, 12 ldmfd ip, {r4, r5, r6} ldr r7, =__NR_mmap2 swi #0 ldmfd sp!, {r4, r5, r6, r7} .cfi_def_cfa_offset 0 cmn r0, #(MAX_ERRNO + 1) bxls lr neg r0, r0 b __set_errno_internal END(__mmap2) __mmap2.S
  81. inline long do_mmap2( unsigned long addr, unsigned long len, unsigned

    long prot, unsigned long flags, unsigned long fd, unsigned long pgoff) { int error = -EINVAL; struct file * file = NULL; down_write(&current->mm->mmap_sem); error = do_mmap_pgoff(file, addr, len, prot, flags, pgoff); up_write(&current->mm->mmap_sem); if (file) fput(file); out: return error; } arch/arm/kernel/sys_arm.c
  82. unsigned long do_mmap_pgoff(struct file *file, unsigned long addr, unsigned long

    len, unsigned long prot, unsigned long flags, unsigned long pgoff) { /* Careful about overflows.. */ len = PAGE_ALIGN(len); if (!len || len > TASK_SIZE) return -ENOMEM; /* offset overflow? */ if ((pgoff + (len >> PAGE_SHIFT)) < pgoff) return -EOVERFLOW; /* Too many mappings? */ if (mm->map_count > sysctl_max_map_count) return -ENOMEM; return mmap_region(file, addr, len, flags, vm_flags, pgoff); } mm/mmap.c
  83. sysctl vm vm.dirty_background_bytes = 0 vm.dirty_background_ratio = 5 ... vm.legacy_va_layout

    = 0 vm.lowmem_reserve_ratio = 256 32 vm.max_map_count = 65530 vm.min_free_kbytes = 6222 vm.min_free_order_shift = 4 vm.mmap_min_addr = 32768 vm.mmap_rnd_bits = 24 ... vm.swappiness = 60 vm.user_reserve_kbytes = 87864 vm.vfs_cache_pressure = 100
  84. unsigned long do_mmap_pgoff(struct file *file, unsigned long addr, unsigned long

    len, unsigned long prot, unsigned long flags, unsigned long pgoff) { /* Careful about overflows.. */ len = PAGE_ALIGN(len); if (!len || len > TASK_SIZE) return -ENOMEM; /* offset overflow? */ if ((pgoff + (len >> PAGE_SHIFT)) < pgoff) return -EOVERFLOW; /* Too many mappings? */ if (mm->map_count > sysctl_max_map_count) return -ENOMEM; return mmap_region(file, addr, len, flags, vm_flags, pgoff); } mm/mmap.c
  85. unsigned long mmap_region(struct file *file, unsigned long addr, unsigned long

    len, unsigned long flags, unsigned int vm_flags, unsigned long pgoff) { /* Check against address space limit. */ if (!may_expand_vm(mm, len >> PAGE_SHIFT)) return -ENOMEM; } int may_expand_vm(struct mm_struct *mm, unsigned long npages) { unsigned long cur = mm->total_vm; /* pages */ unsigned long lim; lim = current->signal->rlim[RLIMIT_AS].rlim_cur >> PAGE_SHIFT; if (cur + npages > lim) return 0; return 1; } mm/mmap.c
  86. Slide with OS and stack size table OS / Thread

    Type Main Thread Regular Thread [Default] 2.3 - 4.4 16 KB 5.0+ 8 MB 1 MB
  87. Rx and Threads • The Scheduler API • Operators and

    Schedulers • Scheduler implementations and implications
  88. Using Schedulers • Predefined, look for @SchedulerSupport @CheckReturnValue @SchedulerSupport(SchedulerSupport.COMPUTATION) public

    static Observable<Long> timer(long delay, TimeUnit unit) { return timer(delay, unit, Schedulers.computation()); }
  89. • Built-in - @SchedulerSupport • Operators’ parameter @CheckReturnValue @SchedulerSupport(SchedulerSupport.CUSTOM) public

    final Observable<T> debounce(long timeout, TimeUnit unit, Scheduler scheduler) { ... } Using Schedulers
  90. • Built-in - @SchedulerSupport • Operators’ parameter • subscribeOn and

    observeOn Observable.fromCallable(() -> 1) .subscribeOn(Schedulers.io()) .observeOn(Schedulers.computation()) .subscribe(); Using Schedulers
  91. subscribeOn Observable.fromCallable(() -> { log("callable"); return 1; }) .doOnSubscribe(d ->

    log("doOnSubscribe")) .subscribeOn(Schedulers.io()) .subscribe(); Output: doOnSubscribe runs on [IoThread-1] callable runs on [IoThread-1] • subscribeOn applies Scheduler to onSubscribe of upstream operations
  92. subscribeOn Observable.fromCallable(() -> 1) .doOnNext(d -> log("doOnNext")) .subscribeOn(Schedulers.io()) .subscribe(); Output:

    doOnNext runs on [IoThread-1] • subscribeOn applies Scheduler to onSubscribe • If observable is synchronous, onNext, onError and onComplete will continue on the same thread
  93. Output: doOnNext runs on [IoThread-1] observeOn Observable.fromArray(1) .observeOn(Schedulers.io()) .doOnNext(d ->

    log("doOnNext")) .subscribe(); • observeOn applies Scheduler to onNext, onError and onComplete
  94. Output: doOnSubscribe runs on [main] doOnNext runs on [IoThread-1] doOnComplete

    runs on [IoThread-1] observeOn Observable.fromArray(1) .observeOn(Schedulers.io()) .doOnSubscribe(d -> log("doOnSubscribe")) .doOnNext(d -> log("doOnNext")) .doOnComplete(() -> log("doOnComplete")) .subscribe(); • observeOn applies Scheduler to onNext, onError and onComplete • observeOn will not affect onSubscribe because it happens before any emission
  95. Worker • Worker is implemented as EventLoopWorker, which uses a

    NewThreadWorker, which is really a single threaded executor service • Generally speaking, worker is abstraction of thread
  96. Scheduler via worker • Operator call createWorker and receives a

    worker, use that worker for its operations.
  97. Worker vs scheduleDirect • Operators that needs serialization holds to

    a worker and use the same worker Observable.fromArray(1) .observeOn(Schedulers.io()) .observeOn(Schedulers.io()) .observeOn(Schedulers.io()) .subscribe();
  98. Into the Scheduler • Single and Completable ◦ Uses scheduleDirect

    because emission happens once. • Observable and Flowable ◦ subscribeOn apply Scheduler to operations during subscribe event ◦ observeOn holds on to worker and uses the same worker
  99. IO Scheduler • Unbounded pool, grow as requested • Cache

    and assign ThreadWorker in FIFO • Evict idle ThreadWorker after 60 second
  100. Closing Thoughts #threadsMatter • Monitor your threads • Know your

    threading approach for frequent flows • Device limitations on number of threads based on their stack size • Use appropriate Schedulers and understand the downsides of each Yohan Hartanto @Yohan Utkarsh Garg @UtkarshGarg17
  101. strace -p <pid> -o output.txt mmap(NULL, 315641856, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1,

    0) = 0x7fdfdaf000 mprotect(0x7fdfdaf000, 4096, PROT_NONE) = 0 prctl(0x53564d41 /* PR_??? */, 0, 0x7fdfdaf000, 4096, "thread stack guard page") = 0 clone(child_stack=0x7ff2ab3440, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHIL _CLEARTID, parent_tidptr=0x7ff2ab3460, tls=0x7ff2ab34e8, child_tidptr=0x7ff2ab3460) = 28238 Overcommit_memory http://engineering.pivotal.io/post/virtual_memory_settings_in_linux_-_the_problem_with_overcommit/, https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/performance_tuning_guide/sect-red_hat_ente prise_linux-performance_tuning_guide-configuration_tools-configuring_system_memory_capacity Linux version - https://www.google.com/search?ei=yqwAXLTuDom50PEP_qmzWA&q=dtb+file+viewer&oq=dtb+file+viewer&gs_l=psy-ab.3..0j0i22i30.11 4.2779..3082...0.0..1.277.1391.0j3j4......0....1..gws-wiz.......0i71j0i67.gZXHaPK-FQA https://android.googlesource.com/device/huawei/angler-kernel/ Clone ENOMEM - http://man7.org/linux/man-pages/man2/clone.2.html https://elixir.bootlin.com/linux/latest/source/security/commoncap.c#L1307 cap_vm_enough_memory Proc pagemap http://man7.org/linux/man-pages/man5/proc.5.html Mmap and swapped https://stackoverflow.com/questions/43541420/when-and-how-is-mmaped-memory-swapped-in-and-out
  102. In David's patch set, the old badness() heuristics are almost

    entirely gone. Instead, the calculation turns into a simple question of what percentage of the available memory is being used by the process. If the system as a whole is short of memory, then "available memory" is the sum of all RAM and swap space available to the system. If, instead, the OOM situation is caused by exhausting the memory allowed to a given cpuset/control group, then "available memory" is the total amount allocated to that control group. A similar calculation is made if limits imposed by a memory policy have been exceeded. In each case, the memory use of the process is deemed to be the sum of its resident set (the number of RAM pages it is using) and its swap usage. This calculation produces a percent-times-ten number as a result; a process which is using every byte of the memory available to it will have a score of 1000, while a process using no memory at all will get a score of zero. There are very few heuristic tweaks to this score, but the code does still subtract a small amount (30) from the score of root-owned processes on the notion that they are slightly more valuable than user-owned processes. One other tweak which is applied is to add the value stored in each process's oom_score_adj variable, which can be adjusted via /proc. This knob allows the adjustment of each process's attractiveness to the OOM killer in user space; setting it to -1000 will disable OOM kills entirely, while setting to +1000 is the equivalent of painting a large target on the associated process.
  103. OOM Killer angler:/proc/13947 # cat oom_score Uberlite - 33 angler:/proc/13947

    # cd ../14085 angler:/proc/14085 # cat oom_score Eats - 56 angler:/proc/14085 # cd ../14278 angler:/proc/14278 # cat oom_score Rider - 89 angler:/proc/14278 # cd ../14560 angler:/proc/14560 # cat oom_score Fleet - 29 angler:/proc/14560 # cd ../14684 angler:/proc/14684 # cat oom_score Driver - 93 Root get 3% discount, essentially 10*percent of RAM used. 1000 means all RAM used by this process https://android.googlesource.com/kernel/common/+/android-3.10.y/mm/oom_kill.c
  104. static int __pthread_start(void* arg) { pthread_internal_t* thread = reinterpret_cast<pthread_internal_t*>(arg); //

    Wait for our creating thread to release us. This lets it have time to // notify gdb about this thread before we start doing anything. // This also provides the memory barrier needed to ensure that all memory // accesses previously made by the creating thread are visible to us. thread->startup_handshake_lock.lock(); __init_alternate_signal_stack(thread); void* result = thread->start_routine(thread->start_routine_arg); pthread_exit(result); return 0; }