Out of memory can be caused by incorrect usage of threads. This talk deep dives into how threading works under the hood in Android and how to avoid it causing a crash in your app.
Presented in Droidcon San Francisco, 2018 by Utkarsh and Yohan
Multiple threading strategies ◦ new Thread() ◦ ExecutorService ◦ AsyncTask ◦ ReactiveX/RxJava • Lots and lots of Out of Memory Crashes • ~40% of which were threading related
Multiple threading strategies ◦ new Thread() ◦ ExecutorService ◦ AsyncTask ◦ ReactiveX/RxJava New App • Unify threading into RxJava • Lots and lots of Out of Memory Crashes • ~40% of which were threading related
Multiple threading strategies ◦ new Thread() ◦ ExecutorService ◦ AsyncTask ◦ ReactiveX/RxJava New App • Unify threading into RxJava • Lots and lots of Out of Memory Crashes • ~40% of which were threading related • Still lots of Out of Memory Crashes • ~50% of which are threading related
again at java.lang.Thread.nativeCreate(Native Method) at java.lang.Thread.start(Thread.java:1078) at com.ubercab.threadlimits.MainActivity$5.run(MainActivity.java:136) at android.os.Handler.handleCallback(Handler.java:751) at android.os.Handler.dispatchMessage(Handler.java:95) at android.os.Looper.loop(Looper.java:154) at android.app.ActivityThread.main(ActivityThread.java:6121) at java.lang.reflect.Method.invoke(Native Method) at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:889) at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:779)
java.lang.Thread.nativeCreate(Native Method) at java.lang.Thread.start(Thread.java:1078) at com.ubercab.threadlimits.MainActivity$5.run(MainActivity.java:136) at android.os.Handler.handleCallback(Handler.java:751) at android.os.Handler.dispatchMessage(Handler.java:95) at android.os.Looper.loop(Looper.java:154) at android.app.ActivityThread.main(ActivityThread.java:6121) at java.lang.reflect.Method.invoke(Native Method) at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:889) at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:779) “Unlimited resources?”
java.lang.Thread.nativeCreate(Native Method) at java.lang.Thread.start(Thread.java:1078) at com.ubercab.threadlimits.MainActivity$5.run(MainActivity.java:136) at android.os.Handler.handleCallback(Handler.java:751) at android.os.Handler.dispatchMessage(Handler.java:95) at android.os.Looper.loop(Looper.java:154) at android.app.ActivityThread.main(ActivityThread.java:6121) at java.lang.reflect.Method.invoke(Native Method) at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:889) at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:779) “Unlimited resources?”
java.lang.Thread.nativeCreate(Native Method) at java.lang.Thread.start(Thread.java:1078) at com.ubercab.threadlimits.MainActivity$5.run(MainActivity.java:136) at android.os.Handler.handleCallback(Handler.java:751) at android.os.Handler.dispatchMessage(Handler.java:95) at android.os.Looper.loop(Looper.java:154) at android.app.ActivityThread.main(ActivityThread.java:6121) at java.lang.reflect.Method.invoke(Native Method) at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:889) at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:779) “Unlimited resources?”
java.lang.Thread.nativeCreate(Native Method) at java.lang.Thread.start(Thread.java:1078) at com.ubercab.threadlimits.MainActivity$5.run(MainActivity.java:136) at android.os.Handler.handleCallback(Handler.java:751) at android.os.Handler.dispatchMessage(Handler.java:95) at android.os.Looper.loop(Looper.java:154) at android.app.ActivityThread.main(ActivityThread.java:6121) at java.lang.reflect.Method.invoke(Native Method) at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:889) at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:779) “Unlimited resources?”
java.lang.Thread.nativeCreate(Native Method) at java.lang.Thread.start(Thread.java:1078) at com.ubercab.threadlimits.MainActivity$5.run(MainActivity.java:136) at android.os.Handler.handleCallback(Handler.java:751) at android.os.Handler.dispatchMessage(Handler.java:95) at android.os.Looper.loop(Looper.java:154) at android.app.ActivityThread.main(ActivityThread.java:6121) at java.lang.reflect.Method.invoke(Native Method) at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:889) at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:779) “Unlimited resources?”
java.lang.Thread.nativeCreate(Native Method) at java.lang.Thread.start(Thread.java:1078) at com.ubercab.threadlimits.MainActivity$5.run(MainActivity.java:136) at android.os.Handler.handleCallback(Handler.java:751) at android.os.Handler.dispatchMessage(Handler.java:95) at android.os.Looper.loop(Looper.java:154) at android.app.ActivityThread.main(ActivityThread.java:6121) at java.lang.reflect.Method.invoke(Native Method) at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:889) at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:779) “Unlimited resources?”
this thread is about to be started * so that it can be added to the group's list of threads * and the group's unstarted count can be decremented. */ group.add(this); started = false; try { nativeCreate(this, stackSize, daemon); started = true; } finally { <Error Handling> } } private native static void nativeCreate(Thread t, long stackSize, boolean daemon); Thread.java
java.lang.Thread.nativeCreate(Native Method) at java.lang.Thread.start(Thread.java:1078) at com.ubercab.threadlimits.MainActivity$5.run(MainActivity.java:136) at android.os.Handler.handleCallback(Handler.java:751) at android.os.Handler.dispatchMessage(Handler.java:95) at android.os.Looper.loop(Looper.java:154) at android.app.ActivityThread.main(ActivityThread.java:6121) at java.lang.reflect.Method.invoke(Native Method) at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:889) at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:779) “Unlimited resources?”
java.lang.Thread.nativeCreate(Native Method) at java.lang.Thread.start(Thread.java:1078) at com.ubercab.threadlimits.MainActivity$5.run(MainActivity.java:136) at android.os.Handler.handleCallback(Handler.java:751) at android.os.Handler.dispatchMessage(Handler.java:95) at android.os.Looper.loop(Looper.java:154) at android.app.ActivityThread.main(ActivityThread.java:6121) at java.lang.reflect.Method.invoke(Native Method) at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:889) at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:779) “Unlimited resources?”
at com.ubercab.threadlimits.MainActivity.overflow(MainActivity.java:81) . . . stack=java.lang.StackOverflowError: stack size 1040KB at com.ubercab.threadlimits.MainActivity$2.run(MainActivity.java:146) at com.ubercab.threadlimits.MainActivity$2.run(MainActivity.java:146) at com.ubercab.threadlimits.MainActivity$2.run(MainActivity.java:146) at com.ubercab.threadlimits.MainActivity$2.run(MainActivity.java:146) . . .
pthread default stack size for native threads, // so include that here to support apps that expect large native stacks. stack_size += 1 * MB; // It's not possible to request a stack smaller than the system-defined PTHREAD_STACK_MIN. if (stack_size < PTHREAD_STACK_MIN) { stack_size = PTHREAD_STACK_MIN; } // add protected and stack overflow region space to stack_size ... // Some systems require the stack size to be a multiple of the system page size, so round up. stack_size = RoundUp(stack_size, kPageSize); return stack_size; } thread.cc
pthread default stack size for native threads, // so include that here to support apps that expect large native stacks. stack_size += 1 * MB; // It's not possible to request a stack smaller than the system-defined PTHREAD_STACK_MIN. if (stack_size < PTHREAD_STACK_MIN) { stack_size = PTHREAD_STACK_MIN; } // add protected and stack overflow region space to stack_size ... // Some systems require the stack size to be a multiple of the system page size, so round up. stack_size = RoundUp(stack_size, kPageSize); return stack_size; } thread.cc
pthread default stack size for native threads, // so include that here to support apps that expect large native stacks. stack_size += 1 * MB; // It's not possible to request a stack smaller than the system-defined PTHREAD_STACK_MIN. if (stack_size < PTHREAD_STACK_MIN) { stack_size = PTHREAD_STACK_MIN; } // add protected and stack overflow region space to stack_size ... // Some systems require the stack size to be a multiple of the system page size, so round up. stack_size = RoundUp(stack_size, kPageSize); return stack_size; } thread.cc
pthread default stack size for native threads, // so include that here to support apps that expect large native stacks. stack_size += 1 * MB; // It's not possible to request a stack smaller than the system-defined PTHREAD_STACK_MIN. if (stack_size < PTHREAD_STACK_MIN) { stack_size = PTHREAD_STACK_MIN; } // add protected and stack overflow region space to stack_size ... // Some systems require the stack size to be a multiple of the system page size, so round up. stack_size = RoundUp(stack_size, kPageSize); return stack_size; } thread.cc
at java.lang.Thread.start(Thread.java:1078) at com.ubercab.threadlimits.MainActivity$5.run(MainActivity.java:136) at android.os.Handler.handleCallback(Handler.java:751) at android.os.Handler.dispatchMessage(Handler.java:95) at android.os.Looper.loop(Looper.java:154) at android.app.ActivityThread.main(ActivityThread.java:6121) at java.lang.reflect.Method.invoke(Native Method) at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:889) at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:779)
Out of memory W art : Throwing OutOfMemoryError "pthread_create (301MB stack) failed: Try again" W libc : pthread_create failed: clone failed: Try again W art : Throwing OutOfMemoryError "pthread_create (1040KB stack) failed: Try again" java.lang.OutOfMemoryError: pthread_create (1040KB stack) failed: Try again at java.lang.Thread.nativeCreate(Native Method) at java.lang.Thread.start(Thread.java:1078) at com.ubercab.threadlimits.MainActivity$5.run(MainActivity.java:136) at android.os.Handler.handleCallback(Handler.java:751) at android.os.Handler.dispatchMessage(Handler.java:95) at android.os.Looper.loop(Looper.java:154) at android.app.ActivityThread.main(ActivityThread.java:6121) at java.lang.reflect.Method.invoke(Native Method) at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:889) at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:779)
java.lang.OutOfMemoryError: pthread_create (1040KB stack) failed: Try again at com.ubercab.threadlimits.MainActivity$5.run(MainActivity.java:136) at android.os.Handler.handleCallback(Handler.java:751) at android.os.Handler.dispatchMessage(Handler.java:95) at android.os.Looper.loop(Looper.java:154) at android.app.ActivityThread.main(ActivityThread.java:6121) at java.lang.reflect.Method.invoke(Native Method) at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:889) at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:779) Force finishing activity com.ubercab.threadlimits/.MainActivity W libc : pthread_create failed: couldn't allocate 315641856-bytes mapped space: Out of memory W art : Throwing OutOfMemoryError "pthread_create (301MB stack) failed: Try again" W libc : pthread_create failed: clone failed: Try again W art : Throwing OutOfMemoryError "pthread_create (1040KB stack) failed: Try again"
unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 9447 9447 processes Max open files 1024 4096 files Max locked memory 67108864 67108864 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 9447 9447 signals Max msgqueue size 819200 819200 bytes Max nice priority 40 40 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us ulimit -a
java.lang.OutOfMemoryError: pthread_create (1040KB stack) failed: Try again at java.lang.Thread.nativeCreate(Native Method) at java.lang.Thread.start(Thread.java:1078) at com.ubercab.threadlimits.MainActivity$5.run(MainActivity.java:136) at android.os.Handler.handleCallback(Handler.java:751) at android.os.Handler.dispatchMessage(Handler.java:95) at android.os.Looper.loop(Looper.java:154) at android.app.ActivityThread.main(ActivityThread.java:6121) at java.lang.reflect.Method.invoke(Native Method) at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:889) at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:779) Force finishing activity com.ubercab.threadlimits/.MainActivity W libc : pthread_create failed: couldn't allocate 315641856-bytes mapped space: Out of memory W art : Throwing OutOfMemoryError "pthread_create (301MB stack) failed: Try again" W libc : pthread_create failed: clone failed: Try again W art : Throwing OutOfMemoryError "pthread_create (1040KB stack) failed: Try again"
len, unsigned long flags, unsigned int vm_flags, unsigned long pgoff) { /* Check against address space limit. */ if (!may_expand_vm(mm, len >> PAGE_SHIFT)) return -ENOMEM; } int may_expand_vm(struct mm_struct *mm, unsigned long npages) { unsigned long cur = mm->total_vm; /* pages */ unsigned long lim; lim = current->signal->rlim[RLIMIT_AS].rlim_cur >> PAGE_SHIFT; if (cur + npages > lim) return 0; return 1; } mm/mmap.c
doOnNext runs on [IoThread-1] • subscribeOn applies Scheduler to onSubscribe • If observable is synchronous, onNext, onError and onComplete will continue on the same thread
runs on [IoThread-1] observeOn Observable.fromArray(1) .observeOn(Schedulers.io()) .doOnSubscribe(d -> log("doOnSubscribe")) .doOnNext(d -> log("doOnNext")) .doOnComplete(() -> log("doOnComplete")) .subscribe(); • observeOn applies Scheduler to onNext, onError and onComplete • observeOn will not affect onSubscribe because it happens before any emission
a worker and use the same worker Observable.fromArray(1) .observeOn(Schedulers.io()) .observeOn(Schedulers.io()) .observeOn(Schedulers.io()) .subscribe();
because emission happens once. • Observable and Flowable ◦ subscribeOn apply Scheduler to operations during subscribe event ◦ observeOn holds on to worker and uses the same worker
threading approach for frequent flows • Device limitations on number of threads based on their stack size • Use appropriate Schedulers and understand the downsides of each Yohan Hartanto @Yohan Utkarsh Garg @UtkarshGarg17
entirely gone. Instead, the calculation turns into a simple question of what percentage of the available memory is being used by the process. If the system as a whole is short of memory, then "available memory" is the sum of all RAM and swap space available to the system. If, instead, the OOM situation is caused by exhausting the memory allowed to a given cpuset/control group, then "available memory" is the total amount allocated to that control group. A similar calculation is made if limits imposed by a memory policy have been exceeded. In each case, the memory use of the process is deemed to be the sum of its resident set (the number of RAM pages it is using) and its swap usage. This calculation produces a percent-times-ten number as a result; a process which is using every byte of the memory available to it will have a score of 1000, while a process using no memory at all will get a score of zero. There are very few heuristic tweaks to this score, but the code does still subtract a small amount (30) from the score of root-owned processes on the notion that they are slightly more valuable than user-owned processes. One other tweak which is applied is to add the value stored in each process's oom_score_adj variable, which can be adjusted via /proc. This knob allows the adjustment of each process's attractiveness to the OOM killer in user space; setting it to -1000 will disable OOM kills entirely, while setting to +1000 is the equivalent of painting a large target on the associated process.
Wait for our creating thread to release us. This lets it have time to // notify gdb about this thread before we start doing anything. // This also provides the memory barrier needed to ensure that all memory // accesses previously made by the creating thread are visible to us. thread->startup_handshake_lock.lock(); __init_alternate_signal_stack(thread); void* result = thread->start_routine(thread->start_routine_arg); pthread_exit(result); return 0; }