Fighting Fires in Swift: Lesson Learned Launching Chelsea Handler's: Gotta Go!

Recently, Yeti launched an app in conjunction with Chelsea Handler’s new Netflix documentary, Chelsea Does: Silicon Valley, on a Friday night. The team awoke to a large fire Saturday morning due to an immediate spike in traffic on the app having to do with the app's SMS and phone call functionality (i.e. Twilio, Nexmo, Plivo). In this talk Rudy will discuss the cause of the fire, how his team put it out, and what they could have done differently, sharing some Swift code and the lessons learned along the way.

http://swift-fires.surge.sh

Transcript

1. FIGHTING FIRES IN SWIFT LESSONS LEARNED LAUNCHING GOTTA GO!
2. Co-founder & Head of Tech @ Yeti Organizer of the SF Django Meetup Polyglot Developer (Python, Swift, JavaScript, Java, C#) Recently launched Tiny Eye, a VR iOS & Android game! RUDY MUTTER
3. LET'S GET SOME CONTEXT. QUICK BACKSTORY!
4. CHELSEA DOES: SILICON VALLEY ON NETFLIX
5. CHELSEA HANDLER'S: GOTTA GO!
6. JAN 22: LAUNCH EVENING
7. JAN 23: LAUNCH MORNING
8. 100s of Veri cation Codes sent Small amount of users veri ed SMS Logs saying "Undelivered" Our phone number was banned WHAT'S GOING WRONG!?
9. Python/Django Backend Celery/RabbitMQ for async Swift iOS App RestKit + Core Data Plivo for Telephony Service REWIND: SYSTEM OVERVIEW
10. User enters their number Server pings telephony service to text code User inputs code to verify their number Pretty standard auth ow VERIFYING YOUR NUMBER
11. RestKit API Call A VIPER Module Core Data Classes @ o b j c ( V e r i f i c a t i o n R e q u e s t ) c l a s s V e r i f i c a t i o n R e q u e s t : N S M a n a g e d O b j e c t { } e x t e n s i o n V e r i f i c a t i o n R e q u e s t { @ N S M a n a g e d v a r c o d e : S t r i n g ? } f u n c p o s t V e r i f y C o d e W i t h C o d e ( c o d e : S t r i n g , s u c c e s s : R e s t K i t S u c c e s s , f a i l u r e : R e s t K i t E r r o r ) { l e t s h a r e d M g r = R K O b j e c t M a n a g e r . s h a r e d M a n a g e r ( ) s h a r e d M g r . r e q u e s t S e r i a l i z a t i o n M I M E T y p e = R K M I M E T y p e J S O N l e t o b j = N S E n t i t y D e s c r i p t i o n . i n s e r t N e w O b j e c t F o r E n t i t y F o r N a m e ( " V e r i f i c a t i o n R e q u e s t " , i n M a n a g e d O b j e c t C o n t e x t o b j . c o d e = c o d e s h a r e d M g r . p o s t O b j e c t ( o b j , p a t h : " v e r i f y _ c o d e / " , p a r a m e t e r s : n i l , s u c c e s s : s u c c e s s , f a i l u r e : f a i l u r e ) } VERIFYING CONT. - VIPER
12. Add/Edit contact with our numbers Allow user to edit name + photo When editing, delete then re-add Built before iOS 9's new contact framework ADDING CONTACTS - ADDRESSBOOK
13. Store numbers appropriate to user's country l e t c o u n t r y = A p p M o d e l . s h a r e d M o d e l . g e t C o u n t r y ( ) l e t a m e r i c a n N u m b e r s : C o n t a c t N u m b e r s = C o n t a c t N u m b e r s ( v o i c e N u m b e r : " ( 3 3 0 ) 2 6 9 - 5 4 7 1 " , t e x t N u m b e r : " 7 9 2 - 2 7 3 " ) l e t c a n a d i a n N u m b e r s : C o n t a c t N u m b e r s = C o n t a c t N u m b e r s ( v o i c e N u m b e r : " ( 6 0 4 ) 4 2 5 - 1 1 5 5 " , t e x t N u m b e r : " ( 6 0 4 ) 4 2 5 - 1 1 5 5 " ) ADDING CONTACTS CONT.
14. Find contact with a matching number s t a t i c f u n c f i n d C o n t a c t ( n u m b e r : S t r i n g ) - > A B R e c o r d R e f ? { i f C o n t a c t . h a s A d d r e s s B o o k A c c e s s ( ) { v a r e r r o r : U n m a n a g e d < C F E r r o r > ? / / c r e a t e a d d r e s s b o o k i n s t a n c e l e t a d d r e s s B o o k : A B A d d r e s s B o o k R e f = A B A d d r e s s B o o k C r e a t e W i t h O p t i o n s ( n i l , & e r r o r ) . t a k e U n r e t a i n e d V a l u e ( / / g e t a l l c o n t a c t s l e t c o n t a c t s = A B A d d r e s s B o o k C o p y A r r a y O f A l l P e o p l e ( a d d r e s s B o o k ) . t a k e R e t a i n e d V a l u e ( ) a s A r r a y f o r r e c o r d i n c o n t a c t s { l e t c u r r e n t C o n t a c t : A B R e c o r d R e f = r e c o r d l e t n u m b e r s : A B M u l t i V a l u e R e f = A B R e c o r d C o p y V a l u e ( c u r r e n t C o n t a c t , k A B P e r s o n P h o n e P r o p e r t y ) . t a k e U n r e t a i n e d V a l u e / / l o o p t h r o u g h c o n t a c t ' s n u m b e r s f o r ( v a r j = 0 ; j < A B M u l t i V a l u e G e t C o u n t ( n u m b e r s ) ; j + + ) { l e t p h o n e N u m b e r = A B M u l t i V a l u e C o p y V a l u e A t I n d e x ( n u m b e r s , j ) . t a k e U n r e t a i n e d V a l u e ( ) a s ! S t r i n g / / i f t h e p h o n e n u m b e r m a t c h e s t h e f i r s t c o n t a c t n u m b e r , w e ' v e g o t a c o n t a c t m a t c h . i f p h o n e N u m b e r = = n u m b e r { r e t u r n c u r r e n t C o n t a c t } } } } r e t u r n n i l } ADDING CONTACTS CONT. II
15. Set name, image, and numbers on new contact f u n c c r e a t e C o n t a c t ( ) - > B o o l { l e t n e w C o n t a c t : A B R e c o r d R e f ! = A B P e r s o n C r e a t e ( ) . t a k e R e t a i n e d V a l u e ( ) v a r e r r o r : U n m a n a g e d < C F E r r o r R e f > ? l e t f i r s t N a m e S u c c e s s = A B R e c o r d S e t V a l u e ( n e w C o n t a c t , k A B P e r s o n F i r s t N a m e P r o p e r t y , s e l f . f i r s t N a m e , & e r r o r l e t l a s t N a m e S u c c e s s = A B R e c o r d S e t V a l u e ( n e w C o n t a c t , k A B P e r s o n L a s t N a m e P r o p e r t y , s e l f . l a s t N a m e , & e r r o r ) i f l e t i m a g e = i m a g e { l e t p n g I m a g e = U I I m a g e P N G R e p r e s e n t a t i o n ( i m a g e ) ! l e t c f D a t a R e f = C F D a t a C r e a t e ( n i l , U n s a f e P o i n t e r ( p n g I m a g e . b y t e s ) , p n g I m a g e . l e n g t h ) A B P e r s o n S e t I m a g e D a t a ( n e w C o n t a c t , c f D a t a R e f , & e r r o r ) } l e t m u l t i S t r i n g P r o p e r t y = A B P r o p e r t y T y p e ( k A B M u l t i S t r i n g P r o p e r t y T y p e ) l e t p h o n e N u m b e r s : A B M u t a b l e M u l t i V a l u e = A B M u l t i V a l u e C r e a t e M u t a b l e ( m u l t i S t r i n g P r o p e r t y ) . t a k e U n r e t a i n e d V a l u e A B M u l t i V a l u e A d d V a l u e A n d L a b e l ( p h o n e N u m b e r s , m a i n P h o n e N u m b e r , k A B P e r s o n P h o n e M a i n L a b e l , n i l ) l e t m a i n N u m b e r S u c c e s s = A B R e c o r d S e t V a l u e ( n e w C o n t a c t , k A B P e r s o n P h o n e P r o p e r t y , p h o n e N u m b e r s , & e r r o r ) A B M u l t i V a l u e A d d V a l u e A n d L a b e l ( p h o n e N u m b e r s , m o b i l e P h o n e N u m b e r , k A B P e r s o n P h o n e M o b i l e L a b e l , n i l ) l e t m o b i l e N u m b e r S u c c e s s = A B R e c o r d S e t V a l u e ( n e w C o n t a c t , k A B P e r s o n P h o n e P r o p e r t y , p h o n e N u m b e r s , & e r r o r ) r e t u r n s a v e C o n t a c t T o P h o n e ( n e w C o n t a c t ) } ADDING CONTACTS CONT. III
16. Craft a believable story Add up to 1 phone call and 7 texts Excuses stored locally in Core Data Schedule it for X mins/hours later SAVING & CREATING EXCUSES
17. Fetch user's excuses from Core Data f u n c g e t E x c u s e s ( ) - > [ E x c u s e ] { l e t f e t c h R e q u e s t = N S F e t c h R e q u e s t ( e n t i t y N a m e : " E x c u s e " ) l e t c r e a t e d S o r t D e s c r i p t o r = N S S o r t D e s c r i p t o r ( k e y : " c r e a t e d A t " , a s c e n d i n g : f a l s e ) f e t c h R e q u e s t . s o r t D e s c r i p t o r s = [ c r e a t e d S o r t D e s c r i p t o r ] r e t u r n ( t r y ! m a n a g e d O b j e c t C o n t e x t ! . e x e c u t e F e t c h R e q u e s t ( f e t c h R e q u e s t ) ) a s ! [ E x c u s e ] } Create default excuses rst time through (Emoji's in Xcode!) n e w E x c u s e = E x c u s e . c r e a t e I n M a n a g e d O b j e c t C o n t e x t ( m a n a g e d O b j e c t C o n t e x t ! , e m o j i : " " , c o l o r : C o n s t a n t s . e x c u s e C e l l C o l o r s [ 5 ] , d e l a y T i m e M i n u t e s : 0 ) l e t l o c k e d O u t M e s s a g e s : [ S t r i n g ? ] = [ " I ' m l o c k e d o u t . " , " T h i n k I m a y h a v e t r i g g e r e d t h e s i l e n t a l a r m . C a n y o u g e t h e r e r i g h t a w a y ? " , " L a s t t i m e I d i d t h i s t h e c o p s s h o w e d u p . P L E A S E H U R R Y ! " ] a d d M e s s a g e s T o E x c u s e ( l o c k e d O u t M e s s a g e s , e x c u s e : n e w E x c u s e ) SAVING & CREATING EXCUSES CONT.
18. SOME CONSIDERATIONS Older OS's don't support new emoji Di erent skin toned emoji Saving emoji preference Grouping emoji together Credit @leemcdole EMOJI KEYBOARD FOR EXCUSES
19. Schedule a task to ping the telephony service X mins later d e f s e n d _ m e s s a g e ( s e l f , t o _ n u m b e r , m i n u t e s , m e s s a g e , c o u n t r y ) : s e n d _ t i m e = t i m e z o n e . n o w ( ) + d a t e t i m e . t i m e d e l t a ( m i n u t e s = m i n u t e s ) r e s u l t = s e n d _ m e s s a g e . a p p l y _ a s y n c ( ( t o _ n u m b e r , m e s s a g e , c o u n t r y , s e l f . p k ) , e t a = s e n d _ t i m e , e x p i r e s = s e n d _ t i m e + d a t e t i m e . t i m e d e l t a ( m i n u t e s = s e t t i n g s . E X P I R A T I O N _ M I N U T E S ) ) s e l f . s a v e ( ) SCHEDULING EXCUSES (SERVER)
20. Plivo provided a library for sending SMS or Voice d e f s e n d _ t e x t ( d e s t i n a t i o n , t e x t , s o u r c e _ p h o n e _ n u m b e r ) : p a r a m s = { " s r c " : s o u r c e _ p h o n e _ n u m b e r , " d s t " : d e s t i n a t i o n , " t e x t " : t e x t } r e t u r n p l i v o . R e s t A P I ( s e t t i n g s . P L I V O _ I D , s e t t i n g s . P L I V O _ S E C R E T ) . s e n d _ m e s s a g e ( p a r a m s ) d e f s e n d _ c a l l ( d e s t i n a t i o n , c o u n t r y ) : s i t e = S i t e . o b j e c t s . g e t _ c u r r e n t ( ) p a r a m s = { " f r o m " : s e t t i n g s . P L I V O _ V O I C E _ N U M B E R [ c o u n t r y ] , " t o " : d e s t i n a t i o n , " a n s w e r _ u r l " : " { } : / / { } / a p i / v 1 / p l i v o _ a n s w e r / " . f o r m a t ( " h t t p s " i f s e t t i n g s . U S I N G _ S S L e l s e " h t t p " , s i t e " t i m e _ l i m i t " : 3 5 } r e t u r n p l i v o . R e s t A P I ( s e t t i n g s . P L I V O _ I D , s e t t i n g s . P L I V O _ S E C R E T ) . m a k e _ c a l l ( p a r a m s ) SCHEDULING EXCUSES (SERVER) CONT.
21. Purchased 10 more numbers Round robin sending texts between them Texts coming from wrong number Tra c on the app was increasing Banned again after an hour FIXING THE FIRES PART #1
22. Bought a Shortcode Meant for high volume texts Much more expensive than before Hindsight 20/20 Weekend had ~20k veri ed users FIXING THE FIRES PART #2
23. iOS app has hardcoded numbers Submitted emergency build to Apple, approved next day FIXING THE FIRES PART #3
24. Contact numbers should be dynamic ^ Served via the API Load balance tested SMS/Voice sending Beta test was not enough messages/second WHAT DID WE DO WRONG?
25. Analytics and Monitoring! 99.9% crash free users Still ~700 users/day WHAT DID WE DO RIGHT?
26. @alekhinen @anyatran @baylee @leemcdole @paolopaolopaolo @rmutter @tonydotsc @walkingtowork @winnietong @ZachGawlik THANKS TO DEVS @ YETI!
27. ANY QUESTIONS?