Process tons of jobs with Swift Tokyo Server Side Swift meetup 2017.06.27 @ainame

Self-intro ● Satoshi Namai ○ https://github.com/ainame ○ https://twitter.com/ainame ● iOS Engineer? ● Love Ruby and Swift ● Heavy user of sidekiq gem for tons of background jobs in previous job

Agenda ● My motivation ● About sidekiq gem ● ainame/Lumpik ● vs Crystal-lang ● Redis client

My motivation ● SSS(Server Side Swift) is very very very hard!! ○ Can’t keep writing without enthusiasm ○ It’s fun for me. ● Learn not only Swift but also the System Programming ○ Swift has compatibility for C-APIs, we can write applications with low-level APIs. ● Be a Hacker ○ Few people try ○ This is an opportunity

SSS except Web ● There are many Web app frameworks in Swift ○ Kitura, Vapor, Zewo, Perfect, etc… ○ https://github.com/swift-server/http ● How about the background job processing? ○ There’s nothing? ○ Opportunity for me

Background job processing ● works with Web app normaly ○ Web app should respond for the client quickly ● works for the time-consuming tasks ○ UPDATE query ○ send push/e-mail notifications ○ video encoding/image processing etc...

Redis Web/Rails app Cron/Rails runner One-off scripts Worker Worker Worker Worker

No content

No content

Sidekiq ● Fastest Ruby’s job system for general purpose ○ works with Redis as a queue for jobs ○ works with rails well ● Why many rubyists choose sidekiq? ○ less memory & low latency ○ nice web admin ○ Pro/Enterprise Support!! http://sidekiq.org/about

https://www.indiehackers.com/businesses/sidekiq

No content

Inside Sidekiq ● Sidekiq has 3 main modules ○ Processor - fetches a job and executes a worker on dedicated thread ○ Poller - poll and enqueue jobs to retry ○ Heatbeat - send statictics metadata to Redis (lib/sidekiq/launcher.rb) ● Definition: Processor/Worker/Job ○ Job is data like arguments of function for Worker ○ Worker is the PORO what you want to do ○ Processor fetches a job and executes

Redis Poller Heatbeat Processor Processor Processor Processor Worker Worker Worker Worker Job Job Job Job Manager Manage retried jobs ● re-enqueue retry jobs ● adjust polling intervals by the number of processes Send process statuses for the admin ● How many do processors work? ● How many did jobs succeeded/failed? ● What job dose processor work with now? Fetch

FIFO queue on the Redis light jobs id=5 id=7 id=27 …. id=58 heavy “abc” “swift” “fuga” …. “ios” > RPUSH “light” 58 > RPUSH “heavy” “ios” > BRLPOP “light” “heavy” 2 id=5 Fetch a job from two queues Append a job to light queue

ainame/Lumpik

ainame/Lumpik Sidekiq clone on the Swift ● Fastest Swift’s job queue system for general purpose? ● Type-safe arguments ● Good concurrency with GCD ● Compatibiity for sidekiq-web

No content

No content

Dependencies ● vapor/redis・・・ Redis client ● ainame/Swift-Daemon・・・Damonize a process ● IBM-Swift/BlueSignals・・・Handle signals ● SwiftBeaver/SwiftBeaver・・・Logger ● kylef/Commander・・・Option parser ● jpsim/Yams・・・YAML

ainame/Swift-Daemon ● Daemonize a Swift process ● How to detach a process from the shell 1. fork 2. setsid 3. dup2 ● Learn from ruby/ruby

Invoke a specified worker from String Swift can’t instatiate any class from String → Generate Router class from source code with Sourcery

How to use ainame/Lumpik $ swift package init $ emacs Package.swift $ swift package resolve $ emacs xxxx/main.swift ... $ brew install sourcery $ sourcery --sources xxx --templates Templates --output xxx

Demo

ainame/Lumpik is a WIP ● Not implemented yet several features ○ dead queue ○ scheduled queue ● Targets for Swift 4 ○ faster String APIs ○ adopt Codable/JSONEncoder, but Swift doesn’t support it on the Linux currently ● Cloud be faster than sidekiq.cr ○ make faster Redis client in Swift

vs sidekiq.cr Original author, @mperham makes faster sidekiq on the Crystal-lang ● Pros ○ mperham/sidekiq.cr is really fast ○ ruby like syntax ○ very fast ● Cons ○ doesn’t work CPU intensive jobs well by multithreading

Benchmarks crystal-redis client is crasy fast... Redis: Crystal vs Ruby vs Node vs ... https://www.stefanwille.com/2015/05/redis-clients-crystal-vs-ruby-vs-c-vs-go/ https://github.com/stefanwille/redis-client-benchmarks https://github.com/ainame/vapor-redis-benchmark I have a plan to make more faster Redis client than crystal-redis but it’s difficult for several reasons...

What I want for Redis client in Swift ● Prefer method signature to enum based signature for command definition ○ redis.set(“key”, value) vs redis.command(.set, [“key”, value]) ● Prefer real typed value to enum based value for response ○ get(_ key: String) throws -> String vs get(_ key: String) -> ResponseValue ● Pipelining / MULTI-EXEC transaction ○ Pipelining is a basic technique to speedup ● Using buffered IO based TCP client ○ buffered IO is also basic technique to speedup

Recap ● Sidekiq is one of the most successful OSS ● Learn how to make good middleware with 写経 ● Swift dosen’t suit for system programming now ● We have to create fast and easy IO library like Ruby or Crystal

Enjoy Server Side Swift!