Process tons of jobs with Swift

45c0e67049c82c238143b82a1660a713?s=47 ainame
June 27, 2017

Process tons of jobs with Swift

Tokyo Server Side Swift meetup #8

45c0e67049c82c238143b82a1660a713?s=128

ainame

June 27, 2017
Tweet

Transcript

  1. Process tons of jobs with Swift Tokyo Server Side Swift

    meetup 2017.06.27 @ainame
  2. Self-intro • Satoshi Namai ◦ https://github.com/ainame ◦ https://twitter.com/ainame • iOS

    Engineer? • Love Ruby and Swift • Heavy user of sidekiq gem for tons of background jobs in previous job
  3. Agenda • My motivation • About sidekiq gem • ainame/Lumpik

    • vs Crystal-lang • Redis client
  4. My motivation • SSS(Server Side Swift) is very very very

    hard!! ◦ Can’t keep writing without enthusiasm ◦ It’s fun for me. • Learn not only Swift but also the System Programming ◦ Swift has compatibility for C-APIs, we can write applications with low-level APIs. • Be a Hacker ◦ Few people try ◦ This is an opportunity
  5. SSS except Web • There are many Web app frameworks

    in Swift ◦ Kitura, Vapor, Zewo, Perfect, etc… ◦ https://github.com/swift-server/http • How about the background job processing? ◦ There’s nothing? ◦ Opportunity for me
  6. Background job processing • works with Web app normaly ◦

    Web app should respond for the client quickly • works for the time-consuming tasks ◦ UPDATE query ◦ send push/e-mail notifications ◦ video encoding/image processing etc...
  7. Redis Web/Rails app Cron/Rails runner One-off scripts Worker Worker Worker

    Worker
  8. None
  9. None
  10. Sidekiq • Fastest Ruby’s job system for general purpose ◦

    works with Redis as a queue for jobs ◦ works with rails well • Why many rubyists choose sidekiq? ◦ less memory & low latency ◦ nice web admin ◦ Pro/Enterprise Support!! http://sidekiq.org/about
  11. https://www.indiehackers.com/businesses/sidekiq

  12. None
  13. Inside Sidekiq • Sidekiq has 3 main modules ◦ Processor

    - fetches a job and executes a worker on dedicated thread ◦ Poller - poll and enqueue jobs to retry ◦ Heatbeat - send statictics metadata to Redis (lib/sidekiq/launcher.rb) • Definition: Processor/Worker/Job ◦ Job is data like arguments of function for Worker ◦ Worker is the PORO what you want to do ◦ Processor fetches a job and executes
  14. Redis Poller Heatbeat Processor Processor Processor Processor Worker Worker Worker

    Worker Job Job Job Job Manager Manage retried jobs • re-enqueue retry jobs • adjust polling intervals by the number of processes Send process statuses for the admin • How many do processors work? • How many did jobs succeeded/failed? • What job dose processor work with now? Fetch
  15. FIFO queue on the Redis light jobs id=5 id=7 id=27

    …. id=58 heavy “abc” “swift” “fuga” …. “ios” > RPUSH “light” 58 > RPUSH “heavy” “ios” > BRLPOP “light” “heavy” 2 id=5 Fetch a job from two queues Append a job to light queue
  16. ainame/Lumpik

  17. ainame/Lumpik Sidekiq clone on the Swift • Fastest Swift’s job

    queue system for general purpose? • Type-safe arguments • Good concurrency with GCD • Compatibiity for sidekiq-web
  18. None
  19. None
  20. Dependencies • vapor/redis・・・ Redis client • ainame/Swift-Daemon・・・Damonize a process •

    IBM-Swift/BlueSignals・・・Handle signals • SwiftBeaver/SwiftBeaver・・・Logger • kylef/Commander・・・Option parser • jpsim/Yams・・・YAML
  21. ainame/Swift-Daemon • Daemonize a Swift process • How to detach

    a process from the shell 1. fork 2. setsid 3. dup2 • Learn from ruby/ruby
  22. Invoke a specified worker from String Swift can’t instatiate any

    class from String → Generate Router class from source code with Sourcery
  23. How to use ainame/Lumpik $ swift package init $ emacs

    Package.swift $ swift package resolve $ emacs xxxx/main.swift ... $ brew install sourcery $ sourcery --sources xxx --templates Templates --output xxx
  24. Demo

  25. ainame/Lumpik is a WIP • Not implemented yet several features

    ◦ dead queue ◦ scheduled queue • Targets for Swift 4 ◦ faster String APIs ◦ adopt Codable/JSONEncoder, but Swift doesn’t support it on the Linux currently • Cloud be faster than sidekiq.cr ◦ make faster Redis client in Swift
  26. vs sidekiq.cr Original author, @mperham makes faster sidekiq on the

    Crystal-lang • Pros ◦ mperham/sidekiq.cr is really fast ◦ ruby like syntax ◦ very fast • Cons ◦ doesn’t work CPU intensive jobs well by multithreading
  27. Benchmarks crystal-redis client is crasy fast... Redis: Crystal vs Ruby

    vs Node vs ... https://www.stefanwille.com/2015/05/redis-clients-crystal-vs-ruby-vs-c-vs-go/ https://github.com/stefanwille/redis-client-benchmarks https://github.com/ainame/vapor-redis-benchmark I have a plan to make more faster Redis client than crystal-redis but it’s difficult for several reasons...
  28. What I want for Redis client in Swift • Prefer

    method signature to enum based signature for command definition ◦ redis.set(“key”, value) vs redis.command(.set, [“key”, value]) • Prefer real typed value to enum based value for response ◦ get(_ key: String) throws -> String vs get(_ key: String) -> ResponseValue • Pipelining / MULTI-EXEC transaction ◦ Pipelining is a basic technique to speedup • Using buffered IO based TCP client ◦ buffered IO is also basic technique to speedup
  29. Recap • Sidekiq is one of the most successful OSS

    • Learn how to make good middleware with 写経 • Swift dosen’t suit for system programming now • We have to create fast and easy IO library like Ruby or Crystal
  30. Enjoy Server Side Swift!