Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Process tons of jobs with Swift

ainame
June 27, 2017

Process tons of jobs with Swift

Tokyo Server Side Swift meetup #8

ainame

June 27, 2017
Tweet

More Decks by ainame

Other Decks in Programming

Transcript

  1. Process tons of jobs
    with Swift
    Tokyo Server Side Swift meetup
    2017.06.27
    @ainame

    View full-size slide

  2. Self-intro
    ● Satoshi Namai
    ○ https://github.com/ainame
    ○ https://twitter.com/ainame
    ● iOS Engineer?
    ● Love Ruby and Swift
    ● Heavy user of sidekiq gem for tons of background jobs in previous job

    View full-size slide

  3. Agenda
    ● My motivation
    ● About sidekiq gem
    ● ainame/Lumpik
    ● vs Crystal-lang
    ● Redis client

    View full-size slide

  4. My motivation
    ● SSS(Server Side Swift) is very very very hard!!
    ○ Can’t keep writing without enthusiasm
    ○ It’s fun for me.
    ● Learn not only Swift but also the System Programming
    ○ Swift has compatibility for C-APIs, we can write applications with low-level APIs.
    ● Be a Hacker
    ○ Few people try
    ○ This is an opportunity

    View full-size slide

  5. SSS except Web
    ● There are many Web app frameworks in Swift
    ○ Kitura, Vapor, Zewo, Perfect, etc…
    ○ https://github.com/swift-server/http
    ● How about the background job processing?
    ○ There’s nothing?
    ○ Opportunity for me

    View full-size slide

  6. Background job processing
    ● works with Web app normaly
    ○ Web app should respond for the client quickly
    ● works for the time-consuming tasks
    ○ UPDATE query
    ○ send push/e-mail notifications
    ○ video encoding/image processing
    etc...

    View full-size slide

  7. Redis
    Web/Rails app
    Cron/Rails runner
    One-off scripts
    Worker
    Worker
    Worker
    Worker

    View full-size slide

  8. Sidekiq
    ● Fastest Ruby’s job system for general purpose
    ○ works with Redis as a queue for jobs
    ○ works with rails well
    ● Why many rubyists choose sidekiq?
    ○ less memory & low latency
    ○ nice web admin
    ○ Pro/Enterprise Support!!
    http://sidekiq.org/about

    View full-size slide

  9. https://www.indiehackers.com/businesses/sidekiq

    View full-size slide

  10. Inside Sidekiq
    ● Sidekiq has 3 main modules
    ○ Processor - fetches a job and executes a worker on dedicated thread
    ○ Poller - poll and enqueue jobs to retry
    ○ Heatbeat - send statictics metadata to Redis (lib/sidekiq/launcher.rb)
    ● Definition: Processor/Worker/Job
    ○ Job is data like arguments of function for Worker
    ○ Worker is the PORO what you want to do
    ○ Processor fetches a job and executes

    View full-size slide

  11. Redis
    Poller
    Heatbeat
    Processor
    Processor
    Processor
    Processor
    Worker
    Worker
    Worker
    Worker
    Job
    Job
    Job
    Job
    Manager
    Manage retried jobs
    ● re-enqueue retry jobs
    ● adjust polling intervals by the number of processes
    Send process statuses for the admin
    ● How many do processors work?
    ● How many did jobs succeeded/failed?
    ● What job dose processor work with now?
    Fetch

    View full-size slide

  12. FIFO queue on the Redis
    light
    jobs
    id=5 id=7 id=27 ….
    id=58
    heavy “abc” “swift” “fuga” ….
    “ios”
    > RPUSH “light” 58
    > RPUSH “heavy” “ios”
    > BRLPOP “light” “heavy” 2
    id=5
    Fetch a job from
    two queues
    Append a job to light queue

    View full-size slide

  13. ainame/Lumpik

    View full-size slide

  14. ainame/Lumpik
    Sidekiq clone on the Swift
    ● Fastest Swift’s job queue system for general purpose?
    ● Type-safe arguments
    ● Good concurrency with GCD
    ● Compatibiity for sidekiq-web

    View full-size slide

  15. Dependencies
    ● vapor/redis・・・ Redis client
    ● ainame/Swift-Daemon・・・Damonize a process
    ● IBM-Swift/BlueSignals・・・Handle signals
    ● SwiftBeaver/SwiftBeaver・・・Logger
    ● kylef/Commander・・・Option parser
    ● jpsim/Yams・・・YAML

    View full-size slide

  16. ainame/Swift-Daemon
    ● Daemonize a Swift process
    ● How to detach a process from the shell
    1. fork
    2. setsid
    3. dup2
    ● Learn from ruby/ruby

    View full-size slide

  17. Invoke a specified worker from String
    Swift can’t instatiate any class from String
    → Generate Router class from source code with Sourcery

    View full-size slide

  18. How to use ainame/Lumpik
    $ swift package init
    $ emacs Package.swift
    $ swift package resolve
    $ emacs xxxx/main.swift
    ...
    $ brew install sourcery
    $ sourcery --sources xxx --templates Templates --output xxx

    View full-size slide

  19. ainame/Lumpik is a WIP
    ● Not implemented yet several features
    ○ dead queue
    ○ scheduled queue
    ● Targets for Swift 4
    ○ faster String APIs
    ○ adopt Codable/JSONEncoder, but Swift doesn’t support it on the Linux currently
    ● Cloud be faster than sidekiq.cr
    ○ make faster Redis client in Swift

    View full-size slide

  20. vs sidekiq.cr
    Original author, @mperham makes faster sidekiq on the Crystal-lang
    ● Pros
    ○ mperham/sidekiq.cr is really fast
    ○ ruby like syntax
    ○ very fast
    ● Cons
    ○ doesn’t work CPU intensive jobs well by multithreading

    View full-size slide

  21. Benchmarks
    crystal-redis client is crasy fast...
    Redis: Crystal vs Ruby vs Node vs ...
    https://www.stefanwille.com/2015/05/redis-clients-crystal-vs-ruby-vs-c-vs-go/
    https://github.com/stefanwille/redis-client-benchmarks
    https://github.com/ainame/vapor-redis-benchmark
    I have a plan to make more faster Redis client than crystal-redis
    but it’s difficult for several reasons...

    View full-size slide

  22. What I want for Redis client in Swift
    ● Prefer method signature to enum based signature for command definition
    ○ redis.set(“key”, value) vs redis.command(.set, [“key”, value])
    ● Prefer real typed value to enum based value for response
    ○ get(_ key: String) throws -> String vs get(_ key: String) -> ResponseValue
    ● Pipelining / MULTI-EXEC transaction
    ○ Pipelining is a basic technique to speedup
    ● Using buffered IO based TCP client
    ○ buffered IO is also basic technique to speedup

    View full-size slide

  23. Recap
    ● Sidekiq is one of the most successful OSS
    ● Learn how to make good middleware with 写経
    ● Swift dosen’t suit for system programming now
    ● We have to create fast and easy IO library like Ruby or Crystal

    View full-size slide

  24. Enjoy Server Side Swift!

    View full-size slide