Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Building Ruby Bots on AWS Lambda

DamirSvrtan
February 21, 2017

Building Ruby Bots on AWS Lambda

Want to build a Ruby Bot without the hassle of provisioning and managing servers? Amazon's got a service for that and it's called AWS Lambda - it executes your code only when needed, scales automatically and you pay only for the compute time you consume.

There's one problem with Lambda - it doesn't support Ruby!
Let's checkout multiple options on how to build a Ruby Bot and package it into an executable which you can run on any machine without the need to actually have Ruby installed.

DamirSvrtan

February 21, 2017
Tweet

More Decks by DamirSvrtan

Other Decks in Programming

Transcript

  1. Building Ruby Bots on
    AWS Lambda

    View Slide

  2. njuskalo.hr

    View Slide

  3. View Slide

  4. # Gemfile
    gem 'httparty'
    gem 'oga'
    gem 'actionmailer'
    gem 'rake'
    gem 'whenever'

    View Slide

  5. github.com/DamirSvrtan/apartment-
    finder

    View Slide

  6. AWS Lambda

    View Slide

  7. Language support
    Java, Node.js, Python

    View Slide

  8. 29 0ct 2015

    View Slide

  9. No Ruby? :(

    View Slide

  10. Traveling Ruby
    MRuby
    JRuby

    View Slide

  11. Traveling Ruby

    View Slide

  12. mruby

    View Slide

  13. JRuby

    View Slide

  14. View Slide

  15. Shrink The Code Base

    View Slide

  16. Gemfile
    gem 'httparty'
    gem 'actionmailer'
    gem 'whenever'
    gem 'rake'
    gem 'oga'

    View Slide

  17. Use net/http
    gem 'httparty'
    gem 'actionmailer'
    gem 'whenever'
    gem 'rake'
    gem 'oga'

    View Slide

  18. Use net/http
    gem 'httparty'
    gem 'actionmailer'
    gem 'whenever'
    gem 'rake'
    gem 'oga'

    View Slide

  19. Use Lambda scheduler
    gem 'httparty'
    gem 'actionmailer'
    gem 'whenever'
    gem 'rake'
    gem 'oga'

    View Slide

  20. There must be an API!

    View Slide

  21. 1. Download the Android apk file.
    2. Decompile the apk file
    3. Reverse engineer the code.

    View Slide

  22. Failure

    View Slide

  23. 1. Install the iOS app
    2. Plug my mobile to a computer
    3. Use Charles to sniff traffic

    View Slide

  24. They have an API!

    View Slide

  25. An HTML API :(

    View Slide

  26. View Slide

  27. 14.02.2017.

    View Slide

  28. Regex it is.

    View Slide

  29. Ruby Bot Code

    View Slide

  30. # main.rb
    require 'net/http'
    require 'time'
    class ApartmentPublishDates
    LAST_SCRAPPED_AT = Time.now - 1 * 60 # 1 minute ago.
    attr_reader :url
    def initialize(url)
    @url = url
    end
    def all
    response_body
    .scan(/datetime\="(.*)" /)
    .map(&:first)
    .map { |time| Time.parse(time) }
    end
    def recent
    all.select { |publish_date| publish_date > LAST_SCRAPPED_AT }
    end
    private
    def response_body
    Net::HTTP.get(URI(url))
    end
    end

    View Slide

  31. class MailgunEmailer
    def self.call
    Net::HTTP.post_form(
    URI("https://api:key-#{MAILGUN_API_KEY}@api.mailgun.net/v3/#{MAILGUN_DOMAIN}/messages"),
    from: "Apartment Bot ",
    to: RECIPIENT,
    subject: 'New Apartments found!',
    html: "Check new apartments"
    )
    end
    end

    View Slide

  32. SEARCH_URL = 'http://iapi.njuskalo.hr?sort=new&categoryId=10920&locationId=2619'.freeze
    RECIPIENT = ENV.fetch('RECIPIENT')
    MAILGUN_API_KEY = ENV.fetch('MAILGUN_API_KEY')
    MAILGUN_DOMAIN = ENV.fetch('MAILGUN_DOMAIN')
    MailgunEmailer.call if ApartmentPublishDates.new(SEARCH_URL).recent.any?

    View Slide

  33. Implementing the Bot with
    Traveling Ruby
    Packaged MRI runtimes

    View Slide

  34. Download the runtimes
    OS X for our dev machine
    Linux x86_64 compatible binary for AWS Lambda

    View Slide

  35. curl -O https://d6r77u77i8pq3.cloudfront.net/releases/traveling-ruby-20141215-2.1.5-linux-x86_64.tar.gz
    curl -O https://d6r77u77i8pq3.cloudfront.net/releases/traveling-ruby-20141215-2.1.5-osx.tar.gz

    View Slide

  36. mkdir ruby-linux-x86_64
    tar -xzf traveling-ruby-20141215-2.1.5-linux-x86_64.tar.gz -C ruby-linux-x86_64
    mkdir ruby-os-x
    tar -xzf traveling-ruby-20141215-2.1.5-osx.tar.gz -C ruby-os-x

    View Slide

  37. traveling-ruby-bot
    - ruby-linux-x86_64/
    - ruby-os-x/

    View Slide

  38. traveling-ruby-bot
    - ruby-linux-x86_64
    - bin/
    - ruby
    - ...
    - ...
    - ruby-os-x
    - bin/
    - ruby
    - ...
    - ...

    View Slide

  39. traveling-ruby-bot
    - ruby-linux-x86_64/
    - ruby-os-x/
    - main.rb

    View Slide

  40. Execute the code
    ruby-os-x/bin/ruby main.rb

    View Slide

  41. Uploading the bot to AWS Lambda

    View Slide

  42. // lambda-function-wrapper.js
    var spawn = require('child_process').spawn;
    exports.handler = function(event, context) {
    var child = spawn('ruby-linux-x86_64/bin/ruby main.rb');
    child.stdout.on('data', function (data) { console.log("stdout:\n"+data); });
    child.stderr.on('data', function (data) { console.log("stderr:\n"+data); });
    child.on('close', function (code) { context.done(); });
    }

    View Slide

  43. zip ruby-bot.zip ruby-linux-x86_64 main.rb lambda-function-wrapper.js

    View Slide

  44. 1. Create a function

    View Slide

  45. 2. Create a trigger

    View Slide

  46. 3. Create a rule for the trigger

    View Slide

  47. 4. Upload the function code

    View Slide

  48. 5. Set a handler

    View Slide

  49. Good Traveling Ruby Tutorials
    hello world app
    app with gems
    app with native extensions such as Nokogiri

    View Slide

  50. Implementing the Bot with
    mruby

    View Slide

  51. Two ways to build apps with
    MRuby:
    - download mruby source directly from Github
    - mruby-cli - platform for building native command line applications
    for Linux, Windows, and OS X.

    View Slide

  52. Building directly from mruby source

    View Slide

  53. git clone https://github.com/mruby/mruby

    View Slide

  54. mruby-bot
    - bin
    - build_config.rb
    - minirake

    View Slide

  55. ./minirake

    View Slide

  56. mruby-bot
    - bin/
    - mruby
    - mirb
    - build_config.rb
    - minirake

    View Slide

  57. > echo "puts 'hello world'" >> hello_world.rb
    > bin/mruby hello_world.rb
    hello world

    View Slide

  58. > bin/ruby main.rb
    main.rb:1: undefined method 'require' for main (NoMethodError)

    View Slide

  59. # build_config.rb
    conf.gem github: 'iij/mruby-env'
    conf.gem github: 'mattn/mruby-curl'
    conf.gem github: 'mattn/mruby-onig-regexp'

    View Slide

  60. ./minirake

    View Slide

  61. Execute the code
    bin/mruby main.rb

    View Slide

  62. CROSS COMPILATION

    View Slide

  63. zip ruby-bot.zip bin/mruby main.rb lambda-function-wrapper.js

    View Slide

  64. Implementing the Bot with JRuby

    View Slide

  65. Download the jruby-complete.jar from
    their official downloads page.

    View Slide

  66. mv jruby-complete-9.1.7.0.jar ruby-bot.jar

    View Slide

  67. cp main.rb ruby-bot.rb

    View Slide

  68. jar -ufe ruby-bot.jar org.jruby.RubyBotMain ruby-bot.rb

    View Slide

  69. Execute the code
    java -jar ruby-bot.jar

    View Slide

  70. Metrics & Numbers

    View Slide

  71. Code Size
    Traveling Ruby 6.7 MB
    mruby 945 kB
    JRuby 23.2 MB

    View Slide

  72. Memory consumption
    Traveling Ruby 25 MB
    mruby 40 MB
    JRuby 150 MB

    View Slide

  73. Execution Time
    Traveling Ruby ~3900 ms
    mruby ~3900 ms
    JRuby ~12000 ms

    View Slide

  74. AWS Lambda Pricing

    View Slide

  75. AWS Lambda Pricing
    the number of requests (i.e. number of times our function is
    invoked)
    the sum of durations our functions took to execute (expressed in
    GB-seconds)

    View Slide

  76. Free Tier
    1M requests & 400,000 GB-seconds per month

    View Slide

  77. 1M times per month
    Rougly every 3 seconds

    View Slide

  78. 400,000 GB-seconds / month
    memory size * duration in seconds

    View Slide

  79. e.g. 128MB * 4s = 0.5GB-second

    View Slide

  80. If we ran the bot every minute
    48,960 GB-seconds

    View Slide

  81. Choosing the right Ruby for the
    job

    View Slide

  82. JRuby
    simple packaging / slow execution

    View Slide

  83. MRuby
    lightweight / good execution / bad
    library support

    View Slide

  84. Traveling Ruby
    good execution / well known
    environment

    View Slide

  85. View Slide