$30 off During Our Annual Pro Sale. View Details »

FaaS Measurement Fundamentals

FaaS Measurement Fundamentals

"Don't Worry About Servers, Still Worry About Metrics: FaaS Measurement Fundamentals"

Talk at Gluecon 2017.

Clay Smith

May 24, 2017
Tweet

More Decks by Clay Smith

Other Decks in Programming

Transcript

  1. Don't Worry About Servers
    Still Worry About Metrics
    FaaS Measurement Fundamentals
    @smithclay
    New Relic
    Gluecon 5/24/17

    View Slide

  2. Metrics are what we
    measure
    *hopefully useful things

    View Slide

  3. λ
    This Thing Appeared
    The Magic Hat
    of Werner Vogels
    How do we
    understand it?

    View Slide

  4. Hyped tech wish list
    Metrics Trends
    Alerting CTA
    Logging Detail
    Tracing Cause
    Analytics All of the above

    View Slide

  5. MALTA Observability Index for FaaS
    Metrics
    Alerting
    Logging
    Tracing
    Analytics
    Maturity Level

    View Slide

  6. Built-in FaaS Metrics*
    Error Count
    Function Invocation Count
    Function Duration (ms)
    * not comprehensive, but the important ones

    View Slide

  7. Why does function
    invocation time vary so
    much?

    View Slide

  8. Event
    Trigger
    1. Invoke
    λ
    2. Run 3. End
    Result
    Error
    Timeout
    or
    or

    View Slide

  9. Cold Start vs Warm Start
    Event
    Trigger
    Handler Code
    Warm
    Function Invocation Time
    Create Initialize Handler Code
    Cold

    View Slide

  10. What's inside AWS
    Lambda?
    "It's containers" — Person waving their hands

    View Slide

  11. λ: Running Commands for Discovery
    const exec = require('child_process').exec;
    exports.handler = (trigger, cb) => {
    exec('whoami', (err, stdout) => {
    console.log(stdout);
    return cb(null);
    });
    }
    [LOG TIME] sbxuser_1066

    View Slide

  12. λ is a UNIX system?!
    I know this!

    View Slide

  13. Let's run SSH in λ
    λ
    ssh process

    SSH Tunnel
    Firewall: no inbound ports

    View Slide

  14. SSH in Lambda Architecture
    λ
    node.js wrapper
    go sshd
    binary (x64) Go SSH Crypto Libs
    process.exec()
    https://github.com/smithclay/faassh

    View Slide

  15. Max Session Length: 5 minutes
    (custom prompt optional)
    https://github.com/smithclay/faassh

    View Slide

  16. Info from /proc
    2x Intel(R) Xeon(R) CPU E5-2666 v3 @ 2.90GHz
    cat /proc/cpuinfo
    3857664 kB
    cat /proc/meminfo
    ixgbevf (EC2 10Gbps Network Driver)
    cat /proc/modules
    c4.large EC2 Compute-Optimized Instance (?)

    View Slide

  17. c4.large instance
    λ
    λ
    λ
    λ
    λ
    λ
    λ
    λ
    λ
    λ
    λ
    λ
    λ
    λ
    λ
    λ
    λ
    λ
    λ
    λ
    λ
    λ
    λ
    λ
    λ
    λ
    λ
    λ
    λ
    λ
    λs in theory on a VM
    10 Gbps
    λ = Running 128 MB Function
    λ
    λ
    λ
    λ
    λ
    λ
    λ
    λ
    λ
    λ
    λ
    λ
    λ
    λ
    λ
    λ
    λ
    λ
    λ
    λ
    λ
    λ
    λ
    λ
    λ
    λ
    λ
    λ
    λ
    λ
    λ
    λ
    λ
    λ
    λ
    λ = Not Running 128 MB Function

    View Slide

  18. Frozen functions help avoid cold starts
    https://www.kernel.org/doc/Documentation/cgroup-v1/freezer-
    subsystem.txt
    cgroup freezer
    subsystem
    λ λ λ

    View Slide

  19. Internals Recap
    It's just containers on a VM
    Functions frozen when not running
    No magic unikernels involved

    View Slide

  20. How do we measure and
    prevent cold starts?
    "Use Kubernetes" — Troll

    View Slide

  21. Cold Start Discovery
    var SO_SO_COLD = true;
    exports.handler = function(trigger, cb) {
    console.log('Cold? %s', SO_SO_COLD);
    SO_SO_COLD = false;
    return callback(cb)
    }
    https://github.com/smithclay/lambda-proc-info

    View Slide

  22. Warming automation
    Scheduled
    Event
    λ Only effective for non-
    concurrent execution!
    4 minute
    interval
    http://docs.aws.amazon.com/lambda/latest/dg/with-scheduled-events.html

    View Slide

  23. Sending cold start events to analytics
    λ
    console.log(coldStart)
    Logs λ
    POST to Event DB
    (Insights)
    CloudWatch
    Log Filter Trigger

    View Slide

  24. Cold Starts Visualized
    ~7 hrs ~8 hrs

    View Slide

  25. λ Host Uptime
    Cold starts happen when hosts change!
    ~8hrs

    View Slide

  26. λ Host Subnet Hopping
    10.13
    10.12 10.11 10.13 10.12 10.12 10.12
    10.
    11
    # of AZs in us-west-2: 3

    View Slide

  27. What's the maximum concurrency of your function?
    one > 1
    A scheduled event
    will warm it
    until host retires.
    More advanced
    strategy needed*

    View Slide

  28. "Advanced" Strategy
    for i in `seq 1 $NUM_EXECUTIONS`;
    do
    echo "[$i] Executing $AWS_LAMBDA_FUNCTION_NAME..."
    aws lambda invoke ...
    done
    https://gist.github.com/smithclay/e89dfe35fe2a4938db56bb12df76777c

    View Slide

  29. Multiple containers running on a
    single host to serve parallel requests
    Tracking /proc/sys/kernel/random/boot_id and hostname
    Confirmed: cold start happens on container init.

    View Slide

  30. So is this just a container PaaS?
    High-availability/multiple zones
    Elastic fleet of compute-optimized VMs
    A very good scheduling algorithm
    Design (freezing, limits, etc) for very fast invocation
    Only if your PaaS has...

    View Slide

  31. FaaS in Production Reality
    λ λ
    Dev Prod
    Orchestration (!?)
    Version/Deploy
    Monitoring
    Security
    Cold Start Mgmt
    The "learning
    cliff"
    Great tweet from @mfdii

    View Slide

  32. FaaS Isn't a Silver Bullet
    "I've got a fast, computationally-intensive task that I need to
    perform occasionally in response to a well-defined event that
    isn't that sensitive to latency."
    —The Ideal FaaS Developer
    // TO DO:
    measure &&
    share results

    View Slide

  33. Thanks.
    @smithclay
    New Relic
    Gluecon 5/24/17

    View Slide