Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Lazy Sequences - Why are they so lazy?

Ramsharan
January 11, 2020

Lazy Sequences - Why are they so lazy?

Here is a very brief outline of my presentation:
- Firstly, I will start of by explaining my failures in understanding bits and pieces of Seq's. Explain with actual function calls.
- The second part of the presentation is on how Lisp Cons Cells work. A brief explanation of how it is a list with 2 functions ie car and cdr. There will be a lot of examples on how sublists can be created and how they can be modified.
- A brief explanation of how Lisp Cons Cells analogues to Clojure Lazy Seq's.
- Concluding with a run through of why Clojure Lazy Seq's are more efficient than normal seq's. This will be majorly talking about how Lazy Seq's does all the heavy lifting with functional computation rather than sitting in memory.

Ramsharan

January 11, 2020
Tweet

Other Decks in Technology

Transcript

  1. Lazy Sequences -
    Why are they so lazy?
    By

    Ramsharan Gorur Jayaraman

    Director Of Engineering
    Quintype
    - Introduction.

    - Pardon me if I am not good at
    public speaking because the last
    time I did this I was talking in a
    Ruby conference about moving to
    Golang and it did not go well.

    - I joined QT and talk about first task.

    View full-size slide

  2. First Experience
    (let [batches (->> items
    (partition-all batch-size))]
    (doall [batch-of-items batches]
    (result-set-fn-wrapper batch-of-items)))
    - I am a big time emacs user and I
    am used to pressing a lot of keys.

    - I did this the first time and pressed
    c-c c-k 100 times, saved the file a
    100 times and hot patched it in
    prod.

    - OOM Kill

    View full-size slide

  3. First Experience
    (let [batches (->> items
    (partition-all batch-size))]
    (doall [batch-of-items batches]
    (result-set-fn-wrapper batch-of-items)))
    (let [batches (->> items
    (partition-all batch-size))]
    (doseq [batch-of-items batches]
    (result-set-fn-wrapper batch-of-items)))
    - OOM Kill

    - I with my emacs fingers went to
    one of my colleagues (Foobar).

    - Then I replaced the doall with
    doseq and ran it on prod, emacsed
    for sometime and it worked like a
    charm.

    - I started googling differences
    between doseq and doall. Then I
    realised that all of them were
    operations on something called a
    lazy sequence.

    View full-size slide

  4. What is a Lazy Seq?
    - I went to Foobar and asked him
    what a lazy seq was.

    - He gave me the snarkiest look in
    his whole life and left the office
    thinking people like me should not
    exist in this world.

    - Then I missed my gym session and
    started googling.

    View full-size slide

  5. Enter Lisp Cons Cells
    Lisp cons cells takes me back to
    when I was studying computer
    science trying to become the useless
    programmer Foobar stated.

    Linked Lists.

    View full-size slide

  6. (setf x '(b))
    - Car is the value of the Cons cell

    - CDR is the pointer to the next cons
    cell or it is nil in case it is the last or
    the first.

    View full-size slide

  7. (setf x '(a b))
    (setf y (cons d x))
    (setf (car x) '(r t))
    A list of cons cells can consist of
    another list of cons cells.

    View full-size slide

  8. Lazy Seqs are not data structures.
    - Coming back to lazy seqs they are
    not data structures.

    - Just because they return a list
    when realised we should not be
    looking at them as data structures.

    - They are a sequence of algorithms
    in my opinion.

    View full-size slide

  9. They are very simple cells like Cons Cells with each
    element consisting of a value and a function.
    “first” calls a function which returns a value and another
    function and then returns the value from the called
    function.
    “rest” calls a function which returns a value and another
    function and then returns the value from the called
    function.

    View full-size slide

  10. Why is it efficient?

    View full-size slide

  11. A lazy seq is an algorithmically generated logical
    sequence that does not have to reside in memory
    user> (time (def partition-large-no-set (partition 3 (range
    1000000))))
    ;;”Elapsed time: 0.165415 msecs"
    user> (time (def partition-large-no-set (doall (partition 3 (range
    1000000))))) ;;"Elapsed time: 3289.272597 msecs"

    View full-size slide

  12. –Anonymous
    “Lazy Seqs make impossible possible.”

    View full-size slide

  13. Lazy Seqs at Quintype

    View full-size slide

  14. • We defined one of our products around the basic
    behaviour of lazy seqs.

    • Quintype imports content from other platforms for
    client acquisition.

    • We built our file structure with one json a line and in
    small files so that we can lazily read data and import
    them.

    View full-size slide

  15. Input
    {"external-id": "external-id-4","headline": "Attractive headline","subheadline": “Attractive
    subheadline","slug": "slug","body": "Some Body”}\n
    {"external-id": "external-id-4","headline": "Attractive headline","subheadline": “Attractive
    subheadline","slug": "slug","body": "Some Body 1”}\n

    View full-size slide

  16. Processing
    (defn- push-content-to-channel
    "Push content to channel"
    [channel]
    (doseq [file (get-list-of-files)]
    (log/info {:message (str "[READING-FILE] " file)})
    (s3/read-file file #(async/>!! channel %)))
    (async/close! channel))
    (def filter-and-decorate
    "Filter & decorate"
    (comp
    (filter verify-content?)
    (map decorate)))
    (defn import-content
    "Import content"
    []
    (let [parallel 5
    batch 100
    input-channel (async/chan 5)
    output-channel (async/chan parallel (partition-all (* parallel
    batch)))]
    (future
    (push-content-to-channel input-channel))
    (async/pipeline 1 output-channel filter-and-decorate input-channel)
    (let [stats (async/(log/info (merge {:message "[FINAL-STATS]"} stats))
    stats)))

    View full-size slide

  17. Mr Foobar’s recent commit

    View full-size slide

  18. References
    • http://theatticlight.net/posts/Lazy-Sequences-in-Clojure/

    • https://cs.gmu.edu/~sean/lisp/cons/

    View full-size slide

  19. Thank You.
    Questions?

    View full-size slide