Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Lazy Sequences - Why are they so lazy?

Ramsharan
January 11, 2020

Lazy Sequences - Why are they so lazy?

Here is a very brief outline of my presentation:
- Firstly, I will start of by explaining my failures in understanding bits and pieces of Seq's. Explain with actual function calls.
- The second part of the presentation is on how Lisp Cons Cells work. A brief explanation of how it is a list with 2 functions ie car and cdr. There will be a lot of examples on how sublists can be created and how they can be modified.
- A brief explanation of how Lisp Cons Cells analogues to Clojure Lazy Seq's.
- Concluding with a run through of why Clojure Lazy Seq's are more efficient than normal seq's. This will be majorly talking about how Lazy Seq's does all the heavy lifting with functional computation rather than sitting in memory.

Ramsharan

January 11, 2020
Tweet

Other Decks in Technology

Transcript

  1. Lazy Sequences - Why are they so lazy? By Ramsharan

    Gorur Jayaraman Director Of Engineering Quintype - Introduction. - Pardon me if I am not good at public speaking because the last time I did this I was talking in a Ruby conference about moving to Golang and it did not go well. - I joined QT and talk about first task.
  2. First Experience (let [batches (->> items (partition-all batch-size))] (doall [batch-of-items

    batches] (result-set-fn-wrapper batch-of-items))) - I am a big time emacs user and I am used to pressing a lot of keys. - I did this the first time and pressed c-c c-k 100 times, saved the file a 100 times and hot patched it in prod. - OOM Kill
  3. First Experience (let [batches (->> items (partition-all batch-size))] (doall [batch-of-items

    batches] (result-set-fn-wrapper batch-of-items))) (let [batches (->> items (partition-all batch-size))] (doseq [batch-of-items batches] (result-set-fn-wrapper batch-of-items))) - OOM Kill - I with my emacs fingers went to one of my colleagues (Foobar). - Then I replaced the doall with doseq and ran it on prod, emacsed for sometime and it worked like a charm. - I started googling differences between doseq and doall. Then I realised that all of them were operations on something called a lazy sequence.
  4. What is a Lazy Seq? - I went to Foobar

    and asked him what a lazy seq was. - He gave me the snarkiest look in his whole life and left the office thinking people like me should not exist in this world. - Then I missed my gym session and started googling.
  5. Enter Lisp Cons Cells Lisp cons cells takes me back

    to when I was studying computer science trying to become the useless programmer Foobar stated. Linked Lists.
  6. (setf x '(b)) - Car is the value of the

    Cons cell - CDR is the pointer to the next cons cell or it is nil in case it is the last or the first.
  7. (setf x '(a b)) (setf y (cons d x)) (setf

    (car x) '(r t)) A list of cons cells can consist of another list of cons cells.
  8. Lazy Seqs are not data structures. - Coming back to

    lazy seqs they are not data structures. - Just because they return a list when realised we should not be looking at them as data structures. - They are a sequence of algorithms in my opinion.
  9. They are very simple cells like Cons Cells with each

    element consisting of a value and a function. “first” calls a function which returns a value and another function and then returns the value from the called function. “rest” calls a function which returns a value and another function and then returns the value from the called function.
  10. A lazy seq is an algorithmically generated logical sequence that

    does not have to reside in memory user> (time (def partition-large-no-set (partition 3 (range 1000000)))) ;;”Elapsed time: 0.165415 msecs" user> (time (def partition-large-no-set (doall (partition 3 (range 1000000))))) ;;"Elapsed time: 3289.272597 msecs"
  11. • We defined one of our products around the basic

    behaviour of lazy seqs. • Quintype imports content from other platforms for client acquisition. • We built our file structure with one json a line and in small files so that we can lazily read data and import them.
  12. Input {"external-id": "external-id-4","headline": "Attractive headline","subheadline": “Attractive subheadline","slug": "slug","body": "<p>Some Body</p>”}\n

    {"external-id": "external-id-4","headline": "Attractive headline","subheadline": “Attractive subheadline","slug": "slug","body": "<p>Some Body 1</p>”}\n
  13. Processing (defn- push-content-to-channel "Push content to channel" [channel] (doseq [file

    (get-list-of-files)] (log/info {:message (str "[READING-FILE] " file)}) (s3/read-file file #(async/>!! channel %))) (async/close! channel)) (def filter-and-decorate "Filter & decorate" (comp (filter verify-content?) (map decorate))) (defn import-content "Import content" [] (let [parallel 5 batch 100 input-channel (async/chan 5) output-channel (async/chan parallel (partition-all (* parallel batch)))] (future (push-content-to-channel input-channel)) (async/pipeline 1 output-channel filter-and-decorate input-channel) (let [stats (async/<!! (log/collect-stats output-channel true))] (log/info (merge {:message "[FINAL-STATS]"} stats)) stats)))