Lazy Sequences - Why are they so lazy?

Here is a very brief outline of my presentation:
- Firstly, I will start of by explaining my failures in understanding bits and pieces of Seq's. Explain with actual function calls.
- The second part of the presentation is on how Lisp Cons Cells work. A brief explanation of how it is a list with 2 functions ie car and cdr. There will be a lot of examples on how sublists can be created and how they can be modified.
- A brief explanation of how Lisp Cons Cells analogues to Clojure Lazy Seq's.
- Concluding with a run through of why Clojure Lazy Seq's are more efficient than normal seq's. This will be majorly talking about how Lazy Seq's does all the heavy lifting with functional computation rather than sitting in memory.

Transcript

1. Lazy Sequences - Why are they so lazy? By Ramsharan Gorur Jayaraman Director Of Engineering Quintype - Introduction. - Pardon me if I am not good at public speaking because the last time I did this I was talking in a Ruby conference about moving to Golang and it did not go well. - I joined QT and talk about first task.
2. First Experience (let [batches (->> items (partition-all batch-size))] (doall [batch-of-items batches] (result-set-fn-wrapper batch-of-items))) - I am a big time emacs user and I am used to pressing a lot of keys. - I did this the first time and pressed c-c c-k 100 times, saved the file a 100 times and hot patched it in prod. - OOM Kill
4. First Experience (let [batches (->> items (partition-all batch-size))] (doall [batch-of-items batches] (result-set-fn-wrapper batch-of-items))) (let [batches (->> items (partition-all batch-size))] (doseq [batch-of-items batches] (result-set-fn-wrapper batch-of-items))) - OOM Kill - I with my emacs fingers went to one of my colleagues (Foobar). - Then I replaced the doall with doseq and ran it on prod, emacsed for sometime and it worked like a charm. - I started googling differences between doseq and doall. Then I realised that all of them were operations on something called a lazy sequence.
5. What is a Lazy Seq? - I went to Foobar and asked him what a lazy seq was. - He gave me the snarkiest look in his whole life and left the office thinking people like me should not exist in this world. - Then I missed my gym session and started googling.
6. Enter Lisp Cons Cells Lisp cons cells takes me back to when I was studying computer science trying to become the useless programmer Foobar stated. Linked Lists.
7. (setf x '(b)) - Car is the value of the Cons cell - CDR is the pointer to the next cons cell or it is nil in case it is the last or the first.
8. (setf x '(a b)) (setf y (cons d x)) (setf (car x) '(r t)) A list of cons cells can consist of another list of cons cells.
9. Lazy Seqs are not data structures. - Coming back to lazy seqs they are not data structures. - Just because they return a list when realised we should not be looking at them as data structures. - They are a sequence of algorithms in my opinion.
10. They are very simple cells like Cons Cells with each element consisting of a value and a function. “first” calls a function which returns a value and another function and then returns the value from the called function. “rest” calls a function which returns a value and another function and then returns the value from the called function.
11. Why is it efficient?
12. A lazy seq is an algorithmically generated logical sequence that does not have to reside in memory user> (time (def partition-large-no-set (partition 3 (range 1000000)))) ;;”Elapsed time: 0.165415 msecs" user> (time (def partition-large-no-set (doall (partition 3 (range 1000000))))) ;;"Elapsed time: 3289.272597 msecs"
13. –Anonymous “Lazy Seqs make impossible possible.”
14. Lazy Seqs at Quintype
15. • We defined one of our products around the basic behaviour of lazy seqs. • Quintype imports content from other platforms for client acquisition. • We built our file structure with one json a line and in small files so that we can lazily read data and import them.
16. Input {"external-id": "external-id-4","headline": "Attractive headline","subheadline": “Attractive subheadline","slug": "slug","body": "<p>Some Body</p>”}\n {"external-id": "external-id-4","headline": "Attractive headline","subheadline": “Attractive subheadline","slug": "slug","body": "<p>Some Body 1</p>”}\n
17. Processing (defn- push-content-to-channel "Push content to channel" [channel] (doseq [file (get-list-of-files)] (log/info {:message (str "[READING-FILE] " file)}) (s3/read-file file #(async/>!! channel %))) (async/close! channel)) (def filter-and-decorate "Filter & decorate" (comp (filter verify-content?) (map decorate))) (defn import-content "Import content" [] (let [parallel 5 batch 100 input-channel (async/chan 5) output-channel (async/chan parallel (partition-all (* parallel batch)))] (future (push-content-to-channel input-channel)) (async/pipeline 1 output-channel filter-and-decorate input-channel) (let [stats (async/<!! (log/collect-stats output-channel true))] (log/info (merge {:message "[FINAL-STATS]"} stats)) stats)))
18. Mr Foobar’s recent commit
19. References • http://theatticlight.net/posts/Lazy-Sequences-in-Clojure/ • https://cs.gmu.edu/~sean/lisp/cons/
20. Thank You. Questions?