Escape from the Callback Hell

Asynchronised operations are good. Callbacks are not. Named callbacks clutter scopes and are disruptive. Anonymous callbacks—if they are supported at all—create deeply nested code and become totally unreadable. There is another way. Let's take a look of how efforts are taken to flatten the code structure for async operations, and how it may be relevant to you.

Transcript

1. Escape from the Callback Hell
2. Me • Call me TP • Follow @uranusjr • uranusjr.com
4. Qt C++ OS X iOS Objective-C Linux Python
5. www. .com
6. CALLBACK When the going gets rough.
7. Callback • “Tell me when you’re done.” • Asynchronous and non-blocking • The task is async • The API is non-blocking
8. Time waiting Sync v. Async
9. Asynchrony • Should I use it? • Can I use it? • How do I use it?
10. Y U No Use Threads • Data-sharing = pain • Not worth it • Cross-platform support
11. Who Haz It • GUI frameworks • ECMAScript • High-level libraries
12. Async with Callbacks
13. import grequests! ! ! def get_data(url, pool):! request = grequests.get(url, hooks={! 'response': handle_data! })! grequests.send(request, pool)! ! ! def handle_data(response):! print response.content https://github.com/kennethreitz/grequests
14. import sys, json, grequests! ! def get_data(url, callback):! request = grequests.get(url, hooks={! 'response': callback! })! grequests.send(request, pool)! ! def handle_initial_data(response):! next_url = json.load(response)['url']! get_data(next_url, print_data)! ! def print_data(response):! print response.content! ! def main():! get_data(sys.argv[1], handle_initial_data)
15. import sys, json, grequests! ! def get_data(url, callback):! request = grequests.get(url, hooks={! 'response': callback! })! grequests.send(request, pool)! ! def handle_initial_data(response):! next_url = json.load(response)['url']! get_data(next_url, print_data)! ! def print_data(response):! print response.content! ! def main():! get_data(sys.argv[1], handle_initial_data) Start here 1 2 3
16. If that seems wrong It’s because it is
17. Anonymous function
18. (function (url) {! $.getJSON(url, function (data) {! var nextUrl = data.url;! $.get(nextUrl, function(data) {! console.log(data);! });! });! })();
19. (function (url) {! $.getJSON(url, function (data) {! var nextUrl = data.url;! $.get(nextUrl, function(data) {! console.log(data);! });! });! })(); 1 2 3 Start here
20. (function (url) {! $.getJSON(url, function (data) {! var nextUrl = data.url;! $.get(nextUrl, function(data) {! console.log(nextUrl);! console.log(data);! });! });! })(); Closure
21. __block AFHTTPRequestOperationManager *manager =! [AFHTTPRequestOperationManager manager];! ! [manager GET:url! parameters:nil! success:^(id operation, id obj) {! NSString *next = [obj objectForKey:'url'];! [manager GET:next! parameters:nil! success:^(id operation, id obj) {! NSLog(@"%@", obj);! }! failure:nil];! }! failure:nil]; “Block” https://github.com/AFNetworking/AFNetworking
22. http://en.wikipedia.org/wiki/Blocks_(C_language_extension)
23. With the help of Clang
24. Not all roses and rainbows
25. Pyramid of Doom $(function () {! $("button").click(function (e) {! $.get(URL, function (data) {! $(".list").each(function () {! $(this).click(function () {! setTimeout(function () {! alert("Top of the world");! }, 1000);! });! });! });! });! });
26. • “Normal” variables are copied • __block variables are referenced • Ivar ref indicates strong to self • Direct access indicates strong to variable (not self) • Yada yada yada… Memory Management
27. I KNOW THAT FEEL
28. Garbage Collection FTW!!!1
29. for (var i = 0; i < urls.length; i++) {! var url = urls[i];! $.getJSON(url, function (data) {! var moreUrls = data.urls;! for (var j = 0; j < moreUrls.length; j++) {! var url = moreUrls[j];! $.get(url, function (data) {! console.log(url + '\n' + data);! });! }! });! } May Not Behave Well
30. for (var i = 0; i < urls.length; i++) {! var url = urls[i];! $.getJSON(url, function (data) {! var moreUrls = data.urls;! for (var j = 0; j < moreUrls.length; j++) {! $.get(moreUrls[j], function (data) {! console.log(url);! console.log(data);! });! }! });! } Long Live the Variable
31. $.get(url, function (data) {! console.log('Do I go first?');! });! ! processSomething();! console.log('Or do I?'); Are You Done?
32. Promise
33. Promise • Common in JavaScript • Various implementations • Native support coming • “Future” • Twisted “Deferred”
34. jQuery Meh • Parallelising • Exception handling • Standard compatibility
35. Q($.getJSON(url))! .then(function (data) {! var nextUrl = data.url;! return Q($.get(nextUrl));! }).then(function (data) {! console.log(data);! }); https://github.com/kriskowal/q
36. Q($.getJSON(url))! .then(function (data) {! var nextUrl = data.url;! return Q($.get(nextUrl));! }).then(function (data) {! console.log(data);! });
37. Q($.getJSON(url))! .then(function (data) {! var nextUrl = data.url;! return Q($.get(nextUrl));! }).then(function (data) {! console.log(data);! });
38. Q($.getJSON(url))! .then(function (data) {! var nextUrl = data.url;! return Q($.get(nextUrl));! }).then(function (data) {! console.log(data);! }); Start here 1 2 3 Intuitive
39. Q($.getJSON(url))! .then(function (data) {! var nextUrl = data.url;! return Q($.get(nextUrl));! }).then(function (data) {! console.log(data);! }); Obvious
40. Q.spread([! Q($.getJSON(url1)),! Q($.getJSON(url2))! ]).then(function (d1, d2) {! var url = d1.host + d2.path;! return Q($.get(url));! }).then(function (data) {! console.log(data);! }); Combination
41. Async with Promises • Obvious flow control • Error handling that makes sense • Easy scoping
42. But…
43. import json! from twisted.internet import reactor! from twisted.web import client! ! def process(content):! client.getPage(! json.loads(content)['url']! ).addCallback(! lambda c: print c! )! ! def getIt(url):! client.getPage(url).addCallback(process)! ! getIt(someUrl)! reactor.run() https://twistedmatrix.com/
44. import json! from twisted.internet import reactor! from twisted.web import client! ! def process(content):! client.getPage(! json.loads(content)['url']! ).addCallback(! lambda c: print c! )! ! def getIt(url):! client.getPage(url).addCallback(process)! ! getIt(someUrl)! reactor.run() (Marginally) acceptable NO.
45. :(
47. async Task GetIt(string url)! {! HttpClient client = new HttpClient();! string content;! ! content = await client.GetStringAsync(url);! ! string next = JsonConvert.DeserializeObject! <Dictionary<string, string>>(content)! ["url"];! ! content = await client.GetStringAsync(next);! Console.WriteLine(content);! } C#
48. async Task GetIt(string url)! {! HttpClient client = new HttpClient();! string content;! ! content = await client.GetStringAsync(url);! ! string next = JsonConvert.DeserializeObject! <Dictionary<string, string>>(content)! ["url"];! ! content = await client.GetStringAsync(next);! Console.WriteLine(content);! } Start here 1 2 3
49. Absolute magic
50. async Task GetIt(string url)! {! HttpClient client = new HttpClient();! string content;! ! content = await client.GetStringAsync(url);! ! string next = JsonConvert.DeserializeObject! <Dictionary<string, string>>(content)! ["url"];! ! content = await client.GetStringAsync(next);! Console.WriteLine(content);! }
51. http://msdn.microsoft.com/en-us/library/hh191443.aspx
52. Microsoft magic
53. Really?
54. import json! from twisted.internet import reactor, defer! from twisted.web import client! ! @defer.inlineCallbacks! def getIt(url):! content = yield client.getPage(url)! url = json.loads(content)['url']! content = yield client.getPage(url)! print content! ! getIt(someUrl)! reactor.run()
55. import json! from twisted.internet import reactor, defer! from twisted.web import client! ! @defer.inlineCallbacks! def getIt(url):! content = yield client.getPage(url)! url = json.loads(content)['url']! content = yield client.getPage(url)! print content! ! getIt(someUrl)! reactor.run() Start here 1 2 3
56. import json! from twisted.internet import reactor, defer! from twisted.web import client! ! @defer.inlineCallbacks! def getIt(url):! content = yield client.getPage(url)! url = json.loads(content)['url']! content = yield client.getPage(url)! print content! ! getIt(someUrl)! reactor.run()
57. Coroutine
58. [Coroutines] allow multiple entry points for suspending and resuming execution at certain locations. http://en.wikipedia.org/wiki/Coroutine
59. async Task GetIt(string url)! {! HttpClient client = new HttpClient();! string content;! ! content = await client.GetStringAsync(url);! ! string next = JsonConvert.DeserializeObject! <Dictionary<string, string>>(content)! ["url"];! ! content = await client.GetStringAsync(next);! Console.WriteLine(content);! }
60. async Task GetIt(string url)! {! HttpClient client = new HttpClient();! string content;! ! content = await client.GetStringAsync(url);! ! string next = JsonConvert.DeserializeObject! <Dictionary<string, string>>(content)! ["url"];! ! content = await client.GetStringAsync(next);! Console.WriteLine(content);! } Yield control to event loop; wait for data.
61. async Task GetIt(string url)! {! HttpClient client = new HttpClient();! string content;! ! content = await client.GetStringAsync(url);! ! string next = JsonConvert.DeserializeObject! <Dictionary<string, string>>(content)! ["url"];! ! content = await client.GetStringAsync(next);! Console.WriteLine(content);! } Control has been yielded back. Resume with retrieved data.
62. async Task GetIt(string url)! {! HttpClient client = new HttpClient();! string content;! ! content = await client.GetStringAsync(url);! ! string next = JsonConvert.DeserializeObject! <Dictionary<string, string>>(content)! ["url"];! ! content = await client.GetStringAsync(next);! Console.WriteLine(content);! }
63. async Task GetIt(string url)! {! HttpClient client = new HttpClient();! string content;! ! content = await client.GetStringAsync(url);! ! string next = JsonConvert.DeserializeObject! <Dictionary<string, string>>(content)! ["url"];! ! content = await client.GetStringAsync(next);! Console.WriteLine(content);! } Yield control.
64. async Task GetIt(string url)! {! HttpClient client = new HttpClient();! string content;! ! content = await client.GetStringAsync(url);! ! string next = JsonConvert.DeserializeObject! <Dictionary<string, string>>(content)! ["url"];! ! content = await client.GetStringAsync(next);! Console.WriteLine(content);! } Resume with data.
65. async Task GetIt(string url)! {! HttpClient client = new HttpClient();! string content;! ! content = await client.GetStringAsync(url);! ! string next = JsonConvert.DeserializeObject! <Dictionary<string, string>>(content)! ["url"];! ! content = await client.GetStringAsync(next);! Console.WriteLine(content);! }
66. async Task GetIt(string url)! {! HttpClient client = new HttpClient();! string content;! ! content = await client.GetStringAsync(url);! ! string next = JsonConvert.DeserializeObject! <Dictionary<string, string>>(content)! ["url"];! ! content = await client.GetStringAsync(next);! Console.WriteLine(content);! }
67. Coroutine • “Pauses” when calling another coroutine • “Resumes” when another coroutine calls it again • Subroutines are special cases of [coroutines] — Knuth
68. import json! from aiohttp import request! ! def get_it(url):! response = yield from request('GET', url)! data = yield from response.read()! next_url = json.loads(data)['url']! response = yield from request('GET', url)! data = yield from response.read()! print(data) https://twistedmatrix.com/trac/browser/trunk/twisted/internet/defer.py
69. https://tw.pycon.org/
70. Generators • “Semi-coroutines” • Always yield to its caller • Primarily used to simply iterators
71. def get_generator():! yield 0! yield 1! ! ! g = get_generator()! print(type(g)) # <type 'generator'>! ! for i in g:! print(i)! # 0! # 1
72. g = get_generator()! while True:! try:! i = next(g) # Python 3! print(i)! except StopIteration:! break! # 0! # 1
73. g = get_generator()! while True:! try:! i = g.send(None)! print(i)! except StopIteration:! break! # 0! # 1 http://www.python.org/dev/peps/pep-0342/
74. def get_generator():! v = yield 0! print('Gen:', v)! v = yield v! print('Gen:', v)! ! g = get_generator()! v = g.send(None) # Starts g.! ! while True:! try:! print('Con:', v)! v = g.send(v + 1)! except StopIteration:! break Con: 0 ! Gen: 1! Con: 1! Gen: 2 Outputs
75. def get_generator():! v = yield 0! print('Gen:', v)! v = yield v! print('Gen:', v)! ! g = get_generator()! v = g.send(None)! ! while True:! try:! print('Con:', v)! v = g.send(v + 1)! except StopIteration:! break Con: 0 ! Gen: 1! Con: 1! Gen: 2 Outputs
76. def get_generator():! v = yield 0! print('Gen:', v)! v = yield v! print('Gen:', v)! ! g = get_generator()! v = g.send(None)! ! while True:! try:! print('Con:', v)! v = g.send(v + 1)! except StopIteration:! break Con: 0 ! Gen: 1! Con: 1! Gen: 2 Outputs
77. def get_generator():! v = yield 0! print('Gen:', v)! v = yield v! print('Gen:', v)! ! g = get_generator()! v = g.send(None)! ! while True:! try:! print('Con:', v)! v = g.send(v + 1)! except StopIteration:! break Con: 0 ! Gen: 1! Con: 1! Gen: 2 Outputs
78. def get_generator():! v = yield 0! print('Gen:', v)! v = yield v! print('Gen:', v)! ! g = get_generator()! v = g.send(None)! ! while True:! try:! print('Con:', v)! v = g.send(v + 1)! except StopIteration:! break Con: 0 ! Gen: 1! Con: 1! Gen: 2 Outputs
79. def get_generator():! v = yield 0! print('Gen:', v)! v = yield v! print('Gen:', v)! ! g = get_generator()! v = g.send(None)! ! while True:! try:! print('Con:', v)! v = g.send(v + 1)! except StopIteration:! break Con: 0 ! Gen: 1! Con: 1! Gen: 2 Outputs
80. def get_generator():! v = yield 0! print('Gen:', v)! v = yield v! print('Gen:', v)! ! g = get_generator()! v = g.send(None)! ! while True:! try:! print('Con:', v)! v = g.send(v + 1)! except StopIteration:! break Con: 0 ! Gen: 1! Con: 1! Gen: 2 Outputs
81. def get_generator():! v = yield 0! print('Gen:', v)! v = yield v! print('Gen:', v)! ! g = get_generator()! v = g.send(None)! ! while True:! try:! print('Con:', v)! v = g.send(v + 1)! except StopIteration:! break Con: 0 ! Gen: 1! Con: 1! Gen: 2 Outputs
82. def get_generator():! v = yield 0! print('Gen:', v)! v = yield v! print('Gen:', v)! ! g = get_generator()! v = g.send(None)! ! while True:! try:! print('Con:', v)! v = g.send(v + 1)! except StopIteration:! break Con: 0 ! Gen: 1! Con: 1! Gen: 2 Outputs
83. def get_generator():! v = yield 0! print('Gen:', v)! v = yield v! print('Gen:', v)! ! g = get_generator()! v = g.send(None)! ! while True:! try:! print('Con:', v)! v = g.send(v + 1)! except StopIteration:! break Con: 0 ! Gen: 1! Con: 1! Gen: 2 Outputs
84. def get_generator():! v = yield 0! print('Gen:', v)! v = yield v! print('Gen:', v)! ! g = get_generator()! v = g.send(None)! ! while True:! try:! print('Con:', v)! v = g.send(v + 1)! except StopIteration:! break Con: 0 ! Gen: 1! Con: 1! Gen: 2 Outputs No more yielding.
85. def get_generator():! v = yield 0! print('Gen:', v)! v = yield v! print('Gen:', v)! ! g = get_generator()! v = g.send(None)! ! while True:! try:! print('Con:', v)! v = g.send(v + 1)! except StopIteration:! break Con: 0 ! Gen: 1! Con: 1! Gen: 2 Outputs No more yielding.
86. def get_generator():! v = yield 0! print('Gen:', v)! v = yield v! print('Gen:', v)! ! g = get_generator()! v = g.send(None)! ! while True:! try:! print('Con:', v)! v = g.send(v + 1)! except StopIteration:! break Con: 0 ! Gen: 1! Con: 1! Gen: 2 Outputs
87. Async with Generators • Yield to event loop • Event loop calls everything • What should they yield?
88. def inlineCallbacks(f):! ! @wraps(f)! def unwindGenerator(*args, **kwargs):! try:! gen = f(*args, **kwargs)! except _DefGen_Return:! raise TypeError("…")! if not isinstance(gen, GeneratorType):! raise TypeError("…")! d = Deferred()! return _inlineCallbacks(None, gen, d)! ! return unwindGenerator https://twistedmatrix.com/trac/browser/trunk/twisted/internet/defer.py
89. def _inlineCallbacks(result, g, deferred):! while True:! try:! result = g.send(result)! except StopIteration:! deferred.callback(None)! return deferred! except _DefGen_Return as e:! deferred.callback(e.value)! return deferred! except:! deferred.errback()! return deferred! if isinstance(result, Deferred):! # <Get result from yielded Deferred>! return deferred https://twistedmatrix.com/trac/browser/trunk/twisted/internet/defer.py
90. def _inlineCallbacks(result, g, deferred):! while True:! try:! result = g.send(result)! except StopIteration:! deferred.callback(None)! return deferred! except _DefGen_Return as e:! deferred.callback(e.value)! return deferred! except:! deferred.errback()! return deferred! if isinstance(result, Deferred):! # <Get result from yielded Deferred>! return deferred https://twistedmatrix.com/trac/browser/trunk/twisted/internet/defer.py
91. def _inlineCallbacks(result, g, deferred):! while True:! try:! result = g.send(result)! except StopIteration:! deferred.callback(None)! return deferred! except _DefGen_Return as e:! deferred.callback(e.value)! return deferred! except:! deferred.errback()! return deferred! if isinstance(result, Deferred):! # <Get result from yielded Deferred>! return deferred https://twistedmatrix.com/trac/browser/trunk/twisted/internet/defer.py
92. def _inlineCallbacks(result, g, deferred):! while True:! try:! result = g.send(result)! except StopIteration:! deferred.callback(None)! return deferred! except _DefGen_Return as e:! deferred.callback(e.value)! return deferred! except:! deferred.errback()! return deferred! if isinstance(result, Deferred):! # <Get result from yielded Deferred>! return deferred https://twistedmatrix.com/trac/browser/trunk/twisted/internet/defer.py
93. Return Values >>> def get_generator():! ... yield 0! ... yield 1! ... return 'I am done.'! ...! File "<stdin>", line 3! SyntaxError: 'return' with argument inside generator
94. class _DefGen_Return(BaseException):! def __init__(self, value):! self.value = value! ! ! def returnValue(val):! raise _DefGen_Return(val) https://twistedmatrix.com/trac/browser/trunk/twisted/internet/defer.py
95. def _inlineCallbacks(result, g, deferred):! while True:! try:! result = g.send(result)! except StopIteration:! deferred.callback(None)! return deferred! except _DefGen_Return as e:! deferred.callback(e.value)! return deferred! except:! deferred.errback()! return deferred! if isinstance(result, Deferred):! # <Get result from yielded Deferred>! return deferred https://twistedmatrix.com/trac/browser/trunk/twisted/internet/defer.py
96. import json! from twisted.internet import reactor, defer! from twisted.web import client! ! ! @defer.inlineCallbacks! def getIt(url):! content = yield client.getPage(url)! url = json.loads(content)['url']! content = yield client.getPage(url)! print content! # We want to return the content! ! ! getIt(someUrl)! reactor.run()
97. import json! from twisted.internet import reactor, defer! from twisted.web import client! ! ! @defer.inlineCallbacks! def getIt(url):! content = yield client.getPage(url)! url = json.loads(content)['url']! content = yield client.getPage(url)! print content! defer.returnValue(content)! ! ! getIt(someUrl)! reactor.run()
98. import json! from twisted.internet import reactor, defer! from twisted.web import client! ! ! @defer.inlineCallbacks! def getIt(url):! content = yield client.getPage(url)! url = json.loads(content)['url']! content = yield client.getPage(url)! print content! raise defer._DefGen_Return(content)! ! ! getIt(someUrl)! reactor.run()
99. WAT Your Captain is not impressed.
100. >>> def get_generator():! ... yield 0! ... yield 1! ... return 'I am done.'! ...! >>> g = get_generator()! >>> while True: print(next(g))! 0! 1! Traceback (most recent call last):! File "<stdin>", line 1, in <module>! StopIteration: I am done Python 3.3
101. class Task(futures.Future):! # …! def _step(self, value=None, exc=None):! # …! try:! if exc is not None:! result = self._coro.throw(exc)! elif value is not None:! result = self._coro.send(value)! else:! result = next(self._coro)! except StopIteration as exc:! self.set_result(exc.value)! # … http://hg.python.org/cpython/file/default/Lib/asyncio/tasks.py
102. ECMA Standard 6 var g = function*() {! yield 0;! yield 1;! }();! ! console.log(g.next().value); // 0! console.log(g.next().value); // 1
103. var g = function* () {! var v = yield 0;! console.log('Gen: ' + v);! var v = yield v;! console.log('Gen: ' + v);! }();! ! var obj = g.next();! while (!obj.done) {! var v = obj.value;! console.log('Con: ' + v);! obj = g.next(v + 1);! } Con: 0 ! Gen: 1! Con: 1! Gen: 2 Outputs
104. Next Generation • Promises — checked. • Generators — checked. • Implementation…
105. co(function* () {! var json =! yield Q($.getJSON(url));! var data =! yield Q($.get(json.url));! console.log(data);! })(); co https://github.com/visionmedia/co
106. co(function* () {! var results = yield Q.all([! Q($.getJSON(url1),! Q($.getJSON(url2)! ]);! var url = results[0].host + ! results[1].path;! var data = yield Q($.get(url));! console.log(data);! })(); Seamless Integration
107. Wrapping Up • Callbacks • Promises • Coroutines / Generators
108. Extra Notes • Doing things in parallel • Exception handling • There are more • Callbacks are not dead
109. Bonus Chatter $('#button').click(function () {! var that = this;! var url = $(this).data('url');! $.getJSON(url, function (data) {! $(that).text(data.content); ! });! });
110. Bonus Chatter $('#button').click(function () {! var that = this;! var url = $(this).data('url');! $.getJSON(url, function (data) {! $(that).text(data.content); ! });! });
111. Bonus Chatter $('#button').click(co(function* () {! var url = $(this).data('url');! var data = yield Q($.getJSON(url));! $(this).text(data.text)! }));
112. Link Dump • Python Coroutines, Present and Future • The difference between yield and yield-from • Deconstructing Deferred • PEPs 255, 342, 380, and 3148 • Python 3.4 asyncio
113. Link Dump • JavaScript Promises: There and Back Again • You’re Missing the Point of Promises • Bringing async/await to life in JavaScript • ECMA-262 6th Edition - DRAFT • task.js (by Mozilla)
114. Link Dump • Callbacks as our Generations’ Go To Statement • Coroutines in C • Futures (Objective-C) • Generators in Objective-C • SHXPromise (Objective-C)
115. FIRE QUESTIONS