Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Escape from the Callback Hell

Escape from the Callback Hell

Asynchronised operations are good. Callbacks are not. Named callbacks clutter scopes and are disruptive. Anonymous callbacks—if they are supported at all—create deeply nested code and become totally unreadable. There is another way. Let's take a look of how efforts are taken to flatten the code structure for async operations, and how it may be relevant to you.

9dafad54b5b4f360b7aae5f482bc1c91?s=128

Tzu-ping Chung

April 12, 2014
Tweet

More Decks by Tzu-ping Chung

Other Decks in Programming

Transcript

  1. Escape from the Callback Hell

  2. Me • Call me TP • Follow @uranusjr • uranusjr.com

  3. None
  4. Qt C++ OS X iOS Objective-C Linux Python

  5. www. .com

  6. CALLBACK When the going gets rough.

  7. Callback • “Tell me when you’re done.” • Asynchronous and

    non-blocking • The task is async • The API is non-blocking
  8. Time waiting Sync v. Async

  9. Asynchrony • Should I use it? • Can I use

    it? • How do I use it?
  10. Y U No Use Threads • Data-sharing = pain •

    Not worth it • Cross-platform support
  11. Who Haz It • GUI frameworks • ECMAScript • High-level

    libraries
  12. Async with Callbacks

  13. import grequests! ! ! def get_data(url, pool):! request = grequests.get(url,

    hooks={! 'response': handle_data! })! grequests.send(request, pool)! ! ! def handle_data(response):! print response.content https://github.com/kennethreitz/grequests
  14. import sys, json, grequests! ! def get_data(url, callback):! request =

    grequests.get(url, hooks={! 'response': callback! })! grequests.send(request, pool)! ! def handle_initial_data(response):! next_url = json.load(response)['url']! get_data(next_url, print_data)! ! def print_data(response):! print response.content! ! def main():! get_data(sys.argv[1], handle_initial_data)
  15. import sys, json, grequests! ! def get_data(url, callback):! request =

    grequests.get(url, hooks={! 'response': callback! })! grequests.send(request, pool)! ! def handle_initial_data(response):! next_url = json.load(response)['url']! get_data(next_url, print_data)! ! def print_data(response):! print response.content! ! def main():! get_data(sys.argv[1], handle_initial_data) Start here 1 2 3
  16. If that seems wrong It’s because it is

  17. Anonymous function

  18. (function (url) {! $.getJSON(url, function (data) {! var nextUrl =

    data.url;! $.get(nextUrl, function(data) {! console.log(data);! });! });! })();
  19. (function (url) {! $.getJSON(url, function (data) {! var nextUrl =

    data.url;! $.get(nextUrl, function(data) {! console.log(data);! });! });! })(); 1 2 3 Start here
  20. (function (url) {! $.getJSON(url, function (data) {! var nextUrl =

    data.url;! $.get(nextUrl, function(data) {! console.log(nextUrl);! console.log(data);! });! });! })(); Closure
  21. __block AFHTTPRequestOperationManager *manager =! [AFHTTPRequestOperationManager manager];! ! [manager GET:url! parameters:nil!

    success:^(id operation, id obj) {! NSString *next = [obj objectForKey:'url'];! [manager GET:next! parameters:nil! success:^(id operation, id obj) {! NSLog(@"%@", obj);! }! failure:nil];! }! failure:nil]; “Block” https://github.com/AFNetworking/AFNetworking
  22. http://en.wikipedia.org/wiki/Blocks_(C_language_extension)

  23. With the help of Clang

  24. Not all roses and rainbows

  25. Pyramid of Doom $(function () {! $("button").click(function (e) {! $.get(URL,

    function (data) {! $(".list").each(function () {! $(this).click(function () {! setTimeout(function () {! alert("Top of the world");! }, 1000);! });! });! });! });! });
  26. • “Normal” variables are copied • __block variables are referenced

    • Ivar ref indicates strong to self • Direct access indicates strong to variable (not self) • Yada yada yada… Memory Management
  27. I KNOW THAT FEEL

  28. Garbage Collection FTW!!!1

  29. for (var i = 0; i < urls.length; i++) {!

    var url = urls[i];! $.getJSON(url, function (data) {! var moreUrls = data.urls;! for (var j = 0; j < moreUrls.length; j++) {! var url = moreUrls[j];! $.get(url, function (data) {! console.log(url + '\n' + data);! });! }! });! } May Not Behave Well
  30. for (var i = 0; i < urls.length; i++) {!

    var url = urls[i];! $.getJSON(url, function (data) {! var moreUrls = data.urls;! for (var j = 0; j < moreUrls.length; j++) {! $.get(moreUrls[j], function (data) {! console.log(url);! console.log(data);! });! }! });! } Long Live the Variable
  31. $.get(url, function (data) {! console.log('Do I go first?');! });! !

    processSomething();! console.log('Or do I?'); Are You Done?
  32. Promise

  33. Promise • Common in JavaScript • Various implementations • Native

    support coming • “Future” • Twisted “Deferred”
  34. jQuery Meh • Parallelising • Exception handling • Standard compatibility

  35. Q($.getJSON(url))! .then(function (data) {! var nextUrl = data.url;! return Q($.get(nextUrl));!

    }).then(function (data) {! console.log(data);! }); https://github.com/kriskowal/q
  36. Q($.getJSON(url))! .then(function (data) {! var nextUrl = data.url;! return Q($.get(nextUrl));!

    }).then(function (data) {! console.log(data);! });
  37. Q($.getJSON(url))! .then(function (data) {! var nextUrl = data.url;! return Q($.get(nextUrl));!

    }).then(function (data) {! console.log(data);! });
  38. Q($.getJSON(url))! .then(function (data) {! var nextUrl = data.url;! return Q($.get(nextUrl));!

    }).then(function (data) {! console.log(data);! }); Start here 1 2 3 Intuitive
  39. Q($.getJSON(url))! .then(function (data) {! var nextUrl = data.url;! return Q($.get(nextUrl));!

    }).then(function (data) {! console.log(data);! }); Obvious
  40. Q.spread([! Q($.getJSON(url1)),! Q($.getJSON(url2))! ]).then(function (d1, d2) {! var url =

    d1.host + d2.path;! return Q($.get(url));! }).then(function (data) {! console.log(data);! }); Combination
  41. Async with Promises • Obvious flow control • Error handling

    that makes sense • Easy scoping
  42. But…

  43. import json! from twisted.internet import reactor! from twisted.web import client!

    ! def process(content):! client.getPage(! json.loads(content)['url']! ).addCallback(! lambda c: print c! )! ! def getIt(url):! client.getPage(url).addCallback(process)! ! getIt(someUrl)! reactor.run() https://twistedmatrix.com/
  44. import json! from twisted.internet import reactor! from twisted.web import client!

    ! def process(content):! client.getPage(! json.loads(content)['url']! ).addCallback(! lambda c: print c! )! ! def getIt(url):! client.getPage(url).addCallback(process)! ! getIt(someUrl)! reactor.run() (Marginally) acceptable NO.
  45. :(

  46. None
  47. async Task GetIt(string url)! {! HttpClient client = new HttpClient();!

    string content;! ! content = await client.GetStringAsync(url);! ! string next = JsonConvert.DeserializeObject! <Dictionary<string, string>>(content)! ["url"];! ! content = await client.GetStringAsync(next);! Console.WriteLine(content);! } C#
  48. async Task GetIt(string url)! {! HttpClient client = new HttpClient();!

    string content;! ! content = await client.GetStringAsync(url);! ! string next = JsonConvert.DeserializeObject! <Dictionary<string, string>>(content)! ["url"];! ! content = await client.GetStringAsync(next);! Console.WriteLine(content);! } Start here 1 2 3
  49. Absolute magic

  50. async Task GetIt(string url)! {! HttpClient client = new HttpClient();!

    string content;! ! content = await client.GetStringAsync(url);! ! string next = JsonConvert.DeserializeObject! <Dictionary<string, string>>(content)! ["url"];! ! content = await client.GetStringAsync(next);! Console.WriteLine(content);! }
  51. http://msdn.microsoft.com/en-us/library/hh191443.aspx

  52. Microsoft magic

  53. Really?

  54. import json! from twisted.internet import reactor, defer! from twisted.web import

    client! ! @defer.inlineCallbacks! def getIt(url):! content = yield client.getPage(url)! url = json.loads(content)['url']! content = yield client.getPage(url)! print content! ! getIt(someUrl)! reactor.run()
  55. import json! from twisted.internet import reactor, defer! from twisted.web import

    client! ! @defer.inlineCallbacks! def getIt(url):! content = yield client.getPage(url)! url = json.loads(content)['url']! content = yield client.getPage(url)! print content! ! getIt(someUrl)! reactor.run() Start here 1 2 3
  56. import json! from twisted.internet import reactor, defer! from twisted.web import

    client! ! @defer.inlineCallbacks! def getIt(url):! content = yield client.getPage(url)! url = json.loads(content)['url']! content = yield client.getPage(url)! print content! ! getIt(someUrl)! reactor.run()
  57. Coroutine

  58. [Coroutines] allow multiple entry points for suspending and resuming execution

    at certain locations. http://en.wikipedia.org/wiki/Coroutine
  59. async Task GetIt(string url)! {! HttpClient client = new HttpClient();!

    string content;! ! content = await client.GetStringAsync(url);! ! string next = JsonConvert.DeserializeObject! <Dictionary<string, string>>(content)! ["url"];! ! content = await client.GetStringAsync(next);! Console.WriteLine(content);! }
  60. async Task GetIt(string url)! {! HttpClient client = new HttpClient();!

    string content;! ! content = await client.GetStringAsync(url);! ! string next = JsonConvert.DeserializeObject! <Dictionary<string, string>>(content)! ["url"];! ! content = await client.GetStringAsync(next);! Console.WriteLine(content);! } Yield control to event loop; wait for data.
  61. async Task GetIt(string url)! {! HttpClient client = new HttpClient();!

    string content;! ! content = await client.GetStringAsync(url);! ! string next = JsonConvert.DeserializeObject! <Dictionary<string, string>>(content)! ["url"];! ! content = await client.GetStringAsync(next);! Console.WriteLine(content);! } Control has been yielded back. Resume with retrieved data.
  62. async Task GetIt(string url)! {! HttpClient client = new HttpClient();!

    string content;! ! content = await client.GetStringAsync(url);! ! string next = JsonConvert.DeserializeObject! <Dictionary<string, string>>(content)! ["url"];! ! content = await client.GetStringAsync(next);! Console.WriteLine(content);! }
  63. async Task GetIt(string url)! {! HttpClient client = new HttpClient();!

    string content;! ! content = await client.GetStringAsync(url);! ! string next = JsonConvert.DeserializeObject! <Dictionary<string, string>>(content)! ["url"];! ! content = await client.GetStringAsync(next);! Console.WriteLine(content);! } Yield control.
  64. async Task GetIt(string url)! {! HttpClient client = new HttpClient();!

    string content;! ! content = await client.GetStringAsync(url);! ! string next = JsonConvert.DeserializeObject! <Dictionary<string, string>>(content)! ["url"];! ! content = await client.GetStringAsync(next);! Console.WriteLine(content);! } Resume with data.
  65. async Task GetIt(string url)! {! HttpClient client = new HttpClient();!

    string content;! ! content = await client.GetStringAsync(url);! ! string next = JsonConvert.DeserializeObject! <Dictionary<string, string>>(content)! ["url"];! ! content = await client.GetStringAsync(next);! Console.WriteLine(content);! }
  66. async Task GetIt(string url)! {! HttpClient client = new HttpClient();!

    string content;! ! content = await client.GetStringAsync(url);! ! string next = JsonConvert.DeserializeObject! <Dictionary<string, string>>(content)! ["url"];! ! content = await client.GetStringAsync(next);! Console.WriteLine(content);! }
  67. Coroutine • “Pauses” when calling another coroutine • “Resumes” when

    another coroutine calls it again • Subroutines are special cases of [coroutines] — Knuth
  68. import json! from aiohttp import request! ! def get_it(url):! response

    = yield from request('GET', url)! data = yield from response.read()! next_url = json.loads(data)['url']! response = yield from request('GET', url)! data = yield from response.read()! print(data) https://twistedmatrix.com/trac/browser/trunk/twisted/internet/defer.py
  69. https://tw.pycon.org/

  70. Generators • “Semi-coroutines” • Always yield to its caller •

    Primarily used to simply iterators
  71. def get_generator():! yield 0! yield 1! ! ! g =

    get_generator()! print(type(g)) # <type 'generator'>! ! for i in g:! print(i)! # 0! # 1
  72. g = get_generator()! while True:! try:! i = next(g) #

    Python 3! print(i)! except StopIteration:! break! # 0! # 1
  73. g = get_generator()! while True:! try:! i = g.send(None)! print(i)!

    except StopIteration:! break! # 0! # 1 http://www.python.org/dev/peps/pep-0342/
  74. def get_generator():! v = yield 0! print('Gen:', v)! v =

    yield v! print('Gen:', v)! ! g = get_generator()! v = g.send(None) # Starts g.! ! while True:! try:! print('Con:', v)! v = g.send(v + 1)! except StopIteration:! break Con: 0 ! Gen: 1! Con: 1! Gen: 2 Outputs
  75. def get_generator():! v = yield 0! print('Gen:', v)! v =

    yield v! print('Gen:', v)! ! g = get_generator()! v = g.send(None)! ! while True:! try:! print('Con:', v)! v = g.send(v + 1)! except StopIteration:! break Con: 0 ! Gen: 1! Con: 1! Gen: 2 Outputs
  76. def get_generator():! v = yield 0! print('Gen:', v)! v =

    yield v! print('Gen:', v)! ! g = get_generator()! v = g.send(None)! ! while True:! try:! print('Con:', v)! v = g.send(v + 1)! except StopIteration:! break Con: 0 ! Gen: 1! Con: 1! Gen: 2 Outputs
  77. def get_generator():! v = yield 0! print('Gen:', v)! v =

    yield v! print('Gen:', v)! ! g = get_generator()! v = g.send(None)! ! while True:! try:! print('Con:', v)! v = g.send(v + 1)! except StopIteration:! break Con: 0 ! Gen: 1! Con: 1! Gen: 2 Outputs
  78. def get_generator():! v = yield 0! print('Gen:', v)! v =

    yield v! print('Gen:', v)! ! g = get_generator()! v = g.send(None)! ! while True:! try:! print('Con:', v)! v = g.send(v + 1)! except StopIteration:! break Con: 0 ! Gen: 1! Con: 1! Gen: 2 Outputs
  79. def get_generator():! v = yield 0! print('Gen:', v)! v =

    yield v! print('Gen:', v)! ! g = get_generator()! v = g.send(None)! ! while True:! try:! print('Con:', v)! v = g.send(v + 1)! except StopIteration:! break Con: 0 ! Gen: 1! Con: 1! Gen: 2 Outputs
  80. def get_generator():! v = yield 0! print('Gen:', v)! v =

    yield v! print('Gen:', v)! ! g = get_generator()! v = g.send(None)! ! while True:! try:! print('Con:', v)! v = g.send(v + 1)! except StopIteration:! break Con: 0 ! Gen: 1! Con: 1! Gen: 2 Outputs
  81. def get_generator():! v = yield 0! print('Gen:', v)! v =

    yield v! print('Gen:', v)! ! g = get_generator()! v = g.send(None)! ! while True:! try:! print('Con:', v)! v = g.send(v + 1)! except StopIteration:! break Con: 0 ! Gen: 1! Con: 1! Gen: 2 Outputs
  82. def get_generator():! v = yield 0! print('Gen:', v)! v =

    yield v! print('Gen:', v)! ! g = get_generator()! v = g.send(None)! ! while True:! try:! print('Con:', v)! v = g.send(v + 1)! except StopIteration:! break Con: 0 ! Gen: 1! Con: 1! Gen: 2 Outputs
  83. def get_generator():! v = yield 0! print('Gen:', v)! v =

    yield v! print('Gen:', v)! ! g = get_generator()! v = g.send(None)! ! while True:! try:! print('Con:', v)! v = g.send(v + 1)! except StopIteration:! break Con: 0 ! Gen: 1! Con: 1! Gen: 2 Outputs
  84. def get_generator():! v = yield 0! print('Gen:', v)! v =

    yield v! print('Gen:', v)! ! g = get_generator()! v = g.send(None)! ! while True:! try:! print('Con:', v)! v = g.send(v + 1)! except StopIteration:! break Con: 0 ! Gen: 1! Con: 1! Gen: 2 Outputs No more yielding.
  85. def get_generator():! v = yield 0! print('Gen:', v)! v =

    yield v! print('Gen:', v)! ! g = get_generator()! v = g.send(None)! ! while True:! try:! print('Con:', v)! v = g.send(v + 1)! except StopIteration:! break Con: 0 ! Gen: 1! Con: 1! Gen: 2 Outputs No more yielding.
  86. def get_generator():! v = yield 0! print('Gen:', v)! v =

    yield v! print('Gen:', v)! ! g = get_generator()! v = g.send(None)! ! while True:! try:! print('Con:', v)! v = g.send(v + 1)! except StopIteration:! break Con: 0 ! Gen: 1! Con: 1! Gen: 2 Outputs
  87. Async with Generators • Yield to event loop • Event

    loop calls everything • What should they yield?
  88. def inlineCallbacks(f):! ! @wraps(f)! def unwindGenerator(*args, **kwargs):! try:! gen =

    f(*args, **kwargs)! except _DefGen_Return:! raise TypeError("…")! if not isinstance(gen, GeneratorType):! raise TypeError("…")! d = Deferred()! return _inlineCallbacks(None, gen, d)! ! return unwindGenerator https://twistedmatrix.com/trac/browser/trunk/twisted/internet/defer.py
  89. def _inlineCallbacks(result, g, deferred):! while True:! try:! result = g.send(result)!

    except StopIteration:! deferred.callback(None)! return deferred! except _DefGen_Return as e:! deferred.callback(e.value)! return deferred! except:! deferred.errback()! return deferred! if isinstance(result, Deferred):! # <Get result from yielded Deferred>! return deferred https://twistedmatrix.com/trac/browser/trunk/twisted/internet/defer.py
  90. def _inlineCallbacks(result, g, deferred):! while True:! try:! result = g.send(result)!

    except StopIteration:! deferred.callback(None)! return deferred! except _DefGen_Return as e:! deferred.callback(e.value)! return deferred! except:! deferred.errback()! return deferred! if isinstance(result, Deferred):! # <Get result from yielded Deferred>! return deferred https://twistedmatrix.com/trac/browser/trunk/twisted/internet/defer.py
  91. def _inlineCallbacks(result, g, deferred):! while True:! try:! result = g.send(result)!

    except StopIteration:! deferred.callback(None)! return deferred! except _DefGen_Return as e:! deferred.callback(e.value)! return deferred! except:! deferred.errback()! return deferred! if isinstance(result, Deferred):! # <Get result from yielded Deferred>! return deferred https://twistedmatrix.com/trac/browser/trunk/twisted/internet/defer.py
  92. def _inlineCallbacks(result, g, deferred):! while True:! try:! result = g.send(result)!

    except StopIteration:! deferred.callback(None)! return deferred! except _DefGen_Return as e:! deferred.callback(e.value)! return deferred! except:! deferred.errback()! return deferred! if isinstance(result, Deferred):! # <Get result from yielded Deferred>! return deferred https://twistedmatrix.com/trac/browser/trunk/twisted/internet/defer.py
  93. Return Values >>> def get_generator():! ... yield 0! ... yield

    1! ... return 'I am done.'! ...! File "<stdin>", line 3! SyntaxError: 'return' with argument inside generator
  94. class _DefGen_Return(BaseException):! def __init__(self, value):! self.value = value! ! !

    def returnValue(val):! raise _DefGen_Return(val) https://twistedmatrix.com/trac/browser/trunk/twisted/internet/defer.py
  95. def _inlineCallbacks(result, g, deferred):! while True:! try:! result = g.send(result)!

    except StopIteration:! deferred.callback(None)! return deferred! except _DefGen_Return as e:! deferred.callback(e.value)! return deferred! except:! deferred.errback()! return deferred! if isinstance(result, Deferred):! # <Get result from yielded Deferred>! return deferred https://twistedmatrix.com/trac/browser/trunk/twisted/internet/defer.py
  96. import json! from twisted.internet import reactor, defer! from twisted.web import

    client! ! ! @defer.inlineCallbacks! def getIt(url):! content = yield client.getPage(url)! url = json.loads(content)['url']! content = yield client.getPage(url)! print content! # We want to return the content! ! ! getIt(someUrl)! reactor.run()
  97. import json! from twisted.internet import reactor, defer! from twisted.web import

    client! ! ! @defer.inlineCallbacks! def getIt(url):! content = yield client.getPage(url)! url = json.loads(content)['url']! content = yield client.getPage(url)! print content! defer.returnValue(content)! ! ! getIt(someUrl)! reactor.run()
  98. import json! from twisted.internet import reactor, defer! from twisted.web import

    client! ! ! @defer.inlineCallbacks! def getIt(url):! content = yield client.getPage(url)! url = json.loads(content)['url']! content = yield client.getPage(url)! print content! raise defer._DefGen_Return(content)! ! ! getIt(someUrl)! reactor.run()
  99. WAT Your Captain is not impressed.

  100. >>> def get_generator():! ... yield 0! ... yield 1! ...

    return 'I am done.'! ...! >>> g = get_generator()! >>> while True: print(next(g))! 0! 1! Traceback (most recent call last):! File "<stdin>", line 1, in <module>! StopIteration: I am done Python 3.3
  101. class Task(futures.Future):! # …! def _step(self, value=None, exc=None):! # …!

    try:! if exc is not None:! result = self._coro.throw(exc)! elif value is not None:! result = self._coro.send(value)! else:! result = next(self._coro)! except StopIteration as exc:! self.set_result(exc.value)! # … http://hg.python.org/cpython/file/default/Lib/asyncio/tasks.py
  102. ECMA Standard 6 var g = function*() {! yield 0;!

    yield 1;! }();! ! console.log(g.next().value); // 0! console.log(g.next().value); // 1
  103. var g = function* () {! var v = yield

    0;! console.log('Gen: ' + v);! var v = yield v;! console.log('Gen: ' + v);! }();! ! var obj = g.next();! while (!obj.done) {! var v = obj.value;! console.log('Con: ' + v);! obj = g.next(v + 1);! } Con: 0 ! Gen: 1! Con: 1! Gen: 2 Outputs
  104. Next Generation • Promises — checked. • Generators — checked.

    • Implementation…
  105. co(function* () {! var json =! yield Q($.getJSON(url));! var data

    =! yield Q($.get(json.url));! console.log(data);! })(); co https://github.com/visionmedia/co
  106. co(function* () {! var results = yield Q.all([! Q($.getJSON(url1),! Q($.getJSON(url2)!

    ]);! var url = results[0].host + ! results[1].path;! var data = yield Q($.get(url));! console.log(data);! })(); Seamless Integration
  107. Wrapping Up • Callbacks • Promises • Coroutines / Generators

  108. Extra Notes • Doing things in parallel • Exception handling

    • There are more • Callbacks are not dead
  109. Bonus Chatter $('#button').click(function () {! var that = this;! var

    url = $(this).data('url');! $.getJSON(url, function (data) {! $(that).text(data.content); ! });! });
  110. Bonus Chatter $('#button').click(function () {! var that = this;! var

    url = $(this).data('url');! $.getJSON(url, function (data) {! $(that).text(data.content); ! });! });
  111. Bonus Chatter $('#button').click(co(function* () {! var url = $(this).data('url');! var

    data = yield Q($.getJSON(url));! $(this).text(data.text)! }));
  112. Link Dump • Python Coroutines, Present and Future • The

    difference between yield and yield-from • Deconstructing Deferred • PEPs 255, 342, 380, and 3148 • Python 3.4 asyncio
  113. Link Dump • JavaScript Promises: There and Back Again •

    You’re Missing the Point of Promises • Bringing async/await to life in JavaScript • ECMA-262 6th Edition - DRAFT • task.js (by Mozilla)
  114. Link Dump • Callbacks as our Generations’ Go To Statement

    • Coroutines in C • Futures (Objective-C) • Generators in Objective-C • SHXPromise (Objective-C)
  115. FIRE QUESTIONS