The Good, the Bad, and the Async

How does an async runtime work? And more importantly, why bother with async at all? Is it worth the trouble and added complexity?

Async is arguably one of Rust's most complex and least understood features. As a wise man once said: "Async Rust is Rust on hard mode".

I will be your guide through this maze. In the process of writing a small async runtime, I'll help you explore and learn the inner workings of the system. This way, you will truly understand how async works, and will be able to use it confidently in your own code when you need to.

Transcript

2. Asynchronous Programming All you need is a single thread? what kind of sorcery are you using!
4. Goals • Understand the benefits of using the async system • Understand the inner workings of the async system in rust • Learn to build an executor from zero • Understand the pitfalls in the async system 4
6. Alternatives 6 • Blocking IO • Non-blocking IO • Event Poll • Language level solution - welcome async
7. Blocking IO fn main() { let listener = TcpListener::bind("127.0.0.1:0").unwrap(); loop { let (conn, _) = listener.accept().unwrap(); std::thread::spawn(|| handle_request(conn)); } } 7
8. The problems with blocking IO • We lose control over the thread • Every io-request is another OS thread • OS threads are expensive to create and tend to pre-allocate a lot of resources 8
9. Non-blocking IO • The program isn’t blocked anymore. More control! More power! • No need to create an OS thread per request 9
10. Non-blocking IO the busy waiting hell fn main() { let listener = TcpListener::bind("127.0.0.1:0").unwrap(); listener.set_nonblocking(true).unwrap(); loop { match listener.accept() { Ok((conn, _)) => todo!() // figure out something Err(err) if err.kind() == ErrorKind::WouldBlock => continue, Err(err) => panic!("{}", err), 10
11. Event poll - Blocking is kinda important but only when we really have nothing else to do, no need to waste cpu cycles! fn main() { let epoll = Epoll::new(EpollCreateFlags ::EPOLL_CLOEXEC) .unwrap(); let listener = TcpListener ::bind("127.0.0.1" ).unwrap(); listener.set_nonblocking (true).unwrap(); epoll.add( listener.as_fd(), EpollEvent::new(EpollFlags::EPOLLIN, listener.as_raw_fd() as u64), ).unwrap(); let mut events = [EpollEvent::empty(); 128]; loop { let count = epoll.wait(&mut events, EpollTimeout ::NONE).unwrap(); for event in &events[..count] { if event.data() == listener.as_raw_fd() as u64 { let (conn, _) = listener.accept().unwrap(); epoll.add( conn.as_fd(), EpollEvent::new( EpollFlags::EPOLLIN | EpollFlags::EPOLLOUT, conn.as_raw_fd() as _)).unwrap(); } } } } 11
12. The problems with event poll • It’s too low-level • It’s prone to errors: deadlocks, data collisions, etc.. • Hard to use and reason about • What about my windows system? 12
13. Language level solution • Threads in user space - lets call them tasks! • User interface - do we actually need control? • What happens to the stack? Is it stackless if the size is a perfect match? • Runtime - someone needs to schedule our tasks… and we don’t want the OS to do it 13
14. What do we have in rust? who cares about other languages • Cooperative scheduling • Built-in types that define traits that power a clearly defined state machines - our tasks • Useful built in futures that don’t rely on a runtime • Syntax sugar that eliminates the need to write a state machine (async/await api) • We don’t have a runtime in std 14
15. Another Solution Async 15
16. async fn async_main() { let mut listener = TcpListener::bind("127.0.0.1:0").unwrap(); loop { let (conn, _) = listener.accept().await.unwrap(); Scheduler::spawn(connection_handler(conn)); } } 16
17. Futures • Tasks are great, but their scope is too big! • Smaller scope means a new term… welcome Futures! 17
18. The future trait pub trait Future { type Output; // Required method fn poll( self: Pin<&mut Self>, cx: &mut Context<'_> ) -> Poll<Self::Output>; } 18
19. Poll flow pub enum Poll<T> { Ready(T), Pending, } I schedule you once. want to be scheduled again? wake yourself, you lazy 19
20. fn async_main() -> impl std::future::Future<Output = ()> { async { let mut listener = TcpListener::bind("127.0.0.1:1663").unwrap(); loop { let (conn, _) = listener.accept().await.unwrap(); Scheduler::spawn(connection_handler(conn)); } } } 20
21. I’m the boss! 21
22. I’m the boss! 22
23. I’m the boss! 23
24. I’m the boss! 24
25. Let’s write a runtime 25
26. 26 I manage epoll! I schedule tasks
27. #[derive(Default)] pub struct Scheduler { next_id: TaskId, tasks: HashMap<TaskId, Task>, pending: Vec<TaskId>, } thread_local! { pub(crate) static SCHEDULER: RefCell<Option<Scheduler>> = RefCell::new(None); } 27
28. #[derive(Debug, Clone, Copy, Hash, PartialEq, Eq)] pub(crate) struct TaskId(usize); pub type Task = Pin<Box<dyn Future<Output = ()>>>; 28
29. impl Scheduler { pub(crate) fn schedule(&mut self, id: TaskId) { self.pending.push(id); } } 29
30. /// Spawn a new task on the current scheduler pub fn spawn(task: impl Future<Output = ()> + 'static) { let task = Box::pin(task); SCHEDULER.with_borrow_mut(|scheduler| { let scheduler = scheduler.as_mut().unwrap(); let id = scheduler.next_id.next(); scheduler.tasks.insert(id, task); scheduler.schedule(id); }); } 30
31. pub fn block_on(self, main_task: impl Future<Output = ()> + 'static) { // inject necessary data into the thread SCHEDULER.with_borrow_mut(|scheduler| { if scheduler.is_some() { panic!("can not spawn more than 1 run time on the same thread" ); } *scheduler = Some(self); }); // spawn main task and start scheduling Self::spawn(main_task); Self::start(); // remove the injected data from thread SCHEDULER.take(); } 31
32. fn start() { loop { let pending = SCHEDULER.with_borrow_mut(|scheduler| { std::mem::take(&mut scheduler.as_mut().unwrap().pending) }); for id in pending { Self::run_task_by_id (id); } match Self::status() { ExecStatus::Work => continue, ExecStatus::NoTasks => return, ExecStatus::NoPending => { todo!(); // block on the reactor } } } 32
33. fn start() { loop { let pending = SCHEDULER.with_borrow_mut (|scheduler| { std::mem::take(&mut scheduler.as_mut().unwrap().pending) }); for id in pending { Self::run_task_by_id(id); } match Self::status() { ExecStatus::Work => continue, ExecStatus::NoTasks => return, ExecStatus::NoPending => { todo!(); // block on the reactor } } } 33
34. fn start() { loop { let pending = SCHEDULER.with_borrow_mut (|scheduler| { std::mem::take(&mut scheduler.as_mut().unwrap().pending) }); for id in pending { Self::run_task_by_id (id); } match Self::status() { ExecStatus::Work => continue, ExecStatus::NoTasks => return, ExecStatus::NoPending => { todo!(); // block on the reactor } } } 34
35. fn start() { loop { let pending = SCHEDULER.with_borrow_mut (|scheduler| { std::mem::take(&mut scheduler.as_mut().unwrap().pending) }); for id in pending { Self::run_task_by_id (id); } match Self::status() { ExecStatus::Work => continue, ExecStatus::NoTasks => return, ExecStatus::NoPending => { todo!(); // block on the reactor } } } 35
36. fn start() { loop { let pending = SCHEDULER.with_borrow_mut (|scheduler| { std::mem::take(&mut scheduler.as_mut().unwrap().pending) }); for id in pending { Self::run_task_by_id (id); } match Self::status() { ExecStatus::Work => continue, ExecStatus::NoTasks => return, ExecStatus::NoPending => { todo!(); // block on the reactor } } } 36
37. fn status() -> ExecStatus { SCHEDULER.with_borrow(|scheduler| { let scheduler = scheduler.as_ref().unwrap(); if scheduler.tasks.is_empty() { return ExecStatus::NoTasks; } else if scheduler.pending.is_empty() { return ExecStatus::NoPending; } else { return ExecStatus::Work; } }) } enum ExecStatus { Work, NoTasks, NoPending, 37
38. fn run_task_by_id(id: TaskId) { let task = SCHEDULER.with_borrow_mut( |scheduler| scheduler.as_mut().unwrap().tasks.remove(&id) ); let Some(mut task) = task else { return; }; let waker: Waker = id.into(); match task.as_mut().poll(&mut Context::from_waker(&waker)) { Poll::Pending => { SCHEDULER.with_borrow_mut (|scheduler| { let scheduler = scheduler.as_mut().unwrap(); scheduler.tasks.insert(id, task); }); } Poll::Ready(()) => {} } } 38
39. fn run_task_by_id(id: TaskId) { let task = SCHEDULER.with_borrow_mut ( |scheduler| scheduler.as_mut().unwrap().tasks.remove(&id) ); let Some(mut task) = task else { return; }; let waker: Waker = id.into(); match task.as_mut().poll(&mut Context::from_waker(&waker)) { Poll::Pending => { SCHEDULER.with_borrow_mut(|scheduler| { let scheduler = scheduler.as_mut().unwrap(); scheduler.tasks.insert(id, task); }); } Poll::Ready(()) => {} } } 39
40. 40 I manage epoll! I schedule tasks ? ✓
41. use nix::sys::epoll::Epoll; pub(crate) struct Reactor { epoll: Epoll, events: HashMap<u64, Waker>, } thread_local! { pub(crate) static REACTOR: RefCell<Option<Reactor>> = RefCell::new(None); } 41
42. The lifetime of a leaf future 1. Register the relevant event in the reactor 2. Update the reactor the latest waker 3. Remove itself from the reactor 42
43. pub fn register<Fd>(&mut self, fd: &Fd, flags: EpollFlags) where Fd: AsFd + AsRawFd, { let id = fd.as_raw_fd() as u64; let _ = self.epoll.add(fd, EpollEvent::new(flags, id)); } 43
44. pub fn update_waker<Fd>(&mut self, fd: &Fd, waker: Waker) where Fd: AsFd + AsRawFd, { let id = fd.as_raw_fd() as u64; self.events.insert(id, waker); } 44
45. pub fn remove<Fd>(&mut self, fd: &Fd) where Fd: AsFd + AsRawFd, { let id = fd.as_raw_fd() as u64; self.events.remove(&id); let _ = self.epoll.delete(fd); } 45
46. pub fn block(&self) { let mut events = [EpollEvent::empty(); 1024]; let count = self.epoll.wait( &mut events, EpollTimeout::NONE ).unwrap(); for event in &events[..count] { let id = event.data(); if let Some(waker) = self.events.get(&id) { waker.wake_by_ref(); } } } 46
47. pub fn block(&self) { let mut events = [EpollEvent::empty(); 1024]; let count = self.epoll.wait( &mut events, EpollTimeout::NONE ).unwrap(); for event in &events[..count] { let id = event.data(); if let Some(waker) = self.events.get(&id) { waker.wake_by_ref(); } } } 47
48. 48 I manage epoll! I schedule tasks ✓ ✓ |
49. pub fn block_on(self, main_task: impl Future<Output = ()> + 'static) { // inject necessary data into the thread SCHEDULER.with_borrow_mut(|scheduler| { if scheduler.is_some() { panic!("can not spawn more than 1 run time on the same thread"); } *scheduler = Some(self); }); REACTOR.with_borrow_mut(|reactor| { *reactor = Some(Reactor::default()); }); // spawn main task and start scheduling Self::spawn(main_task); Self::start(); // remove the injected data from thread SCHEDULER.take(); REACTOR.take(); } 49
50. fn start() { loop { let pending = SCHEDULER.with_borrow_mut (|scheduler| { std::mem::take(&mut scheduler.as_mut().unwrap().pending) }); for id in pending { Self::run_task_by_id (id); } match Self::status() { ExecStatus::Work => continue, ExecStatus::NoTasks => return, ExecStatus::NoPending => { REACTOR.with_borrow(|reactor| reactor.as_ref().unwrap().block()) } } } 50
51. The runtime is done almost 51
52. I’m the boss! 52 ? ? ? ? ? ? ? ? ?
53. Mirroring std’s net module 53
55. pub struct TcpListener { listener: std::net::TcpListener, } 55
56. pub fn bind(addr: impl std::net::ToSocketAddrs) -> std::io::Result<Self> { let listener = std::net::TcpListener::bind(addr)?; listener.set_nonblocking(true)?; Ok(Self { listener }) } 56
57. pub fn accept(&mut self) -> impl Future<Output = ListenerAcceptOutput> + '_ { ListenerAccept::new(self) } pub type ListenerAcceptOutput = std::io::Result<(TcpStream, std::net::SocketAddr)>; 57
58. pub fn accept(&mut self) -> impl Future<Output = ListenerAcceptOutput> + '_ { ListenerAccept::new(self) } pub type ListenerAcceptOutput = std::io::Result<(TcpStream, std::net::SocketAddr)>; 58
59. struct ListenerAccept<'a> { listener: &'a mut TcpListener, } 59
60. The lifetime of a leaf future 1. Register the relevant event in the reactor 2. Update the reactor the latest waker 3. Remove itself from the reactor 60
61. impl<'a> ListenerAccept<'a> { fn new(listener: &'a mut TcpListener) -> Self { REACTOR.with_borrow_mut(|reactor| { reactor.as_mut().unwrap() .register( &listener.listener, EpollFlags::EPOLLIN ) }); Self { listener } }} 61
62. impl<'a> Future for ListenerAccept <'a> { type Output = ListenerAcceptOutput ; fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output> { match self.listener.listener.accept() { Ok((stream, addr)) => Poll::Ready(Ok((TcpStream::new(stream)?, addr))), Err(e) if e.kind() == std::io::ErrorKind::WouldBlock => { REACTOR.with_borrow_mut(|reactor| { reactor .as_mut() .unwrap() .update_waker(&self.listener.listener, cx.waker().clone()) }); Poll::Pending } Err(e) => Poll::Ready(Err(e)), } } } 62
63. impl<'a> Future for ListenerAccept <'a> { type Output = ListenerAcceptOutput ; fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output> { match self.listener.listener.accept() { Ok((stream, addr)) => Poll::Ready(Ok((TcpStream::new(stream)?, addr))), Err(e) if e.kind() == std::io::ErrorKind::WouldBlock => { REACTOR.with_borrow_mut (|reactor| { reactor.as_mut().unwrap() . update_waker(&self.listener.listener, cx.waker().clone()) }); Poll::Pending } Err(e) => Poll::Ready(Err(e)), } } } 63
64. impl<'a> Future for ListenerAccept <'a> { type Output = ListenerAcceptOutput ; fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output> { match self.listener.listener.accept() { Ok((stream, addr)) => Poll::Ready(Ok((TcpStream::new(stream)?, addr))), Err(e) if e.kind() == std::io::ErrorKind::WouldBlock => { REACTOR.with_borrow_mut(|reactor| { reactor .as_mut() .unwrap() .update_waker(&self.listener.listener, cx.waker().clone()) }); Poll::Pending } Err(e) => Poll::Ready(Err(e)), } } } 64
65. impl<'a> Drop for ListenerAccept<'a> { fn drop(&mut self) { REACTOR.with_borrow_mut(|reactor| reactor.as_mut().unwrap() .remove(&self.listener.listener)); } } 65
66. pub struct TcpStream { stream: std::net::TcpStream, } 66
67. Insert more boilerplate here 67
68. Live demo! it actually works, I promise 68
69. Recap • Blocking-io isn’t the most efficient way • To use non-blocking-io we need sth like event poll • Event poll is too low level, we want language level solution • In rust the solution is the async system • In the async system, the game is about futures, not tasks • A task is just a future that was spawned on the runtime 69
70. Tokio • Flexible • Robust - made for production • Will cover any generic need for a io-based application 70
71. Where is my food 71
72. Bonus: common pitfalls 72
73. I’m the boss! 73 What happened to my stack
74. 74 Disconnect branches
75. 75 Disconnect branches
76. Recap: It may be called async fn But it’s fn -> Future • The compiler needs to generate a future from the function body • The future needs to be allocated in memory • If we hold data across .await points, this data needs to live inside the future • Box::pin(future()).await - useful to “cut” a big future from your own future • It’s still too sweet to be good to your health! example 76
77. Where is my food 77
79. Limiting async scope async fn bad_example(some_data: &SomeData) { // use somedata - some housekeeping at the start, no need for async! use_some_data(some_data); // first call to .await some_future().await; // the rest of the function - we don’t use some_data here } 79
80. Limiting async scope fn bad_example<'a>(some_data: &'a SomeData) -> impl Future<Output = ()> + 'a { async { // use somedata - some housekeeping at the start, no need for async! use_some_data(some_data); // first call to .await some_future().await; // the rest of the function } } 80
81. Limiting async scope fn bad_example(some_data: &SomeData) -> impl Future<Output = ()> { // use somedata - some housekeeping at the start, no need for async! use_some_data(some_data); async { // first call to .await some_future().await; // the rest of the function } } 81
82. The coloring problem! Better be color blind with these many colors I really want to use this crate… but we ain’t the same color, this going to be a challenge 1. I need to spawn a runtime to run your async function 2. I already use a runtime, but it can’t run your async function 3. I already use a runtime, so I can’t execute your blocking function 4. How many runtimes do I need to support to make this crate useful? 5. Is this mess even worth it..? 82
83. Tracing how do we collect all these logs 1. Structured application-level diagnostic 2. 2-step approach for gathering information 83
84. Tracing #[tracing::instrument] pub fn shave(yak: usize) -> Result<(), Box<dyn Error + 'static>> { tracing::debug!(excitement = "yay!", "hello! I'm gonna shave a yak."); if yak == 3 { tracing::warn!("could not locate yak!"); return Err(io::Error::new(io::ErrorKind::Other, "shaving yak failed!").into()); } else { tracing::debug!("yak shaved successfully"); } Ok(()) } 84
85. Tracing subscriber 1. Output everything to stdout - tracing_subscriber ::fmt::init(); 2024-03-26T07:27:51.971583Z WARN shave{yak=3}: tracing_example: could not locate yak! 2. Jaeger (web interface) 85
86. Tracing spans & Async scopes async fn scope_example() { // records an event outside of any span context: event!(Level::INFO, "something happened" ); let span = span!(Level::INFO, "my_span"); let _guard = span.enter(); a_cool_future().await; // records an event within "my_span". event!(Level::DEBUG, "something happened inside my_span" ); } 86
87. Tracing spans & Async scopes async fn scope_example() { // records an event outside of any span context: event!(Level::INFO, "something happened" ); let span = span!(Level::INFO, "my_span"); async { a_cool_future().await; // records an event within "my_span". event!(Level::DEBUG, "something happened inside my_span" ); } .instrument(span) .await; } 87
88. Tracing spans & Async scopes impl<T: Future> Future for Instrumented<T> { type Output = T::Output; fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output> { let (span, inner) = self.project().span_and_inner_pin_mut (); let _enter = span.enter(); inner.poll(cx) } } 88
89. Context do we actually need to explain this part? • We don’t actually know anything about its implementation • It gives us a useful function, waker() • The associated waker can be used to… schedule ourself! You asked for cooperation… 89
90. Pin self reference? sure. but you ain’t moving around buddy. • Pinned struct can not be moved around • You can only unpin an unpinned struct • You can only be unpinned if you don’t have self-referential data • We don’t actually need to write self-referential structs by hand… 90
91. Pin why do we need this again? enum SimpleFutureState { State1(SomeBigType), State2(SomeBigType, u8), State3(u8), } 91
92. Pin sorry. why do we need this again? struct SimpleFuture { big1: SomeBigType, state: SimpleFutureState<'reference big1 please>, } enum SimpleFutureState<'a> { State1(&'a SomeBigType), State2(&'a SomeBigType,u8), State3(u8), } 92
93. Syntax sugar where is my cake? async fn connection_handler(mut stream: TcpStream) { let mut buf = [0u8; 1024]; while let Ok(rcount) = stream.read(&mut buf).await { if rcount == 0 { return; } stream.write(&buf[..rcount]).await.unwrap(); } } 93
94. Dynamic binding If you can’t choose one, choose all! But be prepared to pay with safety. • &dyn Trait • Box<dyn Trait> • Rc<dyn Trait> • Arc<dyn Trait> Safety is important… but only people that write runtimes are going to deal with it! 94