Lock in $30 Savings on PRO—Offer Ends Soon! ⏳

Reactive Slick at Scala User Group Vienna May 2...

Reactive Slick at Scala User Group Vienna May 20 2015

The Principles behind Slick 3

The code shown in the talk (not on the slides) was mostly from https://www.typesafe.com/activator/template/hello-slick-3.0

Lutz Hühnken

May 20, 2015
Tweet

More Decks by Lutz Hühnken

Other Decks in Programming

Transcript

  1. JDBC try { // make the connection Class.forName(driver) connection =

    DriverManager.getConnection(url, username, password) // create the statement, and run the select query val statement = connection.createStatement() val resultSet = statement.executeQuery("SELECT host, user FROM user") while ( resultSet.next() ) { val host = resultSet.getString("host") val user = resultSet.getString("user") println("host, user = " + host + ", " + user) } } catch { case e => e.printStackTrace } connection.close() } 2
  2. Reactive Slick JDBC The problem with that code • well

    there are probably many, style etc. But the problem: • Synchronous I/O • Will block the thread it’s running on. • This is true not only for JDBC, but any DB lib running on top of JDBC, Slick included. We like to avoid that. Why? 3
  3. Reactive Slick Digression: Concurrency model Java EE: Threads. Servlet API:

    Thread per Request 4 Note: This is a snapshot at one point in time, not a sequence of events.
  4. Reactive Slick Problems with Threads • Context Switching is expensive

    • Memory overhead per thread • Lock contention when communicating between threads (if you have shared mutable state, which is likely..) 5
  5. Reactive Slick Digression: Concurrency model 6 Source: John Rose, Java

    VM Architect, JFokus, Stockholm, February 2015
  6. Reactive Slick Task level concurrency This does not mean: „no

    threads“. It means: Threads should not be the finest level of concurrency. Not be the level the application developer works on. 7
  7. Reactive Slick Digression: Concurrency model Reactive: Sub-Thread. Play: n threads

    per m requests 8 Note: This is a snapshot at one point in time, not a sequence of events.
  8. Reactive Slick Solution: Use Future / blocking combo • Put

    blocking database calls in Future(blocking( ... )) • Contention for Connections (but may be limited by the ExecutionContext) • A saturated thread pool blocks everything. 9
  9. Reactive Slick Better solution: Isolation 10 Note: This is a

    snapshot at one point in time, not a sequence of events.
  10. Reactive Slick New Question: Size Idea: number of threads <=

    number of connections in pool, so getConnection never blocks. Given you want to handle 10.000 concurrent clients.. what size should your connection pool be? • 10 • 100 • 1.000 • 10.000 11
  11. Reactive Slick It was a trick question… A formula which

    has held up pretty well across a lot of benchmarks for years is that for optimal throughput the number of active connections should be somewhere near ((core_count * 2) + effective_spindle_count). Core count should not include HT threads, even if hyperthreading is enabled. Effective spindle count is zero if the active data set is fully cached, and approaches the actual number of spindles as the cache hit rate falls. ... There hasn't been any analysis so far regarding how well the formula works with SSDs. 12 (From PostgreSQL)
  12. Reactive Slick Slick Every Database contains an AsyncExecutor that manages

    the thread pool for asynchronous execution of Database I/O Actions. 15 mydb = { dataSourceClass = "org.postgresql.ds.PGSimpleDataSource" properties = { databaseName = "mydb" user = "myuser" password = "secret" } numThreads = 10 }
  13. Reactive Slick JDBC (again, it’s the same slide) try {

    // make the connection Class.forName(driver) connection = DriverManager.getConnection(url, username, password) // create the statement, and run the select query val statement = connection.createStatement() val resultSet = statement.executeQuery("SELECT host, user FROM user") while ( resultSet.next() ) { val host = resultSet.getString("host") val user = resultSet.getString("user") println("host, user = " + host + ", " + user) } } catch { case e => e.printStackTrace } connection.close() } 17
  14. Reactive Slick Reactive Streams Overview • How do we: •

    Handle potentially infinite streams of data? • Handle data in a reactive manner? • Achieve asynchronous non-blocking data flow? • Avoid out of memory errors? 19
  15. Reactive Slick Supply and Demand • Data Items Flow Downstream

    • Demand Flows Upstream • Data Items flow only when there is demand. 20
  16. Reactive Slick Reactive Streams Specification • Interface Specification • Java

    interfaces for implementations • TCK • Test Harness to validate implementations • Specification Website • http://www.reactive-streams.org/ 21
  17. Reactive Slick Subscriber public interface Subscriber<T> { public void onSubscribe(

    Subscription subscription); public void onNext(T element); public void onComplete(); public void onError(Throwable cause); } 23
  18. Reactive Slick Dynamic Push-Pull • “Push” behavior when consumer is

    faster • “Pull” behavior when producer is faster • Switches automatically between these • Batching demand allows batching data 25
  19. Reactive Slick Linear Stream Transformations • Deterministic (like for collections)

    • map, filter, collect, grouped, drop, take, groupBy, ... • Time-Based • takeWithin, dropWithin, groupedWithin, ... • Rate-Detached • expand, conflate, buffer, ... • asynchronous • mapAsync, mapAsyncUnordered, ... 28
  20. Reactive Slick Materialization • Akka Streams separate the what from

    the how • declarative Source/Flow/Sink DSL to create blueprint • FlowMaterializer turns this into running Actors 32