High Performance Networking on the JVM - Lessons learned

Slide 1

Slide 1 text

HIGH PERFORMANCE NETWORKING ON THE JVM LESSONS LEARNED

Slide 2

Slide 2 text

NORMAN MAURER Red Hat (JBoss) - EAP Core Team Former contractor for Apple Inc Author - Netty in Action Apache Software Foundation Member Netty / Vert.x Core-Developer and all things NIO Java and Scala Twitter: @normanmaurer Github: https://github.com/normanmaurer

Slide 3

Slide 3 text

GENERAL As always, only optimize if you really need! 1000 concurrent connections != high-scale If you only need to handle a few hundred connections use Blocking IO! Make use of a profiler to find issues, and not best-guess... Always test before and after changes, and not forget to warmup!

Slide 4

Slide 4 text

WHAT YOU WANT “For high-performance with many concurrent connections you WANT to use NIO or NIO.2”

Slide 5

Slide 5 text

WHAT YOU NOT WANT “Create one Thread per connection and let the OS try to deal with thousands of threads” “If you want.... good luck ;)”

Slide 6

Slide 6 text

SOCKET OPTIONS GENERAL Some socket options can have great impact This is true for bad and good impact Only touch them if you know what they do

Slide 7

Slide 7 text

MOST INTERESTING SOCKET OPTIONS TCP_NO_DELAY SO_SNDBUF SO_RCVBUF

Slide 8

Slide 8 text

GC - PRESSURE “Allocate / Deallocate the shit out of it!”

Slide 9

Slide 9 text

SOLVE GC-PRESSURE Try to minimize allocation / deallocation of objects Use static instances where ever possible Ask yourself do I really need to create the instance BUT, only cache/pool where it makes sense as long-living objects may have bad impact on GC as well “Rule of thumb: use static if it's immutable and used often. If its mutable only pool / cache if allocation costs are high!”

Slide 10

Slide 10 text

GC-PRESSURE “Every time I hear allocation / deallocation of objects is a no-brainer a kitten dies!”

Slide 11

Slide 11 text

GC - PRESSURE “But I never had GC-Pressure ....” “ Well, you not pushed your system hard enough! ”

Slide 12

Slide 12 text

SOURCE OF GC-PRESSURE IN ACTION https://github.com/netty/netty/issues/973 BAD BETTER! c h a n n e l I d l e ( c t x , n e w I d l e S t a t e E v e n t ( I d l e S t a t e . R E A D E R _ I D L E , r e a d e r I d l e C o u n t + + , c u r r e n t T i m e - l a s t R e a d T i m e ) ) ; c h a n n e l I d l e ( c t x , I d l e S t a t e E v e n t . R E A D E R _ I D L E _ E V E N T ) ;

Slide 13

Slide 13 text

GARBAGE-COLLECTOR MATTERS The Garbage-Collector really matters Use a CMS-based collector or G1 if you want high- troughput Size different areas depending on your application / access pattern “Stop-the-world GC is your worst enemy if you want to push data hard”

Slide 14

Slide 14 text

GARBAGE COLLECTOR Tune the GC is kind of an "black art" Be sure you understand what you are doing GC-Tuning params are different per App

Slide 15

Slide 15 text

BUFFERS Allocate / Deallocate from direct buffers is expensive Allocate/ Deallocate from heap buffers is cheap “Free up memory of direct buffers is expensive” “Unfortunately zero-out the byte array of heap buffers is not for free too”

Slide 16

Slide 16 text

BUFFERPOOLING TO THE RESCUE Pool buffers if you need to create a lot of them This is especially true for direct buffers There is also Unsafe.... but its " unsafe " ;)

Slide 17

Slide 17 text

MEMORY FRAGMENTATION Memory fragmentation is bad, as you will waste memory More often GC to remove fragmentation. “Can't insert int here as we need 4 slots!”

Slide 18

Slide 18 text

GATHERING WRITES / SCATTERING READS Use Gathering writes / Scattering reads Especially useful for protocols that can be assembled out of multiple buffers IMPORTANT: Gathering writes only works without memory leak since java7 and late java6.

Slide 19

Slide 19 text

USE DIRECT BUFFERS FOR SOCKETS Use direct buffers when you do operations on sockets “WHY ?” “Internally the JDK* will copy the buffer content to a direct buffer if you not use one”

Slide 20

Slide 20 text

MINIMIZE SYSCALLS Also true for other operations that directly hit the OS Batch up things, but as always there is a tradeoff. “Only call Channel.write(...) / Channel.read(...) if you really need!”

Slide 21

Slide 21 text

MEMORY COPIES ARE NOT FOR FREE USE THEM! “ByteBuffer expose operations like slice(), duplicate() for a good reason”

Slide 22

Slide 22 text

ZERO-MEMORY-COPY A.K.A FILECHANNEL Many Operation Systems support it Helps to write File content to a Channel in an efficient way “Only possible if you not need to transform the data during transfer!”

Slide 23

Slide 23 text

THROTTLE READS / WRITES /ACCEPTS Otherwise you will have fun with OOM interestedOps(..) update to the rescue! This will push the "burden" to the network stack https://github.com/netty/netty/issues/1024 “But not call interestedOps(...) too often, its expensive!”

Slide 24

Slide 24 text

DON'T REGISTER FOR OP_WRITE Don't register for OP_WRITE on the Selector by default Only do if you could not write the complete buffer Remove OP_WRITE from interestedOps() after you was able to write “Remember most of the times the Channel is writable!”

Slide 25

Slide 25 text

DON'T BLOCK! Don't block the Thread that handles the IO You may be surprised what can block “I look at you DNS resolution!” “If you really need to block move it to an extra ThreadPool”

Slide 26

Slide 26 text

BLOCKING IN ACTION “RED != Good!”

Slide 27

Slide 27 text

SELECTIONKEY OPERATIONS SelectionKey.interestedOps(....); “This method may be invoked at any time. Whether or not it blocks, and for how long is implementation-dependent”

Slide 28

Slide 28 text

OPTIMIZE BAD BETTER! p u b l i c v o i d s u s p e n d R e a d ( ) { k e y . i n t e r e s t O p s ( k e y . i n t e r e s t O p s ( ) & ~ O P _ R E A D ) ; } p u b l i c v o i d s u s p e n d R e a d ( ) { i n t o p s = k e y . i n t e r e s t O p s ( ) ; i f ( ( o p s & O P _ R E A D ) ! = 0 ) { k e y . i n t e r e s t O p s ( o p s & ~ O P _ R E A D ) ; } }

Slide 29

Slide 29 text

BE MEMORY EFFICIENT “When write a System that handles 100k of concurrent connections every saved memory count for long-living objects”

Slide 30

Slide 30 text

MEMORY EFFICIENT ATOMIC AtomicReference => AtomicReferenceFieldUpdater AtomicBoolean => AtomicIntegerFieldUpdater AtomicLong => AtomicIntegerFieldUpdater AtomicInteger=> AtomicIntegerFieldUpdater “It's ugly, but sometimes you just have to do it!”

Slide 31

Slide 31 text

DATA-STRUCTURE MATTERS Think about what data-structure fits best Linked vs. Array based Access pattern ?!?

Slide 32

Slide 32 text

VOLATILE Volatile reads are cheap.... But still not for free Cache volatile variables if possible to minimize flushes

Slide 33

Slide 33 text

OPTIMIZE BAD BETTER p r i v a t e v o l a t i l e S e l e c t o r s e l e c t o r ; p u b l i c v o i d m e t h o d ( ) . . . . { s e l e c t o r . s e l e c t ( ) ; . . . . } p r i v a t e v o l a t i l e S e l e c t o r s e l e c t o r ; p u b l i c v o i d m e t h o d ( ) . . . . { S e l e c t o r s e l e c t o r = t h i s . s e l e c t o r ; s e l e c t o r . s e l e c t ( ) ; . . . . }

Slide 34

Slide 34 text

MINIMIZE STACKDEPTH Deep stacks are our enemies, because they are expensive Use tail-recursive calls if possible “WHY?” “Everything that needs to be stored till the call is done needs memory...”

Slide 35

Slide 35 text

USE JDK7 IF POSSIBLE Also has some other goodies like: NIO.2 , UDP Multicast, SCTP “Allocation / Deallocation of ByteBuffers is a lot faster now...”

Slide 36

Slide 36 text

WELL DEFINED THREAD-MODEL It makes development easier Reduce context-switching Reduce the need for synchronization in many cases

Slide 37

Slide 37 text

CHOOSE THE CORRECT PROTOCOL UDP TCP SCTP UDT “It's always a trade-off!”

Slide 38

Slide 38 text

PIPELINING IS AWESOME Allow to send / receive more then one message before response This minimize send / receive operations Popular protocols which support Pipelining: HTTP, SMTP, IMAP “ If you write your own protocol think about Pipelining! ”

Slide 39

Slide 39 text

DON'T WANT TO HASSLE Netty Vert.x Xnio Grizzly Apache Mina “There are a few frameworks to rescue....”