Thomas Ballinger - Terminal whispering

Thomas Ballinger - Terminal whispering

Have you ever wanted to add a status bar to your command line program?
Or maybe color the output a bit? Or do you want to write a fullscreen terminal application like ls, top, vim, or emacs? Then you need to speak a bit of terminal! This talk describes how to talk to your terminal from scratch and goes on to show why Python libraries Blessings and Urwid are so awesome.

https://us.pycon.org/2015/schedule/presentation/401/

D5710b3bca38f1233274b4cbc523dc4b?s=128

PyCon 2015

April 18, 2015
Tweet

Transcript

  1. TERMINAL WHISPERING Do more than print text with your terminal

    | | | Thomas Ballinger @ballingt me@ballingt.com tom@recurse.com
  2. TERMINALS ARE COOL

  3. WHAT ARE TERMINALS?

  4. a p p l i c a t i o

    n - - - - - - - - m a y b e c o p p e r ? Λ | | | s i g n a l o v e r m e t a l w i r e | | | t e l e t y p e w r i t e r < - - - - - - - - - - - + Λ | | i n k | | | V | p a p e r | f i n g e r s p r e s s i n g k e y s
  5. a p p l i c a t i o

    n - - - - - - - - m a y b e c o p p e r ? Λ | | | s i g n a l o v e r m e t a l w i r e | | | t y p e w r i t e r < - - - - - - - - - - - - - - + Λ | | i n k | | | V | p a p e r | f i n g e r s p r e s s i n g k e y s
  6. a p p l i c a t i o

    n - - - - - - - - m a y b e c o p p e r ? Λ | | | s i g n a l o v e r m e t a l w i r e | | | t e l e t y p e w r i t e r < - - - - - - - - - - - + Λ | | i n k | | | V | p a p e r | f i n g e r s p r e s s i n g k e y s
  7. a p p l i c a t i o

    n - - - - - - - - m a y b e c o p p e r ? Λ | | | s i g n a l o v e r m e t a l w i r e | | | t e l e t y p e w r i t e r < - - - - - - - - - - - + Λ | | i n k | | | V | p a p e r | f i n g e r s p r e s s i n g k e y s
  8. a p p l i c a t i o

    n - - - - - - - w r i t e ( b y t e s ) Λ | | | r e a d ( ) r e t u r n s b y t e s | | | t e r m i n a l < - - - - - - - - - - - - - - - - - + Λ | | + - - G U I s t u f f - - + | | k e y p r e s s e s V | s c r e e n k e y b o a r d
  9. a p p l i c a t i o

    n - - - - - - - p r i n t ( ) - - + Λ | | | i n p u t ( ) | | | t e r m i n a l < - - - - - - - - - - - - - - - - - + Λ | | + - - G U I s t u f f - - + | | k e y p r e s s e s V | s c r e e n k e y b o a r d
  10. What bytes can we write and what happens? a p

    p l i c a t i o n - - - - - - - w r i t e ( b y t e s ) Λ | | | r e a d ( ) r e t u r n s b y t e s | | | t e r m i n a l < - - - - - - - - - - - - - - - - - + Λ | | + - - G U I s t u f f - - + | | k e y p r e s s e s V | s c r e e n k e y b o a r d
  11. WHAT BYTES CAN WE SEND? ASCII bytes s y s

    . s t d o u t . b u f f e r . w r i t e ( b ' s o m e b y t e s ' ) control characters ([ c h r ( x ) f o r x i n r a n g e ( 3 2 ) ] e.g. ' \ a \ n \ r \ v \ b \ f \ t ' )
  12. i m p o r t t i m e

    , s y s d e f r e p o r t _ p r o g r e s s ( r a t i o , w i d t h = 5 0 ) : f i l l e d = ' = ' * i n t ( r a t i o * w i d t h ) r e s t = ' - ' * ( w i d t h - i n t ( r a t i o * w i d t h ) ) s y s . s t d e r r . w r i t e ( ' \ r | ' + f i l l e d + r e s t + ' | ' ) s y s . s t d e r r . f l u s h ( ) f o r i i n r a n g e ( 1 0 1 ) : r e p o r t _ p r o g r e s s ( i / 1 0 0 . 0 , 5 0 ) t i m e . s l e e p ( . 0 1 )
  13. WHAT BYTES CAN WE SEND? ASCII bytes s y s

    . s t d o u t . b u f f e r . w r i t e ( b ' s o m e b y t e s ' ) control characters ([ c h r ( x ) f o r x i n r a n g e ( 3 2 ) ] or ' \ a \ n \ r \ v \ b \ f \ t ' ) if terminal using an encoding, several bytes to make a character: s y s . s t d o u t . b u f f e r . w r i t e ( b ' \ x e 2 \ x 8 8 \ x 8 2 ' ) s y s . s t d o u t . w r i t e ( u ' ∂ ' ) (via . e n c o d i n g )
  14. WHAT BYTES CAN WE SEND? ASCII bytes s y s

    . s t d o u t . b u f f e r . w r i t e ( b ' s o m e b y t e s ' ) control characters ([ c h r ( x ) f o r x i n r a n g e ( 3 2 ) ] or ' \ a \ n \ v \ b \ f \ t ' ) several bytes may encode a single character: s y s . s t d o u t . b u f f e r . w r i t e ( b ' \ x e 2 \ x 8 8 \ x 8 2 ' ) s y s . s t d o u t . w r i t e ( u ' ∂ ' ) (via . e n c o d i n g ) ANSI escape sequences (' \ x 1 b [ 3 1 m ' )
  15. ANSI ESCAPE SEQUENCES Sequence Meaning \ x 1 b [

    2 3 A move cursor 23 rows up \ x 1 b [ 2 J clear entire screen \ x 1 b [ ? 2 5 l hide the cursor \ x 1 b [ 1 m start writing in bold \ x 1 b [ 3 1 m start writing in red See for more. Wikipedia entry on ANSI escape sequences
  16. COMPATIBILITY CONCERNS termcap (then later terminfo) curses tput ANSI standardization

    efforts
  17. e c h o " I n p u t

    p l e a s e " t p u t c u u 1 t p u t c u u 1 t p u t e l r e a d a k e y t p u t c l e a r code sample from http://aplawrence.com/Basics/tput_stty.html
  18. Blessings - Eric Rose - see "Designing Poetic APIs" from

    last year f r o m b l e s s i n g s i m p o r t T e r m i n a l t = T e r m i n a l ( ) p r i n t t . b o l d ( ' H i t h e r e ! ' ) p r i n t t . b o l d _ r e d _ o n _ b r i g h t _ g r e e n ( ' I t h u r t s m y e y e s ! ' ) w i t h t . l o c a t i o n ( 0 , t . h e i g h t - 1 ) : p r i n t ' T h i s i s a t t h e b o t t o m . '
  19. SECRETS UNLOCKED BY WRITING BYTES TO THE TERMINAL Colored and

    styled text (bold, underline, blink...) Moving the cursor anywhere Alternate screen Hide cursor etc. (scroll regions, save/restore cursor position)
  20. What happens when the user types at the keyboard? a

    p p l i c a t i o n - - - - - - - w r i t e ( b y t e s ) Λ | | | r e a d ( ) r e t u r n s b y t e s | | | t e r m i n a l < - - - - - - - - - - - - - - - - - + Λ | | + - - G U I s t u f f - - + | | k e y p r e s s e s V | s c r e e n k e y b o a r d
  21. What happens when the user types at the keyboard? buffering,

    line editing discipline data not passed to application until the user hits return backspace, delete word, reprint line, delete line - not much else (try cat to see this behavior) echoing back characters
  22. a p p l i c a t i o

    n - - - - - - - - m a y b e c o p p e r ? Λ | | | s i g n a l o v e r m e t a l w i r e | | | t e l e t y p e w r i t e r < - - - - - - - - - - - + Λ | | i n k | | | V | p a p e r | f i n g e r s p r e s s i n g k e y s
  23. + - - - - a p p l i

    c a t i o n - - - - - - - w r i t e ( b y t e s ) | Λ | g e t | | & r e a d ( ) r e t u r n s b y t e s | s e t | | | | | + - - - ⚒ t e r m i n a l < - - - - - - - - - - - - - - - - - + Λ | | + - - G U I s t u f f - - + | | k e y p r e s s e s V | s c r e e n k e y b o a r d
  24. Relevant man pages raw termios ioctl fcntl stty

  25. Turning off echo, using stty $ s t t y

    - e c h o $ e c h o " e c h o i s t u r n e d o f f " $ s t t y e c h o
  26. Turning off echo in Python > > > i m

    p o r t g e t p a s s > > > g e t p a s s . g e t p a s s ( ) P a s s w o r d : ' s e c r e t '
  27. Turning off echo in Python from https://docs.python.org/2/library/termios.html d e f

    g e t p a s s ( p r o m p t = " P a s s w o r d : " ) : i m p o r t t e r m i o s , s y s f d = s y s . s t d i n . f i l e n o ( ) o l d = t e r m i o s . t c g e t a t t r ( f d ) n e w = t e r m i o s . t c g e t a t t r ( f d ) n e w [ 3 ] = n e w [ 3 ] & ~ t e r m i o s . E C H O # l f l a g s t r y : t e r m i o s . t c s e t a t t r ( f d , t e r m i o s . T C S A D R A I N , n e w ) p a s s w d = r a w _ i n p u t ( p r o m p t ) f i n a l l y : t e r m i o s . t c s e t a t t r ( f d , t e r m i o s . T C S A D R A I N , o l d ) r e t u r n p a s s w d
  28. i m p o r t s y s ,

    t e r m i o s c l a s s N o E c h o : d e f _ _ i n i t _ _ ( s e l f , i n _ s t r e a m ) : s e l f . f d = i n _ s t r e a m . f i l e n o ( ) d e f _ _ e n t e r _ _ ( s e l f ) : s e l f . o r i g = t e r m i o s . t c g e t a t t r ( s e l f . f d ) n e w = t e r m i o s . t c g e t a t t r ( s e l f . f d ) n e w [ 3 ] = n e w [ 3 ] & ~ t e r m i o s . E C H O t e r m i o s . t c s e t a t t r ( f d , t e r m i o s . T C S A D R A I N , n e w ) d e f _ _ e x i t _ _ ( s e l f , e x c _ t y p e , e x c _ v a l u e , t r a c e b a c k ) : t e r m i o s . t c s e t a t t r ( s e l f . f d ) , t e r m i o s . T C S A D R A I N , s e l f . o r i g )
  29. w i t h N o E c h o

    ( s y s . s t d i n ) : h = s y s . s t d i n . r e a d l i n e ( ) p r i n t ( h )
  30. SECRET TECHNIQUES MADE POSSIBLE WITH THESE DIALS turn off echo

    send keys immediately current terminal size nonblocking input
  31. What happens when the user types at the keyboard buffering,

    line editing discipline data not passed to application until the user hits return backspace, delete word, reprint line, delete line - not much else to see this, try cat (it doesn't use readline) echoing back characters mapping bytes to keys
  32. EXAMPLE BYTES/KEY MAPPINGS with my particular setup Bytes read Key

    presses b ' a ' a b ' \ x 1 b [ 2 3 ~ ' F11 b ' \ x 1 b [ Z ' Shift-Tab b ' \ x 1 b [ 1 ; 1 0 B ' Meta-Shift-Down b ' \ x 1 b [ 1 ~ ' Home
  33. What happens when the user types at the keyboard buffering,

    line editing discipline data not passed to application until the user hits return backspace, delete word, reprint line, delete line - not much else to see this, try cat (it doesn't use readline) echoing back characters mapping bytes to keys bytes sent without being typed (cursor query)
  34. > > > p r i n t ( "

    \ x 1 b [ 6 n " ) + - - - - a p p l i c a t i o n - - - - - - - w r i t e ( b y t e s ) | Λ | g e t | | & r e a d ( ) r e t u r n s b y t e s | s e t | | | | | + - - - ⚒ t e r m i n a l < - - - - - - - - - - - - - - - - - + Λ | | + - - G U I s t u f f - - + | | k e y p r e s s e s V | s c r e e n k e y b o a r d
  35. > > > p r i n t ( "

    \ x 1 b [ 6 n " )
  36. > > > p r i n t ( "

    \ x 1 b [ 6 n " ) > > > 1 R
  37. > > > p r i n t ( "

    \ x 1 b [ 6 n " ) > > > E S C [ 3 ; 1 R
  38. SIGNALS

  39. How can the terminal notify the application? + - -

    - - a p p l i c a t i o n - - - - - - - w r i t e ( b y t e s ) | Λ ⚡ | g e t | ⚡ | & r e a d ( ) ⚡s i g n a l s ! | s e t | ⚡ | | | ⚡ | + - - - ⚒ t e r m i n a l < - - - - - - - - - - - - - - - - - + Λ | | + - - G U I s t u f f - - + | | k e y p r e s s e s V | s c r e e n k e y b o a r d
  40. Unix signals triggered from the terminal SIGINT SIGTSTP SIGWINCH tons

    of others, see The TTY demystified
  41. COOKED, RAW AND CBREAK MODES

  42. FANCY TERMINAL TRICKS IN PRACTICE

  43. TOOLS Blessings - Eric Rose - see "Designing Poetic APIs"

    from last year Urwid - Ian Ward - a few examples Click - Armin Ronacher Clint - Kenneth Reitz Python Prompt Toolkit - Jonathan Slenders Curtsies - me!
  44. BLESSINGS

  45. URWID

  46. CLICK, CLINT

  47. CURTSIES

  48. PYTHON PROMPT TOOLKIT

  49. TOOLS Blessings - Eric Rose - see "Designing Poetic APIs"

    from last year Urwid - Ian Ward - a few examples Click - Armin Ronacher Clint - Kenneth Reitz Curtsies - me! Python Prompt Toolkit - Jonathan Slenders
  50. RANDOM COOL STUFF termcast asciinema tests I've written

  51. $ t e l n e t t e r

    m c a s t . o r g https://github.com/doy/python-termcast-server https://github.com/doy/python-termcast-client
  52. demo of displaying test failures

  53. FURTHER READING: Terminals, shells, tty, console The TTY Demystified FUN

    VIDEOS http://utcc.utoronto.ca/~cks/space/blog/unix/TypingEOFEffe Loading 4K BASIC with a Teletype Teletype Model 35ASR How The Teleprinter Works- 1940 IMAGE CREDITS Wordcloud credit: https://www.jasondavies.com/wordcloud
  54. THANKS FOR LISTENING! I'll be tweeting terminal tips from @ballingt

    I'll announce when the slides are up on Twitter as well I'll be at the first day of sprints, happy to work on bpython if anyone wants