Thomas Ballinger - Terminal whispering

Have you ever wanted to add a status bar to your command line program?
Or maybe color the output a bit? Or do you want to write a fullscreen terminal application like ls, top, vim, or emacs? Then you need to speak a bit of terminal! This talk describes how to talk to your terminal from scratch and goes on to show why Python libraries Blessings and Urwid are so awesome.

https://us.pycon.org/2015/schedule/presentation/401/

Transcript

1. TERMINAL WHISPERING Do more than print text with your terminal | | | Thomas Ballinger @ballingt me@ballingt.com tom@recurse.com
2. TERMINALS ARE COOL
3. WHAT ARE TERMINALS?
4. a p p l i c a t i o n - - - - - - - - m a y b e c o p p e r ? Λ | | | s i g n a l o v e r m e t a l w i r e | | | t e l e t y p e w r i t e r < - - - - - - - - - - - + Λ | | i n k | | | V | p a p e r | f i n g e r s p r e s s i n g k e y s
5. a p p l i c a t i o n - - - - - - - - m a y b e c o p p e r ? Λ | | | s i g n a l o v e r m e t a l w i r e | | | t y p e w r i t e r < - - - - - - - - - - - - - - + Λ | | i n k | | | V | p a p e r | f i n g e r s p r e s s i n g k e y s
6. a p p l i c a t i o n - - - - - - - - m a y b e c o p p e r ? Λ | | | s i g n a l o v e r m e t a l w i r e | | | t e l e t y p e w r i t e r < - - - - - - - - - - - + Λ | | i n k | | | V | p a p e r | f i n g e r s p r e s s i n g k e y s
7. a p p l i c a t i o n - - - - - - - - m a y b e c o p p e r ? Λ | | | s i g n a l o v e r m e t a l w i r e | | | t e l e t y p e w r i t e r < - - - - - - - - - - - + Λ | | i n k | | | V | p a p e r | f i n g e r s p r e s s i n g k e y s
8. a p p l i c a t i o n - - - - - - - w r i t e ( b y t e s ) Λ | | | r e a d ( ) r e t u r n s b y t e s | | | t e r m i n a l < - - - - - - - - - - - - - - - - - + Λ | | + - - G U I s t u f f - - + | | k e y p r e s s e s V | s c r e e n k e y b o a r d
9. a p p l i c a t i o n - - - - - - - p r i n t ( ) - - + Λ | | | i n p u t ( ) | | | t e r m i n a l < - - - - - - - - - - - - - - - - - + Λ | | + - - G U I s t u f f - - + | | k e y p r e s s e s V | s c r e e n k e y b o a r d
10. What bytes can we write and what happens? a p p l i c a t i o n - - - - - - - w r i t e ( b y t e s ) Λ | | | r e a d ( ) r e t u r n s b y t e s | | | t e r m i n a l < - - - - - - - - - - - - - - - - - + Λ | | + - - G U I s t u f f - - + | | k e y p r e s s e s V | s c r e e n k e y b o a r d
11. WHAT BYTES CAN WE SEND? ASCII bytes s y s . s t d o u t . b u f f e r . w r i t e ( b ' s o m e b y t e s ' ) control characters ([ c h r ( x ) f o r x i n r a n g e ( 3 2 ) ] e.g. ' \ a \ n \ r \ v \ b \ f \ t ' )
12. i m p o r t t i m e , s y s d e f r e p o r t _ p r o g r e s s ( r a t i o , w i d t h = 5 0 ) : f i l l e d = ' = ' * i n t ( r a t i o * w i d t h ) r e s t = ' - ' * ( w i d t h - i n t ( r a t i o * w i d t h ) ) s y s . s t d e r r . w r i t e ( ' \ r | ' + f i l l e d + r e s t + ' | ' ) s y s . s t d e r r . f l u s h ( ) f o r i i n r a n g e ( 1 0 1 ) : r e p o r t _ p r o g r e s s ( i / 1 0 0 . 0 , 5 0 ) t i m e . s l e e p ( . 0 1 )
13. WHAT BYTES CAN WE SEND? ASCII bytes s y s . s t d o u t . b u f f e r . w r i t e ( b ' s o m e b y t e s ' ) control characters ([ c h r ( x ) f o r x i n r a n g e ( 3 2 ) ] or ' \ a \ n \ r \ v \ b \ f \ t ' ) if terminal using an encoding, several bytes to make a character: s y s . s t d o u t . b u f f e r . w r i t e ( b ' \ x e 2 \ x 8 8 \ x 8 2 ' ) s y s . s t d o u t . w r i t e ( u ' ∂ ' ) (via . e n c o d i n g )
14. WHAT BYTES CAN WE SEND? ASCII bytes s y s . s t d o u t . b u f f e r . w r i t e ( b ' s o m e b y t e s ' ) control characters ([ c h r ( x ) f o r x i n r a n g e ( 3 2 ) ] or ' \ a \ n \ v \ b \ f \ t ' ) several bytes may encode a single character: s y s . s t d o u t . b u f f e r . w r i t e ( b ' \ x e 2 \ x 8 8 \ x 8 2 ' ) s y s . s t d o u t . w r i t e ( u ' ∂ ' ) (via . e n c o d i n g ) ANSI escape sequences (' \ x 1 b [ 3 1 m ' )
15. ANSI ESCAPE SEQUENCES Sequence Meaning \ x 1 b [ 2 3 A move cursor 23 rows up \ x 1 b [ 2 J clear entire screen \ x 1 b [ ? 2 5 l hide the cursor \ x 1 b [ 1 m start writing in bold \ x 1 b [ 3 1 m start writing in red See for more. Wikipedia entry on ANSI escape sequences
16. COMPATIBILITY CONCERNS termcap (then later terminfo) curses tput ANSI standardization efforts
17. e c h o " I n p u t p l e a s e " t p u t c u u 1 t p u t c u u 1 t p u t e l r e a d a k e y t p u t c l e a r code sample from http://aplawrence.com/Basics/tput_stty.html
18. Blessings - Eric Rose - see "Designing Poetic APIs" from last year f r o m b l e s s i n g s i m p o r t T e r m i n a l t = T e r m i n a l ( ) p r i n t t . b o l d ( ' H i t h e r e ! ' ) p r i n t t . b o l d _ r e d _ o n _ b r i g h t _ g r e e n ( ' I t h u r t s m y e y e s ! ' ) w i t h t . l o c a t i o n ( 0 , t . h e i g h t - 1 ) : p r i n t ' T h i s i s a t t h e b o t t o m . '
19. SECRETS UNLOCKED BY WRITING BYTES TO THE TERMINAL Colored and styled text (bold, underline, blink...) Moving the cursor anywhere Alternate screen Hide cursor etc. (scroll regions, save/restore cursor position)
20. What happens when the user types at the keyboard? a p p l i c a t i o n - - - - - - - w r i t e ( b y t e s ) Λ | | | r e a d ( ) r e t u r n s b y t e s | | | t e r m i n a l < - - - - - - - - - - - - - - - - - + Λ | | + - - G U I s t u f f - - + | | k e y p r e s s e s V | s c r e e n k e y b o a r d
21. What happens when the user types at the keyboard? buffering, line editing discipline data not passed to application until the user hits return backspace, delete word, reprint line, delete line - not much else (try cat to see this behavior) echoing back characters
22. a p p l i c a t i o n - - - - - - - - m a y b e c o p p e r ? Λ | | | s i g n a l o v e r m e t a l w i r e | | | t e l e t y p e w r i t e r < - - - - - - - - - - - + Λ | | i n k | | | V | p a p e r | f i n g e r s p r e s s i n g k e y s
23. + - - - - a p p l i c a t i o n - - - - - - - w r i t e ( b y t e s ) | Λ | g e t | | & r e a d ( ) r e t u r n s b y t e s | s e t | | | | | + - - - ⚒ t e r m i n a l < - - - - - - - - - - - - - - - - - + Λ | | + - - G U I s t u f f - - + | | k e y p r e s s e s V | s c r e e n k e y b o a r d
24. Relevant man pages raw termios ioctl fcntl stty
25. Turning off echo, using stty $ s t t y - e c h o $ e c h o " e c h o i s t u r n e d o f f " $ s t t y e c h o
26. Turning off echo in Python > > > i m p o r t g e t p a s s > > > g e t p a s s . g e t p a s s ( ) P a s s w o r d : ' s e c r e t '
27. Turning off echo in Python from https://docs.python.org/2/library/termios.html d e f g e t p a s s ( p r o m p t = " P a s s w o r d : " ) : i m p o r t t e r m i o s , s y s f d = s y s . s t d i n . f i l e n o ( ) o l d = t e r m i o s . t c g e t a t t r ( f d ) n e w = t e r m i o s . t c g e t a t t r ( f d ) n e w [ 3 ] = n e w [ 3 ] & ~ t e r m i o s . E C H O # l f l a g s t r y : t e r m i o s . t c s e t a t t r ( f d , t e r m i o s . T C S A D R A I N , n e w ) p a s s w d = r a w _ i n p u t ( p r o m p t ) f i n a l l y : t e r m i o s . t c s e t a t t r ( f d , t e r m i o s . T C S A D R A I N , o l d ) r e t u r n p a s s w d
28. i m p o r t s y s , t e r m i o s c l a s s N o E c h o : d e f _ _ i n i t _ _ ( s e l f , i n _ s t r e a m ) : s e l f . f d = i n _ s t r e a m . f i l e n o ( ) d e f _ _ e n t e r _ _ ( s e l f ) : s e l f . o r i g = t e r m i o s . t c g e t a t t r ( s e l f . f d ) n e w = t e r m i o s . t c g e t a t t r ( s e l f . f d ) n e w [ 3 ] = n e w [ 3 ] & ~ t e r m i o s . E C H O t e r m i o s . t c s e t a t t r ( f d , t e r m i o s . T C S A D R A I N , n e w ) d e f _ _ e x i t _ _ ( s e l f , e x c _ t y p e , e x c _ v a l u e , t r a c e b a c k ) : t e r m i o s . t c s e t a t t r ( s e l f . f d ) , t e r m i o s . T C S A D R A I N , s e l f . o r i g )
29. w i t h N o E c h o ( s y s . s t d i n ) : h = s y s . s t d i n . r e a d l i n e ( ) p r i n t ( h )
30. SECRET TECHNIQUES MADE POSSIBLE WITH THESE DIALS turn off echo send keys immediately current terminal size nonblocking input
31. What happens when the user types at the keyboard buffering, line editing discipline data not passed to application until the user hits return backspace, delete word, reprint line, delete line - not much else to see this, try cat (it doesn't use readline) echoing back characters mapping bytes to keys
32. EXAMPLE BYTES/KEY MAPPINGS with my particular setup Bytes read Key presses b ' a ' a b ' \ x 1 b [ 2 3 ~ ' F11 b ' \ x 1 b [ Z ' Shift-Tab b ' \ x 1 b [ 1 ; 1 0 B ' Meta-Shift-Down b ' \ x 1 b [ 1 ~ ' Home
33. What happens when the user types at the keyboard buffering, line editing discipline data not passed to application until the user hits return backspace, delete word, reprint line, delete line - not much else to see this, try cat (it doesn't use readline) echoing back characters mapping bytes to keys bytes sent without being typed (cursor query)
34. > > > p r i n t ( " \ x 1 b [ 6 n " ) + - - - - a p p l i c a t i o n - - - - - - - w r i t e ( b y t e s ) | Λ | g e t | | & r e a d ( ) r e t u r n s b y t e s | s e t | | | | | + - - - ⚒ t e r m i n a l < - - - - - - - - - - - - - - - - - + Λ | | + - - G U I s t u f f - - + | | k e y p r e s s e s V | s c r e e n k e y b o a r d
35. > > > p r i n t ( " \ x 1 b [ 6 n " )
36. > > > p r i n t ( " \ x 1 b [ 6 n " ) > > > 1 R
37. > > > p r i n t ( " \ x 1 b [ 6 n " ) > > > E S C [ 3 ; 1 R
38. SIGNALS
39. How can the terminal notify the application? + - - - - a p p l i c a t i o n - - - - - - - w r i t e ( b y t e s ) | Λ ⚡ | g e t | ⚡ | & r e a d ( ) ⚡s i g n a l s ! | s e t | ⚡ | | | ⚡ | + - - - ⚒ t e r m i n a l < - - - - - - - - - - - - - - - - - + Λ | | + - - G U I s t u f f - - + | | k e y p r e s s e s V | s c r e e n k e y b o a r d
40. Unix signals triggered from the terminal SIGINT SIGTSTP SIGWINCH tons of others, see The TTY demystified
41. COOKED, RAW AND CBREAK MODES
42. FANCY TERMINAL TRICKS IN PRACTICE
43. TOOLS Blessings - Eric Rose - see "Designing Poetic APIs" from last year Urwid - Ian Ward - a few examples Click - Armin Ronacher Clint - Kenneth Reitz Python Prompt Toolkit - Jonathan Slenders Curtsies - me!
44. BLESSINGS
45. URWID
46. CLICK, CLINT
47. CURTSIES
48. PYTHON PROMPT TOOLKIT
49. TOOLS Blessings - Eric Rose - see "Designing Poetic APIs" from last year Urwid - Ian Ward - a few examples Click - Armin Ronacher Clint - Kenneth Reitz Curtsies - me! Python Prompt Toolkit - Jonathan Slenders
50. RANDOM COOL STUFF termcast asciinema tests I've written
51. $ t e l n e t t e r m c a s t . o r g https://github.com/doy/python-termcast-server https://github.com/doy/python-termcast-client
52. demo of displaying test failures
53. FURTHER READING: Terminals, shells, tty, console The TTY Demystified FUN VIDEOS http://utcc.utoronto.ca/~cks/space/blog/unix/TypingEOFEffe Loading 4K BASIC with a Teletype Teletype Model 35ASR How The Teleprinter Works- 1940 IMAGE CREDITS Wordcloud credit: https://www.jasondavies.com/wordcloud
54. THANKS FOR LISTENING! I'll be tweeting terminal tips from @ballingt I'll announce when the slides are up on Twitter as well I'll be at the first day of sprints, happy to work on bpython if anyone wants