Server Survival

A guide through the annoying parts of servers. This includes slides I couldn't make it through in the actual presentation.

Transcript

1. Server Survival hi! @fideloper serversforhackers.com
2. Server Survival Server Survival
3. Goal: (don’t memorize) understand
4. Programming Stuff Server Mechanics Semi-Ridiculous Chart of Learning Curves
6. annoying server things security supervision network permissions
7. Security enjoying your new server responsibly
9. Protection Process Compliance Security “Levels”
10. Protection
11. access network Security “Levels” basics
12. Security “Levels” don’t be root don’t (only) use passwords user security
13. user & access new user new ssh key $ sudo adduser fideloper $ sudo usermod -a -G sudo fideloper $ ssh-keygen -t rsa -b 4096
14. $ ssh-keygen -t rsa -b 4096 \ -f id_whatever $ ssh-copy-id -i ~/.ssh/id_whatever \ fideloper@<server-ip> (added to ~/.ssh/authorized_keys file) user & access
15. /etc/ssh/sshd_config Port 22 (or) 1234 PermitRootLogin no (or) without-password PasswordAuthentication no AllowGroups some-group ssh access ($ sudo service ssh restart)
16. firewall network
17. sudo iptables -L -v firewall
18. firewall sudo iptables -A INPUT -i lo \ -j ACCEPT sudo iptables -A INPUT -m conntrack \ —ctstate RELATED,ESTABLISHED -j ACCEPT
19. firewall sudo iptables -A INPUT -p tcp --dport 22 \ -j ACCEPT sudo iptables -A INPUT -p tcp --dport 80 \ -j ACCEPT sudo iptables -A INPUT -p tcp --dport 443 \ -j ACCEPT sudo iptables -A INPUT -j DROP
20. firewall
21. firewall drop v reject default policy $ iptables … -j REJECT $ iptables \ --policy INPUT DROP
22. (note: services) $ cat /etc/services | grep http http 80/tcp https 443/tcp http-alt 8080/tcp
23. firewall sudo apt-get install -y \ iptables-persistent sudo service \ iptables-persistent save
24. firewall sudo iptables-save > rules.v4 cat rules.v4 | iptables-restore
25. firewall sudo ip6tables -L -v sudo ip6tables-save > rules.v6 echo rules.v6 | ip6tables-restore
26. $ sudo apt-get install -y \ fail2ban fail2ban 1. monitors logs 2. bans IPs
27. fail2ban
28. auto upgrades APT::Periodic::Unattended-Upgrade "1"; Unattended-Upgrade::Allowed-Origins { "${distro_id}:${distro_codename}-security"; }; Unattended-Upgrade::InstallOnShutdown "false"; Unattended-Upgrade::Automatic-Reboot "false"; $ sudo apt-get install -y \ unattended-upgrades files: /etc/apt/apt.conf.d
29. ¡more! There’s always more SELinux / AppArmor 2FA for SSH Securing “secrets” (.env) Strong PW Enforcement (But don’t freak out about it)
30. Process
31. policy legit, send me that password, kthnx. Hi! I’m *totally*
32. policy Define what & how you’re able to send to people.
33. policy Define what happens when people leave.
34. policy Define what happens when new people come.
35. policy Decide on “key rotation” (and similar access changes)
36. policy policy + automation = time =
37. auditing aggregate logs ()
39. Compliance
40. Security “Levels” regulation • HIPPA/HITECH (health) • PCI (ecommerce/credit cards) • FERPA (education) • Many, many more
41. Security “Levels” regulation • Audits • Paper work • And general security
42. how far to go? “what should I care about?”
43. supervision
44. fid@host:~# sudo systemctl status ssh systemctl status <service> systemctl start <service> systemctl stop <service> systemctl enable <service> systemctl disable <service> systemd
45. fid@host:~# sudo service ssh status • ssh.service - OpenBSD Secure Shell server Loaded: loaded (/lib/systemd/system/ssh.service; enabled; \ vendor preset: enabled) Active: active (running) since Fri 2016-07-22 19:46:40 EDT; 1h 27min ago Main PID: 2493 (sshd) CGroup: /system.slice/ssh.service ├─ 2493 /usr/sbin/sshd -D ├─14218 sshd: root [priv] └─14219 sshd: root [net] Jul 22 21:13:28 host sshd[14114]: Accepted password for root from 76.185.167.253 port 56786 ssh2 Jul 22 21:13:28 host sshd[14114]: pam_unix(sshd:session): session opened for user root by (uid=0) systemd
46. systemd [Unit] Description=OpenBSD Secure Shell server After=network.target auditd.service ConditionPathExists=!/etc/ssh/sshd_not_to_be_run [Service] EnvironmentFile=-/etc/default/ssh ExecStart=/usr/sbin/sshd -D $SSHD_OPTS ExecReload=/bin/kill -HUP $MAINPID KillMode=process Restart=on-failure RestartPreventExitStatus=255 Type=notify [Install] WantedBy=multi-user.target Alias=sshd.service /lib/systemd/system/ssh.service
47. systemd [Unit] Description=Laravel Queue [Service] User=forge Group=forge Restart=on-failure WorkingDirectory=/home/user/forge/mysite.com/current ExecStart=/usr/bin/php artisan queue:work --daemon [Install] WantedBy=multi-user.target /lib/systemd/system/laravel.service
48. fid:~# sudo systemctl enable laravel fid:~# sudo systemctl start laravel fid:~# sudo systemctl status laravel systemd
49. supervisord fid@spr:~# sudo apt-get install -y supervisor fid@spr:~# sudo systemctl start supervisor
50. supervisord fid@spr:~# sudo systemctl status supervisor • supervisor.service - Supervisor process control system for UNIX Loaded: loaded (/lib/systemd/system/supervisor.service; disabled; vendor preset: enabled) Active: active (running) since Tue 2016-07-26 17:13:54 EDT; 3s ago Docs: http://supervisord.org Main PID: 3712 (supervisord) Tasks: 1 Memory: 11.1M CPU: 216ms CGroup: /system.slice/supervisor.service !"3712 /usr/bin/python /usr/bin/supervisord -n -c / etc/supervisor/supervisord.conf Jul 26 17:13:54 spr systemd[1]: Started Supervisor process control system for UNIX.
51. [program:lara_queue] command=php artisan queue:work --daemon directory=/home/forge/app.com/current autostart=true autorestart=true startretries=3 redirect_stderr=true stdout_logfile=/home/forge/…/logs/queue.log user=forge numproc=4 supervisord /etc/supervisor/conf.d/lara_q.conf
52. forge: supervisord
53. forge: supervisord any old daemon
54. Network
55. ifconfig
56. ifconfig private network f@db:~$ ifconfig eth0 Link encap:Ethernet HWaddr 04:01:31:20:63:01 inet addr:162.243.164.216 Bcast:162.243.164.255 Mask:255.255.255.0 inet6 addr: fe80::601:31ff:fe20:6301/64 Scope:Link … eth1 Link encap:Ethernet HWaddr 04:01:31:20:63:02 inet addr:10.136.11.155 Bcast:10.136.255.255 Mask:255.255.0.0 inet6 addr: fe80::601:31ff:fe20:6302/64 Scope:Link … lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host …
57. network binding a tcp socket: <ip-address>:<port> a unix socket: unix://path/to/file.sock
58. network binding forge@site:~$ netstat -ap | grep http tcp 0 0 *:http *:* LISTEN 3797/nginx: worker tcp 0 0 *:https *:* LISTEN 3797/nginx: worker
59. network binding
60. network: mysql unix socket
61. tcp socket network: mysql
62. localhost != 127.0.0.1 (in mysql) network: mysql
63. network: mysql # # Instead of skip-networking the default is now to listen only on # localhost which is more compatible and is not less secure. bind-address = 10.136.11.155 f@db:~$ mysql -h localhost -u root -p Enter password: Welcome to the MySQL monitor. Commands end with ; or \g. f@db:~$ mysql -h 127.0.0.1 -u root -p Enter password: ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ✅
64. network: mysql f@db:~$ mysql -h 10.136.11.155 -u root -p Enter password: ERROR 1130 (HY000): Host '10.136.11.155' is not allowed to connect to this MySQL server f@db:~$ mysql -u root -p -e "create user root@'10.136.11.155' identified by 'root';" Enter password: f@db:~$ mysql -h 10.136.11.155 -u root -p Enter password: Welcome to the MySQL monitor. Commands end with ; or \g. ✅
65. network but normally it is
66. 1.Find networks (interfaces) 2.Learn about socket types 3.See examples of mysql 4.Future: Permissions in Forge network review
67. Permissions
68. who can do things user - file/dir owner group - file/dir group - shared permissions! other - anyone else
69. what can they do read - read file, list directory write - write to file, add new file/dir execute - execute command, cd into
70. usr@hst:~$ chown -R www-data:www-data \ /var/www/example.com usr@hst:~$ chmod -R u=rwx,g=rx,o=rx \ /var/www/example.com usr@hst:~$ chmod -R u=rwx,go=rx \ /var/www/example.com usr@hst:$ chmod ug+x,o-x \ /var/www/example.com/artisan setting permissions
71. d rwx r-x r-x dir user group other permissions
72. - rwx r-x r-x file user group other permissions
73. user@host:/var/www$ ls -lAh total 4.0K drwxrwxr-x 2 deploy www-data 4.0K Jul 10 21:43 example.com example d rwx rwx r-x deploy : www-data
74. usr@host:/var/www$ ps axf o pid,user,group,comm \ | grep -i '[n]ginx\|[p]hp' 4290 root root nginx 4291 www-data www-data \_ nginx 2887 root root php-fpm7.0 2889 www-data www-data \_ php-fpm7.0 2890 www-data www-data \_ php-fpm7.0 not just files
75. user@host:/var/www$ ls -lAh total 4.0K drwxrwxr-x 2 deploy www-data 4.0K Jul 10 21:43 example.com php + web files <?php // run as user and group www-data file_put_contents( '/var/www/example.com/new-file.txt', 'Here is a new line' ); // ✅
76. remember files owned by www-data then run php as www-data $ sudo -u www-data php artisan foo:bar
77. Just Works™
78. web files there’s no place like forge@host:~/store.helpspot.com/current$ ls -lAh drwxrwxr-x 15 forge forge app -rwxrwxr-x 1 forge forge artisan drwxrwxr-x 3 forge forge bootstrap -rw-rw-r-- 1 forge forge composer.json -rw-rw-r-- 1 forge forge composer.lock
79. // File /etc/nginx/nginx.conf user forge; worker_processes auto; pid /run/nginx.pid; events { … } http { … } nginx
80. // File /etc/php5/fpm/pool.d/www.conf listen = /var/run/php5-fpm.sock listen.owner = www-data listen.group = www-data listen.mode = 0666 user = forge group = forge -rw-rw-rw php-fpm
81. // File /etc/php5/fpm/pool.d/www.conf listen = 127.0.0.1:9000 listen.owner = www-data listen.group = www-data listen.mode = 0666 user = forge group = forge php-fpm
82. web files there’s no place like forge@host:~/store.helpspot.com/current$ ls -lAh drwxrwxr-x 15 forge forge app -rwxrwxr-x 1 forge forge artisan drwxrwxr-x 3 forge forge bootstrap -rw-rw-r-- 1 forge forge composer.json -rw-rw-r-- 1 forge forge composer.lock
83. There’s More! ACLs
84. ACL
85. ACL ACL Defaults
86. $ sudo setfacl -Rm \ > g:www-data:rwx,d:g:www-data:rwx \ > /var/www/html ACL
87. ACL Owned by root Group www-data:rwx
88. ACL User-based!
89. Server Survival thanks! @fideloper serversforhackers.com
90. pkg managers 1. searching 2. installing
91. pkg managers apt-get & apt sudo apt-get update sudo apt update sudo apt-get install whatever sudo apt install whatever
92. pkg managers search sudo apt search mysql-server ubuntu@host:~$ apt search mysql-server mysql-server/trusty-updates,trusty-security 5.5.49-0… mysql-server-5.5/trusty-updates,trusty-security MySQL database server binaries and system database setup mysql-server-5.6/trusty-updates,trusty-security MySQL database server binaries and system database setup
93. pkg managers show sudo apt show -a \ mysql-server-5.6 Package: mysql-server-5.6 Version: 5.6.30-0ubuntu0.14.04.1 Package: mysql-server-5.6 Version: 5.6.16-1~exp1
94. pkg managers policy sudo apt-cache policy \ mysql-server-5.6 mysql-server-5.6: Installed: (none) Candidate: 5.6.30-0ubuntu0.14.04.1 Version table: 5.6.30-0ubuntu0.14.04.1 0 500 http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ trusty-updates/universe amd64 Packages 500 http://security.ubuntu.com/ubuntu/ trusty-security/universe amd64 Packages 5.6.16-1~exp1 0 500 http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ trusty/universe amd64 Packages
95. pkg managers package=version sudo apt-get install \ mysql-server-5.6=5.6.16-1~exp1
96. pkg managers repositories sudo add-apt-repository \ ppa:ondrej/php
97. pkg managers ubuntu@host: /etc/apt/sources.list.d $ ls -lah -rw-r--r-- 1 root root ondrej-ubuntu-php-xenial.list ubuntu@host: /etc/apt/sources.list.d $ cat \ ondrej-ubuntu-php-xenial.list deb http://ppa.launchpad.net/ondrej/php/ubuntu xenial main # deb-src http://ppa.launchpad.net/ondrej/php/ubuntu xenial main repositories
98. pkg managers ubuntu@host: ~ sudo apt-key adv --recv-keys --keyserver \ hkp://keyserver.ubuntu.com:80 0xF1656F24C74CD1D8 ubuntu@host: ~ echo 'deb http://ftp.utexas.edu/mariadb/ repo/10.1/ubuntu xenial main' \ | sudo tee /etc/apt/sources.list.d/mariadb.list manual install
99. pkg managers ubuntu@host: /etc/apt $ vim sources.list # See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to # newer versions of the distribution. deb http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ xenial main restricted deb-src http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ xenial main restricted # # Major bug fix updates produced after the final release of the # # distribution. deb http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ xenial-updates main restricted deb-src http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ xenial-updates main \ restricted # # N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu # # team. Also, please note that software in universe WILL NOT receive any # # review or updates from the Ubuntu security team. deb http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ xenial universe deb-src http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ xenial universe included repositories
100. pkg managers repositories
101. –@fideloper “Am I done now?”
102. DNS & Domains
103. DNS: ¯\_(ツ)_/¯ ~~ just *TRY* ~~ to coherently explain controlling domains to the average [non-tech-client-whoever] challenge:
104. ¯\_(ツ)_/¯ (we’ll ignore the seedy secondary market of domain-squatting asshats)
105. you bought a domain DNS
106. …but you control it somewhere else DNS
107. DNS “somewhere else” points [sub]domains… somewhere else…
108. this slide intentionally left blank
109. dig
110. dig mx records
111. nslookup mx records again
112. Domains + Web Server
113. Domains + Web Server
114. set host header
115. default_server
116. (set host header again)
117. meanwhile, fideloper.com
118. Logs
119. /var/log/*
120. wtf, Vagrant? less terrible.
121. fideloper@host  ~  vagrant box Usage: vagrant box <subcommand> [<args>] Available subcommands: add list outdated remove repackage update boxes (servers)
122. where’s my stuff? ~/.vagrant.d/boxes ~/.vagrant.d/tmp C:/Users/[USER]/.vagrant.d/boxes C:/Users/[USER]/.vagrant.d/tmp
123. • ssh by default • can add your own (but we’ll do better) port forwarding SSH Default
124. port forwarding $ vagrant ssh-config Just like ~/.ssh/config
125. port forwarding $ ssh \ > -i /Users/fideloper/…/virtualbox/private_key \ > -p 2222 \ > vagrant@localhost
126. port forwarding
127. port forwarding (aside: It’s common to forward to port 80) config.vm.network "forwarded_port", guest: 80, host: 8000 $> curl -I localhost:8000 HTTP/1.1 302 Found Server: nginx/1.9.9 Content-Type: text/html; charset=UTF-8 Date: Sat, 02 Jul 2016 17:57:49 GMT Location: http://localhost:8000/login
128. port forwarding But, two boxes can’t forward to same port! config.vm.network "forwarded_port", guest: 80, host: 8000 config.vm.network "forwarded_port", guest: 80, host: 8888 ✅ first box: second box:
129. sequel pro 2.5 Ways to Connect to MySQL (without *any* MySQL configuration)
130. sequel pro 1 - Port Forward config.vm.network "forwarded_port", guest: 3306, host: 33060
131. sequel pro config.vm.network "forwarded_port", guest: 3306, host: 33060
132. sequel pro 2 - SSH Tunnel $> ssh -p 2222 \ -i /Users/fideloper/…/virtualbox/private_key \ -L 3306:localhost:3306 vagrant@localhost
133. sequel pro
134. sequel pro 2.5 - SSH Tunnel
135. • 1. Port forwarding (homestead way - easy) • 2. Manual SSH tunnel • 3. Sequel Pro SSH Tunnel
136. sequel pro Remember the SSH Tunnel! You can use it in production to view a database.
137. file sharing config.vm.synced_folder “~/Sites", "/home/vagrant/Sites" default file share slow with a large # files
138. file sharing config.vm.synced_folder “~/Sites", "/home/vagrant/Sites", id: "core", :nfs => true, :mount_options => [‘nolock,vers=3,udp,noatime,actimeo=2,fsc'] network file share handles large # files better
139. file sharing where to run build steps? (especially ones that watch files)
140. file sharing (I’ve actually used Docker for this instead) docker run --rm \ -v ~/Sites/some-project:/opt \ some_node_img:latest \ gulp watch
141. adding projects How I made adding a new project painless (and stopped editing /etc/hosts)
142. adding projects An annoying process:
143. adding projects 1. Share More Files: config.vm.synced_folder "~/Sites/a", "/var/www/a" config.vm.synced_folder "~/Sites/b", “/var/www/b"
144. adding projects 2. Create another server config vagrant@vagrant:/etc/nginx/sites-available$ sudo cp \ laravel-a laravel-b vagrant@vagrant:/etc/nginx/sites-available$ sudo vim \ laravel-b server { listen 80; server_name laravel-b.dev; …
145. adding projects 3. Edit /etc/hosts: 1 ## 2 # Host Database 3 # 4 # localhost is used to configure the loopback interface 5 # when the system is booting. Do not change this entry. 6 ## 7 127.0.0.1 localhost 8 255.255.255.255 broadcasthost 9 ::1 localhost 10 11 192.168.33.10 laravel-a.dev laravel-b.dev
146. adding projects A better way:
147. adding projects 1. One File Share config.vm.synced_folder "~/Sites", "/home/vagrant/Sites"
148. adding projects 2. Install DNSMasq brew install dnsmasq cd $(brew —prefix) # /usr/local echo 'address=/.dev/192.168.33.10' > etc/dnsmasq.conf sudo cp -v $(brew --prefix dnsmasq) \ homebrew.mxcl.dnsmasq.plist /Library/LaunchDaemons sudo launchctl load -w /Library/LaunchDaemons/ \ homebrew.mxcl.dnsmasq.plist sudo mkdir -p /etc/resolver echo "nameserver 127.0.0.1" | sudo tee /etc/resolver/dev
149. adding projects 2. DNSMasq continued fideloper@Christophers-iMac  ~  dig whatever-i-want.dev \ @127.0.0.1 ;; QUESTION SECTION: ;whatever-i-want.dev. IN A ;; ANSWER SECTION: whatever-i-want.dev. 0 IN A 192.168.33.10
150. adding projects 3. Magic Nginx Config server { listen 80; server_name ~^(.*)\.dev$; set $file_path $1; root /home/vagrant/Sites/$file_path/public; index index.html index.htm index.php; # And so on …
151. adding projects cd ~/Sites mkdir -p ~/mysite/public echo “<?php phpinfo();” > mysite/public/index.php
152. [Bonus] Databases • User/Network Security • SSH Tunnel • mysqldump / xtrabackup
153. [Bonus] Vagrant • Port forwarding • Configuration • My Homestead config • NFS cache
154. [Bonus] Philosophy • Be ready to throw out a server (Ansible) • Docker is not your first answer without ops people