Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Server Survival

Server Survival

A guide through the annoying parts of servers. This includes slides I couldn't make it through in the actual presentation.

64b0ae04826716bcc1ca5dfd003145fe?s=128

Chris Fidao

July 27, 2016
Tweet

Transcript

  1. Server Survival hi! @fideloper serversforhackers.com

  2. Server Survival Server Survival

  3. Goal: (don’t memorize) understand

  4. Programming Stuff Server Mechanics Semi-Ridiculous Chart of Learning Curves

  5. None
  6. annoying server things security supervision network permissions

  7. Security enjoying your new server responsibly

  8. None
  9. Protection Process Compliance Security “Levels”

  10. Protection

  11. access network Security “Levels” basics

  12. Security “Levels” don’t be root don’t (only) use passwords user

    security
  13. user & access new user new ssh key $ sudo

    adduser fideloper $ sudo usermod -a -G sudo fideloper $ ssh-keygen -t rsa -b 4096
  14. $ ssh-keygen -t rsa -b 4096 \ -f id_whatever $

    ssh-copy-id -i ~/.ssh/id_whatever \ fideloper@<server-ip> (added to ~/.ssh/authorized_keys file) user & access
  15. /etc/ssh/sshd_config Port 22 (or) 1234 PermitRootLogin no (or) without-password PasswordAuthentication

    no AllowGroups some-group ssh access ($ sudo service ssh restart)
  16. firewall network

  17. sudo iptables -L -v firewall

  18. firewall sudo iptables -A INPUT -i lo \ -j ACCEPT

    sudo iptables -A INPUT -m conntrack \ —ctstate RELATED,ESTABLISHED -j ACCEPT
  19. firewall sudo iptables -A INPUT -p tcp --dport 22 \

    -j ACCEPT sudo iptables -A INPUT -p tcp --dport 80 \ -j ACCEPT sudo iptables -A INPUT -p tcp --dport 443 \ -j ACCEPT sudo iptables -A INPUT -j DROP
  20. firewall

  21. firewall drop v reject default policy $ iptables … -j

    REJECT $ iptables \ --policy INPUT DROP
  22. (note: services) $ cat /etc/services | grep http http 80/tcp

    https 443/tcp http-alt 8080/tcp
  23. firewall sudo apt-get install -y \ iptables-persistent sudo service \

    iptables-persistent save
  24. firewall sudo iptables-save > rules.v4 cat rules.v4 | iptables-restore

  25. firewall sudo ip6tables -L -v sudo ip6tables-save > rules.v6 echo

    rules.v6 | ip6tables-restore
  26. $ sudo apt-get install -y \ fail2ban fail2ban 1. monitors

    logs 2. bans IPs
  27. fail2ban

  28. auto upgrades APT::Periodic::Unattended-Upgrade "1"; Unattended-Upgrade::Allowed-Origins { "${distro_id}:${distro_codename}-security"; }; Unattended-Upgrade::InstallOnShutdown "false";

    Unattended-Upgrade::Automatic-Reboot "false"; $ sudo apt-get install -y \ unattended-upgrades files: /etc/apt/apt.conf.d
  29. ¡more! There’s always more SELinux / AppArmor 2FA for SSH

    Securing “secrets” (.env) Strong PW Enforcement (But don’t freak out about it)
  30. Process

  31. policy legit, send me that password, kthnx. Hi! I’m *totally*

  32. policy Define what & how you’re able to send to

    people.
  33. policy Define what happens when people leave.

  34. policy Define what happens when new people come.

  35. policy Decide on “key rotation” (and similar access changes)

  36. policy policy + automation = time =

  37. auditing aggregate logs ()

  38. None
  39. Compliance

  40. Security “Levels” regulation • HIPPA/HITECH (health) • PCI (ecommerce/credit cards)

    • FERPA (education) • Many, many more
  41. Security “Levels” regulation • Audits • Paper work • And

    general security
  42. how far to go? “what should I care about?”

  43. supervision

  44. fid@host:~# sudo systemctl status ssh systemctl status <service> systemctl start

    <service> systemctl stop <service> systemctl enable <service> systemctl disable <service> systemd
  45. fid@host:~# sudo service ssh status • ssh.service - OpenBSD Secure

    Shell server Loaded: loaded (/lib/systemd/system/ssh.service; enabled; \ vendor preset: enabled) Active: active (running) since Fri 2016-07-22 19:46:40 EDT; 1h 27min ago Main PID: 2493 (sshd) CGroup: /system.slice/ssh.service ├─ 2493 /usr/sbin/sshd -D ├─14218 sshd: root [priv] └─14219 sshd: root [net] Jul 22 21:13:28 host sshd[14114]: Accepted password for root from 76.185.167.253 port 56786 ssh2 Jul 22 21:13:28 host sshd[14114]: pam_unix(sshd:session): session opened for user root by (uid=0) systemd
  46. systemd [Unit] Description=OpenBSD Secure Shell server After=network.target auditd.service ConditionPathExists=!/etc/ssh/sshd_not_to_be_run [Service]

    EnvironmentFile=-/etc/default/ssh ExecStart=/usr/sbin/sshd -D $SSHD_OPTS ExecReload=/bin/kill -HUP $MAINPID KillMode=process Restart=on-failure RestartPreventExitStatus=255 Type=notify [Install] WantedBy=multi-user.target Alias=sshd.service /lib/systemd/system/ssh.service
  47. systemd [Unit] Description=Laravel Queue [Service] User=forge Group=forge Restart=on-failure WorkingDirectory=/home/user/forge/mysite.com/current ExecStart=/usr/bin/php

    artisan queue:work --daemon [Install] WantedBy=multi-user.target /lib/systemd/system/laravel.service
  48. fid:~# sudo systemctl enable laravel fid:~# sudo systemctl start laravel

    fid:~# sudo systemctl status laravel systemd
  49. supervisord fid@spr:~# sudo apt-get install -y supervisor fid@spr:~# sudo systemctl

    start supervisor
  50. supervisord fid@spr:~# sudo systemctl status supervisor • supervisor.service - Supervisor

    process control system for UNIX Loaded: loaded (/lib/systemd/system/supervisor.service; disabled; vendor preset: enabled) Active: active (running) since Tue 2016-07-26 17:13:54 EDT; 3s ago Docs: http://supervisord.org Main PID: 3712 (supervisord) Tasks: 1 Memory: 11.1M CPU: 216ms CGroup: /system.slice/supervisor.service !"3712 /usr/bin/python /usr/bin/supervisord -n -c / etc/supervisor/supervisord.conf Jul 26 17:13:54 spr systemd[1]: Started Supervisor process control system for UNIX.
  51. [program:lara_queue] command=php artisan queue:work --daemon directory=/home/forge/app.com/current autostart=true autorestart=true startretries=3 redirect_stderr=true

    stdout_logfile=/home/forge/…/logs/queue.log user=forge numproc=4 supervisord /etc/supervisor/conf.d/lara_q.conf
  52. forge: supervisord

  53. forge: supervisord any old daemon

  54. Network

  55. ifconfig

  56. ifconfig private network f@db:~$ ifconfig eth0 Link encap:Ethernet HWaddr 04:01:31:20:63:01

    inet addr:162.243.164.216 Bcast:162.243.164.255 Mask:255.255.255.0 inet6 addr: fe80::601:31ff:fe20:6301/64 Scope:Link … eth1 Link encap:Ethernet HWaddr 04:01:31:20:63:02 inet addr:10.136.11.155 Bcast:10.136.255.255 Mask:255.255.0.0 inet6 addr: fe80::601:31ff:fe20:6302/64 Scope:Link … lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host …
  57. network binding a tcp socket: <ip-address>:<port> a unix socket: unix://path/to/file.sock

  58. network binding forge@site:~$ netstat -ap | grep http tcp 0

    0 *:http *:* LISTEN 3797/nginx: worker tcp 0 0 *:https *:* LISTEN 3797/nginx: worker
  59. network binding

  60. network: mysql unix socket

  61. tcp socket network: mysql

  62. localhost != 127.0.0.1 (in mysql) network: mysql

  63. network: mysql # # Instead of skip-networking the default is

    now to listen only on # localhost which is more compatible and is not less secure. bind-address = 10.136.11.155 f@db:~$ mysql -h localhost -u root -p Enter password: Welcome to the MySQL monitor. Commands end with ; or \g. f@db:~$ mysql -h 127.0.0.1 -u root -p Enter password: ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ✅
  64. network: mysql f@db:~$ mysql -h 10.136.11.155 -u root -p Enter

    password: ERROR 1130 (HY000): Host '10.136.11.155' is not allowed to connect to this MySQL server f@db:~$ mysql -u root -p -e "create user root@'10.136.11.155' identified by 'root';" Enter password: f@db:~$ mysql -h 10.136.11.155 -u root -p Enter password: Welcome to the MySQL monitor. Commands end with ; or \g. ✅
  65. network but normally it is

  66. 1.Find networks (interfaces) 2.Learn about socket types 3.See examples of

    mysql 4.Future: Permissions in Forge network review
  67. Permissions

  68. who can do things user - file/dir owner group -

    file/dir group - shared permissions! other - anyone else
  69. what can they do read - read file, list directory

    write - write to file, add new file/dir execute - execute command, cd into
  70. usr@hst:~$ chown -R www-data:www-data \ /var/www/example.com usr@hst:~$ chmod -R u=rwx,g=rx,o=rx

    \ /var/www/example.com usr@hst:~$ chmod -R u=rwx,go=rx \ /var/www/example.com usr@hst:$ chmod ug+x,o-x \ /var/www/example.com/artisan setting permissions
  71. d rwx r-x r-x dir user group other permissions

  72. - rwx r-x r-x file user group other permissions

  73. user@host:/var/www$ ls -lAh total 4.0K drwxrwxr-x 2 deploy www-data 4.0K

    Jul 10 21:43 example.com example d rwx rwx r-x deploy : www-data
  74. usr@host:/var/www$ ps axf o pid,user,group,comm \ | grep -i '[n]ginx\|[p]hp'

    4290 root root nginx 4291 www-data www-data \_ nginx 2887 root root php-fpm7.0 2889 www-data www-data \_ php-fpm7.0 2890 www-data www-data \_ php-fpm7.0 not just files
  75. user@host:/var/www$ ls -lAh total 4.0K drwxrwxr-x 2 deploy www-data 4.0K

    Jul 10 21:43 example.com php + web files <?php // run as user and group www-data file_put_contents( '/var/www/example.com/new-file.txt', 'Here is a new line' ); // ✅
  76. remember files owned by www-data then run php as www-data

    $ sudo -u www-data php artisan foo:bar
  77. Just Works™

  78. web files there’s no place like forge@host:~/store.helpspot.com/current$ ls -lAh drwxrwxr-x

    15 forge forge app -rwxrwxr-x 1 forge forge artisan drwxrwxr-x 3 forge forge bootstrap -rw-rw-r-- 1 forge forge composer.json -rw-rw-r-- 1 forge forge composer.lock
  79. // File /etc/nginx/nginx.conf user forge; worker_processes auto; pid /run/nginx.pid; events

    { … } http { … } nginx
  80. // File /etc/php5/fpm/pool.d/www.conf listen = /var/run/php5-fpm.sock listen.owner = www-data listen.group

    = www-data listen.mode = 0666 user = forge group = forge -rw-rw-rw php-fpm
  81. // File /etc/php5/fpm/pool.d/www.conf listen = 127.0.0.1:9000 listen.owner = www-data listen.group

    = www-data listen.mode = 0666 user = forge group = forge php-fpm
  82. web files there’s no place like forge@host:~/store.helpspot.com/current$ ls -lAh drwxrwxr-x

    15 forge forge app -rwxrwxr-x 1 forge forge artisan drwxrwxr-x 3 forge forge bootstrap -rw-rw-r-- 1 forge forge composer.json -rw-rw-r-- 1 forge forge composer.lock
  83. There’s More! ACLs

  84. ACL

  85. ACL ACL Defaults

  86. $ sudo setfacl -Rm \ > g:www-data:rwx,d:g:www-data:rwx \ > /var/www/html

    ACL
  87. ACL Owned by root Group www-data:rwx

  88. ACL User-based!

  89. Server Survival thanks! @fideloper serversforhackers.com

  90. pkg managers 1. searching 2. installing

  91. pkg managers apt-get & apt sudo apt-get update sudo apt

    update sudo apt-get install whatever sudo apt install whatever
  92. pkg managers search sudo apt search mysql-server ubuntu@host:~$ apt search

    mysql-server mysql-server/trusty-updates,trusty-security 5.5.49-0… mysql-server-5.5/trusty-updates,trusty-security MySQL database server binaries and system database setup mysql-server-5.6/trusty-updates,trusty-security MySQL database server binaries and system database setup
  93. pkg managers show sudo apt show -a \ mysql-server-5.6 Package:

    mysql-server-5.6 Version: 5.6.30-0ubuntu0.14.04.1 Package: mysql-server-5.6 Version: 5.6.16-1~exp1
  94. pkg managers policy sudo apt-cache policy \ mysql-server-5.6 mysql-server-5.6: Installed:

    (none) Candidate: 5.6.30-0ubuntu0.14.04.1 Version table: 5.6.30-0ubuntu0.14.04.1 0 500 http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ trusty-updates/universe amd64 Packages 500 http://security.ubuntu.com/ubuntu/ trusty-security/universe amd64 Packages 5.6.16-1~exp1 0 500 http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ trusty/universe amd64 Packages
  95. pkg managers package=version sudo apt-get install \ mysql-server-5.6=5.6.16-1~exp1

  96. pkg managers repositories sudo add-apt-repository \ ppa:ondrej/php

  97. pkg managers ubuntu@host: /etc/apt/sources.list.d $ ls -lah -rw-r--r-- 1 root

    root ondrej-ubuntu-php-xenial.list ubuntu@host: /etc/apt/sources.list.d $ cat \ ondrej-ubuntu-php-xenial.list deb http://ppa.launchpad.net/ondrej/php/ubuntu xenial main # deb-src http://ppa.launchpad.net/ondrej/php/ubuntu xenial main repositories
  98. pkg managers ubuntu@host: ~ sudo apt-key adv --recv-keys --keyserver \

    hkp://keyserver.ubuntu.com:80 0xF1656F24C74CD1D8 ubuntu@host: ~ echo 'deb http://ftp.utexas.edu/mariadb/ repo/10.1/ubuntu xenial main' \ | sudo tee /etc/apt/sources.list.d/mariadb.list manual install
  99. pkg managers ubuntu@host: /etc/apt $ vim sources.list # See http://help.ubuntu.com/community/UpgradeNotes

    for how to upgrade to # newer versions of the distribution. deb http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ xenial main restricted deb-src http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ xenial main restricted # # Major bug fix updates produced after the final release of the # # distribution. deb http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ xenial-updates main restricted deb-src http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ xenial-updates main \ restricted # # N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu # # team. Also, please note that software in universe WILL NOT receive any # # review or updates from the Ubuntu security team. deb http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ xenial universe deb-src http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ xenial universe included repositories
  100. pkg managers repositories

  101. –@fideloper “Am I done now?”

  102. DNS & Domains

  103. DNS: ¯\_(ツ)_/¯ ~~ just *TRY* ~~ to coherently explain controlling

    domains to the average [non-tech-client-whoever] challenge:
  104. ¯\_(ツ)_/¯ (we’ll ignore the seedy secondary market of domain-squatting asshats)

  105. you bought a domain DNS

  106. …but you control it somewhere else DNS

  107. DNS “somewhere else” points [sub]domains… somewhere else…

  108. this slide intentionally left blank

  109. dig

  110. dig mx records

  111. nslookup mx records again

  112. Domains + Web Server

  113. Domains + Web Server

  114. set host header

  115. default_server

  116. (set host header again)

  117. meanwhile, fideloper.com

  118. Logs

  119. /var/log/*

  120. wtf, Vagrant? less terrible.

  121. fideloper@host  ~  vagrant box Usage: vagrant box <subcommand>

    [<args>] Available subcommands: add list outdated remove repackage update boxes (servers)
  122. where’s my stuff? ~/.vagrant.d/boxes ~/.vagrant.d/tmp C:/Users/[USER]/.vagrant.d/boxes C:/Users/[USER]/.vagrant.d/tmp

  123. • ssh by default • can add your own (but

    we’ll do better) port forwarding SSH Default
  124. port forwarding $ vagrant ssh-config Just like ~/.ssh/config

  125. port forwarding $ ssh \ > -i /Users/fideloper/…/virtualbox/private_key \ >

    -p 2222 \ > vagrant@localhost
  126. port forwarding

  127. port forwarding (aside: It’s common to forward to port 80)

    config.vm.network "forwarded_port", guest: 80, host: 8000 $> curl -I localhost:8000 HTTP/1.1 302 Found Server: nginx/1.9.9 Content-Type: text/html; charset=UTF-8 Date: Sat, 02 Jul 2016 17:57:49 GMT Location: http://localhost:8000/login
  128. port forwarding But, two boxes can’t forward to same port!

    config.vm.network "forwarded_port", guest: 80, host: 8000 config.vm.network "forwarded_port", guest: 80, host: 8888 ✅ first box: second box:
  129. sequel pro 2.5 Ways to Connect to MySQL (without *any*

    MySQL configuration)
  130. sequel pro 1 - Port Forward config.vm.network "forwarded_port", guest: 3306,

    host: 33060
  131. sequel pro config.vm.network "forwarded_port", guest: 3306, host: 33060

  132. sequel pro 2 - SSH Tunnel $> ssh -p 2222

    \ -i /Users/fideloper/…/virtualbox/private_key \ -L 3306:localhost:3306 vagrant@localhost
  133. sequel pro

  134. sequel pro 2.5 - SSH Tunnel

  135. • 1. Port forwarding (homestead way - easy) • 2.

    Manual SSH tunnel • 3. Sequel Pro SSH Tunnel
  136. sequel pro Remember the SSH Tunnel! You can use it

    in production to view a database.
  137. file sharing config.vm.synced_folder “~/Sites", "/home/vagrant/Sites" default file share slow with

    a large # files
  138. file sharing config.vm.synced_folder “~/Sites", "/home/vagrant/Sites", id: "core", :nfs => true,

    :mount_options => [‘nolock,vers=3,udp,noatime,actimeo=2,fsc'] network file share handles large # files better
  139. file sharing where to run build steps? (especially ones that

    watch files)
  140. file sharing (I’ve actually used Docker for this instead) docker

    run --rm \ -v ~/Sites/some-project:/opt \ some_node_img:latest \ gulp watch
  141. adding projects How I made adding a new project painless

    (and stopped editing /etc/hosts)
  142. adding projects An annoying process:

  143. adding projects 1. Share More Files: config.vm.synced_folder "~/Sites/a", "/var/www/a" config.vm.synced_folder

    "~/Sites/b", “/var/www/b"
  144. adding projects 2. Create another server config vagrant@vagrant:/etc/nginx/sites-available$ sudo cp

    \ laravel-a laravel-b vagrant@vagrant:/etc/nginx/sites-available$ sudo vim \ laravel-b server { listen 80; server_name laravel-b.dev; …
  145. adding projects 3. Edit /etc/hosts: 1 ## 2 # Host

    Database 3 # 4 # localhost is used to configure the loopback interface 5 # when the system is booting. Do not change this entry. 6 ## 7 127.0.0.1 localhost 8 255.255.255.255 broadcasthost 9 ::1 localhost 10 11 192.168.33.10 laravel-a.dev laravel-b.dev
  146. adding projects A better way:

  147. adding projects 1. One File Share config.vm.synced_folder "~/Sites", "/home/vagrant/Sites"

  148. adding projects 2. Install DNSMasq brew install dnsmasq cd $(brew

    —prefix) # /usr/local echo 'address=/.dev/192.168.33.10' > etc/dnsmasq.conf sudo cp -v $(brew --prefix dnsmasq) \ homebrew.mxcl.dnsmasq.plist /Library/LaunchDaemons sudo launchctl load -w /Library/LaunchDaemons/ \ homebrew.mxcl.dnsmasq.plist sudo mkdir -p /etc/resolver echo "nameserver 127.0.0.1" | sudo tee /etc/resolver/dev
  149. adding projects 2. DNSMasq continued fideloper@Christophers-iMac  ~  dig

    whatever-i-want.dev \ @127.0.0.1 ;; QUESTION SECTION: ;whatever-i-want.dev. IN A ;; ANSWER SECTION: whatever-i-want.dev. 0 IN A 192.168.33.10
  150. adding projects 3. Magic Nginx Config server { listen 80;

    server_name ~^(.*)\.dev$; set $file_path $1; root /home/vagrant/Sites/$file_path/public; index index.html index.htm index.php; # And so on …
  151. adding projects cd ~/Sites mkdir -p ~/mysite/public echo “<?php phpinfo();”

    > mysite/public/index.php
  152. [Bonus] Databases • User/Network Security • SSH Tunnel • mysqldump

    / xtrabackup
  153. [Bonus] Vagrant • Port forwarding • Configuration • My Homestead

    config • NFS cache
  154. [Bonus] Philosophy • Be ready to throw out a server

    (Ansible) • Docker is not your first answer without ops people