Home sshd DDOS measures

Your home server is safe? Check yours to secure from DDOS!
COSCUP 2023発表資料
https://coscup.org/2023/en/session/GNXGDH

Transcript

1. Home sshd DDOS measures Shunsuke Yoshida
2. Agenda/概要 • Who I am/自己紹介 • Main
3. Who I am • Shunsuke Yoshida – @koedoyoshida : Twitter / Facebook • PyCon JP Association Board Member – PyCon JP Staff 2014 - – PyCon JP 2017-2019 Chair – 2023- PyCon JP Association Vice-Chair • KoedoLUG – Japan Local Linux User Group (Monthly) • Tokyo Debian study group / DebianJP • Work: MIRACLE LINUX / Cybertrust Japan Co., Ltd.
4. My 10 Year PyCon APAC/JP history • 2013 PyCon APAC 2013 in Japan – 1st Attendee/初参加、参加者 • 2014 PyCon JP2014 – Start Staff/スタッフ活動を開始 • 2017-2019 PyCon JP Chair – 日本の他の(Mini)PyConに積極的に参加 – PyCon APAC(アジア開催)は重点的に参加 – PyCon US 2017,PyCon Taiwanなど • 2019- PyCon JP Association Board Members • 2023- PyCon JP Association Vice-Chair • 2023 PyCon APAC 2023 in Japan
5. The PyCon JP Association team was awarded the 2021 Q4 Community Service Award. Python Software Foundation award the 2021 Q4 Community Service Award to the following members of the PyCon JP Association for their work in organizing local and regional PyCons, the work of monitoring our trademarks, and in particular organizing the "PyCon JP Charity Talks" raising more than $25,000 USD in funds for the PSF: Manabu Terada, Takanori Suzuki, Takayuki Shimizukawa, Shunsuke Yoshida, Jonas Obrist. https://pyfound.blogspot.com/2022/05/
6. PSF Community Service Awards @ PyCon US 2022
7. PyCon APAC 2023 in Japan
8. PyCon APAC 2023 in Japan Status • Talk Speaker: Closed • Sponsor: Open • Attendee registration: Not yet/準備中 • Travel Grant: Not yet/準備中 • VISA: Not yet/準備中 • https://2023-apac.pycon.jp/ • https://pyconjp.blogspot.com/
9. PyCon APAC 2023: Call for proposals for community poster sessions • We are now accepting proposals for community poster sessions. Community poster sessions are intended for Python communities • Form: QRCODE • Or PyCon JP blog
10. Main
11. As an example of cyber attack countermeasures • サイバー攻撃対策の一例として紹介
12. 発端/ beginning • I'm using my home server's sshd for external access. – 自宅サーバのraspiのsshdをルータのポートフォワードで 自宅アクセス用に公開している • However, in 2022, after the beginning of the year, that can not access from external access occurs several times – しかし、2022年、年明け以降該当ポートにアクセスしても 接続できない現象が数回発生 • Can't access sshd when trying to access from home network – 自宅ネットワークからアクセスしようとしてもsshdにアクセ スできない
13. log:netstat -nap • tcp 57 0 192.168.0.203:443 109.74.204.123:37936 CLOSE_WAIT - • tcp 25 0 192.168.0.203:443 46.183.222.42:60698 CLOSE_WAIT - • tcp 23 0 192.168.0.203:443 92.255.85.135:21832 CLOSE_WAIT - • tcp 183 0 192.168.0.203:443 185.180.143.79:43684 CLOSE_WAIT - • tcp 42 0 192.168.0.203:443 219.123.27.194:54812 CLOSE_WAIT - • tcp 145 0 192.168.0.203:443 192.241.217.127:50962 CLOSE_WAIT - • tcp 231 0 192.168.0.203:443 109.74.204.123:34828 CLOSE_WAIT - • tcp 177 0 192.168.0.203:443 109.74.204.123:34834 CLOSE_WAIT - • tcp 145 0 192.168.0.203:443 192.241.203.171:38696 CLOSE_WAIT - • tcp 42 0 192.168.0.203:443 219.123.27.194:54150 CLOSE_WAIT - • tcp 23 0 192.168.0.203:443 92.255.85.135:59922 CLOSE_WAIT - • tcp 25 0 192.168.0.203:443 185.29.8.36:56558 CLOSE_WAIT - • tcp 42 0 192.168.0.203:443 219.123.27.194:58756 CLOSE_WAIT - • tcp 244 0 192.168.0.203:443 45.146.165.37:43440 CLOSE_WAIT - • tcp 23 0 192.168.0.203:443 92.255.85.135:37314 CLOSE_WAIT - • tcp 23 0 192.168.0.203:443 92.255.85.135:57212 CLOSE_WAIT - • tcp 105 0 192.168.0.203:443 164.52.24.162:40520 CLOSE_WAIT - • tcp 244 0 192.168.0.203:443 45.146.165.37:34442 CLOSE_WAIT - • tcp 244 0 192.168.0.203:443 45.146.165.37:38198 CLOSE_WAIT - • tcp 23 0 192.168.0.203:443 92.255.85.135:32044 CLOSE_WAIT - • tcp 1 0 192.168.0.203:443 8.210.22.46:65475 CLOSE_WAIT - • tcp 514 0 192.168.0.203:443 178.62.194.176:38000 CLOSE_WAIT - • tcp 518 0 192.168.0.203:443 109.74.204.123:40360 CLOSE_WAIT - • tcp 244 0 192.168.0.203:443 45.146.165.37:44962 CLOSE_WAIT - • tcp 23 0 192.168.0.203:443 92.255.85.135:24970 CLOSE_WAIT - • tcp 42 0 192.168.0.203:443 219.123.27.194:56552 CLOSE_WAIT - • tcp 23 0 192.168.0.203:443 92.255.85.135:49884 CLOSE_WAIT - • root@rpi:~# cat closewaitlist.log | grep CLOSE |wc • 105 736 10605
14. sshd was receiving DDOS attack. sshdがDDOS攻撃を受けていました • Netstat shows more than 100 CLOSE_WAIT sessions – Netstatで見るとCLOSE_WAITのセッションが100件以 上 • Many accesses from unknown IPs – 全然知らないIPからのアクセスが多数 • Should CLOSE_WAIT sessions be deleted early? – とりあえずの対策はCLOSE_WAITセッションを早く消 せば良いのでは？
15. distributed denial-of-service attack (DDoS attack) • cyber-attack in which the perpetrator seeks to make a machine or network resource unavailable to its intended users by temporarily or indefinitely disrupting services of a host connected to a network. • https://en.wikipedia.org/wiki/ Denial-of-service_attack
16. workaround とりあえずの復旧 • restart sshd • # systemctl restart ssh – Takes a few minutes to restart – It's late, isn't it? – サービス再起動に数分にかかった – 遅いよね?
17. Should CLOSE_WAIT be erased early? CLOSE_WAITを早く消せば良い？ • https://hirofukami.com/2008/08/11/close- wait/
18. sysctl • # vi /etc/sysctl.conf • net.ipv4.tcp_keepalive_time = 10 • net.ipv4.tcp_keepalive_probes = 2 • net.ipv4.tcp_keepalive_intvl = 3 • # sysctl -p
19. A few days later 数日後 • can't connect again – また繋がらない • reinvestigation – 現象再調査 • About 100 CLOSE_WAIT with the same phenomenon – 同様の現象でCLOSE_WAITが100件程度
20. Try Ban4ip Ban4ip を試してみる • fail2banをやめてBan4ipにしませんか？ | Linux Install Memo • https://qiita.com/_takeuchi_/items/c06f0688752 1ca4684e7 – URL env:Debian9 Myenv Debian11 • Custom Debian9 instructions for Debian 11 – Change php-7.0 to php7.4 and basically install as above – 環境はDebian 11なので読み替えてphpリビジョンの み変更してインストール
21. sshd log • Feb 6 22:33:57 rpi sshd[8724]: error: kex_exchange_identification: Connection closed by remote host • Feb 6 22:33:57 rpi sshd[8724]: Connection closed by 104.248.236.25 port 61953 • Feb 6 22:53:20 rpi sshd[12201]: error: kex_exchange_identification: banner line contains invalid characters • Feb 6 22:53:20 rpi sshd[12201]: banner exchange: Connection from 192.241.205.201 port 38756: invalid format • Feb 6 22:53:22 rpi sshd[12209]: error: kex_exchange_identification: banner line contains invalid characters • Feb 6 22:53:22 rpi sshd[12209]: banner exchange: Connection from 192.241.210.213 port 45306: invalid format • Feb 6 23:13:37 rpi sshd[15840]: error: kex_exchange_identification: banner line contains invalid characters • Feb 6 23:13:37 rpi sshd[15840]: banner exchange: Connection from 193.118.53.210 port 41310: invalid format • Feb 6 23:17:39 rpi sshd[16558]: error: kex_exchange_identification: Connection closed by remote host • Feb 6 23:17:39 rpi sshd[16558]: Connection closed by 46.101.169.203 port 43242 • Feb 6 23:44:24 rpi sshd[21403]: error: kex_exchange_identification: banner line contains invalid characters • Feb 6 23:44:24 rpi sshd[21403]: banner exchange: Connection from 45.146.165.37 port 35700: invalid format • Feb 7 00:27:53 rpi sshd[29161]: error: kex_exchange_identification: banner line contains invalid characters • Feb 7 00:27:53 rpi sshd[29161]: banner exchange: Connection from 45.146.165.37 port 42230: invalid format • Feb 7 00:52:25 rpi sshd[1115]: error: kex_exchange_identification: banner line contains invalid characters • Feb 7 00:52:25 rpi sshd[1115]: banner exchange: Connection from 139.162.145.250 port 46242: invalid format • Feb 7 01:03:06 rpi sshd[3002]: error: kex_exchange_identification: banner line contains invalid characters • Feb 7 01:03:06 rpi sshd[3002]: banner exchange: Connection from 192.241.212.170 port 46332: invalid format • Feb 7 01:03:06 rpi sshd[3003]: error: kex_exchange_identification: banner line contains invalid characters
22. ban4ip setting/設定 • # cat /etc/ban4ip/sshd.conf • [sshd_section] • target_service = 'sshd-error' • #target_log = '/var/log/secure' • target_log = '/var/log/auth.log' • target_protcol = 'tcp' • target_port = 443 • target_rule = 'DROP' • target_str[] = '/sshd¥[.*¥]: Failed .* for .* from (.*) port /' • target_str[] = '/sshd¥[.*¥]: .* from (.*) not allowed because /' • target_str[] = '/sshd¥[.*¥]: Connection closed by (.*) port .*/' • [Overwrite_Section] • ;findtime = 300 • ;maxretry = 3 • ;bantime = 3600 • findtime = 600 • maxretry = 3 • bantime = 3600
23. But A few days later また数日後 • can't connect again – また繋がらない • The access sources are distributed, and the number of accesses is about 2 times at each location. – アクセス元が分散していて、アクセス回数も各所時間 を空けて2回程度 • It did not meet the conditions of ban4ip, so there was no effect of ban4ip – つまりban4ipの条件に合致しないので、ban4ipの効 果が無かった
24. Re-examine from the error message エラーメッセージから再調査 • /var/log/auth.log • [ssh_exchange_identification: Connection closed by remote host] • You think this message in the server log just means that the client closed the connection, right? – あなたはサーバのログに出ていたこのメッセージの 意味をクライアントが接続をクローズしたというだけだ と思いますよね? • It had other meanings. – 他の意味がありました。
25. Sshd [MaxStartups] • https://qiita.com/ssssasaki/items/11123677e2c154dcfe1f • Sshd [MaxStartups] • Many such as RH series and Debian Default value 10:30:100 – RH系やDebianなど多くのデフォルト設定では 10:30:100 になっています。 • This means that when the number of unauthenticated connections exceeds 10, subsequent connections will be rejected with a probability of 30%, and when the number reaches 100, all subsequent connections will be rejected. – これは、非認証な接続が 10 を超えるとそれ以降の接続を 30% の確率 で拒絶し、さらに 100 に達した時点でそれ以降の接続を全て拒絶すると いう設定になります • Normally, 10 unauthenticated connections come in at once in most environments. – 通常、非認証な接続が一気に 10 も来ることはほとんどの環境では無い • ssh_exchange_identification: Connection closed by remote host • During the rejection pattern above, I get the above error. – 上記の 拒絶 パターンの際に、上記のエラーが表示されます。
26. Is it possible to detect this? これを検知できるのでは？ • Restart sshd if a specific string appears in the log – 特定の文字列がログに出たらsshdを再起動する • log monitor script – ログ監視スクリプト – https://qiita.com/Qrg/items/107928672569a8141222 • Start your own shell script as a Linux systemd service – 自作したシェルスクリプトを Linux の systemd サービ スとして起動する – https://genzouw.com/entry/2021/07/05/154156/270 1/
27. Create a script from sample code and log ほぼ流用したスクリプト、ログ取りも • # cat /usr/local/bin/logchecker.sh • #!/bin/sh • # https://qiita.com/Qrg/items/107928672569a8141222 • # 検出対象ログファイル • TARGET_LOG="/var/log/auth.log" • # 検出文字列 • _error_conditions="error: kex_exchange_identification: Connection closed by remote host" • # ログファイルを監視する関数 • hit_action() { • while read i • do • echo $i | grep -q "${_error_conditions}" • if [ $? = "0" ];then • # アクション • netstat -nap > /tmp/netstat.log.`date +%Y%m%d%H%M%S` • ss -nap > /tmp/ss.log.`date +%Y%m%d%H%M%S` • systemctl restart sshd • fi • done • } • # main • if [ ! -f ${TARGET_LOG} ]; then • touch ${TARGET_LOG} • fi • tail -n 0 --follow=name --retry $TARGET_LOG | hit_action
28. For systemd script from sample code systemd 用こちらもほぼそのまま • root@rpi:~# cat /etc/systemd/system/monitor-sshd.service • [Unit] • Description=sshd monitoring shell script service • Documentation= • #After=networking.service • [Service] • Type=simple • User=root • Group=root • TimeoutStartSec=0 • Restart=on-failure • RestartSec=30s • #ExecStartPre= • ExecStart=/usr/local/bin/logchecker.sh • SyslogIdentifier=ssh_monitoring • #ExecStop= • [Install] • WantedBy=multi-user.target
29. Result log 結果ログ • root@rpi:~# ls /tmp/*.log.* -ltr • -rw-r--r-- 1 root root 21205 Feb 13 14:40 /tmp/netstat.log.20220213144044 • -rw-r--r-- 1 root root 94107 Feb 13 14:40 /tmp/ss.log.20220213144044 • -rw-r--r-- 1 root root 22095 Feb 14 12:46 /tmp/netstat.log.20220214124652 • -rw-r--r-- 1 root root 91235 Feb 14 12:46 /tmp/ss.log.20220214124653 • -rw-r--r-- 1 root root 21306 Feb 14 16:04 /tmp/netstat.log.20220214160413 • -rw-r--r-- 1 root root 87191 Feb 14 16:04 /tmp/ss.log.20220214160413 • -rw-r--r-- 1 root root 21306 Feb 14 19:09 /tmp/netstat.log.20220214190924 • -rw-r--r-- 1 root root 87191 Feb 14 19:09 /tmp/ss.log.20220214190924 • -rw-r--r-- 1 root root 21306 Feb 14 21:21 /tmp/netstat.log.20220214212131 • -rw-r--r-- 1 root root 87191 Feb 14 21:21 /tmp/ss.log.20220214212131 • -rw-r--r-- 1 root root 21751 Feb 15 00:12 /tmp/netstat.log.20220215001204 • -rw-r--r-- 1 root root 80640 Feb 15 00:12 /tmp/ss.log.20220215001204 • -rw-r--r-- 1 root root 21650 Feb 15 00:45 /tmp/netstat.log.20220215004517 • -rw-r--r-- 1 root root 82077 Feb 15 00:45 /tmp/ss.log.20220215004517 • -rw-r--r-- 1 root root 21549 Feb 15 05:48 /tmp/netstat.log.20220215054852
30. But A few days later また数日後 • Again, again, it won't connect... – またまた繋がらない... • Research… – 現象再調査... • Now when I look at netstat, there are no remaining sessions – 今度はnetstatを見ても残セッション無し • Continuing research... – 継続調査すると...
31. sshd failed to restart sshdが再起動失敗 • # systemctl status sshd • • ssh.service - OpenBSD Secure Shell server • Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled) • Active: failed (Result: start-limit-hit) since Wed 2022-02-23 11:18:19 UTC; 1 day 11h ago • Docs: man:sshd(8) • man:sshd_config(5) • Process: 13523 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS) • Process: 13524 ExecStart=/usr/sbin/sshd -D $SSHD_OPTS (code=exited, status=0/SUCCESS) • Feb 23 11:18:19 rpi sshd[13524]: Received signal 15; terminating. • Feb 23 11:18:19 rpi systemd[1]: Stopping OpenBSD Secure Shell server... • Feb 23 11:18:19 rpi systemd[1]: ssh.service: Succeeded. • Feb 23 11:18:19 rpi systemd[1]: Stopped OpenBSD Secure Shell server. • Feb 23 11:18:19 rpi systemd[1]: ssh.service: Start request repeated too quickly. • Feb 23 11:18:19 rpi systemd[1]: ssh.service: Failed with result 'start-limit-hit'. • Feb 23 11:18:19 rpi systemd[1]: Failed to start OpenBSD Secure Shell server.
32. Start request repeated too quickly. • [https://tex2e.github.io/blog/linux/systemd-restart-config] • systemd has a function to give up restarting when restarting occurs many times in a short time. Specifically, after StartLimitBurst restarts occur during StartLimitInterval , systemd will stop automatically restarting. By default, it will reboot up to 5 times in 10 seconds, after which it will give up. • Systemdには短い時間で何回も再起動が発生すると再起動を諦め る機能があります。 具体的には StartLimitInterval の間に StartLimitBurst の回数だけ再起動が起きると、systemd は自動的 に再起動するのを止めます。デフォルトでは 10 秒の間に 5 回まで 再起動が行われ、それを超えると再起動を諦めます。 • /etc/systemd/system.conf • [Manager] • #DefaultStartLimitInterval=10s • #DefaultStartLimitBurst=5
33. sshd log • Feb 23 07:36:06 rpi sshd[32282]: debug1: Received SIGCHLD. • Feb 23 07:36:06 rpi sshd[32282]: debug1: session_by_pid: pid 32295 • Feb 23 07:36:06 rpi sshd[32282]: debug1: session_exit_message: session 0 channel 0 pid 32295 • Feb 23 07:36:06 rpi sshd[32282]: debug1: session_exit_message: release channel 0 • Feb 23 07:36:06 rpi sshd[32218]: debug1: session_by_tty: session 0 tty /dev/pts/7 • Feb 23 07:36:06 rpi sshd[32218]: debug1: session_pty_cleanup2: session 0 release /dev/pts/7 • Feb 23 07:36:06 rpi sshd[32282]: Received disconnect from 127.0.0.1 port 55180:11: disconnected by user • Feb 23 07:36:06 rpi sshd[32282]: Disconnected from user syoshida 127.0.0.1 port 55180 • Feb 23 07:36:06 rpi sshd[32282]: debug1: do_cleanup • Feb 23 07:36:06 rpi sshd[32282]: debug1: temporarily_use_uid: 1001/1001 (e=1001/1001) • Feb 23 07:36:06 rpi sshd[32282]: debug1: restore_uid: (unprivileged) • Feb 23 07:36:06 rpi sshd[32218]: debug1: do_cleanup • Feb 23 07:36:06 rpi sshd[32218]: debug1: PAM: cleanup • Feb 23 07:36:06 rpi sshd[32218]: debug1: PAM: closing session • Feb 23 07:36:06 rpi sshd[32218]: pam_unix(sshd:session): session closed for user syoshida • Feb 23 07:36:06 rpi sshd[32218]: debug1: PAM: deleting credentials • Feb 23 07:36:06 rpi sshd[32218]: debug1: temporarily_use_uid: 1001/1001 (e=0/0) • Feb 23 07:36:06 rpi sshd[32218]: debug1: restore_uid: 0/0 • Feb 23 07:36:06 rpi sshd[32218]: debug1: audit_event: unhandled event 12 • Feb 23 07:36:06 rpi sshd[29015]: debug1: main_sigchld_handler: Child exited • Feb 23 07:36:06 rpi systemd-logind[320]: Session 2448 logged out. Waiting for processes to exit. • Feb 23 07:36:06 rpi systemd-logind[320]: Removed session 2448. • Feb 23 07:36:11 rpi sshd[11671]: debug1: server_input_global_request: rtype keepalive@openssh.com want_reply 1
34. Prevent sshd from rerunning too soon sshdが短時間で再実行されないように • # cat /usr/local/bin/logchecker.sh • (略) • # ログファイルを監視する関数 • while read i • do • echo $i | grep -q "${_error_conditions}" • if [ $? = "0" ];then • # アクション • netstat -nap > /tmp/netstat.log.`date +%Y%m%d%H%M%S` • ss -nap > /tmp/ss.log.`date +%Y%m%d%H%M%S` • systemctl restart sshd • sleep 300 • fi • done
35. This is the solution! これで解決！ • Stable! – 安定！ • By the way, where did the attack come from – ところで、どこからアタックされてたんだろう？
36. Continuously Coming IP 継続的に来ているIP • Feb 20 14:57:33 rpi sshd[25762]: Connection from 45.146.165.37 port 35236: invalid format • Feb 20 16:15:43 rpi sshd[27355]: Connection from 45.146.165.37 port 37156: invalid format • Feb 20 17:31:22 rpi sshd[28597]: Connection from 45.146.165.37 port 50254: invalid format • Feb 20 18:16:26 rpi sshd[29368]: Connection from 45.146.165.37 port 42220: invalid format • Feb 20 19:34:56 rpi sshd[30845]: Connection from 45.146.165.37 port 36502: invalid format • Feb 20 20:22:35 rpi sshd[31756]: Connection from 45.146.165.37 port 42334: invalid format • Feb 20 20:35:15 rpi sshd[31961]: Connection from 45.146.165.37 port 37422: invalid format • Feb 20 21:47:39 rpi sshd[808]: Connection from 45.146.165.37 port
37. where are you coming from どこからアタックが来ているか？ • Whois • IPAddress ： 27.82.198.176 • NetRange: 192.241.128.0 - 192.241.255.255 CIDR: 192.241.128.0/17 NetName: DIGITALOCEAN-192-241-128-0 NetHandle: NET-192-241-128-0-1 Parent: NET192 (NET- 192-0-0-0-0) NetType: Direct Allocation OriginAS: AS14061 Organization: DigitalOcean, LLC (DO-13) RegDate: 2013-06- 10 Updated: 2020-04-03 Comment: Routing and Peering Policy can be found at https://www.as14061.net Comment: Comment: Please submit abuse reports at https://www.digitalocean.com/company/contact/#abuse Ref: https://rdap.arin.net/registry/ip/192.241.128.0
38. where are you coming from どこからアタックが来ているか？ • IP Address 45.146.165.37 • This is the RIPE Database query service. % The objects are in RPSL format. % % The RIPE Database is subject to Terms and Conditions. % See http://www.ripe.net/db/support/db-terms- conditions.pdf % Note: this output has been filtered. % To receive output for a database update, use the "-B" flag. % Information related to '45.146.164.0 - 45.146.165.255' % Abuse contact for '45.146.164.0 - 45.146.165.255' is 'abuse@hostway.ru' inetnum: 45.146.164.0 - 45.146.165.255 netname: RU-ITRESHENIYA country: RU → Russian Federation （ロシア連邦） org: ORG- ITR1-RIPE admin-c: ITR30-RIPE tech-c: ITR30-RIPE status: ASSIGNED PA mnt-by: IP-RIPE mnt-routes: MNT-SELECTEL created: 2020-09- 07T16:45:55Z last-modified: 2021-10-05T19:15:10Z source: RIPE •
39. DDOS true cause? DDOS真の原因? • February 2022 was just around the time the Ukrainian war started... – 2022年2月はちょうどウクライナ戦争が開戦した 頃でしたね… • It was a cyber attack from Russia... – ロシアからのサイバー攻撃でした…
40. 終わり/Appendix
41. PyCon APAC 2023 in Japan Status • Talk Speaker: Closed • Sponsor: Open • Attendee registration: Not yet/準備中 • Travel Grant: Not yet/準備中 • VISA: Not yet/準備中 • https://2023-apac.pycon.jp/ • https://pyconjp.blogspot.com/
42. PyCon APAC 2023: Call for proposals for community poster sessions • We are now accepting proposals for community poster sessions. Community poster sessions are intended for Python communities • Form: QRCODE • Or PyCon JP blog