Self-Documenting Architectures

Self-Documenting Architectures a.k.a: How to perpetuate laziness By: Hany Fahim
(@iHandroid)

Background Our job at VM Farms is to manage our
client’s architectures. An essential component of managing any environment is proper documentation. We’ve worked towards this goal using a number techniques...

Background... We standardize on all environments - one method to
rule them all. We use an extensive conﬁguration manager (mix of Puppet and our own tech). Perusing through a conﬁg reveals the architecture and layout. We monitor and trend EVERYTHING. One look at the dashboard for a server reveals important services.

Something’s still missing For suﬃciently large systems (>10 servers, multiple
stacks, techs, etc...), glancing at a conﬁg or monitoring control panel is not enough. Some form of visualization would help greatly. Network diagrams are useful in such cases.

EXAMPLE OF UGLY, BUT FUNCTIONAL, NETWORK DIAGRAM

ANOTHER UGLY, BUT FUNCTIONAL, DIAGRAM

Problem: How do you create and maintain a network diagram
for ever evolving systems?

netstat to the rescue netstat is a Linux utility that
can inspect and report on active network and socket connections. Reads in data from /proc Lists incoming connections, outgoing connections, listening sockets, ports, IPs, etc...

/proc/net/tcp # cat /proc/net/tcp sl local_address rem_address
st tx_queue rx_queue tr tm-‐>when retrnsmt uid timeout inode 0: 00000000:006F 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 8422 1 ffff8800047ff400 99 0 0 10 -‐1 1: 00000000:1111 00000000:0000 0A 00000000:00000000 00:00000000 00000000 495 0 22274 1 ffff88003d9fd5c0 99 0 0 10 -‐1 2: 00000000:0016 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 8861 1 ffff88003e0a9440 99 0 0 10 -‐1 3: 00000000:D978 00000000:0000 0A 00000000:00000000 00:00000000 00000000 495 0 124889478 1 ffff88003b7b66c0 99 0 0 10 -‐1 4: 0100007F:0019 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 43799744 1 ffff880012fea480 99 0 0 10 -‐1 5: 00000000:ECFD 00000000:0000 0A 00000000:00000000 00:00000000 00000000 495 0 124889336 1 ffff88003d8c6040 99 0 0 10 -‐1 6: 00000000:C822 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 145564322 1 ffff88003e0a8740 99 0 0 10 -‐1 7: 00000000:1622 00000000:0000 0A 00000000:00000000 00:00000000 00000000 496 0 100632608 1 ffff88003e0a8dc0 99 0 0 10 -‐1 8: 0100007F:B192 0100007F:1111 01 00000000:00000000 00:00000000 00000000 495 0 124889338 1 ffff880012feab00 20 3 30 10 -‐1 9: 0100007F:1111 0100007F:B192 01 00000000:00000000 00:00000000 00000000 495 0 124889339 1 ffff88003da09400 20 3 29 10 -‐1 10: E2010A0A:84B2 1D020A0A:07D4 01 00000000:00000000 00:00000000 00000000 0 0 123087019 1 ffff88003b7b73c0 20 0 0 10 -‐1 11: 301D3348:AF34 98A41F32:0050 06 00000000:00000000 03:000008E2 00000000 0 0 0 2 ffff88003daf2580 12: 301D3348:0016 16C52D60:C17D 01 00000000:00000000 02:000022D6 00000000 0 0 146400073 4 ffff88003d8c6d40 20 3 19 8 7 IP Address - in hex Port - in hex

netstat Example # netstat -‐n Active Internet connections (w/o
servers) Proto Recv-‐Q Send-‐Q Local Address Foreign Address State tcp 0 0 1.2.3.4:44605 4.3.2.1:80 TIME_WAIT tcp 0 0 127.0.0.1:45458 127.0.0.1:4369 ESTABLISHED tcp 0 0 127.0.0.1:4369 127.0.0.1:45458 ESTABLISHED tcp 0 176 1.2.3.4:22 96.45.197.22:49533 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55278 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55280 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55263 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55276 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55269 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55290 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55288 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55267 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55274 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55261 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55265 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55259 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55286 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55282 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55284 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55271 ESTABLISHED LIST ACTIVE CONNECTIONS

netstat Example # netstat -‐n Active Internet connections (w/o
servers) Proto Recv-‐Q Send-‐Q Local Address Foreign Address State tcp 0 0 1.2.3.4:44605 4.3.2.1:80 TIME_WAIT tcp 0 0 127.0.0.1:45458 127.0.0.1:4369 ESTABLISHED tcp 0 0 127.0.0.1:4369 127.0.0.1:45458 ESTABLISHED tcp 0 176 1.2.3.4:22 96.45.197.22:49533 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55278 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55280 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55263 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55276 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55269 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55290 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55288 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55267 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55274 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55261 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55265 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55259 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55286 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55282 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55284 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55271 ESTABLISHED Who is connecting to who?

netstat Example # netstat -‐nlpt Active Internet connections (only
servers) Proto Recv-‐Q Send-‐Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 1098/rpcbind tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1216/sshd tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1383/master tcp 0 0 0.0.0.0:51234 0.0.0.0:* LISTEN 17572/python tcp 0 0 0.0.0.0:5666 0.0.0.0:* LISTEN 1307/nrpe tcp 0 0 :::111 :::* LISTEN 1098/rpcbind tcp 0 0 :::80 :::* LISTEN 13997/httpd tcp 0 0 :::22 :::* LISTEN 1216/sshd tcp 0 0 :::443 :::* LISTEN 13997/httpd # netstat -‐nlpt Active Internet connections (only servers) Proto Recv-‐Q Send-‐Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 1065/rpcbind tcp 0 0 0.0.0.0:4369 0.0.0.0:* LISTEN 2437/epmd tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1178/sshd tcp 0 0 0.0.0.0:55672 0.0.0.0:* LISTEN 23755/beam.smp tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 29134/master tcp 0 0 0.0.0.0:60669 0.0.0.0:* LISTEN 23755/beam.smp tcp 0 0 0.0.0.0:51234 0.0.0.0:* LISTEN 3954/python tcp 0 0 0.0.0.0:5666 0.0.0.0:* LISTEN 2312/nrpe tcp 0 0 :::5672 :::* LISTEN 23755/beam.smp tcp 0 0 :::111 :::* LISTEN 1065/rpcbind tcp 0 0 :::22 :::* LISTEN 1178/sshd FIND THE LISTENING PROCESSES

netstat Example # netstat -‐n | grep 5672 tcp
0 0 172.16.60.5:5672 172.16.60.6:55278 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55280 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55263 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55276 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55269 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55290 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55288 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55267 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55274 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55261 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55265 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55259 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55286 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55282 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55284 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55271 ESTABLISHED # netstat -‐nlpt | grep 5672 tcp 0 0 0.0.0.0:55672 0.0.0.0:* LISTEN 23755/beam.smp tcp 0 0 :::5672 :::* LISTEN 23755/beam.smp MATCH THE LISTENERS TO THE CONNECTIONS These would represent the edges of a graph (source and destination

netstat Example server1# netstat -‐n | grep 5672 tcp
0 0 172.16.60.5:5672 172.16.60.6:55278 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55280 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55263 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55276 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55269 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55290 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55288 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55267 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55274 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55261 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55265 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55259 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55286 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55282 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55284 ESTABLISHED tcp 0 0 172.16.60.5:5672 172.16.60.6:55271 ESTABLISHED server2# netstat -‐n | grep 5672 tcp 0 0 172.16.60.6:55261 172.16.60.5:5672 ESTABLISHED tcp 0 0 172.16.60.6:55288 172.16.60.5:5672 ESTABLISHED tcp 0 0 172.16.60.6:55265 172.16.60.5:5672 ESTABLISHED tcp 0 0 172.16.60.6:55274 172.16.60.5:5672 ESTABLISHED tcp 0 0 172.16.60.6:55286 172.16.60.5:5672 ESTABLISHED tcp 0 0 172.16.60.6:55267 172.16.60.5:5672 ESTABLISHED tcp 0 0 172.16.60.6:55278 172.16.60.5:5672 ESTABLISHED tcp 0 0 172.16.60.6:55271 172.16.60.5:5672 ESTABLISHED tcp 0 0 172.16.60.6:55259 172.16.60.5:5672 ESTABLISHED tcp 0 0 172.16.60.6:55284 172.16.60.5:5672 ESTABLISHED tcp 0 0 172.16.60.6:55282 172.16.60.5:5672 ESTABLISHED tcp 0 0 172.16.60.6:55280 172.16.60.5:5672 ESTABLISHED tcp 0 0 172.16.60.6:55290 172.16.60.5:5672 ESTABLISHED tcp 0 0 172.16.60.6:55263 172.16.60.5:5672 ESTABLISHED tcp 0 0 172.16.60.6:55269 172.16.60.5:5672 ESTABLISHED tcp 0 0 172.16.60.6:55276 172.16.60.5:5672 ESTABLISHED TWO SIDES TO EACH CONNECTION

Tying it altogether To generate a graph: Grab a list
of all servers in an architecture - these are the nodes. Log into each server and grab a list of both listening processes, and active connections. For each listening process, ﬁnd all connections from external servers connecting to it. These are the edges. Output this data to a format of your choosing (.dot ﬁle, CSV, etc...)

The result?

Questions? Hany Fahim (@iHandroid)

Self-Documenting Architectures

Self-Documenting Architectures

VM Farms Inc.

Transcript