Keep an Eye on the Sky with Nerves and Phoenix

As part of the next generation air traffic control system, regulators in the United States and abroad will require most aircraft to be equipped with avionics by 2020 that automatically broadcast identity, position, and velocity data at regular intervals using a standard radio messaging protocol. This technology, known as Automatic Dependent Surveillance Broadcast (ADS-B), is already in use by the majority of aircraft today and can easily be received on the ground by anyone using cheap Software Defined Radio (SDR) hardware purchased online.

Because ADS-B messages can come from hundreds (if not thousands) of aircraft operating in the sky in parallel at any given time, Elixir is well suited to elegantly handle the problem of tracking them all concurrently in realtime. In this talk, we'll explore highly concurrent approaches to solving this problem using the tools and techniques offered by OTP and GenStage. We'll examine how SDR hardware support is provided using Nerves on an embedded Linux platform such as Raspberry Pi. Finally, we'll take a look at some ways we can visualize all of this flight activity on the web in realtime using Phoenix.

Links

Squitter app: https://github.com/electricshaman/squitter
Parts List: https://www.dropbox.com/s/prkyoin528ylvcz/Squitter%20Parts%20List.pdf?dl=0
ADS-B on FlightAware: http://flightaware.com/adsb/
ADS-B Decoding Guide: https://adsb-decode-guide.readthedocs.io/en/latest/index.html
Custom Nerves System for SDR on Raspberry Pi 3:
https://hex.pm/packages/nerves_system_rpi3_sdr
LeafletJS: http://leafletjs.com/
Hobbyist's Guide to the RTL-SDR: https://www.amazon.com/Hobbyists-Guide-RTL-SDR-Software-Defined/dp/1514716690
RTL-SDR Amazon.ca: https://www.amazon.ca/gp/product/B01GDN1T4S
RTL-SDR Amazon.com: https://www.amazon.com/dp/B01GDN1T4S

Transcript

1. KEEP AN EYE ON THE SKY WITH NERVES AND PHOENIX PHOTO: Chris-Håvard Berge
2. Software Engineer at Previously: Ultimate Software, Oscium, Chesapeake Energy, LexisNexis, … Nerves core team member ❤ Space exploration and astronomy, STEM, hockey, and waffles
 Github and Twitter: @electricshaman Elixir Slack: @jeffsmith ABOUT ME
3. SQUITTER CODE github.com/electricshaman/squitter PARTS
 Squitter Parts List (Dropbox)
4. TRACKING AIRCRAFT FOR FUN (AND NOT MUCH PROFIT) Learn about the operational aspects of aviation Learn about RF using Software Defined Radio (SDR) Experiment with simple DIY antennas Impress your friends and coworkers Scare your neighbors
6. TRACKING AIRCRAFT FOR FUN (AND NOT MUCH PROFIT) Learn about the operational aspects of aviation Learn about RF using Software Defined Radio (SDR) Experiment with simple DIY antennas Impress your friends and coworkers Scare your neighbors Get a free FlightAware enterprise account*
8. Con Air? Creator of Elixir
9. WHAT IS ADS-B? Automatic Dependent Surveillance Broadcast Satellite-based (GPS) successor to radar “ADS-B Out” mandatory by 2020 in US High-rate broadcast of operational data by aircraft on 1090 MHz (1090ES) Message-based: Different types encoded by type code values
10. CORE MESSAGE TYPES FOR TRACKING Message Type Broadcast Frequency Fields of Interest Aircraft Identification 1x ~ 5 sec Callsign Aircraft Category Airborne Position 2x ~ 1 sec Altitude Latitude Longitude Airborne Velocity 2x ~ 1 sec Heading Airspeed Vertical Rate
11. Format 5 bits Capability 3 bits Address 24 bits Payload 56 bits Parity 24 bits ADS-B Message 112 bits MESSAGE STRUCTURE <<format :: 5, ca :: 3, addr :: 3-bytes, payload :: 7-bytes, par :: 24>>
12. Format 5 bits Capability 3 bits Address 24 bits Payload 56 bits Parity 24 bits ADS-B Message 112 bits MESSAGE STRUCTURE Format 5 bits Capability 3 bits Address 24 bits Payload 56 bits Parity 24 bits ADS-B Message 112 bits 17
13. Format 5 bits Capability 3 bits Address 24 bits Payload 56 bits Parity 24 bits ADS-B Message 112 bits 17 5 MESSAGE STRUCTURE
14. Format 5 bits Capability 3 bits Address 24 bits Payload 56 bits Parity 24 bits ADS-B Message 112 bits 17 5 0x4840D6 MESSAGE STRUCTURE
15. Format 5 bits Capability 3 bits Address 24 bits Payload 56 bits Parity 24 bits ADS-B Message 112 bits 17 5 0x4840D6 4 Type: Aircraft Identification Category: None 11 12 13 49 48 50 51 32 Callsign: KLM1023_ 0 MESSAGE STRUCTURE
16. Format 5 bits Capability 3 bits Address 24 bits Payload 56 bits Parity 24 bits ADS-B Message 112 bits 17 5 0x4840D6 4 Type: Aircraft Identification Category: None 11 12 13 49 48 50 51 32 Callsign: KLM1023_ 0 0x576098 MESSAGE STRUCTURE https://adsb-decode-guide.readthedocs.io/en/latest/content/identification.html
18. ~5,000 aircraft in the air over the US at any given moment ✅ Concurrency Isolated units of behavior with transient state and lifetimes ✅ Processes Asynchronous message-based protocol ✅ Messaging passing Linear/serialized state-based and time-based rules ✅ GenServer, gen_statem Receiver should be as stable and fault tolerant as possible ✅ BEAM Peak air traffic is a firehose ✅ Backpressure, caching Background tasks and lookups needed for decorating data ✅ Stream, Flow, ETS Intense math needed for digital signal processing (DSP) ❌ ELIXIR + ADSB
19. ELIXIR + ADSB: HANDLING DSP ✅ Elixir supports multiple ways of delegating to external applications and libraries DATA CONTROL CONTROL DATA PORTS
 NIFS C NODES
20. ELIXIR + ADSB: HANDLING DSP ✅ Elixir supports multiple ways of delegating to external applications and libraries RTL SDR CONTROL CONTROL DATA PORTS
 NIFS C NODES
21. ELIXIR + ADSB: HANDLING DSP ✅ Elixir supports multiple ways of delegating to external applications and libraries RTL SDR SQUITTER CONTROL DATA PORTS
 NIFS C NODES
22. ELIXIR + ADSB: HANDLING DSP ✅ Elixir supports multiple ways of delegating to external applications and libraries RTL SDR SQUITTER CONTROL DATA dump1090 Note: this is similar to how Erlang interacts with telecom switching hardware.
23. HIGH LEVEL FLOW Squitter dump1090 Decode Messages Raw ADS-B Frames Route Message to Aircraft Process Decoded ADS-B Messages Aircraft Process Aircraft Process Aircraft Process Aircraft GenServer Process ETS Table Squitter.Web Phoenix Channels Phoenix Channels Phoenix Channels Browser Leaflet JS State Projections Active Aircraft Projections Projections Projections
24. Decoder Stage Frame Source Stage Partition Dispatcher Decoder Stage Decoder Stage Decoder Stage Aircraft: ABE03E Aircraft: 4CA9E3 Aircraft: ADE09E Aircraft: 12D10A Aircraft: A7E05B Frame Source (e.g.: dump1090) DECODING PIPELINE: FRAMES PRODUCER
25. PARTITIONING BY ADDRESS def init([host, port, partitions]) do send(self(), :connect) hash_function = fn({_time, frame} = envelope) -> {:ok, address} = Squitter.Decoding.ModeS.icao_address(frame) partition = :erlang.phash2(address, length(partitions)) {envelope, partition} end {:producer, %{host: host, port: port, socket: nil, buffer: []}, dispatcher: {GenStage.PartitionDispatcher, hash: hash_function, partitions: partitions}} end https://github.com/electricshaman/squitter/blob/master/squitter/lib/squitter/avr_stage.ex
26. def handle_info({:tcp, _socket, data}, %{buffer: buffer} = state) do time = System.monotonic_time(:microseconds) {frames, next_buffer} = AVR.split_frames(buffer ++ data) # Use time + index as a logical ordering value indexed_frames = frames |> Enum.map(&hex_to_bin/1) |> Enum.filter(&is_valid_frame/1) |> Enum.with_index(time) |> Enum.map(fn {frame, time} -> {time, frame} end) {:noreply, indexed_frames, %{state | buffer: next_buffer}} end RAW FRAMES IN THROUGH :gen_tcp https://github.com/electricshaman/squitter/blob/master/squitter/lib/squitter/avr_stage.ex
27. Decoder Stage Frame Source Stage Partition Dispatcher Decoder Stage Decoder Stage Decoder Stage Aircraft: ABE03E Aircraft: 4CA9E3 Aircraft: ADE09E Aircraft: 12D10A Aircraft: A7E05B Frame Source (e.g.: dump1090) DECODING PIPELINE: DECODER CONSUMER
28. Decoder Stage Frame Source Stage Partition Dispatcher Decoder Stage Decoder Stage Decoder Stage Aircraft: ABE03E Aircraft: 4CA9E3 Aircraft: ADE09E Aircraft: 12D10A Aircraft: A7E05B Frame Source (e.g.: dump1090) AIRCRAFT GENSERVERS
29. AIRCRAFT UNIQUE REGISTRY def start_link(address) do name = {:via, Registry, {Squitter.AircraftRegistry, address}} GenServer.start_link(__MODULE__, [address], name: name) end https://github.com/electricshaman/squitter/blob/master/squitter/lib/squitter/aircraft.ex
30. DELIVERING THE MESSAGES @doc """ Dispatch the decoded `msg` to an aircraft process registered to `address`. """ def dispatch(address, msg) do cast(address, {:dispatch, msg}) end defp cast(address, msg) do get_aircraft(address) |> GenServer.cast(msg) end defp get_aircraft(address) do case Supervisor.start_child(__MODULE__, [address]) do {:ok, pid} -> pid {:error, {:already_started, pid}} -> pid end end https://github.com/electricshaman/squitter/blob/master/squitter/lib/squitter/aircraft_supervisor.ex
31. RECEIVING & HANDLING THE MESSAGES def handle_cast({:dispatch, msg}, state) do case handle_msg(msg, state) do {:ok, new_state} -> new_state = set_received(new_state) new_state |> build_view |> report {:noreply, new_state} {:error, _} -> {:noreply, state} end end https://github.com/electricshaman/squitter/blob/master/squitter/lib/squitter/aircraft.ex
32. BUILDING AND STASHING THE VIEW defp build_view(state) do state |> Map.take([:callsign, :registration, :latlon, :msgs, :category, :altitude, :velocity_kt, :heading, :vr, :address, :age, :distance, :country]) |> Map.put(:position_history, Enum.reverse(state.position_history)) end defp report(view) do {:ok, range_limit} = Site.range_limit() if view.msgs > 1 && length(view.position_history) > 0 && view.distance <= range_limit do StateReport.state_changed(view.address, view) end end https://github.com/electricshaman/squitter/blob/master/squitter/lib/squitter/aircraft.ex
33. BUILDING AND STASHING THE VIEW def state_changed(address, ac_state) do GenServer.cast(__MODULE__, {:state_changed, address, ac_state}) end def handle_cast({:state_changed, address, ac_state}, {table, aircraft}) do :ets.insert(table, {address, ac_state}) {:noreply, {table, aircraft}} end https://github.com/electricshaman/squitter/blob/master/squitter/lib/squitter/state_report.ex
34. CLEANING UP supervisor(Registry, [:unique, Squitter.AircraftRegistry, [partitions: System.schedulers_online(), listeners: [Squitter.StateReport]]]) def handle_info({:register, Squitter.AircraftRegistry, address, pid, _}, {table, aircraft}) do ref = Process.monitor(pid) new_aircraft = Map.put(aircraft, ref, address) {:noreply, {table, new_aircraft}} end def handle_info({:unregister, Squitter.AircraftRegistry, address, _pid}, {table, aircraft}) do # Delete from ETS and demonitor. # (truncated) {:noreply, {table, new_aircraft}} end def handle_info({:DOWN, ref, :process, _pid, _reason}, {table, aircraft}) do # Delete from ETS. # (truncated) {:noreply, {table, new_aircraft}} end https://github.com/electricshaman/squitter/blob/master/squitter/lib/squitter/state_report.ex
35. NERVES WITH RTL SDR ON RASPBERRY PI 3 Custom system {:nerves_system_rpi3_sdr, "~> 0.15.1”} Packages included • librtlsdr • gpsd • Liquid DSP • dump1090 (custom package) https://github.com/electricshaman/nerves_system_rpi3_sdr https://hex.pm/packages/nerves_system_rpi3_sdr
36. ADDING A CUSTOM PACKAGE TO A NERVES SYSTEM: STEP 1 Add Config.in file to a new custom package/ subfolder in your system’s source config BR2_PACKAGE_DUMP1090 bool "dump1090" depends on BR2_TOOLCHAIN_HAS_THREADS select BR2_PACKAGE_LIBRTLSDR help Simple Mode S decoder for RTLSDR devices. https: //github.com/mutability/dump1090 comment "dump1090 needs a toolchain w/ threads" depends on !BR2_TOOLCHAIN_HAS_THREADS e.g.: nerves_system_rpi3_sdr/package/dump1090/Config.in https://github.com/electricshaman/nerves_system_rpi3_sdr/blob/master/package/dump1090/Config.in
37. ADDING A CUSTOM PACKAGE TO A NERVES SYSTEM: STEP 2 Add makefile to the custom package/ subfolder in your system’s source e.g.: nerves_system_rpi3_sdr/package/dump1090/dump1090.mk (full contents on next slide) https://github.com/electricshaman/nerves_system_rpi3_sdr/blob/master/package/dump1090/dump1090.mk
38. ################################################################################ # # dump1090 # ################################################################################ DUMP1090_VERSION = b155fdb458c3241ab375d1f2b12fbb6a9f8a8a3a DUMP1090_SITE = $(call github,mutability,dump1090,$(DUMP1090_VERSION)) DUMP1090_LICENSE = GPL-2.0+ DUMP1090_LICENSE_FILES = COPYING DUMP1090_DEPENDENCIES = libusb librtlsdr define DUMP1090_BUILD_CMDS $(MAKE) $(TARGET_CONFIGURE_OPTS) -C $(@D) all endef define DUMP1090_INSTALL_TARGET_CMDS $(INSTALL) -D -m 0755 $(@D)/dump1090 $(TARGET_DIR)/usr/bin $(INSTALL) -D -m 0755 $(@D)/view1090 $(TARGET_DIR)/usr/bin endef $(eval $(generic-package)) nerves_system_rpi3_sdr/package/dump1090/dump1090.mk (see Buildroot docs here)
39. ADDING A CUSTOM PACKAGE TO A NERVES SYSTEM: STEP 3 Add Config.in to your system source’s root that sources custom Config.in files e.g.: nerves_system_rpi3_sdr/Config.in source "$NERVES_DEFCONFIG_DIR/package/dump1090/Config.in" https://github.com/electricshaman/nerves_system_rpi3_sdr/blob/master/Config.in
40. ADDING A CUSTOM PACKAGE TO A NERVES SYSTEM: STEP 4 Add external.mk file to your custom source’s root that includes other makefiles e.g.: nerves_system_rpi3_sdr/external.mk include $(sort $(wildcard $(NERVES_DEFCONFIG_DIR)/package/*/*.mk)) https://github.com/electricshaman/nerves_system_rpi3_sdr/blob/master/external.mk
41. ADDING A CUSTOM PACKAGE TO A NERVES SYSTEM: STEP 5 Select your custom package in your nerves_defconfig file e.g.: nerves_system_rpi3_sdr/nerves_defconfig +BR2_PACKAGE_DUMP1090=y https://github.com/electricshaman/nerves_system_rpi3_sdr/blob/master/nerves_defconfig
42. ADDING A CUSTOM PACKAGE TO A NERVES SYSTEM: STEP 6 PROFIT!
43. PHOENIX CHANNEL defmodule Squitter.Web.AircraftChannel do use Squitter.Web, :channel require Logger def join("aircraft:" <> _key, _payload, socket) do {:ok, socket} end def handle_in("roger", payload, socket) do # We only want to send position history on the first message push(socket, "state_report", %{aircraft: state_reports(true)}) schedule_report_push() {:reply, {:ok, payload}, socket} end def handle_in(_other, _payload, socket) do {:noreply, socket} end
44. def handle_info(:send_report, socket) do push(socket, "state_report", %{aircraft: state_reports()}) schedule_report_push() {:noreply, socket} end defp state_reports(position_history \\ false) do :ets.tab2list(:state_report) |> Enum.map(fn({_key, state}) -> state end) |> Enum.map(fn(r) -> if position_history, do: r, else: Map.delete(r, :position_history) end) end defp schedule_report_push do Process.send_after(self(), :send_report, 1000) end end PHOENIX CHANNEL (continued) https://github.com/electricshaman/squitter/blob/master/web/lib/squitter_web/channels/aircraft_channel.ex
45. DEMO