Upgrade to Pro — share decks privately, control downloads, hide ads and more …


19dda97830e691285bec1201483741d3?s=47 green
January 03, 2020




January 03, 2020


  1. Tesla: a study of a security trainwreck Me Myname TBD

    * Some names and brands may be claimed as the property of others. Internal use only
  2. Tesla Model S (and later X) • First available for

    sale in mid 2012 • Sophisticated (and somewhat excessive) internal network • Multiple Linux and other OS nodes • Very secretive vendor that keeps silent about internal details • First confirmation the system runs Linux obtained on Mar 2, 2014 • https://teslamotorsclub.com/tmc/threads/successful-connection-on-the-model-s- internal-ethernet-network.28185/ • That was the day people found the king had no clothes • Another early story: • https://blog.lookout.com/hacking-a-tesla 2
  3. Original networking structure 3

  4. Findings from 2014 • Both IC and CID are running

    Debian/Ubuntu 12.04 arm version • Almost stock, lots of unneeded services • Open X servers on both • Hardcoded passwords for root and some other users in sudoers • Connecting to “diag ethernet” is all you really need to own the whole car • no code signing anywhere • High level architecture somewhat sensible with critical functionality behind hardware firewall (the gateway) • Several rounds of whack-a-mole ensued immediately. • Diag ethernet locked, default passwords disabled 4
  5. Networking connection 5

  6. 6

  7. Enter me • mid January 2017 I am buying CID

    and IC from a salvaged car on Ebay • To confirm there really is Linux being used and GPL is being violated • System has firmware 2.9.154 which dates it sometime into Jan 2016 • Well known static ssh private key for root on IC still works • The version is old enough to contain a privileged user password from CID • Thus I quickly gain entry onto both systems to explore on my own 7
  8. What I found • Cid and IC run same firmware

    (/usr squashfs) • Different initramfs for minimal /etc and /bin,/sbin • Every binary comes with full debugging symbols • I certainly appreciate the convenience! • Both nodes run updater process • Built from same source determines what to do based on own filename. cid- updater/ic-updater/…. • Single entry point for all systems • Every car process runs a web server with soap-like variables exposed, no authentication • Cannot install newer firmware updates due to missing components. 8
  9. What I found - continued • Kernels are still unsigned

    • But image is obfuscated with reversed order of targets and compressed with unconventional encryptor (lzf) • /usr Squashfs integrity only checked at update install time, never again. • Tesla1/2 users are the only active users. Password rotated every 24 hours • New password is then sent to Tesla over VPN • The password is stored in clear text for other uses • Static ssh host key on both nodes • Easy to spoof the password with ssh mitm proxy 9
  10. What I found - continued • Lots of development functionality:

    • Gdb, strace, … • Lots of development scripts, many unused, with priceless comments about functionality of various pieces • Rich fat chance! • It was a trap 10
  11. What I found - updater • Compromise once use everywhere

    • Basic downgrade protection at update install time • With critical flaws • Shows quite a bit of status and allows things like firmware updates with no authentication • A lot of functionality hidden behind authentication • /var/etc/sacces/tesla[12] contain the auth token • Allows to run anything via system() when authenticated • If no token present, allows any commands with no authentication. • Exposes cleartext http and cli interfaces 11
  12. What I found – recovery mode • Firmware contained cid-recovery-kernel.img

    • Unpacking initramfs from there we can see it has a shell on serial console and updater with no tokens in the image • Looking for a way to trigger recovery mode found Gateway analysis from TenCent: • https://2016.zeronights.ru/wp- content/uploads/2016/12/Gateway_Internals_of_Tesla_Motors_v6.pdf • Fascinating read. Hardcoded telnet password! • To trigger recovery mode you just send a single two-byte UDP packet 12
  13. What I found – Gateway config • Gateway contains a

    clear text unsigned car configuration file • Lists hardware and features the car has, including “soft” features • No integrity checks • Rich scripts to manipulate the features and otherwise put/get files from gateway are present in CID. • Changing this file allows you to: • Make the car faster • Increase battery capacity (for software-locked batteries) • Enable autopilot • Switch car region • Many more desirable things 13
  14. What I found – Development practices • Single codebase is

    used for all car products • As such constant leaks about upcoming functionality / hw updates • Updated performance and battery models (P100D existence) • Updated hardware (hw2.5, model 3 details, …) • All happening at worst possible moments and hurts Tesla business • Security practices nonexistent • Regular reintroduction of security problems • No security thinking outside of small overworked ”Security” group. • Security by obscurity is a paramount paradigm. 14
  15. What I found – Development practices 2 • Little to

    no integration testing • Leading to regular breakage on new firmwares deployed to customer cars. • Unwittingly corroborated by a former QA team member admitting they only do unit tests. https://teslamotorsclub.com/tmc/posts/2173773/ • Firmware releases deployed into weekends • Extra fun when it’s the one that breaks everybody • Big disconnect from SW team to support team • Service center people don’t know much if anything about inner software workings, need to talk to overworked SW engineers • Software bugs sometimes are taken for HW bugs leading to unnecessary repairs/parts replacements. 15
  16. The car • I got a car on March 31,

    2017 • Latest firmware, latest autopilot hardware • Autopilot hardware is yet another Linux, this time arm64 • The static ssh key on IC still works! • But no CID access tokens there anymore • Updater still buggy • Recovery mode still there 16
  17. Updated findings with new firmware • Still same old base

    Ubuntu • Kernel updated to 4.4 in May 2017 (but nothing else) • Updater includes tftp server now • Allows to fetch any(!) file – Fixed in May 2017 • Updater includes a module to download and install autopilot firmware • Different versioning scheme • No other major security-related changes 17
  18. Updated car network since Oct 2016 18

  19. New autopilot (APE) • CID updater stores autopilot firmware in

    the clear in /home/cid-updater • So we get to see it’s arm64 Nvidia PX2 platform • It’s custom tailored Buiildbox ][ 2016.2 based distro with nothing unnecessary included • It has the updater (!) • All useful development functionality is either missing or is behind fuse checks. • Pretty verbose init scripts • Full debug symbols enabled binaries. • Every component is signed • Firmware is only verified at install time and there is no downgrade protection 19
  20. Autopilot (APE) - continued • Root with no password by

    default on boot (!) • But if token exists (generated at factory), the root is then locked. • Initially token was generated once at factory and forever • As a safety CID would periodically try to login to ape as root with no password and set a random token • In Aug 2017 Tesla finally switches to once per boot tokens on APE • Root has authorized_keys with a public key in there. • Recovery mode (!) • Everything is shrouded in secrecy at Tesla side to hide how deficient the new system is 20
  21. Autopilot – the break-in • Convenient updater bug that tricks

    it into serving any file • Download the tesla1 token, authenticate with updater • Use updater system commands to allow passwordless root login. • Break-in #2 after token policy change: • Downgrade the firmware to one with the bug above and reexploit it again • Use Linux filesystem property to prevent token modifications by boot process 21
  22. Autopilot – the leaks • Code is shared across all

    modern Tesla vehicles • A lot of details about updated autopilot hardware v2.5 becomes known before model 3 reveal • Forces Tesla hand to acknowledge updated hardware and it’s eventual coming to models S and X (many unhappy owners, Tesla tries to downplay importance of the update). • One of the versions contains a spurious private ssh key • The public version later surfaces in authorized_keys file for root on the added backup autopilot node making it trivially exploitable. 22
  23. Autopilot – the security struggle • Once Autopilot compromise becomes

    known, furious activity to keep the hacker out • Switch to CA-based ssh keys • Rotated tokens at every reboot • Getting away from tokens and switching to challenge response based on public key cryptography in updater • All the above ineffective since the entire premise of no persistence was wrong. • In mid December 2017 finally dm-verity is deployed • Still ineffective due to like I would tell you 23
  24. Tesla Network Security • Sloppy security practices evident by: •

    Constant leakage of sensitive information due to improper permission controls • No insider oversight leading to insiders in low positions having access to and leaking high value information • AWS servers for map data exposing private sshd keys • Map data is synced by rsync, failure to contain rsync reach • ’Toolbox’ servers serving diag unlock tokens and other car data to all comers, no authentication checks. Later, no account validity checks. 24
  25. Security – the sudden change • Mid 2017 evident security

    practices appear • Stripped binaries • Improvements in updater auth logic • Coincident with Intel Gordon-peak based model 3 system release • Could it be more than just coincidence? • But CID/IC appears to be a write-off • Silently dropped from bounty program • Many security holes remain unclosed. 25
  26. Tesla bounty program • Good on the surface • Decent

    payout up to $10k per vulnerability • Interfering with researches – a solid shot in the foot • Everybody reporting serious car security issues get on a secret “black list” • Loses access to regular firmware updates • Service becomes a problem – sudden long delays • SC tools stop working until “Engineering” provides approval for every mundane operation. • Remote access to test systems is never available 26
  27. Questions? •Questions? 27