James Abel – PyBay Aug 2017 Latus® A Personal Cloud Storage App written in Python www.github.com/jamesabel/latus James Abel Aug 12, 2017 [email protected] @jamesabel www.abel.co Latus is a registered trademark of James Abel

James Abel – PyBay Aug 2017 Intro • James Abel • HW/SW consultant • Former Intel Principal Engineer • Some Python for several years (otherwise C/C++, ASM, etc.) • Python ‘go to’ programming language for the last few years

James Abel – PyBay Aug 2017 Agenda • What is latus? • Why should I care? • Cloud based File Synchronization • File system events • Event database • What’s in the cloud • Synchronization • Encryption • Task Bar Application • Using latus • Next Steps • Summary latus is on github at www.github.com/jamesabel/latus

James Abel – PyBay Aug 2017 What is latus? • File storage and synchronization across your computers • AKA cloud storage • Automatically mirrors files across your computer and to the cloud • Make files accessible • Inherent backup • App that runs in the background (‘client sync’) • Open Source (GPLv3) • “Zero Knowledge” Encryption • Inherent versioning • Python 3.6 • Utilizes lots of awesome packages! Latus is a cloud file sync app written in Python

James Abel – PyBay Aug 2017 Why should I care? • You like cloud storage file sync, but you want .. • Open Source • Python • Zero Knowledge Encryption to the cloud • Control over exactly where files are stored in the cloud (e.g. for regulatory, compliance and/or business requirements) • Control over versioning history • Control over what takes up space in the cloud in the long term • Custom integrated applications • ‘Independent Web’ • Provide a customized branded (‘white label’) cloud storage/sync offering • Or just example uses of several awesome packages and capabilities Open Source File Sync written in Python!

James Abel – PyBay Aug 2017 What is file sync? Computer A Computer B The Internet a.txt “latus” folder “latus” folder

James Abel – PyBay Aug 2017 What is file sync? Computer A Computer B The Internet a.txt a.txt “latus” folder “latus” folder

James Abel – PyBay Aug 2017 Cloud Based File Sync File System Events  Database  Cloud  Synchronization Local File System (PC/Mac) Watchdog Event Database (AWS DynamoDB + local SQLite cache) Events Sync Engine File Cloud Storage (AWS S3) Files Computer A Cloud Computer B Local File System (PC/Mac)

James Abel – PyBay Aug 2017 Cloud Based File Sync Local File System (PC/Mac) Watchdog Event Database (AWS DynamoDB + local SQLite cache) Events Sync Engine File Cloud Storage (AWS S3) Files (encrypted) Computer A Cloud Computer B Local File System (PC/Mac) Step 1: File “a.txt” created

James Abel – PyBay Aug 2017 Cloud Based File Sync Local File System (PC/Mac) Watchdog Event Database (AWS DynamoDB + local SQLite cache) Events Sync Engine File Cloud Storage (AWS S3) Computer A Cloud Computer B Local File System (PC/Mac) Step 2.a.: “a.txt Created” Event Step 2.b.: “a.txt” to S3 Files (encrypted)

James Abel – PyBay Aug 2017 Step 3: “a.txt” mirrored Cloud Based File Sync Local File System (PC/Mac) Watchdog Event Database (AWS DynamoDB + local SQLite cache) Events Sync Engine File Cloud Storage (AWS S3) Computer A Cloud Computer B Local File System (PC/Mac) Files (encrypted) Events Files (decrypted)

James Abel – PyBay Aug 2017 mivui – Monotonically Increasing Value • Events are strictly sequenced • Monotonically Increasing Value in Micro-Seconds (uS) since Epoch as an Integer • Equivalent to: int(round(time.time()*1E6)) • Integer allows reliable comparison and DB indexing/sort key • Server provides mivui • Optionally can be locally created (but is then not absolutely guaranteed to be monotonically increasing) • http://api.abel.co/miv { "mivui": 1502435085769804, "toc": 0, "mivf": 1502435085.769804, "dur": 0.0018029212951660156, "success": true, "mivs": "1502435085.769804“ }

James Abel – PyBay Aug 2017 File System Event Database Items • originator – node (computer) where this event was detected (normally a UUID) • mivui – monotonically increasing integer in micro-seconds from epoch (from server) • detection – how was event detected – initial scan, watchdog, periodic poll • event_type – create, delete, move, modify • file_hash – SHA512 of file • file_path – file path in the latus folder • mtime – mod time of the file • size – file size (in bytes) • src_path – source in the case of moves (otherwise unused)

James Abel – PyBay Aug 2017 Cloud and Local event databases Cloud database is ‘the’ database – local is a cache Cloud (AWS DynamoDB) Local Computer Nodes (SQLite cache) a b AWS SQLite AWS DynamoDB

James Abel – PyBay Aug 2017 Synchronization • Sync is hard • Compute (algorithm) on the client • All nodes see the events on all other nodes and use the same algorithm • ‘winner’ is based on ordered events • Default algorithm is that the most recent event (across all nodes) wins • Other algorithms are possible • Files are available in the cloud to provide them locally • Encrypted in the cloud

James Abel – PyBay Aug 2017 Sync example $vi a.txt $vi b.txt $mv a.txt c.txt $rm b.txt DynamoDB S3 class DetectionSource(IntEnum): unknown = 0 initial_scan = 1 watchdog = 2 periodic_poll = 3 class LatusFileSystemEvent(IntEnum): created = 1 modified = 2 deleted = 3 moved = 4 Winner!

James Abel – PyBay Aug 2017 Encryption • Keep unencrypted files off the internet/cloud • Uses the most excellent Python cryptography package • Key exists on client nodes (computers) • Local computer is assumed secure • In the cloud file contents are stored as AWS S3 objects • S3 object names are SHA512 hashes of the files contents ‘salted’ with encryption key • Thwarts dictionary lookup attacks • For security, the encryption key should not be transferred over the internet • Keep the key ‘out of band’ • Once generated, a key can be exported to a file (simple JSON format) • USB stick works well (put it in a safe place!) • On first run: • Makes a new encryption key if no one exists or • Asks the user for an existing key to be imported

James Abel – PyBay Aug 2017 boto3 • Amazon’s Python library to access Amazon Web Services (AWS) • AWS credentials stored on the local file system or passed in explicitly to boto3 APIs • ~/.aws/credentials [default] aws_access_key_id = YOUR_ACCESS_KEY aws_secret_access_key = YOUR_SECRET_KEY region=us-west-1 # means Northern California • Access AWS import boto3 s3 = boto3.resource('s3’) # Upload a new file data = open('test.jpg', 'rb') s3.Bucket('my-bucket').put_object(Key='test.jpg', Body=data)

James Abel – PyBay Aug 2017 AWS S3 (with encryption) ‘a’ This object’s contents contains the bytes corresponding to a file with contents ‘b’ latusstorage S3 bucket

James Abel – Pyninsula Mar 2017 latus is a Task Bar App …

James Abel – Pyninsula Mar 2017 What is a Task Bar Application? • A taskbar is an element of a graphical user interface which has various purposes. It typically shows which programs or applications are running on the device, as well as provide links or shortcuts to other programs or places, such as a start menu, notification area, and clock. https://en.wikipedia.org/wiki/Taskbar • AKA System Tray

James Abel – Pyninsula Mar 2017 PyQt • Qt (/kjuːt/ "cute"[7][8][9]) is a cross-platform application framework that is used for developing application software that can be run on various software and hardware platforms with little or no change in the underlying codebase, while still being a native application with native capabilities and speed. • https://en.wikipedia.org/wiki/Qt_(software) • What is PyQt? • PyQt is a set of Python v2 and v3 bindings for The Qt Company's Qt application framework and runs on all platforms supported by Qt including Windows, OS X, Linux, iOS and Android. • https://riverbankcomputing.com/software/pyqt/intro • Currently using PyQt5

James Abel – Pyninsula Mar 2017 Hello World Task Bar App class HelloWorldSystemTray(QSystemTrayIcon): def __init__(self): import icons icon = QIcon(QPixmap(':icon.png')) super().__init__(icon) menu = QMenu() menu.addAction("About").triggered.connect(self.about) menu.addAction("Exit").triggered.connect(self.exit) self.setContextMenu(menu) def about(self): about_box = QDialog() layout = QGridLayout(about_box) layout.addWidget(QLabel('hello world')) about_box.setLayout(layout) about_box.show() about_box.exec_() def exit(self): QApplication.exit() app = QApplication(sys.argv) app.setQuitOnLastWindowClosed(False) # so popup dialogs don't close the system tray icon system_tray = HelloWorldSystemTray() system_tray.show() app.exec_()

James Abel – PyBay Aug 2017 •Latus Task Bar App class LatusSystemTrayIcon(QSystemTrayIcon): def __init__(self, app, latus_appdata_folder): self.app = app self.latus_appdata_folder = latus_appdata_folder menu = QMenu(parent) menu.addAction("Open Latus Folder").triggered.connect(self.open_latus_folder) menu.addAction("Preferences").triggered.connect(self.preferences) menu.addAction("Export Latus Key").triggered.connect(self.export_latus_key) menu.addAction("About").triggered.connect(self.about) menu.addAction("Nodes").triggered.connect(self.nodes) menu.addAction("Exit").triggered.connect(self.exit) self.setContextMenu(menu) def start_latus(self): self.sync = latus.aws.sync_aws.Sync(self.latus_appdata_folder) def show(self): QSystemTrayIcon.show(self) def exit(self): self.sync.request_exit() QApplication.exit() app = QApplication(sys.argv) system_tray = LatusSystemTrayIcon(app, latus_appdata_folder) system_tray.start_latus() system_tray.show() app.exec_()

James Abel – PyBay Aug 2017 Preferences • Local preferences kept in preferences.db • SQLite DB • Latus crypto key • Latus folder path • AWS configuration (credentials, region) • Accessible via icon pulldown

James Abel – PyBay Aug 2017 Using latus • Can be cloned and executed from source • www.github.com/jamesabel/latus • latus as a native app • latus uses many packages – it can be a challenge to freeze/install • cryptography and PyQt can be problematic to freeze • Embedded Python interpreter tends to work best • Freezers/Installers • Windows: pynsist, osnap using the ‘embedded’ Python interpreter • MacOS: osnap currently latus can be run for source or as a native app for Windows and MacOS

James Abel – PyBay Aug 2017 Hacking latus • Fork from • www.github.com/jamesabel/latus • Run regression tests • Uses py.test, localstack • Has create, delete, move, etc. • But can use more tests. ☺ • Pull requests • File issues

James Abel – PyBay Aug 2017 Some Next Steps … • Add AWS pubsub to get file system events instantly (AWS SNS/SQS) •Cloud storage efficiency enhancements • Select folders for ‘offline only’ •Node management • Filter out special files (e.g. “.*”) • Code review ☺ Try out latus! Its on github at www.github.com/jamesabel/latus

James Abel – PyBay Aug 2017 Summary and Thank You • latus is an Open Source Cloud Storage app in Python 3.6 • Hack on latus at SF Python Project Night, chat at Pyninsula or Bay Piggies • Thank you • Glyph • Thomas Kluyver Try out latus! Its on github at www.github.com/jamesabel/latus