Yin Yang Ranch: Building A Distributed Computer Vision Pipeline using Python, OpenCV and ZMQ Jeff Bass

Yin Yang Ranch: A Distributed Computer Vision System • Introduction: Me, my Farm & my Distributed Computer Vision Pipeline • Overall Project Design: Automating the Farm with Raspberry Pi PiCameras • Raspberry Pi “Nodes”: Gathering, selecting & sending images • ZMQ and imageZMQ: High speed, low latency image transfer • Hub Computers: Receiving and storing images & event messages • Questions: Let’s ask & answer questions as issues on my GitHub! (See below)

Building a Small Permaculture Farm in Suburbia • Permaculture is a collection of practices to make farming more sustainable by building living soil, emulating old growth forests and recycling nutrients and water • I am turning my 2 acre suburban lot into a small permaculture farm called Yin Yang Ranch • It is an evolving science project and a demonstration garden; lots of shared learnings with others • Growing a plant polyculture: figs, pears, pomegranates, plums, grapes, avocados, oranges, mulberries, blackberries… • … and a bumper crop of Raspberry Pi’s. • Using Computer Vision as a tool to help manage the farm

Jeff Bass Bio Bullets (College, Career, Retirement) • Started college pursuing EE, got drafted, fixed electronics and meteorological systems while in the army (Electronics) • Went to grad school on GI Bill; dissertation research morphed from econometrics to computer science & became a software company (Computer Science) • Spent 20+ years doing statistics & data science in Big Biotech; large datasets in multiple aspects of healthcare (Big Data) • Currently retired from income producing endeavors: making software, building the farm, bike touring the California Coast • Learning about computer vision, sensors, Raspberry Pi computers as tools to better manage my permaculture farm

How Computer Vision helps manage my small permaculture farm •Reading the water meter; optimizing water use •Counting bees, butterflies and other pollinators •Tracking coyotes, rabbits, raccoons, bobcats and other critters •Monitoring the garage, barn doors and driveway (is that a delivery?) •Tracking sunlight hours, sunlight intensity, clouds, shadows (photosynthesis strength and availability) •Monitoring many non-camera sensors: temperature, humidity, PIR motion sensors, solar panel power output (a large 21KW array) Water Meter Picture

A Distributed Computer Vision Pipeline • Raspberry Pi’s as nodes • 1 or 2 cameras per RPi • 8 to 12 RPi’s per Hub • ZMQ Messaging passes images from RPi’s to Hub • Multiple Hubs feed the Librarian • Communications Hub

• Computer Vision (CV) is a set of of tools and techniques, e.g.: • Capture Images, resize images, transform images, etc. • Detect objects in images (Car? Pedestrian?), read signs, etc. • A Computer Vision Pipeline (CVP) is a Sequence of CV Programs e.g.: • Initialize camera; take picture; resize picture; convert to grayscale, send image. • Receive image; use Convolutional Neural Network to recognize a Coyote, etc. • A Distributed Computer Vision Pipeline (DCVP) is a CVP that runs on multiple computers distributed across a local network…or even distributed across the internet. • My Yin Yang Ranch project is about building a DCVP to manage my small farm. It’s a set of Python programs distributed among 20+ RPi’s & 3 Mac & Linux hubs. What is a Distributed Computer Vision Pipeline?

My Tools for Computer Vision in Python • Software: • Python 3.5 + (we are all using Python 3 now, right?) • Raspbian Linux for 20+ RPis • Linux & MacOS for 3 hubs • Numpy (my best array friend since Fortran, frankly) • OpenCV (2500+ tools for CV; the backbone for doing Python things in CV) • ZMQ and PyZMQ (high speed rail for sending images around the farm) • PiCamera (taught me how digital camera sensors work. Thanks!) • imutils (convenience & helper functions for OpenCV; also taught me a lot) • Electronics: • Raspberry Pi $35 single board computers (RPi’s) and PiCamera modules!! • MOSFETs, PIR motion sensors, DS18B20 temperature sensors, DHT22s • LEDs of many shapes and sizes, including powerful infrared LED floodlights

Image Computing Toolset: OpenCV •Open Source Computer Vision Library •Thousands of CV tools for real time apps •Enables image processing pipelines •Python bindings: “import cv2” •Runs well on Raspberry Pi computers •OpenCV images are Numpy Arrays & easily sent in network messages

imagenode <-> imageZMQ <-> imagehub Pseudocode for imagenode ForeverEventLoop: grab camera image put image in queue analyze queue if SomethingHappened: send event message send images process hub response Pseudocode for imagehub ForeverEventLoop: receive (message, image) store event message store images send response

1 # run this program on the Mac to display image streams from multiple RPis 2 import cv2 3 import imagezmq 4 image_hub = imagezmq.ImageHub() 5 while True: # show streamed images until Ctrl-C 6 rpi_name, image = image_hub.recv_image() 7 cv2.imshow(rpi_name, image) # 1 window for each RPi 8 cv2.waitKey(1) 9 image_hub.send_reply(b'OK') 1 # run this program on each RPi to send a labelled image stream 2 import socket 3 import time 4 from imutils.video import VideoStream 5 import imagezmq 6 7 sender = imagezmq.ImageSender(connect_to='tcp://jeff-macbook:5555') 8 9 rpi_name = socket.gethostname() # send RPi hostname with each image 10 picam = VideoStream(usePiCamera=True).start() 11 time.sleep(2.0) # allow camera sensor to warm up 12 while True: # send images as stream until Ctrl-C 13 image = picam.read() 14 sender.send_image(rpi_name, image) Simple DCV Pipeline Demo— Multiple RPi’s to a Single Mac Node Code (RPi) Hub Code (Mac)

Simple DCV Pipeline Demo— Multiple RPi’s to a Single Mac An 8 RPi Simulcast in 19 lines of Python (11 per RPi; 8 on a Mac)

DCVP Example: Reading my Water Meter What the RPi does: •Turns on LEDs to light the meter •Continuously grabs frames •Does ROI cropping, conversion to grayscale, thresholding, dilation •Detects needle & leak spinner motion •Sends event msg (“water flowing”) •Sends images of event •Xfers images & messages via ZMQ What the Mac does: •Acts as ZMQ hub for multiple RPi’s •Receives and acknowledges images and messages from multiple RPi’s •Adds events messages to events log •Stores and indexes images •Extracts digits to read the meter •Keeps history of events; does stats •Responds to queries about status

DCVP Example: Reading my Water Meter Daylight view with the water meter cover off Full Frame from RPi PiCamera. Lit with 24 low power LEDs to minimize reflections and glare RPi does frame to frame differencing to detect motion of spinning bits in the rectangle Mac does digit masking, digit classification to read the value of the digits (Gotcha: rotating needle can occlude digits)

Distributed Computer Vision Pipeline (revisited) • Raspberry Pi’s as nodes • 1 or 2 cameras per RPi • 8 to 12 RPi’s per Hub • ZMQ Messaging passes images from RPi’s to Hub • Multiple Hubs feed the Librarian • Communications Hub

Raspberry Pi’s are Great (at some things) •Cheap, reliable, fanless, dust tolerant •Physically small (Pi Zero) for sensor enclosures •Very low power; quick restart after power failures •PiCamera has helpful adjustable settings •BUT… •SD Cards not reliable for writing large image files •SD Card writing is VERY slow •Limited memory and limited processing capacity •Slower USB bus: slow Ethernet; slow WiFi •USB webcams on RPi are especially slow and cranky

Macs are Great (at other things) • More computing power for elaborate processing of images • Digits recognition; feature extraction; classification • But…one Mac laptop uses as much power as 8 or 10 RPis • Macs are expensive and not suitable for outdoors • Fast WiFi & Ethernet • Fast SSD storage (for lots of images)

DCVP Example: Reading my Water Meter Raspberry Pi imagenode at Water Meter uses PiCamera to capture & sends images & event messages Linux laptop imagehub receives images & event messages. Then computes water flow & answers SMS text queries imageZMQ methods transfer event messages & images peer-to-peer

Imagenode Code Snippet (Runs on a RPi) 1 # imagenode.py snippet 2 # (all the imports, logging setup, etc. are hidden to fit slide) 3 4 settings = Settings() # get settings for node cameras, ROIs, GPIO 5 node = ImageNode(settings) # start ZMQ, cameras and other sensors 6 7 # forever event loop; capture images; send images & some text about event 8 while True: 9 node.get_sensor_data() # grab camera images and other sensor data 10 node.process_sensor_data() # detect motion; select which images send 11 while len(node.msg_q) > 0: 12 try: 13 with Patience(settings.patience): # wait for hub response 14 text, image = node.msg_q.popleft() 15 hub_reply = node.send_frame(text, image) 16 except Patience.Timeout: # except no timely response from hub 17 hub_reply = node.fix_comm_link() 18 node.process_hub_reply(hub_reply) 19 # ……

There are many imagenode CV settings… Parameter Tuning to optimize detection • Settings can be fine tuned for: • Node name, queue sizes, wait times • Hub addresses for multiple hubs • Cameras • Viewname, resolution, framerate, send quantity • Detectors (motion, light, color, etc) • ROI dimensions and corners • Detection parameters (delta_threshold, area) • Lights (under GPIO control) • Sensors (temperature, humidity, PIR) • I am using YAML files for storing settings • Feels more Pythonic than json or configs • Large settings file becomes a nested Python dict

For each image captured, imagenode: • Applies Transformations: • Resizing the image, Gaussian blur, convert to grayscale as needed • Thresholding, cropping, rotating, selecting a Region of Interest (ROI) • Compares images and Detects Things (may be one or more of): • Detect Motion: Image to image background averaging & subtraction • Detect light or color or changes in either • Detect objects (simple object detection models can be used on RPi) • For each detector specified in the YAML settings file: • Sends images that meet the detector criteria • Sends event messages when the detector state changes • The 2 MOST IMPORTANT jobs for imagenode: 1. Use detector methods to decide which images matter 2. Send ONLY the images that matter (e.g. Water meter starts or stops) • (With many imagenodes sharing the same network, this is crucial) • (Water meter: only sends a few images when needle starts moving or stops)

Image Transport: ZMQ •Fast, memory efficient, low latency •Peer to Peer: No central server or broker process needed •ZMQ makes a great concurrency manager for multiple RPis (using the REQ / REP pattern) •Communications protocol is flexible, but messaging protocol must be well designed •Alternatives considered: RabbitMQ, ROS, AMQP, MQTT …

imageZMQ: Fast Network Transport for images imageZMQ is a set of Python classes that transport OpenCV images from one computer to another using ZMQ messaging. • ImageZMQ allows sending OpenCV images in their native Numpy array format without encoding them to something else. • ImageZMQ uses PyZMQ and builds on PyZMQ code. • ImageZMQ uses the REQ/REP message pattern by default. PUB/SUB is an option. • ImageZMQ can optionally send images which have been compressed to jpgs. • I developed imageZMQ for my own yin-yang-ranch DCVP project. • It has been tested by myself and others for over 3 years. • Production release 1.0.1 is currently pip installable from PyPI. • imageZMQ is open source and available on GitHub. • imageZMQ is pure Python. • imageZMQ’s GitHub repository has the full API, documentation, test programs and examples

1 # run this program on the Mac to display image streams from multiple RPis 2 import cv2 3 import imagezmq 4 image_hub = imagezmq.ImageHub() 5 while True: # show streamed images until Ctrl-C 6 rpi_name, image = image_hub.recv_image() 7 cv2.imshow(rpi_name, image) # 1 window for each RPi 8 cv2.waitKey(1) 9 image_hub.send_reply(b'OK') 1 # run this program on each RPi to send a labelled image stream 2 import socket 3 import time 4 from imutils.video import VideoStream 5 import imagezmq 6 7 sender = imagezmq.ImageSender(connect_to='tcp://jeff-macbook:5555') 8 9 rpi_name = socket.gethostname() # send RPi hostname with each image 10 picam = VideoStream(usePiCamera=True).start() 11 time.sleep(2.0) # allow camera sensor to warm up 12 while True: # send images as stream until Ctrl-C 13 image = picam.read() 14 sender.send_image(rpi_name, image) Simple DCV Pipeline Example (showing imagezmq API) imagezmq sender API imagezmq hub API

Imagehub Code Snippet (Runs on a Mac) 1 # imagehub.py snippet 2 # (all the imports, logging setup, etc. are hidden to fit slide) 3 4 settings = Settings() # settings for Patience, image directories, etc. 5 hub = ImageHub(settings) # start ZMQ, ImageWriter thread, Timers, etc. 6 7 # forever event loop; receive and process images & text from imagenodes 8 while True: 9 try: 10 with Patience(hub.patience): # wait for hub response 11 text, image = hub.receive_next() 12 except Patience.Timeout: # except no timely message from any node 13 hub.handle_timeout() 14 continue 15 reply = hub.process(text, image, settings) 16 hub.send_reply(reply) 17 # …… The imagehub forever loop is somewhat simpler than the imagnode one:

The imagehub receives and stores images The Imagehub loop has just 4 tasks: 1. Receive each incoming (message, image) pair 2. Immediately send the “OK” acknowledgment 3. Store the image (running in a separate thread) 4. Store the message (running in a separate thread) • Each imagehub receives images from 8-12 RPi’s • imagehubs do not do any image analysis, message decoding or reporting; librarian program does those. • The message portion of each (message, image) tuple contains a specific event message format that is decoded by the librarian program • Fast and robust to power failures: laptop with SSD

For each tuple received, imagehub: • Saves image and event message: • Images are typically saved as jpgs (much less disk space) • Event messages are saved using Python logging() • Returns “OK” acknowledgement • The received message tuple: • Contains (message, image) • Message is simple text • Image can be either: • OpenCV image (LARGE), or • jpg encoded from image (small) • imagenode option sets send type • choosing jpg lowers network load • Every message sent MUST: • Identify sender and detector type

DCVP end to end example: the Barn Coyote Cam Glass blocks Infrared, so Mason jars can’t be used for Infrared Cameras. But a PiNoir camera does fine outdoors under a couple of old cedar shingles.

DCVP end to end example: Coyotes & Bobcat by Barn Cam Images taken with an imagenode RPi, using a “PiNoir” Infrared camera, and a 12V 8W Infrared floodlight. These are from the RPi on the back of my barn. (see previous slide) The RPi imagenode captures over 1 million frames a day. But on the 3 days shown here, this imagenode sent only about 300-500 frames total per day. This is an example of a distributed pipeline. The RPi imagnode logic selects a FEW images to send. Which images? The images that meet specific detector criteria set in the imagenode YAML settings file. Fully Distributed CV Pipeline Imagenode running on RPi. Imagehub on a Linux laptop. Object detection on a Mac. Object Detection (confidence) Left: coyote (63%) Right: coyote (47%) Below: bobcat (14%)

Communicating to End Users (me ;-) •Communicating “conclusions" from my computer vision pipeline is done with simple text summaries: •“Water is off; last time flowing was at 7:15AM” •“Garage is closed; last open at 8:20PM yesterday” •“Animal behind Barn at 3:15AM; coyote (47%)” •Communicating uses multiple channels: •Simple terminal CLI text chat interface for testing •SMS texting interface has proved most useful so far •Future: Website “dashboard” for images and data charts

Yin Yang Ranch DCVP: Lessons Learned So Far • Experiment & observe before optimizing; change, document carefully & iterate. Use formal experimental designs and hypothesis testing when making A/B design decisions. • Tune imagenode computer vision parameters carefully (Gaussian blur, thresholds, ROI pixel area, etc.) YAML settings files are a big help. • Get the lighting right! Control the lighting brightness with RPi GPIO (use PWM & MOSFETs). Shadows can cause lots of false positives for motion detectors. Adding a 2nd floodlight can help (it did with my barn cam). • Use PiCameras rather than Webcams: the PiCamera() module has lots of methods to fine tune image capture, including exposure mode, analog gain, digital gain. • Optimize network load: • Send the smallest image size that will get the job done. Many object detection models only need 300px images or less. • Compress images to jpgs; this alone can lower network load 75%. (In many ways, a jpg conversion is like a Gaussian Blur). • Use hardwired ethernet rather than WiFi where possible. Especially for imagehubs. • Send FEWEST images possible to perform a task; for example, imagenode detectors send only “important” images per detector settings. • Reliability planning: recovery from power failures, internet outages, etc.; all imagenodes and imagehubs should recover gracefully and quickly. Run imagehubs on laptops; battery will bridge most power outages.

Hardware: Example Light Types White floodlight (18W), 12V power cube, Infrared floodlight (8W) Simple MOSFET circuit provides RPi GPIO control for floodlights. MOSFET is essential to allow RPi GPIO to control 18W, 12V light!

Hardware Enclosures: Use Existing Light Fixtures! … and temperature sensors!

Hardware Enclosures: Fake Security Cam for $5 This $5 fake security cam can be modified to hold a RPi Zero with PiCamera. It is waterproof and has tripod quality adjustable view angle. A plain RPi zero case is usually $5.

Learnings: Some Hardware Tips and Tricks • PiCams vs. WebCams: PiCamera Python module allows setting exposure mode, iso, contrast, brightness and saturation adjustment. • Waterproof enclosures, e.g. water meter cam, critter cam: mason jars and old shingles are my DIY favorites; using existing outdoor lighting fixtures on house and barn work well also. • InfraRed lighting for critters: multiple choices for size and on/off control. • LED lighting under RPi control: MOSFETs rule! Don’t use your GPIO without one. • Temperature / humidity sensors: DS18B20 & W1ThermSensor; DHT 22s for humidity. • Laptops keep working during power failures; RPi’s don’t; an imagehub design consideration. • Power: Many advantages of 12V over 5V: use 12V to distribute power to RPi’s and Floodlights; then use cheap converters to get 5V for RPi’s (yes, those cheap car chargers for your phone).

Next: Use Deep Learning to Detect Indicator Plant States The most effective soil moisture test is an “indicator” plant that has deep roots and wilts before the plants around it do. Observation of such an “indicator” plant is a great use case for a computer vision Deep Learning model. Next Project Goal.

Yin Yang Ranch DCVP projects are Open Source & on GitHub • yin-yang-ranch: My overall DCVP project design. • GitHub: https://github.com/jeffbass/yin-yang-ranch • imageZMQ: Fast Network Transport for images. Pip installable: • pip install imagezmq • GitHub: https://github.com/jeffbass/imagezmq • imagenode: Captures images, runs detectors, sends (text, image) pairs. • GitHub: https://github.com/jeffbass/imagenode • imagehub: Receives (text, image) pairs from multiple imagenodes, saves them. • GitHub: https://github.com/jeffbass/imagehub

Questions about this talk? Get Answers! PyCon 2020 is happening Online because of the COVID-19 crisis. I was, of course, planning on taking questions at the end of my talk. Let’s try an experiment together: here’s a new way for you to ask me questions about my PyCon 2020 talk. Let’s try a new "post an issue to pose a question" method in the yin-yang-ranch GitHub repository. You'll need a (free) GitHub account to ask me a question. Here’s how to Ask a Question about this talk: 1. Open a browser & navigate to https://github.com/jeffbass/yin-yang-ranch 2. Click the Issues tab. (2nd tab from the left on main repository page). 3. Click the green "New Issue" button. 4. In the Title area of the New Issue page, put a SHORT version of your question. 5. In the Text area of the New Issue page, write a little more about your question. Click the green Submit new issue button. 6. I'll reply; you'll get an email from GitHub letting you know I've replied. 7. This is Open Source! If you can help with an answer to a question, post a comment in the question's issue thread. You may have a better answer than I do. Thanks! [Please Remember to Be Courteous. Also, please Stay on the Topic of my Talk. This is another one of my many Yin Yang Ranch experiments. Thanks for participating.]

Helpful Resource Links • Electronics: Adafruit has extremely helpful tutorials as well as a deep selection of electronic parts and kits: https://www.adafruit.com/ • ZMQ: Python docs: https://pyzmq.readthedocs.io/en/latest/ • OpenCV: Python Tutorials: https://opencv-python-tutroals.readthedocs.io/en/ latest/py_tutorials/py_tutorials.html • Computer Vision Tutorials: Adrian Rosebrock’s tutorials & books got me started in Computer Vision coding: https://www.pyimagesearch.com/ Also, see Adrian’s imutils for some helpful OpenCV tools: https://github.com/jrosebr1/imutils • Permaculture: Toby Hemenway, Gaia’s Garden A Guide to Home-Scale Permaculture. Also by Toby: The Permaculture City: Regenerative Design for Urban, Suburban and Town Resilience Link: http://a.co/izt7Xun