• Cloud Application Architect, Practice Lead at World Wide Technology • ~15 years of industry experience • Software Development • .NET (C# and VB.NET) • JVM (Scala) • JavaScript • Data Streaming Architectures • Kafka, Kinesis, Event Hubs • Application Architecture • AWS, Azure, GCP, and on-prem solutions • Incessant traveler with a new-found skiing addiction • I Live in Denver, by way of St Louis, and grew up in TN INTRO: KEVIN TINN 2
• Source code for AWS infrastructure and the kinesis producer are available at https://github.com/kevasync/aws-meetup-group-data-services • This link is available in the comments of the Meetup deets https://www.meetup.com/AWSMeetupGroup/events/269768602/ Please join the Meetup if you haven’t already REPO INFO 4
• AWS IoT Core is a managed cloud service that lets connected devices easily and securely interact with cloud applications and other device • AWS IoT Core can support billions of devices and trillions of messages, and can process and route those messages to AWS endpoints and to other devices reliably and securely • With AWS IoT Core, your applications can keep track of and communicate with all your devices, all the time, even when they aren’t connected. COMPONENTS: IoT CORE 5
• AWS IoT Greengrass seamlessly extends AWS to edge devices so they can act locally on the data they generate, while still using the cloud for management, analytics, and durable storage. • With AWS IoT Greengrass, connected devices can run AWS Lambda functions, Docker containers, or both, execute predictions based on machine learning models, keep device data in sync, and communicate with other devices securely – even when not connected to the Internet. • Over the air updates! COMPONENTS: GREENGRASS 6
• AWS IoT Analytics is a fully-managed service that makes it easy to run and operationalize sophisticated analytics on massive volumes of IoT data without having to worry about the cost and complexity typically required to build an IoT analytics platform. • Data transformation and enrichment services that are optimized for IoT • Ad hoc query capabilities • Stored in time series DB • Automated scaling COMPONENTS: IoT ANALYTICS 7
• Extract OS • Mount disk using sudo mkdir /mnt && sudo mount -t msdos /dev/disk2 /mnt • Download OS: • Noobs • Pi OS (Raspbian) • I initially tried Pi OS, but for some reason openssh wasn’t working so I tried a fresh OS. The Noobs worked out so I never looked back. • For Noobs I just decompressed the zip file and copied the extracted files to the SD Card in finder • There is also a disk imager utility that can be used, follow instructions from links here to do so: https://www.raspberrypi.org/downloads/ • Complete Pi setup and enable ssh • On the Pi: sudo raspi-config • Navigate to Interfacing Options and select P2 SSH • Restart the Pi: sudo restart • Get the hostname: hostname -I • Do us all a solid: sudo apt install vim -y • Change the hostnames on the pi: • sudo vim /etc/hostname -> change raspberrypi to sensor-pi • sudo vim /etc/hosts -> update to 127.0.1.1 sensor-pi • sudo reboot RASPBERRY PI SETUP 10
• Create policy: • IaC: Pulumi (Available in repo) • or… Navigate to AWS IoT > Secure > Policies > Create: • Update the ARN to have the topic name we will use for sensor data • Name your policy, set Effect to allow and Action to iot:* DEMO: COMMUNICATING WITH IoT CORE DIRECTLY 11
• Create IoT Core instance, create a IoT device, certificates, IAM policy, and attach policy to cert • IaC: Pulumi (Available in repo) • Navigate to IoT Core > Manage > Things: • Create > Single Thing: • Name it and give it a type (You may have to create the type): DEMO: COMMUNICATING WITH IoT CORE DIRECTLY 12
• Create a certificate (This can be done ahead of time): • Don’t forget to activate the cert (I made this mistake.. /facepalm) • Download the root CA file, the device pem file, and the private key file • curl https://www.amazontrust.com/repository/AmazonRootCA1.pem > AmazonRootCA1.pem • Use the links to download the thing pem and the private key, put them in same dir as the CA file • If using Pulumi, stack output command can be used to get device cert and private key: • pulumi stack output IoTPrivateKey > private.pem • pulumi stack output IoTDevicePem > device.pem.crt DEMO: COMMUNICATING WITH IoT CORE DIRECTLY 13
• Associate the certificate with the policy we created: • Get your Thing Shadow form IoT Core > Manage > Things > sensor-pi > Interact: DEMO: COMMUNICATING WITH IoT CORE DIRECTLY 14
• If you manually downloaded cert files, rename them to device.pem.crt and private.pem. Move these and the root ca into same directory and change directories into that directory. • Copy the certs to the sensor-pi using scp (Secure Copy): scp AmazonRootCA1.pem private.pem device.pem.crt [email protected]:~ • ssh into the sensor-pi: ssh [email protected] DEMO: COMMUNICATING WITH IoT CORE DIRECTLY 15
Verify openssl is available within python: python import ssl ssl.OPENSSL_VERSION quit() Get producer source code: git clone https://github.com/kevasync/aws-meetup-group- data-services.git cd aws-meetup-group-data-services/pi DEMO: COMMUNICATING WITH IoT CORE DIRECTLY 16
• MQTT: (MQ Telemetry Transport or Message Queuing Telemetry Transport): is an open OASIS and ISO standard (ISO/IEC 20922) lightweight, publish-subscribe network protocol that transports messages between devices. The protocol usually runs over TCP/IP; however, any network protocol that provides ordered, lossless, bi-directional connections can support MQTT. It is designed for connections with remote locations where a "small code footprint" is required or the network bandwidth is limited. • Setup AWS IoT Py SDK: mkdir sdk cd sdk curl -o aws-iot-sdk.zip https://s3.amazonaws.com/aws-iot-device-sdk- python/aws-iot-device-sdk-python-latest.zip unzip aws-iot-sdk.zip sudo python setup.py install cd .. cp –rf ./sdk/AWSIoTPythonSDK • Run the producer with your interaction url: python pi-sensor-producer.py -ats.iot..amazonaws.com DEMO: GET PY MQTT LIB AND RUN PRODUCER 17
• What we’re working with: DHT22 digital humidity and temperature sensor • Pins: • 1: VCC (Power) • 2: DATA • 3: NULL • 4: GRN (Ground) • Traditional setup: • Place a 10k resistor between Pin 1 and Pin 2 of the DHT22 • Wire Pin 1 of the DHT22 to Physical Pin 1 (3v3) on the Pi • Wire Pin 2 of the DHT22 to Physical Pin 7 (GPIO4) on the Pi • Wire Pin 4 of the DHT22 to Physical Pin 6 (GND) on the Pi • Our device has the resistor on-board, so we can go directly to the board: • Pin 1 -> Pi pin 1 • Pin 2 -> Pi pin 7 # todo: why is this coded in as pin 4? • Pin 3 -> Pi pin 6 • Connect to sensor-pi and prepare it to get sensors readings in our pi-producer code ssh [email protected] cd aws-meetup-group-data-services/pi sudo apt update sudo apt upgrade #takes a while sudo pip3 install Adafruit_Python_DHT # todo: used pip3, test the instructions cp pi-sensor-producer.py pi-sensor-producer-dht22.py vim pi-sensor-producer-dht22.py DEMO: CONNECTING SENSOR TO PI 18
• In the code add an import and reference to the sensor and the data pin on the pi: import Adafruit_DHT sensor = Adafruit_DHT.DHT22 piDataPin = 4 • In the loop that produces to IoT Core, get the temp and humidity from the sensor and add the temp reading in the message payload: while loopCount < 10: humidity, temperature = Adafruit_DHT.read_retry(sensor, piDataPin) message = {} message['SITE_ID'] = clientId message['SENSOR_TYPE'] = 'TEMPERATURE' message['SENSOR_READING_VALUE'] = temperature • Run the producer, see temps (in Celsius): python pi-sensor-producer-dht22.py -ats.iot..amazonaws.com DEMO: CONNECTING SENSOR TO PI 19
• Output form pi producer: • View data in AWS IoT: Core > Test > Subscribe to a topic > sensor-topic (You much run producer again as well): DEMO: VIEW DATA FROM PRODUCER 20
• Let’s connect this sensor data to the data pipeline we created a few Meetups ago (Links: event, repo, slides) DEMO: CONNECTING TO EXISTING DATA PIPELINE 21
• IoT Core MQTT topic rule sends data to Kinesis • Setup up Kinesis Data Stream action and create policy: DEMO: CONNECTING TO EXISTING DATA PIPELINE 23
• Upload weather.csv and start temperature analytics app. Enriched data from the sensor-pi is enriched with the site’s outdoor temperature. DEMO: CONNECTING TO EXISTING DATA PIPELINE 24
• IoT rules offer a integration with a ton of AWS services: • Augment or filter data received from a device. • Write data received from a device to an Amazon DynamoDB database. • Save a file to Amazon S3. • Send a push notification to all users using Amazon SNS. • Publish data to an Amazon SQS queue. • Invoke a Lambda function to extract data. • Process messages from a large number of devices using Amazon Kinesis. • Send data to the Amazon Elasticsearch Service. • Capture a CloudWatch metric. • Change a CloudWatch alarm. • Send the data from an MQTT message to Amazon Machine Learning to make predictions based on an Amazon ML model. • Send a message to a Salesforce IoT Input Stream. • Send message data to an AWS IoT Analytics channel. • Start execution of a Step Functions state machine. • Send message data to an AWS IoT Events input. • Send message data an asset property in AWS IoT SiteWise. • Send message data to a web application or service. DEMO: CONNECTING TO EXISTING DATA PIPELINE 25
• Create Greengrass group and save credentials bundle: https://docs.aws.amazon.com/greengrass/latest/developerguide/gg- config.html CONFIGURE GREENGRASS 29
• Create Greengrass group and save credentials bundle: https://docs.aws.amazon.com/greengrass/latest/developerguide/gg-config.html • Copy the security bundle and root CA from earlier to the greengrass pi, ssh in and extract files: scp ~/Downloads/3664d60716-setup.tar.gz [email protected]:~ scp ~/src/aws-meetup-group/iot/certs/AmazonRootCA1.pem [email protected]:~ ssh [email protected] tar -zxvf 3664d60716-setup.tar.gz CONFIGURE GREENGRASS 30
• Complete Greengrass use case • Bi-directional communication • Device provisioning • Security • Edge-based ML • IoT Greengrass Stream Manager connection to Kinesis or IoT Analytics NEXT TIME… PART 2: GG, IoT ANALYTICS, AND AWS IoT ADVANCED TOPICS 34
• In this session, we have covered: • Overview of AWS IoT Services • Demo of setting up IoT Core and Raspberry Pi w/ sensor • Connecting IoT topic to data pipeline via IoT rule • Thank you for viewing! • Please talk to me if you have further questions • References: • Raspberry pi to IoT core: https://www.mitrai.com/tech-guide/connecting-your-raspberry-pi-with-aws-internet-of- things-iot-service/ • Setting up Greengrass on Raspberry Pi: https://docs.aws.amazon.com/greengrass/latest/developerguide/setup- filter.rpi.html • https://docs.aws.amazon.com/greengrass/latest/developerguide/install-ggc.html#ggc-package-manager • Lambdas for gg: https://docs.aws.amazon.com/greengrass/latest/developerguide/create-lambda.html • DHT 22 hookup: https://pimylifeup.com/raspberry-pi-humidity-sensor-dht22/ CONCLUSION 35
• AWS data lakes using Glue and Athena • First Wednesday of September – Austin Loveless • Glue • Athena • AWS Data ETL patterns • IoT Part 2 • First Wednesday of October – Kevin Tinn • AWS IoT Greengrass full implementation • IoT Analytics • Advanced IoT capability overview UPCOMING MEETUPS 36
kevasync
replu
danielkatz
coconala_engineer
lycorptech_jp
iamshunta
yukimatsuyoshi
minorun365
iselegant
takanorig
satoshihirose
knsg16
trishagee
nonsquared
sachag
edds
danielanewman
lemiorhan
holman
tanoku
eitanlees
panda_program
cassininazir
jasonvnalue