the world’s CO 2 emissions. • Einride’s mission ◦ Create sustainable solutions for movement. • Einride’s strategy ◦ Digitalize. ◦ Electrify. ◦ Automate. With a clean energy mix, battery electric vehicles reduce total emissions by up to 95%. Going electric and autonomous requires connected vehicles with centralized and algorithmic transport management. Cost reductions from autonomous technology is a key enabler for widespread electrification. Up to 95% less emissions Up to 40% lower transport cost About Einride: The Company
enabling shippers to purchase transport-as-a-service. ◦ Strategic, data-driven electrification of existing diesel transports through analytics on historical data.[1] • Electric Transport ◦ Connected electric trucks operated by carrier partners. ◦ Available for all customers today. ◦ Optimized transport plans[2] delivered to drivers in real-time via mobile app. • Autonomous Electric Transport ◦ First all-electric autonomous transport vehicle to operate on public road and up to 85 km/h. ◦ Available for selected customers today. ◦ Scale-up and industrialization in the next few years. About Einride: The Technology [1]: Transportation infrastructure sustainability (2019), Mehdi Akbarian (Ph.D), Einride [2]: Preparing for autonomous electric truck operations (2020), Benjamin Kickhöfer (Ph.D), Einride
from transport data scientists to autonomous drive engineers to electromechanical and vehicle engineers. ◦ Supported and enabled by generalist software and data engineers. • Jack of all trades, master of some ◦ Designing the end-to-end solution for a sustainable, electric and autonomous transport service. ◦ Buying and integrating from the best - inventing the rest. ◦ Software ties everything together - from mobile apps to data pipelines to steering actuators. • Start-up mode: 80/20 solutions all day every day ◦ Choosing wisely where to focus our effort. Safety is top priority. ◦ Leveraging Open Source heavily. Primarily Go. ◦ Cloud-native, serverless approach in Google Cloud Platform. About Einride: The Team
representing senders and receivers of goods. ◦ “Lanes” are weighted edges, representing the total flow of goods during a given time window. • Strategic transport demand forecasting ◦ Large time window (month/quarter/year) and large parts of the transport network. ◦ Example use case: “How much total transport capacity will be required next quarter”? • Tactical transport demand forecasting ◦ Small time window (day/week) and small parts of the transport network (lane/site). ◦ Example use case: “How many outgoing truckloads will be required this week?”. Transport Demand Forecasting
One time series for every combination of time window and site(s) or lane(s). • Future transport demand with timeseries forecasting ◦ Generate forecasts for each time series. ◦ Challenge: Number of time series might be quadratic in number of sites, for each customer. ◦ Need a scalable solution for continously generating updated forecasts for every time series. • 80 / 20 solution: Unsupervised timeseries forecasting ◦ Leverage standard Google Cloud and Open Source tools to get something up and running quick and easy. ◦ Fully automated MVP good enough until we need higher accuracy! ◦ Will need human-in-the-loop forecasting to guarantee that forecasts are actionable. Collaborator Alfred Gunnargård [email protected] Software/Data Engineer, Einride
SARIMAX ◦ Search key: “level trend seasonality” ◦ Led to a great blog series on the statsmodels SARIMAX implementation. • Requires manual hyperparameter search ◦ Non-trivial to implement efficiently, too much time overhead to search on each time series. • Want a higher-level, hands-off solution for non-experts ◦ Preferably with no tuning required at all. ◦ Blog posts are from 2017-2018, better automated methods must have become available since then!
about it. • Batteries included ◦ Spark (and PySpark) included in the image. ◦ Drivers for reading/writing to/from BigQuery and Cloud Storage. ◦ Happy path: pip install fbprophet and off to the races! • At the end of the day, it’s still Hadoop ◦ Debugging issues is where the fun begins. ◦ Many layers of software (YARN, Spark, Arrow, Pandas, Prophet, etc). ◦ Need to manage version dependencies carefully to ensure compatibility. ◦ Runs on a cluster of dedicated VMs. Must keep an eye on billing and only run it on-demand. Dataproc: Hadoop in a box
each step in the data pipeline. Task lifecycles are tracked, failures are automatically retried. The pipeline is modeled as a Directed Acyclic Graph (DAG). The DAG and all its tasks are described with a Python DSL.
from Spanner dump. Airflow’s standard library of pipeline operators has great support for BigQuery. Airflow’s Python DSL models dependencies for each pipeline step.
(UDFs). Type annotations enable conversion of UDF result from Pandas in-memory data frames to Spark distributed data frames. Plain old Python (with Prophet)! https://facebook.github.io/prophet/docs/quick_start.html
trivial to create, but non-trivial to operate. ◦ Hadoop and Airflow: a basement and attic both full of wonders. ◦ Can get by with minimal knowledge by keeping things simple. • Airflow feels a bit heavy-weight for the purpose ◦ Less focus on idempotency than Luigi. ◦ Want lighter solution for Google Cloud-native data pipelines… • Stable and reliable, but slightly brittle ◦ Has been running as of now (Nov 2020) weekly for the last 6 months. ◦ No operational issues (only the occasionally funny forecast...) ◦ Tried to bump Prophet and Dataproc versions, something broke, still needs investigation... • Doable with a minimal team ◦ All in all, a great setup for scaling Python models with little effort! Summary: A working combination
Dataproc cluster management. ◦ Less dependency and version management. ◦ Lower cost...? ◦ Faster and more scalable...? • Generally available a few months ago ◦ Is it as easy to use for non-experts as Prophet? ◦ We’ll find out soon! Next steps: Try BigQuery ML
odsod
tomokusaba
terurou
kworkdev
kash1064
sansantech
puhitaku
mohab1
iktakahiro
y0hgi
pfn
jtakeda
codmoninc
apolaine
jonyablonski
lynnandtonic
jonoalderson
zakiyama
greggifford
denniskardys
kneath
jeffersonlam
rkaga
marketingsoph
lara
![Thank you! Oscar Söderlund, Chief Software Architect [email protected] @odsoderlund @odsod](https://files.speakerdeck.com/presentations/afd8e00b80d047f79c3b94348bf9f830/slide_23.jpg){kind=link}