>> Developer Advocate at AWS, Coding Ambassador @ coders(hq) >> 18 years and counting, Backend engineer to Solutions Architect and Team lead >> Community builder @ASEAN and @MENA >> Mentorship in technical leadership and how to build a career path in tech 3
simpler times… A developer would design a basic UI and focus on the backend logic A DB architect would manage the schema A sys admin would manage the provisioning and maintenance of servers Developer Sys admin Architect
evolution Browsers are the new OS, internet the new delivery medium The rise of multiple programming languages, frameworks More compute options, Distributed systems over networks, Purpose-built databases Open-source tooling Virtualization and containerization Infrastructure democratization through cloud
engineer? Some call them: • BI developers • Research engineers • Machine Learning engineers • DataOps • Analytics engineers (aka unhappy data analysts) • Data engineers • And sometimes… Software Engineer 10 Photo by Brendan Church on Unsplash
2022, Amazon Web Services, Inc. or its affiliates. A data engineer solves data-related engineering problems in a maintainable way. Also talks a lot. 16
Data Science Hierarchy of Needs 18 AI, Deep Learning A/B testing, Experimentation, Simple ML Algorithms Analytics, Metrics, Segments, Aggregates, Features, Training data Cleaning, Anomaly Detection, Preparation Reliable Data Flow, Infrastructure, Pipelines, ETL, Structured and unstructured data storage Instrumentation, Logging, Sensors, External data, User generated content
Data Science Hierarchy of Needs 19 AI, Deep Learning A/B testing, Experimentation, Simple ML Algorithms Analytics, Metrics, Segments, Aggregates, Features, Training data Cleaning, Anomaly Detection, Preparation Reliable Data Flow, Infrastructure, Pipelines, ETL, Structured and unstructured data storage Instrumentation, Logging, Sensors, External data, User generated content
Data Science Hierarchy of Needs 20 AI, Deep Learning A/B testing, Experimentation, Simple ML Algorithms Analytics, Metrics, Segments, Aggregates, Features, Training data Cleaning, Anomaly Detection, Preparation Reliable Data Flow, Infrastructure, Pipelines, ETL, Structured and unstructured data storage Instrumentation, Logging, Sensors, External data, User generated content Machine Learning Engineer Data Scientist Data Analyst Data Engineer Data Infrastructure Engineer
Data Science Hierarchy of Needs 21 AI, Deep Learning A/B testing, Experimentation, Simple ML Algorithms Analytics, Metrics, Segments, Aggregates, Features, Training data Cleaning, Anomaly Detection, Preparation Reliable Data Flow, Infrastructure, Pipelines, ETL, Structured and unstructured data storage Instrumentation, Logging, Sensors, External data, User generated content Machine Learning Engineer Data Scientist Data Analyst Data Engineer Data Infrastructure Engineer
past is the future, choose boring • Data modeling (1960s) • UNIX shell (1971) • SQL (1974) • Python (1991), Java (1995) • Kubernetes YAML (2014)? ”The longer a technology lives, the longer it can be expected to live.” - Nassim N. Taleb (way of Mandelbrot, aka Lindy effect) 23 Photo by Lukas on Unsplash
“DevOps is the combination of cultural philosophies, practices, and tools that increases an organization’s ability to deliver applications and services at high velocity.” Source: https://aws.amazon.com/devops/what-is-devops/
Pilots vs Operationalising Pilot phase Operational phase Purpose: Put the system in production and achieve desired business value ML Code Configuration Data Collection ETL Data Verification Analysis & Evaluations Infrastructure Management Process Management Serving Infrastructure Monitoring Testing Automation CI/CD Machine Learning Code Monitoring Serving Infrastructure Configuration Management Tools Automation Continuous Integration Continuous Deployment Testing Data Verification Continuous Data Collection Model Evaluation Experiments Purpose: Answer the question “Is this possible, and should we proceed?”
is a journey Initial Repeatable Reliable Scalable MLOps Maturity Models in Production Establish the experimentation environment Standardise code repositories and ML solution deployment Introduce testing, monitoring, and multi-account deployment Templatise and productionise multiple ML solutions
while architecting data projects Principle Example Flexibility Use decoupled services Reproducibility Use infrastructure as code (IaC) to deploy your services Reusability Use libraries and references in a shared manner Scalability Choose service configurations to accommodate any data load Auditability Keep an audit trail by using logs, versions, and dependencies 42 https://docs.aws.amazon.com/prescriptive-guidance/latest/modern-data-centric-use-cases/data-engineering-principles.html
advice Build foundations with the boring stack, DS hierarchy of needs Look at the hidden details of the role and responsibilities behind a job title Ramp up on automating data processes and deployments to production It’s a journey, with moving goal posts, and newer responsibilities being added as technology and businesses evolve
faz
nomizone
lycorptech_jp
yoshiki0705
shibuiwilliam
chanyou0311
chacco38
jnchito
shuadachi
taddy_919
oomatomo
kenta_hi
ueponx
michielstock
mkilby
tonyrice
smashingmag
katiecoart
davidcarrasco
axbom
carmenintech
chikuwait
gurzu
twada
aleyda
