ROI of AI

Measuring the Impact: The ROI of AI in Engineering Work
fl ows Michael Stahnke VP Engineering at Flox (flox.dev) @stahnma

@stahnma What is ROI?

@stahnma R Return

@stahnma O On

@stahnma I Investment

@stahnma

@stahnma Is this better with AI?

@stahnma To know this, I have to know something about
current ROI of Engineering

@stahnma Spoiler: We know little as an industry

@stahnma

@stahnma None of this is ROI

@stahnma ROI comes from an obsession for efficiency

@stahnma Edu

@stahnma

@stahnma 1. …

@stahnma

@stahnma Mental Model for Metrics Baseline DevOps Metrics I don’t
feel elite. Move Quickly with Confidence Throughput Quality Normalization Fungibility Rollbacks (or forwards) Deployments per Week Deploys / Dev / Week Successful deploys Narrow vs Wide Time to 5th PR Changes Understood Start Status Questions People Metric Metric Metric Themes Eng Metrics Framework Talk: https://www.youtube.com/watch?v=A1SCGGbHgZ0

@stahnma // it’s a better than average proxy metric developer_productivity
!= roi

@stahnma deploys / dev / week

@stahnma Throughput

@stahnma If you hire somebody, what do you get?

@stahnma

@stahnma ROI at large

@stahnma This is actually easier the more zoomed out

@stahnma Investment

@stahnma Sum of payroll, tools, facilities, travel

@stahnma Return: Profit

@stahnma

@stahnma Difficulties

@stahnma Return for a specific team, area, feature

@stahnma Enter AI Tooling

@stahnma

@stahnma cyn·i·cism noun an inclination to believe that people are
motivated purely by self-interest; skepticism

@stahnma hating is easier than trying

@stahnma All developer tools come down to two promises

@stahnma Save time Save money

@stahnma if ( time == money ) { return money
}

@stahnma Other Returns

@stahnma Number one feature of AI:

@stahnma Number one feature of AI: shitpost enablement 💩

@stahnma Prompt: Generate me a meme image with the theme
of "the most effective use of AI is to make memes faster and in higher quality"

@stahnma But, AI kind of sucks at coding.

@stahnma Hating is easier than trying

@stahnma What is AI engineering good at?

@stahnma Writing code

@stahnma For some values of code

@stahnma What is it less good at?

@stahnma Software Engineering.

@stahnma Coding ✅ Software Engineering ⚠

@stahnma

@stahnma ‣ Writing code ‣ Testing ‣ Debugging ‣ Observability
‣ Understanding Code ‣ Software Architecture ‣ Security ‣ Work breakdown ‣ Prioritization, dependencies ‣ Dealing with ambiguity ‣ Reliability, delivery accountability ‣ Evaluating Tradeoffs ‣ Delivering Feedback ‣ Seeking and receiving feedback ‣ Effective communication ‣ Knowledge Sharing ‣ Teamwork ‣ Relationship building ‣ Handling disagreement ‣ Decision making ‣ Driving alignment ‣ Process thinking ‣ Facilitation ‣ Mentoring ‣ Business acumen ‣ Strategic work ‣ Product Thinking

@stahnma Coding isn’t the hardest part.

@stahnma Your value is not time spent in your editor

@stahnma

@stahnma Public CircleCI Software Engineer Career Matrix

@stahnma More senior-level work doesn’t require as much coding The
more senior the engineer, the less they like AI code

@stahnma What is AI good at?

@stahnma Finding Ambiguity

@stahnma It’s a super power

@stahnma One of my favorite prompts: What ambiguity remains?

@stahnma Learning new tech

@stahnma I’ve never worked with tailwind and HTMX before, could
you walk me through making a landing page?

@stahnma Finding new tools/libraries

@stahnma Tool trade-off explanations

@stahnma is there something like postgREST for mysql compatible databases?

@stahnma

@stahnma Understanding code: what would make this easier to test?

@stahnma Drives Product Thinking… if you start with specifications

@stahnma Improved documentation?

@stahnma Work Breakdown

@stahnma Rubber ducking Wow, this one time, I actually meant
ducking.

@stahnma Saving Time

@stahnma How much time can you save with a few
tab-completes?

@stahnma How much can you save by having simple functions
written?

@stahnma How much time do you spend figuring exact properties,
syntax, methods, API calls?

@stahnma How much time is spent attempting future-proof code?

@stahnma

@stahnma How much time is spent on “technical debt”?

@stahnma How much “technical debt” is created due to attempting
to future-proof?

@stahnma Code you don’t like is not technical debt

@stahnma Save time with YAGNI

@stahnma “I don’t like AI Code because it’s not how
I would do it.”

@stahnma If you’re using AI exactly like you did traditionally
software engineering, you won’t save much time.

@stahnma Saving Money

@stahnma Does it take money to save money?

@stahnma Have you spent money?

@stahnma Yes.

@stahnma

@stahnma talks.stahnma.us/roi

@stahnma In Closing

@stahnma Summary • If you’re not using AI tools, you’re
may get lapped by those who are. • If you’re only using them, you probably don’t know what you’re doing. • If you’re saving time, you’re probably recovering cost. • If you’re learning and leveling up, you’re doing really well.

@stahnma

@stahnma Deterministic vs Probabilistic outcomes

@stahnma

@stahnma When the code has unknown quality and risk levels,
we need to treat it as hostile.

@stahnma Control points need to be created and developed

@stahnma Developers have been kingmakers for ~15 years

@stahnma Coming soon.

@stahnma

@stahnma CircleCI Career Matrix (Public) ROI Calculator References ROI of
Platform Engineering Talk Metrics Framework Talk Speakerdeck Let 1,000 Flower Bloom This deck

@stahnma We’re hiring We’re post-geographic.

@stahnma Availability  Cost for Downtime COGS  Cost for Services in
support of Gross Margin Security   Attach Rate and dollar value for compliance docs and security questionnaires Developer Productivity  Hiring Tradeoffs, time to value, throughput Developer Sentiment Developer Adoption Platform ROI Talk: https://www.youtube.com/watch?v=iVrq0CcMx0E

@stahnma Mental Model for Metrics Baseline DevOps Metrics I don’t
feel elite. Move Quickly with Confidence Throughput Quality Normalization Fungibility Rollbacks (or forwards) Deployments per Week Deploys / Dev / Week Successful deploys Narrow vs Wide Time to 5th PR Changes Understood Start Status Questions People Metric Metric Metric Themes Eng Metrics Framework Talk: https://www.youtube.com/watch?v=A1SCGGbHgZ0

@stahnma All models are flawed. Some are useful. — George
E. P. Box

ROI of AI

ROI of AI

More Decks by Michael Stahnke

Other Decks in Technology

Featured

Transcript