T o m e r G a b e l Kraków, 15-17 May 2019 // Not Your Grandmother’s On-Prem T o m e r G a b e l Porto, April 28th 2022 //

What’s all this, then? • “On-premise” • Used to be taken literally: “at the customer’s location” • This is no longer relevant • Most customers don’t own their own data center • Most customers don’t own their own hardware • Most customers don’t even lease long-term • Customers are likely running “on someone else’s prem” • A more useful interpretation: • “Running on hardware outside of my control”

Let’s get it out in the open… • On-prem is a swear-word these days • For good reasons! • Are they, though? • It’s 2022 • Much has changed • And I’ve recently wrapped a 6-month on-prem project • Let me tell you all about it!

Who or what am I? • Freelance software engineer • Been around the block • Up and down the stack • For a bunch of companies you’ve heard of • Old and grumpy

Yeah. So, on-prem

Why is it hard? SaaS On-Prem Release process Simple Complex Direct control Full, in-house Limited, involving 3rd parties Concurrently-deployed versions Small, finite set Potentially many Staff In-house and cooperative 3rd party

Why is it hard? Coordination 🤬

Why do this to yourself? • We all love SaaS • It’s the future, they said • They were right

Why do this to yourself? • We all love SaaS • But in some cases, it’s just not an option • Regulatory concerns (e.g. medical information) • Security considerations (e.g. critical infrastructure) • Physical constraints (e.g. no internet connectivity) • And sometimes it’s not practical • When you actually sell infrastructure • When your customers simply won’t buy SaaS

On-prem, Evolved

On the shoulders of giants • SaaS was a radical new paradigm 20 years ago • New tools evolved to support the ecosystem • Can we leverage these tools on-prem? Hypervisors Deploy heterogenous software in isolation Public cloud Programmable, elastic production hardware Containers Simple software packaging and distribution Container orchestrators Abstract execution environment

Yes we can.

First, a dose of reality • All on-prem projects aren’t created equal • What are your constraints? • Data persistence/retention Your network On-prem Your system Licensing Integrations Telemetry Persistent Storage?

First, a dose of reality • All on-prem projects aren’t created equal • What are your constraints? • Data persistence/retention • Cluster vs single host Your network On-prem Your system You? You? You? Licensing Integrations Telemetry

First, a dose of reality • All on-prem projects aren’t created equal • What are your constraints? • Data persistence/retention • Cluster vs single host • Cloud vs colo Your network VPC Your system Licensing Integrations Telemetry Cloud Provider Storage Identity Messaging DB

First, a dose of reality • All on-prem projects aren’t created equal • What are your constraints? • Data persistence/retention • Cluster vs single host • Cloud vs colo • Networking Your network On-prem Your system Licensing Integrations Telemetry Persistent Storage

Let’s talk tooling

Docker • It’s a godsend • Yeah, I know. Issues and nitpicks • BUT! • Consistent packaging • Consistent artifact hosting • Consistent operational model • Composes well • Reusable in all scenarios

docker-compose • Multiple components, single host • Lightweight orchestration • Dead-simple • Doesn’t get enough credit! • Composes really well with…

Virtual Machine Images • VMs are awesome: • Self-contained package • Consistent platform (-ish) • Consistent deployment model • Supported everywhere • Private/public cloud • Publish as VM image with marketplace offerings

Container orchestration • If you have to run on multiple hosts… • My condolences • It’s going to be hard • Kubernetes/Nomad can help • Especially if utilizing EKS/AKS/GKE • A huge subject requiring its own talk

Public cloud FTW • Do your customers run on AWS, Azure, GCP et al? • Awesome! You get loads built-in • Delivery mechanism (VMs) • Secure integration with customer resources (data stores etc.) • Reduced operational burden via available services • SLA and regulatory guarantees that are not your problem • Customer buy-in ahead of time!

Monitoring • SaaS observability tools • Great if you can use them! • Trusted 3rd party services • Secure and auditable access • Low operational burden • Predictable OpEx • A much easier sell!

Going oldschool • If you can’t ”call home”… • Log shipping is the only way to go • Experience makes for good logs • No silver bullet. Sorry! • Some mitigations • If on cloud, object store + temporary ACLs may help • Zoom/Meet/etc. “remote control” features are amazing

Case study

Product X • Unfortunately defunct • But that means I can talk about it! • A multi-component production system • Python 3.9 • Redis cache • PostreSQL • Front-end static bundle • External integration points

Constraints Customers running on public cloud (initially AWS, Azure) Completely isolated: no inbound or outbound traffic Deployment owned by a business analyst team • Very limited technical knowhow • Uncooperative ops department 01 02 03

Choices Single host (to start with) docker-compose • Lightweight and works • Easy to instruct on operation • Reusable for testing VM image published on marketplace Utilizes cloud RDBMS, blob store Packer targeting CentOS 7.9 01 02 03 04 05

What Worked Well • Trivial setup process • Initial VM deployment with a few clicks via marketplace • Fully integrated with customer’s identity/security • Trivial upgrades • Leveraging Docker tags • Little or no technical know-how required!

Challenges • Ops/SRE expertise • Many disparate tools (Packer, Docker, systemd…) • Multiple targets (AWS/Azure) • Lots of ”glue logic” with shell scripts • Support and diagnostics • No 3rd parties allowed • No direct access to server/database/logs

In Conclusion

Takeaways • The tools that enable SaaS are a boon for on-prem • Public clouds make for compelling on-prem targets • VMs and Docker make deployment almost painless • Supporting on-prem is surprisingly easy with: • 3rd party observability tools (logging, APM) • Modern communication tools (Zoom et al)

Not Your Grandmother’s On-Prem 1. On-prem is harder than SaaS 2. Modern tools make it quite painless 3. Be not afraid!

tomer@substrate.co.il @tomerg https://github.com/holograph Thank you for listening Questions?