WHAT IS SOFTWARE?

JACOB MATHER ABOUT ME • Software Architect for Lease • Spouter of Opposing Ideas • Community Evangelist • Blog: http://jmather.com • Twitter: @thejmather • Medium: @jacobmather

TODAY IS HUGE FOR ME I HAVE FAMILY IN THE AUDIENCE!

WHO AM I?

I’VE BEEN WORKING WITH DISTRIBUTED SYSTEMS A WHILE… WHO AM I? • Started playing turned based multiplayer games on BBSs ~1988. • Started running my own BBSs when I was 9. • Built my own home network between two houses when I was 13. • Started #networking on Undernet IRC when I was 14. • Set up linux servers for my school district when I was 15. • Began freelance software development over the web at 16. • Started my own hosting and software development businesses in 2004, ran until 2014. • Started building things running at scale in the Silicon Valley in 2014.

I’VE BUILT A LOT OF SYSTEMS FOR A LOT OF COMPANIES… WHO AM I?

HOW I HAVE DEFINED MYSELF OVER TIME • I am a Programmer • I am a Developer • I am a Web Developer • I am a Software Engineer • I pretend to be a Software Engineer • I pretend to be a Software Engineer for sufficiently narrow definitions of software • Theoretically I pretend to be a Software Engineer for sufficiently narrow definitions of software

THE CURRENT STATE OF THINGS

MONOLITHIC VS SERVICE ORIENTED ARCHITECTURE • What makes a monolith a monolith • What makes SOA SOA? • Why this distinction matters

MONOLITH ARCHITECTURE VS. SERVICE ORIENTED ARCHITECTURE

MONOLITHS COMMONLY… GENERAL STATEMENTS ABOUT MONOLITHS • are self-contained in a single repository. • solve the solution end-to-end, providing user interface, data backend, and admin system all in one. • provide developers an ORM for interfacing with the data store backend. • house migrations for the data structure and content. • assume all data is “local”

GENERAL PROBLEMS WITH MONOLITHS • Improper abstraction of assumptions leads to very intertwined functionality that eventually becomes very resistant to change. • The longer code is around, the more it rots, especially when the impact of changes becomes difficult to reason about. • Too many cooks in the kitchen. Different developers with different backgrounds use different patterns for … EVERYTHING, and you’ll have to learn them all for everything you touch. • Migrations often use application level functions which change and break over time, without the migrations being updated or removed. • Improperly staged changes can become difficult to release, possibly even requiring downtime in order to ensure proper isolation.

SOAS COMMONLY… GENERAL STATEMENTS ABOUT SOA • are spread across multiple repositories • allow each service to own it’s backend data store • assume some data is “local” and some is “remote” • coordinating the network of “valid” services which cooperate together becomes a new challenge, especially for improperly staged feature releases.

GENERAL PROBLEMS WITH SOA • The data model of the business becomes distributed amongst multiple services and applications. • The complexity of managing compatible versions can become onerous. • It’s easier to accidentally break dependent pieces because without extra effort, the impact of changes within SOA are less visible than those same changes within a Monolith • Without a solid continuous integration implementation, managing releases of each independent component becomes much harder.

MOST SOA ISN’T SOA • What makes a monolith a monolith isn’t that it’s applications and services all rolled into one. A monolith is monolithic because it encapsulates multiple conceptually distinct layers within a single application or service. • If your SOA services each own their own data persistence layers, I believe what you have a Monolithic Service Oriented Architecture. • This just takes all the problems of Monolithic architecture and smears it across your entire engineering operation.

MONOLITHIC ARCHITECTURE • 1 Application • 1 Business Domain, belonging to the Application • Business Processes map to Features • Entire supported business domain reflected in data model.

MONOLITHIC SOA • N Applications • N Business Domains, each bound to their respective Applications • Otherwise, exactly what we have been doing: building monoliths.

PURE SOA • Applications no longer own the Business Domain • The Application is now directly associated with a purposeful documented business process • Features in the application are then built around the interactions required to perform the process

SO YOU WANNA FISTICUFFS, EH?? APPLICATIONS VS ANALYTICS • Data for the business becomes spread across multiple applications • Business needs to analyze this data for many purposes • Capacity planning • Sales forecasting • Trend detection • Hypothesis validation • Analytics group is formed to aggregate the data. • Analytics is frequently broken by application data updates.

HOW NARROW IS MY DEFINITION OF SOFTWARE?

FACTS ABOUT SOFTWARE THAT HAUNT ME • For most development methodologies, every 1,000 lines of code, there will be at least 1 bug. Most reports tend to say industry average is around 15-30 bugs per 1,000 lines of code. • I have a suspicion that the vast majority of these bugs occur when we write code that transforms data formats. Or do anything related to date math. • More code = more bugs. • Static analysis can be as introspective as you’re willing to invest. • Proper isolation means you can test all scenarios before launching to production. • Production should only ever go from one known good state to another known good state.

THE QUEST FOR A DEFINITION OF SOFTWARE • Everyone seems happy to write a lot more code than I do. • Everyone seems happy to rewrite what I see as core functionality. • Everywhere seems to have a difficult time managing the complexity that change over time brings. • Almost every system ends up with assumptions spread across the code, which become more and more complex to change later. • Every business eventually seems to equate software releases with risking the stability of the production system.

BUILDING A DEFINITION OF SOFTWARE • Software is comprised primarily of two components: • Configuration Data • Execution Code • A software solution implementation is then composed of: • Configuration Data • Execution Code • Runtime Environment

THOUGHTS AROUND VALIDATING SOFTWARE • Working in games really reinforced the difference in risk between shipping configuration changes vs code changes. • Configuration can be validated, generally quickly and efficiently. • Code can be validated, but cost of managing that validation goes up exponentially with complexity as permutations increase. • If configuration is cheaper and easier to validate than code, then why do we write so little configuration and so much code? • Is there anything out there that we could use for a reference implementation, to demonstrate this pattern is robust?

MEANWHILE IN THE OUTSIDE WORLD…

GENETIC MUTATIONS • I discovered I have a (fairly common) genetic mutation of an enzyme configuration named MTHFR. • The “operation” is a “reduction” which is the exchange of a hydrogen (atomic weight 1) molecule for an oxygen (a.w. 8) molecule.

FROM DIET TO ESSENTIAL NUTRIENTS TO MOLECULES SO I APPLIED ENGINEERING TO HEALTH • Mapped essential nutrients to their most common food sources. • Mapped favorite foods to food sources as ingredients. • Followed the metabolic path of how our bodies produce energy. • Did 23&Me, downloaded my genome, got third party reports. • Came to view biology as a very advanced, robust, and simple software system which allows for nearly infinite complex expression from simple building blocks.

THE HUMAN BODY AS A SOFTWARE SYSTEM • One of the most basic biological molecular operations is the exchange of 1 for 8. It allows you to trade a negatively charged hydrogen molecule for a positively charged oxygen molecule. • Is it any surprise, then, that we use bits to signify 1, and bytes to signify 8? • Perhaps, since we know we project ourselves onto other people when we interact with them, just maybe, we project ourselves onto our other creative activities. Like, say, designing a computer?

DNA THE HUMAN BODY AS A SOFTWARE SYSTEM • DNA is made up of 4 nucleotides, we all know as T, A, C, and G. • These nucleotides are actually pairs, T pairs to A, and C to G. • This effectively mimics Raid 1 (mirrored) storage. Instead of 0, 1, 2, and 3, think of them as 0, 0’, 1, and 1’. • DNA is read in codons, which are a set of three nucleotides. • One of those codons is stop. • DNA is not code it is config, because it is interpreted statically.

ENZYMES THE HUMAN BODY AS A SOFTWARE SYSTEM • Enzymes enable chemical reactions to happen faster than they would otherwise happen. • Enzymes are built on the fact that nitrogen (and nitrogen like molecules, which only have) within a compound creates a small “energy imbalance”. They enable a compound to temporarily hold on to extra electrons, which provides the “fuel” for the reactions the enzymes enable.

BIOLOGY AS A PROGRAMMING LANGUAGE • DNA is your Genetic Configuration. • Enzymes are built-in functions. • Mitochondria, and other cellular biological organisms are third party libraries. • Water is the runtime environment. • I can’t help but want to call it bio-assembler.

BIOLOGY AS A DISTRIBUTED SYSTEM • Organically, our bodies are very much like machines, following sets of instructions, operating on local resources. • We understand the body as a set of closed systems: nervous system, cardiovascular system, lymph system, etc… • We can survive for a long time with incorrect inputs because the failure of one system doesn’t necessarily mean failure of the entire machine. • Hormones are long distance messages, the nervous system, blood vessels and lymph paths are the network, the brain is the central hub, for monitoring, logging and coordination.

WHAT BIOLOGY TEACHES US • Software is comprised of both configuration and code, executed within an environment. • Configuration should be large, each environment should maintain it’s own full copy, and have validation mechanisms built in. • Code should be small, serve a single purpose, and assumed to be disposable. • The safest systems for handling change over time have strong boundaries and clear responsibilities. • Complex systems should be made by aggregating the functionality of simple systems.

WHAT IF WE RETHINK OUR JOBS?

BUILDING FOR CHANGE OVER TIME RETHINKING SOFTWARE ENGINEERING • Traditionally, software engineering seems to be viewed as “building the thing”, but the only thing we know for sure at the beginning of the project is that eventually, our definition of “thing” will no longer be congruent with our implementation. • Basically, the only thing we know is that eventually we’ll be wrong, if we aren’t already operating in some known shade of wrong. • If we pull back, and instead look at it as a longer play, it allows us to cut our concerns differently, in a way that I believe becomes much more robust.

WHAT IS AN API?

STRUCTURE VS INSTANCE WHAT IS AN API? • An API is a definition of structure. Not an implementation. • An API may have one or more implementations. This may be an SDK, or a server exposing the API via REST, GraphQL, RPC, or similar, over any number of protocols (like HTTP or HTTPS).

WHAT IS AN API? • A function name. • A defined structure of inputs. • A defined structure of outputs. • A process which is executed: • against an instance of an application. • within a context. • with defined resources.

HOW IS AN API ACCESSED? • An API will be implemented via one or more interfaces. • REST, JSON-RPC, and GraphQL are examples of API interfaces. • An API implementation will then be exposed via a protocol, such as HTTPS.

WHAT IF WE RETHINK OUR REQUIREMENTS?

THE PURPOSE OF BUSINESS APPLICATIONS • To enable a company to solve problems for their customers better than the customer is able to do with their resources alone. • We build business applications to help the business solve problems more efficiently. • The first problem a business encounters is tracking customer data.

BUSINESS APPLICATION REQUIREMENTS • A need to store, modify, and retrieve data. • A need to understand this data to grow the business. • A need to expose only certain data to certain customers. • A need to communicate information to customers.

BUSINESS APP APIS AS CAPABILITIES • Accept web requests • Validate incoming requests • Persist and recall data • Perform stateful modifications of existing data • Notify an action has been performed

A Business API broken down into pieces

How that model is then exposed

HOW CUSTOM APPLICATIONS ARE BUILT ON TOP

DATA AS A SERVICE

WHY I THINK DATA IS A SOLVED DOMAIN

WHY I THINK DATA IS A SOLVED DOMAIN • Business objects can be appropriately defined using an extended form of JSON Schema. • State machine definitions can be written for the business objects. • Objects can be collected into repositories which define relationships and access patterns. • DB Schema can be generated from the repository definitions. • Best practices can constrain DB Schema modifications to be safe. • APIs can be generated from business objects and state machines.

COMPELLING ARGUMENTS THAT DATA IS SOLVED • Model schema to table generation is solved. • Migration generation for table structure is solved. • Exposing an API in a language which has an introspect-able runtime model is solved. • Exposing an API via configuration, attached to an introspect-able runtime model has been solved. • Data modeling is solved VERY WELL by tools in that domain, which can be made to export compatible model schema definitions. • State machine modeling is better communicated by tools in that domain, which can be made to export compatible state machine schema I suspect. • Generating the required list of operations for a given data model is solved. • Componentizes a clear reusable need. • Keeps analytics from dealing with a fragmented business model understanding.

THEN IT SHOULD BE INFRASTRUCTURE. IF DATA IS SOLVED… • Data should be a first class service within your Service Oriented Architecture. • All applications should store and retrieve data the same way. • Data structure is then encapsulated within it’s own layer. • All data definitions live together, meaning half of analytics job is done. • Validation can easily detect when deprecating a field impacts analytics reports and block release accordingly.

TAKEAWAYS

SOFTWARE IS CODE & CONFIGURATION

MORE DESCRIPTIVE CONFIGURATION + SMALLER PIECES OF CODE = NATURALLY RESILIENT SOFTWARE

IS YOUR SOA REALLY SOA? MAYBE NOT.

CONSIDER DATA AS A SERVICE I PREDICT IT’LL BE A THING FIND OUT MORE AT: HTTPS://GITHUB.COM/APITHETIC/APITHY

LEAVE COMMENTS! HTTP://BIT.LY/CCC2017-WIS