Write Code In Production

Teams building data-oriented applications against sensitive or large-scale information often face the problem of how to conduct discovery and work iteratively during development. Often, developers are cut-off from production data sets for security and operational reasons. But this doesn't have to be the case. You can build your system in such a way that analysts can work securely with production data, separate from customers, sharing their work, and performing quality control with real, live, production data. Save on operational overhead and complexity while maintaining security by moving your development environment to production instead of your production environment to your laptop.

Transcript

1. W R I T E C O D E I N P R O D U C T I O N S A M W I L S O N
2. W R I T E C O D E I N P R O D U C T I O N S A M W I L S O N ^some
3. w h o a m i • CTO @ Bainbridge Health • Optimizing Medication Safety and Stewardship for Hospitals • Payments, Education, Finance, Marketing Automation, E-Commerce, Media • @numbsafari
4. “ B O R I N G ” P R O B L E M S • Getting paid, paying bills, compliance, security • “Boring” problems often become the truly differentiating and innovative part of a business.
5. H T T P S : / / E N . W I K I P E D I A . O R G / W I K I / C O N F I G U R AT I O N _ M A N A G E M E N T # / M E D I A / F I L E : C O N F I U R AT I O N A C T I V I T Y M O D E L . P N G
6. • Industry survey and market data analysis • Speed + Stability • DevOps
7. H I G H P E R F O R M E R S • Greater commercial success • Market cap growth, profitability, and market share • Greater non-commercial success • Effectiveness, efficiency, customer satisfaction
8. S P E E D + S TA B I L I T Y • Speed: Higher frequency of deploys • Speed: Lower lead-time to delivery (commit to deploy) • Stability: Lower MTTR • Stability: Lower change fail rate
9. H O W ? D E V O P S • Paraphrasing Wikipedia: integrating development and operational practices to shorten SDLC to deliver change faster and more often in close alignment with organizational objectives • Paraphrasing Twitter: automate all the things!
10. C O N T I N U O U S D E L I V E RY Continuous Delivery is the ability to get changes of all types—including new features, configuration changes, bug fixes and experiments—into production, or into the hands of users, safely and quickly in a sustainable way. http://continuousdelivery.com
11. C O N T I N U O U S D E L I V E RY H T T P S : / / U P L O A D . W I K I M E D I A . O R G / W I K I P E D I A / C O M M O N S / C / C 3 / C O N T I N U O U S _ D E L I V E RY _ P R O C E S S _ D I A G R A M . S V G
12. D TA P Development Testing Acceptance Production
13. W H Y ? • To maintain stability. • To maintain security. • To maintain secrecy. • To maintain control.
14. M A N D AT E D • Industry and Legal Frameworks • Explicit and Implicit
15. I M P L I C I T (DSI-05) Production data shall not be replicated or used in non-production environments. Any use of customer data in non-production environments requires explicit, documented approval from all customers whose data is affected, and must comply with all legal and regulatory requirements for scrubbing of sensitive data elements. (IVS-08) Production and non-production environments shall be separated to prevent unauthorized access or changes to information assets. Separation of the environments may include: stateful inspection firewalls, domain/realm authentication sources, and clear segregation of duties for personnel accessing these environments as part of their job duties. Cloud Security Alliance, Cloud Controls Matrix
16. I M P L I C I T (10.i) Test data shall be selected carefully, and protected and controlled in non-production environments. HiTRUST CSF 9.2
17. I M P L I C I T (6.4.1) Separate development/test environments from production environments, and enforce the separation with access controls. PCI-DSS 3.2
18. E X P L I C I T (BAI07.04) Establish a test environment. Define and establish a secure test environment representative of the planned business process and IT operations environment, performance and capacity, security, internal controls, operational practices, data quality and privacy requirements, and workloads. COBIT5
19. G O L D E N R U L E
20. P H O T O B Y U LV I S A FA R I O N U N S P L A S H
21. H T T P S : / / W W W. A P P L E M U S T. C O M / A P P L E - C O N T I N U E S - T O - I N V E S T- I N - S I N G A P O R E - I N D O N E S I A - C O D I N G - TA L E N T /
22. H T T P S : / / T W I T T E R . C O M / D O S N O S TA L G I C / S TAT U S / 8 0 8 1 0 6 1 2 4 1 3 8 4 7 9 6 1 6 / P H O T O / 1
24. Developer User
25. Developer User
26. B O O T S T R A P P I N G How can I build a report without any sample data? The requirements are in the data.
27. S A M P L I N G • Subsets (horizontal sampling) • De-identification (vertical sampling)
28. S C A L I N G • Laptops aren’t datacenters • Why would we want data all over everyone’s laptops? • What surprises are missing from the sample?
29. Why is the report treated the same as the rest of the application? Why can’t you do proper exploration using your production application?
30. What if we took you to production, instead of bringing production to your laptop?
31. C U S T O M I Z A B I L I T Y • Macros • Plugins • Extensions • “Internal Reprogrammability” • Web Pages, Emacs, ERPs, EHRs
32. “ D E V E L O P M O D E ” Most web app boilerplates include an “admin” console. Perhaps they should also include a “develop” console.
33. S A N D B O X E S A P P L I C AT I O N S H E L L I A M U I D ATA D E FA U LT B U S I N E S S L O G I C S A N D B O X E S E X P E R I M E N T 1 E X P E R I M E N T 2
34. C H A L L E N G E S • Maintaining Security and Integrity • DoS / Resource Exhaustion • Bugs • What about … change management?
35. S T R AT E G I E S • Functional Core, Imperative Shell • Declarative Programming • Immutable Data • Traffic Routing and Service Meshes
36. F U N C T I O N A L C O R E , I M P E R AT I V E S H E L L • Phrase coined by Gary Bernhardt of Destroy All Software • https://www.destroyallsoftware.com/talks/boundaries • The application core is implemented in purely functional logic. It makes no mutations. • The application shell cannot be modified in production, so the dangerous parts are isolated.
37. D E C L A R AT I V E P R O G R A M M I N G • SQL • Configuration files • Business Rules • Tell the shell what to do, not how it should do it.
38. I M M U TA B L E D ATA • Make data pipelines non-destructive • Leverage tools like Pachyderm (pachyderm.io) • Also allows you to quickly promote changes to “full production” because they will have been built on a branch and verified first. • Also helps with data provenance
39. A L I A S I N G R AW D A I LY D A I LY ’ D A S H B O A R D ?
40. T R A F F I C R O U T I N G • Associate the selected branch with the user’s authenticated session • Route their requests based on that attribute • If using a service oriented architecture, a service mesh may be helpful to ensure full isolation
41. P R I O R A R T • Looker (BI) • CMS (WordPress, SquareSpace) • Marketing Automation (Optimizely, Monetate)
42. PA A S ? • Shares a lot of similarities • You (probably) shouldn’t build your own Pass • Doesn’t mean you can’t have extensibility in your own application.
43. S TA R T S M A L L • Feature Flags • Business Rules • Marketing Automation
44. C O N F I G U R AT I O N I S C O D E • Lots of apps have a database table with “configuration” • Make it configuration in your existing app and pipeline • Move it to its own git repository • Then put that in production!
45. T H A N K Y O U