Applying Graph Theory to Infrastructure As Code

As given at HashiConf EU 2016 in Amsterdam

Graphs are mathematical structures used to model relationships between objects. Modern infrastructures can be thought of as graphs — compute resource depend on network resources, which in turn depend on access control resources, and so on. At a certain scale, the relationships between all resources in an infrastructure become impractical to reason about.

Terraform builds a graph to model the relationships between resources so operators can safely manage and change infrastructure resources across bare metal, IaaS, PasS, and SaaS. Terraform models and potentially prevents that simple change with unforeseen consequences so operators don't have to.

In this talk, we will explore the graph theory at the heart of Terraform's orchestration engine.

Transcript

1. Graph Theory to Applying Infrastructure as Code HashiConf EU 2016
2. Infrastructure as Code Graph Theory to Applying
3. Maturity
4. Manual Everything
5. Basic Automa>on #!/bin/bash
6. Machine Virtualiza>on
7. Commodi>za>on of Infrastructure
8. Datacenter-as-Computer
12. Capability Complexity
14. Managing Complexity A strategy for
16. 535 :global_includes, 536 :gmail_domain 537 ].each do |key| 538 params[:account][:settings].try(:delete, key) 539 end 540 end 541 542 if params[:account][:settings] && params[:account][:settings].has_key?(:trusted_referers) 543 if trusted_referers = params[:account][:settings].delete(:trusted_referers) 544 @account.trusted_referers = trusted_referers if @account.root_account? 545 end 546 end 547 548 if sis_id = params[:account].delete(:sis_source_id) 549 if !@account.root_account? && sis_id != @account.sis_source_id && @account.root_account.grant 550 if sis_id == '' 551 @account.sis_source_id = nil 552 else 553 @account.sis_source_id = sis_id 554 end 555 end 556 end 557 558 process_external_integration_keys 559 560 can_edit_email = params[:account][:settings].try(:delete, :edit_institution_email) 561 if @account.root_account? && !can_edit_email.nil? 562 @account[:settings][:edit_institution_email] = value_to_boolean(can_edit_email) 563 end 564 565 if @account.update_attributes(params[:account]) 566 format.html { redirect_to account_settings_url(@account) } 567 format.json { render :json => @account } 568 else 569 flash[:error] = t(:update_failed_notice, "Account settings update failed") 570 format.html { redirect_to account_settings_url(@account) } 571 format.json { render :json => @account.errors, :status => :bad_request } 572 end 573 end 574 end 575 end 576 577 def settings 578 if authorized_action(@account, @current_user, :read) 579 @available_reports = AccountReport.available_reports if @account.grants_right?(@current_user, @se 580 if @available_reports 581 @last_complete_reports = {} 582 @last_reports = {} 583 if AccountReport.connection.adapter_name == 'PostgreSQL' 584 scope = @account.account_reports.select("DISTINCT ON (report_type) account_reports.*").order( 585 @last_complete_reports = scope.last_complete_of_type(@available_reports.keys, nil).preload(:a 586 @last_reports = scope.last_of_type(@available_reports.keys, nil).index_by(&:report_type) 587 else 588 @available_reports.keys.each do |report| 589 @last_complete_reports[report] = @account.account_reports.last_complete_of_type(report).fir 590 @last_reports[report] = @account.account_reports.last_of_type(report).first 591 end 592 end 593 end 594 load_course_right_side 595 @account_users = @account.account_users 596 ActiveRecord::Associations::Preloader.new(@account_users, user: :communication_channels).run 597 order_hash = {} 598 @account.available_account_roles.each_with_index do |role, idx| 599 order_hash[role.id] = idx 600 end 601 @account_users = @account_users.select(&:user).sort_by{|au| [order_hash[au.role_id] || CanvasSort 602 @alerts = @account.alerts HASHICORP
17. 535 :global_includes, 536 :gmail_domain 537 ].each do |key| 538 params[:account][:settings].try(:delete, key) 539 end 540 end 541 542 if params[:account][:settings] && params[:account][:settings].has_key?(:trusted_referers) 543 if trusted_referers = params[:account][:settings].delete(:trusted_referers) 544 @account.trusted_referers = trusted_referers if @account.root_account? 545 end 546 end 547 548 if sis_id = params[:account].delete(:sis_source_id) 549 if !@account.root_account? && sis_id != @account.sis_source_id && @account.root_account.grant 550 if sis_id == '' 551 @account.sis_source_id = nil 552 else 553 @account.sis_source_id = sis_id 554 end 555 end 556 end 557 558 process_external_integration_keys 559 560 can_edit_email = params[:account][:settings].try(:delete, :edit_institution_email) 561 if @account.root_account? && !can_edit_email.nil? 562 @account[:settings][:edit_institution_email] = value_to_boolean(can_edit_email) 563 end 564 565 if @account.update_attributes(params[:account]) 566 format.html { redirect_to account_settings_url(@account) } 567 format.json { render :json => @account } 568 else 569 flash[:error] = t(:update_failed_notice, "Account settings update failed") 570 format.html { redirect_to account_settings_url(@account) } 571 format.json { render :json => @account.errors, :status => :bad_request } 572 end 573 end 574 end 575 end 576 577 def settings 578 if authorized_action(@account, @current_user, :read) 579 @available_reports = AccountReport.available_reports if @account.grants_right?(@current_user, @se 580 if @available_reports 581 @last_complete_reports = {} 582 @last_reports = {} 583 if AccountReport.connection.adapter_name == 'PostgreSQL' 584 scope = @account.account_reports.select("DISTINCT ON (report_type) account_reports.*").order( 585 @last_complete_reports = scope.last_complete_of_type(@available_reports.keys, nil).preload(:a 586 @last_reports = scope.last_of_type(@available_reports.keys, nil).index_by(&:report_type) 587 else 588 @available_reports.keys.each do |report| 589 @last_complete_reports[report] = @account.account_reports.last_complete_of_type(report).fir 590 @last_reports[report] = @account.account_reports.last_of_type(report).first 591 end 592 end 593 end 594 load_course_right_side 595 @account_users = @account.account_users 596 ActiveRecord::Associations::Preloader.new(@account_users, user: :communication_channels).run 597 order_hash = {} 598 @account.available_account_roles.each_with_index do |role, idx| 599 order_hash[role.id] = idx 600 end 601 @account_users = @account_users.select(&:user).sort_by{|au| [order_hash[au.role_id] || CanvasSort 602 @alerts = @account.alerts HASHICORP Infrastructure as Code HASHICORP
18. 535 :global_includes, 536 :gmail_domain 537 ].each do |key| 538 params[:account][:settings].try(:delete, key) 539 end 540 end 541 542 if params[:account][:settings] && params[:account][:settings].has_key?(:trusted_referers) 543 if trusted_referers = params[:account][:settings].delete(:trusted_referers) 544 @account.trusted_referers = trusted_referers if @account.root_account? 545 end 546 end 547 548 if sis_id = params[:account].delete(:sis_source_id) 549 if !@account.root_account? && sis_id != @account.sis_source_id && @account.root_account.grant 550 if sis_id == '' 551 @account.sis_source_id = nil 552 else 553 @account.sis_source_id = sis_id 554 end 555 end 556 end 557 558 process_external_integration_keys 559 560 can_edit_email = params[:account][:settings].try(:delete, :edit_institution_email) 561 if @account.root_account? && !can_edit_email.nil? 562 @account[:settings][:edit_institution_email] = value_to_boolean(can_edit_email) 563 end 564 565 if @account.update_attributes(params[:account]) 566 format.html { redirect_to account_settings_url(@account) } 567 format.json { render :json => @account } 568 else 569 flash[:error] = t(:update_failed_notice, "Account settings update failed") 570 format.html { redirect_to account_settings_url(@account) } 571 format.json { render :json => @account.errors, :status => :bad_request } 572 end 573 end 574 end 575 end 576 577 def settings 578 if authorized_action(@account, @current_user, :read) 579 @available_reports = AccountReport.available_reports if @account.grants_right?(@current_user, @se 580 if @available_reports 581 @last_complete_reports = {} 582 @last_reports = {} 583 if AccountReport.connection.adapter_name == 'PostgreSQL' 584 scope = @account.account_reports.select("DISTINCT ON (report_type) account_reports.*").order( 585 @last_complete_reports = scope.last_complete_of_type(@available_reports.keys, nil).preload(:a 586 @last_reports = scope.last_of_type(@available_reports.keys, nil).index_by(&:report_type) 587 else 588 @available_reports.keys.each do |report| 589 @last_complete_reports[report] = @account.account_reports.last_complete_of_type(report).fir 590 @last_reports[report] = @account.account_reports.last_of_type(report).first 591 end 592 end 593 end 594 load_course_right_side 595 @account_users = @account.account_users 596 ActiveRecord::Associations::Preloader.new(@account_users, user: :communication_channels).run 597 order_hash = {} 598 @account.available_account_roles.each_with_index do |role, idx| 599 order_hash[role.id] = idx 600 end 601 @account_users = @account_users.select(&:user).sort_by{|au| [order_hash[au.role_id] || CanvasSort 602 @alerts = @account.alerts HASHICORP
19. Graph Theory to Applying Infrastructure as Code
20. Discrete Mathema>cs Combinatorics Graph Theory
21. Vertex / Node
22. Edge
23. Directed Edge
24. Directed Graph
25. Cycle
26. Directed Acyclic Graph (DAG)
27. Walk 1 2 3 4 5
28. Parallel Walk 1 3 2 2 3
29. Graph Transform T G G'
31. Graph Theory to Applying Infrastructure as Code HashiConf EU 2016
32. Terraform's Internals: Structure Core Providers Providers Providers Upstream APIs Plugins
33. Terraform's Internals: Provider API Separa>on Core Providers Providers Providers Upstream APIs Plugins Diff() Apply() Refresh() Create() Read() Update() Delete()
34. Terraform's Internals: Core Concepts Config: Target Reality State: Current Reality Diff: {Config - State} Plan: Presents Diff Apply: Resolves Diff
35. Terraform's Internals: Core Packages Core dag config terraform reads makes
36. Terraform's Internals: Core Packages Core dag config terraform reads makes
37. Building the Graph T T T
38. Building the Graph Config ConfigTransformer T
39. Building the Graph aws_instance aws_elb dnsimple_record aws dnsimple
40. Building the Graph ProviderTransformer T
41. Building the Graph aws_instance aws_elb dnsimple_record aws dnsimple
42. Building the Graph T T T
43. Building the Graph T T RootTransformer R
44. Building the Graph aws_instance aws_elb dnsimple_record aws dnsimple root
45. Walking The Graph aws_instance aws_elb dnsimple_record aws dnsimple root 1 2 2 3 3 4
46. What do we get from the graph?
47. Separa>on of Concerns Core dag terraform
48. Parallel Walk
49. Solu>ons to Borrow Transitive Reduction
50. Solid Theore>cal Grounding + Room to Mature & Explore
51. Graph Theory to Applying Infrastructure as Code HashiConf EU 2016
52. by Jason Santos via CloudcraZ A[ribu>ons