ANSIBLETO MAY 2019 - OPENSTACK-ANSIBLE 

WHAT IS OPENSTACK? 

“OpenStack is a cloud operating system that controls large pools of compute, storage, and networking resources throughout a datacenter, all managed and provisioned through APIs with common authentication mechanisms.” - https://www.openstack.org/software/ WHAT IS OPENSTACK? 

No content

No content

HYPE CYCLE 

● Originally thought it would be heavily used in Enterprise ○ ...nothing lives up to the total hype (see Kubernetes) ○ But did not see expected success there ● Currently most growth in telecom/Network Function Virtualization (NFV) ○ By most accounts, vast majority of telecoms will run OpenStack in some capacity WHAT IS OPENSTACK? 

● It is also a foundation -- The OpenStack Foundation, OSF ● The OSF supports and governs more than just OpenStack: ○ Zuul - Advanced CI/CD ○ Kata Containers - Containers that are VMs (huh?) ○ Airship - Declarative k8s and OpenStack for Telecoms ○ StarlingX - Edge k8s and OpenStack ■ FYI, I am on the StarlingX Technical Steering Committee WHAT IS OPENSTACK? 

● Moved projects from openstack.org to opendev.org ● OpenStack has the Open Infra Summit AND the Project Team Gathering (PTG) ○ Summit is presentations, vendor marketplace, forum, etc ○ The PTG is where developers get together face to face and discuss their projects WHAT IS OPENSTACK? 

OPENDEV 

“If one is going to argue that OpenStack has no opportunity in front of it, then, it is necessary to explain either why there will be no private infrastructure or why in a world in which private infrastructure remains a viable option that OpenStack will not be a candidate for those workloads.” - RedMonk May 2019 WHAT IS OPENSTACK? 

OPENSTACK-ANSIBLE 

THE PEOPLE THAT MAKE OSA * Note: I’m not one of them...haven’t contributed any code 

$ git log $(git log --pretty=format:%H|tail -1) commit 6f6e75f5496910724209929e32730f5927373585 Author: d34dh0r53 Date: Tue Aug 26 18:08:15 2014 -0500 Initial Commit WHAT IS OPENSTACK? 

● “Our project does not have a fancy name” - It deploys OpenStack using Ansible (surprise!) ○ A way to install and manage OpenStack using Ansible via containers or bare metal ○ Who is is used by? ■ Hosting companies ■ Upstream developers, sometimes instead of devstack ■ 20+ different companies (May 2018) ■ Rackspace typically main contributor, but participation is less over time ● Diverse affiliation ● Independants have a big part to play OPENSTACK-ANSIBLE 

STACKALYTICS https://www.stackalytics.com/ 

STACKALYTICS https://www.stackalytics.com/ 

OPENSTACK-ANSIBLE ARCHITECTURE 

● Overall concept is to use lxc containers to run various OpenStack control plane components ○ lxc containers are used as “system containers” ie, more like virtual machines than what we see with “application containers” ala docker ○ Also some support for systemd-nspawn ■ Did you know you get a container management system with systemd? ○ Does add complexity, especially around networking ○ Can also deploy to bare metal ● Compute nodes are bare metal ● Recent project updates suggest they may move away from containers and make deploying without containers, ie. bare metal, the default 

LXC COMPARED TO DOCKER 

EXAMPLE PRODUCTION ARCHITECTURE https://docs.openstack.org/openstack-ansible/latest/user/prod/example.html 

● Historically based on deploying from source ○ ie. not using operating system packages ● A large part of the deployment, time wise, was building a “cheese shop” for Python wheels and compiling the wheels ○ OpenStack is mostly python ○ This time has been reduced recently by doing builds in parallel to the deployment ○ Just in time compilation ● You can use operating system packages instead ○ Ubuntu Cloud Archive ○ RDO 

● Current stable - OpenStack Rocky ○ https://releases.openstack.org/rocky/#rocky-openstack-ansible ○ 18.x.y - 18.1.6 ● Upcoming release - OpenStack Stein ○ https://releases.openstack.org/stein/#stein-openstack-ansible ● Future - OpenStack Train... 

OPEN INFRA SUMMIT WORKSHOPS - DENVER 2019 

● Infrastructure provide by packet.com ○ Baremetal public cloud ○ l3 by default, makes for interesting networking as most projects default to expecting layer 2 ○ Fairly good provisioning support in Ansible, better support with Terraform ● All the nodes provisioned with Terraform ● OpenStack is then actually deployed by openstack-ansible project, which is one of the largest, most Advanced Ansible systems available OPENSTACK-ANSIBLE WORKSHOP 

OPENSTACK-ANSIBLE WORKSHOP 

No content

# tf apply Apply complete! Resources: 9 added, 0 changed, 3 destroyed. Outputs: Cloud ID Tag = e100d03f Compute public IPs = [ IP ] Infra/Control public IPs = [ IP ] Project ID = ID SSH Access to compute0 = ssh root@IP -i default.pem SSH Access to infra0 = ssh root@IP -i default.pem real 6m18.732s user 0m2.798s sys 0m1.467s EXAMPLE TERRAFORM RUN Thanks Packet! 

● Workshop code: https://github.com/OpenStackSanDiego/osa-workshop ○ NOTE: Currently having an issue with VXLAN and linuxbridge in Packet, can boot instances but no vxlan connectivity (yet) 

DEPLOYING OPENSTACK-ANSIBLE 

● OpenStack-Ansible doesn't manage the bare metal like some other deployment tools do ○ Eg. TripleO...using Ironic ● Need to setup the hosts on which OpenStack will be deployed to ○ Bare metal OS to be installed ○ Bare metal networking needs to be setup ● Also need somewhere to run OpenStack-Ansible from - Usually a VM REQUIREMENTS 

● Deployment host apt-get install -y aptitude build-essential git ntp ntpdate python-dev sudo git clone -b 18.1.1 https://git.openstack.org/openstack/openstack-ansible /opt/openstack-ansible cd /opt/openstack-ansible scripts/bootstrap-ansible.sh cp -R /opt/openstack-ansible/etc/openstack_deploy/ /etc/openstack_deploy ● Configure /etc/openstack_delpoy/openstack_user_config.yml and potentially /etc/openstack_deploy/user_variables.yml ● Networking is typically the most challenging to get configured properly ○ Need to make decisions about networking, configure networking, configure OSA networking... SETUP DEPLOYMENT HOST 

cd /opt/openstack-ansible # Create passwords ./scripts/pw-token-gen.py --file /etc/openstack_deploy/user_secrets.yml cd /opt/openstack-ansible/playbooks/ # Test configuration openstack-ansible setup-infrastructure.yml --syntax-check # Deploy! openstack-ansible setup-hosts.yml openstack-ansible setup-infrastructure.yml openstack-ansible setup-openstack.yml # or, use setup-everything.yml DEPLOY OPENSTACK 

EXPLORING OPENSTACK-ANSIBLE...ER...OPENSTACK 

ssh p2 tmux cd ~ cat osa-playbooks.sh bash osa-playbooks.sh DEMO - LAUNCH OPENSTACK ANSIBLE 

ssh p1 lxc-ls lxc-attach infra0_utility_container-88288177 source ~/openrc alias os=openstack os server list ACCESS EXISTING DEPLOYMENT 

source ~/openrc wget http://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_ 64-disk.img openstack image create --file cirros-0.4.0-x86_64-disk.img --disk-format qcow2 --container-format bare --public cirros rm cirros-0.4.0-x86_64-disk.img openstack flavor create --ram 512 --disk 1 --vcpus 1 m1.tiny BOOT A VM IN OPENSTACK 

source ~/openrc alias os=openstack os network create --provider-network-type vxlan test-vxlan os subnet create --network test-vxlan --subnet-range 192.168.0.0/24 test-vxlan-subnet os server create --flavor m1.tiny --image cirros cirros_1 os server list BOOT A VM IN OPENSTACK 

EXPLORE OPENSTACK-ANSIBLE...ER...ANSIBLE 

root@infra0:~# ansible --version ansible 2.5.14 SNIP! OPENSTACK-ANSIBLE COMMAND LINE 

root@infra0:/opt/openstack-ansible/playbooks# which openstack-ansible /usr/local/bin/openstack-ansible root@infra0:/opt/openstack-ansible/playbooks# file `which openstack-ansible` /usr/local/bin/openstack-ansible: Bourne-Again shell script, ASCII text executable root@infra0:~# ls /usr/local/bin ansible ansible-inventory lxc-veth-check ansible-config ansible-playbook openstack-ansible ansible-connection ansible-pull openstack-ansible.rc ansible-console ansible-vault openstack-host-hostfile-setup.sh ansible-doc hatop ansible-galaxy lxc-system-manage OPENSTACK-ANSIBLE COMMAND LINE 

root@infra0:/opt/openstack-ansible# ls -F ansible-role-requirements.yml LICENSE.txt scripts/ ansible-role-requirements.yml.example openstack_ansible.egg-info/ setup.cfg bindep.txt osa_toolkit/ setup.py deploy-guide/ playbooks/ test-requirements.txt doc/ README.rst tests/ etc/ releasenotes/ tox.ini global-requirement-pins.txt requirements.txt Vagrantfile inventory/ run_tests.sh* zuul.d/ DIRECTORY STRUCTURE 

● Several different kinds of Ansible “plugins” used ○ Strategy: https://docs.ansible.com/ansible/latest/plugins/strategy.html ○ Filters ○ Callbacks ○ Connection ● https://docs.openstack.org/openstack-ansible-plugins/latest/ ● https://opendev.org/openstack/openstack-ansible-plugins PLUGINS 

root@controller-01:/opt/openstack-ansible# ls /etc/ansible/roles/ ansible-hardening keepalived os_blazar os_magnum os_swift rsyslog_server apt_package_pinning lxc_container_create os_ceilometer os_manila os_tacker sshd bird lxc_hosts os_cinder os_masakari os_tempest systemd_mount ceph-ansible memcached_server os_congress os_mistral os_trove systemd_networkd ceph_client nspawn_container_create os_designate os_neutron plugins systemd_service config_template nspawn_hosts os_glance os_nova python_venv_build unbound etcd opendaylight os_gnocchi os_octavia qdrouterd galera_client openstack_hosts os_heat os_panko rabbitmq_server galera_server openstack_openrc os_horizon os_placement repo_server haproxy_endpoints os_aodh os_ironic os_rally resolvconf haproxy_server os_barbican os_keystone os_sahara rsyslog_client AFTER BOOTSTRAP ANSIBLE - INSTALLED ROLES To find all OpenStack-Ansible roles: https://opendev.org/explore/repos?q=openstack-ansible&tab=&sort=recentupdate 

root@controller-01:/opt/openstack-ansible/playbooks# ls ceph-install.yml healthcheck-infrastructure.yml os-gnocchi-install.yml os-swift-sync.yml ceph-rgw-install.yml healthcheck-openstack.yml os-heat-install.yml os-tacker-install.yml ceph-rgw-keystone-setup.yml infra-journal-remote.yml os-horizon-install.yml os-tempest-install.yml common-playbooks listening-port-report.yml os-ironic-install.yml os-trove-install.yml common-tasks lxc-containers-create.yml os-keystone-install.yml qdrouterd-install.yml containers-deploy.yml lxc-containers-destroy.yml os-magnum-install.yml rabbitmq-install.yml containers-lxc-create.yml lxc-hosts-setup.yml os-manila-install.yml repo-install.yml containers-lxc-destroy.yml memcached-install.yml os-masakari-install.yml roles containers-lxc-host.yml openstack-hosts-setup.yml os-mistral-install.yml rsyslog-install.yml containers-nspawn-create.yml os-aodh-install.yml os-neutron-install.yml security-hardening.yml containers-nspawn-destroy.yml os-barbican-install.yml os-nova-install.yml setup-everything.yml containers-nspawn-host.yml os-blazar-install.yml os-octavia-install.yml setup-hosts.yml defaults os-ceilometer-install.yml os-panko-install.yml setup-infrastructure.yml etcd-install.yml os-cinder-install.yml os-placement-install.yml setup-openstack.yml galera-install.yml os-congress-install.yml os-rally-install.yml unbound-install.yml haproxy-install.yml os-designate-install.yml os-sahara-install.yml utility-install.yml healthcheck-hosts.yml os-glance-install.yml os-swift-install.yml OSA PLAYBOOKS 

Took a random look at openstack_hosts and the rabbitmq role INTERESTING ANSIBLE SNIPPETS 

root@controller-01:/opt/openstack-ansible/playbooks# cat setup-hosts.yml --- SNIP! - import_playbook: openstack-hosts-setup.yml - import_playbook: security-hardening.yml - import_playbook: containers-deploy.yml IMPORT_PLAYBOOK 

root@controller-01:/opt/openstack-ansible/playbooks# cat openstack-hosts-setup.yml SNIP! pre_tasks: - name: Ensure python is installed register: result raw: | if which apt-get >/dev/null && ! which python >/dev/null ; then apt-get -y install python exit 2 else exit 0 fi changed_when: "result.rc == 2" failed_when: "result.rc not in [0, 2]" SNIP! ENSURE PYTHON2... 

- name: Gather variables for each operating system include_vars: "{{ item }}" with_first_found: - "{{ ansible_distribution | lower }}-{{ ansible_distribution_version | lower }}.yml" - "{{ ansible_distribution | lower }}-{{ ansible_distribution_major_version | lower }}.yml" - "{{ ansible_os_family | lower }}-{{ ansible_distribution_major_version | lower }}.yml" - "{{ ansible_distribution | lower }}.yml" - "{{ ansible_os_family | lower }}.yml" tags: - always WITH_FIRST_FOUND 

# Proxy configuration applies to all nodes - name: Add global_environment_variables to environment file blockinfile: dest: "/etc/environment" state: present marker: "# {mark} Managed by OpenStack-Ansible" insertbefore: EOF block: "{{ lookup('template', 'environment.j2') }}" tags: - openstack_hosts-config BLOCKINFILE 

- name: Remove the blacklisted packages package: name: "{{ openstack_hosts_package_list | selectattr('state','equalto','absent') | map(attribute='name') | list }}" state: absent PACKAGE 

# When updating the cache in the apt_repository # task, and the update fails, a retry does not # detect a change the second attempt and therefore # does not update the cache, resulting in a changed # repo config, but no updated cache. To work around # this bug we implement the change of repo config # and the cache update as two seperate tasks. - name: Add/Remove/Update standard and user defined repositories apt_repository: repo: "{{ repo.repo }}" state: "{{ repo.state | default('present') }}" filename: "{{ repo.filename | default(omit) }}" update_cache: no with_items: "{{ openstack_hosts_package_repos }}" loop_control: loop_var: repo register: _adding_apt_repo PACKAGE - name: Update Apt cache apt: update_cache: yes when: - _adding_apt_repo is changed register: _update_apt_cache until: _update_apt_cache is success changed_when: false retries: 5 delay: 2 

# unbound-clients.yml - name: Apply resolver sorting set_fact: resolver_list: |- {% set _var = [] %} {% for host in groups[resolvconf_resolver_group] %} {% if physical_host is defined and hostvars[host]['physical_host'] is defined and physical_host == hostvars[host]['physical_host'] %} {% set _prio = 100 %} {% else %} {% set _prio = 50 %} {% endif %} {% set _ = _var.append({ 'host': hostvars[host]['ansible_host'], 'priority': _prio }) %} {% endfor %} {{ _var | sort(reverse=true, attribute='priority') | map(attribute='host') | list }} SNIP! SET_FACT WITH A TEMPLATE 

- name: Drop hosts file entries script locally template: src: "openstack-host-hostfile-setup.sh.j2" dest: "/var/tmp/openstack-host-hostfile-setup.sh" mode: "0755" delegate_to: localhost run_once: true vars: ansible_python_interpreter: "/usr/bin/python" - name: Copy templated hosts file entries script template: src: "/var/tmp/openstack-host-hostfile-setup.sh" dest: "/usr/local/bin/openstack-host-hostfile-setup.sh" mode: "0755" - name: Stat host file stat: path: /etc/hosts register: stat_hosts SOMETIMES YOU HAVE TO RUN SCRIPTS - name: Update hosts file command: "/usr/local/bin/openstack-host-hostfile-setup.sh" register: update_hosts changed_when: not stat_hosts.stat.exists or stat_hosts.stat.checksum | string != update_hosts.stdout | string 

- name: Stop rabbitmq app shell: | rabbitmqctl stop_app sleep 5 args: executable: /bin/bash tags: # This task must use shell, otherwise the rabbitmqctl # command somehow thinks that the sleep command is a # parameter. Due to this, we skip ansible-lint checks # on this task. - skip_ansible_lint SHELL COMMAND 

- include_tasks: rabbitmq_restart.yml when: rabbit_config is changed or rabbitmq_plugin is changed or cookie_set is changed or rabbitmq_upgrade | bool INCLUDE_TASKS WHEN.... 

- name: Download the RabbitMQ package get_url: url: "{{ rabbitmq_package_url }}" dest: "{{ rabbitmq_package_path }}" mode: "0644" checksum: "sha256:{{ rabbitmq_package_sha256 }}" register: package_download retries: 3 delay: 10 until: package_download is success when: rabbitmq_install_method == 'file' tags: - rabbitmq-package-deb - rabbitmq-package-deb-get - rabbitmq-package-rpm - rabbitmq-package-rpm-get RETRIES...INTERNET IS UNRELIABLE... 

- name: Wait for rabbitmq to be started wait_for: host: 0.0.0.0 port: 5672 WAIT_FOR 

# See playbooks/roles/rabbitmq_server/defaults/main.yml to provide custom # subject material for certificates or specify a user-provided certificate and # key pair. - name: Create self-signed ssl cert command: > openssl req -new -nodes -sha256 -x509 -subj "{{ rabbitmq_ssl_self_signed_subject }}" -days 3650 -keyout {{ rabbitmq_ssl_key }} -out {{ rabbitmq_ssl_cert }} -extensions v3_ca creates={{ rabbitmq_ssl_cert }} tags: - rabbitmq-ssl MULTI-LINE COMMAND 

- name: Store ssl cert slurp: src: "{{ rabbitmq_ssl_cert }}" register: _rabbitmq_ssl_cert changed_when: false tags: - rabbitmq-ssl SLURP 

OPENSTACK TESTING 

● https://review.opendev.org/#/q/project:openstack/openstack-ansible ● ARA output: http://logs.openstack.org/51/661651/1/check/openstack-ansible-deploy-aio_ basekit-centos-7/3144d54/logs/ara-report/ EXAMPLE OUTPUT OF OPENSTACK CI/CD 

RECENT AND UPCOMING CHANGES IN OPENSTACK-ANSIBLE 

● Improved testing ● Documentation refactor - Wizard like ● systemd-nspawn groundwork ● Separation of install and configure - tags ● “convergence” - reduction of # of containers; opposite of microservices; move neutron agents to bare metal ● SUSE support ● 18.04 experimental support ● Full install with distribution package ● Reduction of # of variables QUEENS FEATURES - FROM OSA UPDATE MAY 2018 * These notes made from watching Ansible updates from OpenStack/Open Infra summits - may not be 100% accurate, please check with official project 

● May 2018 cont’d ○ Update to ansible 2.5 ○ New roles: Panko, Masakari, Congress, Blazar ○ Repobuild will go away ○ New library roles - ansible-config_template ● Dec update ○ Future - Offline support, python 3 (centos7 is a problem) ● May 2019 ○ Testing mitogen ○ Internationalization ○ Python venv refactor ○ Repo build refactor ○ Increased stability for CentOS ROCKY PLANS - FROM OSA UPDATE MAY 2018 * These notes made from watching Ansible updates from OpenStack/Open Infra summits - may not be 100% accurate, please check with official project 

● Integrate with mitogen - https://github.com/dw/mitogen ● Re-imagining roles - “Over time we started having a lot of variables, but if we have overrides why do we need all these variables...” ● Using pre-built wheels by default - OpenStack project builds wheels already ● Improving bare metal (and changing default architecture) ● Python 3 ● ceph-ansible - ceph nautilus GOALS FOR OPENSTACK TRAIN * These notes made from watching Ansible updates from OpenStack/Open Infra summits - may not be 100% accurate, please check with official project 

CONCLUSION 

● Email me: [email protected] ● Twitter: @ccollicutt ● Meetups ○ Toronto Open Source Networking Meetup ■ Only one so far, was hosted at Arctiq ○ Toronto Open Infra meetup: https://www.meetup.com/OpenStackTO/