Building an Ansible AAP cluster - Design and implementation

Designing an Ansible Automation Platform cluster is not as easy as it looks. Specifically when everything is spread over the globe and should be high available.

Transcript

1. Ton Kersten
   Velp / The Netherlands / 2023
   T.Kersten@ATComputing.nl
   Building an AAP cluster
   Design decisions and implementation
   

   View Slide

2. v1a – 2
   $ who am i
   Name: Ton Kersten
   From: Groesbeek / The Netherlands
   ●
   UNIX/Linux consultant and Trainer @ AT Computing
   ●
   UNIX freak (started in 1986 with SunOS)
   ●
   Linux Geek (started in 1992 with 0.96α)
   ●
   Configuration Management Addict
   ●
   Red Hat Certified System Engineer
   ●
   Ansible user and contributor since 2012
   ●
   Member of the Ansible Organization on Github
   ●
   Ansible Ambassador since 2015
   ●
   Co-organizer of the Ansible Benelux Meetup Group
   ●
   Free and Open Source Software Enthusiast
   

   View Slide

3. v1a – 3
   Ansible Automation Platform
   What Red Hat says:
   Red Hat® Ansible® Automation Platform elevates automation
   across your organization, expanding your possibilities. It's a
   flexible, security-focused foundation to build and deploy
   automation that helps your business accelerate, orchestrate, and
   innovate.
   What I say:
   Red Hat® Ansible® Automation Platform is a webgui and tools
   around Ansible to help automate tasks. It supports RBAC for fine
   grained control and a scheduler to automate tasks.
   

   View Slide

4. v1a – 4
   $BIGCORP wants AAP
   aap.ansilab.nl
   

   View Slide

5. v1a – 5
   $BIGCORP wants AAP Cluster – Step 1
   aap01.ansilab.nl aap02.ansilab.nl
   dc1 – baluchitherium
   192.168.x.x
   dc2 – c’mon everybody
   172.16.x.x
   High availability is required
   Baluchitherium - Van Halen
   Ansible 1.0
   C'mon Everybody - Led Zeppelin
   Ansible 2.14 - AAP 2.3 version
   

   View Slide

6. v1a – 6
   $BIGCORP wants AAP Cluster – Step 2
   aap01.ansilab.nl aap02.ansilab.nl
   ee01.ansilab.nl ee02.ansilab.nl
   Separate execution environments
   dc2 – c’mon everybody
   172.16.x.x
   dc1 – baluchitherium
   192.168.x.x
   

   View Slide

7. v1a – 7
   $BIGCORP wants AAP Cluster – Step 3
   aap01
   ee01
   db01
   aap02
   ee02
   db02
   NO HA
   PostgreSQL PostgreSQL
   Separate database servers
   dc2 – c’mon everybody
   172.16.x.x
   dc1 – baluchitherium
   192.168.x.x
   

   View Slide

8. v1a – 8
   $BIGCORP wants AAP Cluster – Step 4
   aap01
   ee01
   db01
   aap02
   ee02
   db02
   Virtual IP
   Not routable
   PostgreSQL
   +
   EFM
   PostgreSQL
   +
   EFM
   Add Enterprise Failover Manager
   dc2 – c’mon everybody
   172.16.x.x
   dc1 – baluchitherium
   192.168.x.x
   

   View Slide

9. v1a – 9
   $BIGCORP wants AAP Cluster – Step 5
   aap01
   ee01
   db01
   aap02
   ee02
   db02
   EFM
   Bouncer
   EFM
   Bouncer
   bn01 bn02
   Add Bouncer nodes
   dc2 – c’mon everybody
   172.16.x.x
   dc1 – baluchitherium
   192.168.x.x
   

   View Slide

10. v1a – 10
    $BIGCORP wants AAP Cluster – Step 6
    aap01
    ee01
    db01
    aap02
    ee02
    db02
    bn01 bn02
    ah01 ah02
    Shared storage
    required
    Split AAP into CN and AH
    dc2 – c’mon everybody
    172.16.x.x
    dc1 – baluchitherium
    192.168.x.x
    

    View Slide

11. v1a – 11
    $BIGCORP wants AAP Cluster – Intermezzo
    aap01
    ee01
    db01
    aap02
    ee02
    db02
    bn01 bn02
    ah01 ah02
    gluster01 gluster02
    No free GlusterFS for
    RHEL
    CentOS/Rocky/Alma
    not allowed
    dc2 – c’mon everybody
    172.16.x.x
    dc1 – baluchitherium
    192.168.x.x
    

    View Slide

12. v1a – 12
    $BIGCORP wants AAP Cluster – Final
    aap01
    ee01
    db01
    aap02
    ee02
    db02
    bn01 bn02
    ah01 ah02
    Use shared storage from other department
    dc2 – c’mon everybody
    172.16.x.x
    store01
    dc1 – baluchitherium
    192.168.x.x
    

    View Slide

13. v1a – 13
    $BIGCORP IP addresses
    aap01 – 10.10
    ee01 – 10.20
    db01 – 10.30
    aap02 - 10.10
    ee02 - 10.20
    db02 - 10.30
    bn01 – 10.50 bn02 - 10.50
    ah01 – 10.40 ah02 – 10.40
    dc2 – c’mon everybody
    172.16.x.x
    dc1 – baluchitherium
    192.168.x.x
    https://aap.ansilab.nl
    https://ah.ansilab.nl
    bn.ansilab.nl:6432
    

    View Slide

14. v1a – 14
    Configuration - Database servers
    listen_addresses = '*'
    postgresql.conf
    # Ansble AAP Cluster - Control nodes
    host awx awx 192.168.10.10/32 md5
    host awx awx 172.16.10.10/32 md5
    # Ansible AAP Cluster - Automation Hubs
    host autohub autohub 192.168.10.40/32 md5
    host autohub autohub 172.16.10.40/32 md5
    # Database replication - Needed for EFM HA
    host replication replica 192.168.10.30/32 md5
    host replication replica 172.16.10.30/32 md5
    # PostgreSQL EFM cluster check
    host clustcheckdb clustchecker 192.168.10.30/32 trust
    host clustcheckdb clustchecker 172.16.10.30/32 trust
    pg_hba.conf
    

    View Slide

15. v1a – 15
    Configuration - Database servers - EFM
    192.168.10.30
    172.168.10.30
    efm.nodes
    db.user=clustchecker
    db.password.encrypted=xxxxxx
    db.port=5432
    db.database=clustcheckdb
    db.service.owner=postgres
    db.config.dir=/var/lib/pgsql/13/data
    bind.address=192.168.10.30:7800
    admin.port=7809
    is.witness=false
    local.period=10
    local.timeout=60
    local.timeout.final=10
    pg_hba.conf
    

    View Slide

16. v1a – 16
    Configuration - Bouncer nodes
    [databases]
    awx= host=192.168.10.30
    autohub= host=192.168.10.30
    edb-pgbouncer-databases.ini
    Active database node
    "awx" "SCRAM-SHA-256$xxxxxx"
    "autohub" "SCRAM-SHA-256$4096:xxxxxx"
    userlist.txt
    psql -Atq -U postgres d postgres -c \
    "SELECT concat('\"', usename, '\" \"', passwd, '\"') FROM pg_shadow"
    Generate userlist.txt
    

    View Slide

17. v1a – 17
    Configuration - AAP Inventory - Part 1
    [automationcontroller]
    aap01.ansilab.nl node_type=control
    aap02.ansilab.nl node_type=control
    [automationcontroller:vars]
    peers=execution_nodes
    [execution_nodes]
    ee01.ansilab.nl node_type=execution
    ee02.ansilab.nl node_type=execution
    [automationhub]
    ah01.ansilab.nl
    ah02.ansilab.nl
    [automationcatalog]
    [database]
    [sso]
    inventory
    Do not specify database nodes
    

    View Slide

18. v1a – 18
    Configuration - AAP Inventory - Part 2
    [all:vars]
    admin_password='salami'
    #
    pg_host='bn.ansilab.nl'
    pg_port=6432
    pg_database='awx'
    pg_username='awx'
    pg_password='salami'
    pg_sslmode='prefer'
    #
    registry_url='https://registry.redhat.io'
    registry_username='RedHatAccount'
    registry_password='salami'
    receptor_listener_port=27199
    #
    automationhub_admin_password='salami'
    automationhub_pg_host='bn.ansilab.nl'
    automationhub_pg_port=6432
    automationhub_pg_database='autohub'
    automationhub_pg_username='autohub'
    automationhub_pg_password='salami'
    automationhub_pg_sslmode='prefer'
    inventory
    

    View Slide

19. v1a – 19
    Install AAP
    ./setup.sh
    And wait!!!
    

    View Slide

20. v1a – 20
    AAP Cluster overview
    

    View Slide

21. v1a – 21
    Questions
    – T.Kersten@ATComputing.nl
    – https://www.atcomputing.nl
    – https://www.tonkersten.com
    – https://github.com/tonk
    – https://speakerdeck.com/tonk
    – @TonKersten on Twitter
    - @tonk@mastodon.social on Mastodon
    – TKersten on IRC
    Where to find me
    Working at AT Computing
    hr@atcomputing.nl
    

    View Slide