Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Automated Configuration of Firmware

Puppet Labs
September 27, 2012

Automated Configuration of Firmware

Puppet Labs

September 27, 2012
Tweet

More Decks by Puppet Labs

Other Decks in Technology

Transcript

  1. APOLLO GROUP
    Automated Configuration of Firmware
    Michael Arnold
    Principal Systems Engineer
    27 September 2012

    View Slide

  2. 2
    APOLLO GROUP
    Who
    Tools
    Example Workflow
    Example Code
    Links
    Questions
    Agenda
    © 2012 Apollo Group

    View Slide

  3. 3
    APOLLO GROUP
    APOLLO GROUP
    Who
    © 2012 Apollo Group

    View Slide

  4. 4
    APOLLO GROUP
    Who is Apollo?
    © 2012 Apollo Group
    Apollo Group is a leading provider of higher
    education programs for working adults.

    View Slide

  5. 5
    APOLLO GROUP
    Systems Administrator
    Automation geek
    13 years in IT
    Hadoop Cluster Admin
    I deal with:
    –Server hardware specification/configuration
    –Server firmware
    –Server operating system
    –Hadoop Cluster performance and availability
    Who is Michael Arnold?
    © 2012 Apollo Group

    View Slide

  6. 6
    APOLLO GROUP
    It has nothing to do with Puppet.
    –Unless someone writes a module to program
    firmware from the running OS.
    • Which could work for RAID controllers.
    How does firmware tie in with Puppet?
    © 2012 Apollo Group

    View Slide

  7. 7
    APOLLO GROUP
    APOLLO GROUP
    Tools
    © 2012 Apollo Group

    View Slide

  8. 8
    APOLLO GROUP
    HP SmartStart Scripting Toolkit
    Dell OpenManage Deployment Toolkit
    IBM ServerGuide Scripting Toolkit
    Fujitsu ServerView Scripting Toolkit
    Hardware Toolkits
    © 2012 Apollo Group

    View Slide

  9. 9
    APOLLO GROUP
    Configuration of
    –BIOS
    –BMC (DRAC/iLO/RSA/etc.)
    –Hardware RAID
    –FC HBA
    What do the tools provide?
    © 2012 Apollo Group

    View Slide

  10. 10
    APOLLO GROUP
    Manually configure reference hardware
    Capture the firmware configuration
    Clone the configuration to multiple machines
    How does the toolkit work?
    © 2012 Apollo Group

    View Slide

  11. 11
    APOLLO GROUP
    Toolkit Components
    © 2012 Apollo Group
    Kernel &
    Initrd
    Firmware
    Tools
    Toolkit

    View Slide

  12. 12
    APOLLO GROUP
    Toolkit Requirements
    © 2012 Apollo Group
    Toolkit
    DHCP
    TFTP
    NFS

    View Slide

  13. 13
    APOLLO GROUP
    APOLLO GROUP
    Example Workflow
    © 2012 Apollo Group

    View Slide

  14. 14
    APOLLO GROUP
    Hardware is racked and cabled
    Switch ports are up
    Server is manually powered on
    Firmware defaults are to boot from network if no
    other boot device is found.
    –No other boot devices are found
    Assumptions
    © 2012 Apollo Group

    View Slide

  15. 15
    APOLLO GROUP
    All server hardware is from the same vendor (HP)
    Constraints
    © 2012 Apollo Group

    View Slide

  16. 16
    APOLLO GROUP
    Firmware
    Update
    Hardware
    Discovery
    Firmware
    Configuration
    Buildsystem
    Registration
    The Process
    © 2012 Apollo Group

    View Slide

  17. 17
    APOLLO GROUP
    Screencast
    © 2012 Apollo Group

    View Slide

  18. 18
    APOLLO GROUP
    Unknown system netboots
    Workflow
    © 2012 Apollo Group

    View Slide

  19. 19
    APOLLO GROUP
    Unknown system netboots
    Boot into HP SmartStart Scripting Toolkit
    Workflow
    © 2012 Apollo Group

    View Slide

  20. 20
    APOLLO GROUP
    Unknown system netboots
    Boot into HP SmartStart Scripting Toolkit
    Gather hardware information (CPU/RAM/disk/NIC)
    via hpdiscovery
    Workflow
    © 2012 Apollo Group

    View Slide

  21. 21
    APOLLO GROUP
    Unknown system netboots
    Boot into HP SmartStart Scripting Toolkit
    Gather hardware information (CPU/RAM/disk/NIC)
    via hpdiscovery
    If this is HP hardware, then
    –Update firmware
    –Configure BIOS
    –Configure iLO
    –Configure RAID
    Workflow
    © 2012 Apollo Group

    View Slide

  22. 22
    APOLLO GROUP
    Unknown system netboots
    Boot into HP SmartStart Scripting Toolkit
    Gather hardware information (CPU/RAM/disk/NIC)
    via hpdiscovery
    If this is HP hardware, then
    –Update firmware
    –Configure BIOS
    –Configure iLO
    –Configure RAID
    cobbler-register to the hardware profile
    Workflow
    © 2012 Apollo Group

    View Slide

  23. 23
    APOLLO GROUP
    Unknown system netboots
    Boot into HP SmartStart Scripting Toolkit
    Gather hardware information (CPU/RAM/disk/NIC) via
    hpdiscovery
    If this is HP hardware, then
    –Update firmware
    –Configure BIOS
    –Configure iLO
    –Configure RAID
    cobbler-register to the hardware profile
    System reboots
    Workflow
    © 2012 Apollo Group

    View Slide

  24. 24
    APOLLO GROUP
    Modify Cobbler system profile, giving it an identity
    (hostname, IP, OS metadata)
    Workflow
    © 2012 Apollo Group

    View Slide

  25. 25
    APOLLO GROUP
    Modify Cobbler system profile, giving it an identity
    (hostname, IP, OS metadata)
    Known system netboots
    Workflow
    © 2012 Apollo Group

    View Slide

  26. 26
    APOLLO GROUP
    Modify Cobbler system profile, giving it an identity
    (hostname, IP, OS metadata)
    Known system netboots
    Boot into OS installer
    Workflow
    © 2012 Apollo Group

    View Slide

  27. 27
    APOLLO GROUP
    Modify Cobbler system profile, giving it an identity
    (hostname, IP, OS metadata)
    Known system netboots
    Boot into OS installer
    Install OS
    Workflow
    © 2012 Apollo Group

    View Slide

  28. 28
    APOLLO GROUP
    Modify Cobbler system profile, giving it an identity
    (hostname, IP, OS metadata)
    Known system netboots
    Boot into OS installer
    Install OS
    Install Puppet
    Workflow
    © 2012 Apollo Group

    View Slide

  29. 29
    APOLLO GROUP
    Modify Cobbler system profile, giving it an identity
    (hostname, IP, OS metadata)
    Known system netboots
    Boot into OS installer
    Install OS
    Install Puppet
    …
    Workflow
    © 2012 Apollo Group

    View Slide

  30. 30
    APOLLO GROUP
    Modify Cobbler system profile, giving it an identity
    (hostname, IP, OS metadata)
    Known system netboots
    Boot into OS installer
    Install OS
    Install Puppet
    …
    Profit!
    Workflow
    © 2012 Apollo Group

    View Slide

  31. 31
    APOLLO GROUP
    APOLLO GROUP
    Example Code
    © 2012 Apollo Group

    View Slide

  32. 32
    APOLLO GROUP
    LABEL dtk-install_hadoop
    kernel /images/dtk-3.5.1/SA.1
    append initrd=/images/dtk-3.5.1/SA.2
    ramdisk_size=156482 Stage3_type=cdrom quiet DEBUG=0
    share_location=192.168.1.1:/srv/nfs/ro/dtk-3.5.1
    share_type=nfs selinux=0 share_opts=ro,nolock
    share_script=install_hadoop.sh cprofile=Hadoop
    ipappend 2
    Dell TFTP Config
    © 2012 Apollo Group

    View Slide

  33. 33
    APOLLO GROUP
    LABEL sstk-install_hadoop
    kernel /images/sstk-8.50/vmlinuz
    append initrd=/images/sstk-8.50/initrd.img
    root=/dev/ram0 rw ramdisk_size=184248 ide=nodma
    ide=noraid pnpbios=off network=1
    sstk_mount=192.168.1.1:/srv/nfs/ro/sstk-8.50
    sstk_mount_type=nfs sstk_mount_options=ro,nolock
    sstk_script=install_hadoop.sh cprofile=Hadoop
    ipappend 2
    HP TFTP Config
    © 2012 Apollo Group

    View Slide

  34. 34
    APOLLO GROUP
    cd /opt/dell/toolkit/template/scripts
    /opt/dell/toolkit/template/scripts/syscap.sh /tmp/syscap
    /opt/dell/toolkit/template/scripts/raidcap.sh /tmp/raidcap
    /opt/dell/toolkit/template/scripts/raccap.sh /tmp/raccap
    # MegaCLI does not come with the DTK so "install" it.
    cp -a /opt/dell/toolkit/systems/MegaCLI/opt/MegaRAID \
    /opt/MegaRAID
    /opt/MegaRAID/MegaCli/MegaCli -CfgSave -a0 -f /tmp/mega.0
    Dell Config Capture
    © 2012 Apollo Group

    View Slide

  35. 35
    APOLLO GROUP
    cd /TOOLKIT
    ./conrep -s -f /tmp/conrep.dat # BIOS
    ./hpacuscripting -c /tmp/cpqacuxe.dat # RAID
    ./hplpcfg /s /tmp/hplpcfg.dat # Emulex HBA
    ./hpqlarep /S /tmp/hpqlarep.dat # Qlogic HBA
    if ./ifhw /tmp/hw_discovery.dat allboards.xml \
    "PCI:Integrated Lights-Out" 2> /dev/null ; then
    ./hponcfg -w /tmp/hponcfg.dat # iLO 2/3
    else
    ./lo100cfg -o /tmp/lo100cfg.dat # LO100i
    fi
    HP Config Capture
    © 2012 Apollo Group

    View Slide

  36. 36
    APOLLO GROUP
    *** Detecting system type ***
    System Information:
    Name : ProLiant DL360 G5
    Serial : USE123D456
    Arch : x86_64
    LNAME : ProLiantDL360G5
    SNAME : DL360 G5
    Asset Tag : 01234567
    Platform Detection
    © 2012 Apollo Group

    View Slide

  37. 37
    APOLLO GROUP
     HP SSSTK has a function library that sets many variables.
    –$PRODUCT_NAME =
    •"ProLiant SL170z G6"
    •"ProLiant DL360 G6"
    –$SERIAL_ID = hardware serial number
    –$PRODUCT_LNAME =
    •"ProLiantSL170zG6"
    •"ProLiantDL360G6"
    –$PRODUCT_SNAME =
    •"SL170z G6"
    •"DL360 G6"
    –$TOOLKIT_MNTPNT = /mnt/main
    –$TOOLKIT_SERVER_IP = NFS server IP
    –$HWDISC_FILE = /TOOLKIT/hpdiscovery.xml
    Some Available Variables
    © 2012 Apollo Group

    View Slide

  38. 38
    APOLLO GROUP
    Set Variables Based on Hardware Type
    export "`./hwquery ${HWDISC_FILE} allboards.xml SERVERNAME=SystemName`"
    case "${SERVERNAME}" in
    "ProLiant DL160 G6"* )
    CONREPDAT=${TOOLKIT_MNTPNT}/data_files/dl160_conrep.dat
    LO100CFGDAT=${TOOLKIT_MNTPNT}/data_files/dl160_lo100cfg.dat
    ;;
    "ProLiant SL170z G6"* )
    CONREPDAT="${TOOLKIT_MNTPNT}/data_files/sl170z_conrep.dat -x \
    ${TOOLKIT_MNTPNT}/conrep_xml/conrep_SL170zg6_20100401.xml"
    LO100CFGDAT=${TOOLKIT_MNTPNT}/data_files/sl170z_lo100cfg.dat
    ;;
    #...
    Inside install_hadoop.sh
    © 2012 Apollo Group

    View Slide

  39. 39
    APOLLO GROUP
    #...
    "ProLiant DL360 G5"* )
    CONREPDAT=${TOOLKIT_MNTPNT}/data_files/dl360g5_conrep.dat
    if ./ifhw ${HWDISC_FILE} allboards.xml "PCI:Smart Array E200i Controller" \
    2> /dev/null ; then
    HPACUDAT=${TOOLKIT_MNTPNT}/data_files/dl360g5_saE200i_cpqacuxe.dat
    fi
    if ./ifhw ${HWDISC_FILE} allboards.xml "PCI:Smart Array P400i Controller" \
    2> /dev/null ; then
    HPACUDAT=${TOOLKIT_MNTPNT}/data_files/dl360g5_saP400i_cpqacuxe.dat
    fi
    HPONCFGDAT=${TOOLKIT_MNTPNT}/data_files/dl360g5_hponcfg.dat
    ;;
    esac
    Inside install_hadoop.sh
    © 2012 Apollo Group

    View Slide

  40. 40
    APOLLO GROUP
    if [ -n "$CONREPDAT" ] ; then
    echo "** Applying BIOS Configuration"
    ./conrep -l -f ${CONREPDAT}
    fi
    if [ -n "$HPACUDAT" ] ; then
    echo "** Clearing RAID Configuration"
    ./hpacucli/hpacuscripting -i ${HPACUDAT}_clear; sleep 5
    echo "** Applying RAID Configuration"
    ./hpacucli/hpacuscripting -i ${HPACUDAT}
    fi
    if [ -n "$LO100CFGDAT" ] ; then
    echo "** Applying BMC Configuration"; ./lo100cfg -i ${LO100CFGDAT}
    elif [ -n "$HPONCFGDAT" ] ; then
    echo "** Applying iLO Configuration"; ./hponcfg -f ${HPONCFGDAT}
    fi
    Inside install_hadoop.sh
    © 2012 Apollo Group

    View Slide

  41. 41
    APOLLO GROUP
    echo "** Registering with cobbler"
    # Koan does not come with SSSTK, so "install" it.
    ${TOOLKIT_MNTPNT}/koan/install
    # cprofile is set via kernel boot parameter and points at part of a
    # Cobbler profile name.
    # SERIAL_ID and PRODUCT_LNAME are set via SSSTK.
    # PRODUCT_LNAME = ProLiantSL170zG6, ProLiantDL360G6, etc.
    # SERIAL_ID = hardware serial number
    echo "** Registering to profile ${cprofile}-${PRODUCT_LNAME}"
    cobbler-register --server=${TOOLKIT_SERVER_IP} --fqdn=${SERIAL_ID} \
    --profile=${cprofile}-${PRODUCT_LNAME} --batch
    Register with Buildsystem
    © 2012 Apollo Group

    View Slide

  42. 42
    APOLLO GROUP
    $ cat dl360g5_saE200i_cpqacuxe.dat_clear
    ; Version: 8.50.6.0
    ; http://h30499.www3.hp.com/t5/ProLiant-Servers-ML-DL-
    SL/CPQACUXE-reset-utility-within-WinPE/td-p/3939475
    Action= Reconfigure
    Method= Custom
    Controller= SLOT 0
    ClearConfigurationWithDataLoss= Yes
    Clear the HP RAID Controller
    © 2012 Apollo Group

    View Slide

  43. 43
    APOLLO GROUP
    $ cat dl360g5_saE200i_cpqacuxe.dat
    ; Version: 8.50.6.0
    Action= Configure
    Method= Custom
    ; Controller Specifications
    ; Controller HP Smart Array E200i
    ; Firmware Version 1.82
    Controller= SLOT 0
    ReadCache= 100
    WriteCache= 0
    RebuildPriority= Medium
    ExpandPriority= Medium
    SurfaceScanDelay= 15
    DriveWriteCache= Disabled
    ; Array Specifications
    Array= A
    ; Array Drive Type is SAS
    ; 1I:1:1 (72 GB), 1I:1:2 (72 GB)
    Drive= 1I:1:1, 1I:1:2
    OnlineSpare= No
    ; Logical Drive Specifications
    LogicalDrive= 1
    RAID= 1
    Size= 69973
    Sectors= 32
    StripeSize= 128
    ArrayAccelerator= Enabled
    Configure the HP RAID Controller
    © 2012 Apollo Group

    View Slide

  44. 44
    APOLLO GROUP
    You may have to test for any failures when
    applying configurations.
    When reusing HP hardware, adding a user to the
    BMC that already exists will cause a failure to
    apply all of the configuration.
    Newer firmware can have different config options
    that will cause an older capture to fail.
    Error Handling
    © 2012 Apollo Group

    View Slide

  45. 45
    APOLLO GROUP
    Firmware
    Update
    Hardware
    Discovery
    Firmware
    Configuration
    Buildsystem
    Registration
    Summary
    © 2012 Apollo Group

    View Slide

  46. 46
    APOLLO GROUP 46
    APOLLO GROUP
     HP SmartStart Scripting Toolkit
    http://h18013.www1.hp.com/products/servers/management/toolkit/index.html
     Dell OpenManage Deployment Toolkit
    http://en.community.dell.com/techcenter/systems-management/w/wiki/1772.dell-
    openmanage-deployment-toolkit.aspx
     IBM ServerGuide Scripting Toolkit
    http://www-03.ibm.com/systems/be/management/sgstk/
     Fujitsu ServerView Scripting Toolkit
    http://www.fujitsu.com/fts/products/computing/servers/primergy/management/de
    ploy/
     Cobbler - Linux Installation Server
    http://cobbler.github.com/
     LSI MegaRAID MegaCLI
    http://www.lsi.com/downloads/Public/MegaRAID%20Common%20Files/8.02.24_
    MegaCLI.zip
    Links
    © 2012 Apollo Group

    View Slide

  47. 47
    APOLLO GROUP 47
    APOLLO GROUP
     https://intelligentsysadmin.wordpress.com/
     https://github.com/razorsedge
     http://forge.puppetlabs.com/razorsedge
    Contact
    © 2012 Apollo Group

    View Slide

  48. 48
    APOLLO GROUP
    APOLLO GROUP
    Questions?
    © 2012 Apollo Group

    View Slide