Automated Configuration of Firmware

Transcript

1. APOLLO GROUP
   Automated Configuration of Firmware
   Michael Arnold
   Principal Systems Engineer
   27 September 2012
   

   View Slide

2. 2
   APOLLO GROUP
   Who
   Tools
   Example Workflow
   Example Code
   Links
   Questions
   Agenda
   © 2012 Apollo Group
   

   View Slide

3. 3
   APOLLO GROUP
   APOLLO GROUP
   Who
   © 2012 Apollo Group
   

   View Slide

4. 4
   APOLLO GROUP
   Who is Apollo?
   © 2012 Apollo Group
   Apollo Group is a leading provider of higher
   education programs for working adults.
   

   View Slide

5. 5
   APOLLO GROUP
   Systems Administrator
   Automation geek
   13 years in IT
   Hadoop Cluster Admin
   I deal with:
   –Server hardware specification/configuration
   –Server firmware
   –Server operating system
   –Hadoop Cluster performance and availability
   Who is Michael Arnold?
   © 2012 Apollo Group
   

   View Slide

6. 6
   APOLLO GROUP
   It has nothing to do with Puppet.
   –Unless someone writes a module to program
   firmware from the running OS.
   • Which could work for RAID controllers.
   How does firmware tie in with Puppet?
   © 2012 Apollo Group
   

   View Slide

7. 7
   APOLLO GROUP
   APOLLO GROUP
   Tools
   © 2012 Apollo Group
   

   View Slide

8. 8
   APOLLO GROUP
   HP SmartStart Scripting Toolkit
   Dell OpenManage Deployment Toolkit
   IBM ServerGuide Scripting Toolkit
   Fujitsu ServerView Scripting Toolkit
   Hardware Toolkits
   © 2012 Apollo Group
   

   View Slide

9. 9
   APOLLO GROUP
   Configuration of
   –BIOS
   –BMC (DRAC/iLO/RSA/etc.)
   –Hardware RAID
   –FC HBA
   What do the tools provide?
   © 2012 Apollo Group
   

   View Slide

10. 10
    APOLLO GROUP
    Manually configure reference hardware
    Capture the firmware configuration
    Clone the configuration to multiple machines
    How does the toolkit work?
    © 2012 Apollo Group
    

    View Slide

11. 11
    APOLLO GROUP
    Toolkit Components
    © 2012 Apollo Group
    Kernel &
    Initrd
    Firmware
    Tools
    Toolkit
    

    View Slide

12. 12
    APOLLO GROUP
    Toolkit Requirements
    © 2012 Apollo Group
    Toolkit
    DHCP
    TFTP
    NFS
    

    View Slide

13. 13
    APOLLO GROUP
    APOLLO GROUP
    Example Workflow
    © 2012 Apollo Group
    

    View Slide

14. 14
    APOLLO GROUP
    Hardware is racked and cabled
    Switch ports are up
    Server is manually powered on
    Firmware defaults are to boot from network if no
    other boot device is found.
    –No other boot devices are found
    Assumptions
    © 2012 Apollo Group
    

    View Slide

15. 15
    APOLLO GROUP
    All server hardware is from the same vendor (HP)
    Constraints
    © 2012 Apollo Group
    

    View Slide

16. 16
    APOLLO GROUP
    Firmware
    Update
    Hardware
    Discovery
    Firmware
    Configuration
    Buildsystem
    Registration
    The Process
    © 2012 Apollo Group
    

    View Slide

17. 17
    APOLLO GROUP
    Screencast
    © 2012 Apollo Group
    

    View Slide

18. 18
    APOLLO GROUP
    Unknown system netboots
    Workflow
    © 2012 Apollo Group
    

    View Slide

19. 19
    APOLLO GROUP
    Unknown system netboots
    Boot into HP SmartStart Scripting Toolkit
    Workflow
    © 2012 Apollo Group
    

    View Slide

20. 20
    APOLLO GROUP
    Unknown system netboots
    Boot into HP SmartStart Scripting Toolkit
    Gather hardware information (CPU/RAM/disk/NIC)
    via hpdiscovery
    Workflow
    © 2012 Apollo Group
    

    View Slide

21. 21
    APOLLO GROUP
    Unknown system netboots
    Boot into HP SmartStart Scripting Toolkit
    Gather hardware information (CPU/RAM/disk/NIC)
    via hpdiscovery
    If this is HP hardware, then
    –Update firmware
    –Configure BIOS
    –Configure iLO
    –Configure RAID
    Workflow
    © 2012 Apollo Group
    

    View Slide

22. 22
    APOLLO GROUP
    Unknown system netboots
    Boot into HP SmartStart Scripting Toolkit
    Gather hardware information (CPU/RAM/disk/NIC)
    via hpdiscovery
    If this is HP hardware, then
    –Update firmware
    –Configure BIOS
    –Configure iLO
    –Configure RAID
    cobbler-register to the hardware profile
    Workflow
    © 2012 Apollo Group
    

    View Slide

23. 23
    APOLLO GROUP
    Unknown system netboots
    Boot into HP SmartStart Scripting Toolkit
    Gather hardware information (CPU/RAM/disk/NIC) via
    hpdiscovery
    If this is HP hardware, then
    –Update firmware
    –Configure BIOS
    –Configure iLO
    –Configure RAID
    cobbler-register to the hardware profile
    System reboots
    Workflow
    © 2012 Apollo Group
    

    View Slide

24. 24
    APOLLO GROUP
    Modify Cobbler system profile, giving it an identity
    (hostname, IP, OS metadata)
    Workflow
    © 2012 Apollo Group
    

    View Slide

25. 25
    APOLLO GROUP
    Modify Cobbler system profile, giving it an identity
    (hostname, IP, OS metadata)
    Known system netboots
    Workflow
    © 2012 Apollo Group
    

    View Slide

26. 26
    APOLLO GROUP
    Modify Cobbler system profile, giving it an identity
    (hostname, IP, OS metadata)
    Known system netboots
    Boot into OS installer
    Workflow
    © 2012 Apollo Group
    

    View Slide

27. 27
    APOLLO GROUP
    Modify Cobbler system profile, giving it an identity
    (hostname, IP, OS metadata)
    Known system netboots
    Boot into OS installer
    Install OS
    Workflow
    © 2012 Apollo Group
    

    View Slide

28. 28
    APOLLO GROUP
    Modify Cobbler system profile, giving it an identity
    (hostname, IP, OS metadata)
    Known system netboots
    Boot into OS installer
    Install OS
    Install Puppet
    Workflow
    © 2012 Apollo Group
    

    View Slide

29. 29
    APOLLO GROUP
    Modify Cobbler system profile, giving it an identity
    (hostname, IP, OS metadata)
    Known system netboots
    Boot into OS installer
    Install OS
    Install Puppet
    …
    Workflow
    © 2012 Apollo Group
    

    View Slide

30. 30
    APOLLO GROUP
    Modify Cobbler system profile, giving it an identity
    (hostname, IP, OS metadata)
    Known system netboots
    Boot into OS installer
    Install OS
    Install Puppet
    …
    Profit!
    Workflow
    © 2012 Apollo Group
    

    View Slide

31. 31
    APOLLO GROUP
    APOLLO GROUP
    Example Code
    © 2012 Apollo Group
    

    View Slide

32. 32
    APOLLO GROUP
    LABEL dtk-install_hadoop
    kernel /images/dtk-3.5.1/SA.1
    append initrd=/images/dtk-3.5.1/SA.2
    ramdisk_size=156482 Stage3_type=cdrom quiet DEBUG=0
    share_location=192.168.1.1:/srv/nfs/ro/dtk-3.5.1
    share_type=nfs selinux=0 share_opts=ro,nolock
    share_script=install_hadoop.sh cprofile=Hadoop
    ipappend 2
    Dell TFTP Config
    © 2012 Apollo Group
    

    View Slide

33. 33
    APOLLO GROUP
    LABEL sstk-install_hadoop
    kernel /images/sstk-8.50/vmlinuz
    append initrd=/images/sstk-8.50/initrd.img
    root=/dev/ram0 rw ramdisk_size=184248 ide=nodma
    ide=noraid pnpbios=off network=1
    sstk_mount=192.168.1.1:/srv/nfs/ro/sstk-8.50
    sstk_mount_type=nfs sstk_mount_options=ro,nolock
    sstk_script=install_hadoop.sh cprofile=Hadoop
    ipappend 2
    HP TFTP Config
    © 2012 Apollo Group
    

    View Slide

34. 34
    APOLLO GROUP
    cd /opt/dell/toolkit/template/scripts
    /opt/dell/toolkit/template/scripts/syscap.sh /tmp/syscap
    /opt/dell/toolkit/template/scripts/raidcap.sh /tmp/raidcap
    /opt/dell/toolkit/template/scripts/raccap.sh /tmp/raccap
    # MegaCLI does not come with the DTK so "install" it.
    cp -a /opt/dell/toolkit/systems/MegaCLI/opt/MegaRAID \
    /opt/MegaRAID
    /opt/MegaRAID/MegaCli/MegaCli -CfgSave -a0 -f /tmp/mega.0
    Dell Config Capture
    © 2012 Apollo Group
    

    View Slide

35. 35
    APOLLO GROUP
    cd /TOOLKIT
    ./conrep -s -f /tmp/conrep.dat # BIOS
    ./hpacuscripting -c /tmp/cpqacuxe.dat # RAID
    ./hplpcfg /s /tmp/hplpcfg.dat # Emulex HBA
    ./hpqlarep /S /tmp/hpqlarep.dat # Qlogic HBA
    if ./ifhw /tmp/hw_discovery.dat allboards.xml \
    "PCI:Integrated Lights-Out" 2> /dev/null ; then
    ./hponcfg -w /tmp/hponcfg.dat # iLO 2/3
    else
    ./lo100cfg -o /tmp/lo100cfg.dat # LO100i
    fi
    HP Config Capture
    © 2012 Apollo Group
    

    View Slide

36. 36
    APOLLO GROUP
    *** Detecting system type ***
    System Information:
    Name : ProLiant DL360 G5
    Serial : USE123D456
    Arch : x86_64
    LNAME : ProLiantDL360G5
    SNAME : DL360 G5
    Asset Tag : 01234567
    Platform Detection
    © 2012 Apollo Group
    

    View Slide

37. 37
    APOLLO GROUP
     HP SSSTK has a function library that sets many variables.
    –$PRODUCT_NAME =
    •"ProLiant SL170z G6"
    •"ProLiant DL360 G6"
    –$SERIAL_ID = hardware serial number
    –$PRODUCT_LNAME =
    •"ProLiantSL170zG6"
    •"ProLiantDL360G6"
    –$PRODUCT_SNAME =
    •"SL170z G6"
    •"DL360 G6"
    –$TOOLKIT_MNTPNT = /mnt/main
    –$TOOLKIT_SERVER_IP = NFS server IP
    –$HWDISC_FILE = /TOOLKIT/hpdiscovery.xml
    Some Available Variables
    © 2012 Apollo Group
    

    View Slide

38. 38
    APOLLO GROUP
    Set Variables Based on Hardware Type
    export "`./hwquery ${HWDISC_FILE} allboards.xml SERVERNAME=SystemName`"
    case "${SERVERNAME}" in
    "ProLiant DL160 G6"* )
    CONREPDAT=${TOOLKIT_MNTPNT}/data_files/dl160_conrep.dat
    LO100CFGDAT=${TOOLKIT_MNTPNT}/data_files/dl160_lo100cfg.dat
    ;;
    "ProLiant SL170z G6"* )
    CONREPDAT="${TOOLKIT_MNTPNT}/data_files/sl170z_conrep.dat -x \
    ${TOOLKIT_MNTPNT}/conrep_xml/conrep_SL170zg6_20100401.xml"
    LO100CFGDAT=${TOOLKIT_MNTPNT}/data_files/sl170z_lo100cfg.dat
    ;;
    #...
    Inside install_hadoop.sh
    © 2012 Apollo Group
    

    View Slide

39. 39
    APOLLO GROUP
    #...
    "ProLiant DL360 G5"* )
    CONREPDAT=${TOOLKIT_MNTPNT}/data_files/dl360g5_conrep.dat
    if ./ifhw ${HWDISC_FILE} allboards.xml "PCI:Smart Array E200i Controller" \
    2> /dev/null ; then
    HPACUDAT=${TOOLKIT_MNTPNT}/data_files/dl360g5_saE200i_cpqacuxe.dat
    fi
    if ./ifhw ${HWDISC_FILE} allboards.xml "PCI:Smart Array P400i Controller" \
    2> /dev/null ; then
    HPACUDAT=${TOOLKIT_MNTPNT}/data_files/dl360g5_saP400i_cpqacuxe.dat
    fi
    HPONCFGDAT=${TOOLKIT_MNTPNT}/data_files/dl360g5_hponcfg.dat
    ;;
    esac
    Inside install_hadoop.sh
    © 2012 Apollo Group
    

    View Slide

40. 40
    APOLLO GROUP
    if [ -n "$CONREPDAT" ] ; then
    echo "** Applying BIOS Configuration"
    ./conrep -l -f ${CONREPDAT}
    fi
    if [ -n "$HPACUDAT" ] ; then
    echo "** Clearing RAID Configuration"
    ./hpacucli/hpacuscripting -i ${HPACUDAT}_clear; sleep 5
    echo "** Applying RAID Configuration"
    ./hpacucli/hpacuscripting -i ${HPACUDAT}
    fi
    if [ -n "$LO100CFGDAT" ] ; then
    echo "** Applying BMC Configuration"; ./lo100cfg -i ${LO100CFGDAT}
    elif [ -n "$HPONCFGDAT" ] ; then
    echo "** Applying iLO Configuration"; ./hponcfg -f ${HPONCFGDAT}
    fi
    Inside install_hadoop.sh
    © 2012 Apollo Group
    

    View Slide

41. 41
    APOLLO GROUP
    echo "** Registering with cobbler"
    # Koan does not come with SSSTK, so "install" it.
    ${TOOLKIT_MNTPNT}/koan/install
    # cprofile is set via kernel boot parameter and points at part of a
    # Cobbler profile name.
    # SERIAL_ID and PRODUCT_LNAME are set via SSSTK.
    # PRODUCT_LNAME = ProLiantSL170zG6, ProLiantDL360G6, etc.
    # SERIAL_ID = hardware serial number
    echo "** Registering to profile ${cprofile}-${PRODUCT_LNAME}"
    cobbler-register --server=${TOOLKIT_SERVER_IP} --fqdn=${SERIAL_ID} \
    --profile=${cprofile}-${PRODUCT_LNAME} --batch
    Register with Buildsystem
    © 2012 Apollo Group
    

    View Slide

42. 42
    APOLLO GROUP
    $ cat dl360g5_saE200i_cpqacuxe.dat_clear
    ; Version: 8.50.6.0
    ; http://h30499.www3.hp.com/t5/ProLiant-Servers-ML-DL-
    SL/CPQACUXE-reset-utility-within-WinPE/td-p/3939475
    Action= Reconfigure
    Method= Custom
    Controller= SLOT 0
    ClearConfigurationWithDataLoss= Yes
    Clear the HP RAID Controller
    © 2012 Apollo Group
    

    View Slide

43. 43
    APOLLO GROUP
    $ cat dl360g5_saE200i_cpqacuxe.dat
    ; Version: 8.50.6.0
    Action= Configure
    Method= Custom
    ; Controller Specifications
    ; Controller HP Smart Array E200i
    ; Firmware Version 1.82
    Controller= SLOT 0
    ReadCache= 100
    WriteCache= 0
    RebuildPriority= Medium
    ExpandPriority= Medium
    SurfaceScanDelay= 15
    DriveWriteCache= Disabled
    ; Array Specifications
    Array= A
    ; Array Drive Type is SAS
    ; 1I:1:1 (72 GB), 1I:1:2 (72 GB)
    Drive= 1I:1:1, 1I:1:2
    OnlineSpare= No
    ; Logical Drive Specifications
    LogicalDrive= 1
    RAID= 1
    Size= 69973
    Sectors= 32
    StripeSize= 128
    ArrayAccelerator= Enabled
    Configure the HP RAID Controller
    © 2012 Apollo Group
    

    View Slide

44. 44
    APOLLO GROUP
    You may have to test for any failures when
    applying configurations.
    When reusing HP hardware, adding a user to the
    BMC that already exists will cause a failure to
    apply all of the configuration.
    Newer firmware can have different config options
    that will cause an older capture to fail.
    Error Handling
    © 2012 Apollo Group
    

    View Slide

45. 45
    APOLLO GROUP
    Firmware
    Update
    Hardware
    Discovery
    Firmware
    Configuration
    Buildsystem
    Registration
    Summary
    © 2012 Apollo Group
    

    View Slide

46. 46
    APOLLO GROUP 46
    APOLLO GROUP
     HP SmartStart Scripting Toolkit
    http://h18013.www1.hp.com/products/servers/management/toolkit/index.html
     Dell OpenManage Deployment Toolkit
    http://en.community.dell.com/techcenter/systems-management/w/wiki/1772.dell-
    openmanage-deployment-toolkit.aspx
     IBM ServerGuide Scripting Toolkit
    http://www-03.ibm.com/systems/be/management/sgstk/
     Fujitsu ServerView Scripting Toolkit
    http://www.fujitsu.com/fts/products/computing/servers/primergy/management/de
    ploy/
     Cobbler - Linux Installation Server
    http://cobbler.github.com/
     LSI MegaRAID MegaCLI
    http://www.lsi.com/downloads/Public/MegaRAID%20Common%20Files/8.02.24_
    MegaCLI.zip
    Links
    © 2012 Apollo Group
    

    View Slide

47. 47
    APOLLO GROUP 47
    APOLLO GROUP
     https://intelligentsysadmin.wordpress.com/
     https://github.com/razorsedge
     http://forge.puppetlabs.com/razorsedge
    Contact
    © 2012 Apollo Group
    

    View Slide

48. 48
    APOLLO GROUP
    APOLLO GROUP
    Questions?
    © 2012 Apollo Group
    

    View Slide