Automated Configuration of Firmware

Transcript

1. APOLLO GROUP Automated Configuration of Firmware Michael Arnold Principal Systems

   Engineer 27 September 2012

2. 2 APOLLO GROUP Who Tools Example Workflow Example Code Links

   Questions Agenda © 2012 Apollo Group

3. 3 APOLLO GROUP APOLLO GROUP Who © 2012 Apollo Group

4. 4 APOLLO GROUP Who is Apollo? © 2012 Apollo Group

   Apollo Group is a leading provider of higher education programs for working adults.

5. 5 APOLLO GROUP Systems Administrator Automation geek 13 years in

   IT Hadoop Cluster Admin I deal with: –Server hardware specification/configuration –Server firmware –Server operating system –Hadoop Cluster performance and availability Who is Michael Arnold? © 2012 Apollo Group

6. 6 APOLLO GROUP It has nothing to do with Puppet.

   –Unless someone writes a module to program firmware from the running OS. • Which could work for RAID controllers. How does firmware tie in with Puppet? © 2012 Apollo Group

7. 7 APOLLO GROUP APOLLO GROUP Tools © 2012 Apollo Group

8. 8 APOLLO GROUP HP SmartStart Scripting Toolkit Dell OpenManage Deployment

   Toolkit IBM ServerGuide Scripting Toolkit Fujitsu ServerView Scripting Toolkit Hardware Toolkits © 2012 Apollo Group

9. 9 APOLLO GROUP Configuration of –BIOS –BMC (DRAC/iLO/RSA/etc.) –Hardware RAID

   –FC HBA What do the tools provide? © 2012 Apollo Group

10. 10 APOLLO GROUP Manually configure reference hardware Capture the firmware

    configuration Clone the configuration to multiple machines How does the toolkit work? © 2012 Apollo Group

11. 11 APOLLO GROUP Toolkit Components © 2012 Apollo Group Kernel

    & Initrd Firmware Tools Toolkit

12. 12 APOLLO GROUP Toolkit Requirements © 2012 Apollo Group Toolkit

    DHCP TFTP NFS

13. 13 APOLLO GROUP APOLLO GROUP Example Workflow © 2012 Apollo

    Group

14. 14 APOLLO GROUP Hardware is racked and cabled Switch ports

    are up Server is manually powered on Firmware defaults are to boot from network if no other boot device is found. –No other boot devices are found Assumptions © 2012 Apollo Group

15. 15 APOLLO GROUP All server hardware is from the same

    vendor (HP) Constraints © 2012 Apollo Group

16. 16 APOLLO GROUP Firmware Update Hardware Discovery Firmware Configuration Buildsystem

    Registration The Process © 2012 Apollo Group

17. 17 APOLLO GROUP Screencast © 2012 Apollo Group

18. 18 APOLLO GROUP Unknown system netboots Workflow © 2012 Apollo

    Group

19. 19 APOLLO GROUP Unknown system netboots Boot into HP SmartStart

    Scripting Toolkit Workflow © 2012 Apollo Group

20. 20 APOLLO GROUP Unknown system netboots Boot into HP SmartStart

    Scripting Toolkit Gather hardware information (CPU/RAM/disk/NIC) via hpdiscovery Workflow © 2012 Apollo Group

21. 21 APOLLO GROUP Unknown system netboots Boot into HP SmartStart

    Scripting Toolkit Gather hardware information (CPU/RAM/disk/NIC) via hpdiscovery If this is HP hardware, then –Update firmware –Configure BIOS –Configure iLO –Configure RAID Workflow © 2012 Apollo Group

22. 22 APOLLO GROUP Unknown system netboots Boot into HP SmartStart

    Scripting Toolkit Gather hardware information (CPU/RAM/disk/NIC) via hpdiscovery If this is HP hardware, then –Update firmware –Configure BIOS –Configure iLO –Configure RAID cobbler-register to the hardware profile Workflow © 2012 Apollo Group

23. 23 APOLLO GROUP Unknown system netboots Boot into HP SmartStart

    Scripting Toolkit Gather hardware information (CPU/RAM/disk/NIC) via hpdiscovery If this is HP hardware, then –Update firmware –Configure BIOS –Configure iLO –Configure RAID cobbler-register to the hardware profile System reboots Workflow © 2012 Apollo Group

24. 24 APOLLO GROUP Modify Cobbler system profile, giving it an

    identity (hostname, IP, OS metadata) Workflow © 2012 Apollo Group

25. 25 APOLLO GROUP Modify Cobbler system profile, giving it an

    identity (hostname, IP, OS metadata) Known system netboots Workflow © 2012 Apollo Group

26. 26 APOLLO GROUP Modify Cobbler system profile, giving it an

    identity (hostname, IP, OS metadata) Known system netboots Boot into OS installer Workflow © 2012 Apollo Group

27. 27 APOLLO GROUP Modify Cobbler system profile, giving it an

    identity (hostname, IP, OS metadata) Known system netboots Boot into OS installer Install OS Workflow © 2012 Apollo Group

28. 28 APOLLO GROUP Modify Cobbler system profile, giving it an

    identity (hostname, IP, OS metadata) Known system netboots Boot into OS installer Install OS Install Puppet Workflow © 2012 Apollo Group

29. 29 APOLLO GROUP Modify Cobbler system profile, giving it an

    identity (hostname, IP, OS metadata) Known system netboots Boot into OS installer Install OS Install Puppet … Workflow © 2012 Apollo Group

30. 30 APOLLO GROUP Modify Cobbler system profile, giving it an

    identity (hostname, IP, OS metadata) Known system netboots Boot into OS installer Install OS Install Puppet … Profit! Workflow © 2012 Apollo Group

31. 31 APOLLO GROUP APOLLO GROUP Example Code © 2012 Apollo

    Group

32. 32 APOLLO GROUP LABEL dtk-install_hadoop kernel /images/dtk-3.5.1/SA.1 append initrd=/images/dtk-3.5.1/SA.2 ramdisk_size=156482

    Stage3_type=cdrom quiet DEBUG=0 share_location=192.168.1.1:/srv/nfs/ro/dtk-3.5.1 share_type=nfs selinux=0 share_opts=ro,nolock share_script=install_hadoop.sh cprofile=Hadoop ipappend 2 Dell TFTP Config © 2012 Apollo Group

33. 33 APOLLO GROUP LABEL sstk-install_hadoop kernel /images/sstk-8.50/vmlinuz append initrd=/images/sstk-8.50/initrd.img root=/dev/ram0

    rw ramdisk_size=184248 ide=nodma ide=noraid pnpbios=off network=1 sstk_mount=192.168.1.1:/srv/nfs/ro/sstk-8.50 sstk_mount_type=nfs sstk_mount_options=ro,nolock sstk_script=install_hadoop.sh cprofile=Hadoop ipappend 2 HP TFTP Config © 2012 Apollo Group

34. 34 APOLLO GROUP cd /opt/dell/toolkit/template/scripts /opt/dell/toolkit/template/scripts/syscap.sh /tmp/syscap /opt/dell/toolkit/template/scripts/raidcap.sh /tmp/raidcap /opt/dell/toolkit/template/scripts/raccap.sh

    /tmp/raccap # MegaCLI does not come with the DTK so "install" it. cp -a /opt/dell/toolkit/systems/MegaCLI/opt/MegaRAID \ /opt/MegaRAID /opt/MegaRAID/MegaCli/MegaCli -CfgSave -a0 -f /tmp/mega.0 Dell Config Capture © 2012 Apollo Group

35. 35 APOLLO GROUP cd /TOOLKIT ./conrep -s -f /tmp/conrep.dat #

    BIOS ./hpacuscripting -c /tmp/cpqacuxe.dat # RAID ./hplpcfg /s /tmp/hplpcfg.dat # Emulex HBA ./hpqlarep /S /tmp/hpqlarep.dat # Qlogic HBA if ./ifhw /tmp/hw_discovery.dat allboards.xml \ "PCI:Integrated Lights-Out" 2> /dev/null ; then ./hponcfg -w /tmp/hponcfg.dat # iLO 2/3 else ./lo100cfg -o /tmp/lo100cfg.dat # LO100i fi HP Config Capture © 2012 Apollo Group

36. 36 APOLLO GROUP *** Detecting system type *** System Information:

    Name : ProLiant DL360 G5 Serial : USE123D456 Arch : x86_64 LNAME : ProLiantDL360G5 SNAME : DL360 G5 Asset Tag : 01234567 Platform Detection © 2012 Apollo Group

37. 37 APOLLO GROUP  HP SSSTK has a function library

    that sets many variables. –$PRODUCT_NAME = •"ProLiant SL170z G6" •"ProLiant DL360 G6" –$SERIAL_ID = hardware serial number –$PRODUCT_LNAME = •"ProLiantSL170zG6" •"ProLiantDL360G6" –$PRODUCT_SNAME = •"SL170z G6" •"DL360 G6" –$TOOLKIT_MNTPNT = /mnt/main –$TOOLKIT_SERVER_IP = NFS server IP –$HWDISC_FILE = /TOOLKIT/hpdiscovery.xml Some Available Variables © 2012 Apollo Group

38. 38 APOLLO GROUP Set Variables Based on Hardware Type export

    "`./hwquery ${HWDISC_FILE} allboards.xml SERVERNAME=SystemName`" case "${SERVERNAME}" in "ProLiant DL160 G6"* ) CONREPDAT=${TOOLKIT_MNTPNT}/data_files/dl160_conrep.dat LO100CFGDAT=${TOOLKIT_MNTPNT}/data_files/dl160_lo100cfg.dat ;; "ProLiant SL170z G6"* ) CONREPDAT="${TOOLKIT_MNTPNT}/data_files/sl170z_conrep.dat -x \ ${TOOLKIT_MNTPNT}/conrep_xml/conrep_SL170zg6_20100401.xml" LO100CFGDAT=${TOOLKIT_MNTPNT}/data_files/sl170z_lo100cfg.dat ;; #... Inside install_hadoop.sh © 2012 Apollo Group

39. 39 APOLLO GROUP #... "ProLiant DL360 G5"* ) CONREPDAT=${TOOLKIT_MNTPNT}/data_files/dl360g5_conrep.dat if

    ./ifhw ${HWDISC_FILE} allboards.xml "PCI:Smart Array E200i Controller" \ 2> /dev/null ; then HPACUDAT=${TOOLKIT_MNTPNT}/data_files/dl360g5_saE200i_cpqacuxe.dat fi if ./ifhw ${HWDISC_FILE} allboards.xml "PCI:Smart Array P400i Controller" \ 2> /dev/null ; then HPACUDAT=${TOOLKIT_MNTPNT}/data_files/dl360g5_saP400i_cpqacuxe.dat fi HPONCFGDAT=${TOOLKIT_MNTPNT}/data_files/dl360g5_hponcfg.dat ;; esac Inside install_hadoop.sh © 2012 Apollo Group

40. 40 APOLLO GROUP if [ -n "$CONREPDAT" ] ; then

    echo "** Applying BIOS Configuration" ./conrep -l -f ${CONREPDAT} fi if [ -n "$HPACUDAT" ] ; then echo "** Clearing RAID Configuration" ./hpacucli/hpacuscripting -i ${HPACUDAT}_clear; sleep 5 echo "** Applying RAID Configuration" ./hpacucli/hpacuscripting -i ${HPACUDAT} fi if [ -n "$LO100CFGDAT" ] ; then echo "** Applying BMC Configuration"; ./lo100cfg -i ${LO100CFGDAT} elif [ -n "$HPONCFGDAT" ] ; then echo "** Applying iLO Configuration"; ./hponcfg -f ${HPONCFGDAT} fi Inside install_hadoop.sh © 2012 Apollo Group

41. 41 APOLLO GROUP echo "** Registering with cobbler" # Koan

    does not come with SSSTK, so "install" it. ${TOOLKIT_MNTPNT}/koan/install # cprofile is set via kernel boot parameter and points at part of a # Cobbler profile name. # SERIAL_ID and PRODUCT_LNAME are set via SSSTK. # PRODUCT_LNAME = ProLiantSL170zG6, ProLiantDL360G6, etc. # SERIAL_ID = hardware serial number echo "** Registering to profile ${cprofile}-${PRODUCT_LNAME}" cobbler-register --server=${TOOLKIT_SERVER_IP} --fqdn=${SERIAL_ID} \ --profile=${cprofile}-${PRODUCT_LNAME} --batch Register with Buildsystem © 2012 Apollo Group

42. 42 APOLLO GROUP $ cat dl360g5_saE200i_cpqacuxe.dat_clear ; Version: 8.50.6.0 ;

    http://h30499.www3.hp.com/t5/ProLiant-Servers-ML-DL- SL/CPQACUXE-reset-utility-within-WinPE/td-p/3939475 Action= Reconfigure Method= Custom Controller= SLOT 0 ClearConfigurationWithDataLoss= Yes Clear the HP RAID Controller © 2012 Apollo Group

43. 43 APOLLO GROUP $ cat dl360g5_saE200i_cpqacuxe.dat ; Version: 8.50.6.0 Action=

    Configure Method= Custom ; Controller Specifications ; Controller HP Smart Array E200i ; Firmware Version 1.82 Controller= SLOT 0 ReadCache= 100 WriteCache= 0 RebuildPriority= Medium ExpandPriority= Medium SurfaceScanDelay= 15 DriveWriteCache= Disabled ; Array Specifications Array= A ; Array Drive Type is SAS ; 1I:1:1 (72 GB), 1I:1:2 (72 GB) Drive= 1I:1:1, 1I:1:2 OnlineSpare= No ; Logical Drive Specifications LogicalDrive= 1 RAID= 1 Size= 69973 Sectors= 32 StripeSize= 128 ArrayAccelerator= Enabled Configure the HP RAID Controller © 2012 Apollo Group

44. 44 APOLLO GROUP You may have to test for any

    failures when applying configurations. When reusing HP hardware, adding a user to the BMC that already exists will cause a failure to apply all of the configuration. Newer firmware can have different config options that will cause an older capture to fail. Error Handling © 2012 Apollo Group

45. 45 APOLLO GROUP Firmware Update Hardware Discovery Firmware Configuration Buildsystem

    Registration Summary © 2012 Apollo Group

46. 46 APOLLO GROUP 46 APOLLO GROUP  HP SmartStart Scripting

    Toolkit http://h18013.www1.hp.com/products/servers/management/toolkit/index.html  Dell OpenManage Deployment Toolkit http://en.community.dell.com/techcenter/systems-management/w/wiki/1772.dell- openmanage-deployment-toolkit.aspx  IBM ServerGuide Scripting Toolkit http://www-03.ibm.com/systems/be/management/sgstk/  Fujitsu ServerView Scripting Toolkit http://www.fujitsu.com/fts/products/computing/servers/primergy/management/de ploy/  Cobbler - Linux Installation Server http://cobbler.github.com/  LSI MegaRAID MegaCLI http://www.lsi.com/downloads/Public/MegaRAID%20Common%20Files/8.02.24_ MegaCLI.zip Links © 2012 Apollo Group

47. 47 APOLLO GROUP 47 APOLLO GROUP  https://intelligentsysadmin.wordpress.com/  https://github.com/razorsedge

     http://forge.puppetlabs.com/razorsedge Contact © 2012 Apollo Group

48. 48 APOLLO GROUP APOLLO GROUP Questions? © 2012 Apollo Group