What I am Working On: Zero Touch Firmware Updates with HPE sutesxi on HPE Gen 9 Hardware – Part 1 . . . Probably

Showing the Process: A Different Tack

First, my disclaimer is that there is some assumed knowledge of HPE OneView and Hardware for this one, but even if you don’t run HPE hardware, I should hope the concept and approach I talk about here is valuable, since auto-staging firmware updates is a thing. I also have some background in automating Dell firmware updates, so hit me up for advice, if you wish.

I am taking a different tack on this one, and buckle up because I am going to describe the only acceptable method my desired approach for updating firmware. Usually, I only post topics about projects or tasks that are “done”. But I am getting a bit experimental here and hoping that this one will be more about the process. There are two major phases to this one:

1. Get the iSUT in place and working in conjunction with vCenter and OneView (more on both of these later). This is complete.
2. Create a “code pipeline,” or “workflow” that will schedule the firmware updates. The requirements are as follows:
   1. The pipeline must be easy to maintain; no need to change lots of lines of code or code in multiple places.
   2. The pipeline must have phases and must also be granular in the infrastructure it updates: test infra first, then non-prod, then low-to-high risk.
   3. The firmware must always be staged. None of that “rebooting while not in maintenance mode with VMs on it” hullaballoo. Because that would be bad.
   4. It must also happen in the background and automatically.

Another disclaimer is that what I post here is specifically for ESXi HPE Gen 9 hardware. Gen 10 has its own approach and built-in tools. HPE Gen 9 has a special challenge because, as I understand it, the iLO on Gen 9 was never built to “manage firmware” on ESXi hosts: you need an intermediary.

As If I Have to Ask: Why Zero Touch Firmware Updates?

As I have stated in previous posts I am too lazy to link here, applying patches is, “not moving your team or organization forward.” Yes it needs to be done, and yes it’s an engineer’s job to make sure they happen one way or another. But, again, time is money. The goals for automation are to make sure the trivial is trivial. “It is this blogger’s opinion” . . . I have always wanted to say that . . . that firmware should be trivial for compute nodes. It should just happen in the background. What I have here should at least get your brain in motion on how to accomplish this.

So, How is Firmware Staging Done in the World of HPE?

Well, it does depend on the platform, but for ESXi on HPE Gen 9 hardware, you install iSUT for ESXi and HPE Gen8/Gen 9 on a Linux/Windows Server and configure it to authenticate to your vCenter(s) and OneView Appliance(s). Once new HPE firmware SPPs are released, you create a new firmware baseline from it and simply apply it to the Server Profile through OneView. iSUT does the rest at polled intervals, as you can see from this diagram I stole am using legitimately from HPE’s website:

That’s Great Bryan, but What About the Coded Workflow?

Yeah, that’s still in progress. I envision making a call to the OneView API to apply the new firmware baseline to server profiles. You know . . . Powershell . . . or maybe Ansible to apply the new firmware baseline to the profile. There are some resources out there for this already; my current thought is to use this script and schedule it through Jenkins across the organization at regular intervals.

Then all we have to do, once the new Firmware bundle is available, tested, and approved, is make a new baseline available in OneView.

The Powershell and Jenkins or Ansible playbook(s) would apply the new baseline to all profiles on a schedule in a phased approach.

At the very least, rather than doing HPE SPP installations manually at . . . an hour and 10 minutes per host . . . the idea is that all you have to do is create a new firmware baseline and apply it to each Server Profile . . . and then . . . wait. If you have the resources available, you could “automate by mechanical Turk,” which is a phrase I use when other human beings do things. It’s still automation if someone else does it right?

RIGHT?

That’s Also Great Bryan, But What About the Reboots?

A very common approach is to piggy-back software patching onto the process described above. You are probably going to need upgraded drivers anyway, so use whatever process you have to do software patching and when the system reboots, firmware gets upgraded as well as the software. I have found vSphere Update Manager to be adequate for this. Assuming you have Fully Automated DRS and VMs that are vMotion capable, this is a single touch process. Maybe zero touch if you automate it. At the very least, you are looking at only one reboot instead of two.

Enough with the Process Bryan, Let’s Do It!

You can download the needed software here. You should also take a look at the documentation here.

You’ll need a Linux box. You can use Windows too, but . . . that’s . . . like . . . not what I used. I am using a CentOS 7.8.2003 machine; YMMV with other distros. To install:

[user@server ~] sudo yum install unzip # Not installed on CentOS by default.
[user@server ~] sudo rpm -ivh sutesxi-2.5.0-7.linux.x86_64.rpm # Or whatever version you get.
[user@server ~] sudo /opt/sutesxi/scripts/sutesxi_diagnostic.sh # This will prompt for vCenter Creds and OneView creds - and will regurgitate a bunch of things that will "look bad" - it's normal.
[user@server ~] sudo sutesxi -start # This is registered in chkconfig to start at boot.
[user@server ~] sudo sutesxi -set mode=AutoDeploy # This is important!

Most of the above is straightforward, but the most important part is setting the mode to AutoDeploy. The default is AutoDeployReboot. What do you suppose that does?

Now that it’s installed, the next step is to import a new Firmware baseline and apply it to each OneView Server Profile. This can be automated, as discussed previously:

What it will do is poll the registered OneView Appliance(s) every 5 minutes. As long as the host has a profile in OneView and is part of the iSUT registered vCenter, Firmware will be staged automatically by the iSUT Appliance we have setup. Any changes to the Firmware Baseline in OneView leads to an automated staging of the firmware defined in that baseline. You just need to monitor it in OneView. This is what a successful update looks like (read from the bottom up):

You can also view the progress from the iSUT appliance using the sutesxi -status command.

So this is probably Part 1 of at least one more. Stay tuned!

Hit me up on twitter @RussianLitGuy or email me at bryansullins@thinkingoutcloud.org. I would love to hear from you!