A Method for Auto-Generating Ansible Inventory Files for ESXi Hosts with PowerCLI

TL;DR: https://github.com/bryansullins/myvmwarekit/blob/master/MY.VMWAREKIT/StandaloneScripts/HostInventoryFiles.ps1

First, a Rant

Did I ever tell you how often I run into vendors who are all about automation for VMs/Apps but have jack squat for automating ESXi Hosts?

I get why. The glory and the money is in the “customer-facing” parts of the infrastructure. I would imagine a lot of it has to do with ignorance about dealing with ESXi hosts at scale as well.

And to give VMware credit, we do have PowerCLI, which can do just about everything we need.

It is comical when I attend sessions or talks about automation because I am, “that guy” who, “sticks around at the end” to interrogate ask questions about automating vSphere Infrastructure at the hypervisor layer (ESXi). It’s usually a predictable “Who’s on First”-esque exchange:

Me: I was wondering if you have any out-of-the-box modules for automating the configuration of ESXi hosts?
Speaker: Uh. What do you mean? Don’t you just configure it once through vCenter and then it’s done?
Me: Kind of? First of all I want to do a bunch at a time, say a few hundred, but they can also drift. Do you have anything out of the box that can do that?
Speaker (somewhat perplexed): What kinds of things do you need to set?
Me: NTP for example . . .
Speaker: Oh yeah. We had an example earlier for setting NTP on a guest OS in a VM . . .
Me (mentally face-palming): No. I am talking about NTP on the ESXi host, not in a VM.
Speaker (still somewhat perplexed. Pauses, then the inevitable question): These ESXi hosts, do they have ssh?
Me (having to suppress an eyeroll): Yes, but every automation vendor does ssh. I don’t need you for that. What do you have programmatically that can configure ESXi, out of the box?
Speaker: Let me put you in touch with so-and-so blankety-blank . . .

And so on . . .

But, What About Ansible?

I do my best to continuously evaluate vendor offerings. To give the Ansible Community credit, it seems to me that passing the threshold of version 2.8 and since, there have been a plethora of ESXi host-level additions that make Ansible the most robust automation tool out-of-the-box for ESXi configuration, outside of PowerCLI itself.

Although, it’s not without its flaws, as you are about to see . . .

Scenario: Auto-populating NetBox with Devices

The challenge I ran into this week has to do with populating ESXi host data into NetBox, our chosen DCIM tool. The good news about this recent development is that we have an opportunity to do DCIM right, from the ground up. It means we need to automate the population of ESXi host data into NetBox for both existing ESXi hosts and when new ESXi hosts are added into vCenter(s), and do I really need to tell you it’s a bad idea to depend on humans to enter this data manually? In other words, this needs to be zero touch and the robots need to do this for us automatically and on a schedule.

And yes, 38 followers of my blog, I will post something about NetBox at some point, I promise.

So what’s the best way to do this? Well, it turns out NetBox has both a REST API and a set of Ansible modules, including this one. We can also use Powershell/PowerCLI to connect using this same API. So, options abound. Honestly I am still entertaining different options, which are as follows:

1. Powershell/PowerCLI <-> Jenkins (scheduler) -> NetBox
2. PowerCLI query of all vCenters/ESXi hosts and write to Ansible Inventory files <-> Jenkins (scheduler) <-> Ansible Playbook <-> AWX (scheduler) -> NetBox

I am leaning towards #2 because we have some buy-in for the use of Ansible/AWX across our teams. Right now, I am entertaining both methods, so we’ll see. Let’s focus on option 2 for now:

The Challenge: Automagically Maintaining Ansible Inventories of the ESXi Hosts

I have to make my usual, “this is a method, not the method” disclaimer on this one. I am not 100% up to date on how automated Ansible inventories are done in the year 2020. What I describe here may strike you as “old-school” based on things I am hearing, but it works, so there’s that.

As dependable as I am, it’s a bad idea for me (or anyone for that matter) to keep the Ansible ESXi host inventories up to date manually. YUCKY! We need to schedule a script that will query vCenter for all Clusters and Hosts, then format that output as a valid Ansible inventory file. That way, as we add ESXi Hosts, the workflow I created (as above) will simply add it into the Ansible inventory file automatically. Then Ansible will “find it”, extract the host facts, and write them all into NetBox as a new Device.

The problem is that there is no native out-of-the box way to automatically inventory your ESXi hosts for Ansible, of which I am aware . . . .

“Wait just a minute there, Bryan . . . what about Ansible dynamic inventories?”

That might actually work. I’ll either have to find one or code it through Python. I don’t know Python, but maybe this will work? I’ll have to try it; but this comes with its own nuances as well. Maybe that’s an addendum to this post . . . There is vmware_vm_inventory that ships with Ansible, but, you guessed it, that’s only for VMs.

“What about using Smart Inventory in AWX/Tower? Those can auth to vCenter and find the hosts, right?”

Nope. It will not. It will only inventory . . . you guessed it again . . . the VMs.

So . . . What Then?

Back in the day, as a young strapping automator, my buddy Kevin Seales wrote a PowerCLI script that will inventory defined vCenters at regular intervals and will create the Ansible inventory files automatically.

I took that and ran with it. What the script does:

1. Loops through each vCenter in the vCenterList.csv file and finds all Clusters and Hosts. Need to add all hosts from a vCenter, or, for that matter, stand up a new vCenter? Add it into the .csv file and . . . voilá . . . now those hosts will be added into a new Ansible Inventory file named after their vCenter FQDN, and then used by Ansible to perform actions on those hosts, like, you know, add them as devices into NetBox at the next scheduled interval . . . automaticamenté.
2. As a result, when you add new hosts into these vCenters, they will automatically be added into these inventory files.
3. The script breaks up the hosts into Ansible inventory groups based on their Cluster names.
4. Adds an [all:vars] section at the end of the inventory file for use with Ansible. Although, you could get more specific if you needed to for more specific variables that are limited by group scope.
5. Adds the vSphere tag for "site" found by the script into the [all:vars] section for use when Ansible Facts falls short. This is a list that will grow, I’m sure, so we can synthesize tagged objects with the facts we can pull from the hosts and HPE OneView.

I still have some work to do, such as getting the host state (I think I took that part out; sorry Kevin!), and I have not figured out the decomm process yet, but this is version 1.2. Baby steps.

One additional piece of new territory for me is to make this “Jenkins ready”. I’ll be sure to post that if and when I get that straightened out.

You should be able to pull things apart based on what I’ve described above:

<#
Author: Kevin Seales
Contributor: Bryan Sullins - bryansullins@thinkingoutcloud.org
Date: 6-26-2020
Version: 1.2
#>

# Setup vars and connection(s) to vCenter
Set-PowerCLIConfiguration -InvalidCertificateAction ignore -confirm:$false
Set-PowerCLIConfiguration -DefaultVIServerMode Multiple -confirm:$false
$vCenterList = Get-Content "vCenters/vCenterList.csv"
$Creds = Get-Credential

# Loop through all vCenters brought in from the CSV file and connect.
ForEach ($vCenter in $vCenterList) {

    Connect-VIServer -Server $vCenter -credential $Creds -WarningAction 0 -ErrorAction SilentlyContinue
    # Setup vars for use with Inventory - lines written to the inventory file and addtional vars for use in plays.
    $AllClusters = Get-Cluster | Sort-Object Name
    $VarsLine = "[all:vars]"
    $vCenterVar = "vcenter_hostname" + "=" + '"' + "$vCenter" + '"'
    $SiteTag = "site" + "=" + '"' + (Get-TagAssignment -Entity Datacenters | Select @{N='Site';E={$_.Tag.Name }}).Site + '"'
    # Sets Inventory Filename for all found Hosts
    $inventory_filename = "inventory/$vCenter"

    # Removes old inventory files
    Remove-Item $inventory_filename -ErrorAction SilentlyContinue
    # New line for separating Clusters:
    $NewLine = "`r"
    # Loop through hosts and add hosts to inventory files
    ForEach ($Cluster in $AllClusters) {
        $ClusterGroupName = "["+$Cluster.Name+"]" -replace '\s','' # -replace gets rid of spaces in Cluster names.
        Out-File -Append -FilePath $inventory_filename -InputObject $ClusterGroupName -Encoding ASCII
        $AllHostsinCluster = Get-Cluster -Name "$Cluster" | Get-VMHost | Sort-Object Name
        ForEach ($h in $AllHostsinCluster) {
            Out-File -Append -FilePath $inventory_filename -InputObject $h.Name -Encoding ASCII
        }
        Out-File -Append -FilePath $inventory_filename -InputObject $NewLine -Encoding ASCII
   }
   Out-File -Append -FilePath $inventory_filename -InputObject $VarsLine -Encoding ASCII
   Out-File -Append -FilePath $inventory_filename -InputObject $vCenterVar -Encoding ASCII
   Out-File -Append -FilePath $inventory_filename -InputObject $SiteTag -Encoding ASCII
   Disconnect-VIServer -Server $vCenter -confirm:$false
}

I made a sample inventory file that gets created for your reference in the same repo, anonymized, of course:

[VMwareCluster01]
esxihost01.yourdomain.name
esxihost02.yourdomain.name
esxihost03.yourdomain.name

[VMwareCluster02]
esxihost04.yourdomain.name
esxihost05.yourdomain.name
esxihost06.yourdomain.name

[all:vars]
vcenter_hostname="vcenter.yourdomain.name"
site="SITETAGHERE"

This should get your creativity flowing. Once I code out the Ansible playbook for writing data into NetBox I will share that too: trust me; I always need material . . . .

Hit me up on twitter @RussianLitGuy or email me at bryansullins@thinkingoutcloud.org. I would love to hear from you!