Building Kubernetes-enabled Tower/AWX Execution Environments Using AWX-EE and ansible-builder

Disclaimer: this post is a bit more on the intermediate/advanced side, but you can always email me if you need help at bryansullins@thinkingoutcloud.org.

The title for this post was originally Building Tower/AWX Execution Environments Using AWX-EE and ansible-builder without the K-word. But I know how to use Key Words to my advantage to get traffic to my site. HECK YES I DO!

Note to self: I should just title my blog posts after what I think people will literally type into Google.

Anyway, there has been a nice change to Ansible Tower/AWX of late. Actually it happened quite some time ago, but in my usual fashion, I am a bit late to the table on this one.

It’s still relevant because it combines two trends I am seeing in our industry:

1. Leveraging Kubernetes-enabled pods to do all the things.

OK, so that’s one trend . . . I guess.

But still, if you have dealt with Ansible Tower/AWX for a while you are aware that previous to Execution Environments (aforelinkedto . . . it’s a word . . .), you had to inject all of the Python modules as a venv and what not into the Tower/AWX implementation (the difficulty of which depends on how you are implementing it).

What if you need to make a change? Well then, some a-hole (and that a-hole might just be you) has to go in and modify the venv to include it.

But what if I told you there is a way to 1) Allow for AWX/Tower users to self-manage their venvs themselves and 2) have this be integrated into Kubernetes so that the environment is on-demand and ephemeral.

And we all know what happens when something is ephemeral, right?

What You’ll Need (Yeah, I Know . . . It’s A lot) . . .

. . . but there is no reason you can’t run this locally on your laptop with minikube, as a test.

1. Kubernetes.
2. Ansible Tower/AWX as a Kubernetes Operator.
3. Ideally, a Docker repository like harbor.
4. The Ansible AWX Execution Environment Repo.
5. Docker Engine >= 17.06 (Multi-Stage Builds)
6. Admin-level priveleges to your Job Templates in AWX.
7. Refresher on how to create containers written by mwa.

Preparing for the Docker Build

So bear in mind that what you are doing here is building a docker image that will run all of your ansible playbooks/code in a separate Pod in Kubernetes each time a Job runs in Tower/AWX. It is completely self-contained and has everything it needs. It will also be deleted once it completes. You might be asking, “How does Tower/AWX know how to launch its Execution Environment as a Pod?”

Well, it’s all baked into the Tower/AWX cake. If you are aware of how Kubernetes works, you should be able to reverse engineer it. I would start here. If you have any other questions, hit me up at bryansullins@thinkingoutcloud.org.

1. Once you pull down the awx-ee repo, edit the requirements.txt to reflect your required Python packages. Do not remove the singular default entry for ansible-builder. Simply add the Python packages you require to this file. For example

git+https://github.com/ansible/ansible-builder.git@devel#egg=ansible-builder
jmespath==0.10.0
lxml==4.5.1
psphere==0.6.0
pynetbox==4.3.1
. . .
# and so on

2. Edit the _build\requirements.yml file to reflect the required Ansible Collections you will need. I recommend commenting out the ones you do not need and only adding the collections you require. Otherwise the resultant container will be larger than it needs to be. The default entries are as follows:

---
collections:
  - name: awx.awx
  - name: community.general
#  - name: azure.azcollection
#  - name: amazon.aws
#  - name: theforeman.foreman
#  - name: google.cloud
#  - name: openstack.cloud
  - name: community.vmware
#  - name: ovirt.ovirt
  - name: kubernetes.core
  - name: ansible.posix
#  - name: ansible.windows
#  - name: redhatinsights.insights

3. It is unlikely that you will need to alter the Dockerfile (I didn’t), but if you know what you are doing, you can certainly make changes.

Running Docker Build

One caveat I have here is that the awx-ee repo, by default, uses a tool called tox, which allows you to test/run packaged python code. If you are familiar with it, then you can certainly use it.

But because I am a cool guy who knows things, I know you don’t need it, so I am just going to run good ‘ol docker build.

1. Now we can do a docker build to create the EE docker container:

docker build -t your_ee_name_here:1.0.0 .

2. Now that it’s built, you can tag it for upload to your registry of choice:

docker tag your_ee_name_here:1.0.0 whatever.your.harbor.repo.is.com/project-name/your_ee_name_here:1.0.0

3. And finally push it to harbor:

docker push whatever.your.harbor.repo.is.com/project-name/your_ee_name_here:1.0.0

Testing the Execution Environment

Now that you have it in harbor, it’s time to test it. I am assuming you can run the container locally and test the code if you have the playbooks available to run in this standalone container, but it would be more realistic (and arguably easier) if you just go ahead and run it as it is supposed to be run in Tower/AWX.

Log in to your Ansible Tower/AWX environment, select Execution Environments on the left Navigation Pane and click the Add button:

Now enter the Name, Image (harbor url location), Pull (I do “Always pull container before running”), Description, which Organization this is associated with, and Registry Credentials (if any). If you have any questions, there are some default examples from RedHat’s quay registry.

Next, simply create a Job Template, and when you do, you should have the option to choose your EE using the Magnifying Glass.

One thing we noticed while running these Execution Environments is that they are available in a very granular and hierarchical way. They can be available globally, or within an organization, or within the Job Template.

Now run the Job the way you would normally to see if it runs successfully. If it doesn’t, then you’ll need to troubleshoot and make adjustments to your AWX EE Docker image, and rinse repeat uploading it to your registry, and running it again. Hopefully it will work the first time for you!

Questions? Hit me up on twitter @RussianLitGuy or email me at bryansullins@thinkingoutcloud.org. I would love to hear from you.