Openshift 3.5 installation headaches (and where to find the aspirin)

I've been in the container business for over a year now, and have worked with Docker and Kubernetes. So when the opportunity arose to work with Openshift, I figured it would be a great skill to add, and should be pretty easy since it runs on Kubernetes, and I'm already familiar with that.

Well, the headaches started almost immediately. This isn't due to the nature of the product, but rather the installation documentation. If you go to Red Hat and look up the quickstart guide, it gives instructions for installing Openshift 3.0. Well, I wanted 3.5, so I decided to try changing things up to make it work.

If you follow the instructions, you're likely to end up with errors regarding openvswitch during the atomic-openshift-installer process. It all comes back to our old friend, dependencies.

Here's a basic outline of what you need to do to install Openshift. I'm not going into details, except to call out where things need to be done differently from the documentation:

Ensure your hosts are running RHEL 7.5 and have either XFS for overlay storage, or have a dedicated block device you can use for Openshift storage.

Ensure DNS resolution

Install prerequisites - This is the important part:

subscription-manager attach --pool=<your pool id here> subscription-manager repos --disable=* subscription-manager repos --enable=rhel-7-server-rpms --enable=rhel-7-server-optional-rpms --enable=rhel-7-server-extras-rpms --enable=rhel-7-server-ose-3.5-rpms --enable=rhel-7-fast-datapath-rpms yum -y update

Notice the last repo, rhel-7-fast-datapath-rpms. This repo contains the current version of openvswitch.

Create a root ssh key (or a key for a service account) and copy it to all nodes.

Install docker and configure docker storage

Install dependency packages:

yum -y install atomic-openshift-docker-excluder atomic-openshift-excluder atomic-Openshift-utils bridge-utils git iptables-services net-tools wget openvswitch

Notice the excluders in the list. These are handy for preventing yum from updating Openshift or docker before you're ready to update them. These are also optional and won't prevent the Openshift install if they are omitted.

Install Openshift:

atomic-openshift-excluder unexclude atomic-openshift-installer install

The excluder line is only necessary if you installed the excluder in the first place. If you unexclude openshift, make sure you exclude it again when you're done.

Adding your custom images to MaaS

In my last two posts, we covered creating custom Windows and RHEL images for MaaS. Now we'll deal with uploading them for use. The process is relatively simple:

1. Upload your image to the MaaS server

2. Import the image to MaaS

The first step is easy enough, just use SCP (or a program like WinSCP in Windows) to upload the file:

To Import the image in MaaS, use the following command:

maas $profile boot-resources create name=custom/$imagedisplayname architecture=amd64/generic content=@$tgzfilepath

In the code, I've used variables to make explanation simpler. In real life, I'd just input the actual values. Following is an explanation of each value:

$profile - When using the maas command, you must first log in. This variable represents your maas profile name.

$imagedisplayname - You can put any value here. Preceding it by custom/ means that the image will show up under custom images in MaaS with the name specified following the /

$tgzfilepath - This is the full path to the image file on the MaaS server. If you are admin2 and you kept an image namede my-image.tgz in a directory called images under your home directory, this value would be /home/admin2/images/my-image.tgz

Creating a custom RHEL image for MAAS

A little change from our typical programming, today's post is about how to add an image to an Ubuntu MaaS instance. So what is a Sr Consultant, specializing in Azure and containerization doing working with Linux in an on-premises environment?

As far-removed as it seems from the norm of what we do at Avanade, there is still a lot of demand for on-prem solutions. Many cases are like ours; we have a large lab full of equipment we've already paid for and it makes fiscal sense to do much of our dev work there.

In our lab, we utilize MAAS (Metal as a Service) to manage our hardware. Out of the box (for the free version of MAAS at least), MAAS comes with images for Ubuntu and CENTOS. With the paid version of MAAS, you get Windows and RHEL images as well.

A brief glance around the internet showed that there used to be a tool called MAAS-image-builder that allows for the custom creation of RHEL images. Ubuntu seems to have burried many of the references to this once they started using images as a differentiation between free and paid versions. There are actually a few branches of this, none of which worked out of the box.

The version we eventually got to work was this one: https://code.launchpad.net/~ltrager/maas-image-builder/update_ks

To make it work, you must build it as per the instructions, and once installed, make a few changes:

1. Add the following to /usr/lib/maas-image-builder/contrib/rhel/rhel7-amd64.ks in the %packages section:

Without the python2-oauthlib.noarch and python2-requests-oauthlib.noarch packages, the install will complete, but MAAS will register it as a failed deployment.

Additionally, as anyone familiar with kickstart files will probably tell you, it's advisable to create a standard user account, and register your subscription in addition to any other customization you want to add to your image.

One caveat however: Keep in mind that no matter what you do with the part command in your kickstart, it won't actually make a difference to the final install process in MAAS. MAAS manages storage layouts on its own, and won't allow a custom storage layout for any OS other than Ubuntu.

WALinuxAgent issue in a Linux VM on Azure Stack

I was deploying some Linux RHEL systems yesterday on Azure Stack. Everything was working and then it wasn't.  The VM had deployed fine but the custom script extension had failed.  It takes 2 hours for the deployment to timeout. Deployment Failure Timeout 

Failed Script Extension

Failed Extensions Details

If you cancel the deployment and attempt to delete the resource group you will have to wait for the custom extension to timeout anyway, which takes two hours.  Microsoft recommends against canceling deployments and let them timeout, especially on Linux.  On the next deployment attempt, I got the same results.  On closer inspection of the stdout for custom script extension 2.0.6 (/var/lib/waagent/custom-script/download/0/stdout), I found this error.  warning: %postun(WALinuxAgent-2.2.18-1.el7.noarch) scriptlet failed, signal 15

Stdout log

Reviewing the bash script, the first line that was running was

yum -y clean all && yum -y clean metadata && yum -y clean dbcache && yum -y makecache && yum update -yvq

This appears to be a known issue, here is something someone has written with some information trying to explain what is happening https://github.com/Azure/WALinuxAgent/issues/178

I believe this explains the timeout.  The Agent had stopped and Azure Stack can no longer talk to the agent in the VM.  If you experience this specific issue adding an exclusion should leave the existing binaries alone.  If you don't have this specific error but are still experiencing a timeout it is probably worth checking the WALinuxAgent is still running.

yum update -y --exclude=WALinuxAgent