Over the last twelve months it seems to be a common scenario where customers find that the password for the root account of vCenter Server expires and then they start observing weird behavior within SDDC Manager.

By default following the initial deployment of a vCenter Server appliance, the root password will expire after 90 days, it’s possible to adjust these default policy settings with no impact to the operation of VMware Cloud Foundation but it’s often over looked until its too late.

If you do find yourself in this awkward position the good news is that from vCenter Server 7.0u1 on-wards, the vSphere engineering team has made our lives just a little bit easier by improving the process of resetting the root password without the need to go through the GRUB boot process.

Here I provide a detailed procedure of how to recover from a root password expiring ensuring that we not only resolve the immediate issue but also ensuring there is no further impact to VMware Cloud Foundation (NOTE: Tested in VMware Cloud Foundation 4.2).

Root Password Reset

The first step is to reset the password of root account so that we can actually log into the console. As mentioned before the vSphere engineering team introduced an improvement here where we no longer need to reboot the vCenter Server appliance which of course incurs downtime, instead we can simply connect over SSH and we will be forced to provide a new password.

Step 1 – Connect to the vCenter Server appliance using and SSH client such as Putty using the root user and the original expired password.

Step 2 – The system will force you to enter a new password, follow the on-screen prompts (remember that simplistic or dictionary passwords will not work, I used 1Upp1$pecMin8 during my testing so I know this works)

Step 3 – Type shell

We have now successfully reset the password and gained access to the system, but now our root credentials do not match the details stored within the SDDC Manager inventory.

Revert Root Password

Now we have access we need to make sure that the root credentials and SDDC Manager are in sync, here we have a couple of options, we could log into SDDC Manager and perform an update on the root credential details or an alternative is to revert the root password back to the same as it was before it expired. I’m going to show you the latter method as I want to keep the original password for now as this is my home lab and I want to keep things consistent.

Step 1 – In order to revert to the previous password we must first make an adjustment to the operating system pam settings, as by default we cannot enter a password if its one of the last 5 used. Edit the /etc/pam.d/system-password using vi and comment out the following line and save the file.

password  required    pam_pwhistory.so  debug use_authtok enforce_for_root remember=5

Step 2 – Enter the original password for the root account by entering passwd and following the prompts.

Step 3 – Edit the /etc/pam.d/system-password using vi again and remove the comment out for the line from Step 1 and save the file.

And there you have it, you have now reverted the password back to the original known by SDDC Manager and everything should be operational again.

Don’t forget to then modify the password policy for the local accounts as desired, enable the email alert of possible and ensure that you include a process in your operational run book to rotate the passwords via SDDC Manager on a periodic bases prior to the expiry policy.