Challenges are one of the reasons that keeps my interest in IT. I’ve supported many products in my career. Every product I’ve supported have one thing in common…..UPGRADES.. ( Typed that in upper case on purpose).
There are couple strategies to upgrades.
- Blue / green deploys, a recent cloudy term to stand up new, deploy, test and cutover. Before this was a term, I prefer this method. If there is an issue, it’s easy to revert to original environment.
- In-place upgrade of existing environment. This is NOT my favorite method, but necessary sometimes.
This post shares experiences on a recent upgrade from vRealize Automation 7.1 to 7.2. These are my notes, they include links to formal documentation, which in my experiences with any product, there are a few tips / tricks between the lines docs don’t cover. I’ve included my raw notes below. I hope this helps in your adventure.
I was fortunate enough to have a separate environment to test before doing the real environment. Since this was an in-place upgrade, going through the experience helped prepare what to expect, as well as to revert back to original environment.
With the magic of cloning Linux appliances, snapshots on IaaS windows machines, and a SQL Server backup. It helped simulate the ability to go back to original infrastructure. It’s doesn’t make me as comfortable standing up new, but was better than no backout plan at all.
- Read the upgrade docs, search forums for others reporting issues
- Test the upgrade multiple times. Also, think about if something goes wrong, how to revert to original version. Do this multiple times until you are sick of it, that means you’ll be prepared
- Have backups, clones and use snapshots on windows machines
- Disable Backups before starting. Contention on machines could cause issues
- If you have support with VMware, open a proactive case and have them review your environment.
- Clean-up any un-submitted requests
- Clean-up any In-Progress requests that are orphaned
- Give yourself 4 to 8 hours. Communicate to your end users. (Under promise, over deliver)
- Coordinate with your users to test use cases after upgrade
- Remember external systems like vRealize Business, Log Insight that access vRA.
- Make sure end users disable automated build requests.
Here are my raw notes including links including steps I followed. These were reminders to help the overall process, the order of operations. I’ve found having these types of notes help to refer to.
Prep work, review documents and blog posts.
- http://theithollow.com/2016/11/24/upgrade-vra-7-1-7-2/
- http://pubs.vmware.com/vrealize-automation-72/index.jsp#com.vmware.vrealize.automation.doc/GUID-E052080A-9B61-40B9-AE56-98BAFA1463BD.html
Items I did as prep work.
- Verify all services, nodes are healthy before beginning
- Check disk space on psc, vcenter, vRA / vRO appliances (df -h)
- Check alarms in vCenter
- Make sure nightly backups of PostgreSQL are on NFS share. I run a dump on databases for extra backups about the same time as SQL Server.
- Run vRealize performance test = run a config test
- https://blogs.vmware.com/management/2015/04/new-health-monitor-tool-vrealize-suite-vrealize-production-tool.html
Launch config using the command: java.exe -jar <vRPT Jar file> config
- Download updated management agent. KB article with updated management agent, Workaround issue on upgrade on IaaS components
http://kb.vmware.com/kb/2147926
Load-balancing
The network guys will wonder, what?! Mine did, just tell them the vendor requires the change, double and triple check this step.
- Disable secondary nodes, health checks on VIPs (Same setup as installation)
https://pubs.vmware.com/vrealize-automation-72/topic/com.vmware.ICbase/PDF/vrealize-automation-72-load-balancing.pdf
Cloning each machine vRA to upgrade
- Turn off vRA appliances – Clone each machine (good to have backups of original appliances)
- Turn off IaaS machines, snapshot.
- Turn on all machines, verify services both IaaS and vRA are healthy (both appliances) I learned by testing if you clone windows, I see extra machines in the VAMI you’ll see “several called clone”
- https://<vra-url>:5480/#cafe-services (make sure all services are started)
- Close any opened files in the pgdata directory and remove any files with a .swp suffix. (on primary vRA appliance)
-/var/vmware/vpostgres/current/pgdata/ - Backup SQL Server database (Get with DBA to schedule ahead of time)
- Upload ISO to datastore, mount to primary and secondary appliances if you don’t have internet access for updates.
- Adjust settings in VAMI on primary and secondary to use cd-rom
- ***- Run on a remote machine well connected to network, not laptop on wireless***
- ***Disable vCenter backups on the cluster so snapshots aren’t taken***
- De-register vRB via vRB appliance (because we haven’t used it yet) – might not apply to you.
- Run update on primary (Look for text similar listed below when completed)
- Reboot primary and secondary vRA appliances after upgrade completed
- Verify both appliances upgraded
- Deploy updated Management agent on IaaS machines
- Deploy updated java (version 1.8)
- Reboot each IaaS machine
- Verify services on IaaS and appliances are healthy
- Create upgrade.properties on primary ( Backup file = cp -p upgrade.properties upgrade.properties.password)
- run ./upgrade (cross your fingers) – I had to run three times and my install FINALLY upgrade all six windows machines.
Raw text after upgrade
Version 7.2.0.381 Build 4660246
Last Check:
Tuesday, 2017 January 31 15:46:07 UTC-5 (Using update CD found on: /dev/sr0)
VA-check: finished
Pre-install: finished
After all appliances are upgraded, ssh to the master appliance and go to /usr/lib/vcac/tools/upgrade. Populate all the required data in upgrade.properties and execute ./upgrade script
Replica nodes are upgraded successfully. Reboot master node to trigger the reboot of replica nodes
Post-install: finished
Update finished successfully.
WARNING: Immediately update any vRealize Automation IaaS nodes after reboot to avoid product version mismatches.
Last Install:
Tuesday, 2017 January 31 16:29:07 UTC-5
VA-check: finished
Pre-install: finished
After all appliances are upgraded, ssh to the master appliance and go to /usr/lib/vcac/tools/upgrade. Populate all the required data in upgrade.properties and execute ./upgrade script
Replica nodes are upgraded successfully. Reboot master node to trigger the reboot of replica nodes
Post-install: finished
Update finished successfully.
WARNING: Immediately update any vRealize Automation IaaS nodes after reboot to avoid product version mismatches.
Hope this helps,
Steve Schofield
#vExpert 2017
@steveschofield
http://vsteve.me