Upgrade vRA 7.1 to 7.2 journey

Challenges are one of the reasons that keeps my interest in IT.  I’ve supported many products in my career.  Every product I’ve supported have one thing in common…..UPGRADES..  ( Typed that in upper case on purpose).

There are couple strategies to upgrades.

  • Blue / green deploys, a recent cloudy term to stand up new, deploy, test and cutover.   Before this was a term, I prefer this method.  If there is an issue, it’s easy to revert to original environment.
  • In-place upgrade of existing environment.  This is NOT my favorite method, but necessary sometimes.

This post shares experiences on a recent upgrade from vRealize Automation 7.1 to 7.2.    These are my notes, they include links to formal documentation, which in my experiences with any product, there are a few tips / tricks between the lines docs don’t cover.  I’ve included my raw notes below.  I hope this helps in your adventure.

I was fortunate enough to have a separate environment to test before doing the real environment.  Since this was an in-place upgrade, going through the experience helped prepare what to expect, as well as to revert back to original environment.

With the magic of cloning Linux appliances, snapshots on IaaS windows machines, and a SQL Server backup.  It helped simulate the ability to go back to original infrastructure.  It’s doesn’t make me as comfortable standing up new, but was better than no backout plan at all.

  • Read the upgrade docs, search forums for others reporting issues
  • Test the upgrade multiple times.  Also, think about if something goes wrong, how to revert to original version.  Do this multiple times until you are sick of it, that means you’ll be prepared
  • Have backups, clones and use snapshots on windows machines
  • Disable Backups before starting.  Contention on machines could cause issues
  • If you have support with VMware, open a proactive case and have them review your environment.
  • Clean-up any un-submitted requests
  • Clean-up any In-Progress requests that are orphaned
  • Give yourself 4 to 8 hours.  Communicate to your end users.  (Under promise, over deliver)
  • Coordinate with your users to test use cases after upgrade
  • Remember external systems like vRealize Business, Log Insight that access vRA.
  • Make sure end users disable automated build requests.

Here are my raw notes including links including  steps I followed.   These were reminders to help the overall process, the order of operations.  I’ve found having these types of notes help to refer to.

Prep work, review documents and blog posts.

Items I did as prep work.

Launch config using the command: java.exe -jar <vRPT Jar file> config

  • Download updated management agent. KB article with updated management agent, Workaround issue on upgrade on IaaS components
    http://kb.vmware.com/kb/2147926

Load-balancing

The network guys will wonder, what?!  Mine did, just tell them the vendor requires the change, double and triple check this step.

Cloning each machine vRA to upgrade

  • Turn off vRA appliances – Clone each machine (good to have backups of original appliances)
  • Turn off IaaS machines, snapshot.
  • Turn on all machines, verify services both IaaS and vRA are healthy (both appliances) I learned by testing if you clone windows, I see extra machines in the VAMI you’ll see “several called clone”
  • https://<vra-url>:5480/#cafe-services (make sure all services are started)
  • Close any opened files in the pgdata directory and remove any files with a .swp suffix. (on primary vRA appliance)
    –/var/vmware/vpostgres/current/pgdata/
  • Backup SQL Server database (Get with DBA to schedule ahead of time)
  • Upload ISO to datastore, mount to primary and secondary appliances if you don’t have internet access for updates.
  • Adjust settings in VAMI on primary and secondary to use cd-rom
  • ***- Run on a remote machine well connected to network, not laptop on wireless***
  • ***Disable vCenter backups on the cluster so snapshots aren’t taken***
  • De-register vRB via vRB appliance (because we haven’t used it yet) – might not apply to you.
  • Run update on primary (Look for text similar listed below when completed)
  • Reboot primary and secondary vRA appliances after upgrade completed
  • Verify both appliances upgraded
  • Deploy updated Management agent on IaaS machines
  • Deploy updated java (version 1.8)
  • Reboot each IaaS machine
  • Verify services on IaaS and appliances are healthy
  • Create upgrade.properties on primary ( Backup file = cp -p upgrade.properties upgrade.properties.password)
  • run ./upgrade (cross your fingers) – I had to run three times and my install FINALLY upgrade all six windows machines.

Raw text after upgrade
Version 7.2.0.381 Build 4660246

Last Check:
Tuesday, 2017 January 31 15:46:07 UTC-5 (Using update CD found on: /dev/sr0)

VA-check: finished

Pre-install: finished

After all appliances are upgraded, ssh to the master appliance and go to /usr/lib/vcac/tools/upgrade. Populate all the required data in upgrade.properties and execute ./upgrade script

Replica nodes are upgraded successfully. Reboot master node to trigger the reboot of replica nodes

Post-install: finished

Update finished successfully.
WARNING: Immediately update any vRealize Automation IaaS nodes after reboot to avoid product version mismatches.

Last Install:
Tuesday, 2017 January 31 16:29:07 UTC-5

VA-check: finished

Pre-install: finished

After all appliances are upgraded, ssh to the master appliance and go to /usr/lib/vcac/tools/upgrade. Populate all the required data in upgrade.properties and execute ./upgrade script

Replica nodes are upgraded successfully. Reboot master node to trigger the reboot of replica nodes

Post-install: finished

Update finished successfully.
WARNING: Immediately update any vRealize Automation IaaS nodes after reboot to avoid product version mismatches.

Hope this helps,

Steve Schofield
#vExpert 2017

@steveschofield
http://vsteve.me

Getting used to web client only world…Using VMware HTML 5 fling….

Looking for some hope?  This post hopefully will make you smile, give you some hope.  Grab your favorite beverage and let’s begin.  I’ve been working in an environment with multiple vCenters.   Many were either 5.5 or 6.0.   We still had access to the famed ‘full C# client’, even though the Flash Client was available, many didn’t use and would continue to use C# client until we were forced to change (me included).

For long-time admins, the full client is like comfort food or that favorite beverage they are used to, don’t make me change. With anything in IT, change is part of the job.

In one evening, we upgraded multiple (5) vCenters to 6.5, putting the C# out to pasture.  On one hand, we were thrilled the upgrades and migrations from windows to appliance worked (couple bumps, but we were able to get past).  for those wondering what bumps, we had to remove / re-add the PSC to Active Directory.

On the other hand, there was a small empty feeling.  I try look at the bright side in any situation. ( I really do although there are others who would disagree).

As part of the 6.5 rollout, there is two clients.

  • The Flash client (full functionality and some stresses to using it!)
  • The new HTML 5 shiny client

The links are accessible within the landing page when navigating to the vCenter by name. I’ll give VMware credit putting the wording (partial functionality) on the landing page.   This blog post isn’t here to debate Flash vs HTML5, that has been settled elsewhere.  Remember, this blog post is about giving hope. 🙂

html5-1

Did I say this blog post was about providing some hope.

Dennis Lu apparently likes taking on big challenges.  He is a frequent contributor and main person for something called HTML 5 fling (more info here)  For those unaware or haven’t checked this in a while, it’s grown up.

As part of our rollout, I deployed a separate HTML 5 fling appliance. The appliance is used on more frequently used vCenters accessed by customers. Plus, you can give the appliance a handy DNS name. We call ours vhtml.example.com (have to get a little “v” in the name)

When I first explored the HTML 5 fling, the appliance required a re-deploy every time. Although the HTML 5 fling was “kewl”, it wasn’t functional enough to use in our environment.

Fast forward, the current release is 3.9 as of this blog post. A few weeks ago, I deployed the 3.3x release appliance.  I’ve used the update feature twice without issues (remember to snapshot before upgrade).  Good Job Dennis and crew!  Handy feature here.

html5-2

To access this functionality, go to https://<ApplianceIPorName>:5490 (note 5490, not 5480 like I type a few times).  Login and click update.

The update will take a few minutes.  I noticed the finalized update status appears to not always notify when done..   I waited a few minutes and refreshed my browser (Chrome is my preferred one).

Here is a screenshot of the update in progress.

html5-3

The reason we deployed the extra appliance was to give ability to have a client end users could access, that gets updated more frequently than the HTML 5 client hosted on the vCenter appliance.

To update the HTML 5 hosted on vCenter, requires an upgrade as far as I know.  Would be handy to do it separately from a vCenter upgrade.  (@VMware hint hint!)

We generally try to limit upgrading vCenter to once or twice a year.  Using an external appliance, we get new features faster, safer with less hassle and risk upgrading vCenters.

I hope you enjoyed this slice of hope, there is part of me that misses the C# as my every day tool, we have a couple of vCenters it still works on, although there is little need to access them regularly.

Thanks Dennis and team for providing this option.  It’s made the transition a little less painful.  The disclaimer is use at your own risk, test in a non-production environment first.

PS – The appliance appears it needs internet access, so you’ll have to check with the security group or whoever manages the firewalls to download updates. I’m not sure if there is a way to do offline updates to an existing appliance, probably a reload is required.

Enjoy,

Steve Schofield
#vExpert 2017

@steveschofield
http://vsteve.me

What is this section for?  It’s a separate way to share ideas to pass along that I thought of while typing my blog post.  If you know some of the answers, I’m on twitter at @steveschofield

  • Love to have the appliance automatically redirect port 80 to 443.  We have to type in https:// (maybe a browser issue now HTTPS is more common)
  • Ability to externalize web client on multiple machines, load-balance vs. being a single point of failure + the authentication window that appears on a Platform service controller
  • Update HTML 5 client / Flash web client separately from vCenter appliance
  • Single Appliance access multiple, separate vCenters hosted in separate SSO domains.

 

 

 

Change Docker default network to persist reboots and vRealize Automation 7.2

image

Containers are coming to a company near you! Containers are all the rage.  They are one of the hottest technologies in IT.  In all seriousness, all technologies have to mature, fit a business need.  Docker is a leading company in this space.

Within vRealize Automation 7.2, there is a container option.  Here are docs about containers and vRealize Automation 7.2.   As a vRA admin, I want to understand all features.  To help achieve my goal, I wanted to setup a catalog item similar to these articles.

Mark’s article was very helpful.   His article uses a DHCP scope (which is ok) and default networking in Photon assumes DHCP.  My article uses a vRO workflow, script on the template to set networking based on ip settings handled by vRA.

My article is related to vRealize Orchestrator, but the concept is the same.  Maybe I’ll blog my Photon example later although it’s similar to Mark’s article.   Here are my Photon workflows and addnetwork.sh I used on Photon vRA example

Regardless of how you setup your template, one of the features of Docker has it’s own internal networking.  The default is 172.17.0.0  (more info here). For some enterprises, this can conflict with existing non-routed internet address ranges ( 10.x, 172.x, and 192.168.x).

I ran into this and needed to adjust my default docker network.   My docker network wouldn’t persist reboots.  I initially found out how to change default docker network, but it wouldn’t persist a reboot.  (Links are listed below)

I wanted to setup my Photon template, used by vRA, with a persistent docker network that wouldn’t revert back to 172.17.x.x after reboots.  Follow Marks or my article to setup a Photon template, catalog items in vRA, then adjust your Photon template using instructions below.

After working with VMware and some experimentation.  This worked for me.

Photon OS use systemd-networkd to manage the network. Here is the external documentation on how to setup a bridge with systemd-networkd: https://wiki.archlinux.org/index.php/Systemd-networkd#Bridge_interface

Following steps:

# cd /etc/systemd/network
# vi 10-static-docker0.netdev

[NetDev]
Name=docker0
Kind=bridge

# vi 10-static-docker0.network

[Match]
Name=docker0
[Network]
Address=192.168.3.0/24

# chmod 755 10-static-docker0*
# systemctl restart systemd-networkd.service
# systemctl restart docker

Modify whatever you want, I left 192.168.3.0/24 as that will work in my network.

Here are other links that helped along the journey.

There is a few ideas.
http://www.vmtocloud.com/how-to-enable-docker-remote-api-on-photon-os/

Showed how to adjust the docker networking, didn’t persist reboots though
https://support.zenoss.com/hc/en-us/articles/203582809-How-to-Change-the-Default-Docker-Subnet

Known issue, I applied this hotfix to vRA
https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2148212

Enjoy,

Steve Schofield
https://vsteve.me

vRO workflows

<< back to main article

Download vRO package

Download vro-org.vsteve.me.package on Github

There are two workflows, one action you’ll import into vRO.  The workflows are used by the Event Broker in vRA to setup networking on .  The workflows are available to download.

Go to the landing page on vRA

2017-01-01_20-27-08

Download vRealize Orchestrator client

Type in user id and password

default is vcoadmin / vcoadmin

You’ll need Java

2017-01-01_20-30-51

Import package

Here is an article by Jonathan Medd to import a packages into vRO

2017-01-01_20-37-26

Adjust the root password on the Template-vRO template.

The setting is on vRO Run in Guest workflow

image

Back to vRA to setup Event Broker

Enjoy,

Steve Schofield
http://vsteve.me

Setup Template-vRO catalog item

<< back to main article

Here are steps to publish in the vRO template as a catalog item.    if you want more information on setting up Catalog items, Entitlements, check out Eric Shanks vRealize Automation guide.

Create a Service called vRO-App

2017-01-01_19-56-22

Go to Catalog items,

Select Template-vRO blueprint

2017-01-01_19-58-09

Add catalog item to the vRO-App Service

2017-01-01_19-59-29

Entitlement the item to vRO-App service.   For this example, I entitled just the configuration administrators (configurationadmin by default).  If you have this attached to a LDAP source, you could provision based on LDAP group membership.

2017-01-01_20-02-02

The Template-vRO72 catalog item will show up after entitled.

2017-01-01_20-04-20

Enjoy,

Steve Schofield
http://vsteve.me

vRO setup Event Broker

<< back to main article

vRA introduced the Event Broker feature.   We’ll setup a subscription to fire to run the vRO-Assign-Network workflow.

Click New

2017-01-02_21-09-24

Select Machine.Provisioning option

2017-01-02_21-09-45

Add the following conditions or adjust to fix your environment

2017-01-02_21-10-10

Select vRO-Assign-Network workflow

2017-01-02_21-10-29

Click Finish

2017-01-02_21-11-20

Don’t forget to Publish to make the subscription live.

Enjoy,

Steve Schofield
http://vsteve.me

 

 

2017-01-02_21-09-24

Add Key-State-Changes Property group, add to blueprint

<< back to main article

vRO needs the payload properties bucket, which contains all information about the request, including network information.   There are custom properties added to blueprints to expose this information.

The attacked example are the list of properties I use on blueprints.  I encourage you to investigate each item to understand which data is made available.

Go to Administration > Property Groups

2017-01-01_21-05-15

Add to the property group

2017-01-01_21-00-12

Save

2017-01-01_21-06-43

Edit your blueprint

Add on custom properties page, Property Groups

We will cover in another article how to expose the properties and use meta data,.

2017-01-01_21-07-07

Enjoy,

Steve Schofield
http://vsteve.me

Add Blueprint, network for Template-vRO7

<< back to main article

Sign-into vRA as account with permissions to create / publish blueprints, add items to catalog.

Click Design

New Blueprint

I adjusted “1 to 60 days” option

2017-01-01_19-18-25

Drag vSphere Machine on canvas

2017-01-01_19-22-56

Select Build Information,

Change to Linked Clone in Action field

2017-01-01_19-23-35

Select the “….”

2017-01-01_19-41-39

A popup window will appear

Select Template-vRO7

2017-01-01_19-41-12

Select Network and Security

2017-01-01_19-46-46

Drag existing network to canvas

2017-01-01_19-47-54

Select network profile

In our example, it’s internal network

2017-01-01_19-48-17

Select Network on Blueprint

Follow steps in picture

2017-01-01_19-51-07

Close Blueprint

Make sure Publish

2017-01-01_19-53-14

Enjoy,

Steve Schofield
http://vsteve.me

Add Script to Template-vRO7 VM

<< back to main article

After the vRO OVF  is deployed and running.  Open the VM in VMRC (VMware Remote Console), login.  I add a bash shell script to the template vRO will execute to adjust network settings passed by vRA.  Once the script is added, permissions are adjusted, take a snapshot

Open VMRC, login as root

2017-01-02_16-44-33

In VI, craate a file named addnetwork.sh placed on the root folder.   The file can be located in any folder, we just chose the root folder.

2017-01-02_16-45-25

Type settings listed in the image, or copy and paste from listed below

image

Type command to give vRO the ability to execute the script

Chmod 755 /addnetwork.sh

Notice perms after adjusting

2017-01-02_16-48-33

Last STEP

  • Shutdown VM
  • Take a single snapshot

image

—————————–

Code from step 3

mv /etc/HOSTNAME /etc/HOME.original
echo $1 >> /etc/HOSTNAME
mv /etc/sysconfig/networking/devices/ifcfg-eth0 /etc/sysconfig/networking/devices/ifcfg-eth0.original
echo “DEVICE=eth0” >> /etc/sysconfig/networking/devices/ifcfg-eth0
echo “BOOTPROTO=’static'” >> /etc/sysconfig/networking/devices/ifcfg-eth0
echo “STARTMODE=’auto'” >> /etc/sysconfig/networking/devices/ifcfg-eth0
echo “TYPE=Ethernet” >> /etc/sysconfig/networking/devices/ifcfg-eth0
echo “USERCONTROL=’no'” >> /etc/sysconfig/networking/devices/ifcfg-eth0
echo “IPADDR=’$2′” >> /etc/sysconfig/networking/devices/ifcfg-eth0
echo “NETMASK=’$3′” >> /etc/sysconfig/networking/devices/ifcfg-eth0
mv /etc/sysconfig/network/routes /etc/sysconfig/network/routes.original
echo “default $4 – -” >> /etc/sysconfig/network/routes
mv /etc/resolve.conf /etc/resolve.conf.original
echo “nameserver $5” >> /etc/resolve.conf
echo “nameserver $6” >> /etc/resolve.conf
echo “domain $7” >> /etc/resolve.conf
echo “search $7” >> /etc/resolve.conf
service network restart

Enjoy,

Steve Schofield
http://vsteve.me

Make sure Data Collection is working within vRA

<< back to main article

This is a short step.  There are a few assumptions

  • A connection to vCenter where Template-VRO7 is deployed
  • The vRA / vRO has connectivity and permissions to same vCenter where the Template was deployed
  • Reservations, Business groups and other items are setup
  • The Template-vRO7 vm deployed with a single snapshot (for linked clones)

Sign into vRA with administration permissions, kick off a data collection

2017-01-01_19-12-24

Look for a successful data collection, this will pull in the Template-VRO7 the blueprint will use.

2017-01-01_19-13-10

Enjoy,

Steve Schofield
http://vsteve.me