I’ve been using VMware HCX lately while working on migrating a data center. HCX offers a few different ways to migrate workloads from the source vCenter to the destination vCenter (whether that destination be in VMC on AWS or another vSphere environment). The primary way I’ve been migrating workloads is using HCX’s Bulk Migration.

A little bit about HCX…

Bulk Migration is one of two different HCX migration types that piggybacks on vSphere Replication. The other is Replication-Assisted vMotion. The primary difference is the switchover workflow. Both use vSphere Replication for the initial sync of data and then enter continuous replication once the initial sync is complete. For both replication types, they will remain in this state until the specified maintenance window (which is selected when initiating the replication).

During Switchover, Replication-Assisted vMotion performs a vMotion of the virtual machine from Source to Destination. Once completed, the VM no longer resides in the Source site. During this migration, the desire for the business is to have an artifact remain behind in the Source data center in the event there is a need to fail back. This benefits from the switchover workflow used by Bulk Migration. When Bulk Migration reaches the maintenance window and begins Switchover: the Source VM is powered off, the VM is registered/powered-on in the Destination vCenter, and cleanup operations are taken in the Source data center (the VM is moved to a “VMsMigratedToCloud” folder, the VM is renamed with random numbers appended to it, and the NIC is removed to prevent accidentally having duplicates on the network). This leaves an artifact in the Source data center – which happened to be the desired/required approach to allow for a point in time “fallback plan.”

For any of HCX’s migration types, the maintenance window can be scheduled well in advance. This is particularly important for Bulk Migration and Replication-Assisted vMotion so they can seed the initial VM data. After the initial sync, VMs enter a state of continuous replication.

Important note/distinction: Continuous replication does not indicate that replication is happening with every I/O – there is no copy-on-write to both Source and Destination storage. Instead, continuous replication is a delta sync of changed blocks every two hours. Once the system enters continuous replication, the HCX UI indicates that the system is “Waiting for maintenance window” but does not provide any details on the delta syncs – whether they’re in progress, completed, or their overall size.

The Problem

With many, if not all, systems in continuous replication, I was left in a state of waiting for the business-approved maintenance window to initiate migration of the systems. Based on the steps stated above, you might have guessed that there’s a short period of downtime for each system. While waiting for the maintenance window, we suffered a brief network outage between Source and Destination sites which, unfortunately, failed some of the Bulk Migrations. With only some of the replications failed, the question came up: “How can we verify that the other replications are properly reporting that they’re still successfully replicating data and that they haven’t failed?”

The Solution

Let’s use some PowerCLI! The HCX UI doesn’t report on continuous replication data, but HCX is piggybacking on vSphere Replication to perform delta syncs. This means that events are written to vCenter each time a delta sync is started and completed for a given VM.

While some VMs may have failed during the network interruption, I figured that it would be reasonable enough to assume that others were still running successfully if they were still performing their delta sync operations. I couldn’t find any of this information in the HCX UI. I couldn’t find this information by querying HCX via PowerCLI. Here’s what I came up with:

Connect-HCXServer hcx-manager.mueller-tech.com -Credential Get-Credential
Connect-VIServer vcenter.mueller-tech.com -Credential Get-Credential

$vms = Get-HCXMigration | Where {$_.State -eq "WAITING_FOR_MAINT_WINDOW"} | Select VM

foreach($vm in $vms){
Get-VIEvent -Entity $vm.vm.name | Where {$_.EventTypeId -eq "hbr.primary.DeltaCompletedEvent"} | Select -First 1 | Select @{Name="Name";Expression={$_.vm.name}},CreatedTime | Export-CSV .\ReplicationStatus.csv -append
}

To break down what the above code is doing:

• Connecting to the HCX Manager
• Connecting to vCenter Server
• Querying HCX to find all migrations that are finished with their initial sync and in continuous replication
• For each VM reported, search for the first (most recent) delta sync completed event
• From that event, select the name of the VM and the time the event was created
• Export the information to CSV for reporting

The information doesn’t necessarily need to be piped out to CSV, but I found it easier to manipulate and share data than displaying to console.

Bonus

Prior to this incident, I found myself trying to figure out just how large the latest delta sync was for each of the systems in continuous replication. I couldn’t determine the number of changed blocks from the HCX UI nor using the HCX PowerCLI module. I found the information in the same Delta Completed event. Adding just a little to the query above:

Get-VIEvent -Entity $vm.vm.name | Where {$_.EventTypeId -eq "hbr.primary.DeltaCompletedEvent"} | Select -First 1 | Select @{Name="Name";Expression={$_.vm.name}}, @{Name="KB Transferred";Expression={($_.Arguments.Value)/1000}}

I hope you find this information helpful. Let me know if you’ve found a better way to gather this data!