Verify metadata migrations

Discover discrepancies (inconsistencies) between source and target metadata migration paths by performing a Verification on Live, Completed or Stopped metadata migrations.

Limitations and considerations

• In Data Migrator 3.2.1+, only one verification can be run at a time
• You must wait until one verification is completed before triggering a new one
• Only Hive -> Hive verifications are supported (Both local Hive agent and remote agent)

Prerequisites

Migrations must have one of the following statuses to perform verification scans:

• Live (real-time event stream notifications for changes to the source that are replicated to the target)
• Complete (one-time migration without event stream notifications)
• Stopped (a user has stopped the migration manually)

note

It is not recommended to run a verification when a migration is in progress as it will likely provide inaccurate or misleading results as the target is actively changing.

Objects Verified

The following object types are verified as part of each verification:

• Database
• Table
• Partition
• Constraint
• Materialized View

Verify migrations with the UI

View the migration status

1. From the Dashboard, select the metadata migration you want to verify.

2. On the Metadata Migration Verification panel, you can see:

   • View the Verification Summary Report
   • Verification Status - Not Started, In Progress, Complete.
   • Total Inconsistencies - Number of discrepancies between the source and target paths.

Verify a migration

Use the following options to create a new verification for a migration:

1. Select Migration Verification from the sidebar menu.

2. Cick Start Verification button.

Cancel a verification

You can cancel a verification that is in progress. After you select Start verification, you can simply select Cancel check.

View a verification summary report

After you select Start verification, the Verification Summary Report panel is updated with information from the source and target found during the verification scan. You can view this while the verification is in progress or when it's complete or canceled. When the verification starts checking the source and target, you can compare:

• The number of objects found on the source and the target
• Total number of inconsistencies found, including object mismatches, missing or extra objects.

Verifications Report Directory

Completed verification reports for metadata can be found in /opt/wandisco/hivemigrator/verifications/.

caution

When a metadata migration is Reset, the older migration verification reports contained within verfiication's report directory will not be removed

Download a full verification report

After the verification is complete, you can view and expand reports for completed verifications under Last x Verification Reports. Select Download all files to view, share, and analyze the results of the metadata migration verification. This will give you a full report as a tar archive file containing summary.json and the following files:

• verification-discrepancy.json
• verification-full-content.json
• verification-summary.json

tip

Reports are downloaded as .gz files. Use a tool like gunzip or any compatible decompression utility to extract the file before viewing.

The verification-discrepancy report shows all discrepancies (inconsistencies) between these two metastores while the full-content report shows discrepancies and all checks.

To download the full verification report, under Last x Verification Reports, expand the main verification report you need, select the download all files icon. Reports which have failed or been cancelled cannot be downloaded, these can be identified by noting that there is no data and time in the 'Report Gerneated' column.

Review the full-content report

Example full-content report.

full-content.json example

 "summaryList" : [ {
  "verifyResult" : "OK",
  "metadataObjectName" : "database1"
  }, {
    "verifyResult" : "MISSING_ON_SOURCE",
    "metadataObjectName" : "database1.table1"
  }, {
    "verifyResult" : "MISSING_ON_TARGET",
    "metadataObjectName" : "database1.table2"
  }, {
    "verifyResult" : "PROPERTY_MISMATCH",
    "metadataObjectName" : "database1.table3"
  } ]

scanResult	reason	Object	Explanation	Inconsistency
OK	OK	database1	Object exists on source and target and is consistent	No Inconsistency
MISSING_ON_SOURCE	Object is missing from the source but is present on the target	database1.table1	Object dropped from source after migration	Inconsistency
MISSING_ON_TARGET	Object is missing on the target but is present on the source	database1.table2	Object added to source after migration	Inconsistency
Mismatch	Table owner was different	database1.table3	Table owner changed after migration	Inconsistency

scanResult and reason types

scanResult	Description
OK	Object exists on both source and target metastore and is consistent
MISSING_ON_TARGET	Object exists on source but not on target
MISSING_ON_SOURCE	Object exists on target but not on source
MISMATCH	Object exists on both but inconsistent

Verify migrations with the CLI

Use the following commands to manage verifications.

hive migration verification start

Start a new verification for a migration.

Example

Trigger a new verification for a migration

hive migration verification start --name migration1

hive migration verification list

List summaries for all or specified verifications.

Example

List summaries for all verifications.

hive migration verification list

hive migration verification show

Show the status of a specific migration verification.

Example

Example status of a completed verification

hive migration verification show --verification-id migration1-1752065069284

hive migration verification stop

Stop a queued or in-progress migration verification.

Example

Stop a migration verification

hive migration verification stop --verification-id migration1-1752065069284

hive migration verification report

Download a full verification report.

Example

Download a verification report

hive migration verification report --verification-id migration1-1752065069284 --out-dir /user/exampleVerificationDirectory