Release notes
| Product Version | LiveData UI | LiveData Migrator | Hive Migrator | CLI |
|---|---|---|---|---|
| 3.2.1 | 14.12.4 | 3.2.5 | 2.8.7 | 2.2.4 |
Release Highlights
Parallel Scan
Scanning files and directories is central to the data migration process. Challenges with this scanning include where there are large filesystems, with high latency, and significant client activity. To meet these challenges we have re-engineered our scanning approach and introduced Parallel Scan.
In this release Parallel Scan is now approved for production use. Parallel Scan can improve scanning performance, reduce migration times, and enhance the management of your system resources through the introduction of a Global Scan Pool for Target Match migrations. Any improvements will depend on your particular environment, the number of migrations, and the structure of your data in terms of files and directories.
Parallel Scan can be enabled or disabled from the UI or CLI when creating or resetting a migration. There are configuration options which can be set via the API. Please consult with support for additional guidance.
HVM Metadata Verification
We are excited to announce the availability of the Metadata Verification functionality in this release. This is designed to help users validate the availability and consistency of metadata between a source Hive metastore and a target Hive metastore.
With this release, users can run verification jobs against existing metadata migrations that scan and compare metadata objects such as databases, tables and partitions to determine whether they are present and consistent, missing from either environment, or structurally mismatched. This gives users increased confidence in their migrations by identifying gaps and discrepancies before they cause downstream issues.
For more information on metadata verification see here.
Other Improvements
Recheck and Repair options for Verification now available from UI
You can now benefit from the verification repair and recheck options being available in the UI. Users can now utilize repair on the UI to address any inconsistencies identified by verification and recheck to ensure the inconsistencies have been addressed. This will provide additional ways to easily and efficiently resolve inconsistent data that may have been caused by unforeseen environment issues or maintenance windows and verify they have been addressed before proceeding with the migrations.
For more information on verification repair see here.
Resolved Issues
Data Migrator Core
Hive Migrator
HVM-5569 Errors creating databricks migrations from CLI
HVM-5566 Databricks - Table dropped while copy into retries causes copyIntoManager to continue to poll
HVM-5512 [Snowflake/Iceberg] Resets of metadata migrations freeze and can't progress
HVM-5383 Checksum comparison issue between HDP 2.6.5 and HDP 3.1.5 client
UI
ONEUI-8514 Display error snackbar when trying to trigger multiple verifications
ONEUI-8434 UI Display Parallel Scan Enabled/Disabled for migrations in Bulk Action table
ONEUI-8428 Table page doesn't update when data removed
ONEUI-8388 Restructure the H2 database upgrade to leave the existing database untouched until the new one is successfully written