Release notes

Product Version	LiveData UI	LiveData Migrator	Hive Migrator	CLI
3.3.0	14.14.1	3.3.4	2.10.2	3.3.3

Release Highlights

Live Ozone

Apache Ozone is a scalable, distributed object store designed for lakehouse workloads, AI/ML, and cloud-native applications. Unlike HDFS, Ozone separates metadata and data layers, making it more scalable and suitable for modern cloud-native and containerized environments.

In this release we are delighted to announce our support for the Ozone filesystem. This facilitates seamless, zero-downtime migration of your data from HDFS to Ozone or from Ozone to Ozone by enabling real-time replication of data changes. This minimizes risk, eliminates operational disruption, and accelerates your move to scalable, modern storage.

Configure an Ozone target describes how you can modernize your data storage by migrating to Ozone.

HVM Metadata Verification

This release includes enhancements to the metadata verification functionality to improve usability and overall user experience:

Verification Queueing – When a verification is triggered while another is running, it will automatically enter a queue and start once the running verification is completed.

Automatic Garbage Collection – HVM now holds a maximum of ten completed verifications per migration. After 10, each time a new verification completes, older reports are automatically deleted for storage efficiency.

Notifications – Users now receive UI notifications when a verification starts and when it completes, making it easier to monitor progress without the need for manual checks.

For more information on metadata verification see here.

HVM Metadata Repairs [Preview]

We've introduced a preview version of the Metadata Verification Repair capability, which allows users to trigger a repair against inconsistencies identified during metadata verification.

In this release, the repair functionality focuses on resolving cases where an object is present on the target metastore but does not exist on the source metastore. It achieves this by removing the redundant objects from the target to restore consistency between environments.

This feature is currently in preview and is intended for evaluation and testing only. It should not be used in production environments, as it supports a limited set of repair scenarios. For more information on how to enable and use see here.

Live Ceph

The Ceph filesystem is built on top of Ceph's distributed object store. It is designed to provide high performance, scalability, fault tolerance, and flexibility. Common use cases include cloud infrastructure, data lakes and enterprise storage for AI/ML.

With this release, we welcome support for Ceph filesystems, enabling seamless, zero-downtime data migration to or between Ceph clusters. Real-time replication ensures your target environment stays perfectly in sync, even as users actively interact with the data.

Take a look at Configure a Ceph source for more information.

CDP 7.3.1 Support

Data Migrator now supports Cloudera Data Platform 7.3.1. Please note, Cloudera has introduced Hive capabilities that allow users to transform their tables from Hive tables to Iceberg tables. Data Migrator will not be able to replicate this action from source to target as it is a transformation of format on source. However, Data Migrator will continue to support migrations from Hive to Iceberg targets.

Other Improvements

UI Enhancements

The UI has been enhanced in this release across several key areas. Improvements to the usability of both data and metadata verifications have been introduced. Within bulk actions, we have simplified the reset action so that parallel scan is no longer a separate option. Advances in the display and responsiveness of our user interface provide you with an even better user experience.

Resolved Issues

Data Migrator Core

LM2-9083 Can't add to management group on DM rule create Hive Migrator

LM2-9024 Reset live ADLS2 source migration to one way scan

LM2-8764 Adls2 source overwrite action policy inconsistent directory permissions

LM2-8963 Unable to create GCS source/target filesystem

Hive Migrator

HVM-5761 Liquibase upgrades broken

HVM-5743 Verifications incorrectly report as queued when they are in progress

HVM-5719 Remote agent logging "Vulnerable protobuf generated type in use"

HVM-5704 Drop create bug on remote agents and dataproc

HVM-5687 Remote agent timeouts set non-retryable exceptions, leading to reset

HVM-5665 objectType for materialized view shows as table in verification reports

HVM-5663 Retrieving record counts for files in a batch of partitions in Iceberg migrations can take longer than the READ_TIMEOUT of the LM2 API

HVM-5618 Add object type to verification reports

HVM-5605 Fix CVE-2024-29869

HVM-5528 Hive to Databricks Schema Evolution: Evolving a parquet table by expanding structs and adding fields results in failures on COPY INTO commands

HVM-4915 HVM remote agent unhealthy on first deployment

HVM-4816 Null pointer exception in HVM log when migrating a managed table that already has an "altered add constraint" in its history

UI

ONEUI-8566 Changing RBAC filter silently disables management groups

ONEUI-8565 UI not purging historic data

ONEUI-8542 Verification reports cannot be downloaded if Migration Management Groups are enabled

ONEUI-8541 Notifications for HVM Verifications link to incorrect migration

ONEUI-8531 HVM Verification status only updates on refresh

ONEUI-8527 Instance column on metadata migrations within single instance is redundant

ONEUI-8524 A verification in a failed state causes the instance overview and migration overview not to render

ONEUI-8523 Target Match stat missing from Parallel Scan migrations

ONEUI-8526 Instance name in migration table does not update

ONEUI-8519 Some verification stats are using singular instead of plural

ONEUI-8515 Failed verification reports still offer a download option and error

ONEUI-8003 Inaccurate error message when license has expired

ONEUI-7982 Prompt in the UI RBAC page when LDAP Group Filter is empty

ONEUI-7961 Messaging error when adding target hdfs

ONEUI-7793 Recurring LM2 migration can prevent migrator removal

ONEUI-5969 Migration stats left in incorrect state after migration is reset

CLI

LDMC-731 "hive migration verification list" has two duplicate parameters, making it fail to run

LDMC-715 Command 'hive migration verification delete' confirms deletes which did not succeed

LDMC-701 Verifications: downloaded report is tar, not tar.gz

LDMC-668 LDM installer can't find 'gettext' on RHEL 9