Release notes

Product Version	LiveData UI	LiveData Migrator	Hive Migrator	CLI
3.0	14.0.4	3.0.4	2.3.6	1.22.3

Release Highlights

Verification Improvements and Extension - Repair

In this release, we've enhanced the verification feature to include a repair option. If inconsistencies are found during verification, users can now immediately repair them directly from the CLI. This improvement enables faster and more efficient issue resolution. Additionally, verification performance has been significantly improved with multi-threading, delivering quicker results in this release.

More information on Verification can be found here.

Iceberg Integration Update

In this release we’ve addressed issues when adding columns to partitioned tables, column and table renames, and inconsistent health check status. We’ve also included improvements in handling ORC tables, external tables, and overall performance for migrations to Iceberg on the Watsonx.data platform. For more information on the bugs addressed please refer to the resolved issues below.

More information on the Iceberg metadata agent can be found here.

Secrets Store HashiCorp Updates

This release enhances secrets store integration, enabling configuration of HashiCorp Vault as a property source for sensitive application properties used in the UI and Hive Migrator. Additionally, the UI has been improved to better protect sensitive filesystem configuration, allowing you to reference credentials securely for S3 and ADLS Gen 2 filesystems.

For more information on secrets store integration see here.

Recurring Migrations Scheduling

Users can now create scheduled recurring migrations for their sources. This is best utilized when there is no event feed or the user chooses not to provide a method of event notification. Without an event feed, users can't utilize the "Live" functionality of Data Migrator, but with recurring migrations users can still take advantage of a scheduled scan to migrate changes over time. This feature has extended to GCS, AWS S3, GPFS, and S3 compatible sources.

Find more information on recurring migrations and scheduled recurring migrations here.

RBAC support in API updates

In this release, RBAC is fully supported across all Data Migrator components. Users can now configure migration management groups for their metadata migrations in the API with RBAC. Users can also push existing migration management configurations for their metadata migrations to the API as well. This allows existing users who have configured LDAP with RBAC to easily have the same configurations in the API and ensure consistency. In this release, we have also improved logging by providing an audit trail of users using the API. This allows organizations to easily track product usage for individuals who are utilizing the API for their migrations.

Find more information on API LDAP access control here and activity logs here.

RHEL 9 Support

Data Migrator 3.0 is now supported on RHEL 9.

CDP 7.1.9 Support

Data Migrator 3.0 is now supported on CDP 7.1.9.

Other Improvements

Databricks

Additional properties have been added to allow configuration of concurrent threads to optimize migration performance and control concurrency. More information can be found here.

Metadata Migration

Metadata migration batch size is now configurable via the API. More information can be found here.

Resolved Issues

Data Migrator Core

LM2-8525 In-progress verification set to “complete” when migration is deleted

LM2-8478 User is unable to authenticate if license is expired

LM2-8466 Migration verification is not recorded in LM2 audit logs

LM2-8456 Resetting a data migration will remove the migration management group flag on the migration

LM2-8446 Incorrect behaviour when assigning non existent management groups

LM2-8445 Recurring migrations cause LM2 to fail completely and stop rescanning but appear healthy

LM2-8436 Audit logger is recording unnecessary authentication success in the LM2 DB

LM2-8432 LDM LocalFS implementation should support copying symlinks

LM2-8405 migrationPath and filesytemName are not specified in the UI notification

LM2-8386 MetadataUpdateEvent(s) creating noise in livedata-migrator.log

LM2-8377 Remove excessive logging from RegionDependentTracker

LM2-8372 Talkback trap inconsistent reporting

LM2-8354 Can't delete target FS with migration in UNSCHEDULED state

LM2-8345 Capture Bandwidth Rate Limit in Talkback

LM2-8338 Audit metadata returns incorrect actions

LM2-8330 Skip Action Policy - improve comparing files and directories

LM2-8302 Scanner is stuck at waiting for marker that is never received

LM2-8298 TWO_WAY_SCAN should not enter directories that are regex excluded

LM2-8284 Improve skipIfSizeMatch Action policy to be more efficient

LM2-8281 Reduce RPC calls in LocalFS open/create/mkdir operations

LM2-8262 ContentSummary future is swallowing exceptions silently

LM2-8237 LDM fails to start due to permissions error on audit log

LM2-8235 File System Controller delete by path should be restricted to target only

LM2-8051 AuthenticationController sends incorrect response when Basic Auth configured

LM2-8022 Talkback should check that commands exist before proceeding

LM2-8021 Talkback fails when not running as root

LM2-7181 Talkback should collect verification information

LM2-7103 Talkback should always include threaddumps

Hive Migrator

HVM-5130 Databricks agent can't run blocking task due to AgentMetaException

HVM-5128 NPE thrown in getUpdatedLocations method on local listeners

HVM-5127 Simpler and broad retry mechanism for LM2 rest failures

HVM-5114 When LDAP enabled HVM is unable to authenticate a user if it belongs to a group that isn't part of any role

HVM-5112 Data insertion errors Databricks Agents

HVM-5109 HVM doesn't require restart only for disable/enable MMG

HVM-5099 Incorrect behaviour when assigning non existent management groups

HVM-5098 Audit logger logging events twice

HVM-5094 Migration manager rule unable to create metadata migrations when the migration management group is disabled

HVM-5092 HVM is not retrying as per the configured connectionMaxRetries before failing a LM2 API call

HVM-5090 When login to HVM using CLI, HVM audit log doesn't record the user logged in

HVM-5089 HVM records the wrong role for LDAP users login

HVM-5085 503 causing Databricks ORC fallback transformation

HVM-5083 Databricks exceptions thrown when stopping active migrations

HVM-5077 Scan operations on source placing unnecessary load on metastore

HVM-5076 HVM doesn't insert audit logs in the log file

HVM-5073 Listener event ordering in database drop cascade causing error

HVM-5071 Race condition when a Drop table purge is followed by an Alter table changing the location to the purged location

HVM-5067 Listener operations can be executed out of order causing incorrect metadata on target

HVM-5066 Bad error message in CLI for the LDAP users that doesn't have sufficient authority

HVM-5063 Allow hive migration batch size to be configurable per migration via the API

HVM-5061 Unable to fetch audit logs when security is disabled

HVM-5054 Databricks issue with partition inserts on external tables CSV

HVM-5041 AUDIT_DIR breaks remote server install

HVM-5039 Remote Server upgrades - config file has wrong name / upstart edge case

HVM-5037 LDAP login leaks password to audit log

HVM-5033 Fix “list of one” partition causes stats collection issue

HVM-5019 Start/start all on a migration not assigned to the login user will throw error show the migration

HVM-5011 Databricks unable to migrate table with spark table properties without converting to delta

HVM-5010 Legacy Databricks migration failure seen if the partitioned table exists on the target

HVM-5009 Convert to Delta will create managed Delta tables

HVM-5005 Alter Table Set Location command fails for external tables to Databricks with unity catalog

HVM-4994 Pre-existing metadata objects count resets and counts up when issue is encountered

HVM-4959 Endpoint to fetch all management groups should return the current users groups

HVM-4928 HVM not issuing drop commands for DBFS on a legacy agent

HVM-4883 Hivemigrator stops sending metrics

HVM-4861 user agent is not being recognized by Databricks

HVM-4807 HVM Debian upgrades don't update config

HVM-4648 Migration Audit logs report a successfully applied create event after each failed create event

UI

ONEUI-8247 TokenProvider dumps fingerprint hash to log if there is a fingerprint mismatch

ONEUI-8213 RemoteLoginService login should send unauthenticated requests to log in

ONEUI-8206 Deleting migration management groups while migration management group is disabled across all the components will not update the group list in HVM & LM2

ONEUI-8187 Metadata migration reports live in UI during prolonged initial scan

ONEUI-8181 Date & Time picker is still using standard input styling

ONEUI-8177 Add LDM and HVM UUID to Data migrator overview page

ONEUI-8173 LM2 unhealthy notifications not being generated by UI

ONEUI-8171 No Usage on Daily Breakdown for last day of the month

ONEUI-8160 Metadata rule search bar on migration create takes up full width

ONEUI-8154 Reset button is not removed in reset metadata migrations bulk actions post actioning

ONEUI-8148 Misalignment on API Access Control Page

ONEUI-8145 "Couldn't communicate with Hive Migrator" stays on the target agent page when the agent becomes healthy

ONEUI-8136 Metadata migrations display '100% Complete' in the ready state

ONEUI-8123 Unnecessary WARN message happens in the UI log with couldn't connect to HVM via HTTP while HVM is on HTTPS

ONEUI-8113 LDAP group name in management group is case-sensitive

ONEUI-8109 LDM_MIGRATION_VERIFICATION_SUMMARY sync fails with lots of migrations

ONEUI-8102 Improve metastore agent deletes

ONEUI-8100 Details stored in local storage need to be moved to session storage

ONEUI-8090 Sync issues with management groups in LM2 & HVM

ONEUI-8088 Renew license link opens in new tab/window showing login screen

ONEUI-8064 Iceberg Hive Agent "Metastore URI" default text and description uses inconsistent terminology and port

ONEUI-8063 Iceberg Hive Agent "Configuration Path" description text is incorrect

ONEUI-8061 Reset metadata migrations

ONEUI-8058 LDM backups are duplicated in the UI

ONEUI-8056 Successfully restoring from HVM backup should display correct message

ONEUI-8033 Improve handling of API request timeouts

ONEUI-7958 UI dumps full stacktrace in log when migration path is not found

CLI

LDMC-536 Update metadata migration batch size via the CLI

LDMC-532 Add Migration Stats --watch command to view migration stats

LDMC-528 Typo in verification help message

LDMC-526 Improve logic for handling error responses from HVM

LDMC-517 Consistent deletes for metastores and filesystems