Release notes

Product Version	LiveData UI	LiveData Migrator	Hive Migrator	CLI
2.4.3	13.7.2	1.27.3	1.18.3	1.18.2

Release Highlights

Target Match

Target Match is now available in the UI and provides the additional option for migrations to scan both source and target to not only add but also remove any extraneous files. Data Migrator provides migration capabilities and utilization of filesystem events to ensure changes made on source are also made on target. However, situations can occur such as network failures, maintenance periods, unexpected changes, or even lack of confidence in the consistency of data. With Target Match, discrepancies (i.e. extra files or missing files) by such events will be resolved alongside the migration to ensure the source and target are 100% identical. More information can be found here.

Live Catalog

This release is also introducing Live Catalog. This feature extends Data Migrator’s metadata functionality by providing additional flexibility in managing metadata migration rules. Previously, rules that define what metadata is migrated can only be created when creating a metadata migration. With Live Catalog, the definition of what metadata is migrated can now be created separately. This allows the association of the definitions or "rules" for the metadata migration to be better managed by the user. More information can be found here.

Other Improvements

Recurring Migration for ADLS Gen2 Source

For ADLS Gen2 sources, users now have the additional migration type "Recurring". This can be used similar to One-time migrations, when there is no event feed for the source Gen2 storage. Existing data on the source will be moved to the target, then the migration scan will be repeated to discover and migrate new changes after a user defined set of time.

License Usage Metrics

Updates and new metrics in the usage to reflect data migration have been made in this release. The UI will now display the lifetime usage of the data migrations.

UI Updates

In this release a variety of UI updates have been made to improve the consistency of Cirata branding and experience. This includes some updates in the page layouts, icons, and language.

Resolved Issues

Data Migrator Core

LM2-7802 ADLS as source does not support recurring migrations

LM2-7656 SyncMetadataAction permission action is performed on targetPath instead of source path

LM2-7639 Excluding ApiSchemaVersion is deprecated

LM2-7627 Exception when deleting an adls2 source migration

LM2-7616 Changing time intervals for recurring migrations do not take effect when target is unavailable

LM2-7574 Lack of logging with path conflicts in S3 source

LM2-7573 Migration Progress details slow to load with ADLS2 live source

LM2-7572 Talkback script exits when "livedata-migrator" is in the UI's JVM args

LM2-7571 AbandonWindowException refers to source filesystem by default even when target filesystem is impacted

LM2-7552 update user experience when creating data migrations with S3 filesystem using JCKES as Authentication method

LM2-7534 Adls2 source path ACL changes ignored in event stream

LM2-7529 Inconsistencies in migrating data from GPFS source to target

LM2-7524 Running Migration limit to be reverted to default of 10

LM2-7522 Talkback snapshots reduce the frequency to 12 hours and the directory Path requires updated

LM2-7513 SysInfo fails with specific configurations

LM2-7491 INFO level messages in LM2 logs showing in ERROR level incorrectly

LM2-7487 Remove diagnostics from TalkbackAPIDataWriter and talkback.sh

LM2-7475 Migrations do not retry or recover if there is an interruption to the target connection

LM2-7473 TalkbackAPIDataWriter is inefficiently using memory

LM2-7472 Incorrect notification sent on entering the grace period of a usage based license

LM2-7469 Only one ADLS2 source can be added to an instance of LDM

LM2-7467 Inaccurate count of pre-existing files when ALDS2 live is the source

LM2-7466 Directory rename not picked up in ALDS2 live migration with skip if size match

LM2-7454 Creating migration with a file path on a single file is causing LDM and UI errors

LM2-7441 LM2 doesn't bind to port 18080 following connection issue with S3 target, cannot be restored

LM2-7434 Excessive Logging when scanning

LM2-7373 Source Cleanup Constants still present in LDM core

LM2-7366 Possible for DTAManager to throw OOB exceptions

LM2-7349 /fs/{role}/{fsId}/listPaths endpoint not handling timestamps correctly

LM2-7328 Error around /etc/xinetd.d/* in output of sysinfo.sh

LM2-7327 Issue with loop over /var/log/sa/ in sysinfo.sh script

LM2-7326 Two way scan path deletions should contribute to scan pending items

LM2-7299 Upgrade has potential to loose events on already running migrations

LM2-7293 MigrationScheduler does not take state into account before queueing migrations

LM2-7282 Appears to be possible to resume/start a migration in a STOPPING state

LM2-7257 Update Guava to resolve CVE-2023-2976

LM2-7256 Update SpringBoot library to resolve vulnerabilities

LM2-7253 Prometheus Metrics endpoint returns zero for all metrics

LM2-7250 Fix lock ordering issues with fsLockManager

LM2-7231 ADLS2 smaller buffer optimisation

LM2-7212 Typo in talkback

LM2-7194 Canceled verifications should not require restart

LM2-7189 Modifying a file that is part of a migration to ADLS2 results in failures

LM2-7182 INotifyEventSourceIsNotAvailableError repeated constantly in notifications

LM2-7178 Verification Report is becoming stuck and not able to cancel

LM2-7164 Create and remove DTA BandwidthPolicy entity

LM2-7139 Items not migrated due to default exclusions should not be reported as discrepancies on the verification report

LM2-7132 Excessive migrations in RUNNING state

LM2-7110 Scheduled migrations are not started by MigrationStarter after restart

LM2-7109 Inherent Deadlock in UniquePriorityBlockingQueue

LM2-7104 Confusing log messages - "Transferring removal failed"

LM2-7102 Should not log stack trace for file not found

LM2-7101 Verification Report is referencing ".livemigrator" folder

LM2-7100 Object reference rather than Object printed in log message

LM2-7073 Able to add unhealthy DTA to the LDM, as a result unable to remove the DTA or migrate data

LM2-7072 LDM can leave actions from deleted/resetted migrations in the DB

LM2-7060 Hadoop properties over 4KiB get truncated when decrypted

LM2-7049 Can't upgrade DTA from 1.23.4.1-4147 to 1.25.0-4586

LM2-7012 event-logger.log is created but never written to

LM2-6896 Improve DTA notification behaviour

LM2-6885 LDM not working with 16GB Heap

LM2-6645 Incorrect usage @Async in FileSystemHealthManager

LM2-7890 Scanner stopped working, falling behind event stream MissingEventsException

Hive Migrator

HVM-4644 Starting a new metadata migration triggers retries in an unrelated, pre-existing metadata migration

HVM-4636 HVM failures spamming the UI Notifications repeatedly

HVM-4635 Fix file paths for databricks legacy copy into

HVM-4623 Restarting convert to delta databricks migration causes truncate on target

HVM-4619 HVM stops migration after 25secs if it can't reach LDM

HVM-4590 While the HVM loses the connection or blocked the connection to its target, won't migrate the metadata and will ignore the metadata that were missed after the blockage is removed

HVM-4489 Duplicated Not Null constraints after migration if the target table already has it on the same column

HVM-4479 Cleanup filetracking on migration reset

HVM-4472 Correct logic in Datalake Facade

HVM-4463 Connection pool limit being reached when moving partitioned tables to databricks and snowflake

HVM-4455 Migration failure error results in whole table schema for each partition in migration audit log

HVM-4453 Copy into statements contain full partition in Migration audit logs to Databricks

HVM-4422 Data lost on partitioned tables where a column rename occurs

HVM-4393 Hive source delta tables won't migrate to Databricks unity catalog successfully

HVM-4318 Can't Parse Incoming payload Error for Txn_id

HVM-4092 HVM Live Migration : Exception generated during Live HVM migration (Hive -> Hive)

UI

ONEUI-7816 Rescan with skip if the size match will not report the files and directories removed

ONEUI-7746 Hive Migration is deleted but remains in DB

ONEUI-7723 Sync of notifications from multiple HVM instances can fail

ONEUI-7691 Mark 'Migration Type' as a required field

ONEUI-7688 Cannot untick 'Override JDBC Connection Properties' in hive agents on the UI

ONEUI-7648 LDMNotificationSyncDataProcessor and LDMStorageSyncDataProcessor use incorrect locking strategy

ONEUI-7566 NPE in sync service tracker

ONEUI-7332 Remove persisted 'Migration Errors Report' following migration reset

ONEUI-7299 Improve the notification for the metadata agent error

ONEUI-7244 HelperText on Target FS Form incorrectly mentions migrating data

ONEUI-7049 "Resource unavailable" page seen for a split second when clicking on the migration

CLI

LDMC-400 Add Path mapping CLI command missing source FS option