Skip to main content
Version: 3.1.1 (latest)

Release notes

Product VersionLiveData UILiveData MigratorHive MigratorCLI
3.1.114.6.53.1.32.5.62.0.4

Release Highlights​

Iceberg REST Catalog Support​

In Iceberg, the REST Catalog provides a standalone, lightweight solution that allows for metadata management through a simple API. Since the REST catalog is decoupled from the underlying storage and compute layer, support for this catalog increases flexibility and interoperability across diverse systems. In this release, we are expanding our Iceberg capabilities by introducing support for the REST catalog. Data Migrator can now migrate tables from the Hive Metastore into any Iceberg Catalog that provides a REST API.

More information on how to configure REST API Iceberg Agent here.

Parallel Scan​

Scanning files and directories is central to the data migration process. Challenges with this scanning include where there are large filesystems, with high latency, and significant client activity. To meet these challenges we have completely re-engineered our scanning approach and would like to introduce what we call Parallel Scan.

In this release we have an early version of our novel Parallel Scan approach, which can be enabled as a beta feature. We do not recommend this yet for production use cases however it can be utilized in your other environments. Parallel Scan will improve scanning performance, reduce your migration times and enhance the management of your system resources. We continue to build out this feature and more will follow in future releases.

The Parallel Scan beta feature is exclusively available to customers who are members of our innovation board for early testing only. Please consult with support for more information.

Checksum Matching​

The ability to check that data is consistent between source and target filesystems is critical for any data migration. This verification has up until now used the name of the file or directory, and the file size to determine consistency.

This release introduces an optional, additional feature leveraging a checksum approach for supported filesystems. There is now the option to use skip if checksums match, which is of course a stricter condition than simply skip if the size of the file matches. If this new feature is enabled for a migration, any verification will also confirm if the checksums are consistent.

Users must carefully consider whether to enable this stronger consistency check for their data. As with any additional benefit, depending on your data and filesystems, there will be costs in terms of processing overhead and impact on license usage. The clear benefits are those of stricter control and therefore reduced risk.

For more information on Checksum Matching see here.

For migration between local filesystems, the default behaviour is for symbolic links (symlinks) to be resolved, and linked data to be migrated as a copy to the target. By default, broken links that do not resolve are ignored.

A new feature has now been introduced which allows symlinks to be migrated to the target without resolving the link, so that even if the link is broken it will not be followed, and only the link itself will be copied to the target regardless of any data which that link may reference. Therefore with this feature enabled for local filesystems, all symlinks can be migrated to the target.

More information on configuring this feature can be found here.

Other Improvements​

New UI graph for data migrated per day​

Users can now see the amount of data migrated per day for each migration on the UI. This provides a more granular view of what is happening per individual migration and allows users to visualise and identify patterns and anomalies in their migration behaviour.

Multiple local filesystems as source​

Users will now be able to add more than one local filesystem sources on the UI or CLI with a single instance of Data Migrator. For more information on how to add source local filesystems see here.

Resolved Issues​

Data Migrator Core​

LM2-8470 Inconsistencies in error message to migration managers

LM2-8474 Restrict migration managers from setting migration priority

LM2-8489 Add defensive check for policy in test scenario

LM2-8490 Handle test scenario for non CRON migration

LM2-8498 'filesystem update gcs' command requires unnecessary parameter

LM2-8501 ChecksumActionPolicy - file re-migrated on fresh install

LM2-8505 Inaccurate recurring scans fields

LM2-8507 Verification output doesn't record finishedAt

LM2-8508 Verification condensed reports subPaths is double the value it should be

LM2-8512 Add new management group update endpoint

LM2-8522 Migration verification repair re-migrates root directory

LM2-8529 HDFS filesystem wrapper throws exception if no checksum

LM2-8551 Check initial paths for Verification

LM2-8566 Track per Migration Latency metrics

LM2-8573 Increase the allowed verification paths limit

LM2-8577 Track bytes and files transferred per migration in Micrometer

LM2-8584 Migration Management issue with usernames in group search filters

LM2-8589 LocalFS symlink replication mode should also apply to content summary

LM2-8592 Inconsistent error message reported with empty values for flag on ADLS2

LM2-8597 License usage and stats fail to update when using local filesystem

LM2-8651 Verification to use default lister

Hive Migrator​

HVM-4397 Improve HVM tolerance of failed license checks

HVM-4989 Correct the log entry when we convert to delta on the migration level

HVM-5102 Disable LDAP in the UI, then disable LDAP on LM2 & HVM via API

HVM-5126 Add new management group update endpoint

HVM-5136 Correct behaviour on target when dropping and renaming table

HVM-5140 Closed client pool preventing table create events on Iceberg

HVM-5149 Limitations of migrating timestamp partitions to Iceberg

HVM-5152 External partitioned tables fail to populate on Iceberg following rename

HVM-5156 Resetting metadata migrations to Iceberg cause migration errors

HVM-5157 Hive : Databricks : Rule match issue

HVM-5173 Migration Management fix for usernames in group search filters

HVM-5190 Prevent updates to metadata migrations outside my group

HVM-5191 HVM fails to start with exception

UI​

ONEUI-7624 Update daily usage graph even if there is no gateway connection

ONEUI-8170 Failed path provider times-out with many migrations

ONEUI-8190 Configurable batch size per migration

ONEUI-8196 API version of the newest LM2 component in the notification

ONEUI-8212 If only one instance, select that instance by default and disable input

ONEUI-8228 Recurrence history table is not updated after a new recurrence

ONEUI-8231 Display of hours/minutes field within recurring migration settings

ONEUI-8232 Update text colour in set recurring schedule helper text

ONEUI-8261 Improve logic to determine a recurring migration

ONEUI-8266 Link misalignment on Register page

ONEUI-8277 For ADLS2 LIVE source, align UI to CLI for creation of non-live migration

ONEUI-8280 Limit display of number of paths for verification

ONEUI-8283 Migration Management fix for usernames in group search filters

ONEUI-8347 Display all days of the selected month on the migration usage graph

CLI​

LDMC-415 Don't default action policy on reset

LDMC-534 Update data migration management group via CLI

LDMC-535 Update meta data migration management group via CLI

LDMC-546 Can't upload license via CLI because license is invalid

LDMC-547 HVM should be authenticating with LDAP

LDMC-552 Cannot add GCS target

LDMC-567 Migration Stats enhancements

LDMC-568 LDM File Transfers enhancements

LDMC-599 Remove warning for scanOnly flag with a LIVE ADLS2 source

LDMC-602 Mask Iceberg sensitive parameters in command history