Skip to main content
Version: 2.6 (latest)

Release notes

Product VersionLiveData UILiveData MigratorHive MigratorCLI
2.613.17.21.29.42.2.41.21.2

Release Highlights​

Databricks Generally Available​

In this release, Databricks is now generally available with the support of Unity Catalog and a variety of improvements addressed. More information on the Databricks metastore agent can be found here.

Secrets Store, HashiCorp Support​

Data Migrator now supports integration with the HashiCorp Vault secrets management system, allowing you to protect specific sensitive filesystem configuration values and sensitive application configuration properties reducing the risk of unauthorized access and enhancing overall security. More information on Secrets stores and HashiCorp Vault can be found here.

RBAC support in API for metadata migrations​

API access control has been extended to provide LDAP authentication to the Hive Migrator API. More information on API Access Control can be found here.

Other Improvements​

Iceberg Support​

In this release, we are introducing support for Iceberg provided by the IBM Watsonx.data platform. We provide the additional selection option in the agent drop-down and configuration to connect to an Iceberg target utilizing the Apache Hive catalog and S3a filesystem. Data Migrator then migrates and converts the metadata to the Apache Iceberg table format. More information on the Iceberg agent can be found here.

Resolved Issues​

Data Migrator Core​

LM2-8223 Warning message for relative path in absolute URI

LM2-8195 IBMCos does not check kafka details on fs update

LM2-8169 Remove the reference to P12 file for the API commands

LM2-8163 Separate application.properties between JAR and installer

LM2-8160 Persist LOST region

LM2-8152 Remove attempts at actioning CREATE event on file

LM2-8140 EventListener Abort migration action should not effect scan only migrations

LM2-8138 Recurring Migration cannot recover from Auto Stop

LM2-8135 Inconsistency due to failure to add failed path to LOST set

LM2-8123 Files with colons in the name do not work with LFS

LM2-8116 Exclude access time (atime) events from replication by default

LM2-8102 Always recreate the marker file, so we can't miss messages from the event stream, that cause migrations to get stuck in SCHEDULED

LM2-8064 Conflicting property names for RBAC defaultToReadOnly

LM2-8061 Swagger API doesn't work through /ldm-api.html

LM2-8037 Kafka Consumer not always shutdown correctly

LM2-8034 Pushing the LDAP configs to the LM2 will print out unsupported LDAP properties in the logs

LM2-8030 "\" in the Administrator Bind Username pushed from the UI is not read correctly in LM2

LM2-8029 Hive migrations stop and can't be restarted after applying LDAP & Access control in the UI

LM2-7996 After stopping and deleting a migration with GPFS live source, SOMETIMES the UI throws an error which mis lead the user

LM2-7611 Possible to have internal path mappings on TWO_WAY_SCAN migration

LM2-7097 Should not require Schema header

LM2-3741 Put the OOM errors in the /var/log/wandisco/component directories

Hive Migrator​

HVM-4944 Liquibase changeset invalid checksums cause the upgrade to fail

HVM-4940 Look into remote agent failures with transactional tables

HVM-4924 Single incompatible table causes DOS

HVM-4881 Fix broken Datalake Insert

HVM-4869 Missing property and incorrect default in HVM application.properties

HVM-4831 Hive CDP Source agent can fail if hive-site has JDBC connection details

UI​

ONEUI-8059 Client creates a new websocket every time you navigate around the app

ONEUI-8050 Auto starting a HVM metadata migration is causing an internal server error

ONEUI-8020 HVM rule list is slow to update after creating a migration

ONEUI-7979 With the LDAP enabled on the LM2, the user can't download the migration verification reports from the UI

ONEUI-7973 Target Match is enabled by default for non-live ADLSgen2 source

ONEUI-7959 LDMStorageResource logs requests for migration file paths at INFO level

ONEUI-7955 Migration Manager can't access LM2 with basic auth enabled

ONEUI-7939 Duplicate migration error is displayed incorrectly

ONEUI-7938 NPE on creation of UC non-convert external metadata migration

ONEUI-7937 "bad credentials" INFO level log in the LM2 can be ignored when login to the UI using the root credentials

ONEUI-7930 Miscounting Verification Report Exclusion files on UI

ONEUI-7914 If the source is GPFS live, remove the GPFS option to add more GPFS filesystem from the UI options

ONEUI-7894 Incorrect error msg when deleting a metadata agent

ONEUI-7830 Internal Server Error in the UI

CLI​

LDMC-475 After LDAP is enabled in the LM2, the CLI is throwing a NPE error

Configuration property changes​

/etc/wandisco/hivemigrator/application.properties​

micronaut.security.intercept-url-map[1].access=isAuthenticated() has been replaced with micronaut.security.intercept-url-map[1].access=isAnonymous()

micronaut.security.intercept-url-map[2].access=isAuthenticated() has been replaced with micronaut.security.intercept-url-map[2].access=isAnonymous()

/etc/wandisco/hivemigrator/vars.sh​

HIVE_MIGRATOR_SERVER_USER=hive has been removed.

HIVE_MIGRATOR_SERVER_GROUP=hadoop has been removed.

/etc/wandisco/livedata-migrator/application.properties​

gcs.fs.type.masked.properties=fs.gs.auth.service.account.private.key.id,fs.gs.auth.service.account.private.key,privateKey,privateKeyId,jsonKeyFile,p12KeyFile has been replaced with gcs.fs.type.masked.properties=fs.gs.auth.service.account.private.key.id,fs.gs.auth.service.account.private.key,privateKey,privateKeyId,jsonKeyFile

/etc/wandisco/ui/application-prod.properties​

spring.dataSource.password=ENC(yLeo5UlhUisq6AwywfI8MrDMwjKeiYByT3wzRi1HUwI\=) has been replaced with spring.dataSource.password=ENC(x2D7SNYInWEefzu6DLZa8rr1YLRwcvMao1EtHBRCz+4\=)