Release notes
Product Version | LiveData UI | LiveData Migrator | Hive Migrator | CLI |
---|---|---|---|---|
2.5.4 | 13.11.4 | 1.28.7 | 2.0.4 | 1.20.4 |
Release Highlights
Live GPFS Source
In this release, we are introducing a new supported source, IBM Spectrum Scale(GPFS). This source is supported for both Live and One-time migrations in both CLI and UI. To utilize the Live capabilities, Kafka must be installed and configured to send notifications on the GPFS paths. Users can now migrate from their GPFS sources to all supported Data Migrator targets. More information on how to configure an IBM Spectrum Scale(GPFS) source can be found here.
RBAC for Data Migrator API
We’ve now introduced support for Role Based Access Control (RBAC) for the API. This will be via the already configured LDAP users for the UI and will support the same roles: admin, read-only, and migration manager. In this release, the API will only be supported for data migrations, not metadata migrations. The support for metadata migrations will come in a future release. More information on how to enable API access control can be found here.
Other Improvements
Databricks Improvements
In this release, we’ve made significant improvements to Databricks and Unity Catalog support. When creating the Databricks agent, there is now a choice of metastore type where the user can choose Unity Catalog or Workspace Hive Metastore agent types. We’ve addressed a variety of bugs to make the support of Unity Catalog more stable. We now allow users to choose whether they want their tables to be managed or external when using the Unity agent on the target. Additionally, we support migrating of managed and external tables as long as they're not transactional. We don't support transactional managed tables yet. More information on Databricks agents can be found here.
License Usage Report Download CLI Support
Users are now able to download the license usage report via the Data Migrator CLI. More information on the license report command can be found here.
Resolved Issues
Data Migrator Core
LM2-8111 Session cookie persists when Basic Auth is enabled
LM2-8080 EventListener can leak resources
LM2-8079 Possible to get migration stuck in SCHEDULED
LM2-8071 Disable session cookie when LDAP is enabled
LM2-8070 LM2 delivers replication factor 1 to HDFS target
LM2-8007 Verifications should be cancelled on restart rather than shutdown
LM2-8001 Log spam - Collecting ActionQueueDiagnostic diagnostics
LM2-7889 LM2 Migration Events fail to process
LM2-7844 No health check for ADLS2 target with 0auth2
LM2-7841 log missing from file tracker log file
Hive Migrator
HVM-4869 Missing property and incorrect default in HVM application.properties
HVM-4818 SERDE properties and table KV pairs should be configurable
HVM-4797 HVM/Remote agent upgrade fails to run successfully
HVM-4788 Column comments are not being migrated when converting to delta
HVM-4711 Remote agent GRPC calls droppage caused by unrelated failure
HVM-4696 HVM upgrade failing due to liquibase change
HVM-4689 Incorrectly formed SQL statement in Databricks SHOW SCHEMAS
HVM-4670 Failed to complete the metadata migration while the jdbc connection is blocked and losing data even after the connection is established
HVM-4650 Recurring data migration on a Delta table's data when a recurrence happens cause triggering the HVM to do copy into on that table again which cause the data to be lost on Databricks
HVM-4647 Live Data Events on Delta Source Tables trigger COPY INTOs
HVM-4642 Not able to migrating tables with time stamp type partitioned column
HVM-4640 Migration level override takes no effect for External Location
HVM-4639 Databricks External Location COPY INTO scope and construction issues
HVM-4621 Checksum calculation between equivalent hive table and delta table fails
HVM-4609 Unrecoverable error occurred: java.lang.StackOverflowError
HVM-4471 Copy Into not with force when migration is restarted when convertToDelta is true
HVM-4046 Remote agent InstanceId file being searched for in wrong directory
HVM-3999 /stats/{hour} endpoint time format doesn't seem to work
UI
ONEUI-7854 Error message when checking metadata rule validation has incorrect message
ONEUI-7842 ActionStore and MigrationAndThroughput Diagnostic calls timeout with 1000 long-running migrations
ONEUI-7837 Clicking on an empty field in line with "Learn More" opens the docs
ONEUI-7833 If the user doesn't have HVM installed or added in the UI, we still can see the metadata section in the side menu
ONEUI-7810 Inconsistent behaviour of required parameters on databricks agent form
ONEUI-7807 External location in the Databricks unity catalog agent in the UI should be empty
ONEUI-7802 Change to Migration Type modifies Skip or Overwrite Settings value
ONEUI-7800 Displaying wrong value for convertToDelta on metadata agent view page
ONEUI-7775 Hive Rule names and migration names containing & or ; can't be deleted
ONEUI-7649 LDMDiagnosticsSyncDataProvider times out with 500 long-running migrations
ONEUI-7593 UI client requests bandwidth graph data when UI in read-only mode
ONEUI-7544 LDMFailedPathsSyncDataProvider fails if migration does not exist on LM2
CLI
LDMC-446 Regression adding hive migration in CLI
LDMC-435 Add the missing fields to the usage report in the CLI to make it consistent with the UI
LDMC-372 Create migration should allow you to create with more than one exclusion
LDMC-347 Limited selection of possible email notification types available for autofill