Skip to main content
Version: 2.4.3

Create a metadata migration

Metadata migrations transfer existing metadata, as well as any subsequent changes made to the source metadata (in the scope of the migration), while Hive Migrator keeps working.

caution

If you're using MariaDB or MySQL, add the JDBC driver to the classpath manually.

info

Ensure migrations exist for the data (databases and tables) you want to migrate, this is not optional for transactional tables which will not be populated on the target unless the migration for the data exists first.

You need both the data and associated metadata before you can successfully run queries on migrated databases.

Create a metadata migration with the UI​

tip

Before creating your metadata migration, create a metadata rule to define it's scope.

  1. From your Dashboard, select the instance under Instances.
  2. Under Migrations, select Metadata Migrations.
  3. Select Create metadata migration.
  4. Under Migration Name, enter a name for this migration.
  5. Under Source, select a source Metadata Agent.
  6. Under Target, select a target Metadata Agent.
If a Databricks Unity Catalog Metastore Agent is selected as target

Use the options under Target Agent Configuration Overrides to override your Databricks target agent configuration for this migration.

  • Catalog: Enter the name of your Databricks Unity Catalog.
  • External Location: Specify the external location by appending or adjusting the pre-populated URI.
  • Delete after conversion: Select to delete raw data after it has been converted to Delta format and migrated to Databricks.
    info

    Only use this option if you're performing one-time migrations for the underlying table data. The Databricks agent doesn't support continuous (live) updates of table data if you're converting to Delta Lake in Databricks.

If a Databricks Workspace Hive Metastore (Legacy) Metastore Agent is selected as target

Use the options under Target Agent Configuration Overrides to override your Databricks target agent configuration for this migration.

  • Convert to delta format: Select to convert your tables to Delta Lake format after migrating to Databricks.

  • Delete after conversion: Select to delete the underlying table data and metadata from the Filesystem Mount Point location after it has been converted to Delta Lake in Databricks.

    info

    Only use this option if you're performing one-time migrations for the underlying table data. The Databricks agent doesn't support continuous (live) updates of table data if you're converting to Delta Lake in Databricks.

  • Filesysytem Mount Point: The filesystem that contains your data you want to migrate must be mounted onto your DBFS.
    Enter the mounted container's path on the DBFS.

  • (Optional) - Enter path for Default Filesystem Override.

    • If you select Convert to Delta Lake , enter the location on the DBFS to store the tables converted to Delta Lake. To store Delta Lake tables on cloud storage, enter the path to the mount point and the path on the cloud storage.

      Example: Location on the DBFS to store tables converted to Delta Lake
      dbfs:<location>
      Example: Cloud storage location
      dbfs:/mnt/adls2/storage_account/

    • If you don't select Convert to Delta, leave blank or a enter a file system mount point.

      Example: Filesystem mount point
      dbfs:<value of Fs mount point>
  1. Under Metadata rules, select a metadata rule to define the scope of the migration.
  2. Select Start migration automatically to start the migration automatically or leave clear to start manually after creation.
  3. Select Create to create the metadata migration.
info

Metrics shown on the metadata migration content summary page don’t show correct results following certain metadata migration failures. See the Known issue for more information.

Create a metadata migration with the CLI​

Migrate metadata from your source metastore to a target metastore using the hive migration command.

Define the source and target using the hive agent names detailed in the Connect to metastores section, and apply the hive rule names to the migration.

Follow the command links to learn how to set the parameters and see examples.

  1. Create a new metadata migration:

    hive migration add

    Apply the --auto-start parameter if you would like the migration to start right away.

  2. If you don't have auto-start enabled, manually start the migration:

    hive migration start