Skip to main content
Version: 3.1.1 (latest)

Configure Iceberg REST Catalog Target

Data Migrator supports metadata migration to Iceberg REST Catalogs. Follow the instructions to create and configure an Iceberg agent to your Iceberg REST Catalog for your migrations with either the UI or CLI.

Add an Iceberg metastore agent on UI​

  1. From the Dashboard, select an instance under Instances.
  2. Under Filesystems & Agents, select Metastore Agents.
  3. Select Connect to Metastore.
  4. Select the filesystem.
  5. Select Iceberg as the Metastore Type.
  6. Enter a Display Name.
  7. Select/confirm REST Catalog as the Catalog Type
  8. Enter the Metastore URI. (ex https://example.iceberg.rest.catalog:8213/iceberg)
  9. Select the Authentication Type: Basic Authentication, API Key Parameters, OAuth, or None.
    1. Basic Authentication: Provide the authentication user and password.
    2. API Key Parameters: Provide key.
    3. Oauth: Provide the OAuth/token server URI connection (ex.oauth.cloud.com/oauth/token), the client ID, and client secret.
      1. Scope: Token access scope on OAuth server.
      2. (Optional) Provide Audience for the token.
      3. (Optional) Provide Resource for the token.
  10. (Optional) Add HTTP Custom Headers for Hive Migrator to communicate with the REST API or proxy.
  11. (Optional) Enter a filesystem URI into Default Filesystem Override to override the default filesystem URI.

Add an Iceberg metastore agent with the CLI​

To add an Iceberg agent with the CLI use the hive agent add iceberg rest CLI command:

Iceberg agent add example
hive agent add iceberg rest -auth-type NONE --custom-headers header=value,header1=value2 --file-system-id storage-target --name no-auth-iceberg --metastore-uri https://iceberg.rest.api/iceberg

Update an existing Iceberg metastore agent with the UI​

  1. From the Dashboard, select an instance under Instances.
  2. Under Filesystems & Agents, select Metastore Agents.
  3. Select your Iceberg agent from the list.
  4. Update your desired parameters on the Metastore Connection.

Any updates to agents will require authentication input to be provided again.

Update an Iceberg metastore agent with the CLI​

To update an Iceberg agent with the CLI, use the hive agent configure iceberg rest CLI command:

Example update of an existing Iceberg agent
hive agent configure iceberg rest --name ice1 --username admin2
info

An Iceberg agent health check status may report incorrectly if updated repeatedly. See the following Known issue for more information.

tip

Remember to define your target filesystem and add any accompanying data migrations for the tables and databases you need to migrate.

Next steps​

If you have already added Metadata Rules, create a Metadata Migration. You can also add metadata rules with the hive rule add CLI command to define the scope then create a metadata migration with hive migration add.