Skip to main content
Version: 2.4.3

Supported sources and targets

Sources

Select a source below to learn more about which migration types and event notifications it supports.

When you configure Data Migrator to transfer data from a source, you must select one of the following migration types:

  • One-time
    Data Migrator scans the existing source data once and migrates the data to the target. After the data is transferred, the migration is complete and no further changes are migrated.

    No event stream is required for one-time migrations.

  • Live
    After Data Migrator performs an initial content scan, it moves existing data to the target. Any changes made to the source filesystem are migrated in real time using the notification system defined for this storage.

    You must have an event stream set up in your environment as detailed below. Expand the sections below to learn more about which event streams work for the sources and Data Migrator.

  • Recurring
    After existing data is moved, the migration scan is repeated to discover new changes. Changes are then migrated to the target.

    No event stream is required for recurring migrations as Data Migrator performs scans of the source.

Amazon S3

Amazon S3 as a source supports live migrations.

Set up your source bucket to use Simple Queue Service (SQS) to handle event notifications.

See the Amazon documentation on enabling messages to be published to an SQS.

We don't support versioning, metadata migrations, or object locks.

*Supported HDFS target versions. (Other HDFS-compatible environments can be targeted without change. For details, contact Support.)

HDP: 2.6.x, 3.1.
CDH: 5.11, 5.13, 5.14, 5.16, 6.2, 6.3.
CDP: 7.1.4, 7.1.6, 7.1.7, 7.1.8
TargetOne-time migrationLive migrationRecurring migration
Alibaba Cloud Object Storage Service
Amazon S3
Azure Data Lake Storage Gen2
Google Cloud Storage
Hadoop Distributed File System*
IBM Cloud Object Storage
Local Filesystem
Oracle Object Storage
S3
Azure Data Lake Storage Gen2

*Live migration: Additional third party requirements are necessary. Learn more and contact Support for more information.

*Supported HDFS target versions. (Other HDFS-compatible environments can be targeted without change. For details, contact Support.)

HDP: 2.6.x, 3.1.
CDH: 5.11, 5.13, 5.14, 5.16, 6.2, 6.3.
CDP: 7.1.4, 7.1.6, 7.1.7, 7.1.8
TargetOne-time migrationLive migration*Recurring migration
Alibaba Cloud Object Storage Service
Amazon S3
Azure Data Lake Storage Gen2
Google Cloud Storage
Hadoop Distributed File System*
IBM Cloud Object Storage
Local Filesystem
Oracle Object Storage
S3
Google Cloud Storage

Google Cloud Storage (GCS) as a source supports one-time migrations.

*Supported HDFS target versions. (Other HDFS-compatible environments can be targeted without change. For details, contact Support.)

HDP: 2.6.x, 3.1.
CDH: 5.11, 5.13, 5.14, 5.16, 6.2, 6.3.
CDP: 7.1.4, 7.1.6, 7.1.7, 7.1.8
TargetOne-time migrationLive migrationRecurring migration
Alibaba Cloud Object Storage Service
Amazon S3
Azure Data Lake Storage Gen2
Google Cloud Storage
Hadoop Distributed File System*
IBM Cloud Object Storage
Local Filesystem
Oracle Object Storage
S3
Hadoop Distributed Filesystem

Hadoop Distributed Filesystem (HDFS) supports all migration types including live migrations.

Data Migrator reads events from a HDFS cluster's NameNode to track changes to data on the filesystem.

For more information, see Configure your HDFS cluster.

Supported HDFS source versions. (All HDFS versions from Hadoop 2.6 are likely to function without issue. For details on additional support, contact Support.)

HDP: 2.6.3, 2.6.5, 3.1.0
CDH: 5.13, 5.15, 5.16, 6.2, 6.3.
CDP: 7.1.4, 7.1.6, 7.1.7, 7.1.8

*Supported HDFS target versions. (Other HDFS-compatible environments can be targeted without change. For details, contact Support.)

HDP: 2.6.x, 3.1.
CDH: 5.11, 5.13, 5.14, 5.16, 6.2, 6.3.
CDP: 7.1.4, 7.1.6, 7.1.7, 7.1.8
TargetOne-time migrationLive migrationRecurring migration
Alibaba Cloud Object Storage Service
Amazon S3
Azure Data Lake Storage Gen2
Google Cloud Storage
Hadoop Distributed File System*
IBM Cloud Object Storage
Local Filesystem
Oracle Object Storage
S3
IBM Cloud Object Storage

IBM Cloud Object Storage (IBM COS) supports live migrations.

IBM COS has Apache Kafka event streaming which allows us to handle event notifications.

See the IBM documentation on event streams and how to configure Kafka for Data Migrator in Configure IBM COS as a source.

One-time migration support can be enabled by configuring IBM COS as a generic S3 source filesystem.

*Supported HDFS target versions. (Other HDFS-compatible environments can be targeted without change. For details, contact Support.)

HDP: 2.6.x, 3.1.
CDH: 5.11, 5.13, 5.14, 5.16, 6.2, 6.3.
CDP: 7.1.4, 7.1.6, 7.1.7, 7.1.8
TargetOne-time migrationLive migrationRecurring migration
Alibaba Cloud Object Storage Service
Amazon S3
Azure Data Lake Storage Gen2
Google Cloud Storage
Hadoop Distributed File System*
IBM Cloud Object Storage
Local Filesystem
Oracle Object Storage
S3
Local Filesystem

A local filesystem is a filesystem mounted on a Linux server on which Data Migrator runs. This can be any filesystem supported by the operating system ideal for migrating small to midsize business data, for example.

Data Migrator must have sufficient privileges to access the filesystem. This often means running as root.

Data Migrator doesn't migrate file permissions and access control lists (ACLs). For example, migrating a network file system version 4 (NFS v4) file share with ACLs to S3 migrates the data and gives control to anyone who has access to the target bucket.

*Supported HDFS target versions. (Other HDFS-compatible environments can be targeted without change. For details, contact Support.)

HDP: 2.6.x, 3.1.
CDH: 5.11, 5.13, 5.14, 5.16, 6.2, 6.3.
CDP: 7.1.4, 7.1.6, 7.1.7, 7.1.8
TargetOne-time migrationLive migrationRecurring migration
Alibaba Cloud Object Storage Service
Amazon S3
Azure Data Lake Storage Gen2
Google Cloud Storage
Hadoop Distributed File System*
IBM Cloud Object Storage
Local Filesystem
Oracle Object Storage
S3
Network-Attached Storage

Add network-attached storage as a local filesystem.

*Supported HDFS target versions. (Other HDFS-compatible environments can be targeted without change. For details, contact Support.)

HDP: 2.6.x, 3.1.
CDH: 5.11, 5.13, 5.14, 5.16, 6.2, 6.3.
CDP: 7.1.4, 7.1.6, 7.1.7, 7.1.8
TargetOne-time migrationLive migrationRecurring migration
Alibaba Cloud Object Storage Service
Amazon S3
Azure Data Lake Storage Gen2
Google Cloud Storage
Hadoop Distributed File System*
IBM Cloud Object Storage
Local Filesystem
Oracle Object Storage
S3
S3

Generic S3 covers all S3 sources other than Amazon S3 and IBM COS. This includes cloud storage from other providers, for example, on-premise S3 storage such as MinIO, Dell EMC PowerScale, Scality, and Cloudian.

These providers don't support live migrations yet.

*Supported HDFS target versions. (Other HDFS-compatible environments can be targeted without change. For details, contact Support.)

HDP: 2.6.x, 3.1.
CDH: 5.11, 5.13, 5.14, 5.16, 6.2, 6.3.
CDP: 7.1.4, 7.1.6, 7.1.7, 7.1.8
TargetOne-time migrationLive migrationRecurring migration
Alibaba Cloud Object Storage Service
Amazon S3
Azure Data Lake Storage Gen2
Google Cloud Storage
Hadoop Distributed File System*
IBM Cloud Object Storage
Local Filesystem
Oracle Object Storage
S3

Metastores

Select a filesystem below to learn which metastores are supported:

Azure Data Lake Storage Gen2
Google Cloud Storage
Hadoop Distributed Filesystem
* = In preview.
Supported Hive versions.
Hive 1.1.0, 1.2.1, 2.1.1, 3.1.0, and 3.1.3, as used in the supported Hadoop distributions
Apache HiveDatabricks*Snowflake*AWS GlueMicrosoft Azure SQL DatabaseGoogle Cloud Dataproc
S3