Supported sources and targets
Sources
Select a source below to learn more about which migration types and event notifications it supports.
When you configure Data Migrator to transfer data from a source, you must select one of the following migration types:
One-time
Data Migrator scans the existing source data once and migrates the data to the target. After the data is transferred, the migration is complete and no further changes are migrated.No event stream is required for one-time migrations.
Live
After Data Migrator performs an initial content scan, it moves existing data to the target. Any changes made to the source filesystem are migrated in real time using the notification system defined for this storage.You must have an event stream set up in your environment as detailed below. Expand the sections below to learn more about which event streams work for the sources and Data Migrator.
Recurring
After existing data is moved, the migration scan is repeated to discover new changes. Changes are then migrated to the target.No event stream is required for recurring migrations as Data Migrator performs scans of the source.
Amazon S3
Amazon S3 as a source supports live migrations.
Set up your source bucket to use Simple Queue Service (SQS) to handle event notifications.
See the Amazon documentation on enabling messages to be published to an SQS.
We don't support versioning, metadata migrations, or object locks.
*Supported HDFS target versions. (Other HDFS-compatible environments can be targeted without change. For details, contact Support.)
HDP: 2.6.x, 3.1.
CDH: 5.11, 5.13, 5.14, 5.16, 6.2, 6.3.
CDP: 7.1.4, 7.1.6, 7.1.7, 7.1.8
Azure Data Lake Storage Gen2
*Live migration: Additional third party requirements are necessary. Learn more and contact Support for more information.
*Supported HDFS target versions. (Other HDFS-compatible environments can be targeted without change. For details, contact Support.)
HDP: 2.6.x, 3.1.
CDH: 5.11, 5.13, 5.14, 5.16, 6.2, 6.3.
CDP: 7.1.4, 7.1.6, 7.1.7, 7.1.8
Google Cloud Storage
Google Cloud Storage (GCS) as a source supports one-time migrations.
*Supported HDFS target versions. (Other HDFS-compatible environments can be targeted without change. For details, contact Support.)
HDP: 2.6.x, 3.1.
CDH: 5.11, 5.13, 5.14, 5.16, 6.2, 6.3.
CDP: 7.1.4, 7.1.6, 7.1.7, 7.1.8
Hadoop Distributed Filesystem
Hadoop Distributed Filesystem (HDFS) supports all migration types including live migrations.
Data Migrator reads events from a HDFS cluster's NameNode to track changes to data on the filesystem.
For more information, see Configure your HDFS cluster.
Supported HDFS source versions. (All HDFS versions from Hadoop 2.6 are likely to function without issue. For details on additional support, contact Support.)
HDP: 2.6.3, 2.6.5, 3.1.0
CDH: 5.13, 5.15, 5.16, 6.2, 6.3.
CDP: 7.1.4, 7.1.6, 7.1.7, 7.1.8
*Supported HDFS target versions. (Other HDFS-compatible environments can be targeted without change. For details, contact Support.)
HDP: 2.6.x, 3.1.
CDH: 5.11, 5.13, 5.14, 5.16, 6.2, 6.3.
CDP: 7.1.4, 7.1.6, 7.1.7, 7.1.8
IBM Cloud Object Storage
IBM Cloud Object Storage (IBM COS) supports live migrations.
IBM COS has Apache Kafka event streaming which allows us to handle event notifications.
See the IBM documentation on event streams and how to configure Kafka for Data Migrator in Configure IBM COS as a source.
One-time migration support can be enabled by configuring IBM COS as a generic S3 source filesystem.
*Supported HDFS target versions. (Other HDFS-compatible environments can be targeted without change. For details, contact Support.)
HDP: 2.6.x, 3.1.
CDH: 5.11, 5.13, 5.14, 5.16, 6.2, 6.3.
CDP: 7.1.4, 7.1.6, 7.1.7, 7.1.8
Local Filesystem
A local filesystem is a filesystem mounted on a Linux server on which Data Migrator runs. This can be any filesystem supported by the operating system ideal for migrating small to midsize business data, for example.
Data Migrator must have sufficient privileges to access the filesystem. This often means running as root.
Data Migrator doesn't migrate file permissions and access control lists (ACLs). For example, migrating a network file system version 4 (NFS v4) file share with ACLs to S3 migrates the data and gives control to anyone who has access to the target bucket.
*Supported HDFS target versions. (Other HDFS-compatible environments can be targeted without change. For details, contact Support.)
HDP: 2.6.x, 3.1.
CDH: 5.11, 5.13, 5.14, 5.16, 6.2, 6.3.
CDP: 7.1.4, 7.1.6, 7.1.7, 7.1.8
Network-Attached Storage
Add network-attached storage as a local filesystem.
*Supported HDFS target versions. (Other HDFS-compatible environments can be targeted without change. For details, contact Support.)
HDP: 2.6.x, 3.1.
CDH: 5.11, 5.13, 5.14, 5.16, 6.2, 6.3.
CDP: 7.1.4, 7.1.6, 7.1.7, 7.1.8
S3
Generic S3 covers all S3 sources other than Amazon S3 and IBM COS. This includes cloud storage from other providers, for example, on-premise S3 storage such as MinIO, Dell EMC PowerScale, Scality, and Cloudian.
These providers don't support live migrations yet.
*Supported HDFS target versions. (Other HDFS-compatible environments can be targeted without change. For details, contact Support.)
HDP: 2.6.x, 3.1.
CDH: 5.11, 5.13, 5.14, 5.16, 6.2, 6.3.
CDP: 7.1.4, 7.1.6, 7.1.7, 7.1.8
Metastores
Select a filesystem below to learn which metastores are supported:
Azure Data Lake Storage Gen2
Google Cloud Storage
Hadoop Distributed Filesystem
Supported Hive versions.
Hive 1.1.0, 1.2.1, 2.1.1, 3.1.0, and 3.1.3, as used in the supported Hadoop distributions
| Apache Hive | Databricks* | Snowflake* | AWS Glue | Microsoft Azure SQL Database | Google Cloud Dataproc |
|---|---|---|---|---|---|
| ✅ |