Configure a HDFS source

You can migrate data from a Hadoop Distributed File System (HDFS) by configuring it as your source filesystem for Data Migrator. After you install Data Migrator to a HDFS, Data Migrator automatically configures it as a source for data migrations. However, if you have Kerberos enabled or if your Hadoop configuration isn't in a default location, you'll have to manually configure your HDFS source using the steps below.

You can also use these steps to set up a custom HDFS source instead of the default.

note

If you deleted the HDFS source that Data Migrator detected automatically, and you want to detect it again, run the command filesystem auto-discover-source hdfs in the CLI. For more information about this CLI command, see the command reference.

Prerequisites

You need the following:

• An HDFS cluster running Hadoop 2.6 or above.
• If Kerberos is enabled on your filesystem, a valid keytab containing a suitable principal for the HDFS superuser must be available on the Linux host.
• Oracle Big Data Services (BDS) - If running with Oracle's Distribution of Apache Hadoop (ODH), Data Migrator must provide fully qualified hostnames for DNS. Ensure that the following configuration property overrides are added for the source filesystem:
  Configuration property overrides are an option under Additional Configuration on the Filesystem Configuration screen.

  dfs.client.use.datanode.hostname=true
  dfs.datanode.use.datanode.hostname=true

Configure a HDFS source filesystem with the UI

1. Connect to the UI.

2. From the Dashboard, select an instance under Instances.

3. In the Filesystems & Agents menu, select Filesystems.

4. Select Add source filesystem

5. Select Hadoop Distributed File System (HDFS) from the Filesystem Type dropdown list.

6. Enter the following details:

   • Display Name - Enter a name for your source filesystem.
   • Default Filesystem - Enter the default filesystem property value (fs.defaultFS) from your HDFS configuration. For example, hdfs://examplenameservice1 or hdfs://node1.example.com:8020.
   • Kerberos Configuration
     • Kerberos Principal - Enter a principal that will map to the HDFS superuser using auth_to_local rules, or add the Data Migrator user principal to the superuser group on the Hadoop cluster you're using.
       • For example: Create the Kerberos principal ldmuser@realm.com. Using auth_to_local rules, ensure the principal maps to the user hdfs, or that the user ldmuser is added to the superuser group.
     • Kerberos Keytab Location - Enter the path to the Kerberos keytab file containing the Kerberos Principal. The keytab file must be accessible to the local system user running the Data Migrator service (the default is hdfs) and must be accessible from the edge node where Data Migrator is installed.
       • For example: Copy the ldmuser.keytab file (where ldmuser is your intended user) containing the Kerberos principal into the /etc/security/keytabs/ directory on the edge node running Data Migrator, make its permissions accessible to the HDFS user running Data Migrator, and enter the /etc/security/keytabs/ldmuser.keytab path during Kerberos configuration for the filesystem.
   • Additional Configuration - Enter override properties or enter additional properties by adding key/value pairs.
     • Configuration Property File Paths - Enter the directory or directories containing your HDFS configuration (such as the core-site.xml and hdfs-site.xml) on your Data Migrator host's local filesystem. This is required if you have Kerberos or a High Availability (HA) HDFS.
       note

       Data Migrator reads core-site.xml and hdfs-site.xml once, during filesystem creation, applying any configuration within paths added under Configuration Property File Paths. After creation, these paths are no longer visible in the UI. You can see all filesystem properties using the API.

     • Configuration Property Overrides (Optional) - Enter override properties or additional properties for your HDFS filesystem by adding key/value pairs.
   • Success File - Enter the file name or glob pattern that Data Migrator will use to recognize client application success files when they are created in migration directories. These files will be migrated last after all other data in the directory has been successfully migrated. You can use these files to confirm the directory they're in has finished migrating.
   • Filesystem Options
     • Select Live Migration to include Live as a migration type when creating migrations. Or select One-time Migration to limit migration types available to one-time and recurring. See migration types to learn more about each type.

7. Select Save.

   You can now migrate data from the HDFS source.

For more information about configuring Kerberos, see the section below. If you have problems configuring Kerberos, see the troubleshooting section.

Configure an HDFS source filesystem with the CLI

To create an HDFS source, run the filesystem add hdfs command in the Data Migrator CLI with the --source parameter:

Add a Hadoop Distributed File System

filesystem add hdfs       [--file-system-id] string  
                          [--default-fs] string  
                          [--user] string
                          [--kerberos-principal] string  
                          [--kerberos-keytab] string  
                          [--source]  
                          [--scan-only]  
                          [--success-file] string
                          [--properties-files] list  
                          [--properties] string

Mandatory parameters

• --file-system-id The ID to give the new filesystem resource.
• --default-fs A string that defines how Data Migrator accesses HDFS. You can enter it in the following ways:
  1. As a single HDFS URI, such as hdfs://192.168.1.10:8020 (using an IP address) or hdfs://myhost.localdomain:8020 (using a hostname).
  2. As an HDFS URI that references a nameservice if the NameNodes have high availability. For example, hdfs://mynameservice. For more information, see HDFS High Availability.
• --properties-files Reference a list of existing properties files that contain Hadoop configuration properties in the format used by core-site.xml or hdfs-site.xml.
  note

  If you're using a HA HDFS filesystem, you must include this parameter. Define the absolute paths to the core-site.xml and hdfs-site.xml files. For example, --properties-files /etc/hadoop/conf/core-site.xml,/etc/hadoop/conf/hdfs-site.xml.

Optional parameters

• --user The name of the HDFS user to be used when performing operations against the filesystem. In environments where Kerberos is disabled, this user must be the HDFS superuser, such as hdfs.
• --kerberos-principal The Kerberos principal to authenticate with and perform migrations as. This principal should map to the HDFS superuser using auth_to_local rules.
• --kerberos-keytab The Kerberos keytab that contains the principal defined for the --kerberos-principal parameter. This must be accessible to the local system user running the Data Migrator service (default is hdfs).
• --source Enter this parameter to use the filesystem resource created as a source. This is referenced in the UI when configuring the Unknown source.
• --scan-only Enter this parameter to create a static source filesystem for use in one-time migrations. Requires --source.
• --properties Enter properties to use in a comma-separated key/value list.
• --success-file Enter a file name or glob pattern for files that Data Migrator will migrate last from the directory they're contained in. For example, --success-file /mypath/myfile.txt or --success-file /**_SUCCESS. You can use these files to confirm the directory they're in has finished migrating.

Examples

Example without Kerberos

filesystem add hdfs     --file-system-id mysource
                        --default-fs hdfs://sourcenameservice
                        --properties-files /etc/hadoop/conf/core-site.xml,/etc/hadoop/conf/hdfs-site.xml
                        --source

Example with Kerberos enabled

filesystem add hdfs     --file-system-id mysource
                        --default-fs hdfs://sourcenameservice
                        --properties-files /etc/hadoop/conf/core-site.xml,/etc/hadoop/conf/hdfs-site.xml
                        --kerberos-keytab /etc/security/keytabs/hdfs.headless.keytab
                        --kerberos-principal hdfs@SOURCEREALM.COM
                        --source

For more CLI commands, see the command reference.

Configure Kerberos

From the tabs below, select which filesystem(s) you want to set up Kerberos on to view the relevant instructions.

Source only

To set up Kerberos on a source filesystem, enter the Kerberos details for your source filesystem in the Kerberos parameters above.

Target only

To migrate data from a source filesystem without Kerberos to a target with Kerberos:

1. Copy the krb5.conf file with the configuration and keytabs for your target HDFS to your source HDFS.
2. Open /etc/wandisco/livedata-migrator/vars.env and add the file path for your krb5.conf file to LDM_EXTRA_JVM_ARGS. For example:

   LDM_EXTRA_JVM_ARGS="-Djava.security.krb5.conf=/etc/remote/krb5.conf"
3. Restart Data Migrator.
4. Fill in the Kerberos parameters during target HDFS creation with the details of the Kerberos configuration you moved to your source HDFS.

Source and target with cross-realm trust

Use Kerberos on both filesystems with cross-realm trust

To use cross-realm trust to migrate data from a Kerberos-enabled source filesystem to a Kerberos-enabled target filesystem, fill in the parameters above with the details of a Kerberos configuration that has the correct cross-realm trust settings.

See the links below for Kerberos configuration guidance for common Hadoop distributions:

• CDH
• CDP
• Red Hat (Unmanaged)
• HDP

Source and target without cross-realm trust

Use Kerberos on both filesystems without cross-realm trust

See Configure Kerberos.

Next steps

Configure a target filesystem to migrate data to. Then create a migration.