Configure a Google Cloud Storage target

You can migrate data to a Google Cloud Storage bucket by configuring it as a target filesystem for Data Migrator.

Follow these steps to create a Google Cloud Storage target:

Configure a Google Cloud Storage target filesystem

Prerequisites

You need the following:

• A Google Cloud Storage bucket.
• A Service Account with at least Storage Admin role permissions to your Google Cloud Storage bucket.

UI

Configure a Google Cloud Storage target filesystem with the UI

1. Connect to the UI.

2. From the Dashboard, select an instance under Instances.

3. In the Filesystems & Agents menu, select Filesystems.

4. Select Add target filesystem.

5. Enter the following details:

   • Filesystem Type - The type of filesystem target. Select Google Cloud Storage.
   • Display Name - Enter a name for your target filesystem.
   • Bucket Name - Enter the name of your Google Cloud Storage bucket.
   • Key File Options - Select whether to upload a JSON or a P12 key file to authenticate to your Google Cloud Storage bucket. Drag and drop your key file into the box below or select Browse to browse your system for it.
     • Email Address - If you chose Upload P12 Key File, enter the email address for your Google account.
   info

   Data Migrator imports Google Cloud Storage credentials from your JSON key file, stores them internally as configuration properties, then removes the file.

6. Select Save. You can now use your Google Cloud Storage target in data migrations.

CLI

Configure a Google Cloud Storage target filesystem with the CLI

To create a Google Cloud Storage target, run the filesystem add gcs command in the Data Migrator CLI:

Add a Google Cloud Storage filesystem

filesystem add gcs        [--file-system-id] string  
                          [--service-account-json-key-file] string  
                          [--service-account-p12-key-file] string  
                          [--service-account-json-key-file-server-location] string  
                          [--service-account-p12-key-file-server-location] string  
                          [--service-account-email] string  
                          [--bucket-name] string  
                          [--properties-files] list  
                          [--properties] string
                          [--source] 

Mandatory parameters

• --file-system-id The ID to give the new filesystem resource. In the UI, this is called Display Name.
• --bucket-name The bucket name of a Google Cloud Storage account. In the UI, this is called Bucket Name.

Service account key parameters

info

Enter your service account key for the Google Cloud Storage bucket by choosing one of the parameters below.

A service account key must be supplied to add the file system.

You can also upload the service account key directly when using the UI (this isn't supported through the CLI).

• --service-account-json-key-file-server-location The absolute filesystem path on the Data Migrator server of your service account key file in JSON format. You can either create a Google Cloud Storage service account key or use an existing one.
  In the UI, this is called Key File and becomes visible when you select Key File Options -> Provide a Path. Use this parameter if you are running the CLI on a different host to your Data Migrator server.

• --service-account-p12-key-file-server-location The absolute filesystem path on the Data Migrator server of your service account key file in P12 format. You can either create a Google Cloud Storage service account key or use an existing one. In the UI, this is called Key File and becomes visible when you select Key File Options -> Provide a Path. Use this parameter if you are running the CLI on a different host to your Data Migrator server.

• --service-account-json-key-file The absolute filesystem path on the host running the Data Migrator CLI of your service account key file in JSON format.

  info

  Data Migrator imports Google Cloud Storage (GCS) credentials from your --service-account-json-key-file, stores them internally as configuration properties, then removes the file.

  You can delete your JSON key file after successfully creating your GCS target.

• --service-account-p12-key-file The absolute filesystem path on the host running the CLI of your service account key file in P12 format.

Optional parameters

• --service-account-email The email address linked to your Google Cloud Storage service account. In the UI, this is called Email address and is required when selecting the Upload P12 Key File option.
• --properties-files Reference a list of existing properties files, each containing Hadoop configuration properties in the format used by core-site.xml or hdfs-site.xml.
• --properties Enter properties to use in a comma-separated key/value list.
• --source Used when adding the file system as a source.

Example

filesystem add gcs      --file-system-id gcsAgent
                        --bucket-name myGcsBucket
                        --service-account-p12-key-file-server-location /user/hdfs/targetStorage/myAccountKey.p12
                        --service-account-email user@mydomain.com

Next steps

If you haven't already, configure a source filesystem from which to migrate data. Then, you can create a migration to migrate data to your new Google Cloud Storage target.