Configure an Oracle Object Storage target

You can migrate data to an Oracle Cloud Infrastructure Object Storage by configuring it as a target filesystem in Data Migrator.

Follow these steps to create an Oracle Object Storage target.

info

To configure a storage bucket which isn't in your default region, specify a fs.s3a.endpoint.region=<region> in the S3 Properties or with the --properties flag when adding the filesystem with the CLI.

See S3A properties below and Oracle Cloud Infrastructure Docs for more information.

Prerequisites

You need the following:

• An Oracle Object Storage.

• Authentication details for the target storage. See below for more information.

Configure an Oracle Object Storage as a target filesystem in the UI

1. From the Dashboard, select an instance under Instances.

2. In the Filesystems & Agents menu, select Filesystems.

3. Select Add target filesystem.

4. Enter the following details:

   • Filesystem Type - The type of filesystem target. Select Oracle Object Storage.
   • Display Name - Enter a name for your target filesystem.
   • Access Key - Enter the Oracle Object Storage access key. For example, RANDOMSTRINGACCESSKEY.
   • Secret Key - Enter the secret key that corresponds with your access key. For example, RANDOMSTRINGPASSWORD.
   • Bucket Name - The reference name of your Oracle Object Storage bucket.
   • Endpoint - The Oracle Object Storage endpoint (URL) for Data Migrator to connect to the bucket.
   • S3 Properties - Add optional properties to your target as key/value pairs.

5. Select Save.
   You can now use your Oracle Object Storage target in data migrations.

Configure an Oracle Object Storage as a target filesystem in the CLI

To create an Oracle Object Storage as a target in the Data Migrator CLI, run the filesystem add s3a command:

Add an S3 filesystem

filesystem add s3a          [--access-key] string
                            [--aws-config-file] string  
                            [--aws-profile] string
                            [--bootstrap.servers] string 
                            [--bucket-name] string
                            [--credentials-provider] string
                            [--endpoint] string                             
                            [--file-system-id] string 
                            [--properties] string   
                            [--properties-files] list 
                            [--s3type] string  
                            [--scan-only]
                            [--secret-key] string  
                            [--source]  
                            [--sqs-endpoint] string
                            [--sqs-queue] string                                                    
                            [--topic] string

Oracle Object Storage mandatory parameters

• --s3type Enter oracle.
• --file-system-id The ID for the new filesystem resource.
• --bucket-name The name of your Oracle Object Storage bucket.
• --access-key The Oracle Object Storage bucket access key. For example, RANDOMSTRINGACCESSKEY.
• --secret-key The secret key to use with your access key. For example, RANDOMSTRINGPASSWORD.
• --endpoint The endpoint for your Oracle object Storage bucket.

Oracle Object Storage optional parameters

• --properties Enter properties to use in a comma-separated key/value list. See the S3A properties section for more information.

S3A properties

Enter additional properties for Oracle object Storage filesystems by adding them as key/value pairs in the UI or as a comma-separated key/value pair list with the --properties parameter in the CLI. You can overwrite default property values or add new properties.

Default properties

These properties are defined by default when you add an Oracle object Storage filesystem. Overwrite them by specifying their keys with new values in key/value pairs.

• fs.s3a.impl (default org.apache.hadoop.fs.s3a.S3AFileSystem): The implementation class of the S3A filesystem.
• fs.AbstractFileSystem.s3a.impl (default org.apache.hadoop.fs.s3a.S3A): The implementation class of the S3A abstract filesystem.
• fs.s3a.user.agent.prefix (default APN/1.0 WANdisco/1.0 LiveDataMigrator/1.11.6): Sets a custom value that is prepended to the user-agent header sent in HTTP requests to the S3 backend by the S3A filesystem.
• fs.s3a.impl.disable.cache (default true): Disables the S3 filesystem cache when set to 'true'.
• hadoop.tmp.dir (default tmp): The parent directory for other temporary directories.
• fs.s3a.connection.maximum (default 120) Defines the maximum number of simultaneous connections to the S3 filesystem.
• fs.s3a.threads.max (default 150): Defines the total number of threads made available for data uploads or any other queued filesystem operation.
• fs.s3a.max.total.tasks (default 60): Defines the number of operations that can be queued for execution at a time.
• fs.s3a.healthcheck (Default true): Allows you to switch the health check off by changing the value from true to false. This option is useful for setting up Data Migrator while cloud services are offline. If you disable the check, it's harder to diagnose any issues with the S3A configuration.

Additional properties

These additional properties are not defined by default. Add more properties by specifying their key-value pairs.

• fs.s3a.fast.upload.buffer (default disk): Defines how the filesystem buffers the upload.
• fs.s3a.fast.upload.active.blocks (default 8): Defines how many blocks a single output stream can have uploading or queued at a given time.
• fs.s3a.block.size (default 32M): Defines the maximum size of blocks during file transfer. Use the suffix K, M, G, T, or P to scale the value in kilobytes, megabytes, gigabytes, terabytes, or petabytes, respectively.
• fs.s3a.buffer.dir (default tmp): Defines the directory used by disk buffering.
• fs.s3a.endpoint.region (default Current region): Explicitly sets the bucket region.

Find an additional list of S3A properties in the S3A documentation.

Additional properties example

See the following example using additional properties to set the bucket region when adding an Oracle Cloud Storage filesystem.

• fs.s3a.endpoint=myproject.compat.objectstorage.ap-sydney-1.oraclecloud.com
• fs.s3a.path.style.access=true
• fs.s3a.list.version=1
• fs.s3a.endpoint.region=ap-sydney-1

filesystem add s3a --bucket-name australia --file-system-id ocs-australia --properties fs.s3a.path.style.access=true,fs.s3a.endpoint=myproject.compat.objectstorage.ap-sydney-1.oraclecloud.com,fs.s3a.list.version=1,fs.s3a.endpoint.region=ap-sydney-1 --s3type oracle --credentials-provider com.amazonaws.auth.DefaultAWSCredentialsProviderChain --access-key 123 --secret-key 123

Upload buffering

Migrations using an S3 target destination buffer all uploads. By default, the buffering occurs on the local disk of the same system on which Data Migrator is running. Uploads are buffered to the /tmp directory.

Data Migrator automatically deletes the temporary buffering files once they are no longer needed.

To use a different type of buffering, change the property fs.s3a.fast.upload.buffer. Enter one of the following values:

Buffering option	Details	Property value
Array buffer	Buffers the uploaded data in memory instead of on the disk, using the Java heap.	array
Byte buffer	Buffers the uploaded data in memory instead of on the disk, but doesn't use the Java heap.	bytebuffer
Disk buffering	The default option. This property buffers the upload to the disk.	disk

Both the array and bytebuffer options may consume large amounts of memory. To avoid using a lot of memory and fine-tune your migrations, use properties such as fs.s3a.fast.upload.active.blocks.

note

If you run out of disk space on which to buffer the migration, the migration will stall with a series of errors. To avoid this, ensure the filesystem containing the directory used for buffering (/tmp by default) has enough remaining space to facilitate the transfer.

Next steps

If you haven't already, configure a source filesystem from which to migrate data. Then, you can create a migration to migrate data to your new Oracle Object Storage target.