Create a live migration

Changes made to the source filesystem will be migrated in real time using the notification system defined for this storage.

When you start a live migration, the initial scan will identify files and migrate them to target. After this initial scan has completed successfully the migration will reach a Live state. When live, any changes made to the source filesystem will be migrated in real time using the notification system defined for this storage.

Create a live migration with the UI

If your source file system supports it, to create a live migration with the UI, select the Live option under Migration Type when creating a migration. Live migrations will display a status of Live after their initial scan.

tip

To delete a migration, stop it first. You can stop a migration in a Live state. When stopped no further changes will be migrated, you can then Delete, Reset or Resume the migration as needed.

Create a live migration with the CLI

If your source file system supports it and you've enabled Live Migration in it's Filesystem Options, add a live migration using the migration-add cli command. Not using the --scan-ony or --recurring-migration options will create a live migration.

Example: Create a live migration with HDFS source

migration add --name liveMigration871 --path /data/871 --source sourceHDFS --target aT

Example: Show a live migration state.

Show the live state of a migration with the migration-show cli command.

migration show --name liveMigration00871

"migrationId": "test2",
  "internalId": "9374239c-ae67-4f46-a22b-e336c2b085e3",
  "path": "/data/3",
  "target": "aT",
  "state": "LIVE",
  "migrationStartTime": "2023-11-13T12:29:46.575Z",
  "migrationEdge": null,
  "scannerSummary": {

For more information, see the migration add and the optional parameters sections of the Command reference.

Stopping and resuming Live data migrations

When you stop a Live data migration, the transfer of data is suspended, but the migration will continue to accumulate events from its event stream. When the migration is resumed these events can then be processed.

By default, a Live migration is configured to accumulate 100K events from the event stream when stopped before it becomes unable to be resumed. A migration in this state must be reset before it can be started again.

The number of events an individual migration can accumulate (when stopped) before it becomes no longer resumable is set with it's maxStoppedUnscheduledEventBufferSize property.

tip

View a migrations maxStoppedUnscheduledEventBufferSize using the migration show CLI command with the --detailed option.

Adjusting maxStoppedUnscheduledEventBufferSize

In scenarios where a larger limit is required (for example, a path with high event activity or a migration stopped for an extended period), adjust the maxStoppedUnscheduledEventBufferSize on a per-migration basis via the API.

caution

Adjusting the maxStoppedUnscheduledEventBufferSize for any migration can result in additional memory consumption.

For example:

curl -X 'PUT' 'http://<Data Migrator Host>:18080/migrations/MyMigration8/maxStoppedUnscheduledEventBuffer/100001' -H 'accept: */*'

See the following endpoint: /migrations/{migrationId}/maxStoppedUnscheduledEventBuffer/{maxEvents} in your Swagger API interface for more info.