Create a recurring migration
Recurring migrations let you continuously scan for changing data if your source doesn't support or isn’t configured to support notification-based live changes. Existing data on the source will be moved to the target, then the migration scan will be repeated to discover new changes.
As recurring migrations repeatedly rescan, so they don't necessitate additional rescanning. Therefore, a Rescan is not permitted for recurring migrations.
Data Migrator supports recurring migrations for the following filesystems:
- Source: Azure Data Lake Storage (ADLS) Gen2, Hadoop Distributed File System (HDFS), local filesystem, and Mounted Network-Attached Storage (NAS).
Find all migration types and target file systems supported here.
Dangling files.
Recurring migrations repeatedly scan source and copy to target, files subsequently deleted from source will therefore remain present (dangling) at target. See Target Match for more options when dealing with dangling files.
Create a recurring migration with the UI
To create a recurring migration with the UI, select Recurring Migration under Migration Type when you create a migration.
Set an optional Rescan Limit by selecting the minimum number of minutes or hours from the start of the first scan to the start of the next scan.
When using exclusions with a recurring migration, the count of number of files excluded reported in the migration status show a cumulative total for each recurrence of the migration.
Create a recurring migration with the CLI
Create a recurring migration with the CLI by using the migration add
command with the --recurring-migration
, --recurring-period
, and --scan-only
parameters.
--recurring-migration
Enter this parameter to create a recurring migration.--recurring-period
Define the minimum number of minutes or hours from the start of the first scan to the start of the next scan. For example, 4H (four hours) or 10M (ten minutes). If not specified, the default is0
, and the scan will immediately recur after the last scan completes.--scan-only
Enter this parameter to create a recurring migration.
Example
migration add --path /repl1 --target mytarget –-migration-id myNewMigration --recurring-migration --recurring-period 12H --scan-only
Rescan limit
Recurring migrations scan continuously by default, which can cause performance degradation on the source.
Set a rescan limit to reduce risks to performance by selecting the minimum number of minutes or hours that must pass from the start of the first scan to the start of the next scan.
Only one scan can run at once for a migration.
Examples
- If you set a rescan limit of 12 hours and your scan takes 1 hour, the next one won't start for another 11 hours.
- If you set a rescan limit of 24 hours and your scan takes 25 hours, the next one will start immediately.
If a limit isn't specified, it defaults to 0
, and the scan will immediately recur after the last scan completes.
You can amend the rescan limit at any time for an existing migration.
Check the current rescan limit for a migration by selecting the migration from your Migrations panel, then selecting the Recurring Scan tab.
Contact Support for guidance on adjusting this setting.
Recurring scan metrics
Go to the Scan Progress panel of the Recurring Scan tab of a migration to see metrics on recurring scans:
- The number of scans in the last 24 hours
- The number of scans in the last 7 days
- The average time taken for all scans
- The number of times the source has been scanned, including the initial scan
The Last 10 Scans table provides detailed metrics on the start timestamps and duration of the latest 10 scans.