Create a recurring migration
Recurring migrations allow continuous scanning for changing data if your source doesn't support or isn’t configured to support notification-based live changes. During migration, the source filesystem paths are scanned, data is migrated to the target, the scan will be repeated to discover new changes.
Choose the Recurring Schedule option to set a schedule of recurring scans or the Recurring Limit option to limit frequency of rescanning with a minimum number of minutes or hours from the start of the first scan to the start of the next scan.
As recurring migrations repeatedly rescan, they don't necessitate additional rescanning. Therefore, a manual Rescan is not permitted for recurring migrations.
Find all migration types and target file systems supported here.
To ensure predictable and regular recurrence, recurring migrations ignore the maximum number of running migrations. However, without this limit, consider the number and recurrence of migrations carefully to avoid any potential excessive load that concurrent scanning could place on the namenode.
Dangling files.
Recurring migrations repeatedly scan source and copy to target, files subsequently deleted from source will therefore remain present (dangling) at target. See Target Match for more options when dealing with dangling files.
Create a recurring migration with the UI
To create a recurring migration with the UI, select Recurring Migration under Migration Type when you create a migration.
Select the Set Recurring Schedule option to set a schedule of recurring scans or the Set Recurring Limit option to limit frequency of rescanning with a minimum number of minutes or hours from the start of the first scan to the start of the next scan.
When using exclusions with a recurring migration, the count of number of files excluded reported in the migration status show a cumulative total for each recurrence of the migration.
Recurring schedule
Select the Set Recurring Schedule option and enter a cron expression to set a rescan schedule for your recurring migration.
Set your recurring schedule based on the timezone of the machine where Data Migrator is running. However, note that times, such as those in Completed Recurrences, are reported in UTC.
You don't need to manually start a migration on a recurring schedule, subsequent rescans (recurrences) automatically start on the schedule defined by the cron expression.
Data Migrator won't be able to start a specific scheduled rescan if the previous one is still in progress or the Data Migrator service was unavailable at the time the occurrence was scheduled. If Data Migrator is unable to start a scheduled rescan occurrence, the occurrence is skipped/missed and the next occurrence will be scheduled as normal.
Use the Recurring scan status and Completed recurrences to find information on skipped and missed occurrences.
Cron expression format
Use cron expressions consisting of the following five fields when setting a recurring scan schedule:
┌───────────── minute (0 - 59)
│ ┌───────────── hour (0 - 23)
│ │ ┌───────────── day of the month (1 - 31)
│ │ │ ┌───────────── month (1 - 12) (or JAN-DEC)
│ │ │ │ ┌───────────── day of the week (0 - 7)
│ │ │ │ │ (0 or 7 is Sunday, or MON-SUN)
│ │ │ │ │
* * * * *
Examples
The examples below show some common date and time schedules and their corresponding cron expressions.
- Set a migration to rescan at the top of every hour of every day:
0 * * * * - Set a migration to rescan at 10:15am every day:
15 10 ? * * - Set a migration to rescan at 16:35pm every Thursday:
35 16 ? * THU - Set a migration to rescan at 14:55pm every Thursday, Saturday, Monday:
55 14 ? * THU,SAT,MON
A field may be an asterisk (*), which always stands for "first-last". For the "day of the month" or "day of the week" fields, a question mark (?) may be used instead of an asterisk.
Ranges of numbers are expressed by two numbers separated with a hyphen (-). The specified range is inclusive.
Recurring limit
Recurring migrations scan continuously by default, which can potentially cause performance degradation on the source.
Reduce the impact of continually scanning the source by setting a sufficient Recurring limit on your recurring migration. The Recurring limit value sets a minimum number of minutes or hours that must pass from the start of the scan to the start of the next scan.
Only one scan can run at once for a migration.
Examples
- If you set a limit of 12 hours and your scan takes 1 hour, the next one won't start for another 11 hours.
- If you set a limit of 24 hours and your scan takes 25 hours, the next one will start immediately.
If a limit isn't specified, it defaults to 0, and the scan will immediately recur after the last scan completes.
You can amend the rescan limit at any time for an existing migration.
Check the current limit for a migration by selecting the migration from your Migrations panel, then selecting the Recurring Scan tab.
Contact Support for guidance on adjusting this setting.
Recurring scan status
To view the status of an existing recurring migration, any previous rescans and adjust configuration:
- From the Dashboard, select your Data Migrator instance under Instances.
- Under Migrations, select Data Migrations.
- Select your migration from the Data Migrations panel.
- Select Recurring Scan
Recurring scan configuration
To adjust the current Recurring Limit value, or Recurring Schedule fields of your recurring migration: Under Recurring Scan, enter a new value and select Save.
You can't change a recurring migration's recurrence type from a Recurring schedule to a Recurring limit.
Latest Recurrences
View the status of the current rescan, the next scheduled scan and manually start a scheduled scan.
To manually begin a rescan before the next recurrence while it's waiting for the next recurrence, select Start Recurrence now.
Starting a migration manually could cause it to overrun, depending on its schedule and other parameters.
Completed Recurrences
View the Completed Recurrences panel for information about all the migration's recurrences and more detail about each recurrence.
Review the table to find information on previous recurrences including it's status. The following describes each status:
- Complete - Recurrence completed successfully.
- Failed - Recurrence was unable to complete successfully.
- Missed - This occurrence on a recurring schedule was missed due to downtime(e.g. the service was unavailable.).
- Running - This recurrence is currently underway.
- Skipped - This occurrence on a recurring schedule was skipped due to the overrun of the previous scan.
- Stopped - User stopped migration.
The Start Timestamp for each recurrence is reported in UTC.
Recurring migration with the CLI
Create a recurring migration with recurring limit
Create a recurring migration and specify a recurring limit with the CLI by using the migration add command and the --recurrence-type OFFSET, --recurring-period, and --scan-only options.
migration add --migration-id NewMigration1 --source MySource --target MyTarget --path /repl1 --recurrence-type OFFSET --recurring-period 12H --scan-only
Create a recurring migration with recurring schedule
Create a recurring migration and specify a recurring schedule with the CLI by using the migration add command and the --recurrence-type CRON, --cron-schedule and --scan-only options.
Supply your cron expression inside double quotes.
migration add --migration-id NewMigration2 --source MySource --target MyTarget --path /repl2 --recurrence-type CRON --cron-schedule "45 16 * * *" --scan-only
Update a recurring migration
Use the migration update configuration command to change an existing schedule or recurring limit for a recurring migration.
migration update configuration --name Mig3 --cron-schedule "14,15,16 * * * *"
migration update configuration --name Mig4 --recurring-period 14H
Manually start a recurring schedule migration now
Use the migration recurring start CLI command to manually begin a scheduled rescan before the next scheduled recurrence while it's waiting for the next recurrence.
migration recurring start --migration-id Mig1_noon
Starting a migration manually could cause it to overrun, depending on its schedule and other parameters.
Review the recurrence history of a migration
Use the migration recurrence history command to view the recurrence history of your recurring migration.
migration recurrence history --migration-id mig2 --hide-stats --limit 5