Configure a secrets store

Data Migrator supports integration with secrets management systems, allowing you to securely store and manage passwords and other sensitive information, reducing the risk of unauthorized access and enhancing overall security.

With a secrets store configured with Data Migrator, you can protect specific sensitive filesystem configuration values using references to secrets store keys. You can also protect sensitive application configuration properties by adding key-value pairs to your secrets store and defining a property source in each component's secrets store configuration.

The following secrets management systems are currently supported in this release: HashiCorp Vault.

• To integrate Data Migrator with your secrets store, see Data Migrator secrets store configuration.
• To integrate Hive Migrator with your secrets store, see Hive Migrator secrets store configuration.
• To integrate the UI with your secrets store, see UI secrets store configuration.
• To learn how to use secrets store reference and protect sensitive values in application and filesystem configuration, see Using secrets store references.

note

In this release you can reference application configuration for Data Migrator, Hive Migrator and the UI, and reference sensitive filesystem configuration with Data Migrator.

Limitations

• Automatically rotated secrets are not supported. Automatically or manually changing referenced values stored in secrets store locations may have undesired effects and require additional actions to take effect. For example, changes to application property key-value pairs stored in a secrets store require a restart.
• HashiCorp Dynamic Secrets are not supported.
• Different secrets stores for different Data Migrator components are not supported. A single secrets store instance must be used for all components.
• The use of binary secret data, such as the binary content of an SSL Keystore stored in the secrets store, is not supported.
• HashiCorp Vault using Unix domain socket listener mode is not supported. TCP listener mode is supported, see HashiCorp Vault listener for more information on listening modes.

tip

It's generally considered a best practice to use TLS for Vault in production environments, review HashiCorp Vault with TLS before you apply your Vault integration configuration.

Data Migrator secrets store configuration

Integrate your secrets store with Data Migrator by adding configuration to Data Migrator application.properties. See the configuration steps for supported secrets management systems and their authentication types below.

See configuration steps for Hive Migrator and the UI if you require secrets store integration for other components.

HashiCorp Vault

Data Migrator supports Token and AppRole(push mode) authentication methods with HashiCorp Vault.

See the steps below applicable to each authentication type.

info

The HashiCorp AppRole pull mode authentication method is currently not supported.

HashiCorp Vault using token authentication

If your Vault server is configured to use Token authentication, configure Data Migrator to integrate with your HashiCorp Vault using token authentication.

1. Open /etc/wandisco/livedata-migrator/application.properties.
2. Add the following properties to the end of your application.properties file:

HashiCorp Vault using token authentication properties

    spring.cloud.vault.enabled
    spring.cloud.vault.uri
    spring.cloud.vault.authentication
    spring.cloud.vault.token

3. Optionally, add the following property to specify secrets store locations used to store configuration key-value pairs.

    spring.config.import

See Secrets store properties for descriptions of each property.

For example:

/etc/wandisco/livedata-migrator/application.properties TOKEN

    spring.cloud.vault.enabled=true
    spring.cloud.vault.uri=http://127.0.0.1:8200
    spring.cloud.vault.authentication=TOKEN
    spring.cloud.vault.token=hvs.t0k3nex4mp13
    spring.config.import=vault://secret/ldm/config, vault://secret/ldm/node1/config

4. Save the file.
5. Restart the Data Migrator service. See System service commands - Data Migrator.

tip

The spring.config.import property is optional, use this property to import additional sensitive Data Migrator application property key-value pairs from your secrets store. Use references to the paths on your secrets store containing multiple key-value pairs. See the Application property reference format for more info.

HashiCorp Vault using AppRole(push mode) authentication

If your Vault server is configured to use AppRole(push mode) authentication configure Data Migrator to integrate with your HashiCorp Vault using AppRole(push mode) authentication.

1. Open /etc/wandisco/livedata-migrator/application.properties.
2. Add the following properties to the end of your application.properties file:

HashiCorp Vault using AppRole(push mode) authentication properties

    spring.cloud.vault.enabled
    spring.cloud.vault.uri
    spring.cloud.vault.authentication
    spring.cloud.vault.app-role.role-id
    spring.cloud.vault.app-role.secret-id 

3. Optionally, add the following property to specify secrets store locations used to store configuration key-value pairs.

    spring.config.import

See Secrets store properties for descriptions of each property.

For example:

/etc/wandisco/livedata-migrator/application.properties APPROLE

    spring.cloud.vault.enabled=true
    spring.cloud.vault.uri=http://127.0.0.1:8200
    spring.cloud.vault.authentication=APPROLE
    spring.cloud.vault.app-role.role-id=ex4mp13r013va1u3ex4mp13va1u3
    spring.cloud.vault.app-role.secret-id=ex4mp13s3cretva1u3ex4mp13va1u3
    spring.config.import=vault://secret/ldm/config, vault://secret/ldm/node1/config

4. Save the file.
5. Restart the Data Migrator service. See System service commands - Data Migrator.

HashiCorp Vault with TLS

If your Vault server is configured to use TLS (as is typically recommended), specify a trust store to allow Data Migrator to trust the server's certificate and use https in your spring.cloud.vault.uri. To specify a trust store in your Data Migrator Vault configuration, include the following properties in /etc/wandisco/livedata-migrator/application.properties:

HashiCorp Vault with TLS properties

    spring.cloud.vault.ssl.trust-store
    spring.cloud.vault.ssl.trust-store-password
    spring.cloud.vault.ssl.trust-store-type

See Secrets store properties for descriptions of each property.

For example:

/etc/wandisco/livedata-migrator/application.properties TLS

    spring.cloud.vault.uri=https://127.0.0.1:8200
    ...
    ...
    spring.cloud.vault.ssl.trust-store=file:/etc/cirata/tls/truststore.jks
    spring.cloud.vault.ssl.trust-store-password=truststore_password
    spring.cloud.vault.ssl.trust-store-type=JKS

Ensure the trust store file is accessible to the local system user running the Data Migrator service and restart the Data Migrator service for changes to take effect.

tip

Remember to specify https in your spring.cloud.vault.uri if your Vault server uses TLS.

Vault TLS in JVM arguments

Alternatively, if you already specify a trust store in your additional Data Migrator JVM arguments in /etc/wandisco/livedata-migrator/vars.env configuration, include the Vault server's certificate in the trust store.

For example:

/etc/wandisco/livedata-migrator/vars.env

   LDM_EXTRA_JVM_ARGS="-Djavax.net.ssl.trustStore=/etc/wandisco/tls/truststore.jks -Djavax.net.ssl.trustStorePassword=truststore_password"

caution

Be aware, specifying a trust store using the JVM argument -Djavax.net.ssl.trustStore applies to the entire JVM instance. If applicable, you can also install the CA certificate globally to your default cacerts.

Hive Migrator secrets store configuration

Create a /etc/wandisco/hivemigrator/bootstrap.properties configuration file and add your secrets store configuration to integrate your secrets store with Hive Migrator.

See the configuration steps, supported secrets management systems and their authentication types below.

HashiCorp Vault

HashiCorp Vault using token authentication

1. Create/open file /etc/wandisco/hivemigrator/bootstrap.properties.
2. Add the following properties:

HashiCorp Vault using token authentication properties

    spring.cloud.vault.enabled
    spring.cloud.vault.uri
    spring.cloud.vault.authentication
    spring.cloud.vault.token
    spring.config.import

See Secrets store properties for descriptions of each property.

For example:

/etc/wandisco/hivemigrator/bootstrap.properties TOKEN

    spring.cloud.vault.enabled=true
    spring.cloud.vault.uri=http://127.0.0.1:8200
    spring.cloud.vault.authentication=TOKEN
    spring.cloud.vault.token=hvs.t0k3nex4mp13
    spring.config.import=vault://secret/hvm/config, vault://secret/hvm/node1/config

4. Save the file.
5. Ensure the bootstrap.properties file is owned by your Hive Migrator user.
6. Restart the Hive Migrator service. See System service commands - Hive Migrator.

tip

Use the spring.config.import property to import any sensitive Hive Migrator application property key-value pairs from your secrets store. Use references to the paths on your secrets store containing multiple key-value pairs. See the Application property reference format for more info.

HashiCorp Vault using AppRole(push mode) authentication

If your Vault server is configured to use AppRole(push mode) authentication, configure Hive Migrator to integrate with your HashiCorp Vault using AppRole(push mode) authentication.

1. Open or create file: /etc/wandisco/hivemigrator/bootstrap.properties.
2. Add the following properties:

HashiCorp Vault using AppRole(push mode) authentication properties

    spring.cloud.vault.enabled
    spring.cloud.vault.uri
    spring.cloud.vault.authentication
    spring.cloud.vault.app-role.role-id
    spring.cloud.vault.app-role.secret-id
    spring.config.import

See Secrets store properties for descriptions of each property.

For example:

/etc/wandisco/hivemigrator/bootstrap.properties APPROLE

    spring.cloud.vault.enabled=true
    spring.cloud.vault.uri=http://127.0.0.1:8200
    spring.cloud.vault.authentication=APPROLE
    spring.cloud.vault.app-role.role-id=ex4mp13r013va1u3ex4mp13va1u3
    spring.cloud.vault.app-role.secret-id=ex4mp13s3cretva1u3ex4mp13va1u3
    spring.config.import=vault://secret/hvm/config, vault://secret/hvm/node1/config

4. Save the file.
5. Ensure the bootstrap.properties file is owned by your Hive Migrator user.
6. Restart the Hive Migrator service. See System service commands - Hive Migrator.

HashiCorp Vault with TLS

If your Vault server is configured to use TLS (as is typically recommended), include the server's CA cert in your hosts cacerts to allow Hive Migrator to trust the server's certificate and use https in your spring.cloud.vault.uri.

note

You currently can not use configuration properties to supply your truststore information with Hive Migrator and HashiCorp Vault. See the following Known Issue for more info.

For example:

/etc/wandisco/hivemigrator/bootstrap.properties TLS

    spring.cloud.vault.uri=https://127.0.0.1:8200
    ...
    ...

Restart the Hive Migrator service for changes to take effect.

tip

Remember to specify https in your spring.cloud.vault.uri if your Vault server uses TLS.

Vault TLS in JVM arguments

Alternatively, if you already specify a trust store in your additional Hive Migrator JVM arguments in /etc/wandisco/hivemigrator/vars.sh configuration, include the Vault server's certificate in the trust store.

For example:

/etc/wandisco/hivemigrator/vars.sh

   HVM_EXTRA_JVM_ARGS="-Djavax.net.ssl.trustStore=/etc/wandisco/tls/truststore.jks -Djavax.net.ssl.trustStorePassword=truststore_password"

caution

Be aware, specifying a trust store using the JVM argument -Djavax.net.ssl.trustStore applies to the entire JVM instance. If applicable, you can also install the CA certificate globally to your default cacerts.

UI secrets store configuration

Integrate your secrets store with your Data Migrator UI by adding configuration to application-prod.properties. See the configuration steps for supported secrets management systems and their authentication types below.

HashiCorp Vault

See the steps below applicable to each authentication type.

HashiCorp Vault using token authentication

If your Vault server is configured to use Token authentication. Configure the UI to integrate with your HashiCorp Vault using token authentication.

1. Open /etc/wandisco/ui/application-prod.properties.
2. Add the following properties:

HashiCorp Vault using token authentication properties

    spring.cloud.vault.enabled
    spring.cloud.vault.uri
    spring.cloud.vault.authentication
    spring.cloud.vault.token
    spring.config.import

See Secrets store properties for descriptions of each property.

For example:

/etc/wandisco/ui/application-prod.properties TOKEN

    spring.cloud.vault.enabled=true
    spring.cloud.vault.uri=http://127.0.0.1:8200
    spring.cloud.vault.authentication=TOKEN
    spring.cloud.vault.token=hvs.t0k3nex4mp13
    spring.config.import=vault://secret/ui/config, vault://secret/ui/node1/config

4. Save the file.
5. Restart the UI service. See System service commands - UI.

HashiCorp Vault using AppRole(push mode) authentication

If your Vault server is configured to use AppRole(push mode) authentication configure the UI to integrate with your HashiCorp Vault using AppRole(push mode) authentication.

1. Open /etc/wandisco/ui/application-prod.properties.
2. Add the following properties to the end of your application.properties file:

HashiCorp Vault using AppRole(push mode) authentication properties

    spring.cloud.vault.enabled
    spring.cloud.vault.uri
    spring.cloud.vault.authentication
    spring.cloud.vault.app-role.role-id
    spring.cloud.vault.app-role.secret-id 
    spring.config.import

See Secrets store properties for descriptions of each property.

For example:

/etc/wandisco/ui/application-prod.properties APPROLE

    spring.cloud.vault.enabled=true
    spring.cloud.vault.uri=http://127.0.0.1:8200
    spring.cloud.vault.authentication=APPROLE
    spring.cloud.vault.app-role.role-id=ex4mp13r013va1u3ex4mp13va1u3
    spring.cloud.vault.app-role.secret-id=ex4mp13s3cretva1u3ex4mp13va1u3
    spring.config.import=vault://secret/ui/config, vault://secret/ui/node1/config

3. Save the file.
4. Restart the UI service. See System service commands - UI.

HashiCorp Vault with TLS

If your Vault server is configured to use TLS (as is typically recommended), specify a trust store to allow the UI to trust the server's certificate and use https in your spring.cloud.vault.uri.

To specify a trust store in your UI Vault configuration, include the following properties in /etc/wandisco/ui/application-prod.properties:

HashiCorp Vault with TLS properties

    spring.cloud.vault.ssl.trust-store
    spring.cloud.vault.ssl.trust-store-password
    spring.cloud.vault.ssl.trust-store-type

For example:

/etc/wandisco/ui/application-prod.properties TLS

    spring.cloud.vault.uri=https://127.0.0.1:8200
    ...
    ...
    spring.cloud.vault.ssl.trust-store=file:/etc/cirata/tls/truststore.jks
    spring.cloud.vault.ssl.trust-store-password=truststore_password
    spring.cloud.vault.ssl.trust-store-type=JKS

Ensure the trust store file is accessible to the local system user running the UI service and restart the UI service for changes to take effect.

tip

Remember to specify https in your spring.cloud.vault.uri if your Vault server uses TLS.

Vault TLS in JVM arguments

Alternatively, if you already specify a trust store in your additional UI JVM arguments in /etc/wandisco/ui/vars.env configuration, include the Vault server's certificate in the trust store.

For example:

/etc/wandisco/livedata-migrator/vars.env

   LDUI_EXTRA_JVM_ARGS="-Djavax.net.ssl.trustStore=/etc/wandisco/tls/truststore.jks -Djavax.net.ssl.trustStorePassword=truststore_password"

caution

Be aware, specifying a trust store using the JVM argument -Djavax.net.ssl.trustStore applies to the entire JVM instance. If applicable, you can also install the CA certificate globally to your default cacerts.

With your secrets store integrated with the UI, see how to Use application property references and the UI example to secure specific UI configuration properties.

Using secrets store references

With a secrets store configured, you can protect sensitive filesystem values and application properties using secrets store references. Add your secrets to your store before using the referenced locations in configuration.

note

For HasiCorp Vault AppRole authentication, ensure the path to your secret adheres to the respective KV1 or KV2 secrets engines in the AppRole policy, see HasiCorp Vault policies to learn more.

Reference format

For filesystem property configuration specify the path and the key, for application property references, specify the path containing the key-value pairs. See details on both formats below.

Filesystem property reference format

Use the following format to reference secret values in filesystem configuration:

<scheme>://<secret_engine_mount_path>/<path_to_secret>/<key>

For example:

Example: Reference the value with a key of 'json' on the secrets engine mounted at 'secret' and a path of 'ldm/gcs'.

vault://secret/ldm/gcs/json

• scheme - Defines the secrets store type configured. In this release only the vault scheme is supported.
• secret_engine_mount_path - Specifies the mount path of your HashiCorp Vault kv secrets engine.
• path_to_secret - Defines the location of the secret on your secrets store.
• key - Specifies the key of the secret on the path.

note

Both kv1 and kv2 HashiCorp Vault secrets engines are supported. See HashiCorp Vault secrets engines for more information.

tip

HashiCorp Vault allows the use of forward slashes in key names. If the key you need to reference contains a slash, explicitly specify the key by appending the path_to_secret with ?key=. For example: vault://secret/ldm/gcs?key=json

Application property reference format

When referencing application properties you can store multiple key-value pairs on your secrets store and use the path in the reference without the need to specify the key. Use the following format to reference multiple secret application key-value pairs from your secrets store in application configuration:

<scheme>://<secret_engine_mount_path>/<path_to_secrets>

For example:

Example: Reference the key-value pairs on the secrets engine mounted at 'secret' and a path of 'ldm/config'.

vault://secret/ldm/config

• scheme - Defines the secrets store type configured. In this release only the vault scheme is supported.
• secret_engine_mount_path - Specifies the mount path of your HashiCorp Vault kv secrets engine.
• path_to_secrets - Defines the location of the key-value secrets on your secrets store.

Use filesystem configuration references

To protect specific sensitive filesystem configuration values, add secret values to a path in your secrets store then reference this path and key when adding your filesystem.

note

Automatically rotated secrets are not supported see limitations for more info.

Using secrets store references in the UI

After configuring your secrets store, the UI will clearly indicate which fields can accept secrets store references. When adding your filesystem, you can either input references or enter values directly. Refer to the steps for your specific filesystem to identify the fields that support references.

note

In this release, Google Cloud Storage, ADLS Gen2, Amazon S3 and S3 allow the use of secrets store references in the UI.

Using secrets store references in the CLI

The following filesystem configuration commands currently support secrets store references with the CLI:

• filesystem add gcs with the --service-account-json-vault-reference option.
• filesystem add adls2 sharedKey with the --shared-key option.
• filesystem add adls oauth with the --oauth2-client-id and --oauth2-client-secret options.
• filesystem add s3a with the --access-key and --secret-key options.

Example

Add an ADLS Gen2 source using the filesystem add adls sharedKey command with a reference to a shared key stored in the secrets store.

Example: ADLS gen2 source filesystem with secrets store reference

filesystem add adls2 sharedKey --file-system-id adls1 --container-name container1 --storage-account-name storageAC1 --shared-key vault://secret/ldm/azure/key --scan-only --source

note

While you can use secrets store references for sensitive values when using the filesystem add command, the GCS filesystem add gcs command uses a specific --service-account-json-vault-reference option to supply a secrets store reference, see the GCS example below.

Google cloud storage example

Use the filesystem add gcs command with the --service-account-json-vault-reference option to add a GCS filesystem using a secrets store reference.

Example: Add GCS target with reference to key file in secrets store

filesystem add gcs --name gcs1 --bucket-name examplebucket --service-account-json-vault-reference vault://secret/ldm/production/keyfile

Filesystem properties which do not accept references

The following filesystem properties don't accept references to values in your secrets store. If your filesystem configuration requires properties that don't allow references, use the explicit values in your configuration.

info

For example, it's possible to use these properties when supplied as additional properties or with the equivalent CLI option. Additional properties may be properties added to configuration in the CLI with the --properties and --properties-files options or as additional key-value pairs supplied in the UI.

Filesystem	Properties	CLI option
ADLS Gen2	fs.defaultFS, fs.container.name, fs.account.endpoint	--container-name
GPFS	fs.defaultFS, kafka.topic, kafka.bootstrap.servers	--default-fs, --kafka-topic, --kafka-bootstrap-servers
HDFS	fs.defaultFS	--default-fs
LocalFs	fs.root	--fs-root
IBMCOS	topic, bootstrap.severs, bucketName, fs.s3a.endpoint
S3A	bucketName, fs.s3a.endpoint, sqsQueue, sqsQueue.endpoint	--bucket-name, --endpoint, --sqs-queue, --sqs-endpoint

Use application property references

To protect sensitive application properties (values typically stored in application.properties), add sensitive key-value pairs to a secrets store location, then reference this location with the spring.config.import property. On service restart, these spring.config.import reference locations are read as a configuration source(Property source) for the specific key-value pairs stored.

info

Ensure the key names stored on your secrets store use the same property name as used in application properties.

Example: Secure Data Migrator's server.ssl.key-store-password

For example, to secure the server.ssl.key-store-password property by referencing it from your secrets store property source:

1. Configure your secrets store with Data Migrator. See Data Migrator secrets store configuration.
2. Add the property to your secret store with key: server.ssl.key-store-password and its value containing the key store password.
3. Remove the property now provided by your secrets store from application-properties.

note

If you don't already have this property source defined, open /etc/wandisco/livedata-migrator/application.properties. Add the source location to spring.config.import.

/etc/wandisco/livedata-migrator/application.properties

spring.config.import=vault://secret/ldm/config, vault://secret/ldm/node1/config, vault://secret/ldm/new/example

4. Restart the Data Migrator service. See System service commands - Data Migrator. Upon restart, Data Migrator will identify keys and apply their values.

Example: Secure UI LDAP properties

For example, secure specific LDAP properties by referencing them from your secrets store property source:

1. Integrate your secrets store with your Data Migrator UI by adding configuration to application-prod.properties including your property source location. See UI secrets store configuration.
2. Configure the LDAP Authentication, Access Control and/or API Access Control.
   info

   Performing this configuration adds/writes/updates properties to your local application-prod.properties.

3. Place any required LDAP properties currently defined in the application-prod.properties into your secrets store. For example, encoded application.ldap.managerPassword.

note

Some LDAP property values may be formatted with a backslash escape character when contained in application-prod.properties, use your actual property value and omit any backslashes when placing the value in your secrets store. For example: For the property application.ldap.userSearchBase=ou\=people the value with the escape character omitted would be ou=people.

4. Remove these properties now provided by your secrets store from your application-prod.properties file.
5. Restart the UI service. See System service commands - UI. Upon restart, the UI will identify keys and apply their values.

info

Important. Secrets store UI property source

Clicking 'Apply' rewrites the LDAP config locally when configuring LDAP, Access Control and API Access Control. If you've configured a secrets store as a UI property source and reference any sensitive properties, remove the referenced properties from the local application-prod.properties file and update them in your secrets store.