Skip to main content
Version: 2.6 (latest)

Configure HTTP proxy

Proxy without authentication​

Forward traffic through the proxy using Java HTTP properties in the /etc/wandisco/livedata-migrator/vars.env configuration file.

HTTP proxy JVM Arguments​

ArgumentDescription
http.proxyHostThe hostname of the proxy server to be used for HTTP traffic.
https.proxyHostThe hostname of the proxy server to be used for HTTPS traffic.
http.proxyPortThe port the proxy server for HTTP is listening on.
https.proxyPortThe port the proxy server for HTTPS is listening on.
http.nonProxyHostsHosts (or IPs) which should be reached directly, not through the proxy.

Configure Data Migrator to use a HTTP proxy​

Add your proxy server arguments to Data Migrator configuration to enforce the use of a proxy server.

  1. Open/edit /etc/wandisco/livedata-migrator/vars.env in a text editor.
  2. Supply your proxy configuration on a new line starting with LDM_EXTRA_JVM_ARGS=. Ensure all arguments are enclosed in quotation marks.
    LDM_EXTRA_JVM_ARGS="-Dhttp.proxyHost=<proxy-hostname> -Dhttp.proxyPort=<proxy-port> -Dhttps.proxyHost=<proxy-hostname> -Dhttps.proxyPort=<proxy-port> -Dhttp.nonProxyHosts=\'localhost|127.0.0.1|<cluster-name-*>\'"
  3. Save the changes.
  4. Restart the Data Migrator service using the systemctl restart <service> command.
    systemctl restart livedata-migrator

Configure Hive Migrator to use a HTTP proxy​

Add your proxy server arguments to Hivemigrator configuration to enforce the use of a proxy server.

  1. Open/edit /etc/wandisco/hivemigrator/vars.sh in a text editor.
  2. Supply your proxy configuration on a new line starting with HVM_EXTRA_JVM_ARGS=. Ensure all arguments are enclosed in quotation marks.
    HVM_EXTRA_JVM_ARGS="-Dhttp.proxyHost=<proxy-hostname> -Dhttp.proxyPort=<proxy-port> -Dhttps.proxyHost=<proxy-hostname> -Dhttps.proxyPort=<proxy-port> -Dhttp.nonProxyHosts=\'localhost|127.0.0.1|<cluster-name-*>\'"
  3. Save the changes.
  4. Restart the Hive Migrator service using the systemctl restart <service> command.
    systemctl restart hivemigrator

Proxy with authentication​

note

Supplying proxy properties is supported with S3a filesystems only.

Supply S3 proxy properties when adding or updating your S3a filesystem to use a proxy server.

S3a Proxy properties​

Use the following key and value pairs when adding or updating your S3a filesystem to use a proxy server.

keyvalue
fs.s3a.proxy.hostHostname of the (optional) proxy server for S3 connections.
fs.s3a.proxy.portProxy server port.
fs.s3a.proxy.usernameUsername for authenticating with proxy server.
fs.s3a.proxy.passwordPassword for authenticating with proxy server.

Find more information on s3a properties from the official Hadoop documentation here.

Configure proxy with S3a properties with the UI​

Include your proxy configuration when adding a S3a filesystem by following the steps outlined in any of the S3 Source, S3 target, Amazon S3 source or Amazon S3 target configuration sections.

Supply your proxy configuration key and value pairs under S3A Properties when adding the filesystem.

info

You can also add proxy configuration to existing S3a filesytems.

Select your S3 filesystem from Filesystems & Agents > Filesystems. Then add your proxy key and value pairs under S3A Properties.

Configure proxy with S3a properties in the CLI​

Use the --properties flag to include proxy configuration when adding your S3a filesystem with filesystem add s3a CLI command.

Add a S3a filesystem with proxy config example​

filesystem add s3a (--source or --target) --access-key *** --secret-key *** --file-system-id <naming-FS> --endpoint <AWS-endpoint> --bucket-name <name-of-bucket> --properties "fs.s3a.proxy.host=<proxy-host> , fs.s3a.proxy.port=<proxy-port> , fs.s3a.proxy.username=<proxy-user> , fs.s3a.proxy.password=<proxy-pass>"

Update a S3a filesystem with proxy config example​

Use the --properties flag to include proxy configuration when updating your S3a filesystem with filesystem update s3a CLI command.

filesystem update s3a --properties "fs.s3a.proxy.host=<proxy-host> , fs.s3a.proxy.port=<proxy-port> , fs.s3a.proxy.username=<proxy-user> , fs.s3a.proxy.password=<proxy-pass>"
tip

Use commas to separate each property key and value pair and enclose them with quotation marks. --properties "key=<value> , key=<value>"