logo

WANDISCO FUSION®
PLUGIN FOR LIVE S3

Version 2.1.0 Build 578 - 23rd November 2018
Last updated: 20th December 2018

1. Welcome

1.1. Product overview

Use the Fusion Plugin for Live S3 to replicate data among S3 buckets. Fusion Plugin for Live S3 provides continuous, active replication of LiveData across multiple buckets, which can span regions in AWS, or can be a mix of on-premise S3-compatible storage systems and cloud-hosted services. Applications can modify and access S3 data in any of these buckets while the Fusion Plugin for Live S3 ensures data are available across all of them.

1.2. Documentation guide

This guide contains the following:

Welcome

This chapter introduces this user guide and provides help with how to use it.

Release Notes

Details the latest software release, covering new features, fixes and known issues of which you should be aware.

Concepts

Explains how Fusion Plugin for Live S3 through WANdisco Fusion uses WANdisco’s LiveData platform.

Installation

Covers the steps required to install and set up Fusion Plugin for Live S3 into a WANdisco Fusion deployment.

Operation

The steps required to run, reconfigure and troubleshoot Fusion Plugin for Live S3.

Developer

A guide for developers looking to develop and incorporate their own software for Fusion Plugin for Live S3.

Reference

Additional Fusion Plugin for Live S3 documentation, including documentation for the available REST API.

1.2.1. Symbols in the documentation

In the guide we highlight types of information using the following call outs:

The alert symbol highlights important information.
The STOP symbol cautions you against doing something.
Tips are principles or practices that you’ll benefit from knowing or using.
The i symbol shows where you can find more information, such as in our online Knowledgebase.

1.3. Contact support

See our online Knowledgebase which contains updates and more information.

If you need more help raise a case on our support website.

1.4. Give feedback

If you find an error or if you think some information needs improving, raise a case on our support website or email docs@wandisco.com.

2. Release Notes

2.1. Live S3 2.1 Build 578

23 November 2018

WANdisco is pleased to present the first major revision to the Fusion Plugin for Live S3. The 2.1 release of the Fusion Plugin for Live S3 supports the latest version of WANdisco Fusion, 2.12.3. It includes a handful of new features, issue resolutions and other enhancements. These release notes include specific information about the product improvements, and should be read in conjunction with the product documentation.

2.1.1. Highlighted New Features

User Interface

Simplified User Interface for common tasks such as replication rules management (create, list, update, delete), consistency check, consistency report and repair.

High Availability

Supports Fusion high availability deployment in a Live S3 environment. This allows for the Live S3 Plugin and its Proxy to operate in a High Availability configuration.

Improved Scalability

Horizontal scalability capability for Live S3 environments enables large number of simultaneous network connections to access the service.

Replication across firewall

Enables Fusion to replicate data from on-premises to AWS in a DMZ configuration, where on-premises Fusion Server can see and make connections to AWS Fusion server. However, the AWS Fusion server is completely firewalled from the on-premises network. No inbound connections are allowed on any port.

Enhanced Selective Replication

Replication rules will provide inclusion & exclusion patterns. User can be selective about content to be replicated at a finer level of granularity than a bucket.

2.1.2. System Requirements

Before installing or upgrading, ensure that your systems, software, and hardware meet the requirements. The requirements for WANdisco Fusion are found in the User Guide at link: Pre-requisites

Fusion Plugin for Live S3 is tested on a more limited number of operating systems then the main product. These are:

  • RHEL 6 x86_64

  • RHEL 7 x86_64

  • Oracle Linux 6 x86_64

  • Oracle Linux 7 x86_64

  • CentOS 6 x86_64

  • CentOS 7 x86_64

  • Ubuntu 16.04LTS

  • SLES 11 x86_64

  • SLES 12 x86_64

Unsupported:

  • Ubuntu 14.04LTS

  • Ubuntu 18.04LTS

2.1.3. Known Issues

Some S3 concepts do not map directly to an environment that replicates buckets—e.g., bucket creation or deletion, and operations that act on objects by unique identifier provided by the S3 endpoint. There are also some aspects of the S3 API that are not yet supported—e.g., browser-based object upload/creation.

  • WD-SPX-9 - Deletion of specific object versions is not supported because independent S3 buckets assign different version identifiers to replicated instances of an object.

  • WD-SPX-11 - Deletion of tagging for a specific object version is not supported because independent buckets assign different version identifiers to replicated instances of an object.

  • WD-SPX-48 - POST Object restore allows restoration of a temporary copy of an archived object. It is not supported by the Fusion Plugin for Live S3.

  • WD-SPX-98 - Complete Multipart Upload won’t replicate when Server-side Encryption is Enabled.

  • WD-SPX-104 - AWS KMS SSE is not yet supported.

  • WD-SPX-123 - Multi-chunk payloads are not yet supported.

  • WD-SPX-125 - Authentication via an IAM role is not yet supported.

  • WD-SPX-125 - Buckets configured with MFA serial security are not yet supported.

  • WD-SPX-347 - POST is an alternative to PUT for browser-based uploads. The Fusion Plugin for Live S3 does not support Post object replication with virtual bucket.

  • WD-SPX-965 - Performing consistency checks which yield large number of inconsistencies (e.g. hundreds of thousands) consume large amounts of memory (~1.5kb per inconsistency). When the number of inconsistencies is larger then ~1.3 million, then the server will crash with OutOfMemory error. See Memory requirements

    Workaround: if it’s known the buckets are widely inconsistent, perform the consistency check on prefix to reduce the scope, or perform the repair first.

  • SPX-981 - The installer’s restarts may, in some cases, not actually restart services. TO fix, perform restarts before the installation begins, or manually start services if a restart doesn’t complete.

3. Concepts

3.1. Product concepts

Familiarity with the following concepts will improve your use of the Fusion Plugin for Live S3.

Virtual Bucket

A virtual bucket is a bucket accessible through the Fusion Plugin for Live S3 that retains and shares data across multiple underlying buckets. Access your data with automatic, continuous and consistent replication using the Fusion Plugin for Live S3. Applications use the virtual bucket when interacting with S3 via the Fusion Plugin for Live S3.

S3 Proxy

Applications use the Fusion Plugin for Live S3 via one or more S3-compatible endpoints that it provides in the S3 proxy, which is the runtime component that proxies access to underlying S3-compatible storage services on behalf of the Fusion Plugin for Live S3.

WANdisco Fusion Plugin

The Fusion Plugin for Live S3 extends the Fusion server to support the operation of the S3 proxy, extending in later versions to support a range of cloud storage types.

Version 1.0 coordinates activities among multiple S3 proxy instances with the WANdisco Fusion Plugin that is installed for each WANdisco Fusion server.

Dependencies

A virtual bucket is associated with a replication rule that is used for all coordination against operations performed on that virtual bucket, and with one underlying s3 bucket per zone.

3.2. S3 Plugin Architecture

Fusion Plugin for Live S3 provides a LiveData architecture, where data are stored and used in multiple locations, while data are replicated with guaranteed consistency across them all.

The Fusion Plugin for Live S3 is a distributed network proxy for the S3 API that uses WANdisco Fusion to replicate data. Replication is performed selectively for S3 buckets, allowing any bucket to have replicas in other locations or S3 providers, including Amazon S3, Dell EMC Elastic Cloud Storage, IBM Cloud Object Storage, Amazon Snowball, Snowball Edge, Virtustream Storage Cloud and more.

Requests made from client applications to the S3 endpoints provided by the proxy are coordinated so that activities performed against a single S3 bucket make content consistent across multiple buckets. These buckets can span separate AWS regions, or even be provided by alternative S3 implementations.

Multiple applications can use any of the replicated S3 endpoints at the same time, while WANdisco Fusion ensures that all activities are replicated with active-active consistency across all environments. Each application need only communicate with its local replicated S3 endpoint for objects that it uses to be made consistent across all S3 buckets.

Live S3 Architecture
Figure 1. Live S3 Architecture

Build distributed systems that use S3 with the Fusion Plugin for Live S3. Applications can operate against the same set of S3 objects in multiple locations, reading and writing against their local endpoint, while WANdisco Fusion ensures that the objects are accessible in every location.

Unlike cross-region replication that is native to Amazon S3, the Fusion Plugin for Live S3 supports the use of object replicas:

  • across multiple regions without the use of Object Versioning

  • between buckets within the same region if needed

  • in a multi-directional manner among as many buckets as you need

  • without granting Amazon S3 IAM roles just for the purpose of replication

  • without any need for maintaining metadata information that is specific to replication

  • preventing conflicting modifications to the bucket

  • with S3-compatible systems other than Amazon S3

  • between different S3 providers

3.3. Supported Functionality

3.3.1. S3 Features

The Fusion Plugin for Live S3 supports a broad range of S3-compatible features, including:

Virtual-hosted style URLs

Where the virtual bucket name is part of the domain name in a virtual-hosted–style URL. For example: http://virtualbucket.s3proxyhost.yourdomain.com.

Path style URLs

Where the bucket name is not included in the domain unless as a region-specific endpoint. For example: http://s3proxyhost.yourdomain.com/virtualbucket.

Client request signature validation

Requests are validated for correct credentials before being executed against an underlying S3-compatible storage. Versions 2 and 4 of AWS signatures are supported:

  1. AWS V4 Path Style

  2. AWS V4 Virtual Hosted Style

  3. AWS V4 Pre-signed URLs

  4. AWS V2 Path Style

  5. AWS V2 Virtual Hosted Style

  6. AWS V2 Pre-signed URLs

The Fusion Plugin for Live S3 supports versions 2 and 4 of AWS signatures in Path Style. To enable virtual hosted style, follow the below steps:

  1. Add a tag <path-style-access>false</path-style-access> in /etc/wandisco/fusion/plugins/live-s3/proxy-plugin-site.xml

  2. Add a tag for IHC pull <direct-pull-enabled>true</direct-pull-enabled> in /etc/wandisco/fusion/plugins/live-s3/proxy-plugin-site.xml.

  3. Add the DNS entry for virtual-hosted–style URL in /etc/hosts. Open /etc/hosts file and add the virtual-hosted–style URL, e.g 0.0.0.0 virtualbucket.s3proxyhost.yourdomain.com
Default config values in AWS CLI

Signature type: v4
Path addressing style: auto

Run the commands below to set the required signature type and addressing style:

Signature type v4: aws configure set default.s3.signature_version s3v4
Signature type v2: aws configure set default.s3.signature_version s3
Enable Path Style: aws configure set default.s3.addressing_style path
Enable Virtual Style: aws configure set default.s3.addressing_style virtual
Payload Options

S3 Payloads can be signed or unsigned in a single chunk. Multi-chunk payloads are not yet supported.

Server-side Encryption

AES-256 server-side encryption is supported. AWS KMS SSE is not yet supported.

IAM role authentication

IAM role-based authentication is not yet supported.

MFA Serial authentication

Buckets configured with MFA serial security are not yet supported.

3.3.2. WANdisco LiveData Features

The Fusion Plugin for Live S3 provide support for additional features beyond those of standard S3 endpoints as a result of supporting LiveData functionality. These include:

Consistency Check

Determine and report on differences in content between replicated buckets.

Repair

Resolve any differences in the content among multiple buckets automatically.

3.4. Deployment models

3.4.1. Use Cases for the Fusion Plugin for Live S3

Use the Fusion Plugin for Live S3 for a variety of reasons, including:

Heterogenous storage

Your applications may benefit from accessing S3 data in different storage systems, perhaps both on-premises and in the cloud, or with multiple cloud providers to take advantage of cost arbitrage.

Multi-geo applications

Applications that operate in multiple, geographically-separate locations can work with a local S3 endpoint in each location, and ensure that each location has access to the same data.

Improving performance

By having a local replica of a bucket, applications can operate more efficiently than if they need to work with data that are not physically close.

Improved availability

The impact of the failure of a single source of S3 objects can be eliminated by having a strongly-consistent replica of those objects in another source.

Regulatory compliance

Your compliance needs may require that you store multiple copies of data in different locations, or with different service providers.

The Fusion Plugin for Live S3 automates the replication of data across S3 buckets and ensures that they store exactly the same information, even when applications change content in any of the replicated buckets.

3.5. Application integration

Important note about access/secret keys

The access key and secret key of s3Plugin and the S3proxy server should be same.
The replicated rule should be as follows:

  • HDP HDFS + S3Plugin ⇒ Replicated Rule

  • ASF + ASF ⇒ Replicated Rule

The fs.fusion.s3.region property should be set with the region value.

The fs.fusion.s3.pathStyleAccess value should be set to 'true'. - Reason: The machine where s3plugin is running may not resolve the <hostname>://<bucket_name>.<S3Proxy_url>:<S3Proxy_port>, by adding /etc/hosts entry can set the fs.fusion.s3.pathStyleAccess value as false.

3.5.1. ASF Environment

Diagram
Live S3 with ASF Environment
Figure 2. Live S3 with ASF Environment
Procedure
Install ASF

  1. Set up an ASF Cluster

Install Fusion on ASF

  1. Set up Fusion ASF (2 zones)

  2. Create a membership

  3. Create a replication rule

Install S3Proxy Plugin on ASF

  1. Install S3Proxy Plugin.

  2. Configure the S3 Proxy Plugin.

Install S3Proxy Server on ASF

  1. Install S3Proxy Server on ASF.

  2. Configure the S3Proxy Server.

3.5.2. LFS Environment

Diagram
Live S3 with ASF Environment
Figure 3. Live S3 with ASF Environment
Install LFS

  1. Set up LFS Cluster (2 zones).

Install Fusion on LFS

  1. Set up Fusion LFS (2 zones).

  2. Create a membership.

  3. Create a replication rule.

Install S3Proxy Plugin on LFS

  1. Install S3 Proxy Plugin.

  2. Install S3Proxy Server on LFS.

LFS Environment
Diagram
Live S3 with ASF Environment
Figure 4. Live S3 with ASF Environment
Install LFS

  1. Setup LFS Cluster (2 zones)

Install Fusion on LFS

  1. Setup Fusion LFS (2 zones)

  2. Create a membership

  3. Create a replication rule.

Install S3Proxy Plugin on LFS

  1. Installation.

  2. Configuration.

Install S3Proxy Server on LFS

  1. Installation.

  2. Configuration.

3.5.3. HDP + S3 Environment

Diagram
Live S3 with ASF Environment
Figure 5. Live S3 with ASF Environment
Install HDP + Fusion

  1. Set up a HDP cluster.

Install S3Plugin Fusion

  1. Create a bucket.

  2. Create the ec2 instance with s3 template to install Fusion with the bucket you created.

  3. Induct both HDP and S3p zones.

  4. Create a membership.

  5. Credate a replication rule.

3.5.4. Connect Fusion S3 Plugin with s3Proxy

  1. In S3Plugin Fusion UI, update the Access Key and secret key’s

    Live S3 AWS Credentials
    Figure 6. Live S3 AWS Credentials

3.5.5. Connect Fusion S3 Plugin with s3Proxy

  1. In S3Plugin Fusion UI, update the Access Key and secret key’s

    Live S3 AWS Credentials
    Figure 7. Live S3 AWS Credentials

3.5.6. High availability

Configuration requirements

The following configuration is required to avoid "Bad Gateway" response to "aws s3 ls" command:

Add the following property to /etc/wandisco/live-s3-proxy/proxy-server-site.xml

  1. Open /etc/wandisco/live-s3-proxy/proxy-server-site.xml

    vi /etc/wandisco/live-s3-proxy/proxy-server-site.xml

  2. The parameters can be added as below:

    <max-proxy-threads>{  }</max-proxy-threads> // default 200
<max-client-threads>{  }</max-client-threads> // default 256
<max-client-request-queue>{  }</max-client-request-queue> // default 1024
<timeout>{  }</timeout> // default 0
<idle-timeout>{  }</idle-timeout> // default 60000

  3. Restart the proxy server.

    /etc/init.d/s3proxy-server restart

4. Installation

4.1. Pre-requisites

Along with the standard product requirements that you can find on the WANdisco Fusion Deployment Checklist, you also need to ensure that you have available:

  • WANdisco Fusion 2.12.2 for S3. See the S3 installation of the WANdisco Fusion User Guide for more information.

    Fusion’s S3 bucket shall be used for internal storage and is required for the correct operation. Please make sure that the bucket is not deleted.

  • Java 1.8.

  • One or more compatible providers of an S3 endpoint: AWS S3, AWS Snowball, Virtustream Storage Cloud or HGST Activescale. Note that other providers may be fully compatible with the WANdisco Fusion, and WANdisco will continue to test and validate functionality for a broad range of S3 implementations.

  • Credentials for accessing the S3 endpoints among which replication is required. For AWS S3, this will be in the form of an Access Key and Secret Access Key.

  • Details of the endpoint URL by which applications access the S3 service normally, e.g. s3-us-west-1.amazonaws.com.

  • The name of each bucket in use.

  • Access to the hosts on which each WANdisco Fusion server is operating for the purpose of installation.

  • The names of the WANdisco Fusion zones across which replication will occur. Also note that when using the Live S3 proxy, authentication can be limited depending on your configuration:

  • The bucket configured as shared bucket in S3Plugin, should not be used as replicated bucket in Live S3.

  • For buckets with an assigned virtual bucket, only the accessKey/secretKey pair which is defined in the vbucket configuration can be used to access the proxy. Other accessKey/secretKey will be refused as invalid credentials (even though they might actually be valid with the underlying storage).

  • For non-replicated buckets, any accessKey/secretKey pair that appears in any vbucket configuration can be used to access the proxy. The same pair will be used when request is passed to the underlying storage.

4.1.1. Memory requirements

Live S3 currently requires a larger memory buffer than is specified for WANdisco Fusion running without Live S3, it requires this additional system memory for pulling data (64MB+ part per thread). The amount of memory that we recommend is based on the following guideline:

Fusion recommended + 2*64MB*NumberOfExecutors + MAX(512 * numOfObjectInBucket, 1400 * numberOfInconsistenciesInBucket)
Fusion recommended

48GB - small cluster, 64GB - large cluster

MAX

Is for a single consistency check, running in parallel.

numOfObjectInBucket

Number of objects in the bucket

numberOfInconsistenciesInBucket

This is the number of inconsistencies found in the bucket, a number greater than 1400 (1.5GB) would potentially represent a problem in its own right.

4.2. Installation

Ensure you have read all known issues and pre-requisites before beginning installation.

4.2.1. Install the plugin

All WANdisco Fusion servers which participate in S3 replication need to have the plugin installed. You can be selective about which WANdisco Fusion zones will have S3 objects replicated for each virtual bucket, but every WANdisco Fusion server in the network needs the plugin in order to function.

Ensure that you repeat the installation on all WANdisco Fusion nodes in the zone. Once Live S3 is installed on all nodes within a zone, it will automatically activate.

Use the following steps to complete an installation using the installer file, this needs to be carried out on every required Fusion server. This requires an administrator to enter details throughout the procedure.

  1. Open a terminal session on your WANdisco Fusion node.

  2. Download the relevant installer from customer.wandisco.com.

  3. Run the Live S3 plugin installer using an account with appropriate permissions:

    # ./live-s3-installer-2.1.0.0.sh Enter

    The installer will now start.

    Verifying archive integrity... All good.
Uncompressing WANdisco Live S3..................

    ::   ::  ::     #     #   ##    ####  ######   #   #####   #####   #####
   :::: :::: :::    #     #  #  #  ##  ## #     #  #  #     # #     # #     #
  ::::::::::: :::   #  #  # #    # #    # #     #  #  #       #       #     #
 ::::::::::::: :::  # # # # #    # #    # #     #  #   #####  #       #     #
  ::::::::::: :::   # # # # #    # #    # #     #  #        # #       #     #
   :::: :::: :::    ##   ##  #  ## #    # #     #  #  #     # #     # #     #
    ::   ::  ::     #     #   ## # #    # ######   #   #####   #####   #####

You are about to install WANdisco Live S3 version 2.1.0.0

Do you want to continue with the installation? (Y/n)

    The installer will perform an integrity check and confirm the product version that will be installed. Enter "Y" to continue the installation.

  4. Follow the instructions to add all required Fusion servers

    This plugin requires knowledge of all the fusion servers for this zone.
You can add Fusion servers and their request ports one by one with a 'hostname:port' format.
The hostname should be of a form that will allow the local proxy server to contact the specified hostname.
No checking is currently performed to validate any server addresses entered here.
Multiple entries can be added by separating entries with a ','.
Adding the local fusion server to the fusion server list
The current fusion server list is:
  '<your.fusion.hostname>:<port>'

  Enter 'P' to print the current fusion server list.
  Entering a blank value ends the fusion server collection.
  Host and port to add (format: 'hostname:port')  []:

    Enter the hostname and ports for each node in the current zone, then press Enter.

  5. Confirm the hostname and port for the virtual server. Enter the details or press enter to go with the default value: 0.0.0.0:8081.

    Enter the proxy server listen host and port [0.0.0.0:8081]:

  6. Enter hostname for the Proxy Server(s).

    Enter DNS compatible host names for the Proxy Server.
Multiple entries can be added by separating with a ','.
The current proxy virtual host list is:
'localhost'
'127.0.0.1'

Enter 'P' to print the current proxy virtual host list.
Entering a blank value ends proxy virtual host collection.
New Virtual Host []:

  7. Decide whether you want SSL encryption for the proxy

    Do you wish to enable proxy ssl? (Y/n): n

  8. Installation will then occur:

    Installing requested plugin components...
  wd-live-s3-plugin-2.1.0.0.tar.gz ... Done
  live-s3-plugin-ui-server-2.1.0.0-dist.tar.gz ... Done
  fusion-s3-plugin-asf-2.5.0-2.1.0.0-547.noarch.rpm ... Done
  fusion-s3-ihc-plugin-asf-2.5.0-2.1.0.0-547.noarch.rpm ... Done
  fusion-s3-proxy-asf-2.5.0-2.1.0.0-547.noarch.rpm ... Done
All requested components installed.

Running additional post install actions...

Running fusion server plugin configuration

Running proxy server configuration
Moving existing '/etc/wandisco/live-s3-proxy/proxy-server_env.sh' to '/etc/wandisco/live-s3-proxy/proxy-server_env.sh.29491.backup'

  9. A restart of the fusion-server is required, enter "y" to continue the installation.

    Restarting fusion-server is required as part of plugin activation
Do you wish to restart fusion-server now? (y/N) y
Restarting WANdisco Fusion Server: fusion-server
Stopped WANdisco Fusion Server process 5849 successfully.
Started WANdisco Fusion Server process successfully.

  10. A restart of the Fusions IHC servers is required, enter "y" to trigger a restart.

    Restarting fusion-ihc-server-asf_2_5_0 is required as part of plugin activation
Do you wish to restart fusion-ihc-server-asf_2_5_0 now? (y/N) y
Restarting WANdisco Fusion IHC Server: fusion-ihc-server-asf_2_5_0
Stopped WANdisco Fusion IHC Server process 5951 successfully.
Started WANdisco Fusion IHC Server process successfully.

  11. A restart of the s3proxy server is required, enter "y" to trigger a restart.

    Restarting WANdisco Live S3Proxy Server: live-s3-proxy
No WANdisco Live S3Proxy Server to stop.

Started WANdisco Live S3Proxy Server process successfully.

Installation complete

  12. The Live S3 Plugin is now installed. You can confirm this by checking the Plugin section of the Fusion UI Settings tab.

    Live S3 Status
    Figure 8. Live S3 Status

4.2.2. Configure the proxy

Configuring the proxy needs to be done where the proxy is installed.

Change current directory to /etc/wandisco/live-s3-proxy:

# cd /etc/wandisco/live-s3-proxy Enter

Execute the configuration script configure-proxy-server. Provide details for the operation of the S3 proxy:

Server listen host

The network interface on which the proxy should listen for client connections. This can be a specific IP address, network name, or the 0.0.0.0 value if the proxy should listen on all available interfaces.

Server listen port

The IP port used by the proxy to accept client connections.

Enable SSL

The proxy supports HTTP or HTTPS access. SSL should be enabled to offer clients the option of communicating with the proxy via HTTPS. Specifying yes to this setting will require further information on the keystore path and password.

Virtual host name

Proxy server’s DNS compatible host names.

Fusion server host name

The host name of the WANdisco Fusion server associated with this proxy instance.

Fusion server request port

The request port offered by the WANdisco Fusion server.

An example:

# ./configure-proxy-server Enter
Enter the proxy server listen host [0.0.0.0]: s3proxydemo.wandisco.com Enter
Enter the proxy server listen port [8081]: 8081 Enter
Do you want to enable ssl (yes/no)?
  [If yes, you need to provide the keystore path and password]: no Enter
Enter Proxy server’s DNS compatible host names (ie., virtual host) [localhost,127.0.0.1]: s3proxydemo.wandisco.com Enter
Enter the fusion server host and port [host:port]: s3proxydemo.wandisco.com:8023 Enter
Is Fusion SSL enabled (yes/no)?
 [If yes, you need to provide the keystore path and encrypted password]: yes Enter
Enter the Truststore file path: /opt/fusionssl/wandisco.ks Enter
Enter the Truststore keytype: JKS Enter
Please enter the password to be encrypted
>
 ------------------------------------------------------------------------------------------------
 * S3Proxy server details *
Proxy server listen host: s3proxydemo.wandisco.com
Proxy server listen port: 8081
Proxy server SSL: true
Keystore path: /etc/wandisco/live-s3-proxy/ssl/wandisco.ks
Keystore password: 
S3Proxy server DNS compatible host names: localhost,127.0.0.1,s3proxydemo.wandisco.com
 * Fusion server details *
Fusion server host and port: s3proxydemo.wandisco.com:8023
Fusion server SSL: true
Keystore path: /opt/fusionssl/wandisco.ks
Keystore type: JKS
Keystore password: *
 ------------------------------------------------------------------------------------------------
 ------------------------------------------------------------------------------------------------
Which user should Live S3Proxy run as? [root]: root Enter
Which group should Live S3Proxy run as? [root]: root Enter
Enter the minimum memory(-Xms) for Live S3Proxy (in MB) [512]: Enter
Enter the maximum memory(-Xmx) for Live S3Proxy (in MB) [1024]: Enter
 -------------------------------------------------
 * Live S3Proxy environment details *
 Run as User: root
 Run as Group: root
 Minimum memory: 512m
 Maximum memory: 1024m
  -------------------------------------------------
Do you confirm the s3proxy server configuration details (yes/no): yes Enter
S3Proxy server configuration done successfully, start 's3proxy-server' to load the configuration
--------------------------------------------------------------------------------------------------------
Note: You can edit the configuration values anytime in: /etc/wandisco/live-s3-proxy/core-site.xml, /etc/wandisco/live-s3-proxy/proxy-server-site.xml
      The s3proxy-server must be restarted for the changes to take effect
  -------------------------------------------------------------------------------------------------------

Ensure that you repeat the installation on all WANdisco Fusion nodes in the zone. Once Live S3 is installed on all nodes within a zone, it will automatically activate.

4.2.3. Manually IHC plugin Installation

In Live S3 1.0, the IHC plugin is now included in the installer script, so these manual steps should not be required.

4.2.4. Install the IHC plugin

When the direct-pull-enabled configuration is set to false for one or more buckets, the IHC plugin is required. The plugin will attempt to use the IHC to pull the object being replicated to the destination zone and so the Live S3 IHC plugin must be installed so the IHC can find the object being pulled.

To install the IHC plugin:

Install the plugin on each WANdisco Fusion server:
# rpm -i fusion-s3-plugin-localfs-2.7.0-2.0.1.0-328.noarch.rpm Enter

or

# dpkg -i fusion-s3-proxy-localfs-2.7.0-2.0.1.0-328.noarch.rpm Enter

4.2.5. Enabling SSL

To enable SSL for the proxy server follow the steps below:

  1. Generate SSL certificates in /etc/wandisco/live-s3-proxy/generate-keystore.sh

  2. Reconfigure the proxy server in /etc/wandisco/live-s3-proxy/configure-proxy-server

  3. Connect to the S3Proxy without --no-verify-ssl. The 's3proxy server' can be connected via ssl without --no-verify-ssl by one of the following approaches.

Approach 1: Passing the trusted CA root certificate using the --ca-bundle command line argument

aws s3 ls s3:// --endpoint-url https://s3proxydemo.wandisco.com:8081 --ca-bundle /etc/wandisco/live-s3-proxy/ssl/ca.crt
2018-05-21 13:38:04 vbucket

Approach 2: Set the environment variable AWS_CA_BUNDLE with the absolute path to the trusted CA root certificate.

export AWS_CA_BUNDLE=/etc/wandisco/live-s3-proxy/ssl/ca.crt
echo $AWS_CA_BUNDLE
/etc/wandisco/live-s3-proxy/ssl/ca.crt
aws s3 ls s3:// --endpoint-url https://s3proxydemo.wandisco.com:8081
2018-05-21 13:41:04 vbucket

Approach 3: In the .aws/config file, set the variable ca_bundle with the absolute path of the trusted CA root certificate.

ca_bundle = /etc/wandisco/live-s3-proxy/ssl/ca.crt

cat .aws/config
[default]
output = json
region = us-east-1
ca_bundle = /etc/wandisco/live-s3-proxy/ssl/ca.crt
s3 =
signature_version = s3
addressing_style = path
aws s3 ls s3:// --endpoint-url https://s3proxydemo.wandisco.com:8081
2018-05-21 13:41:04 vbucket

4.3. Validation

After installation and configuration is complete we recommend that you create a test replication rule and test data to ensure that everything is working as it should.

4.4. Upgrade

Use the following procedure to update via package manager. This example is CentOS/RMPs.

  1. Stop the fusion server.

    service fusion-server stop

  2. Stop the fusion IHC server.

    service fusion-ihc-server stop

  3. Stop the s3 proxy server.

    service s3proxy-server stop

  4. Upgrade the plugin RPM.

    rpm -U fusion-s3-plugin-asf-2.x.x-2.0.1.x-123.noarch.rpm

  5. Upgrade the proxy RPM.

    rpm -U fusion-s3-proxy-asf-2.x.x-2.0.1.x-123.noarch.rpm

  6. 6. Upgrade the ihc server.

    rpm -U fusion-s3-ihc-plugin-asf-2.7.0-2.0.1.0-xyz.noarch.rpm

  7. Start the Fusion server.

    service fusion-server start

  8. Start the Fusion IHC server.

    service fusion-ihc-server start

  9. Start the s3 proxy server.

    service s3proxy-server start

4.5. Uninstallation

  1. Stop fusion server.

  2. Stop s3proxy server.

  3. Uninstall using below command.

    For centOS:
# rpm -e {fusion-s3-plugin-asf-2.7.0-2.0.1.0-xyz.noarch.rpm}
# rpm -e {fusion-s3-proxy-asf-2.7.0-2.0.1.0-xyz.noarch.rpm}
# rpm -e {fusion-s3-ihc-plugin-asf-2.7.0-2.0.1.0-xyz.noarch.rpm}

For Ubuntu:
# dpkg -r {fusion-s3-plugin-asf-2.7.0-2.0.1.0-xyz.noarch.deb}
# dpkg -r {fusion-s3-proxy-asf-2.7.0-2.0.1.0-xyz.noarch.deb}
# dpkg -r {fusion-s3-ihc-plugin-asf-2.7.0-2.0.1.0-xyz.noarch.deb}

  4. Uninstall Live-s3 from UI server.

    rm -rf /opt/wandisco/fusion-ui-server/plugins/live-s3-plugin-ui-server-2.1.0.1/
rm -rf /opt/wandisco/fusion-ui-server/ui-client-platform/plugins/wd-live-s3-plugin/

  5. Start Fusion server.

    service fusion-server start

  6. Start the Fusion IHC server.

    service fusion-ihc-server start

5. Operation

5.1. Configuration

Once configured, restart the WANdisco Fusion server to use the configuration applied:

# service fusion-server restart Enter

Then start each instance of the S3 proxy:

# service s3proxy-server start Enter

You can validate operation against the virtual buckets defined for the environment using standard S3 client applications, such as the AWS CLI tools, or tools like s3cmd.

5.1.1. Configuring Applications to use the Fusion Plugin for Live S3

Client applications can be configured to use the virtual buckets provided by the Fusion Plugin for Live S3 in multiple ways:

As an HTTP(S) proxy

Applications that would normally communicate directly with the underlying S3 bucket can be directed to use the proxy through standard HTTP(S) proxy configuration. The benefit of this approach is that it requires no change to application code to direct them to the Fusion Plugin for Live S3.

Ensure that the plugin has been configured to accept requests for the original hostname used by client applications. Specify the DNS compatible host names to match when configuring the proxy. Applications can continue to use the original bucket name if it matches the underlying bucket referred to by the proxy.

As a new S3 endpoint

Applications can direct S3 requests directly to the proxy, which provides virtual buckets. Configure your application to refer to the proxy as the S3 endpoint, and use the virtual bucket name.

5.2. Create a rule

Before you can replicate data, you need to set up a replication rule for the resource location in the underlying file system.

  1. Login to the Fusion UI and click on the Replication Rules tab. Click on the Create + button.

    Live S3 Architecture
    Figure 9. Live S3 Create a rule

  2. The Rule creation screen will appear. Enter the following details.

    Live S3 Architecture
    Figure 10. Live S3 Create a rule
    Virtual bucket name

    Choose a name that will be the single identifier for the proxy’s virtual bucket. This name will be used by client applications when interacting with replicate S3 objects, and will be available at the endpoints offered by each of the S3 proxy instances.

    Replication constraints
    Include in replication

    Select which resources you wish to replicate under the rule. Note that by default the pattern ".*" is used which will replicate everything in the bucket.

    Exclude from replication

    Select specific artifacts that you wish to block from replication.

    Zones

    You need to select a minimum of 2 zones, the local zone is automatically selected and is manditory.

  3. Continue adding your settings.

    Live S3 Architecture
    Figure 11. Live S3 Create a rule
    Priority Zone

    The Priority Zone is the zone which is most important, that which is most reliable or up to date. For example if your set up has a production zone and a disaster recovery zone then production would be more important and therefore your priority zone.

Connect to Underlying Buckets
Bucket name

The name of the underlying S3 bucket for this zone.

Access key

Credentials for the underlying bucket.

Secret access key

Further credentials for the underlying bucket.

Region

The region by which the bucket is located.

Endpoint URL

The endpoint by which the bucket can be accessed. This can be in any of the forms: <hostname>, <hostname>:<port>, http(s)://<hostname>, or http(s)://<hostname>:<port>.

Networking

Select whether or not you want to allow both inbound and outbound connections or limit to inbound connections only.

Create Rule

Click to create the rule.

5.2.1. Edit a rule

Once created, you can edit Replication Rules by clicking on the rule resource label.

Live S3 Architecture
Include in replication

Click the edit icon to change.

Exclude from replication

Click the edit icon to change.

Live S3 Architecture

Click the edit icon to make changes to Bucket.

Only the Access Key, Secret Access Key and Networking settings.
Live S3 Architecture

Click update to save any changes that you make.

5.3. Administration

Once configured, applications interact with the virtual buckets that have been configured.

Standard S3 API operations are replicated with strong consistency among the underlying buckets, and content associated with objects created is replicated between these buckets.

Use the Fusion Plugin for Live S3 to provide a LiveData environment, where applications can interact with any of the replicated buckets, and each bucket will provide access to the same content regardless of where change is initiated.

5.3.1. Consistency Check

Because applications that do not interact with the Live S3 environment via the Fusion Plugin for Live S3 can modify bucket content without coordination or replication, the product provides a consistency check feature. Use consistency check to report on any differences among the replicated buckets.

Known Issue
Performing consistency checks which yield large number of inconsistencies (e.g. hundreds of thousands) consume large amounts of memory (~1.5kb per inconsistency). When the number of inconsistencies is larger then ~1.3 million, then the server will crash with OutOfMemory error.
Workaround: if it’s known the buckets are widely inconsistent, perform the consistency check on prefix to reduce the scope, or perform the repair first.
Perform a consistency check
S3 data and eventual consistency

Live S3’s mechanism for dealing with data inconsistency is not instantaneous and operates under constraints imposed by the use of the S3 API. For this reason, all operations to make data consistent between zones run under the eventual consistency model. You may need to wait some time to see the results of a replica convergence.

Use this procedure to complete a check of consistency between one or more zones.

  1. Navigate to Replication tab.

  2. Select a rule on which to perform a consistency check.

  3. Click on the status panel and click on the check now button, then click on one of the available zones to be the source of truth.

    Live S3 Architecture
    Figure 12. Live S3 Consistency Check

    By clicking on a zone you see a graph of the differences between zones. If you select a zone that has data that is not available on the other zone(s), then the bar graphs will show green. Conversely, if the selected zone doesn’t contain some data that is on the other zone(s) then this will show as red on the bar.

    Objects

    Total number of objects.

    Size

    Total file size of replicated data under the rule.

    Directories

    Number of directories replicated under the rule.

    Download detailed report

    Click this link to download a JSON file containing all available data from the consistency check.

sample download report
{"inconsistencies":[{"differences":{"epam01-zone1":{"zoneName":"epam01-zone1","size":21,"directory":false}},"path":"test"},{"differences":{"epam01-zone1":{"zoneName":"epam01-zone1","size":58,"directory":false}},"path":"test"},{"differences":{"epam01-zone1":{"zoneName":"epam01-zone1","size":58,"directory":true}},"path":"test"},{"differences":{"epam01-zone1":{"zoneName":"epam01-zone1","size":58,"directory":true}},"path":"test"},{"differences":{"epam01-zone1":{"zoneName":"epam01-zone1","size":58,"directory":true}},"path":"test"},{"differences":{"epam01-zone1":{"zoneName":"epam01-zone1","size":58,"directory":false}},"path":"test"},{"differences":{"epam01-zone1":{"zoneName":"epam01-zone1","size":21,"directory":false}},"path":"test"},{"differences":{"epam01-zone1":{"zoneName":"epam01-zone1","size":21,"directory":false}},"path":"test"},{"differences":{"epam01-zone1":{"zoneName":"epam01-zone1","size":0,"directory":true}},"path":"test"}]}

5.3.2. Repair

Resolve inconsistencies among buckets with the Repair feature. Initiate a repair with a single REST API invocation, and access the status of the potentially long-running repair task at the location provided in response.

Make a repair

Use this procedure to make a repair when there are inconsistencies in data stored in your virtual buckets, between zones.

  1. Once you have a consistency report that shows there is inconsistency between zones, and you have selected the zone that represents the source of truth, then click the "Repair" button.

    Live S3 Architecture
    Figure 13. Live S3 Repair

  2. Select from the available options, as described below.

    Add missing objects

    [checkbox] When ticked, data that exists at the source of truth but not in other zones will be copied into place to ensure that the data is consistently available.

    Remove extraneous objects

    [checkbox] When ticked, data that exists on nodes that are not the source of truth but not at the source of truth are removed, to ensure that all zones match the source of truth. Use with care, given that this operation is destructive.

    Update objects which differ

    [checkbox] When ticked, a repair will ensure that files from the source of truth zone will overwrite the same files on different zones, should the files not match.

  3. Click Repair. The repair operation will now start. The UI will report when the repair has completed. You should perform another consistency check to ensure that the repair was successful.

5.3.3. Changing the time zone

Logs use UTC timezone by default but this can be manually altered through log4j configuration if required. To alter the timezone the xxx.layout.ConversionPattern property needs to be overwritten.

log4j.appender.xxxxxlog.layout.ConversionPattern=%d{ISO8601}{UTC} %p %c - %t:[%m]%n

{UTC} can be replaced with, for example {GMT} or {ITC+1:30}. If offsetting from a timezone, + or - can be used, hours must be between 0 and 23, and minutes must be between 00 and 59.

This property is located in /etc/wandisco/live-s3-proxy/log4j.properties. After updating the file, the s3proxy-server needs to be restarted for the changes to take effect.

5.4. Troubleshooting

Observe information in the log files generated for the WANdisco Fusion server and the Fusion Plugin for Live S3 to troubleshoot issues at runtime. Exceptions or log entries with a SEVERE label may represent information that can assist in determining the cause of any problem.

As a distributed system, the Fusion Plugin for Live S3 will be impacted by the operation of the underlying S3 endpoints with which it communicates. You may also find it useful to review log or other information from these endpoints.

6. Reference Guide

6.1. API

Fusion Plugin for Live S3 offers increased control and flexibility through a RESTful (REpresentational State Transfer) API.

Below are listed some example calls that you can use to guide the construction of your own scripts and API driven interactions.

Note the following:

  • All calls use the base URI:

    http(s)://<server-host>:8082/plugin/s3proxy/

  • The internet media type of the data supported by the web service is application/xml.

  • The API is hypertext driven, using the following HTTP methods:

Type Action

POST

Create a resource on the server

GET

Retrieve a resource from the server

PUT

Modify the state of a resource

DELETE

Remove a resource

6.1.1. S3 API Support

Broad support for the S3 API is offered by the Fusion Plugin for Live S3. Details of the specific operations that clients of the Fusion Plugin for Live S3 can use are provided below. Of note are:

  • Operations that use object identifiers (not keys) being only partially supported

  • POST operations that are partially supported

Operations on Services
Operations on Buckets
Operations on Objects

6.1.2. Consistency Check via the API

Because applications that do not interact with the Live S3 environment via the Fusion Plugin for Live S3 can modify bucket content without coordination or replication, the product provides a consistency check feature. Use consistency check to report on any differences among the replicated buckets.

A consistency check is a potentially long-running task, initiated with a specific REST operation. There are 2 consistency check options:

  1. Provide the path associated with the virtual bucket when initiating the check to specify which virtual bucket to review:

    curl -v -i -X POST "http://hostname:8082/plugin/s3proxy/cc?vbucket=vbucket"* About to connect() to hostname port 8082 (#0)
*   Trying 10.6.120.134...
* Connected to hostname (10.6.120.134) port 8082 (#0)
> POST /plugin/s3proxy/cc?vbucket=vbucket HTTP/1.1
> User-Agent: curl/7.29.0
> Host: hostname:8082
> Accept: /
>
< HTTP/1.1 202 Accepted
HTTP/1.1 202 Accepted
< Content-Location: http://hostname:8082/fusion/task/09676c6f-f13a-11e8-91a9-0e4f03e0bb67
Content-Location: http://hostname:8082/fusion/task/09676c6f-f13a-11e8-91a9-0e4f03e0bb67
< Content-Length: 0
Content-Length: 0
Server: Jetty(6.1.26)

  2. To check a particular directory under the virtual bucket associated with a replication path, add the ccpath to the end of the operation:

    # curl -i -X POST "http://localhost:8082/plugin/s3proxy/cc?path=/repl-path&vbucket=vbucket&ccpath=dir1/" Enter
HTTP/1.1 202 Accepted
Content-Location: http://localhost:8082/fusion/task/84b417f3-ec60-11e7-aa4b-0242ac120002
Content-Length: 0
Server: Jetty(6.1.26)

Access the status of the consistency check with the cc task Id generated during consistency check operation at the location referenced:

# curl -v -X GET "http://localhost:8082/fusion/task/09676c6f-f13a-11e8-91a9-0e4f03e0bb67" Enter
> GET /fusion/task/09676c6f-f13a-11e8-91a9-0e4f03e0bb67 HTTP/1.1
> User-Agent: curl/7.29.0
> Host: localhost:8082:8082
> Accept: /
>
< HTTP/1.1 200 OK
< Content-Length: 1393
< Content-Type: application/xml
< Server: Jetty(6.1.26)
<
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><task><taskId>09676c6f-f13a-11e8-91a9-0e4f03e0bb67</taskId><timeCreated>1543209220319</timeCreated><creatorNodeId>e8987b58-e5cb-49f4-8b7d-e429c79cae6e</creatorNodeId><timeUpdated>1543209225289</timeUpdated><isDone>true</isDone><aborted>false</aborted><properties><entry><key>CC_REPORT_PATH</key><value>MetadataKeyImpl. fileStatus: ExtendedFileStatus. path: /wandisco/live-s3-plugin/vBucket/vbucket/.fusion/2ae4848f-f135-11e8-91a9-0e4f03e0bb67/metadata/09676c6f-f13a-11e8-91a9-0e4f03e0bb67/cc-report, length: 844, isDir: false, owner: null, group: null, perms: unknown, checksum: , acls: SerializableAclList{ACLs unsupported}, usernameTranslated: false, checkPosixPermissions: false, checkOwnerAndGroupName: false, replication: 3, blocksize: 134217728, modificationTime: -1, accessTime: -1, xAttrs: {}</value></entry><entry><key>TOTAL_INCONSISTENCIES_FOUND</key><value>1</value></entry><entry><key>TASK_TYPE</key><value>S3PROXY_CONSISTENCY_CHECK</value></entry><entry><key>LOCAL_COMPLETE</key><value>1543209225280</value></entry><entry><key>DATA_NOT_PULLED_ZONES</key><value>[]</value></entry><entry><key>LOCAL_START</key><value>1543209220319</value></entry><entry><key>CONSISTENCY_CHECK_STATUS</key><value>INCONSISTENT</value></entry></properties><previousTask xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:nil="true"/></task>

To access the consistency check report use the vbucket name which is used during consistency check operation.

# curl -i -X GET "http://localhost:8082/plugin/s3proxy/cc/report/<vbucket-name>?withConsistencyReport=true" Enter
HTTP/1.1 200 OK
< Content-Length: 1031
Content-Length: 1031
< Content-Type: application/xml
Content-Type: application/xml
< Server: Jetty(6.1.26)
Server: Jetty(6.1.26)

<
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><consistencyReport><path>/wandisco/live-s3-plugin/vBucket/vbucket/</path><state>INCONSISTENT</state><taskId>09676c6f-f13a-11e8-91a9-0e4f03e0bb67</taskId><checksumMethod>MD5</checksumMethod><summary><zonesummary><zoneId>zone1-marip02-vm4</zoneId><objectsSize>842</objectsSize><directoriesCount>0</directoriesCount><objectsCount>1</objectsCount></zonesummary><zonesummary><zoneId>xone2-marip02-vm5</zoneId><objectsSize>1684</objectsSize><directoriesCount>0</directoriesCount><objectsCount>2</objectsCount></zonesummary></summary><inconsistencies><inconsistency><uri>vbucket</uri><inconsistencyDetails><inconsistencyDetail><zoneId>xone2-marip02-vm5</zoneId><key>testfile</key><length>842</length><isDir>false</isDir></inconsistencyDetail></inconsistencyDetails></inconsistency></inconsistencies><noInconsistencies>1</noInconsistencies><lastCheck>1543209225289</lastCheck><localStartTime>1543209220319</localStartTime><localEndTime>1543209225280</localEndTime></consistencyReport>

6.1.3. Repair via the API

Resolve inconsistencies among buckets with the Repair feature. Initiate a repair with a single REST API invocation, and access the status of the potentially long-running repair task at the location provided in response.

Specify the ccTaskId and the name of the zone that will be used as the source of truth for repair. Repair ensures that each replicated bucket has the same content as the source of truth. This may introduce object deletion and creation in all other replicated zones, so use this feature with care.

There are 3 types of repair:

Recursive

True A repair with recursive true will repair the contents of any subfolders in the target zone.
False A repair with recursive false will not repair the contents of any subfolders in the target zone.

Replace

True A repair with Replace true will overwrite all duplicate files and directories in target zone.
False A repair with Replace false will not overwrite the duplicate files and directories in target zone.

Preserve

True A repair with Preserve true will not remove any data that exists in target zone.
False A repair with Preserve false will remove all the data that exists in target zone.

To start a repair you need to add the cc vbucket generated from the consistency check e.g.:

# curl -i -X PUT "http://localhost:8082/plugin/s3proxy/repair/<virtual-bucket-name>?srcZone=<Zone_name>&preserve=true&recursive=true&replace=false" Enter
HTTP/1.1 202 Accepted
Content-Location: http://localhost:8082/plugin/fusion/task/0088e14d-ec62-11e7-aa4b-0242ac120002
Content-Length: 0
Server: Jetty(6.1.26)

To access the status of the repair task at the location referenced e.g.:

# curl -i -X GET "http://localhost:8082/fusion/task/0088e14d-ec62-11e7-aa4b-0242ac120002" Enter
HTTP/1.1 200 OK
Content-Length: 679
Content-Type: application/xml
Server: Jetty(6.1.26)

<?xml version="1.0" encoding="UTF-8" standalone="yes"?><task><taskId>0088e14d-ec62-11e7-aa4b-0242ac120002</taskId><timeCreated>1514548109259</timeCreated><creatorNodeId>400b7699-56d2-44c5-b07c-f42a760b966f</creatorNodeId><timeUpdated>1514528530417</timeUpdated><isDone>true</isDone><aborted>false</aborted><properties><entry><key>TASK_TYPE</key><value>REPAIR_TASK</value></entry><entry><key>REPAIR_STATUS</key><value>COMPLETED</value></entry><entry><key>LOCAL_COMPLETE</key><value>1514548109903</value></entry><entry><key>LOCAL_START</key><value>1514548109259</value></entry></properties><previousTask xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:nil="true"/></task>

6.1.4. Dynamic configuration of Virtual buckets

Virtual buckets (vbucket) can be added, listed, modified or deleted from the proxy-plugin-site.xml using the REST APIs below. A TaskId, will be returned, indicating the status of the task e.g. failed, completed or still running. Exclusion and inclusion rules can be used to determine what is replicated.

  • Need to change the ownership as installed in Fusion.

  • The data will not be replicated to other zones if it matches with the exclusion rule.

  • The data will be replicated to other zones when it matches with the inclusion rule. The default inclusion rule will be ".*".

  • When the exclusion and inclusion rules are same, the priority will be given to the exclusion rule.

Add virtual buckets via the API

Follow these steps to add virtual buckets via the API.

  1. Create replication directory in both zones fusion fs -mkdir /Newrepl

  2. Set a permission and ownership for the directory (/Newrepl) from both zones, e.g.:

    fusion fs -chmod 777 /Newrepl
fusion fs -chown fusion:fusion /Newrepl

  3. Get the membership id, e.g.:

    curl -v -X GET "http://<localhost>:<port>/fusion/fs"

An example xml file with the details of virtual bucket to be added in the proxy-plugin-site.xml:

<vbucket name="vbucket1" repl-path="/Newrepl">
   <bucket name="bucket-a">
     <zonename>zone1</zonename>
     <accesskey></accesskey>
     <secretaccesskey>*</secretaccesskey>
     <region>us-east-1</region>
     <endpoint-url>s3.amazonaws.com</endpoint-url>
     <provider>AWS_S3</provider>
   </bucket>
   <bucket name="bucket-b">
     <zonename>zone2</zonename>
     <accesskey></accesskey>
     <secretaccesskey></secretaccesskey>
     <region>eu-west-1</region>
     <endpoint-url>s3-eu-west-1.amazonaws.com</endpoint-url>
     <provider>AWS_S3</provider>
   </bucket>
   <exclusions>
     <exclusion>ex.</exclusion>
   </exclusions>
   <inclusions>
     <inclusion>in.*</inclusion>
   </inclusions>
</vbucket>

This REST API adds virtual buckets to the proxy-plugin-site.xml.

# curl -v -i -X PUT -H "Content-Type: application/xml" -d @config.xml "http://localhost:8082/plugin/s3proxy/config/vbucket?membershipId=57e3ab9a-c3f7-46a4-9ee1-cf9eb8284b1d" Enter
Connected to localhost (10.6.120.125) port 8082 (#0)
> PUT /plugin/s3proxy/config/vbucket?membershipId=57e3ab9a-c3f7-46a4-9ee1-cf9eb8284b1d HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.21 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: localhost:8082
> Accept: /
> Content-Type: application/xml
> Content-Length: 1008
>
< HTTP/1.1 202 Accepted
HTTP/1.1 202 Accepted
< Content-Location: http://localhost:8082/fusion/task/1eb5d6da-c7c8-11e8-be61-e2dcf520c05b
Add virtual bucket status
# curl -X GET http://localhost:8082/fusion/task/1eb5d6da-c7c8-11e8-be61-e2dcf520c05b | xmllint --format - Enter
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
111 888 111 888 0 0 159k 0 -::- -::- -::- 216k
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<dsmTask>
  <taskId>1eb5d6da-c7c8-11e8-be61-e2dcf520c05b</taskId>
  <timeCreated>1538652295742</timeCreated>
  <creatorNodeId>c0c4e839-b44c-42ba-bed8-6fec02c23384</creatorNodeId>
  <timeUpdated>1538652296152</timeUpdated>
  <isDone>true</isDone>
  <aborted>false</aborted>
  <properties>
    <entry>
      <key>PROPOSAL_SENT</key>
      <value>true</value>
    </entry>
    <entry>
      <key>TASK_TYPE</key>
      <value>UPDATE_VBUCKET_CONFIG_BY_PATH</value>
    </entry>
  </properties>
  <previousTask>1eb4c569-c7c8-11e8-be61-e2dcf520c05b</previousTask>
  <message>VBucketUpdateProposal [virtualBucketDTO=com.wandisco.s3proxy.dto.VirtualBucketDTO@2e767409, operation=CREATE, path=/repl1, taskIdentity=1eb5d6da-c7c8-11e8-be61-e2dcf520c05b, proposerNodeId=c0c4e839-b44c-42ba-bed8-6fec02c23384, origin=zone2, vBucketName=vbucket]</message>
  <dsmId>1eb4c567-c7c8-11e8-be61-e2dcf520c05b</dsmId>
</dsmTask>
List virtual buckets via the API

This REST API gives a listing of virtual buckets configured in the proxy-plugin-site.xml.

curl -v -X GET http://localhost:8082/fusion/task/09676c6f-f13a-11e8-91a9-0e4f03e0bb67
> GET /fusion/task/09676c6f-f13a-11e8-91a9-0e4f03e0bb67 HTTP/1.1
> User-Agent: curl/7.29.0
> Host: localhost:8082:8082
> Accept: /
>
< HTTP/1.1 200 OK
< Content-Length: 1393
< Content-Type: application/xml
< Server: Jetty(6.1.26)
<
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><task><taskId>09676c6f-f13a-11e8-91a9-0e4f03e0bb67</taskId><timeCreated>1543209220319</timeCreated><creatorNodeId>e8987b58-e5cb-49f4-8b7d-e429c79cae6e</creatorNodeId><timeUpdated>1543209225289</timeUpdated><isDone>true</isDone><aborted>false</aborted><properties><entry><key>CC_REPORT_PATH</key><value>MetadataKeyImpl. fileStatus: ExtendedFileStatus. path: /wandisco/live-s3-plugin/vBucket/vbucket/.fusion/2ae4848f-f135-11e8-91a9-0e4f03e0bb67/metadata/09676c6f-f13a-11e8-91a9-0e4f03e0bb67/cc-report, length: 844, isDir: false, owner: null, group: null, perms: unknown, checksum: , acls: SerializableAclList{ACLs unsupported}, usernameTranslated: false, checkPosixPermissions: false, checkOwnerAndGroupName: false, replication: 3, blocksize: 134217728, modificationTime: -1, accessTime: -1, xAttrs: {}</value></entry><entry><key>TOTAL_INCONSISTENCIES_FOUND</key><value>1</value></entry><entry><key>TASK_TYPE</key><value>S3PROXY_CONSISTENCY_CHECK</value></entry><entry><key>LOCAL_COMPLETE</key><value>1543209225280</value></entry><entry><key>DATA_NOT_PULLED_ZONES</key><value>[]</value></entry><entry><key>LOCAL_START</key><value>1543209220319</value></entry><entry><key>CONSISTENCY_CHECK_STATUS</key><value>INCONSISTENT</value></entry></properties><previousTask xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:nil="true"/></task>
Modify virtual buckets via the API

This REST API modifies the virtual buckets configured in the proxy-plugin-site.xml. The virtual bucket details for modification has to be given as an xml file, from which the details will be taken and specified virtual bucket will be modified in the proxy-plugin-site.xml.

An example xml file with details of virtual bucket to be modified in the proxy-plugin-site.xml:

<vbucket name="vbucket" repl-path="/Newrepl >
   <bucket name="bucketNew1">
     <zonename>zone1</zonename>
     <accesskey></accesskey>
     <secretaccesskey>*</secretaccesskey>
     <region>us-east-1</region>
     <endpoint-url>s3.amazonaws.com</endpoint-url>
     <provider>AWS_S3</provider>
   </bucket>
   <bucket name="bucketNew2">
     <zonename>zone2</zonename>
     <accesskey></accesskey>
     <secretaccesskey></secretaccesskey>
     <region>eu-west-1</region>
     <endpoint-url>s3-eu-west-1.amazonaws.com</endpoint-url>
     <provider>AWS_S3</provider>
   </bucket>
   <exclusions>
     <exclusion>exfile.</exclusion>
     <exclusion>./.fusion/.</exclusion>
   </exclusions>
   <inclusions>
     <inclusion>infile.*</inclusion>
   </inclusions>
</vbucket>
# curl -v -i -X POST -H "Content-Type: application/xml" -d @config.xml "http://localhost:8082/plugin/s3proxy/config/vbucket" Enter
> POST /plugin/s3proxy/config/vbucket HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.21 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: localhost:8082
> Accept: /
> Content-Type: application/xml
> Content-Length: 1009
>
< HTTP/1.1 202 Accepted
HTTP/1.1 202 Accepted
< Content-Location: http://localhost:8082/fusion/task/9dc0d860-c7b1-11e8-be61-e2dcf520c05b
Content-Location: http://localhost:8082/fusion/task/9dc0d860-c7b1-11e8-be61-e2dcf520c05b
Modify virtual bucket status
# curl -X GET "http://localhost:8082/fusion/task/9dc0d860-c7b1-11e8-be61-e2dcf520c05b" | xmllint --format - Enter
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 908 100 908 0 0 218k 0 -::- -::- -::- 295k
<dsmTask>
  <taskId>9dc0d860-c7b1-11e8-be61-e2dcf520c05b</taskId>
  <timeCreated>1538642630460</timeCreated>
  <creatorNodeId>c0c4e839-b44c-42ba-bed8-6fec02c23384</creatorNodeId>
  <timeUpdated>1538642630549</timeUpdated>
  <isDone>true</isDone>
  <aborted>false</aborted>
  <properties>
  <entry>
    <key>PROPOSAL_SENT</key>
    <value>true</value>
  </entry>
  <entry>
    <key>TASK_TYPE</key>
    <value>UPDATE_VBUCKET_CONFIG_BY_PATH</value>
  </entry>
  </properties>
  <previousTask xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:nil="true"/>
  <message>VBucketUpdateProposal [virtualBucketDTO=com.wandisco.s3proxy.dto.VirtualBucketDTO@22067042, operation=UPDATE, path=/repl, taskIdentity=9dc0d860-c7b1-11e8-be61-e2dcf520c05b, proposerNodeId=c0c4e839-b44c-42ba-bed8-6fec02c23384, origin=zone2, vBucketName=vbucket]</message>
  <dsmId>3155fd27-c79a-11e8-8a9d-5666138c2544</dsmId>
</dsmTask>
Delete virtual buckets via the API

This REST API deletes the virtual buckets configured in the proxy-plugin-site.xml.

  • If deleteRD=false, the virtual bucket will be removed from proxy-plugin-site.xml but the repl directory will remain in the UI.

  • If deleteRD=true, the virtual bucket will be removed from proxy-plugin-site.xml and from the UI.

# curl -v -i -X DELETE "http://localhost:8082/plugin/s3proxy/config/vbucket?vbucket=bucket&deleteRD=true" Enter
* Connected to localhost (10.6.121.44) port 8082 (#0)
> DELETE /plugin/s3proxy/config/vbucket?vbucket=bucket&deleteRD=true HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.21 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: localhost:8082
> Accept: /
>
< HTTP/1.1 202 Accepted
HTTP/1.1 202 Accepted
< Content-Location: http://localhost:8082/fusion/task/1c11fdd8-c604-11e8-8af8-82b20b0178bd
Content-Location: http://localhost:8082/fusion/task/1c11fdd8-c604-11e8-8af8-82b20b0178bd
Delete virtual bucket status
# curl -X GET http://localhost:8082/fusion/task/1c11fdd8-c604-11e8-8af8-82b20b0178bd | xmllint --format - Enter
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
107 861 107 861 0 0 300k 0 --:--:-- --:--:-- --:--:-- 420k
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<dsmTask>
  <taskId>b50f319e</taskId>
  <timeCreated>1538649112012</timeCreated>
  <creatorNodeId>c0c4e839-b44c-42ba-bed8-6fec02c23384</creatorNodeId>
  <timeUpdated>1538649112087</timeUpdated>
  <isDone>true</isDone>
  <aborted>false</aborted>
  <properties>
  <entry>
    <key>PROPOSAL_SENT</key>
    <value>true</value>
  </entry>
  <entry>
    <key>TASK_TYPE</key>
    <value>UPDATE_VBUCKET_CONFIG_BY_PATH</value>
  </entry>
  </properties>
  <previousTask xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:nil="true"/>
  <message>VBucketUpdateProposal [virtualBucketDTO=null, operation=DELETE, path=/repl1, taskIdentity=b50f319e, proposerNodeId=c0c4e839-b44c-42ba-bed8-6fec02c23384, origin=zone2, vBucketName=bucket]</message>
  <dsmId>ca3038fa-c7a1-11e8-825d-5666138c2544</dsmId>
</dsmTask>
1. The specified API contains payload as xml configuration with the dependency on destination bucket which has to be in the same region as the source bucket.
2. The specified API contains payload as xml configuration with the dependency on destination bucket which has to be in the different region from the source bucket.
3. PUT Bucket Notification has payload as xml configuration containing topic arn which has the region that has to be same as the bucket region.
4. Deletion of specific object versions is not supported because independent S3 buckets assign different version identifiers to replicated instances of an object.
5. Deletion of tagging for a specific object version is not supported because independent buckets assign different version identifiers to replicated instances of an object.
6. POST is an alternative to PUT for browser-based uploads. The Fusion Plugin for Live S3 does not support Post object replication with virtual bucket.
7. POST Object restore allows restoration of a temporary copy of an archived object. It is not supported by the Fusion Plugin for Live S3.