SVN: Match to projects and repositories.
MSP Transaction Journal: Equivalent of seven days of changes.

Estimating your disk requirements can be very difficult and there’s no perfect system for making an accurate estimation. Some organizations monitor their repository growth over a period of time and use an extrapolation as a guide. This method works best if your organization is unlikely to see the addition of large new projects that instantly introduce large amounts of extra repository data.
You need to quantify some elements of your deployment:

Provision enough disk space for /opt/wandisco/multisite-plus/replicator/database to cover the expected number of commits for four hours of peak usage, or a busy 3-day weekend (whichever is larger).

MSP replication does not provide a replacement for a periodic backup policy. Repository replication is not a backup solution. Unlike backups, replicated repositories are not static snapshots that incoming changes can’t negatively impact. Any errors written to single copy of a repository may be replicated to all copies. A periodic backup may safeguard against such errors being unrecoverable.

MSP itself is a replicated application. As such, it contains critical up-to-date data and meta-data. While a backup of this information might be useful for a recovery procedure, it is critical to Never restore Subversion MultiSite Plus data from a backup directly into operation. Doing this could cause severe and unrecoverable damage to the replication ecosystem.

See the Release Notes for which operating systems are supported for your MSP version.

Any that are compatible with local SVN servers.

Normally we recommend that all hook scripts be duplicated exactly on all repository replicas however in some circumstances this is not possible. See hooks for more information.

Ensure hard and soft limits are set to 64000 or higher. Check with the ulimit or limit command.

Supported file systems include:

Write barriers should always be enabled.

Replicator logs should be on a journaling file system, for example, ext3 on Linux or VXFS from Veritas.

ext4 can be used as your journaling file system, although it must be configured appropriately. See Using Ext4 filesystem for journaling.

Install the JRE / JDK version shown in the Release Notes for your MSP version.

See the MSP Release Notes for which version is needed.

Set up and configuration requires access through a browser. The browsers listed in the Release Notes are known to work.

We support the implementation of Kerberos for single sign-on. By default Kerberos requires that stronger encryption algorithms be available than are currently provided by default in Java. This is so that Oracle can avoid the complications that arise from countries that place import restrictions on encryption technology.

The stronger encryption algorithms are available as an optional download where the user takes responsibility for compliance with the local laws.
For Java 7: JCE Unlimited Strength Jurisdiction Policy
For Java 8: JCE Unlimited Strength Jurisdiction Policy
When downloaded, extract the contents to (and overwrite the existing contents of) the Java security library directory on all nodes, e.g:

Several ports are reserved by MSP. You should do a port survey of all of your servers to determine which ports are already in use and then select appropriate ports for MSP that will not conflict. You can change these ports after installation, but it is not straightforward and so it is best to do it during installation. To change the ports after installation follow the instruction in Update a node’s properties. The default values suggested during the installation are the following:

If you have a virus scanner running on the system housing your repositories and replicator you should:

In general, virus scanners don’t filter ports: firewalls do that. However, some Anti-Virus products contain firewall-like filtering capabilities - if this is the case in your platform, you should make sure that you understand what impact it could have on your MSP deployment.

MSP requires full network connectivity between all nodes. Ensure that each node’s server is able to communicate with all other servers that will host nodes in your MSP installation on all ports assigned (see above).

Set up IPsec tunnel, and ensure WAN connectivity.

Ensure that your VPN doesn’t reset persistent connections for MSP.

Put your WAN through realistic load testing before going into production. You can then identify and fix potential problems before they impact productivity.

We strongly recommend using Fully Qualified Domain Names (FQDN), not IP addresses, and IP addresses cannot be used if SSL is going to be implemented. It is complex to change from IP addresses to FQDNs after the product is installed and so you should use FQDNs anywhere the product or installation requests IP addresses.

To prevent outages based on an inability to convert from FQDN’s to IP addresses, we suggest you make sure your DNS service robust in the face of failures (it should be replicated).

MSP provides a limited system for monitoring system disk space available. This monitor is intended only to provide a deployment with a last line of defense against running out of storage space. We recommend that you deploy a system-wide monitor that ensures that you quickly identify potential problems that could impact services.

Read our recommendations for system-wide monitoring tools

The use of a correctly configured load balancer can greatly benefit performance in situations where there could be large numbers of concurrent SVN users.
The load-balancer should direct session requests to the same server based solely on the source IP address of a packet. Once the choice of server has been made the load-balancer should only change to a different server if the original chosen server is no longer communicable.

Therefore, MSP requires that any load balancing solution has the following features:

You should deploy a robust implementation of NTP, including monitoring as NTP will not auto-correct if the time is too far off-set from the current time. This is an important requirement because without nodes being in sync there are a number of problems that can occur.

Take careful note of this requirement as many installation problems are caused by running applications with unsuitable or incompatible system accounts.
In most cases you must install MSP with Apache’s username, e.g. apache.

Read a detailed explanation of why this is required: System accounts for running MultiSite.

Read our Replication Strategy Guide for information on how to set up and optimize your replication - Replication Strategy.

To ensure best performance, make sure that MSP can deliver the content of a commit to another local node. MSP normally requires that content reach at least one other node for data integrity purposes. As the content normally represents the bulk of the data in a commit, having a second local node available will improve performance. Furthermore, you may wish to use our scheduling system to modify the voter roles so a proposal may be accepted by local voter nodes during regular working hours. If you need more help with setting up the most efficient deployment please get in touch with our support team.

MSP is supplied through a licensing model based on the number of nodes and users. WANdisco generates a license file matched to your agreed usage model.

Repository replicas must be byte-for-byte mirrors of each other. This stringent requirement did not apply to SVN MultiSite 4.x: the previous tests for whether replicas are identical are not sufficient for FSFSWD replication (see 4.2 vs Plus, below). As a result, you need to recreate your replica repositories using a nominated master repository:

You may need to plan the exact process of copying repositories so that it is practical and achievable. Many production repositories take a long time to checksum. If you are in any doubt about handling the process, talk to your WANdisco account manager.

MSP opens up more options because MultiSite is no longer running as proxy and options that were previously not compatible with MultiSite now are compatible.

Also Access Control Plus (ACP) is the newer product version of Access Control. It provides for Team management and generates authentication, if needed, and authorization files for use with Apache or svnserve. ACP uses MSP to deliver these files. Since ACP is no longer a proxy, things to consider:

This section gives an example Apache configuration. In Apache’s config file, httpd.conf:

If you need to capture a complete record of installer messages, warnings, errors, then you need to set the LOG_FILE environment variable before running the installer. Run:

This file’s permissions must allow being appended to by the installer. Ideally, the file should not already exist (or it should exist and be empty) and its directory should enable the account running the installer to create the file.

If you are installing MSP where the account access auditing functionality for ACP is required then the following information will be required during installation:

For more information about installing Account Access Auditing, see the ACP installation instructions and How to do a manual set up for audit logging.

For information on how to how to upgrade the ACP Flume sender delivered with ACP1.9 and how to set up SSL, see the How to upgrade the ACP sender delivered with ACP1.9 (and above) and how to set up SSL.

The platform you are using will affect which commands you need to use to, for example, start and stop MSP. As of MSP 1.9.5 SystemD commands are used for platforms that support only SystemD (without compatibility mode). On all other platforms the SysV commands are used. See the list below for which commands you need to use:

If you have any queries about your specific set up please contact support.

If you wish to run the tarball installer please run the same script as above but with following extra parameters:

Setting sources

This value sets the sources that flume will monitor: acpSender.sources =

Setting log locations

Settings that apply to SVNServe and Webdav:

The system user that runs MSP MUST have permissions to read all the files that you configure to monitor.
This can be particularly tricky since, for example, /var/log/httpd is normally set to 0700. One way to work-around this would be to set the group of the /var/log/httpd directory to the same group as MSP runs as and then change its permissions to be 0750. This enables proper monitoring with minimal security impact.

For more information see the ACP manual’s section on configuring the Flume Receiver.

Back to Contents - Back to beginning of section

Back to Contents - Back to beginning of section

The following table clearly indicates which FSFS formats support each of the new SVN 1.9 features:

Subversion 1.9 FSFS-backed repositories create, by default, a new on-disk file for each revision added to the repository. Having thousands of these files present on your Subversion server, even when housed in separate shard directories, can be inefficient. Firstly, the OS has to reference many different files over a short period of time, leading to inefficient use of disk caches and, as a result, more time spent seeking across large disks. Because of this, Subversion pays a performance penalty when accessing your versioned data.

A second problem is that filesystems allocate disk space, each file claims more space (from 2 to 16 kilobytes per file) on the disk than it actually uses. This gives FSFS_backed repositories a per-revision disk usage penalty, and the penalty can be noticeable on repositories that have extremely large numbers of small files

Back in Subversion 1.7 (using FSFS format 4), there was support for revision packing, but not for revprop packing and so a packed shard had unpacked revprops. Later formats expect revprops to be packed if revisions are packed. Upgrading format 4 involves packing all the revprops corresponding to the packed revisions, reading those revprops, one file per revision, and writing the revprops, a few files per shard. So if a large number of revisions are packed then the runtime is likely to be proportional to the number of revisions.

Another factor that can delay the process is repository write-lock, although upgrade is no different to commit in this respect.

Note that the upgrade of an unpacked repository is fast as no revprops need to be packed.

Upgrade changes the file db/format, with 1.9 you can use:

One other thing that can cause upgrade to take a long time is some other process holding a write lock:

Back to Contents - Back to beginning of section

Example: service svn-multisite-plus restart

Example: systemctl start wdmsp.target

Note: To obtain version information for MSP on a SystemD governed system, please execute /opt/wandisco/svn-multisite-plus/bin/svn-multisite-plus version.

You can also modify other properties in the application.properties configuration file. By default it is located in /opt/wandisco/svn-multisite-plus/replicator/properties/application.properties.

There are two configurable properties that you can modify as part of fine-tuning an MSP deployment. They are provided to allow you to balance best possible performance against the tolerance of a poor WAN connectivity.

socket.timeout

This is the time, in milliseconds, that a read() call on a socket will wait before timing out. Default value is 15 minutes (90,000 milliseconds).

content.pull.timeout

The content pull timeout sets how long the Content Distribution system will wait for new content to be pulled fully over from a remote node. The default value is 30 seconds (30,000 milliseconds). This default is set on the assumption that there are no problems with the deployment’s WAN connectivity.

content.max.idle.time

This should be set to the amount of time, in milliseconds, that your networking (router) infrastructure will close an idle connection.

See the content disribution policy section for more information.

By default MSP’s database directory (/opt/wandisco/svn-multisite-plus/replicator/database) is monitored - this is the location of MSP’s prevayler database where all data and transactions files for replication are stored.

This built-in monitor runs on all nodes. To add a new monitor click Add. Any additional monitors that you set up will monitor on a per-node basis. Monitors are not replicated so a monitor set up on one node is not applied to any other node.

As well as MSP’s own database folder, there are several directories that might grow very large and potentially consume all available file space.

Consider monitoring the following MSP directories:

/opt/wandisco/svn-multisite-plus/replicator/content
/opt/wandisco/svn-multisite-plus/logs
/opt/wandisco/svn-multisite-plus/replicator/logs

Also monitor /path/to/authz. If you are using Authz to manage authorization and your Authz file is situated on different file system from MSP, then you are recommended to set up monitoring of the Authz file.

For most deployments all these directories reside on the same file system, so that your default monitor would catch if any of them were consuming the available space. However, there are two scenarios where we’d recommend that you set up your own monitor for the content directory:

In both cases, after setting up a monitor you should set up a corresponding email notification that is sent if some or all of your monitor’s trigger conditions are met.

Create additional resource monitors using the following procedure:

If the disk space available to a monitored resource is less than the value you have for a SEVERE event, then the event is logged and the MSP replicator shuts down immediately. By default, the disk space is checked every 10 minutes. You can configure this interval in the applications.properties file:

When the MSP replicator is shutdown, no modifications to the the repositories at that site can be made. Read-only access can continue, depending on how severe the disk shortage becomes. You should immediately take action to make more disk space available. You can then start the replicator up again as soon as the resource that triggered the shutdown has enough available disk space not to shut down again.
To restart use the commands:

or if using SLES 12/SystemD:

The Gateways section stores your email (SMTP) server details. You can set up multiple gateways to ensure that the loss of the server doesn’t prevent alert notifications from being delivered.

The destinations section stores the email address for your notification recipients.

The template section stores email messages. You can create any number of templates, each with its own notification message, triggered by one of a number of trigger scenarios that are set up in the Rule section.

For example if an Admin wanted to receive an email when a new repository is deployed the the body text could be:

In the rules, the template needs to trigger in the event Deploy Repository Succeeded, and the event selected determines which variables are available in the message body - for more information see events and variables.

The Rule section defines which system event should trigger a notification, what message template should be used and which recipients should be sent the notification.

When writing email notification templates, you can insert variables into the template that will be interpolated when the notification is delivered. The variables available depend on the event type selected. The following variables are available for all event types:

{node} - This returns the node name.

{timestamp} - This returns the time at which the event is received (not the time at which the notification is delivered).

{event} -This returns the raw dump of the event.

All events are now listed along with a brief description and the additional variables available for each specific event:

You can modify all account details other than the account name itself. If you need to change the account name then add the new account name and then remove the old one.

MSP supports the use of LDAP authorities for managing admin accounts.

When connecting MSP to available LDAP authorities it is possible to classify the authority as Local, i.e. specific to the node in question or not. If not local, the authority details are replicated to the other nodes within the replication network.

It’s possible to run multiple LDAP authorities that are of mixed type, i.e. using some local authorities along with other authorities that are shared by all nodes. When multiple authorities are used, it’s possible to set what order they are checked for accounts. This enables you to prefer specific account names come from specific LDAP authorities in case your organization has account name collisions between LDAP authorities.

The standard settings are supported for each configured LDAP authority: URL, search base and filter and bind account credentials. Note that the bind account’s password cannot be one-way encrypted using a hash function because it must be sent to the LDAP server in plain text, so for this reason the bind account should be a low privilege user with just enough permissions to search the directory for the account being authenticated. Anonymous binding is permitted for those LDAP servers that support anonymous binding.

It is possible to create single-node replication groups. No replication takes place when you have a single node replication group. In a single node replication group the node must be an Active Voter. All other features work normally.

You may have a single-node configuration in the following situations:

Note: When changing a single node replication group into a multi-node replication group the only helper node will be the original single node. During the synchronization period the repositories in the new replication group will be read-only.

To create a single-node replication group follow the same steps as here. You can ignore the warnings, particularly the one that states A replication group should contain at least 2 nodes, which is applicable if you intend to replicate your repository data.

There may be circumstances in which you need to change the schedule managing node, for example if the current node will be undergoing maintenance and therefore will be unable to rotate the voting population at the normal time.
To change which node is the managing node (schedule manager), click on the node you want to be the new managing node and select Make Schedule Manager.

The schedule, as it is displayed by the UI, is an ideal representation of where the schedule would place the roles, assuming that the nodes are all properly connected to each other, and that you have not stopped the scheduling. If one or more nodes that are members of the replication group are down or unable to communicate (e.g. a network partition event has occurred), then the schedule could be in the process of moving, or be unable to move, if there are a sufficient number of outages.
Therefore it is very important to verify that all nodes that are members of this replication are properly connected and functioning and that the schedule has not been manually stopped before making operational decisions based on the information displayed on the scheduling page.

We will improve this function in future product versions.

If you need to stop any and all scheduled rotations, e.g. in an emergency to prevent losing quorum:

Back to Contents - Back to beginning of section

Apache log4j provides Appenders which are primarily responsible for printing logging messages to different destinations such as consoles, files, sockets, NT event logs, etc.

Appenders always have a name so that they can be referenced from Loggers.

For more information about setting up appenders, see the Apache documentation.

We strongly recommend that you work with WANdisco Support team before making any significant changes to your logging.

If there’s doubt, contact the file owner and ask about the file’s status. When you have identified a stale lock, it can be removed via the svnadmin rmlocks command. Note: the actions taken by the svnadmin rmlocks command are replicated, so specified locks will be removed from all repository replicas.

You can have consistency checks triggered automatically on a predefined schedule. Run scheduled consistency checks are on a per-node basis as follows (this is not a replicated feature):

The EMR process cannot be undone, and it involves major changes to your replication system. Only consider an EMR if the disconnected node cannot be repaired or reconnected in an acceptable amount of time.

The EMR procedure needs to be co-ordinated between sites/nodes. You must not start an EMR if an EMR procedure has already started from another node. Running multiple EMR procedures at the same time can lead to unpredictable results or cause the processes to get permanently stuck requiring a complete re-deployment of your system. Again, please contact WANdisco support to prevent unnecessary damage to your replication ecosystem.

Any replication group which has its membership reduced to one node will continue to exist after the emergency reconfiguration as a non-replicating group. When you have set up a replacement node you should be able to add it back to the group to restart replication.

Note: If EMR is used to remove a node you may be left with a pending task of type tasksTypeREMOVE_STATE_MACHINE_TASK. If this is the case then:

Back to Contents - Back to beginning of section

The repository table lists all the repositories that you add to MSP.

You can create a replication group providing that you have at least one node connected.

For more details, see How to create a replication group.

At the heart of WANdisco’s DConE2 replication technology is an agreement engine that ensures that SVN operations are performed in exactly the same order on each replica, on each node. Any node that has the role of voter becomes part of the agreement engine and together with other voters determine the correct ordering. If there’s high latency between any voters this may adversely affect replication performance. Fortunately it isn’t a requirement that every node takes part in forming agreements. An Active only node can still create proposals (i.e. initiates repository changes) but the agreement engine doesn’t need to wait for its vote. Read more about how replication works in the Replication Strategy Section.

To optimize replication performance it’s common for administrators to remove voter status from a node after their staff leave for the day - a practice commonly known as "Follow The Sun" where far-flung organizations transfer roles and privileges between locations so that those privileges are always held by nodes at actively staffed sites.

If you change any nodes you need to click the Save Schedule button. Any mistakes in node role combinations selected will be detected at this time, and if there are any then the save will fail.

Use the Clear Schedule button to blank out settings that you have changed, returning to the default schedule.

For how to setup a schedule, see How to configure a schedule.

The Resource Monitoring Data settings provide a basic tool for monitoring available disk storage for MSP’s resources.

For more information about setting up monitors, read Setting up resource monitoring.

The notifications system provides SVN administrators with the ability to create event-driven alert emails. Set up one more more gateway (mail servers), add destination emails to specify recipients, create email templates for the content of the alert emails, then set the rules for which event should trigger a specific email.

The Logging Setting lets you quickly add or modify Java loggers via the admin console, rather than making manual edits to the logger file:

Loggers are usually attached to packages, here, the level for each package is available to modify or delete. The global level is used by default, so changes made here are used to override the default values. Changes are applied instantly but in-memory only and are forgotten after a restart of the replicator (unless they are saved). For information about adding or changing loggers, see Logger Setting Tool.

The System Data table provides a list of the editable and read-only settings.

Changing the editable settings causes a replicator restart:

The read-only settings were either provided during setup or have since been applied:

View REST API Documentation
This link takes you to your node’s local copy of the API documentation. This documentation is generated automatically and ties directly into your server’s local resources.
There is also a copy of the latest API documentation available in this user guide. Note that this has been lifted from an installation and will link to resources that will not be available on the website (resulting in dead links).