Administrator Guide

1. Running Git MultiSite

This guide describes how to use Git MultiSite (GitMS).

1.1 Start up

To start the GitMS replicator:

1. Open a terminal window on the server and log in with suitable file permissions.
2. Run the git-multisite service, located in the init.d folder:

   lrwxrwxrwx  1 root root    37 May  9 10:37 git-multisite -> /opt/git-multisite/bin/git-multisite

3. Run the start script:

   [root@localhost init.d]#  ./git-multisite start
   
   
   
   20130520-164811 (24088) [INFO]: Starting WANdisco MultiSite
   20130520-164811 (24088) [INFO]: Started replicator (24100)
   20130520-164811 (24088) [INFO]: Started ui (24110)
   20130520-164811 (24088) [INFO]: Number of errors: 0
   20130520-164811 (24088) [INFO]: Number of warnings: 0
   
       	

4. The two components of GitMS, the replicator and the UI, start up. Read more about the git-multisite init.d script.

1.2 Shut down

To shutdown:

1. Open a terminal window on the server and log in with suitable file permissions.
2. Run the git-multisite service, located in the init.d folder:

   lrwxrwxrwx  1 root root    37 May  9 10:37 git-multisite -> /opt/multisite-plus/bin/git-multisite

3. Run the stop script, i.e.:

   
   [wandisco@ip-10-0-100-7 bin]$  ./git-multisite stop
   
   
   20130520-165704 (24767) [INFO]: Stopping WANdisco MultiSite
   20130520-165704 (24767) [INFO]: Request received to shut down replicator
   20130520-165704 (24767) [INFO]: replicator processes ended
   20130520-165704 (24767) [INFO]: Request received to shut down ui
   20130520-165704 (24767) [INFO]: Sending signal 15 to watched ui process (attempt 1)...
   20130520-165707 (24767) [INFO]: Sending signal 15 to watched ui process (attempt 2)...
   20130520-165710 (24767) [INFO]: ui processes ended
   20130520-165710 (24767) [INFO]: Number of errors: 0
   20130520-165710 (24767) [INFO]: Number of warnings: 0
   
         

4. Both the replicator and the UI processes shut down. Read more about the git-multisite init.d script

1.3 Startup script

The startup script for persistent running of GitMS is in the /etc/init.d folder. Run the script with the help command to list the available commands:

[root@localhost init.d]# ./git-multisite help
    
  Usage: git-multisite {start|stop|status|uistart|uistop|repstart|repstart}

  start      Start WANdisco Multisite service
  stop       Stop WANdisco Multisite service
  status     Display running WANdisco Multisite services
  uistart    Start WANdisco Multisite UI service
  uistop     Stop WANdisco Multisite UI service
  repstart   Start WANdisco Multisite Replicator service
  repstop    Stop WANdisco Multisite Replicator service
    
  

1.4 Change the admin console password

You can change GitMS's login password at any time:

1. Log in to the MultiSite admin console.
   
   

   Login

2. Click the Security tab.
   
   

   Security

3. On the security tab screen you see the Internally Managed Users table. Click the Edit link that corresponds with the Admin account. In the Edit User window that opens, enter a new password. Repeat the entry in the box immediately below.
   
   

   Changed password

4. Click the SAVE button to store the new password. The new password has been accepted if you see a growl message on screen.
   
   

   Growl

1.5 Update your license.key file

Follow this procedure if you need to change your product license, e.g. if you need to increase the number of Git users or the number of replication nodes.

1. Log in to your server's command line, navigate to the properties directory: /opt/wandisco/git-multisite/replicator/properties and rename the license.key to license.20130625:

   
   total 16
   -rw-r--r-- 1 wandisco wandisco 1183 Dec  5 15:58 application.properties
   -rw-r--r-- 1 wandisco wandisco  512 Dec  5 15:05 license.key
   -rw-r--r-- 1 wandisco wandisco  630 Dec 17 15:43 logger.properties
   -rw-r--r-- 1 wandisco wandisco  630 Dec 17 15:45 log4j.properties
   	    

2. Get your new license.key and drop it into the /opt/git-multisite/replicator/properties directory.
3. Restart the replicator by running the GitMS script with the following argument:

   /etc/init.d/git-multisite restart

   This triggers a GitMS replicator restart, which forces GitMS to pick up the new license file and apply any changes to permitted usage.

If you have problems, check the replicator logs (/opt/git-multisite/replicator/logs) for more information.

PANIC: License is invalid com.wandisco.fsfs.licensing.LicenseException: Failed to load filepath. 

1.6 Update a node's properties

The System Data section of the Settings tab lists editable properties that you can quickly update by re-entering, saving, and allowing the GitMS replicator to restart. Note: This may cause a brief disruption for users because in-flight commits will fail.

Editable node properties, involving replicator restart

Node Name

This is the human-readable form of the node's ID. You can change the value of Node Name and reuse it after it has been removed from the replication network. You cannot have two nodes with the same name, but you can reuse a previously removed node name.

Location Latitude

Enter the node's latitude here.

Location Longitude

Enter the node's longitude here.

Hostname / IP Address

Enter/update the hostname or underlying IP address.

DConE Port

The TCP port used for DConE agreement traffic. Do not confuse this with the Content Distribution port which carries the payload repository data.

Dashboard Polling Interval (Minutes)

Sets how often the dashboard messaging is updated. The messaging is populated by Warnings and Errors that appear in the replicator logs file. The default frequency is every 10 minutes.

Dashboard Item Age Threshold (Hours)

Sets how long dashboard messages are maintained. After this amount of time messages are flushed from the dashboard. The default is 96 hours (4 days).

After entering a new value, click Save. A growl message confirms that the change is being replicated. This results in a restart of the replicator which may cause brief disruption to SVN users.

1.6.1 Other property changes

You can also modify other properties in the configuration files.

• Content delivery port
• Task garbage collection
• Node content distribution timeouts

Content delivery port

To change the Content Delivery port:

1. Use the command:

   content.port.<Node id>=<new port>

2. When the file is in place, run the following command (on all the nodes except the one you have changed):

   java -jar git-ms-replicator-updateinetaddress.jar -c <path to application.properties>

3. Go back to the node with the updated properties and Restart MultiSite.
4. Log in to the updated node and check its System Data at the bottom of the Settings tab. Do some test commits to ensure that replication continues successfully.

Task garbage collection

Two configurable properties control how often the task garbage collection process runs. These properties are set during installation. To modify their values, add them to the application.properties file:

task.removal.interval

This controls how often the task garbage collection process should run. The default is 96 hours, noted in milliseconds (i.e. 345600000 milliseconds for 96 hours).

task.expired.interval

This controls how old a successfully run task must be before it is made available for garbage collection. The default is 96 hours, noted in milliseconds (i.e. 345600000 milliseconds for 96 hours).

Recommended Settings

Summary: For large deployments reduce the time from 96 to 24 hours
The recommended settings are suitable for most deployments. However, for deployments with very large numbers (thousands) of repositories and where repository consistency checks are automated, then we recommend that you reduce the setting times, initially to 24 hours (86400000 ms).

Shorter periods result in a corresponding reduction in your ability to troubleshoot problems that involve replicator task history. If you notice large numbers of failed tasks accumulating over time or have any concerns about what settings are right for your specific deployment, contact WANdisco's support team.

Example

For a deployment that replicates several thousand repositories and schedules daily consistency checks it's decided to reduce the task expiry to 48 hours and the garbage collection frequency to 24 hours. The settings would therefore be:

task.removal.interval   86400000L
task.expired.interval   172800000L
  

Node content distribution timeouts

Two configurable properties enable you to balance best possible performance against the tolerance of a poor WAN connectivity. The properties are in the application properties file, by default located in /opt/wandisco/git-multisite/replicator/properties/application.properties.

socket.timeout

socket.timeout=90000

This is the amount of time in milliseconds that the local node waits for the connection to be established before throwing an exception. The exception shows that it failed to connect within that timeout. Default value is 15 minutes (90,000 milliseconds).

Not less than 10 minutes!
DO NOT set socket.timeout to less than 10 minutes (60,000 milliseconds) or you may encounter problems.

content.pull.timeout

content.pull.timeout=300000

This sets how long the Content Distribution system waits for new content to be pulled fully over from a remote node. The default value is 5 minutes (300,000 milliseconds). This default is set with the assumption that there are no problems with the deployment's WAN connectivity.

• Increasing the timeout: This may help if poor connectivity causes the replicator to repeatedly give up on content distribution that would have eventually transferred if there was enough time, i.e. not as a result of a slow network rather than something that has caused a permanent error.
• Decreasing the timeout: We recommend that you do not decrease the timeout value because this is not designed to boost performance, although that may occur in some situations. We recommend that you consult the WANdisco Support team if you want to drop the timeout value below 5000 (5 seconds).

1.7 Set up data monitoring

The Monitoring Data tool monitors the disk usage of GitMS's database directory, providing a basic level of protection against GitMS consuming all disk space. The tool also lets you set up your own monitors for user-selected resources.

1.7.1 Default settings

Click the "View" link to go to a monitor's settings.

By default MultiSite's database directory (/opt/wandisco/git-multisite/replicator/database) is monitored - this is the location of MultiSite's prevayler database where all data and transactions files for replication are stored.

This built-in monitor runs on all nodes. Any additional monitors that you set up will monitor on a per-node basis. Monitors are not replicated so a monitor set up on one node is not applied to any other node.

1.7.2 Additional monitors

As well as GitMS's own database folder, there are several directories that might grow very large and potentially consume all available file space.

Consider monitoring the following MultiSite directories:

• /opt/wandisco/git-multisite/replicator/content
• /opt/wandisco/git-multisite/logs
• /opt/wandisco/git-multisite/replicator/logs

Also monitor /path/to/authz. If you are using Authz to manage authorization and your Authz file is situated on different file system from GitMS, then we recommend that you set up monitoring of the authz file.

For most deployments all these directories will reside on the same file system, so that our default monitor would catch if any of them were consuming the available space. However, there are two scenarios where we'd recommend that you set up your own monitor for the content directory:

1) You wish to set a higher trigger amount than the default monitor (1GiB for warning, 0.09GiB for emergency shutdown).
2) You have placed the content directory on a different filesystem with its own capacity that wouldn't be tracked by the default monitor.

In either case you should follow up the setting up of a monitor with a corresponding email notification that will be sent if some or all of your monitor's trigger conditions are met.

Create additional resource monitors using the following procedure:

1. Log in to the Administrator user interface.
2. Click the "SETTINGS" link on the top menu bar.
3. Monitoring Data is situated below the Administrator Settings. Enter the full path to the resource that you wish to monitor. For example, you might wish to monitor the replicator logs: /opt/wandisco/git-multisite/replicator/logs. Enter the path and click "Add".
   
   

   Add resource path

4. The new resource monitor appears as a new box - it will display "No records found", indicating that it doesn't yet have any monitoring rules set. Click its corresponding "Configure" link.
   

   Configure

5. The screen will update to show the Resource Monitoring screen for your selected resource.
   
   

   Settings

   File Path:

   The full path for your selected resource

   Monitor Identity:

   The unique string that will identify the monitor

   Edit Condition and Event List

   Lists current resource monitors, initially this will state "No records found"

6. Add Conditional and Event to list.

   Storage amount entry field

   Enter an amount of disk space in Gigabytes. e.g. 0.2 would be equal to 200 Megabytes of storage.

   Select an Event from the dropdown:

   SEVERE

   Initiates a shutdown of GitMS and will also write a message to the log and the "SEVERE" logging level. See "When a Shut down is triggered" for more information.

   WARNING

   Writes a message to the log and the "WARNING" level of severity.

   DEBUG

   Writes a message to the log and the "DEBUG" level of severity.

   INFO

   Writes a message to the log and the "INFO" level of severity.

7. When you have added all the trigger points and events that you require for the resource, click "Update". You can then navigate away: Click "Resource Monitoring" on the breadcrumb trail to return to the settings screen.

1.7.3 When a shutdown is triggered

If the disk space available to a monitored resource is less than the value you have for a "Severe" event then the event is logged and MultiSite's replicator will shut down after a set interval of 10 minutes. You can configure the interval in the application.properties file in /opt/wandisco/git-multisite/replicator/properties/application.properties.

resourcemonitor.period.min=10L

Value is minutes.

When shut down, all Git repositories become unavailable to users. You should immediately make more disk space available. The replicator can be restarted using GitMS's service as soon as the resource that triggered the shutdown has enough available disk space not to shut down again.

1.8 Set up email notifications

Email notification is a rules-based system for delivering alerts (based on user-defined templates) over one or more channels to destinations that are based on triggers that are activated by arbitrary system events. Put simply, email notification sends out emails when something happens within the GitMS environment. The message content, trigger rules and destinations are all user-definable.

Automated alert emails

Read about

• Gateways in Set up a Gateway
• Destinations in Set up a Destination
• Templates in Set up a Template
• Rules in Set up a rule

1.8.1 Set up a gateway

The Gateway settings panel stores your email (SMTP) server details. You can set up multiple gateways to ensure that the loss of the server doesn't prevent alert notifications from being delivered.

1. Log in to the admin UI, then click the Settings tab.
2. Click the Gateway section of the Notifications area.
   
   

   Add Gateway

3. Enter your email gateway's settings:
   
   

   Enter settings

   IP/Hostname of SMTP Server:

   your email server's address.

   SMTP Server Port:

   The port assigned for SMTP traffic (Port 25 etc).

   Encryption Type:

   Indicate your server's encryption type - None, SSL (Secure Socket Layer) or TLS (Transport Layer Security). SSL is a commonly used. For tips on setting up suitable keystore and truststore files see Setting up SSL Key pair.

   Keystores?
   

   If you're not familiar with the finer points of setting up SSL keystores and truststores it is recommended that you read the following article: Using Java Keytool to manage keystores.

   Authentication Required:

   Indicate whether you need a username and password to connect to the server - requires either "true" or "false".

   User Name:

   dd>If authentication is required, enter the authentication username here.

   Password:

   dd>If authentication is required, enter the authentication password here.

   Sender Address:

   dd>Provide an email address that your notifications will appear to come from. If you want to be able to receive replies from notifications you need to make sure that this is a valid and monitored address.

   Number of Tries Before Failing:

   Set the number of attempts GitMS makes to send out notifications.

   Interval Between Tries (Seconds):

   Set the time (in seconds) between your server's attempts to send notifications.

4. Click the +Add button. Your gateway appears in the table.
   You can add any number of gateways. GitMS exhausts the "Number of Tries Before Failing" for each registered gateway before moving on down the list to the next. You can use the Test button to verify that your entered details will connect to a mail gateway server.

1.8.2 Set up a destination

The destinations panel stores the email addresses for your notification recipients.

1. Click the Destinations line.
   

   Click Destinations

2. Enter an email address for a notification recipient. Click the + Add link.
   
   

   Email addresses you target for alerting.

3. The destination will appear in a table. Click the Edit or Remove links to change the address or remove it from the system.

1.8.3 Set up a template

The template section is used to store email messages. You can create any number of templates, each with its own notification message, triggered by one of a number of trigger scenarios that are set up in the Rule section.

1. Click the Templates line.
   

   Click Templates

2. Enter a Template Subject line, this will be the subject of the notification email.


3. Enter some Body Text, this will be the message that is sent out when the notification is triggered. The message has a 1024 character limit, you can track the available number of characters at the bottom of the text box.
   

   The message you want to send out

4. When the message has been entered, click the + Add link to save the message template.

Available variables

When writing email notification templates, you can insert variables into the template that will be interpolated when the notification is delivered. The following variables are available for ALL event types:

{node}

This returns the node name.

{timestamp}

This returns the time at which the event is received (not the time at which the notification is delivered).

{event}

This returns the raw dump of the event.

For the event types Disk Monitor Info, Disk Monitor Severe and Disk Monitor Warning:

{event.message}

This returns information about the disk monitoring threshold that was exceeded.

{event.resource.absolutePath}

This returns the monitored file path..

For the event types Deploy Repository Succeeded and Deploy Repository Failed, the following additional variables are available:

{event.proposerNodeId}

This returns the ID of the node that sent the event.

{event.repository.name}

This returns the repository name.

{event.originalProposal.repository.name}

This returns the user-specified name of the repository to which the event pertains.

{event.originalProposal.repository.fSPath}

This returns the location on-disk of the repository that the event pertains to. More events types and event variables will be added in the future.

1.8.4 Set up a rule

Use the Rules section to define the system event that should trigger a notification, the message template that should be used, and the recipients that should receive the notification.

1. Click the + on the Rule line.
2. Choose an Event from the Event drop-down list:
   
   

   Rules options for email notification

Any Repository Global Read-Only Event

In case of any repository entering a global read-only mode.

Global Read-Only Due to Admin Action

In case of any repository entering a global read-only mode as a result of administrator interaction through the admin UI.

Disk Monitor Info

Disk Storage has dropped below the Info level. This will trigger if any data monitor message is written to the logs at the "INFO" level.

Disk Monitor Warning

Disk Storage has dropped below the Warning level.This will trigger if any data monitor message is written to the logs.

Disk Monitor Severe

Disk Storage has hit the Severe level. This will trigger if any "Severe" level data monitor message is written to the logs. At this level, GitMS will have shutdown to ensure that disk space exhaustion doesn't corrupt your system and potentially your Git repositories.

Deploy Repository Failed

A repository added to GitMS has failed to deploy, in which case the repository will not be replicated.

Deploy Repository Succeeded

A repository added to GitMS has successfully deployed. Such an event might be sent to a mail group received by Git users, telling them that their repository is now accessible.

Global Read-Only Due to Consistency Check Failure

In case of any repository entering a global read-only mode as a result of failing a consistency check with its replicas.

Generic file replication error occurred

An error occured with the Generic Replication script (When running GitMS in conjunction with Access Control Plus).

3. Choose a Template from the drop-down list. These are the templates that you have already set up under the Templates section.
4. Choose destinations for your notification from the available destination email addresses. You can make multiple selections so that a message is sent to more than one recipient address.
5. Click on the + Add link to save your rule.
   
   

   Rules you can edit or remove

2. Manage access to GitMS

GitMS supports the following mechanisms to manage access to its admin UI:

• Internally Managed users are admin accounts that are set up from within GitMS's Admin UI.
• LDAP Authorities: You can have GitMS query LDAP services and filter for a suitable group from which to populate admin users.
• Kerberos Security: If your organization uses a Kerberos authentication system you can set up MultiSite to use it.

You can set up multiple administrator accounts to access the GitMS admin console. You can set up the accounts from the admin UI (via the Security tab). These users can then log in to any node's admin UI by providing their username and password.

This section describes how to set up multiple accounts, set up managing LDAP authorities, and export/import the resulting data.

2.1 Add additional users

1. Log in to the Admin UI using an existing admin account.
   

   Log in

2. Click the SECURITY tab, then click Add User.
   

   Add User

3. Enter details for the new administrator, then click the Add User button at the end of the entry bar.
   

   Click Add User to save their details

4. You see a growl message confirming that the user has been added. They are listed on the Internally Managed Users after clicking the Reload button (or refershing your browser session).
   

   New user appears

2.2 Remove or edit user details

You can modify any user details by clicking their corresponding Edit button on the Internally Managed Users table

Remove or Edit users

2.3 LDAP authorities

GitMS supports the use of LDAP authorities for managing admin logging accounts. See our Guide to LDAP.

When connecting GitMS to available LDAP authorities it is possible to classify the authority as "Local", i.e. specific to the node in question or not. In this case, the authority details are replicated to the other nodes within the replication network.

You can run multiple LDAP authorities that are of mixed type, i.e. using some local authorities along with other authorities that are shared by all nodes. When multiple authorities are used, you can set the order that they are checked for users.

The standard settings are supported for each configured LDAP authority: URL, search base and filter and bind user credentials. Note that the bind user's password cannot be one-way encrypted using a hash function because it must be sent to the LDAP server in plain text, so for this reason the bind user should be a low privilege user with just enough permissions to search the directory for the user being authenticated. Anonymous binding is permitted for those LDAP servers that support anonymous binding.

2.3.1 Add authority

Use the Add Authority feature to add one or more LDAP authorities, either local to the node or connected via WAN. Locally LDAP services are treated as having presedence. When Internally managed users are enabled they are first checked when authenticating users - see Admin Account Precedence

To add an authority:

1. Log in to the admin UI, click the Security tab.
2. Click Add Authority.
   

   Add Authority

3. The Authority entry form appears. Enter the following details:
   

   Add Authority

   URL

   Enter your authorities URL. You need to include the protocol ldap:// or ldaps://
   Example (Active Directory:)

   ldap://<server IP>:389

   Bind User DN

   Enter a LDAP admin user account that will be used to query the authority
   Example (Active Directory:)

   cn=Administrator,cn=Users,dc=windows,dc=AD

   Search Base

   Enter the Base DN, that is the location of users that you wish to retrieve.
   Example (Active Directory:)

   CN=Users,DC=sr,DC=wandisco,DC=com?sAMAccountName

   Search Filter

   Optionally add A query filter that will a select user based on relevant LDAP attributes. Currently the search filter must be created so that it filters LDAP query result into unique result. A work around might look something like:
   Example: (Active Directory)

   (&(memberOf=[DN of user group])(sAMAccountName={0}))

   This dynamically swaps the {0} for the ID of the active user. For more information about query filter syntax, consult the documentation for your LDAP server.

   Is Local?

   Tick this checkbox if you want the authority to only apply to the current node and not be replicated to other nodes (which is otherwise done by default).

4. Click the Add Authority. This will save the authority settings that you have just entered. You can click the Test button to verify that the details will successfully connect to the authority without yet adding the authority.
5. When running with multiple authorities, you should determine the order by which MultiSite polls the authorities. Use the +- symbols at the end of each authority entry to push it up (+) or down (-) the list.
   

   Order authorities

2.3.2 Edit authority

Modify an existing authorities settings:

1. Log in to the admin UI, click the Security tab.
2. Click the edit link on the line that corresponds with the authority that you wish to edit.
   

   Edit authorities link

3. Update the settings in the popup box, then click Save.
   

   Edit authorities box

Kerberos security

This section describes the basic requirements for integrating GitMS with your existing Kerberos systems. The procedure requires the following:

• A Key Distribution Center
• A Workstation setup on each node
• A machine with a suitably configured browser. See Enabling Kerberos access on your browser

2.3.3 Configuration

This procedure assumes that you have already set up your DNS service and master Key Distribution Center.

1. On each node, add the service principal:

   # kadmin -p root/admin -q "addprinc -randkey HTTP/node1.example.com"
   # kadmin -p root/admin -q "ktadd -k /opt/krb5.keytab HTTP/node1.example.com"
   # chmod 777 /opt/krb5.keytab
   	    

2. Each node should have installed the add-on JCE Java 6 or Java 7 Unlimited Strength Jurisdiction Policy Files". These can be downloaded from Oracle, subject to your local import rules concerning encryption technology. Once downloaded, extract to the the Java security library, i.e.

   $JAVA_HOME/lib/security/

3. At this point you can install GitMS on each node. If that's already done, then configure the Kerberos settings under the Security tab.
   

   Edit Kerberos box

   Serivce Principal:

   This unique name for an instance of a service, such as HTTP/node1.example.com

   Keytab Location:

   This is the location of the keytab, a file containing pairs of Kerberos principals and encrypted keys (often derived from the Kerberos password). It's used for logging into Kerberos without being prompted for a password.

   Kerberos Config Location:

   The krb5.conf file contains Kerberos configuration information, including the locations of KDCs and admin servers for the Kerberos realms of interest, defaults for the current realm and for Kerberos applications, and mappings of hostnames onto Kerberos realms. Normally, you should install your krb5.conf file in the directory /etc. i.e. /etc/krb5.conf

   Save the settings. Log out. Return to the node in your browser, this time you should log in automatically (in this as user sally@EXAMPLE.COM.)

See Security Reference: Kerberos settings.
See Configure browsers for Kerberos authentication.

3. Nodes

This section describes the functions that manage repository data replication.

3.1 Add a node

To replicate Git repository data between sites, you first first tie the sites together. Start by adding (connecting) sites in an induction process:

1. Log in to the GitMS admin console of the new node that you are connecting to your existing servers.
2. Click the Sites tab.
3. Click the Connect to node button.
   

   Connect to node

4. Enter the details of an existing node. You can get these details from the Settings tab of the existing node.
   

   Enter details from an existing, connected node

   Node ID

   This is the name that you gave the Node during installation. If you log into the node you can see the Node ID on the title of any screen that you view, it also appears in the logged in message: "Welcome to MultiSite, admin. You are connected to <NODE ID>"

   Node Location ID

   A unique string that that a node creates for use as in identifier. You can get this from the node's Settings tab:
   

   System Data table, found on the Settings tab.

   Node IP Address

   The IP Address of the node's server.

   Node Port No

   The TPC port that the node uses for DConE, which handles agreement traffic. The default is 6444. See Reserved Ports.

5. Click the SEND CONNECTION REQUEST button. The new node appears on the active list of Sites. The new node may get stuck in a pending state. If this happens click the Cancel button.

3.2 Remove a node

It's useful to remove a node from a GitMS replication group if you are no longer replicating repository data to its location and want to tidy up your replication group settings.

No ties allowed
You only have the option to remove a node if it is not a member of a replication group. Therefore, you may need to remove and recreate replication groups to make it eligible for removal.

Known issue:
NOTE: If a node is inducted but not in a replication group then you can, from that node, remove other inducted nodes that are in a replication group. Currently, a node is not aware of the membership of any replication group of which it is not a member. This means that it is possible to remove a node that is a member of a replication group, if done from another node that does not have knowledge of the replication group.

Until we fix this issue, do a manual check of any nodes that you want to remove to make sure that it is not a member of a replication group.
A removed node cannot return
Take care when removing nodes! To ensure that your replication network is kept in sync, removed nodes are barred from being re-inducted. The only way that you can bring back a node is to reinstall GitMS using a new Node ID.
1. Log in to the GitMS admin console of any connected node.
2. Click the Nodes tab.
3. Nodes that are eligible for removal have the Remove Node option available in the Action column. In this example, NodeSanFrancisco is eligible for removal because we have removed it from replication groups.
   

   Nodes table under the Nodes tab

4. Click the Remove Node button. Don't forget that this action is irreversible. You must be sure that you want to permanently remove the node.

   Ready to remove NodeSanFrancisco

5. After a refresh of the admin user interface you can still see the removed node if you click the Display Removed Nodes button. Removed nodes are identified by their Removed status.
   

   Node removed

3.3 Stop all nodes

You can bring all nodes to a stop with a single button click (if all associated repositories are replicating/writable).

A stop can't be synchronized if associated repositories are Local Read-only
Before starting a Sync Stop All, make sure that none of your nodes have repositories in a local read-only state.
1. Log into the admin UI and click the Nodes tab.
2. Click the Sync Stop All button.
   

   Stop all nodes

   You get a growl message confirming the stop has been triggered. You see the results when you refresh your browser session.
   

   Stopped

3. On the Node table all nodes show as Stopped. In this state you can do maintenance or repairs without risking your replication getting out-of-sync.
   

   Node removed.

4. The Sync Stop All button has changed to Start all. However, you can start selected nodes by logging in to the admin console of each node that you want to start. Use the Start Node link that appears in the Action column of the nodes table.
Important!
We strongly recommend that you watch the log messages and confirm that all nodes report as stopped. If you suspect that one or more nodes are not going to stop you should investigate immediately. The DASHBOARD messages should report the stop, for example:

Aborted tasksType PREPARE_COORDINATE_STOP_TASK_TYPE
Delete Task
Originating Node: Ld5UYU
tasksPropertyTASK_ABORTING_NODE: Ld5UYU
tasksPropertyTASK_ABORT_REASON: One or more replicas is already stopped.
The replica was: [[[Ld5UYU][bf0c6395-77b6-11e3-9990-0a1eeced110e]]]
    

Look for the message:

Aborted tasksType PREPARE_COORDINATE_STOP_TASK_TYPE

In the replicator.log file you might see the error type:

DiscardTaskProposal <task id etc> message: One or more replicas is already stopped.

3.4 Start all nodes

1. If all nodes have been brought to a stop, click the Start All button to start them replicating again.
   

   Stopped!



2. After a browser refresh, all nodes will now show as running.

3.5 Disconnected/offline nodes

If a node is disconnected you can see this from the UI:

• When you click the REPLICATON GROUPS tab, you can see the groups with nodes that are offline:
  
• When you click View to see a replication group, you are warned that functionality is reduced when a node is disconnected, plus you see which nodes are connected/disconnected:
  
• If you try to create a new replication group that includes a disconnected node you are warned that the node is unavailable and the group is pending:
  
• You cannot add a new node to an existing replication group while a member node is disconnected. You get this message in the UI log on the dashboard:

  Replication group schedule cannot be updated with a new node whilst a member is disconnected.

4. Replication groups

When you click the REPLICATON GROUPS tab you can see if there are disconnected nodes:
See Disconnected/offline nodes.

4.1 Add a replication group

Follow this procedure to add a new Replication Group. You need to add a new replication group when you want to replicate between a new combination of sites, i.e. sites that are not currently replicating in an existing group. If, instead, you want to replicate a new repository between existing sites, you can add a new repository to those sites. In this case, see Add a new repository.

1. Log in to the GitMS browser-based user interface. Click the REPLICATION GROUPS tab, then click the CREATE REPLICATION GROUP button.
   

   Create a replication group

2. Enter a name for the group in the Replication Group Name field, then click the drop-down selector on the Add Sites field. Select the sites that you want to replicate between.
   

   Replication group details

   Replication Ground Rules
   

   • A node can belong to any number of replication groups.
   • A repository can only be part of a single active replication group at any particular time.
   • You can change membership on the fly, moving a repository between replication groups with minimal fuss.
3. Click each node label to set its node type.
   

   Click node labels to change their type

   Advice on creating effective replication groups
   For an understanding of some of the ground rules for replication you should read the section Creating resilient Replication Groups.
   Nodes are automatically added to a group as Active Voters. To understand the differences between the different types of nodes, read Guide to node types.
4. When all sites are in place with your required settings, click CREATE REPLICATION GROUP.
   

   Create Replication Group

5. Newly created replication groups appear on the Replication Group tab, but only on the admin UI of nodes that are themselves members of the new group.
   

   The new replication group appears if you are logged into one of its constituent nodes

Important: Don't cancel replication group creation tasks
If you create a new replication group, then find that the task is stuck in pending because one of your nodes is down, do not use the Cancel Tasks option on the Dashboard's Pending Tasks table.
If, when all nodes are up and running, the replication group creation tasks are still not progressing, please contact the WANdisco support team for assistance.


4.2 Delete a replication group

You can remove replication groups from GitMS, as long as they they have no repositories. For example:

1. We have identified that replication group VineyanRepos is to be removed from GitMS. We can see that it has a single repository associated with it. Click the View to see which one.
   
   

   View

2. On the Replication Group configuration screen we can see that Repo5 is associated with the group. We can see that currently the Delete Replication Group (VinyardRepos) is disabled. You can follow the link to the repositories page to remove the association.
   
   

   Repositories

3. On the Repositories screen, click the associated repository, in this example it's Repo5, then click the EDIT button.
   
   

   Select and Edit

4. On the Edit Repository box, use the Replication Group drop-down to move the repository to a different Replication Group. Then click SAVE.
   
   

   Edit

5. Repeat this process until there are no more repositories assoicated with the Replication Group that you wish to delete. In this example VinyardRepos only had a single repository, so it is now empty, and can be deleted. Click View, then on Configure.
   
   

   Move it

6. Now that Replication Group VinardRepos is effectively empty of replication payload the Delete link is enabled. Click the link Delete Replication Group (VinyardRepos) to remove the replication group, taking note that there's no undo - although no data is removed when a replication group is deleted, it should be easy enough to recreate a group if necessary.
   
   

   Click the Delete link button

7. A growl will appear confirming that the replication group has been deleted.
   
   

   Deleting the replication group

4.3 Add node to replication group

Don't add a node during a period of high replication load
When adding nodes to a replication group that already contains three or more nodes, ensure that there isn't currently a large number of commits being replicated.

Adding a node during a period of high traffic (heavy level of commits) going to the repositories may cause the process to stall.

To add additional nodes to an existing replication group, so that there's minimal disruption to users:

1. Log in to a node, click the REPLICATION GROUPS tab. Go to the replication group to which you will add a new node, click its VIEW.
   
   

   Replication Groups

2. The replication group screen will appear. Click Add Nodes.
   
   

   View the group settings

   Why is the Add Nodes button is disabled?
   The Add Nodes button may be grayed out if the current replication group configuration won't support the addition of a new voter node.
   
   It is also possible that a configuration that is scheduled in the future may block the addition of a new node. Check the schedule if you think that you should otherwise be able to add a new node to the replication group.
3. Select the node that you wish to add to the replication group.
   
   

   Select

4. When there are no further nodes to add to the group, click the Add Nodes button.
   
   
5. At this stage in the process we're ready to select a Helper node from which we'll synch repository data to the new node - select a Helper Node.
   
   

   Helper node

6. Note the warning about not closing the browser or logging out during this process. Otherwise you will need to perform a more lengthy repair procedure. Click the Start Sync button.
   
   

   Start sync

7. You now need to manually synchronize the repositories from the helper node, which is temporarily offline until this process is finished. See Synchronizing repositories using rsync.
   
   The process lets you either do a complete sync or select specific repositories that you wish to sync. Assuming that you have synced all repositories you click Complete All. The helper node is then released from the process, allowing it to catch up with any transactions it missed while taking part in the procedure.
   
   

   Complete all

8. A growl message confirms that the new node is added to the replication group.
   
   

   new node!

9. Return to the Replication Group screen and you can see the new node count.
   
   

   Adding new node complete

4.4 Remove node from replication group

You can remove a node from a replication group, for example, if the developers at one of your nodes are no longer going to contribute to the repositories handled by a replication group. Removing a node from a replication group stops more updates to its repository replicas.

Remove stray repositories
If you remove a node from a replication group, you must delete its copy of the repositories managed by the replication group. An out-of-date stray copy can result in confusion or users working from old data.

You cannot remove a node that is currently assigned as the Managing Node. To remove the managing node, go to the Configure Schedule page and assign a different node as managing node.

1. Log in to the admin console of one of your nodes. The node needs to be a member of the relevant Replication Group, otherwise it does not appear on the tab. Click the Replication Groups tab.
   , then click the View button for the Replication Group from which you plan to remove a node.
   

   Log in and go to REPLICATION GROUPS

2. Click the node that you want to remove from the group. Providing that the removal of the node doesn't invalidate the remaining configuration you see a Remove node from replication group link. Click the link.
   
   

   Remove

3. A dialog opens asking you to confirm the removal of the selected node from the Replication Group. Click Remove.
   

   Confirm remove

4. A growl message confirms that the removal is in progress. You many need to click the Reload button to ensure that the action has been completed on all nodes.
   

   Reload to confirm the updated state

5. The node is removed from the Replication Group. On the Replication Groups panel you now see that the number of nodes has reduced by one.
   

   Less one member node

4.5 Schedule node changes: follow the sun

You can schedule the member nodes of a replication group to change type according to when and where it is most beneficial to have active voters. To understand why you may want to change your nodes read about Node Types.

4.5.1 Schedule node type changes via the public API

Instead of manually setting up schedules through a node's UI you can do it programmatically through calls to the public API.
See Public API ScheduledNodeAPIDTOList element and scheduledNodeAPIDTOList Datatype.

Use the following API call:

http://<ip>:8082/public-api/replicationgroup/{repgroupID}/schedule

e.g.

http://10.0.100.135:8082/public-api/replicationgroup/97913c04-bbad-11e2-877a-028e03094f8d/schedule

PUT with ReplicationGroupAPIDTO XML as body:

To make Node N3 a tie-breaker 'T' FROM 10:00 - 16:00 (GMT) every day of the week with Node N1 as tie-breaker 'T' afterwards.

Times are always in UTC (GMT)
When viewed on a node, times are shifted to the local timezone although internally they are always recorded in UTC.

Example curl command:

Make a text file containing ReplicationgroupAPIDTO XML (as above) called schedule.xml

curl -u username:password -X PUT -d @schedule.xml http://[IP]:[PORT]/public-api/replicationgroup/97913c04-bbad-11e2-877a-028e03094f8d/schedule

Sample schedule.xml file

<ReplicationGroupAPIDTO>
       <replicationGroupName>global</replicationGroupName>
     <replicationGroupIdentity>97913c04-bbad-11e2-877a-028e03094f8d</replicationGroupIdentity>
       <scheduledNodes>
           <dayOfWeek>1</dayOfWeek>
           <hourOfDay>14</hourOfDay>
           <schedulednode>
               <nodeIdentity>N1</nodeIdentity>
               <locationIdentity>c0e486a0-bbab-11e2-863b-028e03094f8e</locationIdentity>
               <isLocal>true</isLocal>
               <isUp>true</isUp>
               <lastStatusChange>0</lastStatusChange>
               <role>AV</role>
           </schedulednode>
           <schedulednode>
               <nodeIdentity>N3</nodeIdentity>
               <locationIdentity>5480f515-bbad-11e2-8301-028e03094f8c</locationIdentity>
               <isLocal>false</isLocal>
               <isUp>true</isUp>
               <lastStatusChange>0</lastStatusChange>
               <role>T</role>
           </schedulednode>
           <schedulednode>
               <nodeIdentity>N2</nodeIdentity>
               <locationIdentity>478c766f-bbad-11e2-877a-028e03094f8d</locationIdentity>
               <isLocal>false</isLocal>
               <isUp>true</isUp>
               <lastStatusChange>0</lastStatusChange>
               <role>AV</role>
           </schedulednode>
  

Download the full sample schedule.xml file.

1. Log in to a node, then click the REPLICATION GROUPS tab. Click the VIEW link for the replication group that you wish to make a schedule.
   

   Scheduling is done through replication group settings

2. The replication group's pop-up window opens, showing the member nodes together, along with their current (scheduled) roles. Click the CONFIGURE button.
   

   Configure

   Membership views show what is scheduled, not necessarily what is currently active
   The roles and membership displayed in the popup is based upon the agreed schedule. It is the setup that should be in place if everything is running smoothly. It may not accurately represent the state of the replication group, due to a delay in processing on a node or if a process has hung. This is not a cause for concern but you must be aware that the displayed membership is an approximation based on the information currently available to the local node.
3. The replication groups configuration screen will appear. You may notice that to the left a Role Schedule is noted. By default this will show as DISABLED. Click the Configure Schedule button, in the right-hand corner.
   

   Role Schedule: Disabled (for now)

4. The Schedule screen appears. The main feature of the screen is a table that lists all the nodes in the replication group, set against a generic day (midnight to midnight) that is divided into hourly blocks. Each hourly block is color-coded to indicate the specific node's type.

   Vanilla Scheduling - no changes to type over time

   Note: In the image below, NodeSanFrancisco is blue, indicating that it is a Passive Voter. The hourly blocks associated with NodeChengdu are magenta, indicating that it is a pure voter. The blocks for NodeParis are yellow, indicating that it is an Active Voter.

5. To change the schedule, click a block. You can click any block because the New Scheduled Configuration form lets you modify any hours for any available node.
   
   

   New Schedule form

   Frequency

   Select from the available frequency patterns: Daily, Weekly, Monday-Friday or Saturday to Sunday.

   From

   The starting hour for the new schedule, e.g. 00 for the start of the day.

   To

   The hour at which the scheduled changes end, e.g. 24 would effectively end the scheduled change at midnight.

   Node list

   The member nodes are listed, in graphical form, colour coded to their type.

6. Click the node icon to change its type.

   Note: In this example NodeSanFransisco is changed to a Tie-breaking Passive Voter, then NodeAuckland is changed into a Tie-breaker.

   
   

   Swapping roles

   When all node changes have been made, click the SAVE button to continue, or the CANCEL button if you change your mind.
7. The schedule view will now change to show the changes that you make. You must click the Save Schedule button for the changes to be applied.
   With all necessary changes made, you need to review the change to the schedule table and then click SAVE SCHEDULE button.
   Changing role of the managing node
   You can change the managaing node to Active, Active Voter and Active Voter Tiebreaker, though not to any passive roles. If you want to make it a passive node you must switch the manager to an active node (A, AV, AVT) because the manager needs to be able to propose schedule changes and therefore be active.

4.5.2 Disable the schedule

If you need to stop any and all scheduled rotations, e.g. in an emergency to prevent losing quorum:

1. Click the REPLICATION GROUPS tab, then select your group:
   
2. Click Disable Schedule:
   
   Note the warning message.

4.6 Single-node configurations

No replication takes place when you have a single node. Other functionality works, but you may have issues relating to quorum.

You may have a single-node configuration in the following situations:

• If the other nodes in a replicaton group are disconnected: contact WANdisco Support.
• To support WANdisco Access Control Plus's standalone mode: for more information see ACP guide.

5. Repositories

5.1 Add a repository

When you have added at least one Replication Group you can add repositories to your node:

1. Click the REPOSITORIES tab, then the ADD button.
   
   

   Repositories > ADD

2. Enter the Repository's name, the file system path (full path to the repository), and use the drop-down to select the replication group.
3. Tick the Global Ready-only checkbox to set the repository to be read-only. You can deselect this later. Click ADD REPO.
   
   

   Repositories > Enter details then click ADD REPO

   Repository stuck in Pending state
   

   If a repository that you added gets stuck in the deploying state, you see this on the Dashboard, in the Replicator Tasks window. You can cancel the deployment and try adding the repository again. To cancel a deployment, go to the Replicator Tasks window and click the Cancel Task link.

4. When added, a repository appears in a list on the REPOSITORIES tab. The list provides the following details.
   
   

   Repositories listed

   Name

   The name of the repository - this will be the same as the folder name in the Git directory.

   Path

   The full path to the Repository.

   Replication Group

   The Replication Group in which the repository may be replicated.

   Size

   The file size of the repository.

   Youngest Rev

   The youngest (latest) revision in the repository. Comparing the youngest revisions between replicas is a quick test that a repository is in the same state on all sites.

   Last Modified

   The timestamp for the last revision, which provides a quick indicator for the last time a Git user made a change.

   Global RO

   Checkbox that indicates whether the repository is globally Read-only, that is Read-only at all sites.

   Local RO

   Checkbox that indicates whether the repository is locally Read-only, that is Read-only to users at this node. The repository receives updates from the replicas on other sites, but never instigates changes itself.

5.1.1 Automated repository deployment

Although you can deploy new repositories automatically using the REST API, the changing state of repositories is not chained to the deployment task. It's therefore possible that a repository that is successfully created may not be fully deployed, and this could result in errors if your automation moves ahead too soon.

Temporary solution for automated repository management
Ensure that a short delay is added to the deployment of new repositories. You can query the newly added repository through the REST API (using the repository endpoint) in order to verify that it is in fact deployed.

5.2 Remove a repository

You can remove repositories from GitMS. Follow this quick procedure.

Important requirement when performing a repository dump / load
Commits may fail if a repository is removed and, in some way, historically edited before re-introducing the repository to GitMS. This may happen, for example, if you use a Git dumpfilter to remove content/revisions due to sensitive content, while maintaining the same UUID. This is caused by existing full-text cache references no longer being applicable.
Workaround: Restart Apache and GitMS before you use the repository.
1. Log in to the admin console of one of your nodes. The node must be a member of a replication group in which the repository is replicated, otherwise it is not listed. Click the Repositories tab to see it.
   
   

   Login

2. On the Repositories tab, click the line that corresponds with the repository that you want to remove.
   
   

   Repositories.

3. When a repository is highlighted (in yellow), the REMOVE button becomes available. Click it.
   

   Remove

   A dialog box appears called Remove repository from replication group. This confirms that removing a repository from a replication group stops any changes that are made to it from being replicated. However, no repository data is removed.

5.3 Edit a repository

To edit a repository's properties after they have been set up in GitMS:

1. Log in to the admin console of one of your nodes. The node must be a member of a replication group in which the repository is replicated, otherwise it is not listed. Click the Repositories tab to see it.
   
   

   Log in

2. On the Repositories tab, click the line that corresponds with the repository that you want to edit. Then click the Edit button. You can also click the repository's name to go directly to the Edit screen.
   
   

   Repositories

3. You now see the Repository screen appear. For here you can trigger a consistency check, bring the repository to a coordinated stop or start a repair if a problem has been detected.
   
   

   Edit Repository.

   Name

   The name of the repository assigned when it was placed under MultiSite control.

   Path

   The absolute path to the repository on each node.

   Replication Group

   I

   Last Modified

   The filesize of the repository, noted here in bytes.

   Size

   The timestamp of the last modification to the repository.

   Sized added today

   Indicates the change in repository size over the last 24 hours, which provides a quick cue for recent traffic levels.

   Youngest Revision

   This entry is not used by GitMS. It is used when MultiSite is configured to run with Subversion.

   Commits made today

   Provides a count of the repository changes that have been made over the last 24 hours. This is used to measure recent activity levels on the repository.

   Global Read-only

   Changes the Read-only setting, enable or disable the repository Global Read-only setting. When enabled, the repository will not be writable either locally or globally. This is used to lock a repository from any changes.

   Local Read-only

   Changes the Read-only setting, enable or disable the repository Local Read-only setting. When enabled, the repository will not be writable, either for local users or for the replication system (that would push changes made to the repository on other nodes). However, changes that come from the other nodes are stored away to be played out as soon as the read-only state is removed.

   Reload

   For a refresh of the repository information to pick up any changes that may have occured since loading the screen.

5.4 Repository synchronized stop

Use the repository Sync Stop function to stop replication between repository replicas. You can do this either:

• On a per-repository basis
• On a replication group basis where replication will be stopped for all associated repositories

To stop all nodes, use the Sync Stop All command via the Nodes tab.

Repository stops are synchronized between nodes using a stop proposal to which all nodes need to agree. Although not all nodes stop at the same time they do all stop at the same point.

1. Log in to a node's browser-based UI and click the Repositories tab. Click the repository that you want to stop replicating.
   
   
2. With the repository selected, click the Sync Stop button. A growl message confirms that a synchronized stop has been requested. Note that the process may not be completed immediately, especially if there are large proposals transferring over a WAN link.
   
   
3. Refresh the screen to see that a successfully sync stopped repository has a status of Stopped and is Local RO (locally read-only) at all nodes.

5.5 Repository synchronized start

When you restart replication after a Synchronized Stop, you must start the stopped replication in a synchronized way.

1. Click a stopped repository and click the Sync Start button.
   
   
2. The repository will stop being Local Read-only on all nodes and will restart replicating again.

5.6 Stop repositories on a node

In some situations, such as performing a repository backup, you may need to stop writes to the local repository replica. You can do this in the following ways:

• Sync-Stop the Node> This takes the node offline because it stops accepting proposals.
• Stop Node via API call: Use the API to stop all repositories on a node. This method is less obtrusive because it doesn't stop the node from otherwise operating.

5.6.1 Stop node via API call

With this method, proposals are still delivered to the node and the node can still participate in voting, but the proposals are not executed until the output is restarted.

This is supported in the API with the RepositoryResource methods:

PUT <server>:<port>/api/repository/{repositoryId}/stopoutput

Read more about stopoutput.

This command takes one argument, NodeListDTO nodeListDTO, which is the list of nodes where the repo output will be stopped. In this case the list only includes NodeX.

Note that while the output is stopped it shows in the UI as LocalReadOnly.

PUT <server>:<port>/api/repository/{repositoryId}
/startoutput

Read more about startoutput.

This command takes one argument, NodeListDTO nodeListDTO, which is the list of nodes where the repo output will be started. In this case the list only includes NodeX.

PUT: <server>:<port>/api/replicator/stopall

If you don't need the stop to be coordinated then the following method is simpler and immediately stops the output of proposals on all repositories on the node it is executed against:

PUT: <server>:<port>/api/replicator/stopall

This stops all repositories on the node on which it is invoked (with no coordination).

To start them all again call:

PUT: <server>:<port>/api/replicator/startall

5.7 Reuse a repository

Take care when reusing a Git repository.
You need to edit the Git configuration file.

You might reuse a repository after improperly removing it from replication, by copying it from backup/restore for example. If so, you need to hand edit the Git configuration file (in repo/config) to change replicated = true to replicated = false or remove the line. Do this before putting it in place on any replicated server as a local, i.e. non-replicated, repository.

6. Back up GitMS data

You can back up GitMS's own database if you want to quickly restore a node.

Only MultiSite Settings are backed-up
This procedure backs up GitMS's internal Prevayler database. It does not back up your Git repository data or any other system files that you should also be backing up.

Create a backup of the current installation by invoking the following API call:

curl --user <username>:<password> -X POST http://[node_ip_address]:8082/api/backup

This creates a backup folder in the [install dir]/git-multisite/replicator/database/backup directory.

6.1 Restore GitMS data

Use the following procedure to restore GitMS settings from your backup after reinstalling and starting a node:

1. Shut down the node.
2. Run the following jar file. This needs to be run from the replicator directory.

   java -jar git-ms-replicator-gitmsrestore.jar -c path/to/application.properties -r path/to/back-up-folder

   path/to/application.properties is properties/application.properties

   path/to/backup by default is /opt/wandisco/git-multisite/replicator/database/backup/

   For example

   java -jar git-ms-replicator-gitmsrestore.jar -c properties/application.properties -r /opt/wandisco/git-multisite/replicator/database/backup/2017-03-15T14:00:07+0000_DConE_Backup

3. Restart the node.

7. Running GitMS with Apache

This section describes how to set up Apache with GitMS.

Using mod_dav or WebDav to administer GitMS is not supported.

7.1 Before you start

Note the following requirements:

• Java JDK 1.7 or later
• Apache HTTP Web Server
• Git 2.0+ (see Yum Configuration)
• GitMS (installed and configured for replication, using gitms as the user):
  Ensure that at least one repository has been replicated successfully and is in the directory /home/gitms/repos/ on each machine.
• Access Control Plus
• Generic File Replication package (used to link Access Control Plus to a GitMS installation)
Notes
If you are using ONLY Apache to access both GitMS and SVN MSP, then you can use just one account.
If you are ONLY accessing either GitMS or SVN MSP via SSH then you can use just one account.
However, if you are using Apache and SSH to access both Git and SVN repositories you need to use two accounts (because each account has only a single authorized_keys file).
When using SSH for both SVN and Git access, the account used to run Apache must be the same account that is used to install SVN MSP.
Running GitMS with the same system account as Apache (for example, via a dedicated apache user account) can cause problems. For production we recommend that you set up a dedicated user account: in Red Hat that's an account with a UID of 500 or higher.
In some cases you may need to run GitMS using the apache user, for example if you are deploying with both Git and SVN repository replication. To make the apache user account suitable for running GitMS you need to ensure that:
• user has a valid shell.
• Apache doesn't invoke the SuexecUserGroup directive, which ensures that apache user is set to run CGI programs.
• Repositories should be set to be owned by apache.

7.2 Install Apache

yum install httpd mod_ssl

Next, ensure that Apache starts on system restart:

chkconfig --add httpd

service httpd start

7.3 Configure SeLinux

If SeLinux is running:

1. Enable access to home directories, i.e. /usr/home/gitms:

   setsebool -P httpd_enable_homedirs on 

2. Install semanage:

   usermod -a -G apache gitms

   yum -y install policycoreutils-python 

3. Allow httpd read/write access to /home/gitms:

   chcon -R -t httpd_sys_rw_content_t /home/gitms

   chcon -R -t httpd_sys_rw_content_t /opt/wandisco/git-multisite/replicator/content_delivery

4. Allow the update script to make a network connection to the Java service:

   setsebool -P httpd_can_network_connect on

   setsebool -P git_system_enable_homedirs on

7.4 Configure IPTables

/etc/sysconfig/iptables should look like:

*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [11:12222]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 6789 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 8082 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 8085 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 9001 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 443 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT
# Completed on Mon Jul 29 13:52:30 2013
Increased the OUTPUT ACCEPT from 2222 to 12222 (allows outgoing connections up to port 12222)
Allow incoming connections on 6789, 8082, 8085, 9001 and 443.
  

7.5 Create your HTTP password file

htpasswd -c /var/www/passwd username

Change the ownership of the passwd file:

chown apache:apache /var/www/passwd

7.6 Create git-http-backend wrapper script

The script executed by suexec must be under /var/www:

mkdir -p /var/www/bin

Add the following to /var/www/bin/git-http-backend:

#!/bin/bash
GIT_PROJECT_ROOT=/home/gitms/repos
# This value should be configured to match the base location of repos on disk
export GIT_BASEDIR=$GIT_PROJECT_ROOT
# This should be set to the gitms users home directory - reset here because SUexec strips it out
export HOME=/home/gitms
export GIT_HTTP_EXPORT_ALL=true
# Execute gitms_shell script
exec /opt/wandisco/git-multisite/bin/gitms_shell $REMOTE_USER
  

The script and the directory it is in must be owned by the user who will be executing the script:

chown -R gitms:gitms /var/www/bin

Make the wrapper script executable:

chmod +x /var/www/bin/git-http-backend

Tell selinux that /var/www/bin has httpd exectuable scripts:

semanage fcontext -a -t httpd_sys_script_exec_t /var/www/bin  

restorecon /var/www/bin

7.7 Add Apache config

Copy the following into /etc/httpd/conf.d/repo.conf:

<VirtualHost *:80>
DocumentRoot /home/gitms/repos
ServerName git.example.com
  

<Directory "/home/gitms/repos">
Allow from All
Options +ExecCGI
AllowOverride All
</Directory>
  

<Location /repos>
AuthType Basic
AuthName "Private Git Access"
AuthUserFile "/var/www/passwd"
Require valid-user
</Location>
  

uexecUserGroup gitms gitms
ScriptAlias /repos /var/www/bin/git-http-backend
</VirtualHost>
  

7.8 Allow Git pushes

Change to the directory your repo is located in:

su - gitms -s /bin/bash
cd /home/gitms/repos/bar.git
git config http.receivepack true
git config core.sharedRepository group
  

Do this in each repo and on all replicas.

7.9 Restart Apache and test

service httpd restart
git clone http://192.168.122.1/repos/bar.git
  

7.10 HTTPS support: Generate certificates

You can use tools such as easy-rsa to generate certificates.

You need to have the Epel rep installed to use this:

wget http://dl.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm
rpm -Uvh epel-release-6-8.noarch.rpm
yum install easy-rsa
cp -r /usr/share/easy-rsa/2.0 .
cd 2.0/
source vars
./clean-all
./build-ca
./build-key-server git-http1     (when prompted for the CommonName use the system IP address if the system does not have a register DNS name)
./build-key-server git-http2
./build-key-server git-http3
  

This generates three server certs/keys as well as the ca cert/key in the ./keys directory.

Copy a cert and key into /etc/apache on each node.

cp ./keys/git-http1.crt /etc/httpd

cp ./keys/git-http1.key /etc/httpd

chown apache:apache /etc/httpd/git-http1.crt /etc/httpd/git-http1.key

7.11 Modify Apache config to support SSL

Edit /etc/httpd/conf.d/repo.conf:

<VirtualHost *:443>
DocumentRoot /home/gitms/repos
ServerName git.example.com
SSLEngine on
SSLCertificateFile /etc/httpd/git-http1.crt
SSLCertificateKeyFile /etc/httpd/git-http1.key

<Directory "/home/gitms/repos">
Allow from All
Options +ExecCGI
AllowOverride All
</Directory>

<Location /repos>
AuthType Basic
AuthName "Private Git Access"
AuthUserFile "/var/www/passwd"
Require valid-user
</Location>

SuexecUserGroup gitms gitms
ScriptAlias /repos /var/www/bin/git-http-backend
</VirtualHost>
  

Restart service:

service httpd restart

7.12 Add the CA certificate to client system

The server certificate generated will not be recognized by your client. You can either:

• Turn off strict SSL checking:

  git config --global http.sslVerify false

• Add the CA certificate to the client machine. You need to add the ./keys/ca.crt file to the set of CA certificates that the client system accepts. Follow these instructions.

7.13 Test

$ git push

WARNING: gnome-keyring:: couldn't connect to: /tmp/keyring-P25n4M/pkcs11: No such file or directory
Password for 'https://test@192.168.122.207':
remote: GitMS - update replicated.
To https://test@192.168.122.207/repos/bar.git
 d434f14..8b7bab0 master -> master$
  

7.14 Manual setup for audit logging

Use this procedure to account for some configuration relating to the audit feature that is currently missing from the installer.

7.14.1 Sender configuration

Setting sources

This value sets the sources that flume will monitor: acpSender.sources =

• Example: To monitor all three set: acpSender.sources = svnServeSource svnWebdavSource gitmsSource
• Example: To monitor just Webdav: acpSender.sources = svnWebdavSource

Setting Log locations

Settings that apply to Webdav and GitMS:

acpSender.sources.svnWebdavSource.type = exec
acpSender.sources.svnWebdavSource.command = tail -F /var/log/httpd/access_log
acpSender.sources.svnWebdavSource.restart = true
acpSender.sources.svnWebdavSource.channels = memChannel

acpSender.sources.gitmsSource.type = exec
acpSender.sources.gitmsSource.command = tail -F /opt/wandisco/git-multisite/replicator/log/gitms.log
acpSender.sources.gitmsSource.restart = true
acpSender.sources.gitmsSource.channels = memChannel
      

The system user that runs GitMS MUST have permissions to read all the locations that you configure.

7.14.2 Avro settings

If you're running with Apache Avro, apply these settings:

Receiver: /opt/wandisco/flume-scm-access-control-plus/conf/flume_acp_receiver.properties
acp_agent.sinks.acpSink.acp_receiver_host = <Access Control Plus IP>
acp_agent.sinks.acpSink.acp_receiver_port = <Access Control Plus PORT>
acp_agent.sources.avroSrc.bind = <Git MultiSite IP>
acp_agent.sources.avroSrc.port = <FLUME PORT>

Sender: /opt/wandisco/flume-git-multisite/conf/acp_sender.conf
acpSender.sinks.acpSink.hostname = <Git MultiSite IP>
acpSender.sinks.acpSink.port = <FLUME PORT>
  

8. GitMS authentication and authorization

GitMS authentication is performed by a third party service. When the authentication is complete, details of the requested git operation and a username are passed back to GitMS.

The operation and username are then checked internally by GitMS to ensure sufficient permissions are available to perform the operation.

Authentication can use either:

• SSH
• HTTP with an authentication mechanism such as htpasswd or LDAP

8.1 Authentication

Authentication can be by either SSH directly or via an HTTP authentication mechanism. It is normal to only use one or the other, but it is possible to use both in parallel. This section describes examples of using either SSH or Apache for authentication.

8.1.1 Authentication through Apache

Apache authentication allows users to communicate via the HTTP(S) protocol. This is beneficial in environments with heavily restricted firewalls, as the usual ports 80 and 443 are used for communication.

Git has two HTTP-based protocols, Git over HTTP and Smart HTTP. GitMS only supports the Smart HTTP protocol, because of its better functionality, especially in speed of operation.

The information given here assumes that:

• GitMS is already set up and configured for replication.
• GitMS runs with the gitms user and gitms group.
• The base directory for managed repositories is /home/gitms/repos/.
• The apache2+suexec package is installed.
• If an htpasswd file is required, it is stored in /var/www/passwd.
• Apache runs as user apache.
• Apache configuration is in its default directory, /etc/httpd/conf.d/.

You can configure Apache to authenticate against either an internal file (htpasswd) or an LDAP directory service.

Authentication by htpasswd

• Create the htpasswd file:

  htpasswd -c /var/www/passwd <username>

  Only use the -c option when you first create an htpasswd file. If you use this to reference a pre-existing file, any details in the file are overwritten with the username you specify.

• Add users to an existing file:

  htpasswd /var/www/passwd <username>

• Configure htpasswd. Add the repo.conf file in the /etc/httpd/conf.d/ directory with the following contents:

  
  <VirtualHost *:80>
  # 80 is the port the webserver will bind to
  DocumentRoot /home/gitms/repos
  # The base directory for repositories managed by GitMS
  ServerName git.example.com
  RewriteEngine On
  RewriteCond %{REMOTE_USER} ^(.*)$
  RewriteRule ^(.*)$ - [E=R_U:%1]
  RequestHeader set X-Remote-User %{R_U}e
  
  <Directory "/home/gitms/repos">
  Allow from All
  Options +ExecCGI
  AllowOverride All
  </Directory>
  
  <Location /repos>
  # This matches the location in the requesting url,
  # for example, matches against request http://<ip>/repos/
  AuthType Basic
  AuthName "Private Git Access"
  AuthUserFile "/var/www/passwd"
  Require valid-user
  </Location>
  
  SuexecUserGroup gitms gitms
  ScriptAlias /repos /var/www/bin/git-http-backend
  # This script alias redirects matches made
  # earlier to a script we will create later
  </VirtualHost>
        

Authentication by LDAP

Apache can use an LDAP directory to authenticate against. Unlike htpasswd, you do not have to maintain a separate passwd file.

Add a file called repo.conf in the /etc/httpd/conf.d/ directory with the following contents:

<VirtualHost *:80>
DocumentRoot /home/gitms/repos
ServerName git.example.com
RewriteEngine On
RewriteCond %{REMOTE_USER} ^(.*)$
RewriteRule ^(.*)$ - [E=R_U:%1]
RequestHeader set X-Remote-User %{R_U}e

<Directory "/home/gitms/repos">
Allow from All
Options +ExecCGI
AllowOverride All
</Directory>

<Location /repos>
AuthType Basic
AuthName "Git Repos"
AuthBasicProvider ldap
AuthzLDAPAuthoritative off
AuthLDAPURL "ldap://LDAP-IP:389/CN=CN-details,DC=DC-details,DC=DC-details?uid"

#If the LDAP directory requires a bind user and password:
AuthLDAPBindDN "CN=Administrator,CN=Users.DC=sr,DC=wandisco,DC=com"
AuthLDAPBindPassword password

Require valid-user
</Location>

SuexecUserGroup gitms gitms
ScriptAlias /repos /var/www/bin/git-http-backend
</VirtualHost>
  

Configuration using HTTPS

You can set up Apache to use HTTPS rather than HTTP. This is preferred in Enterprise settings due to the security benefits.

Using HTTPS with htpasswd

Add a file called repo.conf in the /etc/httpd/conf.d/ directory with the following contents:

<VirtualHost *:443>
DocumentRoot /home/gitms/repos
ServerName git.example.com
RewriteEngine On
RewriteCond %{REMOTE_USER} ^(.*)$
RewriteRule ^(.*)$ - [E=R_U:%1]
# The following two lines will redirect port 80 (HTTP) to 443 (HTTPS)
# if SSL/TLS is always required:
RewriteCond %{HTTPS} off
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}
RequestHeader set X-Remote-User %{R_U}e
SSLEngine on
SSLCertificateFile /etc/httpd/git-http1.crt
SSLCertificateKeyFile /etc/httpd/git-http1.key

<Directory "/home/gitms/repos">
Allow from All
Options +ExecCGI
AllowOverride All
</Directory>

<Location /repos>
AuthType Basic
AuthName "Private Git Access"
AuthUserFile "/var/www/passwd"
Require valid-user
</Location>

SuexecUserGroup gitms gitms
ScriptAlias /repos /var/www/bin/git-http-backend
</VirtualHost>
  

Using HTTPS with LDAP

Add a file called repo.conf in the /etc/httpd/conf.d/ directory with the following contents:

<VirtualHost *:443>
DocumentRoot /home/gitms/repos
ServerName git.example.com
RewriteEngine On
RewriteCond %{REMOTE_USER} ^(.*)$
RewriteRule ^(.*)$ - [E=R_U:%1]
#  The following two lines will redirect port 80 (HTTP) to 443 (HTTPS)
# if SSL/TLS is always required:
RewriteCond %{HTTPS} off
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}
RequestHeader set X-Remote-User %{R_U}e
SSLEngine on
SSLCertificateFile /etc/httpd/git-http1.crt
SSLCertificateKeyFile /etc/httpd/git-http1.key

<Directory "/home/gitms/repos">
Allow from All
Options +ExecCGI
AllowOverride All
</Directory>

<Location /repos>
AuthType Basic
AuthName "Git Repos"
AuthBasicProvider ldap
AuthzLDAPAuthoritative off
AuthLDAPURL "ldap://LDAP-IP:389/CN=CN-details,DC=DC-details,DC=DC-details?uid"

#If the LDAP directory requires a bind user and password:
AuthLDAPBindDN "CN=Administrator,CN=Users.DC=sr,DC=wandisco,DC=com"
AuthLDAPBindPassword password

Require valid-user
</Location>

SuexecUserGroup gitms gitms
ScriptAlias /repos /var/www/bin/git-http-backend
</VirtualHost>
  

Create git-http-backend script

Create a script called git-http-backend as follows:

#!/bin/bash
GIT_PROJECT_ROOT=/home/gitms/repos
# This value should be configured to match the base location of repos on disk
export GIT_BASEDIR=$GIT_PROJECT_ROOT
export GIT_HTTP_EXPORT_ALL=true
# Execute gitms_shell script
exec /opt/wandisco/git-multisite/bin/gitms_shell $REMOTE_USER
  

This script location should match where you set it to be called from in the repo.conf file on the ScriptAlias line - we have used /var/www/bin/git-http-backend. It is important to ensure that this script is executable, and the script and directory it is in are both owned by the "gitms" user - not "apache".

This script is run by suexec, and will therefore run as the user that owns it, which we require to be "gitms":

chmod +x /var/www/bin/git-http-backend
chown -R gitms:gitms /var/www/bin
  

Configuration of SELinux

If you are running SELinux, additional configuration is required, as follows:

#enable access to home directories, i.e. /home/gitms
setsebool -P httpd_enable_homedirs on

usermod -a -G apache gitms

#install semanage
yum -y install policycoreutils-python

#allow httpd read/write access to /home/gitms
chcon -R -t httpd_sys_rw_content_t /home/gitms

#allow httpd read/write access to /home/gitms
chcon -R -t httpd_sys_rw_content_t /opt/wandisco/git-multisite/replicator/content_delivery

#allow the update script make network connection to the Java service
setsebool -P httpd_can_network_connect on

setsebool -P git_system_enable_homedirs on

#configure selinux for suexec on git-http-backend
semanage fcontext -a -t httpd_sys_script_exec_t /var/www/bin
restorecon /var/www/bin
  

9. Install GitWeb

GitWeb is an optional open source component that provides read-only access to your Git repositories via a Web interface. It allows you to review recent commits, logs, history, and other metadata about the git repositories.

1. Install gitweb from yum:

   yum install gitweb 

2. Locate gitweb installation files, e.g. in their own folder:

   mkdir /var/www/gitweb

3. Copy gitweb.cgi, gitweb.perl and the static directory into here.
4. Set permissions (because you are using suexec, this must be wandisco):

   chown -R wandisco:wandisco /var/www/gitweb

5. Modify /etc/gitweb.conf to reflect site changes:

   our $projectroot = "/opt/repos";
   our @git_base_url_list = qw(http://<server-ip-or-name>/gitweb)
         

   Save changes.
6. Configure Apache to use gitweb (see Appendix A example) and restart Apache after making any changes.
7. Browse to http://<server-ip-or-name>/gitweb to verify installation and configuration.

Example GitWeb Configuration

#GitWeb Configuration
Alias /gitweb /var/www/gitweb

<Directory /var/www/gitweb>
  Options +ExecCGI
  AddHandler cgi-script.cgi
  DirectoryIndex gitweb.cgi
</Directory>
  

Final step

After you've finished updating your configuration, the last step is to restart apache:

service httpd restart

10. SSH authentication

SSH authentication is available for Git and supported by GitMS. It is simple to set up and is attached to a service which is often already enabled. Occasionally, firewall rules may block clients and and so this may not be popular with Windows users.

10.1 Requirements

SSH authentication is done using the SSH daemon and public/private keypairs. Requirements are as follows:

• Git operations are run through a single shared user account.
• The user account is the same as the account that runs the GitMS service, for example, "gitms".
• All authentication is done by public/private keypairs. No password or certificate based authentication.
• The ssh daemon uses the authorized_keys file in "~/.ssh/" to do the public key lookup.
• The shared user account must have a regular shell login in /etc/passwd. Using, for example, git-shell, causes the command GitMS relies on to fail.

10.2 Authorized keys

GitMS requires use of the 'command' keyword to be attached to each key in the authorized_keys file. This associates a username with the key used to log in by using it as an argument to the gitms_shell script. By default, the script is found in /opt/wandisco/git-multisite/bin/gitms_shell, but this can vary between installations.

For example:

command="/opt/wandisco/git-multisite/bin/gitms_shell user1" ssh-rsa <SSH_KEY>

If your script is in a different location, update this line accordingly.

11. Authorization

11.1 GitMS configuration

Authorization is not enabled by default. To change this, and any other authorization settings, edit the application.properties file. The location is /opt/wandisco/git-multisite/replicator/properties/ for default installations.

# enable/disable authz module
# Default:
#gitms.authz.enabled=false
gitms.authz.enabled=true

# Set the file location of the authz file
# Default: /opt/wandisco/git-multisite/replicator/properties/auth.authz
#gitms.authz.file=/opt/wandisco/git-multisite/replicator/properties/auth.authz
gitms.authz.file=<filepath>

# Set the default permissions policy to DENY or ACCEPT
# for requests without a specific rule
# Default:
#gitms.authz.policy=DENY;
gitms.authz.policy=<policy>

# Set the polling period to detect changes in the AuthZ file
# NOTE: The trailing L is important, as it indicates this
# number is a Long value
# Default:
#gitms.authz.poll.timer=50L
gitms.authz.poll.timer=<numberInMilliseconds>L
  

11.2 AuthZ File format

Begin with the following header:

# Git AuthZ Version:1.0

To define teams, add entries such as:

[groups]
team1 = user1, user2
team2 = user3, user4
  

To define rules against individual repositories, the repository path is used as a unique identifier, which is added in square brackets. For example:

[/home/gitms/repos/Repo1.git]
user1 = R+W+
@team1 = R+W+C+D+
  

AuthZ rules are applied to users and teams using the following tokens:

• R(+/-) - Allow/Deny user read access
• W(+/-) - Allow/Deny user write access
• C(+/-) - Allow/Deny user create access
• D(+/-) - Allow/Deny user delete access

Note: To create a branch, a user must have both Create and Write access, i.e. C+W+ .

Rules can also be applied at branch level, as follows:

[/opt/gitRepos/Repo1.git:BRANCH/secure]
  rick = W-

A complete example:

# Git AuthZ Version:1.0
[groups]
team1 = wayne, rick
team2 = wayne, allan
[/home/gitms/repos/Repo1.git]
rick = R+W+
@team2 = R+W+C+D+
[/home/gitms/repos/Repo1.git:BRANCH/secure]
rick = W-
  

11.2.1 AuthZ rule application

The AuthZ rules have 2 hierarchies which determine whether a user has a requested level of access:

• Resource-level hierarchy (Repo->Branch|Tag)
• User-level hierarchy (User->Team)

Rule conflicts within these hierarchies are resolved by picking the rule which is most granular. In our above example, "rick" would have Read and Write access to the whole repo, except for the "secure" branch, on which he wouldn't have write access. Apply these rules as follows:

Apply these rules as follows. If a user called Tom is requesting Write access to branch master on repo Repo1.git, the AuthZ rule resolution is:

1. Determine that the repo Repo1.git exists on the local node. If not, error out.
2. Lookup Tom's rules for branch master on Repo1.git
3. If rules exist for Tom which grant/deny the access he needs, apply them
4. If rules do not exist for Tom, check each of the teams Tom is a part of.
5. Tom may be a part of multiple teams which have conflicting rule permissions. One could grant, and one could deny access. In the case where the permissions conflict at the same point on the hierarchy, we always pick the most permissive rule.
6. If Tom's teams have no rules for the master branch either, we move up the Resource hierarchy and check the permissions assigned against Repo1.git.
7. If Tom has permissions assigned against Repo1.git, apply them.
8. If no relevant rules exist for Tom, check again for each of the teams he is a part of.
9. If no permissions exist at this point, apply the default policy permissions. (gitms.authz.policy)

Logging

The GitMS replicator makes several log entries relating to AuthZ acitivities.

INFO: AuthZ: File change detected for <filename>

WARNING: AuthZ: Auth file invalid <errorMessage>

INFO: AuthZ: Time taken for parsing: <timeTaken>

DEBUG: AuthZ: Authorization disabled, accepting request

INFO: "AuthZ: Request [user: <username, repoPath: <repoPath>, ref:<refName>, accessRequested:<accessRequested>] received"

WARNING: AuthZ: There was an error with the Authorization setup, request declined

INFO: AuthZ: <ACCEPT/DENY> Permissions applied for [user: <username, repoPath: <repoPath>, ref:<refName>, accessRequested:<accessRequested>]

INFO: AuthZ: No permissions specified for [user: <username, repoPath: <repoPath>, ref:<refName>, accessRequested:<accessRequested>] using default policy: <ACCEPT/DENY>

AuthZ: Request received for non-existent user: <username> applying default policy: <ACCEPT/DENY>