Setting Up Local Git Cluster

Build a resilient, self-hosted Git repository storage system using Ubuntu and GlusterFS. This tutorial provides a straightforward guide to setting up distributed Git storage across multiple nodes for data redundancy and access through Gitea. An excellent local alternative when enterprise solutions aren’t available.

Setting Local Git Cluster

[embed]Multi-Hypervisor Homelab -Exploring VMWare ESXi Alternatives This post is about my journey of setting up multi hypervisor home lab, exploring ESXi, Proxmox, XCP-ng, Nutanix and…medium.com

As part of my home lab setup, before getting too far with standard services or custom services, I wanted to get source control setup. I did not need anything fancy but still wanted to setup with typical corporate requirements:

• Not single point of failure
• Not disable TLS as shortcut

So, I started with following goals:

• Install Git on three Ubuntu machines
• Load balance using NGINX
• Use some distributed file system like GlusterFS
• Deploy certs for TLS

For distributed file system, I used GlusterFS.

What is GlusterFS

GlusterFS is a scalable, distributed file system that allows us to combine storage resources from multiple servers into a single, unified storage pool. It is designed to handle large amounts of data and provides high availability, fault tolerance, and redundancy by distributing data across multiple nodes. GlusterFS is particularly useful in environments where we need to scale storage capacity and performance easily, such as in cloud, virtualization, and containerized environments

Basic Concepts in GlusterFS

• Bricks: The basic unit of storage in GlusterFS. A brick is a directory on a server that GlusterFS uses to store data.
• Volumes: A logical collection of bricks. Volumes are the units that clients interact with and can be configured for different types of redundancy (e.g., replicated, distributed).
• Replicated Volume: Ensures data redundancy by storing multiple copies of data across different bricks.
• Distributed Volume: Spreads data across multiple bricks without replication, useful for scaling storage capacity without redundancy.
• Distributed Replicated Volume: Combines distribution and replication, spreading data across nodes while maintaining redundancy.

Getting Started with GlusterFS

To start using GlusterFS, we’ll :

1. Install GlusterFS on all participating nodes.
2. Configure the nodes to recognize each other as peers.
3. Create and start volumes by combining bricks from multiple nodes.
4. Mount the volumes on client systems using standard protocols like NFS or directly with the GlusterFS client.

Using Git with GlusterFS (Distributed File System)

GlusterFS allows us to set up a distributed and replicated storage system, which can be mounted on all three VMs, ensuring that our Git repositories are available on each VM.

Step 1: Install Git and GlusterFS on All VMs

# Install Git
sudo apt update
sudo apt install -y git

# Install GlusterFS
sudo apt install -y glusterfs-server
sudo systemctl start glusterd
sudo systemctl enable glusterd

Step 2: Set Up GlusterFS Cluster

Prepare the Bricks: On each VM, create a directory that will be used as a storage brick for GlusterFS.

sudo mkdir -p /data/glusterfs

Peer the Nodes: On one of the VMs, add the other nodes to the GlusterFS cluster.

# Replace vm2 and vm3 with the hostnames or IP addresses of the other VMs

sudo gluster peer probe vm2
sudo gluster peer probe vm3

Create a GlusterFS Volume Create a replicated volume that includes all three nodes.

# Replace vm1, vm2, and vm3 with the hostnames or IP addresses of the VMs
sudo gluster volume create git-volume replica 3 transport tcp vm1:/data/glusterfs vm2:/data/glusterfs vm3:/data/glusterfs
sudo gluster volume start git-volume

Mount the GlusterFS Volume: Mount the GlusterFS volume on each VM to make the Git repositories available

sudo mkdir -p /mnt/git
sudo mount -t glusterfs vm1:/git-volume /mnt/git

Set Up Git on GlusterFS

Create Git Repositories: Initialize Git repositories on the mounted GlusterFS volume

sudo mkdir -p /mnt/git/myrepo.git
cd /mnt/git/myrepo.git
sudo git init --bare

Interface for Git — Gitea

Next, I had to decide what interface to go for Git in my environment. I started with GitLab but decided against that in the first try. There were couple of reasons for this decision:

• Initially, I planned to install these components on same servers as my Git servers and wanted something simple to begin with
• I do have plans to setup GitLab later but on separate servers as those require other components like Redis, Postgres etc. and I did not want to put too much on my Git servers

So, I decided to go with Gitea.

 # download required version
 wget -O gitea https://dl.gitea.com/gitea/1.22.2/gitea-1.22.2-linux-amd64

 # make it executable and move to path
 chmod +x gitea
 sudo mv gitea /usr/local/bin/

 # create required directories
 sudo mkdir -p /mnt/gitea-data/gitea/{custom,data,log}
 sudo chown -R git:git /mnt/gitea-data/gitea
 sudo chmod -R 750 /mnt/gitea-data/gitea

Edit config file: vi /mnt/gitea-data/gitea/custom/conf/app.ini

[database]
DB_TYPE  = sqlite3
PATH     = /mnt/gitea-data/gitea/data/gitea.db

[repository]
ROOT = /mnt/gitea-data/gitea-repositories

[server]
DOMAIN       = <your-domain-or-ip>
HTTP_PORT    = 3000
ROOT_URL     = http://<your-domain-or-ip>:3000/
LFS_CONTENT_PATH = /mnt/gitea-data/lfs
SSH_PORT     = 22  # Change if necessary to avoid conflicts

Edit service file: sudo vi /etc/systemd/system/gitea.service

[Unit]
Description=Gitea
After=syslog.target
After=network.target
After=mysql.service
After=postgresql.service
After=memcached.service
After=redis.service

[Service]
RestartSec=2s
Type=simple
User=git
Group=git
WorkingDirectory=/var/lib/gitea/
ExecStart=/usr/local/bin/gitea web
Restart=always
Environment=USER=git HOME=/home/git GITEA_WORK_DIR=/var/lib/gitea/

[Install]
WantedBy=multi-user.target

sudo systemctl enable gitea
sudo systemctl start gitea

Access Gitea in the Browser

Open your browser and navigate to http://<your-server-ip>:3000. You should see the Gitea setup page.

Note: In my setup, I have load balanced it through NGINX and also setup certificates issues by my local HashiCorp Vault. Hence, in my case I do not have to go through port number in browser.

Complete the setup by following the instructions on the web interface:

• Set your database type and connection details.
• Set the server’s domain and application settings.
• Create an admin account.

This will set up a Git web interface on your server, accessible from a browser. If you have more specific needs or want to use a different tool like GitLab or Gitweb, the setup steps will vary slightly but will follow a similar pattern of installation, configuration, and service setup.

Should see page like below after setup:

Create some repository and navigating to that repository should give us interface that we are mostly used to from GitHub:

Summary

Setting up local Git cluster with distributed file system is pretty simple. Then we can add Gitea or anything we prefer to work with that. This is good alternative if we need local source control system and do not have access to GiHub Enterprise.

Even though data is replicated on three nodes, we can further protect by taking offline backups to some NAS using something like rsync.

Thanks!