Prerequisites

The three virtual machines and the Load Balancer provisioned:

  • Node 1: 10.192.245.1 (Hostname: pnb-rabbitmq-oepay-1) [RHEL 9] CPU: 24 Memory: 48Gi Storage: 1Tb (/appsmcon)

  • Node 2: 10.192.245.2 (Hostname: pnb-rabbitmq-oepay-2) [RHEL 9] CPU: 24 Memory: 48Gi Storage: 1Tb (/appsmcon)

  • Node 3: 10.192.245.3 (Hostname: pnb-rabbitmq-oepay-3) [RHEL 9] CPU: 24 Memory: 48Gi Storage: 1Tb (/appsmcon)

  • Load Balancer: 10.192.245.45 (Ports: 5672, 15672, 15692)

Step 1: Set Hostname

Run the following command on each node to set its hostname:

hostnamectl set-hostname <hostname>
exec bash

Replace <hostname> with the respective hostname for each VM.

Step 2: Configure System Limits

Edit /etc/security/limits.conf and add the following lines:

* soft nproc 1000000
* hard nproc 1000000
* soft nofile 1000000
* hard nofile 1000000

Edit /etc/sysctl.conf and add:

fs.file-max = 1000000

Apply the changes:

sysctl -p

Logout and login to the system and check the limits:

ulimit -a

Step 3: Install Erlang and RabbitMQ

Navigate to the directory containing the RPM packages:

cd /home/oepay/rpm-rabbitmq

Install the required packages:

rpm -ivh erlang-27.3-1.el9.x86_64.rpm rabbitmq-server-4.0.7-1.el8.noarch.rpm

Step 4: Modify RabbitMQ Service Configuration

Edit the systemd service file /usr/lib/systemd/system/rabbitmq-server.service and add:

LimitNOFILE=1000000

Reload systemd and enable RabbitMQ:

systemctl daemon-reload
systemctl enable --now rabbitmq-server.service
systemctl restart rabbitmq-server.service
systemctl status rabbitmq-server.service

Step 5: Enable Required RabbitMQ Plugins

Run the following command to enable the relevant plugins:

rabbitmq-plugins enable rabbitmq_consistent_hash_exchange rabbitmq_management rabbitmq_prometheus rabbitmq_shovel rabbitmq_shovel_management rabbitmq_top

Step 6: Configure Directories and Permissions

Change ownership for RabbitMQ mount point directory:

chown -R rabbitmq:rabbitmq /appsmcon

Modify /etc/rabbitmq/rabbitmq-env.conf to change the base directory of RabbitMQ files and increase file descriptor limits:

RABBITMQ_MNESIA_BASE=/appsmcon/rabbitmq
RABBITMQ_LOG_BASE=/appsmcon/rabbitmq/log
ERL_MAX_PORTS=1000000

Modify /etc/rabbitmq/rabbitmq.conf:

loopback_users = none
vm_memory_high_watermark.relative = 0.8

Step 7: Update Hosts File

Edit /etc/hosts on all nodes:

10.192.245.1 pnb-rabbitmq-oepay-1
10.192.245.2 pnb-rabbitmq-oepay-2
10.192.245.3 pnb-rabbitmq-oepay-3

Stop RabbitMQ service:

systemctl stop rabbitmq-server.service

Set the Erlang cookie on all nodes:

echo -n "KRAWACXQABWGBORYWBUO" > /appsmcon/rabbitmq/.erlang.cookie
echo -n "KRAWACXQABWGBORYWBUO" > /var/lib/rabbitmq/.erlang.cookie

Note: Make sure the cookie is correct by checking the file’s contents.

Ensure correct permissions:

chown rabbitmq:rabbitmq /appsmcon/rabbitmq/.erlang.cookie /var/lib/rabbitmq/.erlang.cookie
chmod 400 /appsmcon/rabbitmq/.erlang.cookie /var/lib/rabbitmq/.erlang.cookie

Restart the RabbitMQ service:

systemctl start rabbitmq-server.service

Step 9: Cluster Configuration

On Node 2 and Node 3, run:

rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl join_cluster rabbit@pnb-rabbitmq-oepay-1
rabbitmqctl start_app

Modify /etc/rabbitmq/rabbitmq.conf to include:

queue_leader_locator = min-master

Restart RabbitMQ service on all nodes:

systemctl restart rabbitmq-server.service
systemctl status rabbitmq-server.service

Step 10: Verify Cluster

Run the following on any node:

rabbitmqctl cluster_status

You should see all three nodes listed in the cluster.

Here, a 3 node RabbitMQ cluster has been setup on baremetal virtual machines with a Load Balancer configured.