In this tutorial series, we are setting up a highly available WordPress web site from scratch.
Part 1 – Introduction, Considerations, and Architecture
Part 2 – Setting Up the VPSes
Part 3 – Setting Up MariaDB Multi-Master Replication
Part 4 – File Replication and Setting Up DRBD
Part 5 – Setting Up OCFS2
Part 6 – Round-Robin DNS, Let’s Encrypt, & Conclusion
File Replication Options
There are a galaxy of file replication technologies. We’re going to use DRBD but let’s first cover why we didn’t choose some others. Our requirements are that the replication is both bi-directional and real-time:
rsync: not bi-directional, not real-time. If you are the only author and the only person who will be uploading media, etc. you might just run rsync after each update (or set it up in a cron job). This is not a bad option but you’ll need to handle the certbot verification a different way. I recommend either using DNS validation or reading this article on certbot in RRDNS configurations.
lsyncd: real-time, but not bi-directional. Only supports one-way syncing.
unison: bi-directional but not not real-time. You could run unison in a cron job every few minutes on both nodes to keep your nodes in sync if you can live with the lag.
glusterfs: – bi-directional (multi-directional, actually) and real-time, but requires a minimum of three nodes.
mirror: doesn’t preserve owner, group, etc.
In this tutorial, we’ll use DRBD which is both bi-directional and real-time. Because we want each node to be capable of writing/changing files, we can’t use ext4 or any other “normal” filesystem but instead need a filesystem that supports shared-disk operations. For our purposes, we’ll use OFCS2 (Oracle Cluster Filesystem 2)
Preparing for DRBD
To use DRBD, we need to have a raw, unformatted partition on each side available. DRBD works a the block level, not the filesystem level. In Linode’s manager, I’ve added a new 10GB block storage disk to each node. If you’re using KVM, you will need to set aside an unused partition when you setup the node.
Start by installing needed packages on both nodes:
apt-get -y install drbd-utils ocfs2-toolsLoad the DRBD kernel module on both nodes:
root@web1:~/mysql# modprobe drbd root@web1:~/mysql# lsmod | grep drbd drbd 421888 0 lru_cache 16384 1 drbd libcrc32c 16384 1 drbd
I checked dmesg to discover that the block storage I’d added was mapped to /dev/sdc:
[17461.350617] scsi 1:0:0:0: Direct-Access Linode Volume 2.5+ PQ: 0 ANSI: 5 [17461.353223] sd 1:0:0:0: Power-on or device reset occurred [17461.355751] sd 1:0:0:0: [sdc] 20971520 512-byte logical blocks: (10.7 GB/10.0 GiB) [17461.357007] sd 1:0:0:0: [sdc] Write Protect is off [17461.357771] sd 1:0:0:0: [sdc] Mode Sense: 63 00 00 08 [17461.357912] sd 1:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [17461.360936] sd 1:0:0:0: Attached scsi generic sg2 type 0 [17461.365013] sd 1:0:0:0: [sdc] Attached SCSI disk
On both nodes, I partitioned /dev/sdc to add a single primary partition that took up the whole disk:
root@web1:/etc# fdisk /dev/sdc Welcome to fdisk (util-linux 2.33.1). Changes will remain in memory only, until you decide to write them. Be careful before using the write command. Device does not contain a recognized partition table. Created a new DOS disklabel with disk identifier 0x477ee7d7. Command (m for help): p Disk /dev/sdc: 10 GiB, 10737418240 bytes, 20971520 sectors Disk model: Volume Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x477ee7d7 Command (m for help): n Partition type p primary (0 primary, 0 extended, 4 free) e extended (container for logical partitions) Select (default p): p Partition number (1-4, default 1): 1 First sector (2048-20971519, default 2048): <return> Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-20971519, default 20971519): Created a new partition 1 of type 'Linux' and of size 10 GiB. Command (m for help): w The partition table has been altered. Calling ioctl() to re-read partition table. Syncing disks.
The partition table now looks like this:
root@web1:/etc# fdisk /dev/sdc Welcome to fdisk (util-linux 2.33.1). Changes will remain in memory only, until you decide to write them. Be careful before using the write command. Command (m for help): p Disk /dev/sdc: 10 GiB, 10737418240 bytes, 20971520 sectors Disk model: Volume Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x477ee7d7 Device Boot Start End Sectors Size Id Type /dev/sdc1 2048 20971519 20969472 10G 83 Linux
Setting up DRBD
Now let’s setup some configuration files. On each node, rename /etc/drbd.d/global_common.conf to /etc/drbd.d/global_common.conf.dist and replace it with this:
global { usage-count yes; } common { net { protocol C; } }
Protocol C is synchronous replication. This is the best choice in our scenario because (a) the servers are in the same datacenter so there is reduced latency, and (b) WordPress is a “few writers, many readers” kind of site, where most operations are read rather than write.
Next, create /etc/drbd.d/r0.res and populate it as follows:
resource r0 { on web1.lowend.party { address 1.1.1.1:7788; device /dev/drbd0; meta-disk internal; disk /dev/sdc1; } on web2.lowend.party { address 2.2.2.2:7788; device /dev/drbd0; meta-disk internal; disk /dev/sdc1; } # startup { # become-primary-on both; # } net { cram-hmac-alg sha1; shared-secret "secret"; # allow-two-primaries yes; after-sb-0pri discard-zero-changes; after-sb-1pri discard-secondary; after-sb-2pri disconnect; } }
A couple important notes:
- The “on” clauses must match the output of the ‘hostname’ command.
- You’ll notice both the “startup” clause and the “allow-two-primaries” part of the “net” clause are commented out. This is intentional. We will enable these directives once we have DRBD initialized.
Now on each node, execute these commands:
drbdadm create-md r0 drbdadm up r0
You can now look at /proc/drbd to see the status, or run drbdadm:
root@web1:/etc/drbd.d# cat /proc/drbd version: 8.4.10 (api:1/proto:86-101) srcversion: 983FCB77F30137D4E127B83 0: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r----- ns:0 nr:0 dw:0 dr:0 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:10484380 root@web1:/etc/drbd.d# drbdadm status r0 r0 role:Secondary disk:Inconsistent peer role:Secondary replication:Established peer-disk:Inconsistent
This output tells use that this volume is in an inconsistent state and both sides of the cluster are secondaries. In essence, both sides are waiting for the primary to update, but neither is primary so nothing is happening. Let’s fix that. On web1, execute this command:
drbdadm primary --force r0Now if you look at /proc/drbd you’ll see that synchronization is beginning:
root@web1:/etc/drbd.d# cat /proc/drbd version: 8.4.10 (api:1/proto:86-101) srcversion: 983FCB77F30137D4E127B83 0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r----- ns:1732 nr:0 dw:0 dr:1732 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:10482648 [>....................] sync'ed: 0.1% (10236/10236)M finish: 1:27:21 speed: 1,732 (1,732) K/sec
Don’t freak out by the estimate 1 hour, 27 minute sync time for a 10G volume. I checked /proc/drbd about 10 seconds later and it looked like this:
root@web1:/etc/drbd.d# cat /proc/drbd version: 8.4.10 (api:1/proto:86-101) srcversion: 983FCB77F30137D4E127B83 0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r----- ns:596272 nr:0 dw:0 dr:596272 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:9888108 [>...................] sync'ed: 5.8% (9656/10236)M finish: 0:07:16 speed: 22,640 (14,904) K/sec
Once the volumes are in sync, you’ll see this:
root@web1:/etc/drbd.d# cat /proc/drbd version: 8.4.10 (api:1/proto:86-101) srcversion: 983FCB77F30137D4E127B83 0: cs:Connected ro:Primary/Primary ds:UpToDate/UpToDate C r----- ns:134792 nr:0 dw:134792 dr:1469 al:120 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Now change /etc/drbd.d/r0.res on both nodes and uncomment out the commented portions. You can do this manually or by running the sed command below (be sure to do this on both nodes):
# systemctl stop drbd # sed -i 's/#//' /etc/drbd.d/r0.res # systemctl start drbd # systemctl enable drbd drbd.service is not a native service, redirecting to systemd-sysv-install. Executing: /lib/systemd/systemd-sysv-install enable drbd
DRBD should now report that it’s both up-to-date (consistent on all nodes) and that all nodes are primaries:
root@web1:/etc# cat /proc/drbd version: 8.4.10 (api:1/proto:86-101) srcversion: 983FCB77F30137D4E127B83 0: cs:Connected ro:Primary/Primary ds:UpToDate/UpToDate C r----- ns:609047 nr:7596 dw:15911 dr:623182 al:4 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0 root@web1:/etc# drbdadm status all r0 role:Primary disk:UpToDate peer role:Primary replication:Established peer-disk:UpToDate
Our DRBD configuration is now complete.
Next Part: Part 5 – Setting Up OCFS2
Related Posts:
I also enjoy using LEBs! I have an empire of little guys for VPNs, larger guys for apps, and big guys for databases and storage. When I'm not in front a screen I'm into German Shepherd dogs, quality knives, target shooting, theology, tabletop roleplaying games, and forest hiking.
I enjoy writing technical articles here on LowEndBox to help people get more out of their systems. You can find me on LowEndTalk @raindog308
- One Week From Tomorrow…THE WORLD WILL LOSE THEIR MINDS!Lines Are Already Forming! - November 21, 2024
- Crunchbits Discontinuing Popular Annual Plans – The Community Mourns! - November 20, 2024
- RackNerd’s Black Friday 2024: Bigger, Better, and Now in Dublin! - November 19, 2024
Very interesting article. Thank you.