« Nuevolution raises $15 million | Main | Farewell, Rick Blake »
October 10, 2003
OpenGFS failover working
WARNING! I think there are some factual errors in this, so be careful - I need to check tomorrow, but the crucial point is still in making the lock server a heartbeat managed cluster resource.
I'm assuming you have two nodes (mine are called fred and barney) in the cluster, both on the LAN and also preferably with a serial connection between them (for the backup heartbeat channel). I'm also assuming you want to use Samba here, if you have a heterogeneous network.
1. The OpenGFS config file should look something like this:
# the pool device that will contain the data(the larger of the 2) datadev: /dev/sda3 # the pool device that will contain the cluster info(the small one) cidev: /dev/sda1 # the IP:Port of the server that is going to be running memexpd lockdev: 192.168.1.100:15697 # the port used for callbacks between cluster nodes(no other programs on your # network should use this port, and it should be different for each OpenGFS # setup on your network) cbport: 3001 # time before a node is considered dead(lower number may mean faster recovery, # but may lead to unnecessary STOMITHing in the case of slow nodes) timeout: 30 # method used for STOMITHing "dead" nodes # Manual - human intervention STOMITH: manual # unique name for this method name: manual # Node Section # Tag: always "node:" # IPAddr IP address of the node # CID sequential numbers starting at 0 # STOMITH Method "SM:" followed by the name from the particular STOMITH # Method you are going to use # Plug/Port if you are using a power switch this should be the plug # that the node is plugged into # if you are using FC hardware this should be the port # that the node is plugged into # A simple example used for testing #Tag IPAddr CID STOMITH Method Plug/Port node: 192.168.1.1 0 SM: manual node: 192.168.1.2 1 SM: manual
Note that there are three IP addresses - one for the lockdev (also know as the "lock server"), which will also be the cluster IP address in this case, and one for each of the nodes. "What?". Well, the lock server IP address will be managed by the heartbeat mechanism - see below.
NB. Make sure the cluster IP address has a DNS entry (eg fscluster).
2. Create the OpenGFS drive, lock partition and external journal as described in the NoPool_HOWTO_build_and_install_OpenGFS document, using the above ogfscf.cf file and the sample journal file from the document. Use /ogfs as a mountpoint.
3. Install the latest heartbeat code from linux-ha.org on both nodes and test the serial connections as described in the heartbeat documentation.
4. Configure ha.cf for serial and net heartbeat (I won't do it here - see the heartbeat documentation for info).
5. Copy the ha_memexpd script from the OpenGFS source directory (under scripts, I think) to /etc/ha.d/resource.d/memexpd
6. Configure Samba with /ogfs as a public, writeable share (eg "ogfs") on both servers. Make sure smbd and nmbd aren't running yet.
7. Configure haresources. Pretty much the only thing you need to have in this file is a single line which looks something like
fred 192.168.1.100 memexpd \ Filesystem::/dev/sda3::/ogfs::ogfs::hostdata=[node IP address] smb
(The node IP address will be 192.168.1.1 on fred, and 192.168.1.2 on barney. This actually goes against the advice in the heartbeat documentation, which says that haresources should be the same on both nodes, but OpenGFS needs the IP address when mounting, in order to manage locks. There are, I'm sure, more elegant ways of passing the IP address.)
What this says is that the preferred node is fred, the cluster IP address is 192.168.1.100 and that three other resources should be managed - the locking daemon (memexpd), the mount for the ogfs device (Filesystem::...) and Samba (smb). These are actually scripts (and parameters separated by "::", in the case of Filesystem). That's why we copied the ha_memexpd script to memexpd in resource.d, because that is how heartbeat stops and starts resources, and resource.d is one of the place that heartbeat looks for scripts.
8. Make sure that the ogfs device isn't mounted on either node, and that the memexpd daemon also isn't running on either node.
9. You can now start the heartbeat service on both nodes (and make sure it's checked to start at bootup, and that you save the service configuration file!)
10. From a Windows client, you should browse for fscluster in the network neighbourhood. With luck, you should see the ogfs share and even be able to map it to a network drive.
11. Now shutdown the fred node. Within 30 seconds, barney should spot that fred is dead and should take over the lock server, mount and samba share! You will see a slight glitch in your Windows client, but after a few seconds, the share comes back.
Still a few things to iron out, but that's about it so far, folks.
WARNING! I think there are some factual errors in this, so be careful - I need to check tomorrow, but the crucial point is still in making the lock server a heartbeat managed cluster resource.
Posted by daen at October 10, 2003 12:08 AM