cat /dev/brain | grep technology >> blog
In Linux, it’s easy to log the output of a command to a timestamped log file:
echo this is easy >> $(date +%s).log
When I tried to accomplish the same thing in Windows, I ended up with this mess:
for /f "tokens=1,2,3,4 delims=/- " %%a in ("%date%") do set ts=%%d%%b%%c echo this sucks >> %ts%.log
I just got back from LinuxWorld in Boston, and I have to say that it wasn’t nearly as geeky as I expected it to be. Judging from pictures I’ve seen from previous LinuxWorlds, I expected it to be like a circus, but I think I saw more business guys in suits than overweight guys with beards. I guess it’s a testament to how mainstream Linux has become over the last few years.
Observations:
These are my notes on installing and configuring GFS from source RPMs. I haven’t worked with GFS in over a year, so I don’t know if this information is still accurate, but I’m posting it anyway in the hope that someone out there will find it useful. When following these instructions, your best bet is to run each command on each node before moving on to the next step (unless otherwise specified, or unless you know what you’re doing).
1.) Get the GFS and perl-Net-Telnet SRPMs from Redhat.
ftp://ftp.redhat.com/pub/redhat/linux/enterprise/3/en/RHGFS/i386/SRPMS/
ftp://ftp.redhat.com/pub/redhat/linux/updates/enterprise/3ES/en/RHGFS/SRPMS/
2.) Install the perl-Digest-HMAC and perl-Digest-SHA1 RPMs.
3.) Build and install the perl-Net-Telnet SRPM.
rpmbuild --rebuild perl-Net-Telnet-3.03-2.src.rpm rpm -Uvh /usr/src/redhat/RPMS/noarch/perl-Net-Telnet-3.03-2.noarch.rpm
4.) Each GFS node needs be running clock synchronization software to prevent unnecessary inode timestamp updates (which according to the manual, will impact performance severely), so you need to download and install the NTP RPM.
rpm -Uvh ntp-4.1.2-4.EL3.1.i386.rpm5.) Sync up your clock for the first time:
ntpdate 10.25.1.36
6.) Add the following lines to /etc/ntp.conf.
restrict pool.ntp.org mask 255.255.255.255 nomodify notrap noquery server pool.ntp.org
7.) Start ntpd.
/etc/init.d/ntpd start
8.) Verify that ntpd is syncing with your NTP server(s). When you do this, you need to make sure that your jitter values are in the lower single digits. They should definately not be 4000 (which means that NTP is not working at all).
ntpq -p9.) Make sure you have the kernel, kernel-smp, and kernel-source RPMs installed.
rpm -q kernel kernel-smp kernel-source10.) Install the GFS SRPM.
rpm -Uvh GFS-6.0.2-25.src.rpm11.) Check your current kernel version.
uname -a
12.) Open /usr/src/redhat/SPECS/gfs-build.spec and look for a line that starts with %define KERNEL_EXTRAVERSION. You may need to change this to match the “extraversion” of your kernel. You should also look for the line that says %define buildhugemem 1 and set it to 0 (unless you have a machine with >16gb memory with the hugemem kernel installed).
13.) Create a new SRPM with all the changes you made.
rpmbuild -bs /usr/src/redhat/SPECS/gfs-build.spec
14.) Build the GFS RPMs. Don’t forget to use the –target i686 option, or the SMP modules will not be installed.
rpmbuild --rebuild --target i686 /usr/src/redhat/SRPMS/GFS-6.0.2-25.src.rpm
15.) Install the GFS RPMs.
rpm -Uvh /usr/src/redhat/RPMS/i686/*6.0.2-25.i686.rpm
16.) Try manually loading the GFS modules into the kernel. If the modules are loaded succesfully, you should see them (along with all the other loaded kernel modules) in the output of lsmod.
depmod -a modprobe pool modprobe lock_gulm modprobe gfs lsmod
17.) At this point, the clustering software is installed, and simply needs to be configured. Now you need to create three config files (cluster.ccs, fence.ccs, and nodes.ccs) for the cluster configuration system (CCS). These files should be placed in a temporary directory by themselves on one node (I used /root/cluster). This is a fairly straightforward process, so I won’t repeat what chapter 6 of the Redhat GFS Administrators Guide already covers in detail.
18.) Once the CCS files have been created on one of the nodes, you should probably run a syntax check on them.
ccs_tool test /root/cluster
19.) Next, you need to create a “cluster configuration archive” (CCA) from the ccs files, and write it to a “cluster configuration archive device” (which is just a fancy name for a partition that all nodes have access to). A pool volume can be used for this, but I had the luxury of a 2.5TB iSCSI storage array, so just created a 2MB partition on that. Use the ccs_tool command to create the archive on the storage device of your choice. Note that ccs_tool writes these files in its own raw format, so there’s no need to format the partition. Also note that I was unable to create new CCS archives without having valid DNS records for all nodes.
ccs_tool create /root/cluster /dev/iscsi/bus0/target0/lun0/part1
20.) Tell Redhat’s init scripts where to find the ccs archive.
echo "CCS_ARCHIVE=\"/dev/iscsi/bus0/target0/lun0/part1\"" >/etc/sysconfig/gfs
21.) Now start the ccs daemons.
service ccsd start
22.) Start the lock_gulm server daemons.
service lock_gulmd start
23.) Create the GFS filesystems from one node.
gfs_mkfs -p lock_gulm -t Cluster1:gfs1 -j 8 /dev/iscsi/bus0/target0/lun0/part2
24.) Add your GFS filesystems to /etc/fstab with a fstype of gfs.
25.) Mount your GFS filesystems. Note that there is a known bug in some versions of GFS (related to mounting shared volumes) where node hostnames must be unique in the first 8 characters.
service gfs start
26.) Congratulations, you have a cluster! At this point, you should test moving files from individual nodes to the shared volume. All other nodes in the cluster should immediately be able to see these files. You might also try comparing the md5 sum of the file before it was moved to the md5 sum of the file after it was moved, just to make sure nothing weird is going on.
From node 1:
md5sum file.tgz cp file.tgz /mnt/volume
From node 2:
md5sum /mnt/volume/file.tgz
Thats it! For more info, RTFM! ;-)
Caveats
1.) I encountered a problem due to network latency in which iSCSI sessions were not consistantly being established before the GFS scripts tried to access the volumes. The ultimate solution was to first disable the init scripts…
chkconfig --level 0123456 iscsi off chkconfig --level 0123456 ccsd off chkconfig --level 0123456 lock_gulmd off chkconfig --level 0123456 gfs off
…then add the following to /etc/rc.local.
sleep 45 service iscsi start service ccsd start service lock_gulmd start service gfs start
2.) The fence_apc fencing method does not officially support the APC switch I was using, but I came up with a workaround (which can be found on the bug report I submitted to Redhat. This workaround was successful on the fence_apc script from version 6.0.0-1.2, but not with the one from 6.0.2-25. When upgrading, I needed to copy over the old fence_apc script (on the master lock server only):
cp /usr/src/redhat/SOURCES/gfs-build/bedrock/fence/agents/apc/fence_apc.pl /sbin/fence_apc
Once upon a time, I needed to grow an ext3 filesystem on a partitioned iSCSI volume. Unfortunately, I was unable to use GNU’s parted tool, because newer ext2/3 partitions weren’t supported yet, so here’s what I did to get around that limitation:
1.) First I unmounted the partition and converted it to ext2.
# umount /mnt/sdb1 # /sbin/tune2fs -O ^has_journal /dev/sdb1 tune2fs 1.32 (09-Nov-2002) # /sbin/e2fsck -y /dev/sdb1 e2fsck 1.32 (09-Nov-2002) /dev/sdb1: clean, 179076/10502144 files, 20365394/20974076 blocks
2.) Then I used fdisk to delete my old partiton and immediately recreate a larger one. The important thing to note here is that the new starting block was the same as the old one.
# fdisk /dev/sdb The number of cylinders for this disk is set to 153600. There is nothing wrong with that, but this is larger than 1024, and could in certain setups cause problems with: 1) software that runs at boot time (e.g., old versions of LILO) 2) booting and partitioning software from other OSs (e.g., DOS FDISK, OS/2 FDISK) Command (m for help): p Disk /dev/sdb: 161.0 GB, 161061273600 bytes 64 heads, 32 sectors/track, 153600 cylinders Units = cylinders of 2048 * 512 = 1048576 bytes Device Boot Start End Blocks Id System /dev/sdb1 1 81930 83896304 83 Linux Command (m for help): d Selected partition 1 Command (m for help): p Disk /dev/sdb: 161.0 GB, 161061273600 bytes 64 heads, 32 sectors/track, 153600 cylinders Units = cylinders of 2048 * 512 = 1048576 bytes Device Boot Start End Blocks Id System Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-153600, default 1): Using default value 1 Last cylinder or +size or +sizeM or +sizeK (1-153600, default 153600): Using default value 153600 Command (m for help): p Disk /dev/sdb: 161.0 GB, 161061273600 bytes 64 heads, 32 sectors/track, 153600 cylinders Units = cylinders of 2048 * 512 = 1048576 bytes Device Boot Start End Blocks Id System /dev/sdb1 1 153600 157286384 83 Linux Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks.
3.) Check the partition.
# e2fsck -f /dev/sdb1 e2fsck 1.32 (09-Nov-2002) Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information /dev/sdb1: 179076/10502144 files (17.8% non-contiguous), 20365394/20974076 blocks
4.) Now it’s possible to grow the filesystem.
# resize2fs /dev/sdb1 resize2fs 1.32 (09-Nov-2002) The filesystem on /dev/sdb1 is now 39321596 blocks long.
5.) The final step is to convert back to ext3 and remount the partition.
# /sbin/tune2fs -j /dev/sdb1 tune2fs 1.32 (09-Nov-2002) Creating journal inode: done This filesystem will be automatically checked every 39 mounts or 180 days, whichever comes first. Use tune2fs -c or -i to override. # mount -t ext3 /dev/sdb1 /mnt/iscsi
Thats it! But this process shows why it’s a good idea not to partition your iscsi volumes at all, if you ever intend on making them larger (ie: use /dev/sda instead of /dev/sda1,2,n). That way, you don’t have to mess with fdisk in order to grow the volume.