Make Sure Your CLUSTER Passes MUSTER.

In today's information technology environment, High Availability Systems (HAS) are becoming more and more important. The demands on these systems are not only continuous availability of important data (365 x 24), but also efficient resource sharing of the components of the HAS. The buzzword behind all this is "clustering." A Server Cluster, simply put, extends RAID technology into the server arena by creating a system in which one server is acting redundant to another and, therefore, increases the security and computing power of the whole system. The end user is oblivious to the Cluster, knowing only that there is uninterrupted access to data. Special Clustering software manages and controls the Cluster and its members. In a Microsoft Windows NT environment, Microsoft Cluster Server (MSCS) is fairly simple to install and set up. The hardware involved, however, can get a bit trickier.

As a hardware support engineer for a major SCSI RAID controller manufacturer, it has become obvious that many so called technicians and system administrators are lacking in the basic knowledge of setting up a SCSI bus, let alone the Clustering concept. It makes me wonder why so much emphasis is placed on software for NT certifications and not a little more on the hardware, but this is a subject for another article, perhaps. Right now, let's make sure that if you are considering setting up an NT Cluster, you muster the necessary hardware and set it up correctly.

MSCS: What Is It?

First, let's look at Microsoft Clustering. What is it and what can it do for you? Or, perhaps more important, what can't it do? In a basic Server Cluster, two servers would share one common shared SCSI bus, usually attached to some sort of disk array. A separate, local disk is required for the installation of NT Server on each. Each server is also attached, not only to the network, but directly to each other, as well, utilizing a second NIC for a monitoring or heartbeat connection between the two. When one server goes down, the second becomes aware of this and takes over the duties of the failed server while still maintaining its own agenda.

It is important to note that the shared SCSI bus and accompanying array cannot be completely shared. Sure, the SCSI bus is attached between the two servers and both servers can share an array of disks attached to the bus, but that is as far as it goes. Any volume or disk space created by one server in a MSCS Cluster on the array can be seen by only one node at a time. Thus, Microsoft's term for this is "shared nothing."

The idea, then, is that two Microsoft Cluster Servers can each run a separate application on the shared bus, e.g. an SQL database from one Node and a Web Sever from the other. If, in this case, the machine running the Web Server fails, the server running the SQL database picks up the slack and also becomes the Web Server by taking over the duties of the failed server node. It is not designed for two machines to access the same data or application(s) at the same time. This seems to be a common misconception regarding Microsoft Cluster Server.

SCSI Basics In MSCS

It is very important, not just in a Cluster environment, but any SCSI environment, to note that a SCSI bus is not just a data bus. There is a difference in types of signal transmission on the SCSI bus that determines cable length and number of devices allowed. Single Ended (SE) devices, usually associated these days with the moniker "Wide/ Ultra," utilize a single wire on which logical information is represented in terms of voltage levels. Small distortions can change the voltage level significantly, which results in transmission errors, which results in data errors. Therefore, the Wide/Ultra SE SCSI bus is limited to just 1.5 meters and, theoretically, 15 devices. I say "theoretically" because, to get this many devices on the bus, you would have to slow the signal down from 40MB/sec to just "Fast SCSI" mode at 20MB/sec. Realistically, you are looking at only four devices at the longer cable length. This is certainly not adequate for Clustering.

Low Voltage Differential (LVD), known also as Wide/Ultra2, or just Ultra2, on the other hand uses two wires with separate voltage levels per signal. Only the difference between the wires represents the logical information. It is automatically deleted during calculation of the difference because distortion has the same effect on both wires. In this way, much longer cable length and higher transfer rates are possible. In fact, you should be able to achieve 15 devices and a cable length of 12 meters with a synchronous data transfer rate of 80MB/sec. You should, however, try to keep cable length as short as possible to cut down any possible signal degradation, but, this is definitely the way to go if Server Clustering is in your future.

The SCSI ID factor is also something to consider. Every device on the SCSI chain must have a unique SCSI ID, even the controller. A SCSI RALD/ Cluster controller will almost always, by default, be set to SCSI ID 7. This leaves the remaining 15 IDs for devices. In a Cluster setup, you will have two controllers, so one must be changed to avoid an ID conflict.

The Terminator

The SCSI bus also has a definite beginning and a definite end. This beginning and end is determined by physical termination at both. This doesn't mean that termination can be just placed somewhere on the bus after the last device, but must be at the physical beginning and end of the cable.

On a normal SCSI bus, the SCSI controller usually supplies the termination on one end and a Single Ended, LVD, or multimode terminator on the other. In a Cluster environment, however, termination cannot be left up to a SCSI controller. Since the object of the Cluster is for one node to take over the duties of the other in the case of failure, if there is a server failure, you potentially have no power to the controller and, therefore, no termination on one end of the bus. Now, you have more trouble than just a failed server; you have possible data corruption on the SCSI bus. To avoid this situation, a SCSI Cluster controller should have an internal channel with a terminator attached, which corresponds to the external connector. In other words, the controller is now just another device on the SCSI cable. When a server fails and there is no power to the controller, there is still termination on the SCSI bus.

But not just any termination will do. As stated before, there is a difference in SCSI bus transmission types, SE, or LVD. You can certainly place a SE terminator on your LVD bus, but by doing so, the entire bus now reverts back to SE mode. In fact, any SE device, whether it is a terminator, hard drive, tape backup, etc., will slow the bus to SE mode. This means that all the rules of the SE SCSI bus apply in relation to number of devices, cable length, etc. It is best, therefore, to stick with good quality LVD or multimode termination and place your legacy tape or CD devices on another channel (if using a multi channel controller) or another host adapter, entirely.

There is even an external UHD SCSI cable available with built-in active multimode termination at one end. This doesn't help during a server failure, but does allow you to disconnect the external SCSI cable from a failed Cluster node without the risk of losing termination or shutting down the remaining server in your Cluster. This is a great option to consider when designing your Cluster.

Termination also requires power. On the typical SCSI setup, the controller normally would handle this. However, in the case of Clustering, term power should be turned over to the disks or, better yet, a disk enclosure. Again, this is important in the event of a failed Cluster node that has lost power.

The Beat Goes On...

Setting up the network portion of MSCS is somewhat simpler for most system administrators since this is what they do most. Two network cards should be used, although there are NICs with multiple connectors. The problem with using a multiple port NIC in the Cluster environment is a loss of redundancy. One of the main points of the Cluster (or your entire network, for that matter) is to try to avoid any single point of failure and maximize up time.

The simplest way to set up the network portion of MSCS is to establish the public connection to your existing Domain during setup of NT Server. By establishing this connection first, you are assured the end users have access and you are not dealing with any potential resource conflicts of two network cards during the installation process. The second NIC can be installed after the fact so that if there are any problems, you can always revert back to that famous "Last known good configuration." Once both Network cards are up in both nodes, the second "private" or heartbeat connection is established using a crossover cable and an internal TCP/IP address. A crossover cable is recommended to keep this a pure private connection and should not support any other network clients. There are several internal IP network addresses

available by agreement with the Internet Assigned Numbers Authority (IANA), which can be used on the private interconnect. They include:

* 10.0.0.0 through 10.255.255.255 (Class A)

* 172.16.0.0 through 172.31.255.255 (Class B)

* 192.168.0.0 through 192.168.255.255 (Class C)

During the install of the MSCS itself, you will need to be able to differentiate between the two networks and their purpose (public, private, or both). The Cluster itself is also assigned an IP address. This requires a valid network IP address on the domain, so make sure there is a free address available for this before starting. It is important to use the LAN subnet mask, not the private interconnect's subnet mask for the Cluster, as well. When it's all said and done, the configuration should look something like the one in the Fig.

Final Assessment

It's an understatement to say that planning is everything, but, well, planning is everything, especially when it comes to high availability networks and Microsoft Clustering. Some of the basic SCSI concepts need to be reconsidered in a Cluster environment in regards to cabling, termination, disk arrays, etc. It is also very important to contemplate the shared storage itself. How will it be distributed between the two Cluster nodes, for example? How much disk space do you need for what applications on which server? It's always good to have a good clear picture, not only in your mind, but on paper, as well. Also make sure you have planned your network connections and protocols to allow for adequate NICs and IP addresses.

Of course last, but certainly not least, make sure you have mustered all of your components together and made sure they are all operational before beginning. There is nothing like a last minute scramble for that one little part at 11:00 o'clock the night before you are to bring your system online.

Frank Poole is the senior technical support engineer of ICP Vortex Corp. (Phoenix, AZ).