Engineering challenges to storage system protocols: diagnosing problems involving multiple protocols presents complex engineering challenges.
Storage system architectures have evolved considerably over the last ten years--from Direct Attached Storage (DAS) to Network Attached Storage (NAS (1) See network access server.
(2) (Network Attached Storage) A specialized file server that connects to the network. A NAS device contains a slimmed-down operating system and a file system and processes only I/O requests by supporting the popular ) and Storage Area Network (SAN) and together, NAS and SAN form the basis of Networked Storage Architecture (NSA NSA
National Security Agency
Noun 1. NSA - the United States cryptologic organization that coordinates and directs highly specialized activities to protect United States information systems and to produce foreign ). With this evolution, engineers face considerable challenges in designing, testing, installing and maintaining complex infrastructures, mixing and matching multiple storage protocols.
The first noticeable trend in the storage market is its rapid migration from DAS to SAN-attached or network storage. By many estimates, about 70% of the current market is DAS, with about 30% externally attached systems. However, in four years those numbers are expected to reverse in favor of external-attached systems. From a test and measurement perspective, this evolution will pose significant challenges at the system level.
Another trend in the storage industry is the move from parallel interfaces to serial interfaces. Emerging technologies like Serial ATA See SATA.
Serial ATA - Serial Advanced Technology Attachment (SATA (Serial ATA) A serial version of the ATA (IDE) interface, which has been the de facto standard hard disk interface for desktop PCs for more than two decades. The original Parallel ATA (PATA) interface was launched in 1986. ) and Serial Attached SCSI See SAS. (SAS (1) (SAS Institute Inc., Cary, NC, www.sas.com) A software company that specializes in data warehousing and decision support software based on the SAS System. Founded in 1976, SAS is one of the world's largest privately held software companies. See SAS System. ) are coming alongside established Fibre Channel Arbitrated Loop A ring topology used in Fibre Channel. Up to 127 devices may be attached in the loop, but only two can communicate at the same time, reflecting the channel nature of Fibre Channel technology. (FC-AL (Fibre Channel-Arbitrated Loop) See Fibre Channel.
FC-AL - Fibre Channel-Arbitrated Loop. ) for storage device level interface. Networked storage interconnects are currently dominated by Fibre Channel Switched Fabric; however, SCSI SCSI
in full Small Computer System Interface
Once common standard for connecting peripheral devices (disks, modems, printers, etc.) to small and medium-sized computers. SCSI has given way to faster standards, such as Firewire and USB. over TCP/IP TCP/IP
in full Transmission Control Protocol/Internet Protocol
Standard Internet communications protocols that allow digital computers to communicate over long distances. (iSCSI) is gaining momentum for low-cost solutions that do not require the high performance provided by FC fabrics.
FC is defined for 1, 2, 4 and 10Gbits/sec speed categories, and iSCSI uses the Ethernet physical layer The Ethernet physical layer is the physical layer component of the Ethernet standard.
The Ethernet physical layer evolved over a considerable time span and encompasses quite a few physical media interfaces and several magnitudes of speed. with 1 and 10Gbits/sec speed categories. It is expected that when 10-Gigabit Ethernet is widely used, iSCSI will be competing for a larger share of the SAN connectivity market against 10-Gbits/sec Fibre Channel.
Serial connections in storage architectures make sense for several reasons. First, serial connectivity is more cost effective, using less copper on the backplane. More importantly, parallel connections become far less efficient as traffic rates increase, making serial connections more scalable. At high speeds, assembling each byte correctly in reference to the clock becomes more difficult, particularly for parallel connections where the clock is running on a separate line. Although there are techniques currently available for getting more speed from parallel connections, the long-term answer is to switch to serial interfaces.
Finally, InfiniBand was expected to make a big splash Big Splash could refer to:
DAS Protocol Evolution
DAS has been in use for a long time as a cost-effective local storage architecture. DAS devices are usually configured in Redundant Array of Independent Disks (RAID) mode (either by hardware, in Host Bus Adaptors (HBA (Host Bus Adapter) See host adapter. ), or by software) for fault tolerance See fault tolerant.
(architecture) fault tolerance - 1. The ability of a system or component to continue normal operation despite the presence of hardware or software faults. This often involves some degree of redundancy.
2. and better performance. Currently, there are two primary protocols used in this architecture at the disk device level: Parallel ATA See PATA. (PATA (Parallel ATA) Refers to the original ATA (IDE) technology that uses a parallel data channel from the controller to the disk drives. After Serial ATA drives became popular, the PATA term was coined to specifically refer to the parallel drives. See IDE and SATA. ) and parallel SCSI--and both are rapidly being replaced. For lower cost implementations, PATA is being replaced by SATA, and parallel SCSI Parallel SCSI (formally, SCSI Parallel Interface, or SPI) is one of the interface implementations in the SCSI family. In addition to being a data bus, SPI is a parallel electrical bus: There is one set of electrical connections stretching from one end of the SCSI bus is giving way to SAS.
Next generation SATA will support 3Gbits/sec wire rates yielding 300MB/sec maximum raw throughput, with future generations planned beyond 3Gbits/sec. A point-to-point host-to-target connection, SATA, can connect multiple devices using port multipliers. Although SATA does not inherently support dual-ported disk interface, several proprietary solutions are emerging. SATA-based RAID storage systems have already started shipping.
The latest specifications for SCSI, referred to as Ultra 320 (U320) supports 320MB/sec of maximum raw data throughput. A peer-to-peer interface U320 can support 16 devices on a single bus. For fault tolerance and shared data configurations, dual-ported disk devices are available. SCSI is defined for high performance applications.
SAS is challenging traditional SCSI. Standards have been planned for higher bandwidth connections up to 6Gbits/sec for the future. Also, the physical layer of SAS is defined as compatible with SATA 3Gbits/sec physical interface.
Both SAS and SATA devices can share a connectivity network using fan-out and edge expanders. SATA commands are transmitted over SAS connections as a "tunneling" protocol called SATA Tunneling Protocol A network protocol that encapsulates packets at a peer level or below. It is used to transport multiple protocols over a common network as well as provide the vehicle for encrypted virtual private networks (VPNs). (STP STP or standard temperature and pressure, standard conditions for measurement of the properties of matter. The standard temperature is the freezing point of pure water, 0°C; or 273.15°K;. ). SAS devices and systems will be in volume production as early as next year.
Fibre Channel Loop protocol is used within a storage system for very high performance and large spindle configurations. These loops, consisting of up to 127 devices, can run at 1- or 2-Gbits/sec serial rates, providing 200MB/sec (full duplex (Computers) arranged so that the information may be transmitted in both directions simultaneously; - of communications channels between computers; contrasted with
SAN/NAS Protocol Evolution
The remote storage concept evolved first for disaster recovery applications with remote backup. The remote storage elements are connected over LAN/WAN LAN/WAN Local Area Network/Wide Area Network networks and perform file-level operations. SAN evolved to support distributed processing The first term used to describe the distribution of multiple computers throughout an organization in contrast to a centralized system. It started with the first minicomputers. Today, distributed processing is called "distributed computing." See also client/server. with shared storage. To share a storage element across different computing elements, connectivity to the storage elements must be available over a network that can move block data very quickly.
A switched architecture satisfies this application and, as a result, Fibre Channel (FC) was adopted. In almost all SAN implementations today, FC switched fabric is the most dominant protocol used.
FC fabric is a point-to-point switched protocol operating at 1 or 2 Gbits/sec serial rates. The switched architecture provides bandwidth multiplication for high performance storage networks. Fibre Channel switched fabric usually consists of several end-point devices (hosts and targets) and several layers of switches.
SATA, SAS and FC-AL are the protocols of choice for internal connectivity within a storage system, whereas external connectivity between hosts and storage systems are FC switched fabric or iSCSI (IP storage) connections. FC switched fabric is a point-to-point protocol See PPP and PPTP.
(communications, protocol) Point-to-Point Protocol - (PPP) The protocol defined in RFC 1661, the Internet standard for transmitting network layer datagrams (e.g. IP packets) over serial point-to-point links. , whereas iSCSI is a networked protocol using classical TCP/IP connectivity layer However, in the first wave of iSCSI adoption, it is likely to be used as a point-to-point connection to avoid many issues (e.g. security, reliability, bandwidth sharing, etc.) related to network connections.
The Storage Network Industry Association (SNIA (Storage Networking Industry Association, San Francisco, CA, www.snia.org) An organization devoted to the advancement of mission critical storage systems. Founded in 1997, its goal is to determine the standards that must be developed to allow hosts and storage systems to interact via ) is attempting to standardize the interlaces so that different components can interact without compatibility problems. But the important thing to realize is that even when the interfaces between different components are standardized, there are multiple protocols (FC, SATA, SAS, FC Loop) that may be mixed and matched throughout NAS and SAN architectures. This mix of protocols requires test and measurement tool manufacturers to provide engineers with products that will enable them to "see through" different protocols to efficiently manage storage systems and easily identify and correct potential problems.
Bus vs. Switched or Loop Protocols
As mentioned in the previous two sections, storage protocols of the future am moving away from "bus" architecture. Test instruments which can monitor traffic related to all the components in a bus will be able to monitor only the traffic related to the end points in a point-to-point connection.
To monitor interactions between multiple components in a switched fabric, multi-channel analysis is required. This is a paradigm shift A dramatic change in methodology or practice. It often refers to a major change in thinking and planning, which ultimately changes the way projects are implemented. For example, accessing applications and data from the Web instead of from local servers is a paradigm shift. See paradigm. in test instrumentation resulting in usage and pricing changes.
Protocol analyzers are a vital tool for engineers to diagnose problems fur all the major protocols from the storage devices to storage systems and applications. These tools are not a part of the network, but sit unobtrusively on the network to silently monitor the data--bit by bit.
At the lowest levels in all serial protocols, frames or packets carry higher-level protocol information. During analysis, all the serial bits must be captured, but only the relevant information should be presented to the engineer. For example, if the user suspects frame-level problems, only the low-level serial data should be presented as it flows through the wires. But if the user must analyze ATAPI-level commands (in SATA interface), the presentation layer should elevate the low-level packets to the ATAPI (AT Attachment Packet Interface) The specification for ATA (IDE) tape drives and CD-ROMs. See IDE.
ATAPI - AT Attachment Packet Interface command level and show a transaction at that level.
There are a few important features required in serial protocol analyzers that would enable engineers to be very productive.
In SAN environments, debug To correct a problem in hardware or software. Debugging software means locating the errors in the source code (the program logic). Debugging hardware means finding errors in the circuit design (logical circuits) or in the physical interconnections of the circuits. can be very complex, with layers of switches and several end-point devices. To isolate faults in a short time in such a network, it would be essential to monitor several point-to-point links at the same time and collect protocol data in a time-correlated manner between all the links.
However, it should be noted that multi-link (multi-channel) analyzer equipment can get prohibitively expensive unless a modular approach is chosen, such as a solution that is enabled by cascaded analyzers. Using the cascaded approach, independent analyzers used in different locations can be brought together for a large configuration debug--practically at no additional cost.
Serial protocols carry idle frame patterns when there is no useful information transferred between the connected devices. This is to maintain a clock reference as well as to indicate that the physical connection is alive and well. Whereas idle frames are essential for physically maintaining a connection, they are not important for higher-level protocol layers that are concerned with actual information exchange.
At high speeds of greater than 1Gbit/sec, the idle frames associated with these serial protocols, if captured in an analyzer, can fill up the capture or trace buffer very quickly--in fact, useful information in this capture may be very minimal. That makes it essential to filter out all unnecessary information relative to the protocol layer being debugged and store only essential information in trace buffer.
For example, if SCSI over Fibre Channel is the layer of interest, READ SCSI commands from a host to a particular disk may be the only data the engineer wants to view. All other data can be filtered out, leaving only these commands in the memory. With effective filtering, the available trace buffer can be used very efficiently.
This is very important if traffic from multiple channels is to be collected in the trace buffer. In a multiple channel scenario, one channel may be carrying more information than another channel. If fixed memory is allocated per channel, the memory used by the channel with less information transfer is not utilized properly. So, memory pooling between several channels can be implemented to increase trace buffer utilization.
Triggering is another feature that enables troubleshooters to narrow down a problem. Sometimes it is difficult to guess the right trigger condition, and engineers may have to do some amount of "trial and error" captures to narrow down to the problem area. If multiple analyzers can be hooked up to the same observation point, setting each for different trigger conditions will reduce the time of sequentially doing multiple runs. This is an expensive solution. However, if a single analyzer can allow for several independent and parallel trigger events to be set, it would greatly enhance the usage of an analyzer.
Setting trigger conditions is not always easy. In some instances, parallel sequences might help reduce the time to debug. In other complex situations, it may be impossible to set trigger conditions. Some problems may be very infrequent with long gaps in between. To debug these problems, the only solution might be to collect data for several hours or days at a stretch and look for problem areas for further drilldown analysis. In these situations, it is not cost effective to collect data in semiconductor memory. Spooling (Simultaneous Peripheral Operations OnLine) The overlapping of low-speed operations with normal processing. Spooling originated with mainframes in order to optimize slow operations such as reading cards and printing. trace data to disk as the data is being collected is one of the best ways to cost effectively implement a solution for catching intermittent problems.
Finally, the visual presentation itself must enable easy viewing of specific information. Presentations are graphical, textual, or a combination of both. The key is in making it easier for engineers to view exactly what they need to locate a problem. Dimming certain colors to highlight a particular field enables faster packet-by-packet viewing by only looking for Looking for
In the context of general equities, this describing a buy interest in which a dealer is asked to offer stock, often involving a capital commitment. Antithesis of in touch with. a particular color.
The purpose of the presentation layer is to use the monitor screen area to present maximum information for the layer in which the debug is being attempted. Engineers can start with the highest layer (for example in a FC fabric, this layer may be FCP (Fibre Channel Protocol) See Fibre Channel.
FCP - Flat Concurrent Prolog.
["Design and Implementation of Flat Concurrent Prolog", C. Mierowsky, TR CS84-21 Weizmann Inst, Dec 1984]. or SSCI-over-layer) to narrow down to a problem FCP exchange. The exchange can then be drilled down to expose what sequences were involved. If the problem is evident, the analysis stops at this layer. If not, the engineer can drill further down to frame level for each of the sequences for that exchange.
In conclusion, as networked storage becomes more complex, using multiple protocols in different areas of the network, engineers will face many challenges in designing, installing, and maintaining storage environments--from devices to very complex SAN and clustered networks. Protocol analyzers must adapt to this changing environment to provide scalable products and enhancements that enable engineers to keep storage architectures running efficiently-regardless of protocol choice.
Srikumar Chandran is senior director of storage business at CATC (Santa Clara, CA)