The distributed laboratory: an interactive visualization environment for electron microscopy and 3D imaging.
The SDMIR microscope is a unique resource. By building an application around it, we hope to achieve two ends. Our primary result will be to extend the capabilities of the microscope. In studying researchers using the microscope, we have found that electron microscopes are used as cameras to collect data photographically for subsequent analysis. Although the SDMIR microscope is equipped for digital image acquisition, the images are still analyzed off-line, away from the microscope. By providing tools for image compositing, comparison, and morphometrics, and by enhancing, via stereopsis and computed tomography, the microscopist's perception of three-dimensional structures during the data collection process at the microscope, we hope to demonstrate a synergy between interactive visualization techniques and a sophisticated scientific instrument. The result should be a more powerful investigative tool.
Our secondary goal is to extend access to the microscope. The software for the MWS is designed to be used either in the same room as the microscope or at a remote site, via a high-speed network. The software is also designed to be used either with the microscope on-line, or independent of the microscope, when the microscope is not available or not necessary for a particular task.
The United States Congress recently passed the High-Performance Computing and Communication Initiative, which will dedicate billions of dollars toward the development of supercomputing technology and the creation of a gigabit-per-second National Research and Education Network (NREN) by 1996. Such advances in network capabilities will facilitate higher-speed connections to unique facilities like SDMIR, and could lead to an extension of localized resources beyond geographical limitations. Providing network access to resources and custom software multiplies single resource, that are evolving into what we have been calling the Distributed Laboratory. While we are a long way from a completely remote-access SDMIR microscope, the Microscopist's Workstation prototype provides a glimpse of how high-speed networking can amalgamate distant sites into a highly interconnected research community.
The Distributed Laboratory
The MWS project is a collaborative effort among several research sites in San Diego: the San Diego Supercomputer Center, the Computer Science department and Medical School of the University of California, San Diego, and the Scripps Research Institute. Each institution is supplying hardware, networking, software development, and design expertise toward the project. One of the immediate spin-offs of this collaboration will be the establishment of higher-speed network connections between all of the sites.
The ongoing collaboration and discussions among the groups at different sites have lead us to begin to conceptualize a Distributed Laboratory that employs high-speed networks to integrate data acquisition, computational resources, visualization, and communication into a seamless, geographically distributed, investigative environment. As an example, consider the Internet. The Internet is not a single-governed entity but an amalgamation of networks, email routers, news distributors, and on-line databases that comprise an invaluable communication and information resource. The Distributed Laboratory would combine scientific facilities and connect researchers both within and across disciplines in the same way. The concept of a distribututed laboratory leads to many areas of possible investigation, three of which are addressed by the MWS project: integrating a scientific instrument into the computing environment, seamless management of supercomputing resources, and multiuser collaboration.
Constructing an environment that accesses an instrument as a computer peripheral allows the device to be used by a researcher at a remote site with nearly the same efficiency available to an on-site researcher, that is, distributing the facility spatially. If the investigative software also operates with the instrument on-line and off-line, either independent of the instrument or with the instrument available for occasional "batch" processing, the facility is also distributed temporally. The result is the multiplication of single resources.
To present the researcher with a unified investigative environment, the management of computing resources must be automatic. Our experience with NetV software , which transparently distributes volume-rendering tasks to highend resources, is a first step toward hiding the computing power behind the interface. In the MWS project, the workstation interface is the visible third of a triumvirate including the computer-controlled microscope and the supercomputing resources which are the remaining, hidden portions.
Focusing on bringing distributed data acquisition and computational resources to the workstation user interface allows the interface itself to be replicated among multiple researchers. This creates a collaborative research environment where two or more researchers can interact with one another and the instrument.
We consider the issues in developing a Distributed Laboratory approach to electron microscopy.
The SDMIR Electron Microscope
SDMIR is a new biology research facility established by the National Center for Research Resources of the National Institutes of Health, located in the Medical School at the University of California, San Diego. This laboratory was developed to support scientific research focused on relating biological function to structure, using state-of-the-art computer imaging and electron microscopy. The facilities of the laboratory are available to researchers throughout the U.S. The central instrument in the lab is a specially designed and equipped transmission electron microscope (TEM). A TEM resolves biological structures that are invisible when examined with a light microscope. It can magnify over 800,000 times and can resolve details in the 0.2 to 0.3 nanometer range. In contrast, a light microscope has a usable range of magnification up to about 1,000 times and a maximum resolution of approximately 200 nanometers. The resolution of a TEM is required, for example, to derive the "wiring diagram" of the nervous system. A light microscope provides a valuable panoramic view of the paths of nerve fibers from one area jungle of nerve fibers arriving from different sources in the vicinity of the input sites of a particular nerve cell. But the resolution of a TEM is required to establish which fibers actually form contacts or synapses on the neuron as well as the particular form of the contact. In addition to studies concerning these connections, scientists are currently using the electron microscope at SDMIR to examine the disruption of nerve cell components resulting from Alzheimer's disease [4, 13], the structural relations of protein molecules invoved in the release of calcium inside neurons , and the three-dimensional form of the Golgi apparatus, where sugars are added to proteins. Examples of such three-dimensional reconstructions are shown in Figure 1.
As shown in Figure 2, many elements of the TEM are analogous to those of the light microscope. Instead of light, the specimen, which is placed in a vacuum column, is irradiated with a beam of electrons to form an image. Electrons emanating from a filament are accelerated toward the specimen. The electron beam is shaped by an electromagnetic condenser lens and directed by deflection coils to uniformly irradiate the specimen. Electroncs penetrate the specimen and are magnified and focused by objective and projection electromagnetic lenses to form an image . The image may be viewed directly on a phosphorous-coated screen, captured on film or digitized using a video system, and displayed on a video monitor. The contrast changes comprising the image formed by the microscope reflect variations in the electron densities within the specimen material. To visualize a particular component, the specimen must be processed to make the component more or less dense than the background. Preparing the specimen so that a particular biological component is differentiated, while its structural integrity is preserved, has been and continues to be the subject of much research.
The SDMIR instrument is one of the few high voltage transmission electron microscopes (HVEM) in the U.S. (see Figure 3). This instrument, a JEOL 4000EX, can employ a higher acceleration voltage (up to 400KV) than a conventional TEM (100KV). The higher accelerating voltage makes possible penetration and imaging of thicker specimens, up to 3 microns compared to 0.1 to 0.25 microns in the conventional TEM.
One focus of research at SDMIR is on the development of computer-assisted methods for deriving and exploring the three-dimensional structure of these thick sections [3, 10]. Because the HVEM has a very large depth of field, the image it forms is an orthogonal projection of the section. The depth dimension is lost. It is, however, possible to regain this dimension by obtaining images from different angles of view either by tilting the electron bean relative to the specimen or by mechanically tilting the specimen. In the simplest use of this principle, a stereo pair of images may be derived. In a more sophisticated approach, equivalent to that used in computer-assisted tomography (CAT), a three-dimensional volume of densities is derived from a series of images collected by successively tilting the specimen in small increments about an axis. This axial tomography procedure requires image processing to achieve proper alignment of the tilt series, and computationally intensive operations to derive the volume from the tilt series via filtered back-projection . The SDMIR microscope has been specially designed to accomplish this task.
Special deflection coils make it possible to vary the beam entry into the material over sufficient angles to obtain pairs of images for stereo views. The objective lens was constructed with a larger gap into which the specimen is inserted, allowing rotation over the wider range of tilt angles required to perform axial tomography.
The HVEM is also well-equipped for automated control. It employs an A/D-D/A converter under microprocessor control to read, set, and display parameters such as acceleration voltage, magnification, focus, beam intensity, and astigmatic correction. These parameters can be manipulated with an array of knobs and buttons on the microscope. The parameters can also be set and read remotely with ASCII commands sent over an RS232 serial port. The specimen holder is mounted in a four-axis (X, Y, Z, and tilting) stage. The positions of the axes can be controlled manually and also by a dedicated IBM PC-AT compatible microcomputer driving four closed-loop motor-position encoders. The microcomputer program can be queried via a serial connection to set or retrieve the current stage position.
Microscope control and image acquisition are handled by a dedicated workstation called the Cap/Con (capture/control). The microscope and stage computer interact via RS232 lines with the Cap/Con, a Sun Microsystems SPARCstation 2 with several special purpose peripherals. Cap/Con also interfaces to an image processor that can acquire microscope images from both a real-time 512 x 512-8-bit video camera and a high-resolution 1,024 x 1,024-14-bit slow scan cooled CCD camera. Processed images may be viewed on a monitor at the microscope. In addition to providing views with better contrast than those seen by direct viewing at the microscope, the image processor may be used to display previously acquired images and stereo views. These capabilities make possible the design of the MWS.
The Microscopist's Workstation
We began our investigation into designing the MWS by talking to the scientists who constructed the SDMIR facility and used the microscope on a regular basis . We also videotaped a microscopic session with an expert user, Tom Deerinck, who is adept not only with the SDMIR HVEM, but also with several other electron microscopes. We then transcribed and analyzed the videotape. The existing user interface to the microscope, consisting of the controls and displays on the microscope console, is the result of nearly five decades of development within the electron microscopy industry, and already provides an excellent ergonomic environment for controlling the microscope. The user interviews and video analysis helped us design a visualization environment centered around users' needs when using the microscope as a research instrument, including tasks currently performed away from the microscope.
A microscope user begins by preparing a sample, usually by chemically staining the structures of interest, and depositing it on a grid, a 3mm diameter plate with a 2mm-diameter usable area. There are a wide assortment of grids available; some have a lattice of support wires and some are open throughout. One of the constraints is to be able to view a grid, in registration, from one microscopy session to the next. Our work, then, relies on grids that have registration marks at the center and at the edge. By calibrating to these marks, we can match the coordinate space of the software with that of the microscope across multiple sessions.
Much of the operation of the HVEM is already automated. The user has control over the X and Y stage position, the height and tilt of the stage, and parameters controlling the electron beam. The beam passes through the entire sample and is not focused on a particular plane, so the stage height is used only to control the axis of rotation for tilting the specimen. The primary positioning controls, then, are the X and Y stage controls. As previously mentioned, stage positions may be set manually or via the stage computer. The microscope computer display shows the coordinates in tenths of microns, in the range of [+ or -]1000 microns in X and Y.
There are three primary beam controls: magnification, which is measured in discrete steps from 100x to 800,000x, brightness, which is measured at the phosphor screen as current density, in units of picoamps per square centimeter, and the focus. In addition, astigmatic distortion can be corrected via stigmator coils and the beam can be shifted in X and Y as well as tilted up to five degrees. The beam controls operate in discrete steps so the user can return to a previous setting. This feature is important for an investigative tool and is incorporated into the design of the MWS software.
All the beam parameters must be adjusted to align and focus the image correctly, not only at the start of each session, but continually, as the microscope is operated. To operate the microscope remotely, a real-time video image of the microscope stage would need to be available to the remote microscopist, in addition to control over all of the beam parameters. Any perceivable lags or discontinuities in the video signal or in the transmission of the beam control information would make tuning the parameters extremely difficult. Since we cannot assume the availability of real-time video, the MWS software design is dependent on the presence of an operator at the microscope site, and includes support for interaction between the on-site operator and the remote researcher.
In studying users and the operation of the SDMIR laboratory, we found that electron microscopes are used mostly as cameras for collecting images either on film or in a digital format. For example, although the microscope's stage computer provides the ability to store several stage positions and return to them later, this capability may be used to record points of interest, but is rarely used to compare structures at different locations on the sample. Structure comparison can be better performed by taking photographs of each location and comparing them side-by-side, rather than by shuttling the stage back and forth. The MWS will extend the capabilities beyond those of a microscopic camera to allow image comparison and analysis while online with the microscope. Currently, tomography computation and display are performed away from the microscope. The workstation software will integrate the microscope's tomography capability into the existing microscopic environment.
Much of what is described in the Sidebar "A Look Forward" is incorporated into the design of the MWS prototype and will be demonstrated at the SIGGRAPH '92 Showcase. The major difference is that the operator on-site at SDMIR will have to be a highly trained electron microscopist, and will have to manually perform some of the tasks that will eventually become automated and incorporated into the Cap/Con software.
The scenario describes the remote researcher requesting a change in magnification by issuing a command that is sent to the Cap/Con workstation. At present, realigning and refocusing the electron beam after changing magnification requires a human's ability to navigate the multidimensional parameter space. The ideal focus is a subjective assessment; the researchers we have interviewed stated that the best images are often obtained by slightly under-focusing. There is no optimal focus condition applicable to every viewing situation, and no algorithmic means of finding an ideal configuration for all of the parameters for differing types of specimens. Automating focus adjustment is one of the challenges facing the electron microscopy field and is an active area of research. The MWS prototype will provide us with a platform on which to investigate approaches to solving this problem.
A human is also required in the computed tomography process, to center the structure of interest within the field of view and to adjust the height of the stage, which controls the axis of rotation for tilting the stage. The remainder of the tomography process is currently semiautomated; we are attempting to automate more of the process.
Two MWS's based on SPARCstations are being assembled. An FDDI ring connects the SDMIR MWS and Cap/Con to SDSC, where they connect to the NSFNet T3 backbone. The MWS's each have a ViCOM VX graphics accelerator board, a tightly coupled ViCOM high-speed bulk memory board (128- to 256MB), and an FDDI interface. The VX accelerator is based on an Intel i860 processor along with 4MB of Intel RAM and a 16MB framebuffer. The bulk memory board connects to the VX via ViCOM's VXC Bus, at 320MB/sec. The resulting configuration affords us a large amount of image storage and manipulation capability.
Our goal in the MWS project is to investigate the potential advantages afforded by tightly coupling a scientific instrument under computer control with interactive visualization software. We are taking a user-centered approach to designing the software, in order to produce an environment that facilitates the researcher's interaction with electron microscopy data. This environment may also become available to users at remote sites, making the microscope a resource in a distributed laboratory.
The authors thank those whose support has contributed to this work, including the state of California, the University of California and the NSF Biological Sciences Directorate. Many thanks to Tom Deerinck and Steve Lamont at SDMIR, Jim Madden at UCSD, Tom Hutton, Paul Love and Mike Bailey at SDSC, and Tom Erickson at Apple Computer, Inc.
. Agar, A., Alderson, R., Chescoe, D. Principles and practice of electron microscope operation. In Practical Methods in Electron Microscopy, A. M. Glauet, Ed., American Elsevier, N.Y., 1974.
. Bancroft, G.V., Plessel, T., Merritt, F., and Walatka, P.P. Scientific visualization in computational aerodynamics at NASA Ames Research Center. Computer 22, 8 (Aug. 1989), 89-95.
. Carragher, B., Hessler, D.F., Hinshaw, J.E., Martone, M., Milligan, R.A., Young, S.J., and Ellisman, M.H. Computer-aided image analysis and graphics in biological microscopy at higher voltages. In Proceedings of the 49th Annual Meeting of the Electron Microscopy of America (1991), p. 438.
. Ellisman, M.H., Lindsey, J.D., Carragher, B.O., Kiyonaga, S.H., McEwen, L.R. and McEwen, B.F. Three-dimensional tomographic reconstructions of components of the Golgi apparatus imaged by selective staining and high voltage electron microscopy. J. Cell Biology, III 5, 2, 199a.
. Ellisman, M.H., Carragher, B., and Martone, M. Three-dimensional microscopy and computer graphic representations of internal membrane system components of neurons. In Proceedings of the 49th Annual Meeting of the Electron Microscopy of America (1991), p. 150.
. Elvins, T.T. A visualization computing environment for a widely dispersed scientific community. State of the Art in Data Visualization, ACM SIGGRAPH 90 Course Notes. Course Number 27 (Dallas, Tex., Aug. 1990), pp. 2.1-2.50.
. Elvins, T.T., Nadeau, D. NetV: An experimental network-based volume visualization system. In Proceedings of the IEEE Visualization '91 Conference, IEEE Computer Society Press, Oct. 1991, pp. 239-245.
. Frank, J., Radermacher, M. Three-dimensional reconstruction of non-periodic macromolecular assemblies from electron micrographs. In Advanced Techniques in Biological Electron Microscopy III, J.K. Koehler, Ed., Springer Verlag Press, 1986.
. Gore, A. HPCC policy champion forsees networked nation. Commun. ACM 34, 11 (Nov. 1991).
. Haber, R.B., McNabb, D.A. Visualization idioms: A conceptual model for scientific visualization systems. In Visualization in Scientific Computing, Nielson and Shriver, Eds. IEEE Computer Society Press, 1990, pp. 74-93.
. Hessler, D., Young, S., Carragher, B., Martone, M., Hinshaw, J., Milligan, R., Masliah, E., Whittacher, M., Lamont, S. and Ellisman, M. SYNU: Software for visualization of three-dimensional biological structures. J. Microscopy Soc. Am. 22, 1 (Mar. 1992).
. Hill, W.C. and Hollan, J.D. Deixis and the future of visualization excellence. In Proceedings of the IEEE Visualization '91 Conference, IEEE Computer Society Press, Oct. 1991, pp. 314-319.
. Martone, M.E., Zhang, Y., Simpliciano, V.M., Carragher, B.O., and Ellisman, M.H. Three-dimensional visualization of the internal membrane system of Avian Pur-kinje cell dendrites. Soc. Neurosciences Abstracts 1991, p. 1573.
. Masliah, E., Ellisman, M., Carragher B., Mallory, M., Young, S., Hansen, L., DeTeresa, R. and Terry, R. Three-dimensional analysis of the relationship between synaptic pathology and neuronal threads in Alzheimer Disease. J. Neuropathology and Exp. Neurology, (1992). To be published.
. McCormick, B.H., DeFanti, T.S. and Brown, M.D. Visualization in scientific computing. Comput. Graph. 21, 6 (Nov. 1987).
. Norman, D.A. and Draper, S.W., User Centered System Design, Lawrence Erlbaum Associates, Hillsdale, New Jersey, 1986.
. Phillips, R.L. Distributed visualization at Los Alamos National Laboratory. Computer 22, 8 (Aug. 1989), 70-77.
CR Categories and Subject Descriptors: C.2.4 [Computer-Communication networks]: Distributed systems; I.3.2 [Computer graphics]: Graphics systems; I.3.6 [Computer graphics]: Methodology and techniques; J.3 [Computer Applications]: Life and medical sciences
General Terms: Design, Experimentation, Human Factors
Additional Key Words and Phrases: Distributed systems, microscopy, telepresence, visualization
|Printer friendly Cite/link Email Feedback|
|Title Annotation:||technical; SIGGRAPH '92 Showcase; includes a related article on how such a system would be used by researchers in the future; the Microscopist's Workstation project at the University of California at San Diego Microscopy and Imaging Resource|
|Author:||Mercurio, Philip J.; Elvins, T. Todd; Young, Stephen J.; Fall, Kevin R.; Ellisman, Mark H.|
|Publication:||Communications of the ACM|
|Article Type:||Cover Story|
|Date:||Jun 1, 1992|
|Next Article:||The CAVE: audio visual experience automatic virtual environment.|