Printer Friendly

A test of the S/P ratio as a correlate for brightness perception using rapid-sequential and side-by-side experimental protocols.


Trichromacy of human vision permits the mixing of three spectral primaries to create white light. Importantly, the choice of the wavelength regions for the primaries is not arbitrary when the goal is to maximize the perceptual response per watt of optical radiation. It has been demonstrated that the spectral regions at or around 450, 530, and 610 nm are uniquely efficient at stimulating the visual system when targeting increased brightness and color perception per watt [for example, Thornton 1992a 1992b 1992c, Houser and Hu 2004, Houser and others, 2004]. The intermediate regions that are centered near 490 and 570 nm are less effective at eliciting perceptions of brightness. Thornton has called the 450-530-610 nm spectral regions the "prime color" regions and the 490 and 570 nm regions the "anti-prime" [Thornton 1992a]. The genesis for these concepts can be traced at least as far back as Thornton's invention of the triphosphor fluorescent lamp [Thornton 1979a 1979b].

When creating white light that is maximally efficient per watt, it is not enough to simply increase the amount of optical radiation in the prime-color spectral regions; perhaps counter intuitively, it is also beneficial to vision to reduce the optical radiation in the anti-prime spectral regions. Houser and his colleagues demonstrated prototype fluorescent lamps that, when compared against conventional lamps, concurrently enhanced perceptions of brightness and color. This benefit was achieved by reapportioning some of the optical radiation from the anti-prime spectral regions to the prime-color regions [Houser and others, 2004].

The emission characteristics of light emitting diodes (LEDs) permit the creation of white-light spectra that are dominated by three spectral peaks. When carefully tuned in consideration of the underlying human visual response, this method has the potential to improve the visual benefit per watt (of optical radiation) in comparison to conventional light sources. The prime-color theory of brightness perception asserts that the most visually effective white light, in terms of both brightness and color perception per radiant watt, will be achieved with a spectrum that is dominated by optical radiation in the 450-530-610 nm spectral regions.

Entirely separate from trichromacy, and without reference to it, Berman has argued that the perception of spatial brightness is related to the ratio of the Scotopic (S) to Photopic (P) lumens [Berman and others, 1990]. The quantity [P.sup.*][(S/P).sup.0.5] has been promoted as a correlate for brightness perception [Berman & Liebel, 1996] and the use of lamps with a high S/P ratio has been promoted as a method for reducing energy consumption. When this expression was originally proposed, it was presumed that the rod photoreceptors were contributing to brightness perception at light levels traditionally considered to be photopic. More recently, however, the discoveries of the intrinsic photosensitivity of some retinal ganglion cells [Berson and others, 2002] and the action spectrum for melatonin suppression [Brainard and others, 2001, Thapan and others, 2001] have led to the proposition of the circadian proxy. Here, Berman advocates that the Cirtopic spectral sensitivity is the true driver of brightness perception while Scotopic sensitivity can explain the effect because the Scotopic and Cirtopic functions have peaks near to each other [Berman 2008]. Berman notes a conversion: [S/P=(0.66C/P).sup.0.74]. However, this relationship would not be accurate for narrow band sources, such as LEDs.

When considered in the context of working interiors, Berman's assertions have often been simplified to the recommendation of employing lamps with higher Correlated Color Temperature (CCT) since such sources tend to have higher S/P ratios. Despite the fact that there are many studies in the peer-reviewed literature that conflict with Berman's conclusions [for example, Smith & Rea 1979, Vrabel and others, 1995, Boyce and others, 2003, Houser and others, 2004, Hu and others, 2006], high CCT lamps are nonetheless being pressed into use because of a desire to reduce energy consumption.

To expand on one of the studies cited above, Houser and his colleagues developed prototype fluorescent lamps based on Thornton's prime color criteria [Houser and others, 2004]. By adjusting the magnitude of optical radiation within the prime-color and anti-prime spectral regions, they created lamps with low and high CCT and lamps that varied in their trichromatic potential. Their experiment demonstrated: 1) a pair of lamps with similar S/P ratios that elicited statistically different perceptions of spatial brightness, and 2) a pair of lamps with very different S/P ratios that elicited equivalent perceptions of spatial brightness. These results cast doubt on the use of the S/P ratio and related measures such as [P.sup.*][(S/P).sup.0.5] as proxies for brightness perception and called into question the practice of using high S/P lamps as an energy saving strategy. Houser and his colleagues suggested that the perception of brightness is more dependent upon the placement of optical radiation within key spectral regions than on the absolute amount of radiation within each of these regions.

There is little disagreement about the opportunity to reduce energy consumption by better aligning the radiant output of electric light sources with the spectral regions that yield the most beneficial visual response. There is not agreement about how such tuning should occur. In North America, many lighting design professionals and building occupants prefer environments to be illuminated with warmer light sources that have lower S/P ratios [Houser and others, 2004]. It has also been shown, both theoretically and experimentally, that higher CCT light sources should not be expected to appear brighter [Hu and others, 2006]. The work presented here provides additional clarity on the best path forward. As will be shown, in a direct test of the S/P ratio using methodologies comparable to those used by Berman in his original work, the S/P ratio was found to relate poorly to the perception of brightness.

This experiment also investigated two of the common methods used to evaluate brightness perception: simultaneous side-by-side comparisons and rapid-sequential comparisons. While many studies have employed one of these methods, we know of only one study that has used both methods to obtain data from a single sample group [McNelis and others, 1985]. McNelis and his colleagues did not test whether the difference between the two methods was significant and they did not provide sufficient data to permit this to be done. Fotios and Houser suggest that these two methods will yield similar results given equal experimental parameters [Fotios & Houser, 2007], but heretofore they had not been directly compared.

In the side-by-side ranking task two stimuli are presented simultaneously in adjacent spaces. Subjects are asked to make a forced choice about a property or properties of the stimuli, such as brightness. Several types of bias must be accounted for to generate reliable results that do not misrepresent the effect being observed. A positional bias may occur if the pairs of stimuli are not counterbalanced [Fotios & Houser, 2007]. While random chance would result in a 50/50 split when two identical stimuli are presented, previous studies that documented null condition trials show that this does not always occur [Taylor & Sucov 1974, Fotios 2001, Houser and others, 2004]. When each stimulus is presented at multiple levels of the independent variable, it is also necessary to consider the potential for stimulus frequency bias [Fotios & Cheal, 2008].

Rapid-sequential ranking is an alternative to side-by-side ranking. Here, a subject is in a single room and two stimuli are presented in a rapidly alternating (< 5 second) sequence. An interval bias is related to the effect of presentation order [Klein 2001] and is manifest as a consistent asymmetry in the direction of the psychophysical response [Yeshurun and others, 2008]. An interval bias is analogous to the positional bias of simultaneous evaluations. Interval biases have been found in experiments that use successive evaluation without repeats [Jakel & Wichmann 2006, Yeshrun and others, 2008]. In such a protocol, the subject observes the first stimulus, then the second stimulus, and then makes a judgment. The rapid-sequential method may remove this bias because the observer can continually refresh his or her memory of both stimuli being evaluated. Previous lighting research that used the rapid-sequential method [Berman and others, 1990, Vrabel and others, 1998] did not use a null condition so there is no opportunity to check for an interval bias in their work. The stimulus frequency bias that may occur with side-by-side ranking is also possible with the rapid-sequential protocol.

Another experimental consideration that is important when varying the light spectrum is the chromatic adaptation of the subject. In side-by-side experiments, the subject adapts to a mixed spectrum where the white point is somewhere between the two stimuli. This adaptation occurs temporally with 60 percent adaptation being reached within 5 seconds, greater than 90 percent adaptation after 60 seconds, and full chromatic adaptation at about 2 minutes [Fairchild & Reniff, 1995]. Since only 60 percent chromatic adaptation occurs after 5 seconds, a subject in a rapid-sequential experiment will similarly be adapted to a mixed condition with a white point between the chromaticity of the two alternating stimuli.

Recent reviews have examined over 50 studies on the relationship between SPD and brightness perception [Fotios & Houser 2008, Fotios and others, 2008, Fotios 2001]. Unaccounted experimental bias renders many of the results unreliable. In those studies deemed reliable, Fotios and Houser note that nearly all find some relationship between SPD and perceived brightness [Fotios & Houser 2008]. This review also found that studies where the subject experienced mixed chromatic adaption reported a stronger effect, while studies where the subject experienced full chromatic adaptation reported a weaker but still significant relationship between SPD and brightness.



The independent variables were spectral power distribution (SPD), luminance, and the experimental protocol. Each of these variables had two levels.

SPD can be manipulated in an infinite number of ways and is therefore a broad and unconstrained variable. It is traditionally characterized with derived metrics including CCT, color rendering index (CRI), gamut area, and the S/P ratio. All of these measures reduce the SPD to a single number. This is convenient for the purpose of analysis, but as with all reductionist approaches it is intrinsically problematic because of the potential to overlook important characteristics of the independent variable that have not been captured in the derived measure. For example, many SPDs will produce the same CCT, but it is well-known that psychophysical responses such as color preference and brightness perception are unrelated to CCT. Many SPDs will also produce the same S/P ratio, yet, interestingly, the fact that such SPDs might produce different perceptions of brightness seems to be unrecognized.


In this experiment the same three spectral primaries, which were generated using narrow emitting LEDs (described below), were used to create light spectra with two different CCTs and, correspondingly, two different S/P ratios. This method minimized the confounding that is characteristic of other work. Berman and his colleagues radically modified the wavelength components in order to generate light of the same chromaticity but with different S/P ratios [Berman and others, 1990]. In such situations, attributing the visual response (brightness perception) to the derived metric (S/P ratio) is an act of faith. The psychophysical response attributed to the S/P ratio of the illuminant could just as plausibly be explained by changes in the wavelengths of the spectral components that combined to form the composite spectra that were evaluated by the subjects.

The emergence of high-brightness narrow-emitting LEDs has made it practical to create light settings with highly structured spectra. In this experiment, one stimulus had a nominal CCT of 2900 K and a corresponding S/P ratio of 1.7. The second stimulus had a nominal CCT of 7200 K and a corresponding S/P ratio of 2.6. Both stimuli were on the blackbody locus. Relative SPDs are shown in Fig. 1. The values for CCT were selected because they represent the practical extremities for white-light that is used in commercial interiors. If Berman's assertion about the relationship between S/P ratio and brightness perception were to hold true, it should have been found in this experiment given the large differences in CCT and S/P ratio.

The second independent variable was luminance, which also had two levels: 24 and 30 cd/[m.sup.2].

Since each variable had two distinct levels, there were four unique light settings: 2900 K at 24 cd/[m.sup.2] (light setting A), 2900 K at 30 cd/[m.sup.2] (light setting B), 7200 Kat 24 cd/[m.sup.2] (light setting C) and 7200 Kat 30 cd/[m.sup.2] (light setting D). These light settings are summarized in Table 1. The four light settings were created by individually controlling the output of the red, green, and blue LEDs. The peak wavelength of each channel remained the same for both the 2900 and 7200 K settings, but the relative output of each channel changed. These combinations were chosen with specific contrasts in mind. For example, the ratio of [P.sup.*][(S/P).sup.0.5] [Berman & Liebel 1996] is 1:1 for the comparison of 2900 K, 30 cd/[m.sup.2] (light setting B) vs. 7200 K, 24 cd/[m.sup.2] (light setting C). If the [P.sup.*][(S/P).sup.0.5] ratio were to hold true then light settings B and C would be selected an equal number of times. The photopic, scotopic and [P.sup.*][(S/P).sup.0.5] ratios are summarized for all pairs in Table 2.

Each room was calibrated for chromaticity coordinates and luminance in advance of the experimental sessions. The chromaticity coordinate targets were on the blackbody locus at 2900 and 7200 K and the luminance targets were 24 and 30 cd/[m.sup.2]. The settings for the DMX control system remained constant for all subjects. Chromaticity was measured using a StellarNet EPP2000c spectrometer. The remote integrating sphere for this apparatus was positioned on a tripod at 114 cm above the floor with the opening facing the back wall of the room. This corresponded to the position of the subjects' eyes during the rapid-sequential presentation mode. Luminance was measured using a Minolta CS-100 chroma meter aimed at a point 114 cm above the floor in the center of the back wall of each room. Because of the physical layout of the test environment and the geometry of the apparatus, the luminance on the back wall was employed as a proxy for illuminance at the plane of the subject's eyes.

Measurements of chromaticity and luminance were taken for each room both before and after running a block of approximately five subjects. The overall precision, considering both rooms and all settings, was [+ or -] 4.6 percent for CCT and [+ or -] 2.1 percent for luminance. Table 3 shows the mean and standard deviations for each of the four light settings in both rooms, summarizing both luminance and CCT. Temperature stability, and in particular the temperature of the p-n junction, is known to influence the chromaticity and lumen output of LEDs. This is believed to be the main cause of the variance summarized in Table 3. It was clear during our calibration that the red LED spectral primary was particularly sensitive to temperature. To minimize temperature effects the LEDs were turned on at least 3 hours before calibration and before a block of subjects was run.

The third independent variable was the presentation mode. All subjects evaluated all pairs of light settings in both a side-by-side and rapid-sequential presentation. Both rooms were used for the side-by-side presentation mode whereas just the left room was used for the rapid-sequential mode.


Subjects were asked to choose which light setting of the pair was brighter. This was a forced choice. If the subject declared that they were equal, then the experimenter instructed him or her that some of the judgments might be difficult, but that an honest assessment of which of the pair appeared brighter was required.


The four light settings result in 16 permutations of stimulus pairs. Each permutation was assigned a comparison number as shown in Table 2. Four of the pairs were null condition trials. There are six permutations that can be presented in two different orders--AB and BA, for example--which accounts for the other twelve pairs. Each subject evaluated the 16 pairs in a random sequence, as well as four fixed pre- and post-trials that were not randomized and were identical for all subjects. Thus, each subject evaluated a total of 24 pairs for the rapid-sequential method and another 24 pairs for the side-by-side method. The sequence of the experimental methods alternated between subjects. The subjects were aware that the first three trials were practice trials, but were not told that the fourth trial and trials 21-24 would not be included in the primary analysis. An example of the recording sheet used by the experimenter is shown in Fig. 2. For analysis, the 12 non-null-condition trials were grouped based on unique pairs: AB was combined with BA, for example. This was done to counter any positional bias with the side-by-side method and interval bias with the rapid-sequential method.


Two empty rooms, enclosed on three sides, with nominal dimensions of 3.05 m (width) x 3.66 m (depth) x 2.74 m (height) were assembled adjacent to one another. All wall surfaces were painted with Munsell N8 spectrally neutral paint, which was purchased from RP Imaging. The rooms were enclosed with a black felt curtain that was behind the subject and out of his or her field of view (FOV). 61 cm x 61 cm acoustical tile was used for the ceiling and gray carpeting was installed on the floor. Indirect pendant luminaires that were approximately 2.5 m long were fitted with the custom LED strips. The luminaires were suspended 38 cm below the ceiling, with four in each room. The LEDs were diffused with translucent Mylar that was fitted across the opening of each luminaire to minimize color striations on the walls and ceiling. The rooms appeared as near to visually identical as reasonably possible. A view of the side-by-side arrangement from behind the subject is shown in Fig. 3. A view from within the left room, looking back toward the subject's position when arranged for the rapid-sequential presentation, is shown in Fig. 4.


Power supplies and drivers for the LEDs were housed above the ceiling. The system was controlled by custom DMX software that was developed in cooperation with Lighting Science Group Corporation. The light settings were triggered by the experimenter from a computer that was outside of the rooms. For the rapid-sequential experiments, the DMX software permitted the light settings to be controlled by timed scripts, ensuring rigorous consistency in the timing of the presentations for all subjects.

For both evaluation modes, the subject was seated at a table with his or her head positioned in a chin / forehead rest. Eye height was approximately 114 cm above the floor, varying slightly with the size of a subject's head. The subject was instructed that he or she was free to move his or her head within the chin / forehead rest, but to focus on the back wall(s) of the room when making the final judgment. For the side-by-side presentation, the table where the subject was seated was positioned against the dividing wall, with the sagittal plane aligned with the middle of the two spaces. For the rapid-sequential presentation, the table was positioned entirely within the room, with approximately 2.85 m from the center of the back wall to the subject's eyes. During the rapid-sequential presentation, the room on the right was simultaneously controlled and set to an identical setting, though this was not visible to the subject.



The subjects were categorized as either expert or naive. Expert subjects were members of: 1) the Illuminating Engineering Society of North America (IESNA) Quality of the Visual Environment (QVE) Committee, 2) the International Association of Light Designers (IALD) Metrics of Quality (MOQ) Committee, or 3) an attendee of the Project CANDLE workshop held at Penn State University in April 2009. The mean age for expert subjects was 47.4 years with a range of 23 to 69 years and a standard deviation of 11.9. The group included 2 females and 15 males. The naive subjects were recruited from Penn State and the surrounding community. The mean age for naive subjects was 26.3 years with a range of 19 to 63 years and a standard deviation of 11.5. The group included 10 females and 20 males.



Upon arrival, subjects read a brief description of the experiment and signed an informed consent form. The Keystone Visual Skills Test was then administered; no subject was excluded from the experiment based on the results of this test, or for any other reason.

After vision screening, the subject was escorted to the experiment room. The illumination in both rooms was set at an average of the four light settings; that is, a CCT of about 5050 K, and 27 cd/[m.sup.2] measured at seated eye level on the back wall of both rooms. The subject was seated, the experimenter read the instructions, and then proceeded to run the 24 pairs using one of the two experimental methods. The instructions and the experimental procedures were always read from a script to minimize variation between subjects. When the first experimental method was complete, the experimenter set both rooms to the average light setting as was used when the subject first entered the space, and then moved the table containing the chin / forehead rest to the location required for the next experimental method. The subject remained in the experiment room while the experimenter repositioned the table. A new set of instruction was read, followed by the subject's evaluation of the 24 pairs of light settings. The table was not moved when the subject was finished and in this way the order of the experimental method alternated from subject to subject.

For the side-by-side presentation, the experimenter advanced the light settings to the next stimulus pair by advancing a pre-written script file. Subjects experienced 150 ms of darkness between each trial. Subjects were instructed to hold their judgment for at least thirty seconds to allow for adaptation to the new settings. The experimenter prompted the subject after 30 seconds, but the subject was allowed as much time as necessary to judge which room appeared brighter. The experimenter recorded the subject's response on a survey form before moving to the next trial.

For the rapid-sequential presentation, for each pair of light settings the experimenter loaded a script that alternated the light in the room every 5 seconds with a dark period of 10 ms between each alternation. This 10 ms dark period was implemented to provide the subject with a visual cue that the settings were changing. The experimenter also spoke aloud "A, B, A, B ..." as the settings changed. The subject was instructed to wait for at least three alternations (that is, ABABAB) before providing a response. Once a response was given, the script was stopped and the judgment was recorded. The experimenter then loaded the next script, which always began with a single dark period of 150 ms.


Data were recorded as left/right (side-by-side) or first/second (rapid-sequential). These data were converted to binary digits (0 or 1) for analysis. Thus, the number of observations is double the number of subjects because the inverse presentations (for example, AB and BA) were combined into one data set. To provide meaningful statistical analyses, counts were summed and percentage values for each condition are reported. Statistical analyses were performed using the number of subjects, rather than the number of observations, since the two observations made for each stimulus pair are not independent. Essentially, the two observations per stimulus pair have been averaged.


As shown in Table 3, the average luminance and CCT in the left room and right room were not identical. Across all light settings, the mean luminance in the left room was 0.32 cd/[m.sup.2] brighter at the calibration point. By presenting all mixed comparisons in both orders, the effect of this difference was distributed evenly across all stimuli. Null condition trials were included to establish the prevalence of bias. The results of these tests are shown in Table 4. Overall, subjects chose the left room 57 percent of the time in the side-by-side null condition trials. This bias just reached statistical significance (p = 0.041). The finding is similar to results found in a previous study (59 percent chose the left room) that used similar methods [Houser and others, 2004]. For the rapid-sequential method, subjects chose the first stimulus 49 percent of the time, a proportion that is not statistical different from 50 percent.

Of the null-conditions trials, AA, BB, and CC were not statistically different, but DD was. Luminance measurements taken at the end of the experimental sessions show that DD had the greatest difference in measured luminance (30.6 cd/[m.sup.2] on the left, 30.1 cd/[m.sup.2] on the right). This luminance difference of 0.5 cd/[m.sup.2] was greater than for the other null conditions (AA=0.2 cd/[m.sup.2], BB=0.3 cd/[m.sup.2], CC=0.3 cd/[m.sup.2]). The left room was always brighter. This likely explains the left side bias and the statistical significance for DD, but it has little relevance to the rest of the experiment since the stimulus presentation in the left and right rooms was counterbalanced. These measurements and statistical findings suggest that subjects were sensitive to small differences.

Since the experimental design provided for complete counterbalancing, random chance suggests an even number of selections for left/right and first/ second for the entire experiment, not just the null condition trials. For the side-by-side method, the left room was selected as the brighter room 51.2 percent of the time. For the rapid-sequential method, the first stimulus in the sequence was selected 47.9 percent of the time. Neither of these proportions is statistically different from 50 percent.

The total number of counts, broken down by either left/right or first/second, also suggests that subjects responded to the light settings rather than to the position (left/right) or interval (first/second). Consider AD, BC, AC, BD, CD, AB for the side-by-side method, where the first letter symbolizes the light setting in the left room and the second letter symbolizes the light setting in the right. The left room was selected 68 times and the right room 214 times. When the counterbalanced comparisons are considered (that is, DA, CB, CA, DB, DC, BA) the left room was selected 209 times and the right room 73. These two pairs of numbers are not statistically different (McNemar test: Z=0.243, which is less than the critical value of [Z.sub.0.05] = 1.96). The results are similar for the rapid-sequential method: AD, BC, AC, BD, CD, AB yielded 84 selections for the first light setting and 198 selections for the second; DA, CB, CA, DB, DC, BA yielded 184 selections for the first light setting and 98 selections for the second. These pairs of numbers are not statistically different (McNemar test: Z=0.716).

Bias, while undesirable, may in practice be unavoidable. A goal should be to minimize the potential for bias through the design of a careful apparatus and the use of careful experimental methods. It is equally important to counter any potential biases through good experimental designs. Our use of counterbalancing means that the small bias found in the side-by-side method was equally distributed among all pairs of light settings; counterbalancing means that the bias did not contribute to the inferences about the main effects. It should be noted that very few studies include null condition trials; such studies are dubious because they do not provide a means with which to gauge the validity of the tests.


McNemar's test [Sheskin 2007] was used to test for differences between the two presentation methods. As in other analyses, the two permutations for each pair of stimuli were grouped together as one combination. McNemar's test only allows for the evaluation of a single dichotomous dependent variable. Thus, the presentation methods as a whole were not tested based on a statistical hypothesis. Rather, each specific paired comparison was tested; for example, AC in the rapid-sequential method vs. AC in the side-by-side method. A sample McNemar test matrix is provided in Fig. 5.

The results for all 10 McNemar tests are provided in Table 5, which show that 2 of the 10 trials, BC and BD, have statistically different results for the two presentation methods. It is important to consider the context of these differences, however. Table 6 shows the percentages chosen for the 6 mixed combinations for both methods. While the McNemar test indicates that the presentation methods yielded a different result for the BC trials, in both methods more subjects found stimulus B (2900K, 30 cd/[m.sup.2]) to be brighter than stimulus C (7200K, 24 cd/[m.sup.2]) and these differences are statistically significant for both experimental methods. Said another way, the proportion 77/23 found with the side-by-side method is different than the proportion 96/4 found with the rapid-sequential, but both proportions are statistically different from 50/50. In the second combination where the McNemar test indicates a statistically significant difference, BD, the two presentation methods also led to the same conclusions: stimulus D (7200 K, 30 cd/[m.sup.2]) was not perceived to be different from stimulus B (2900 K, 30 cd/[m.sup.2]). Importantly, both experimental methods produced the same statistical results for all trials, as described in the next section.


The effect of SPD on brightness perception was examined by analyzing the results of the 6 mixed condition trials. Combinations AB and CD compared two stimuli with the same CCT but different luminance. Combinations AC and BD compared two stimuli with different CCTs but the same luminance; any effect of CCT and S/P ratio should therefore be prominent. Combinations AD and BC compare two stimuli with a different CCT and a different luminance. Of special note is combination BC that compared 2900K, 30 cd/[m.sup.2] to 7500K, 24 cd/[m.sup.2]. The ratio [P.sup.*][(S/P).sup.0.5] for these two stimuli is 1:1.

Each method was analyzed using variance stable rank sums (VSRS), which is an adaptation of two-way analysis of variance by ranks developed by Dunn-Rankin [Dunn-Rankin and others, 2004] and used in previous lighting research [Quellman and Boyce 2002]. Results from the VSRS test for the expert and naive subjects showed that the same comparisons were significantly different for both groups. Therefore, it was possible to pool the expert and naive subject groups. Fig. 6 shows the results of VSRS analysis for the side-by-side and rapid-sequential methods. Of the six stimulus pairs, both methods find that in the four combinations where one stimulus has greater photopic luminance, it was chosen to be brighter by the subject regardless of the CCT and S/P ratio. In both pairs where stimuli had equal photopic luminance, the subjects did not choose one to be brighter with statistical significance, regardless of the CCT and S/P ratio.



In trials where stimuli of the same CCT and different luminance were presented (that is, AB and CD) subjects unanimously chose the stimulus with the higher luminance in the rapid-sequential method, as shown in Table 6. The side-by-side comparison for the same combinations did not result in a unanimous selection, although it was nearly so (Table 6).The results of the null condition trials show that the overall group selected the first and second stimuli almost equally for the rapid-sequential method (Table 4). When the side-by-side method was employed, the subjects had a tendency to select the room on the left, at a level that just reached statistical significance (p = 0.041). Recall that, on average, the left room was measured at a slightly higher luminance level (Table 3), so this finding is consistent with the actual experimental conditions. Importantly, the full counterbalancing of the experimental design effectively distributed this noise so that the main effects were minimally disrupted.

Based on the VSRS test (Fig. 6), both methods conclude that, for the four combinations that had a difference in luminance, one stimulus of the pair was selected significantly more than its counterpart. The VSRS test also shows that for the two combinations with identical luminance neither stimulus was selected more than its counterpart. These trends hold when the overall group is separated into subgroups for expert and naive observers.

McNemar's test (Table 5) indicates that for two of the combinations, BC and BD, the proportions found for the side-by-side and rapid-sequential methods are statistically different. For combination BC, the subjects' choice was the same (Table 6) and the VSRS tests show that they were statistically different for both evaluation modes (Fig. 6). Though the subjects' choice differed for combination BD (Table 6), the VSRS tests show that they were not statistically different with either evaluation mode (Fig. 6). In short, both evaluation modes led to the same general conclusions (Fig. 6), albeit with greater statistical significance for the rapid-sequential method.

The result that the side-by-side and rapid-sequential methods produce similar results can likely be generalized. Presenting the stimuli in a single space eliminates differences in the physical environment and may help to reduce differences in the luminous conditions that can occur when using side-by-side light settings. However, the rapid-sequential presentation method may not always be feasible. For example, the output for some lamps is so variable with temperature that rapidly switching them on-and-off would required a sophisticated apparatus that may be unrealistic. This study shows that the side-by-side method can yield reliable and defensible data. Regardless of the method, null condition trials and counterbalancing should always be employed.


The four light settings that were used in this experiment were specifically chosen to test for the effect of the S/P ratio on brightness perception. Since all stimuli were produced by the same set of LEDs, they all emitted optical radiation in the same three spectral regions. The difference in luminance was less than 1/3 of a log unit, which is often considered to be the minimum meaningful threshold for brightness differences. The difference in CCT (and the corresponding S/P ratios) was as large as practical and was at the extremes of what is used within interiors. The large difference in CCT was selected to give the maximum potential for finding an effect.

When the luminance was different, as with AB, AD, BC, CD, the perception of brightness was based solely on luminance; there was no effect of CCT or S/P ratio. In pairs AC and BD, which had equal luminance but different CCT and S/P ratios, one stimulus was not chosen over the other with statistical significance; the two stimuli must be considered equally bright. These findings were true for both expert and naive subjects.

Of special note is the comparison BC, which has a [P.sup.*][(S/P).sup.0.5] ratio of 1:1, thus predicting that the two stimuli would appear equally bright and that subjects would select each stimulus half the time. For the side-by-side method, subjects chose stimulus B 77 percent of the time. For the rapid-sequential method, subjects chose stimulus B 96 percent of the time: statistical analysis suggests both of these are a significant departure from 50 percent and thus light setting B and C did not appear equally bright. Therefore the [P.sup.*][(S/P).sup.0.5] model did not accurately predict brightness perception for these stimuli. At the very least, it is a model that cannot be generalized beyond the original data from which is was generated. It is likely an incorrect model.

The 1990 work of Berman and his colleagues [Berman and others, 1990] made use of lighting conditions that were approximately metameric. In the years since, their results have been generalized and applied to nonmetameric situations. Where there is a change in CCT, there is a corresponding hue shift and metamerism no longer exists. It is important to lighting practice and to the spectral design of lamplight that the conclusions of Berman and his colleagues about the correlation between brightness perception and [P.sup.*][(S/P).sup.0.5] may only apply at constant chromaticity. We don't dispute the results of their original work, but the data presented herein makes it clear that [P.sup.*][(S/P).sup.0.5] does not generalize to situations where the illumination is not metameric.

Though this experiment was not designed to test a specific hypothesis about trichromacy, it is nonetheless possible to make indirect inferences because of the way that SPD was adjusted. As illustrated in Fig. 1, all SPDs employed in this experiment were comprised on the same three spectral components. Under these conditions, trichromacy and the opponent colors model suggest that luminance will predict brightness perception, at least within the region near the blackbody locus. By maintaining a single set of spectral primaries, the achromatic (luminance) and chromatic (blue-yellow, red-green) channels will be stimulated in the same general ways, though not with identical magnitudes. While motion along the blackbody locus can be expected to affect the perceptions of brightness, creating such movement with the same primary set was shown here to maintain an equivalent perception of brightness. Conversely, if the primary components had been changed, brightness perception can be expected to be different even at equal chromaticity and luminance, as has been previously demonstrated [Houser & Hu 2004].

Reductionist measures that simplify a complex SPD into a single number are intrinsically apocryphal, yet there is no denying their appeal for simplicity of commerce and application. Measures that are unconnected to trichromacy and the opponent-colors model, such as CCT, the S/P ratio, [P.sup.*][(S/P).sup.0.5] and [(0.66C/ P).sup.0.74] cannot be expected to predict brightness in a generalizable way. A successful index will most likely be based upon trichromacy and the opponent colors model, will likely be rooted in a color appearance model [CIE 2004], and may employ differential weightings for the prime-color and anti-prime spectral regions. Work toward the goal of a universal index for spatial brightness is ongoing, including work by the IES Visual Effects of Lamp Spectral Distribution Committee.


This experiment had two primary inquiries: to investigate the relationship between the S/P ratio and spatial brightness perception and to compare directly the rapid-sequential and side-by-side methods of assessing judgments of brightness.

The rapid-sequential and side-by-side methodologies were shown to produce comparable results. Though the side-by-side method has informally been called into question by others, there is no evidence to dismiss it as faulty or invalid. Indeed, many real life situations involve side-by-side comparisons under different lighting conditions. One practical and ubiquitous example is adjacent storefronts within enclosed shopping malls. The rapid-sequential method was also found to produce clear results and we found no reason to exclude it from future psychophysical research. The conclusions of the statistical analyses for the side-by-side and rapid-sequential methods were comparable. The choice of one method verses the other may include experimental practicalities, such as lamp operating characteristics, control capabilities, and available space.

The S/P ratio and CCT were not found to be important predictors of brightness perception. The light settings evaluated were selected to present the practical extremities of CCT (2900Kvs. 7200K) and a large difference in S/P ratio (1.7 vs. 2.6). These very large differences did not lead to statistically different perceptions of brightness at equal luminance with either the rapid-sequential or side-by-side methodologies. Further, when the ratio [P.sup.*][(S/P).sup.0.5] was made equal, by making luminance unequal, subjects overwhelmingly selected the room with the higher luminance as brighter. When considered along with other work, we conclude that brightness perceptions at photopic light levels are unrelated to the S/P ratio of the illuminant.

Within this experiment, brightness perception was not perceived to be unequal when the following two conditions were met: 1) luminance was equal; 2) the spectral lights that combined to form the SPD were altered in relative magnitude, but not wavelength. This provides indirect evidence that spatial brightness perception is dependent upon the placement of optical radiation within key spectral regions, more so than on the magnitude of the radiation within those regions. Because the visual system is fundamentally a three-channel system, light sources have the potential to be most effective if they are designed to harmoniously stimulate the three visual channels that underlie normal human vision. Thornton has demonstrated that these peak spectral sensitivities are near 450-530-610 nm, with minima sensitivities near 500 and 580 nm and at the limits of the visual spectrum [Thornton 1992a, 1992b, 1992c]. Light sources tuned to the visual needs of people will almost certainly exploit these underlying characteristics of normal human vision.


This project was made possible by Project CANDLE partners: Cooper Lighting, Erco Lighting, Fisher Marantz Stone, Gabriel Mackinnon, Horton Lees Brogden Lighting Design, I2 Illuminations, IALD Education Trust, Lighting Design Alliance, Litecontrol Corporation, Lutron Electronics, Naomi Miller Lighting Design, Office for Visual Interaction Inc, Penn State University, Philips Lighting Company, Philips SSL Solutions, Randy Burkett Lighting Design, Schuler Shook, and the US Department of Energy (under PNNL Contract Number 79894). Litecontrol is gratefully acknowledged for the donation of the luminaires, Lumileds for the donation of the LEDs, and Lighting Science Group Corporation for the development of the control system hardware and software. Many thanks to Jamie Devenger, Luke Renwick, and Dan Moynagh for assisting with the apparatus and for their help with running the subjects.


Berman SM, Jewett DL, Fein G, Saika G, Ashford F. 1990. Photopic luminance does not always predict perceived room brightness. Lighting Res. Technol. 22(1):37-41.

Berman SM. 1992. Energy efficiency consequences of scotopic sensitivity. J. Illum. Eng. Soc. 21:3-14.

Berman SM, Liebel B. 1996. Essay by Invitation. Lighting Design and Application. November. 12-17.

Berman SM. 2000. The coming revolution in lighting practice. Energy User News. October: 24-26.

Berman SM. 2008. Correspondence: A new retinal photoreceptor should affect lighting practice. Lighting Res. Technol. 40:373-376.

Bersen DM, Dunn FA, Takao M. 2002. Phototransduction by retinal ganglion cells that set the circadian clock. Science. 295:1070-1073.

Boyce PR, Akashi Y, Hunter CM, Bullough JD. 2003. The impact of spectral power distribution on the performance of an achromatic visual task. Lighting Res. Technol. 35(2): 141-161.

Brainard GC, Hanifin JP, Greeson JM, Byrne B, Glickman G, Gerner E, Rollag MD. 2001. Action spectrum for melatonin regulation in humans: evidence for a novel circadian photoreceptor. Journal of Neuroscience. 21(16):6405-6412.

CIE 159:2004. 2004. A Colour Appearance Model for Colour Management Systems: CIECAM02. Commission Internationale de rEclairage. Vienna, Austria.

Dunn-Rankin P, Knezek GA, Wallace S, Zhang S. 2004. Scaling Methods. 2nd Ed. Mahwah, NJ: Lawrence Erlbaum Associates, Inc.

Fairchild MD, Reniff L. 1995. Time course of chromatic adaptation for color-appearance judgments. J. Opt. Soc. Am. 12(5):824-833.

Fotios SA. 2001. Lamp colour properties and apparent brightness: a review. Lighting Research and Technology 33(3):163-181.

Fotios SA, Houser KW. 2007. Research of lamp SPD effects on the perception of interior spaces: the current state of knowledge. 26th Session of the CIE, Beijing. 1:D1-111 to 114.

Fotios SA, Houser KW, Cheal C. 2008. Counterbalancing needed to avoid bias in side-by-side brightness matching tasks. Leukos. 4(4):207-223.

Fotios SA, Cheal C. 2008. The effect of a stimulus frequency bias in side-by-side brightness ranking tests. Lighting Res. Technol. 40:43-54.

Fotios SA, Houser KW. 2008. Measuring lamp SPD effect on the perception of interior spaces: frequently this is misleading. Balkan Light. R 10:69-78.

Houser KW, Hu X. 2004. Visually matching daylight fluorescent lamplight with two primary sets. Color Res Appl. 29(6):428-437.

Houser KW, Tiller DK, Hu X. 2004. Tuning the fluorescent spectrum for the trichromatic visual response: a pilot study. Leukos. 1(1):7-23.

Hu X, Houser KW, Tiller DK. 2006. Higher color temperature lamps may not appear brighter. Leukos, 3(1)69-81.

Jakel F & Wichmann FA. 2006. Spatial four-alternative forced-choice method is the preferred psychophysical method for naive observers. Journal of Vision. 6:1307-1322.

Klein SA. 2001. Measuring, estimating and understanding the psychometric function: a commentary. Perception and Psychophysics. 63(8):1421-1455.

McNelis JF, Howley JG, Dore GE & DeLaney WB. 1985. Subjective appraisal of colored scenes under various fluorescent lamp colors. Lighting Design & Application, 15(6):25-29.

Poulton EC. 1989. Bias in quantifying judgments. London: Lawrence Erlbaum Associates Ltd.

Quellman EM, Boyce PB. 2002. The light source color preferences of people of different skin tones. Journal of the Illuminating Engineering Society. 31(1):109-118.

Sheskin, DJ. 2007. Handbook of parametric and nonparametric statistical procedures. 4th ed. Boca Raton, FL: Chapman and Hall/CRC, Taylor & Francis Group.

Smith SW, Rea MS. 1979. Relationships between office task performance and ratings of feelings and task evaluations under different light sources and levels. Proceedings of the CIE, 29th Session, Kyoto, Japan.

Taylor LH, Sucov EW. 1974. The movement of people towards lights. J. Illum. Eng. Soc. 3(3): 237-241.

Thapan K, Arendt J, Skene DJ. 2001. An action spectrum for melatonin suppression: evidence for a non-rod, non-cone photoreceptor system in humans. Journal of Physiology. 535(1):261-267.

Thornton WA. 1979a. Method for efficiently generating white light with good color rendition of illuminated objects. US Patent No. 4176299.

Thornton WA. 1979b. Method and device for efficiently generating white light with good rendition of illuminated objects. US Patent No. 4176294.

Thornton WA. 1992a. Toward a more accurate and extensible colorimetry. Part I.

Introduction. The visual colorimeter-spectroradiometer experimental results. Color Res Appl. 17:79-122.

Thornton WA. 1992b. Toward a more accurate and extensible colorimetry. Part II. Discussion. Color Res Appl. 17:162-186.

Thornton WA. 1992c. Toward a more accurate and extensible colorimetry. Part III. Discussion (continued). Color Res Appl. 17:240-262.

Vrabel PL, Bernecker CA, Mistrick RG. 1995. Visual performance and visual clarity under electric light sources: Part I--visual performance. J. Illum. Eng. Soc. 22:150-165.

Yeshurun Y, Carrasco M & Maloney LT. 2008. Bias and sensitivity in two-interval forced choice procedures: Tests of the difference model. Vision Research. 48:1837-1851.

KW Houser (1) PhD, PE, LC, LEED AP, SA Fotios (2) PhD, BEng, CEng, MEI, MSLL, MILE, and MP Royer (1) BAE, MAE

(1) Department of Architectural Engineering, The Pennsylvania State University, USA; (2) School of Architecture, The University of Sheffield, UK

Corresponding author: Kevin W. Houser, Department of Architectural Engineering, The Pennsylvania State University, University Park, PA 16802, (814)863-3555,

doi: 10.1582/LEUKOS.2009.06.02003
TABLE 1. Index of the four light settings (A, B, C, D) and nominal
calibration characteristics


                24   30

CCT(K)   2900   A    B
         7200   C    D

TABLE 2. Sixteen permutation pairs of the four light settings and the
associated ratios for photopic, scotopic and P * [(S/P).sup.0.5].

No.   Pair              Comparison Characteristics

                 Left or First          Right or Second

                CCT      Photopic      CCT     Photopic
                        Luminance              Luminance

 1    AA *    2900 K        24       2900 K       24
 2    AB      2900 K        24       2900 K       30
 3    AC      2900 K        24       7200 K       24
 4    AD      2900 K        24       7200 K       30
 5    BA      2900 K        30       2900 K       24
 6    BB *    2900 K        30       2900 K       30
 7    BC      2900 K        30       7200 K       24
 8    BD      2900 K        30       7200 K       30
 9    CA      7200 K        24       2900 K       24
10    CB      7200 K        24       2900 K       30
11    CC *    7200 K        24       7200 K       24
12    CD      7200 K        24       7200 K       30
13    DA      7200 K        30       2900 K       24
14    DB      7200 K        30       2900 K       30
15    DC      7200 K        30       7200 K       24
16    DD *    7200 K        30       7200 K       30

No.               Ratios

      Photopic   Scotopic   P * [(S/P)

 1      1.00       1.00        1.00
 2      0.80       0.80        0.80
 3      1.00       0.65        0.81
 4      0.80       0.52        0.65
 5      1.25       1.25        1.25
 6      1.00       1.00        1.00
 7      1.25       0.82        1.01
 8      1.00       0.65        0.81
 9      1.00       1.53        1.24
10      0.80       1.22        0.99
11      1.00       1.00        1.00
12      0.80       0.80        0.80
13      1.25       1.91        1.55
14      1.00       1.53        1.24
15      1.25       1.25        1.25
16      1.00       1.00        1.00

* Null Condition Comparisons

TABLE 3. Average Measured Values for CCT and Luminance. Measurements
were taken before and after each Block of 5-6 Subjects using the Same
Procedure that was used during Calibration. The between-room
Differences were Counterbalanced since each Pair of Stimuli was
Presented in both Orders.

                                       LEFT ROOM

                               CCT (K)       Luminance

                            Ave     St.     Ave     St.
                                   Dev.            Dev.

2900 K, 24 cd/[m.sup.2]    2879    40.8    24.5     0.2
2900 K, 30 cd/[m.sup.2]    2912    43.0    30.8     0.3
7200 K, 24 cd/[m.sup.2]    7074   132.5    24.6     0.2
7200 K, 30 cd/[m.sup.2]    7158   124.7    30.6     0.2

                                      RIGHT ROOM

                               CCT (K)       Luminance

                            Ave     St.     Ave     St.
                                   Dev.            Dev.

2900 K, 24 cd/[m.sup.2]    2862    35.9    24.3     0.2
2900 K, 30 cd/[m.sup.2]    2894    36.9    30.5     0.2
7200 K, 24 cd/[m.sup.2]    7252   115.2    24.3     0.1
7200 K, 30 cd/[m.sup.2]    7332   117.7    30.1     0.1

TABLE 4. Summary Data for Null Condition Trials Showing Percentages of
Left-Right or First-Second Selected. The Shaded Cells with Emboldened
Values Represent Results that are Statistically different from Chance
According to the Z-test for Proportions at [alpha] < 0.05. The
Statistical Results are only Shown in the Overall Category Since the
Expert and Naive Categories do not have Enough Subjects to Meet the
Assumptions of the Test.

                                                      SIDE BY SIDE

                                              Expert   Naive   Overall

AA   2900 K, 24 cd/[m.sup.2]    Left/First     53%      57%      55%
     2900 K, 24 cd/[m.sup.2]   Right/Second    47%      43%      45%
BB   2900 K, 30 cd/[m.sup.2]    Left/First     59%      53%      55%
     2900 K, 30 cd/[m.sup.2]   Right/Second    41%      47%      45%
CC   7200 K, 24 cd/[m.sup.2]    Left/First     41%      60%      53%
     7200 K, 24 cd/[m.sup.2]   Right/Second    59%      40%      47%
DD   7200 K, 30 cd/[m.sup.2]    Left/First     47%      77%      66%
     7200 K, 30 cd/[m.sup.2]   Right/Second    53%      23%      34%
    Combined Null Trials        Left/First     50%      62%      57%
    AA + BB + CC + DD          Right/Second    50%      38%      43%

                                                    RAPID SEQUENTIAL

                                              Expert   Naive   Overall

AA   2900 K, 24 cd/[m.sup.2]    Left/First     59%      47%      51%
     2900 K, 24 cd/[m.sup.2]   Right/Second    41%      53%      49%
BB   2900 K, 30 cd/[m.sup.2]    Left/First     53%      57%      55%
     2900 K, 30 cd/[m.sup.2]   Right/Second    47%      43%      45%
CC   7200 K, 24 cd/[m.sup.2]    Left/First     29%      47%      40%
     7200 K, 24 cd/[m.sup.2]   Right/Second    71%      53%      60%
DD   7200 K, 30 cd/[m.sup.2]    Left/First     41%      53%      49%
     7200 K, 30 cd/[m.sup.2]   Right/Second    59%      47%      51%
    Combined Null Trials        Left/First     46%      51%      49%
    AA + BB + CC + DD          Right/Second    54%      49%      51%

TABLE 5. Summary of results for McNemar test analysis comparing side-
by-side and rapid-sequential methods. The values listed for the mixed
condition trials have been pooled for both permutations (for example,
AB and BA). The shaded and emboldened values represent results that
are statistically significant at a < 0.01.

                         Null Condition Trials

                 AA       BB       CC       DD

Z statistic    -0.378   0.000    -1.279   -1.706
  p value       0.705   1.000     0.201    0.088

                              Mixed Condition Trials

                 AB       AC       AD       BC       BD       CD

Z statistic   -1.414    1.761   -0.707    4.025    4.025   -1.414
  p value      0.157    0.078    0.480    0.000    0.004    0.157

TABLE 6. Results expressed as percentages for the mixed condition
trials, broken down by experimental method and class of subject.

                                      SIDE BY SIDE

                               Expert   Naive    Pooled

AB   2900 K, 24 cd/[m.sup.2]     3%       2%        2%
     2900 K, 30 cd/[m.sup.2]    97%      98%       98%
AC   2900 K, 24 cd/[m.sup.2]    26%      32%       30%
     7200 K, 24 cd/[m.sup.2]    74%      68%       70%
AD   2900 K, 24 cd/[m.sup.2]     9%       5%        6%
     7200 K, 30 cd/[m.sup.2]    91%      95%       94%
BC   2900 K, 30 cd/[m.sup.2]    88%      70%       77%
     7200 K, 24 cd/[m.sup.2]    12%      30%       23%
BD   2900 K, 30 cd/[m.sup.2]    35%      32%       33%
     7200 K, 30 cd/[m.sup.2]    65%      68%       67%
CD   7200 K, 24 cd/[m.sup.2]     3%       2%        2%
     7200 K, 30 cd/[m.sup.2]    97%      98%       98%

                                    RAPID SEQUENTIAL

                               Expert   Naive    Pooled

AB   2900 K, 24 cd/[m.sup.2]     0%       0%        0%
     2900 K, 30 cd/[m.sup.2]    100%     100%     100%
AC   2900 K, 24 cd/[m.sup.2]     56%      33%      41%
     7200 K, 24 cd/[m.sup.2]     44%      67%      59%
AD   2900 K, 24 cd/[m.sup.2]      3%       5%       4%
     7200 K, 30 cd/[m.sup.2]     97%      95%      96%
BC   2900 K, 30 cd/[m.sup.2]     97%      95%      96%
     7200 K, 24 cd/[m.sup.2]      3%       5%       4%
BD   2900 K, 30 cd/[m.sup.2]     59%      48%      52%
     7200 K, 30 cd/[m.sup.2]     41%      52%      48%
CD   7200 K, 24 cd/[m.sup.2]      0%       0%       0%
     7200 K, 30 cd/[m.sup.2]    100%     100%     100%

(1) The values listed are the sum of the combination listed and its

Fig. 5. Example McNemar test matrix for comparing the sideby-side and
rapid-sequential results for the AC combination (z = 1.761, p =
0.078). Results for all 10 matrices are summarized in Table 5.

                           Side-By-        Row Sum

                           A     C

Rapid            A        14    25           39
Sequential       C        14    41           55

             Column Sum   28    66           94
                                         Overall Sum

Fig. 6. Summary tables for the VSRS tests [Dunn-Rankin et al. 2004].
The shaded cells with red text are for statistically different pairs.
The same conclusions are drawn with either experimental method. The
critical value is 31.2 at [alpha] 0.05 and 39.0 at [alpha] 0.01.


               A       B       C       D

      Ri      18     97.5     45     121.5
A     18      --
B    97.5    79.5     --
C     45     27.0    52.5     --
D    121.5   103.5   24.0    76.5     --


               A       B       C       D
      Ri     21.5    116.5   29.5    114.5
A    21.5     --
B    116.5   95.0     --
C    29.5     8.0    87.0     --
D    114.5   93.0     2.0    85.0     --
COPYRIGHT 2009 Illuminating Engineering Society
No portion of this article can be reproduced without the express written permission from the copyright holder.
Copyright 2009 Gale, Cengage Learning. All rights reserved.

Article Details
Printer friendly Cite/link Email Feedback
Author:Houser, K.W.; Fotios, S.A.; Royer, M.P.
Article Type:Report
Geographic Code:1USA
Date:Oct 1, 2009
Previous Article:Reflections on the Last One Hundred Years of Lighting in Great Britain.
Next Article:A methodology for locating the maximum vertical illuminance in street lighting.

Terms of use | Privacy policy | Copyright © 2018 Farlex, Inc. | Feedback | For webmasters