Applying an analytic writing rubric to children's hypermedia "Narratives".

Abstract

In an effort designed to guide and improve the assessment of a newly developed writing environment, the reliability and developmental and concurrent validity of a previously validated rubric developed for pen-and-paper-created narratives, Writing What You Read (WWYR), was determined when applied to hypermedia-authored narratives of children in second and third grade. Children (n = 60) from four intact classrooms produced hypermedia narratives (text, audio, graphic, and video elements) over a 4-month period in a school-based computer laboratory. Raters (n = 5) with knowledge of the teaching of process writing and use of hypermedia software judged the hypermedia narrative productions. Raters judged all students' (n = 60) hypermedia narrative productions individually without resolving differences through discussion. Two analyses were used to determine reliability: percentages of agreement and Pearson correlations. Percentages of agreement for the WWYR rubric averaged across 10 pairs of raters found high percentages of agreement among raters (.70 for [+ or -] 0 and .99 for [+ or -] 1). Pearson correlations averaged across 10 pairs of raters found acceptable interrater reliability for four of the five subscales. For the five subscales (Theme, Character, Setting, Plot, and Communication), the r values were .59, .55, .49, .50, and .50, respectively. Developmental validity of the WWYR scores was examined with one-way MANOVA to evaluate the WWYR scores of children grouped as low, medium, or high ability based on their Iowa Test of Basic Skills (ITBS) National Percentile Rank for Literacy Skill. Evidence for the developmental validity of the WWYR scores was supported across the three ability groups, F(2, 36) = 2.59, p <. 01. Concurrent validity was examined through correlational analysis between students' mean WWYR score and ITBS score. Scores from the two measures were positively correlated, r = .83, p < .01, providing evidence of the sensitivity of the WWYR assessment to measure the developmental literacy competency of the third-grade students. Results support teachers' use of a validated rubric developed for pen-and-paper-created narratives applied to hypermedia narratives, despite additional visual and audio narrative elements inherent to hypermedia. Implications for literacy teaching and learning with hypermedia address core questions about the similarities and differences between written textual expression and visual and verbally recorded expression. A revised and expanded WWYR rubric is proposed to begin to address these core questions generated by teachers' use of the hypermedia writing environment.

Introduction

The very notion of "writing" is increasingly being transformed by new digital computer technologies in society, homes, businesses, and schools. Today, one's ability to represent thought electronically is more important than ever. The extreme pace of change in society dictates that early childhood educators consider how to facilitate children in learning to express themselves digitally in computer environments in developmentally appropriate ways. This article addresses the curriculum, instruction, and assessment of children's written expression in hypermedia, a computer environment that supports text, audio, video, and graphics. The hypermedia curriculum and instruction are described, and a technical analysis of a proposed assessment is applied and discussed.

Emerging Trends

Prior research has addressed the reliability and validity of a narrative rubric useful for both teaching and learning (value) and large-scale (utility) literacy assessment. As literacy curriculum and instruction continue to transform to include new computer-based learning environments, researchers must correspondingly develop and technically evaluate assessments for the new environments. Hypermedia represents a powerful electronic environment through which literacy expression can be facilitated by the manipulation of text, graphics, audio, and video elements, and this technology and integration are reflected in the new literacy K-12 standards (IRA and NCTE, 2001). The qualities of hypermedia that support higher-level cognitive processes such as synthesis, organization, evaluation, and reflexivity have been well documented (Yang, 1996; Mott, Sumrall, & Hodges, 1997). However, there is a glaring absence of ways to reliably and validly assess students' hypermedia products. To address this absence, a narrative process writing curriculum and instruction environment was merged with hypermedia. These terms are defined as follows:

* Process writing curriculum and instruction: writing using discrete stages (brainstorming, drafting, conferencing, revising, editing, and publishing) administered via "mini-lessons" applied to address the needs of the individual writers in whole-group instruction. See Graves (1983) for a detailed description.

* Hypermedia: Hypermedia comprises two main components: (1) hyper: the ability to program electronic links, or hyperconnections, to connect information to any other Internet-based source or simply to link locally to a hard drive or diskette, and (2) media: the ability to manipulate multiple meaning-based symbol systems representing a variety of sources--text, graphics, audio, and video clips.

This article addresses the reliability and developmental and concurrent validity of a previously validated narrative writing rubric, Writing What You Read (WWYR) (Wolf & Gearhart, 1994; Novak, Herman, & Gearhart, 1996), applied to hypermedia narratives created by students in grades 2 and 3 (see Table 1). The technical qualities of the process-oriented classroom rubric, valuable for teaching and learning on a day-to-day basis, are linked to the utility of the narrative rubric for measuring elementary students' literacy competencies as identified via a validated large-scale instrument, the Iowa Test of Basic Skills--Literacy Competency. Thus, five main issues are addressed relating to rater judgments of elementary students' hypermedia narratives (stories with text, graphics, audio, and video elements authored using HyperStudio hypermedia software):

* Reliability of interrater judgments is examined using the WWYR of hypermedia narrative quality.

* Developmental validity of the WWYR scores is examined with one-way MANOVA used to evaluate the WWYR scores of students grouped as low, medium, or high ability based on their Iowa Test of Basic Skills (ITBS) National Percentile Rank for Literacy Skill.

* Concurrent validity is examined through correlational analysis of students' mean WWYR scores and ITBS scores.

* Value of the WWYR for use as a teaching tool is summarized through reviewing the genesis of the rubric as demonstrated in its path from creation for pen-and-paper narratives to hypermedia narratives as evaluated in multiple studies.

* Utility of the WWYR applied to students' hypermedia narratives is addressed to reveal possible alignment between the innovative curriculum and instruction addressed in the current study with ITBS- and WWYR-identified literacy levels.

Research Components

Methods

Children (n = 60) from four intact classrooms (two second-grade and two third-grade classrooms) produced hypermedia narratives over a 4-month period in a school-based computer laboratory equipped with 10 Windows-based microcomputers. Rater/(Teachers) (n = 5) with knowledge of the teaching of process writing and use of hypermedia software judged the hypermedia narrative productions. An interactive hypermedia software tutorial program was developed and used to train the teachers (n = 4) in the implementation of process writing techniques in conjunction with the use of hypermedia features as part of their elementary curriculum. Raters participated in a 3-hour training and rating session in a university computer laboratory equipped with five Power Macintosh microcomputers. Raters judged all students' (n = 60) hypermedia narrative productions individually without resolving differences through discussion.

Materials

The WWYR rubric shown in Table 1 contained five evaluative scales designed to assess students' developing competencies in narrative writing: Theme, Character, Setting, Plot, and Communication. The vertical analytical evaluative scales (1-6 for each competency) were designed to enable teachers to make instructional decisions on specific narrative components needing reinforcement and were not intended as a method for assigning a numerical value to a narrative. Teachers merely had to shade a box in the rubric to denote where a child's narrative was along each competency. The ITBS (Linn & Wilson, 1990) Form J was used as a basic battery for grades K-9 and included language skills directly related to writing: word analysis, vocabulary, spelling, and reading comprehension. Reliability coefficients for Form J ranged from .70-.90 for the language skills components. Additionally, the ITBS met high standards of overall technical quality and has been a widely accepted standardized measure of cognitive skill.

Hypermedia Narratives

HyperStudio Presentation Software (Wagner, 1997-2001) was used to support the children in expressing themselves with text, audio, video, and graphic elements in their narrative productions (see Figures 1-4). Children composed on paper and computer with teachers structuring the process with Writing Workshop (Graves, 1983), a method for organizing writing into discrete and recursive stages. These stages are (1) brainstorming, (2) revising, (3) drafting, (4) peer conferencing, (5) editing, and (6) publishing (see Mott & Klomes, 2001, for a detailed description of a program similar to the program addressed in the current study).

[FIGURES 1-4 OMITTED]

HyperStudio was selected for this project for a number of reasons: (1) it supports high-end features such as video and animation; (2) it is relatively easy to use, even by young children (grades 1-3); (3) it contains a flexible interface; and (4) it is the most widely used multimedia/hypermedia platform in elementary education with over 100,000 users (Wagner, 1997-2001). HyperStudio contains several programming features that support children's hypermedia programming (see Figures 5-10). Programming instruction (and writing workshop) occurred through mini-lessons over a 4-month period. Figures 5 through 10 contain examples from a mini-lesson.

[FIGURES 5-10 OMITTED]

Children inserted audio clips, programmed via button-creation, to enrich their text. Audio clips were recorded by the author or downloaded from HyperStudio for special effects such as an alarm clock sound.

Video clips were inserted, programmed via button-creation, to supplement the narrative. One example was a video clip of Mars used in a science fiction narrative by a third-grade student.

Reliability

Percentages of Agreement

Percentages of agreement for the WWYR rubric averaged across 10 pairs of raters found high percentages of agreement among raters (.70 for [+ or -] 0 and .99 for [+ or -] 1) (see Table 2). The [+ or -] 0 and [+ or -] 1 percentages of agreement across 10 pairs of raters were higher than the [+ or -] 0 and [+ or -] 1 agreement levels found in both the Gearhart, Herman, Novack, and Wolf (1995) and Novak, Herman, and Gearhart (1996) WWYR reliability studies (compared in Table 3). The high percentages of agreement found in this study may be attributed to the raters' use of only the first three WWYR rubric evaluative subscale levels. The WWYR rubric contains six subscale levels that are developmentally sequenced according to the varied writing competencies of students in grades K-6. Because students in this study were in second and third grade, raters typically applied only levels 1, 2, and 3 out of five total levels. This narrow range of values independently applied by raters limited the number of choices. Hence, high percentages of agreement between raters would be expected based on the limited number of scale levels used.

The percentages of agreement that were revealed in the current study, although higher than those found in the Gearhart, Herman, Novak, and Wolf (1995) study, should be considered descriptive information. Gearhart et al. remarked that percentages of agreement found for the WWYR should not be interpreted as "strong evidence of reliability" (p. 224). Rather, percentages of agreement can be used to help identify the existence of widely varying patterns of rater judgments, both across WWYR subscales and across all rater pairs. No such widely varying patterns were found in the current study. The limitations of analyses involving percentages of agreement analysis were discussed by Abedi (1994), who argued that, although percentages of agreement can reveal the existence of widely varying patterns of agreement among raters, they can also yield different results from other analyses such as Pearson correlations.

Pearson Correlations

Pearson correlations were used to further examine reliability of rater judgments. Pearson correlations averaged across 10 pairs of raters found acceptable interrater reliability for four of the five subscales. For the five subscales, Theme, Character, Setting, Plot, and Communication, the r values were .59, .55, .49, .50 and .50. Table 4 contains the results of the Pearson correlations for WWYR rubric scoring across all rater pairs for the current study and for the Gearhart et al. (1995) study. An examination of correlation scores for hypermedia narrative productions revealed that interrater reliability for four of the five WWYR subscales (Theme, Character, Setting, and Plot) was comparable to the interrater reliability levels found in the Gearhart et al. (1995) WWYR reliability study for pen-and-paper-created narratives. For the fifth subscale (Communication), however, the correlational coefficient value was .16 higher in the Gearhart et al. study than in the current study. Despite the lower value found in the current study for Communication, Gearhart et al. related that an average subscale correlation higher than .50 could be considered adequate for a rubric such as the WWYR. Table 5 summarizes the comparison of WWYR correlations across all subscales for the current study and those found in the literature (Gearhart et al., 1995; Novak et al., 1996).

The WWYR correlations observed in this study, as well as in the Gearhart et al. (1995) study, demonstrated that ratings were highly correlated across all subscales. The r values were low in this study as well as in the Gearhart et al. (1995) and Novak et al. (1996) studies. However, set guidelines for what is an acceptable level of interrater reliability do not exist. Nonetheless, both Gearhart et al. and Novak et al., whose studies analyzed holistic scores derived from the combined r values of Theme, Character, Plot, Setting, and Communication, argued that r values that fell within the .50 to .70 range were acceptable for analytic writing rubrics. In the current study, the interrater reliability for Theme, Character, Plot, and Communication subscales fell within the .50 and .59 range, but the level of interrater reliability (r = .49) for the Setting subscale did not. It is important to note that, in the Gearhart et al. (1995) study, a low coefficient value for the subscale of Setting was also found (r = .48). (1) The acceptable interrater reliabilities for Theme, Character, Plot, and Communication in this study were comparable to the acceptable levels found in the Gearhart et al. (1995) study, and the r values for the Setting subscale in both this study and the Gearhart et al. (1995) study were not acceptable (albeit by large-scale standards). It is important to note that interrater reliability levels for Theme, Character, and Plot in this study may have been lower than the r values in the Gearhart et al. (1995) study because the researcher applied more stringent rating procedures in this study. Raters in the Gearhart et al. (1995) study were permitted to resolve differences greater than one scale point through discussion, whereas raters in this study were not permitted to resolve differences. In the current study, all ratings were included in the final data set.

The highly correlated rater judgments, along all five WWYR subscales for the current study and for the Gearhart et al. (1995) study, provided further evidence of the reliability of WWYR raters' judgments. The true function of a writing rubric is that it "enables raters to apply standard criteria in making judgments about the quality of students' work" (Abedi, 1994, p. 8). Gearhart et al. (1995), Novak et al. (1996), and Abedi (1994) argued that highly correlated scores across rubric subscales can be viewed as a positive indication that raters' judgments are being consistently applied.

Validity

Developmental Validity

Developmental validity of WWYR scores was examined via one-way MANOVA conducted on the low-, medium-, and high-ability vectors of WWYR subscale scores. The assumption for this analysis was based upon the technical qualities of the ITBS--Literacy measure to delineate the developmental literacy levels of the children. Results indicated a statistically significant difference between the three ability groups (F(2, 36) = 2.59, p = .01). Table 6 provides descriptive statistics, and Table 7 provides an additional summary of these results across each of the five WWYR subscales.

Tukey HSD tests were conducted on the mean vector scores of the three ability groups for all five WWYR subscales to follow up these results. For the WWYR subscale of Theme, low-ability students (M = 2.31, SD = .62) received lower scores than both medium-ability students (M = 2.80, SD = .28) and high-ability students (M = 2.86, SD = .31). For the WWYR subscales of Character, Setting, Plot, and Communication, all differences were significant. Therefore, low-ability students' scores were significantly lower than medium-ability students' scores, which were significantly lower than the high-ability students' scores. The significant differences revealed between low, medium, and high ITBS groups and the WWYR subscale scores provided evidence for the sensitivity of the WWYR to measure the development of students' hypermedia/writing competence. The significant results of the one-way MANOVA suggest that raters' judgments were evaluating students' skills as message producers (communication through text and other meaning-based symbol systems). The one-way MANOVA did not yield results that would enable the researcher to describe the degree of relatedness of raters' WWYR judgments and students' ITBS scores. In order to describe the relationship between WWYR scores and literacy skill (as measured by the ITBS), additional correlational analyses were conducted.

Concurrent Validity

The observed Pearson r correlation revealed a positive relationship between students' average WWYR score (averaged across the subscales of Theme, Character, Setting, Plot, and Communication) and their ITBS National Percentile Rank (literacy skills score), r = .83, p > .001. The positive correlation (r = .83) between students' WWYR scores and ITBS scores revealed in this analysis provided evidence for the concurrent validity (the degree to which test scores are related to the scores on an already established test) of WWYR raters' judgments of hypermedia productions. According to Messick (1992), establishment of the concurrent validity of a measure can be a stepping-stone toward establishment of the content-related validity (the degree to which scores evaluate the specific domain they were designed to evaluate) of a measure. Hence, the developmental and concurrent validities established for WWYR raters' judgments of hypermedia productions represented an important initial attempt toward eventually establishing the content-related validity of the WWYR when applied to hypermedia productions.

The strong, positive, linear relationship between ITBS literacy skill scores and WWYR rater judgments of hypermedia productions indicated that the hypermedia writing curriculum used in the current study involved literacy-based activities. The fact that students in this study expressed themselves through hypermedia features, and not solely through text, indicated that students' literacy skill can be enhanced through student expression via hypermedia and multimedia features. Table 8 provides additional information on the students' utilization of the hypermedia features used in their writing. This finding supported the claims of Daiute and Morse (1994), who observed that students who engaged in hypermedia writing developed literacy skill through the manipulation of text and other symbols. A weakness of the developmental and concurrent validity analyses was that evidence for obtaining the degree to which rater judgments of students' hypermedia productions evaluated textual features as well as textual and other hypermedia features (audio, hypermedia links, graphics, etc.) could not be determined.

Assessment Value and Utility

The results of this study suggest several important implications for the assessment of students' hypermedia products. Having a reliable and valid assessment for evaluating students' hypermedia-based writing serves two general purposes: (1) to enhance classroom instruction (value), and (2) to inform, to a lesser extent, educational policy (utility). The positive results yielded in this study concerning the reliability and validity of the WWYR provide an avenue for teachers to accurately and consistently evaluate their students' hypermedia narrative productions by applying the WWYR assessment. The value of an assessment is the degree to which it enhances teacher instruction by linking teachers' comments to their instructional objectives (Wolf & Gearhart, 1994). Therefore, in order for teachers to properly evaluate both student outcomes and the instructional effectiveness of their hypermedia/writing curricula, it is useful for all educators to apply a reliable and valid instrument. Furthermore, the positive correlation between the students' ITBS literacy skill score and WWYR average score for hypermedia productions indicated that students who were engaged in a hypermedia/writing curriculum improved their literacy skills.

Table 1
Writing What You Read Narrative Rubric (Wolf & Gearhart, 1994)

        Theme                 Character                Setting

Explicit-Implicit       Flat-Round              Backdrop-Essential

Didactic-               Static-Dynamic          Simple-Multifunctional
Revealing

1: Not present or not   1: One or two flat,     1: Backdrop setting
developed through       static characters,      with little or no
other narrative         with little             indication of time or
elements                relationship between    place ("There was a
                        characters              little girl. She liked
                                                candy.")

2: Meaning centered     2: Some  rounding,      2: Skeletal indication
in a series of list-    usually in physical     of  time and place
like statements ("I     description;            often held in past
like my mom. And I      relationship between    time ("Once there
like my dad. And I      characters is action    was ...");  little
like my ...")           driven                  relationship to other
                                                narrative elements

3: Beginning            3: Continued rounding   3: Beginning
statement of theme,     in physical             relationship between
often explicit and      description,            setting and other
didactic ("The mean     particularly            narrative elements
witch chased the        stereotypical           (futuristic setting to
children and she        features ("wart on      accommodate aliens
shouldn't have done     the end of her nose")   and spaceships)
that.")

4: Beginning            4: Beginning insights   4: Setting becomes
revelation of theme     into motivation and     more essential to the
on both explicit and    intention that drive    development of the
implicit levels         the feeling and         story in explicit
through the more        action of main          ways: characters may
subtle things           characters often        remark on the setting,
characters say and do   through limited         or the time and place
                        omniscient point of     may be integral to the
                        view                    plot

5: Beginning use of     5: Further rounding     5: Setting may serve
secondary themes,       (in feeling and         more than one
often tied to           motivation); dynamic    function, and the
overarching theme,      features appear in      relationship between
but sometimes           central characters      functions is more
tangential              and between             implicit and symbolic
                        characters

6: Overarching theme    6: Round, dynamic       6: Setting fully
multilayered and        major characters        integrated with the
complex; secondary      through rich            characters, action,
themes integrally       description of          and theme
related to the          affect,  intention,
primary themes          and motivation

        Theme                   Plot              Communication

Explicit-Implicit       Simple-Complex         Context-bound

Didactic-               Static-Conflict        Literal-Symbolic
Revealing

1: Not present or not   1: One or two events   1: Writing bound to
developed through       with little or no      context (You have to
other narrative         conflict ("Once        be there) and often
elements                there was a cat. The   dependent on drawing
                        cat liked milk.")      and talk to clarify
                                               the meaning

2: Meaning centered     2: Beginning           2: Beginning
in a series of list-    sequence of events,    awareness of reader
like statements ("I     but out-of-sync        considerations;
like my mom. And I      occurrences; events    straightforward
like my dad. And I      without problem;       style and tone
like my ...")           problem without        focused on getting
                        resolution             the information out

3: Beginning            3: Single, linear      3: Writer begins to
statement of theme,     episode with clear     make sense of
often explicit and      beginning, middle,     explanations and
didactic ("The mean     and end; the episode   transitions
witch chased the        contains a problem,    ("because" and
children and she        emotional response,    so); literal style
shouldn't have done     action, and outcome    centers on
that.")                                        description

4: Beginning            4: Plot increases in   4: Increased
revelation of theme     complexity with more   information and
on both explicit and    than one episode;      explanation for the
implicit levels         each episode           reader (linking
through the more        contains problem,      ideas as well as
subtle things           emotional response,    episodes); words
characters say and do   action, and outcome    more carefully
                                               selected to suit the
                                               narrative's purpose

5: Beginning use of     5: Stronger            5: Some
secondary themes,       relationships          experimentation with
often tied to           between episodes       symbolism
overarching theme,      (with resolution in    (particularly
but sometimes           one leading to a       figurative
tangential              problem in the         language), which
                        next)                  shows reader
                                               considerations

6: Overarching theme    6: Overarching         6: Careful crafting
multilayered and        problem and            of choices of story
complex; secondary      resolution supported   structure as well as
themes integrally       by multiple episodes   vocabulary
related to the                                 demonstrate
primary themes                                 considerate
                                               orchestration of all
                                               resources

Table 2
Percentage of Agreement for all Five Subscale of the WWYR Rubric

WWYR Subscale   [+ or -] 0   [+ or -] 1   n

Theme           .70          .96          60
Character       .78          .99          60
Plot            .73          .99          60
Setting         .67          .99          60
Communication   .68          .99          60

Table 3
Percentage of Agreement for the WWYR Rubric Averaged across
All Subscales

WWYR Rating Material and Grade           [+ or -] 0   [+ or -] 1   n

Hypermedia Narratives: Grades 2-3        .71          .98          60
Mott & Sumrall (1998)

Pen-and-Paper Narratives: Grades 1-6     .46          .96          120
Gearhart, Herman, Novak, & Wolf (1995)

Collections of Pen-and-Paper             .25          .94          52
Narratives: Grades 2-5
Novak, Herman, & Gearhart (1996)

Table 4
Average Pearson Correlations for WWYR Rubric Scoring across
10 Pairs of Raters

WWYR Rating Material
and Grade                       Theme   Character   Setting

Hypermedia Narratives:     r    .59     .55         .49
Grades 2-3

Mott & Sumrall, 1998       SD   .25     .31         .25
(n = 60)

Pen-and-Paper Narratives   r    .64     .59         .48
Grades 1-6

Gearhart et al. (1995)     SD   .10     .10         .12
(n = 120)

WWYR Rating Material
and Grade                  Plot   Communication

Hypermedia Narratives:     .50    .50
Grades 2-3

Mott & Sumrall, 1998       .29    .24
(n = 60)

Pen-and-Paper Narratives   .57    .66
Grades 1-6

Gearhart et al. (1995)     .14    .10
(n = 120)

Table 5
Comparison of WWYR Subscale Correlations: Pen-and-Pen Narratives
versus Hypermedia

Subscale                   Theme   Character   Setting

Mott & Sumrall (1998)          Samples (n = 60)
Hypermedia

Theme                      --      .86 *       .79 *

Character                  --      --          .74 *

Setting                    --      --          --

Plot                       --      --          --

Communication              --      --          --

Gearhart et al. (1995)
Pen-and-Paper Narratives       Samples (n = 120)

Theme                      --      .83 *       .81 *

Character                  --      --          .82 *

Setting                    --      --          --

Plot                       --      --          --

Communication              --      --          --

Subscale                   Plot    Communication

Mott & Sumrall (1998)      Samples (n = 60)
Hypermedia

Theme                      .79 *   .73 *

Character                  .74 *   .76 *

Setting                    .75 *   .68 *

Plot                       --      .78 *

Communication              --      --

Gearhart et al. (1995)
Pen-and-Paper Narratives   Samples (n = 120)

Theme                      .83 *   .86 *

Character                  .87 *   .86 *

Setting                    .73 *   .86 *

Plot                       --      .85 *

Communication              --      --

Note: * p < .001.

Table 6
Descriptive Statistics: WWYR Subscales across ITBS Ability Level

                                  Dependent Variables

Statistics                   n    Theme   Character   Setting

Mean Vectors

ITBS Ability Level

  Low                        13   2.31    1.80        1.96

  Medium                     13   2.80    2.34        2.32

  High                       14   2.86    2.60        2.66

Variance-Covariance Matrix

  Theme                           .21     .15         .19

  Character                       --      .23         .13

  Setting                         --      --          .19

  Plot                            --      --          --

  Communication                   --      --          --

                             Dependent Variables

Statistics                   Plot   Communication

Mean Vectors

ITBS Ability Level

  Low                        2.10   2.14

  Medium                     2.66   2.52

  High                       2.74   2.77

Variance-Covariance Matrix

  Theme                      .13    .11

  Character                  .15    .13

  Setting                    .14    .11

  Plot                       .19    .14

  Communication              --     .16

Table 7
Mean WWYR Subscale Scores for Low-, Medium-, and High-Ability
Grade-3 Students

                ITBS NPR/Literacy   Mean
WWYR Subscale   Category            Score   SD    n    F       Sig

Theme           Low                 2.31    .62   16   6.19    .01
                Medium              2.80    .28   10
                High                2.86    .31   13

Character       Low                 1.80    .50   16   10.77   .01
                Medium              2.34    .38   10
                High                2.60    .51   13

Setting         Low                 1.96    .42   16   9.34    .01
                Medium              2.32    .56   19
                High                2.66    .34   13

Plot            Low                 2.10    .54   16   9.28    .01
                Medium              2.66    .34   10
                High                2.74    .34   13

Communication   Low                 2.14    .47   16   9.20    .01
                Medium              2.52    .34   10
                High                2.77    .35   13

Table 8
Frequency of HyperStudio Multimedia Features Used in Students'
Hypermedia Narrative Productions*

                Hypermedia/Multimedia Feature

             Button with   Button   Button
Grade        Hyperdemia    with     with     Text
Level        Link          Audio    Video    Box

2 (n = 20)   100%          91%      0%       100%
3 (n = 40)   100%          100%     5%       100%

              Hypermedia/Multimedia Feature

                                  Graphics
Grade        Graphics   Scanned   Objects
Level        Text       Art       (Clip Art)

2 (n = 20)   45%        96%       92%
3 (n = 40)   64%        100%      100%

* Note. In three out of the four classrooms where
hypermedia/writing occurred, students' use of
hypermedia/multimedia features was controlled by the teachers.

Table 9
Hypermedia-WWYR

Hypermedia    Theme                       Character

Elements      Explicit-Implicit           Flat-Round

              Didactic-Revealing          Static-Dynamic

-Text-        1: Not present or not       1: One or two flat,
-Hypertext-   developed through other     static characters, with
-Graphic-     narrative elements          little relationship
-Audio-                                   between characters
-Video-

-Text-        2: Meaning centered in a    2: Some rounding, usually
-Hypertext-   series of list-like         in physical description;
-Graphic-     statements ("I like my      relationship between
-Audio-       mom. And I like my dad.     characters is action
-Video-       And I like my ...")         driven

-Text-        3: Beginning statement of   3: Continued rounding in
-Hypertext-   theme, often explicit and   physical description,
-Graphic-     didactic ("The mean         particularly
-Audio-       witch chased the children   stereotypical features
-Video-       and she shouldn't have      ("wart on the end of her
              done that.")                nose")

-Text-        4: Beginning revelation     4: Beginning insights
-Hypertext-   of theme on both explicit   into motivation and
-Graphic-     and implicit levels         intention that drives the
-Audio-       through the more subtle     feeling and action of
-Video-       things characters say and   main characters often
              do                          through limited
                                          omniscient point of view

-Text-        5: Beginning use of         5: Further rounding (in
-Hypertext-   secondary themes, often     feeling and motivation);
-Graphic-     tied to overarching         dynamic features appear
-Audio-       theme, but sometimes        in central characters and
-Video-       tangential                  between characters

-Text-        6: Overarching theme        6: Round, dynamic major
-Hypertext-   multilayered and complex;   characters through rich
-Graphic-     secondary themes            description of affect,
-Audio-       integrally related to the   intention, and motivation
-Video-       primary themes

Hypermedia    Setting                     Plot

Elements      Backdrop-Essential          Simple-Complex

              Simple-Multifunctional      Static-Conflict

-Text-        1: Backdrop setting with    1: One or two events with
-Hypertext-   little or no                little or no conflict
-Graphic-     indication of time or       ("Once there was a cat.
-Audio-       place ("There was a         The cat liked milk.")
-Video-       little girl. She liked
              candy.")

-Text-        2: Skeletal indication of   2: Beginning sequence of
-Hypertext-   time and place often held   events, but out-of-sync
-Graphic-     in past time ("Once there   occurrences; events
-Audio-       was ..."); little           without problem; problem
-Video-       relationship to other       without resolution
              narrative elements

-Text-        3: Beginning relationship   3: Single, linear episode
-Hypertext-   between setting and other   with clear beginning,
-Graphic-     narrative elements          middle, and end; the
-Audio-       (futuristic setting to      episode contains a
-Video-       accommodate aliens and      problem, emotional
              spaceships)                 response, action, and
                                          outcome

-Text-        4: Setting becomes more     4: Plot increases in
-Hypertext-   essential to the            complexity with more
-Graphic-     development of the story    than one episode; each
-Audio-       in explicit ways:           episode contains problem,
-Video-       characters may remark on    emotional response,
              the setting, or the time    action, and outcome
              and place may be integral
              to the plot

-Text-        5: Setting may serve more   5: Stronger relationships
-Hypertext-   than one function, and      between episodes (with
-Graphic-     the relationship between    resolution in one leading
-Audio-       functions is more           to a problem in the next)
-Video-       implicit and symbolic

-Text-        6: Setting fully            6: Overarching problem
-Hypertext-   integrated with the         and resolution supported
-Graphic-     characters, action, and     by multiple episodes
-Audio-       theme
-Video-

Hypermedia    Communication

Elements      Context-bound

              Literal-Symbolic

-Text-        1: Writing bound to
-Hypertext-   context (You have to be
-Graphic-     there) and often
-Audio-       dependent on drawing and
-Video-       talk to clarify the
              meaning

-Text-        2: Beginning awareness of
-Hypertext-   reader considerations;
-Graphic-     straightforward style and
-Audio-       tone focused on getting
-Video-       the information out

-Text-        3: Writer begins to make
-Hypertext-   sense of explanations and
-Graphic-     transitions ("because"
-Audio-       and "so"); literal style
-Video-       centers on description

-Text-        4: Increased information
-Hypertext-   and explanation for the
-Graphic-     reader (linking ideas as
-Audio-       well as episodes); words
-Video-       more carefully selected
              to suit the narrative's
              purpose

-Text-        5: Some experimentation
-Hypertext-   with symbolism
-Graphic-     (particularly
-Audio-       figurative language),
-Video-       which shows reader
              considerations

-Text-        6: Careful crafting of
-Hypertext-   choices of story
-Graphic-     structure as well as
-Audio-       vocabulary demonstrate
-Video-       considerate orchestration
              of all resources

Note

(1.) A note on correlations averaged across raters: (1) A relatively small number of raters (n = 5) were used in this study and the Gearhart, Herman, Novak, and Wolf (1995) study, which may have contributed to the lower r values across all subscales. The attenuation of correlational coefficients may be another explanation for the low levels of interrater reliability (Gay, 1996). Accordingly, coefficients tend to be lower when a restricted range of values is utilized (e.g., the narrow range of only 3 out of a possible 6 WWYR subscale levels utilized by raters in this study). Thus, the more narrow the range of scores utilized by raters, the lower the coefficients. On the other hand, Gearhart et al. argued that if the number of raters was statistically increased five-fold, r values in the .50 to .60 range for Theme, Character, Setting, Plot, and Communication would be changed to .87, .89, .82, .86, and .89, respectively. Gearhart et al. used decision-study (multiplication of sample scores and aggregation of the results) coefficients to determine the number of raters needed to attain high reliability coefficients. (2) The r value for the Communication subscale in this study was considerably lower than the r value in the Gearhart et al. study (r = .50 versus .66). This sizable disparity in the level of interrater reliability may have been the result of the contrasting features of hypermedia-created narrative productions versus pen-and-paper-created narratives. The Communication subscale text primarily consisted of evaluative prompts designed to guide teachers in the assessment of writing style (see Table 1). Perhaps, in the current study, raters solely viewed textual features at the expense of the hypermedia features of graphics, sounds, buttons, and scanned art.

References

Abedi, Jamal. (1994). Final report of achievement, section A (report to the National Center for Education Statistics, contract no. RS90159001). Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing.

Daiute, Colette, & Morse, Frances. (1994). Access to knowledge and expression: Multimedia writing tools for students with diverse needs and strengths. Journal of Special Education Technology, 12(3), 221-256. EJ 492 959.

Gay, Lorrie R. (1996). Educational research: Competencies for analysis and application (5th ed.). Englewood Cliffs, NJ: Prentice-Hall.

Gearhart, Maryl; Herman, Joan L.; Novak, John R.; & Wolf, Shelby A. (1995). Toward the instructional utility of large-scale writing assessment: Validation of a new narrative rubric. Assessing Writing, 2(2), 207-242. EJ 562 433.

Graves, Donald. (1983). Writing: Teachers and children at work. Portsmouth, NH: Heinemann. ED 234 430.

International Reading Association (IRA) and National Council of Teachers of English (NCTE). (2001). Standards for the English language arts. Urbana, IL: NCTE; Newark, DE: IRA.

Linn, R., & Wilson, V. (1990). Review of the Iowa Test of Basic Skills form J. In Jane Close Conoley and James C. Impara (Eds.), Mental measurement yearbook (9th ed.). Lincoln: University of Nebraska Press.

Messick, Stephen. (1992). Validity of test interpretation and use. In M. Alkin (Ed.), Encyclopedia of education research (6th ed., pp. 1487-1495). New York: Macmillan.

Mott, Michael S., & Klomes, Jeanine. (2001). The synthesis of writing workshop and hypermedia-authoring: Grades 1-4. Early Childhood Research & Practice [Online], 3(2). Available: http://ecrp.uiuc.edu/v3n2/mott.html [2003, February 6]. ED 458 043.

Mott, Michael S., & Sumrall, William J. (1998). Scientists are presenters: Tech trek: Interactive media. Science Scope, 21(7), 42-45.

Mott, Michael S.; Sumrall, William J.; & Hodges, M. Lee. (1997, November). Process and computer-based elementary writing curriculum: A review of methods and assessments. Paper presented at the annual meeting of the Mid-South Educational Research Association, Memphis, TN. ED 415 255.

Novak, John R.; Herman, Joan L.; & Gearhart, Maryl. (1996). Establishing validity for performance-based assessments: An illustration for collections of student writing. Journal of Educational Research, 89(4), 220-233. EJ 528 634.

Wagner, Roger. (1997-2001). HyperStudio multimedia presentation software. Palo Alto, CA: Roger Wagner Publishing.

Wolf, Shelby A., & Gearhart, Maryl. (1994). Writing What You Read: A framework for narrative assessment. Language Arts, 71(6), 425-445. EJ 490 740.

Yang, S. C. (1996). A dynamic reading-linking-to-writing model for problem solving within a constructive hypermedia learning environment. Journal of Educational Multimedia and Hypermedia, 5(3/4), 263-283.

Dr. Michael Seth Mott is an assistant professor of literacy/early childhood education at Purdue University Calumet. Dr. Mott conducts research in literacy and technology integration and designs materials for teaching both literacy and foundations of education courses.

Dr. Michael Seth Mott

Assistant Professor of Literacy

Purdue University Calumet

2200 169th Street

Hammond, IN 46323

Email: [e-mail]

Cynthia Etsler is a visiting instructor in the School of Education at Purdue University Calumet. In addition to mentoring new faculty in the School of Education, Professor Etsler is designing digital courseware for courses in literacy and science methods for preservice and inservice teachers.

Deondre Drumgold is an administrator in Bank Street College's Computing and Information Services (CIS) Department where he provides computing, audiovisual, telecommunications, and networking support for the College's academic and administrative communities. This article has been accessed 6,944 times through April 1, 2005.