Evaluating online learners in applied psychology.

Abstract

Graduate learners receiving their degree training in applied psychology online learn not only information from their studies, but also critical interpersonal strategies for interacting with colleagues, peers, and clients. This article emphasizes the importance of using a critically reflective stance in online teaching in order to better engage and evaluate these learners on both of these important aspects of their training. Additionally, specific target behaviors for training and assessment of online graduate learners in applied psychology are suggested.

**********

There is always some anecdotal data that come with new territory. The descriptions provided here reflect the experience of this authorship group and its impressions of online graduate learners in applied psychology, representing a glimpse into the very exciting work happening in online course rooms. In this article, some general training and assessment needs of graduate learners in applied psychology are described. Additionally, the value of using a critically reflective teaching approach is emphasized to achieve successful transmission of the information and other skills necessary for work in this discipline.

Who is getting an applied graduate-level psychology education online?

Prospective learners in applied graduate-level psychology are evaluated for admission using specific criteria examining readiness for graduate school involving a combination of academic and personal characteristics. These include things such as the prospective learner's background and training in the specific content area, their interpersonal skills relative to the area of specialization in which they seek to earn a degree, and other key behaviors or indicators central to training in their specialization. In addition, they are evaluated for their specific readiness to do an online degree program because such programs require familiarity with Internet-based technologies.

Anecdotally, online graduate learners in applied psychology are very accepting of using the web as a resource, typically research or locate online learning institutions through the Internet, and tend to be professionals with field experience who are able to multitask. They typically work, have active lives, tend to be highly independent, and self-driven. Many have their own businesses, are managers or directors of business groups or other practices, or are faculty at smaller or more junior educational institutions seeking to upgrade their training. Some are even professionals who are abroad or in rural areas where they would otherwise be unable to seek graduate training. Overall, these learners bring professional experiences to their classes and contribute practical issues for discussion, enhancing the overall educational process they create and experience. It is also important to note that these are individuals who choose to be in an electronic environment where they are interacting with others from around the world. As such, ideas brought to the table for discussion in the electronic course rooms tend to be highly diverse and compelling. Views tend to be global, and learners are afforded the opportunity to dialogue and have ideas explained in detail when requested from sometimes very divergent points of view and experience. Thus this environment tends to be conducive to retaining learners who are reflective and accepting of perspectives and beliefs different than their own.

Evaluating learner performance in online classes

Both the content and process of learner work must be assessed. This entails assessing work products, as well as in-course behavior and relevant extra-class behavior. Work products might include evaluation of tests, take-home exams, written papers or reports, or other projects to be graded, much as in a traditional face-to-face teaching environment. With in-course behavior, this includes written contributions to interactions with peers and the instructor. Extra-class behavior focuses upon learner interaction with the university more generally, whether it is written, by phone, or in person. This includes how the learner acts on an individual level out of class with peers and/or with the instructor, as well as in interactions with the institution as a whole (Bullen, 1998; Dahlgren, 1998).

General assessment issues

Online teaching has many advantages for evaluating learner performance. One such advantage is that evaluation of learner knowledge, interaction, and skills are relatively easy because nearly all communications are kept as written records. This can be true of individual learner responses to assignments, learner to learner communications, and learner to instructor communications in class. It also applies to learner to administration communications within the university, and any general comments learners may make to the community of learners or faculty via discussion lists, or more generally in course room discussions. Having comments in writing does much to assist instructors and mentors in determining what the learner knows, how and when they respond to questions, writing capabilities, organizational capabilities in terms of how they structure their writing and responses, and their general communication style. It also allows instructors set objective criteria and use specific targets of professional writing style in everyday communications, potentially helping online graduate learners in psychology to advance their writing skills more quickly than they might in a traditional face-to-face program where writing is not as frequent or demanding.

In electronic course rooms for specific classes, the first few assignments provide a baseline of data that an instructor can use at any given time to determine learner progress. This keeps the instructor from having to rely on memory or from writing cumbersome notes about each learner. In addition, the instructor can still seek clarification about a learner's thoughts and obtain a written response. Other learners in the class may have the opportunity to read these interactions, benefiting from modeling and feedback to the extent that they monitor the ongoing discussion. And finally, specific issues around promptness of responses, writing skills, and organizational abilities can be examined and used as examples if feedback is necessary.

Evaluating learner process

In a traditional classroom there are four walls (Thomerson & Smith, 1996; Waghid, 1998) yet in a distance learning course room, there are no physical walls. Psychologically though, a wall having both pluses and minuses existing in the virtual course room is that of two-way anonymity. The fact that learners cannot be seen or see others, may affect and even change their behavior. Learners who are shy or inhibited, for instance, may appear more open, as their hesitancy to express verbally may translate into hesitation to express in writing. As a result, they may make shorter contributions to class discussions and possibly be seen as easier to engage, less opinionated, or somewhat less threatening to interact with than a peer who produces voluminous class contributions. And because typical non-verbal feedback is unavailable online, even if these shy or inhibited individuals do not like what their peers have to say, unless they actively engage in writing about this in the discussion, their peers will have no sense of this. Conversely, shyness and inhibition may disappear in an online context due to the social distance afforded by the online environment. And similarly, adaptively assertive individuals may become or appear more aggressive online. A lack of non-verbal feedback to let them know that their communication was experienced could prompt more frequent or intense communication by them in order to elicit a response from their peers. Or they might be verbally assertive, but when it comes to communicating in the written word, their writing might be experienced differently by their peers and possibly seen as aggressive if they are not adept at empathic, assertive writing.

Given these examples, the obvious missing component in a virtual environment is nonverbal face-to-face behavior, as virtual course rooms currently are composed of written text and imported still images (Saxena & Juhaura, 1999). Therefore, the challenge arises as to how the instructor and learners interpret the written word. In one sense this may remove some of the bias regarding appearance and other factors, but it also means instructors will base their view of learners basically on their writing and writing styles.

In each of the examples, another key point is that written and verbal expression skills are different skills. Learners who are otherwise able to express verbally may experience difficulty in portraying this same ability in a written format. This may be due to differences in verbal and written expressive abilities. But then writing skills also may be misinterpreted because of technical deficiencies. Learners less skilled in typing, for example, may likely have less detailed responses. And so, instructors in virtual environments must consider the quality of the responses, look beneath the process, and use care when evaluating learners who may have poorer technical-procedural skills (Bennett, 1998; McDonald & Gibson, 1998; Thomerson & Smith, 1996). Individual communication can be used if the instructor is in doubt or receives mixed messages from learners about any specific course room issue, allowing an opportunity for these things to be checked out before coming to a conclusion regarding the individual (Bullen, 1998; Dahlgren, 1998; Taylor, 1998). Instructors also may need to resort to other means of clarification of responses when they believe learners' performances are being hindered by their technical skills or are otherwise in trouble. They may need to inquire and/or follow up with telephone conversations with their learners, particularly those where the behaviors appear inappropriate, ambiguous, or persisting. And in an effort to avoid problems, instructors should be precise in their instructions to learners to create an atmosphere whereby learners can ask for clarification. And when feedback is given to learners, it should be done constructively and then followed up with monitoring and more feedback.

As instructors become more experienced with the virtual format of the distance learning course room, they will likely come to see the true behavior of the learner based on their overall interactions. Behavior tends to be consistent from situation to situation. Therefore, by observing learners' "writing" in more than one context--individually, in interactions with peers, the instructor, and the institution--it is likely that an accurate mental picture of the individual will develop (Dahlgren, 1998; Pal, 1999). For example, the leadership abilities or anxiety level of a learner may be determined by whether a learner is the first to respond to a question in class or the last. When interacting with peers, do they challenge their peers or simply reflect what is being said? Do they tend to be formal or informal in their responses? Do they incorporate humor and incorporate it appropriately? Do they post in a timely manner or are always late with excuses? Do they tend to be defensive or argumentative? While an instructor should not base a grade on personality per se, it is the job of the instructor to make the course room a safe environment and conducive to learning. And in applied psychology, it is the job of the instructor to note interpersonal behavior, give corrective or reinforcing feedback, and help to shape appropriate interpersonal, professional behavior. So, while instructors would not allow learners in a traditional classroom to display inappropriate behavior, this is also true for the virtual course room. When action regarding learner behaviors is taken, it should be done constructively and privately (Thomerson & Smith, 1996), such as in an individual email or phone call. This allows for specific feedback, both positive and negative, opportunities for modeling, and reinforcement of professional boundaries in communications.

On a class process level, the overall course discussion can be monitored, over and above individual contributions. Additionally, because all discussions are preserved over time, this allows for an examination of course room relationships, class history and process issues, and to ongoing conceptual development and learning at the class level. In some courses, for example, each learner is required to respond to any discussion question posted by the instructor while also being required to respond with thoughts, comments, and insights to at least one peer's response. This interaction connects learners, provides course interaction, and enhances overall learning while providing a unique window into how the learners are digesting, discussing, and synthesizing the course content in a social and professional context. Experientially it also sets an example as to what learners can expect in collegial discussion in the future.

Critical reflection as a tool for navigating the process of online teaching and evaluation

In learning to teach applied psychology online, the development of an evaluative framework for assessing learners' work, as well as one's own work and skill development, is necessary. This is especially important in teaching applied psychology because the instructor must assess not only learners' acquisition of course content, but also learners' process and socialization into the field. Brookfield (1995) suggests that the approach of being critically reflective is valuable in helping to unearth hidden assumptions about teaching so as to avoid having hidden assumptions inadvertently shape teaching style or obscure teaching opportunities. Specifically, he suggests using four perspectives for generating critically reflective approaches to teaching and learning.

Autobiographical reflection

The first of these perspectives involves gaining a sense of autobiographical knowledge, knowledge of one's own experience. By reflecting on one's experience as a learner, one may remember critical incidents to stimulate recollection of assumptions about expectations held for one's learners and/or one's self as an instructor. Once these assumptions are uncovered, they can be examined formally for their validity and value. In online teaching, this would mean examining expectations for learners and one's self as an instructor--and examining expectations both for face-to-face and online learning environments. By comparing and contrasting these expectations, the influence of any subtle biases or beliefs could be examined and tested in future teaching experiences, protecting one's self and one's learners from missed learning opportunities. In the area of applied psychology, this might include consideration of differences in impressions and experiences with online and face-to-face teaching and learning experiences, as well as how these experiences and impressions fare with regard to current professional standards and demands, ethically, legally, or otherwise.

Through the eyes of the learner

A second perspective offered is characterized by making an active effort to view one's own behavior as the instructor through the eyes of one's learners. With this, efforts to gather learner feedback and to imagine how instructor communications are experienced by learners helps to reveal the power and influence of the instructor over relationships in the course room. Instructors must take the time to think through what is considered high quality feedback by learners and how the online learning environment may or may not permit transmission of specific aspects of feedback-- be they emotional, informational, or interactive--in a way that meets the learners' needs. Follow up strategies such as sending additional communications to reiterate, reinforce, or clarify feedback may be required. Recognition of the power of the instructor and his or her words to affect how he or she is perceived as an instructor, by individuals and the class as a whole, and how this perception affects learner adjustment are the primary goals of this perspective-taking exercise. Again though, this is not just an exercise for the imagination, but one that also involves critically examining feedback provided by one's learners. In fact, examination of such feedback may invite opportunities to educate learners about their expectations, how realistic they may or may not be, and facilitating better communications and a more open course room environment more generally. This type of reflection, as a modeling endeavor, is also useful, as such a skill may prove particularly beneficial for learners in their work with clients later in their careers.

Peer feedback

Similarly, collegial feedback and experience can be helpful in learning to manage and evaluate student performance online. By inviting colleagues to observe teaching sessions and other communications with learners, a critical perspective can be gathered to uncover teaching habits about which one might not be aware. It may also initiate a process whereby one can dialogue with colleagues about teaching practices that may or may not be effective, new ways of problem-solving, or alternative strategies for achieving the same goals. For instance, colleagues may provide tips as to how to convey information about a graphic or procedure (such as a specific psychological test strategy). They may have experience as to how to do this when an online course permits only words or when learners may access video clips, animation, or printed material and you are not present immediately to discuss the issues with them. Their experience, planning, and feedback as to how to make the discussion clear enough to convey the concepts of the procedures or graphic may save you time in planning and practice.

Theory and practice

The fourth perspective suggested is that of using theory to help guide practice. As mentioned earlier, distance-learning approaches have a long history upon which online teaching is based. As such, online instructors are well advised to borrow from the literature available that reports on effective distance teaching and learning strategies in applied psychology. Theoretical research in this area can provide tools to apply to teaching situations that may otherwise seem daunting.

Summary

This article presented a variety of issues and strategies for understanding online learners, cultivating good assessment and interaction practices, and making suggestions for how to reach and evaluate learners in this very new teaching context for applied psychology. Further, the use of critical reflection was offered as a useful tool for navigating these challenges so as to avoid the inappropriate generalization of face-to-face teaching strategies, while encouraging adaptation of those behaviors and skills that will work in both environments.

Ultimately, cognizance of the similarities and differences between the two teaching environments, as well as of any participants' skills and competencies, are critical. Instructors will be most prepared if they thoroughly assess their skills for both environments and remain aware of any biases they have about the environments or their prospective learners. This is very important because it is likely that these biases will affect not only the goals chosen for learning, but also their emphases and evaluation. To thoroughly design target behaviors for different courses in this area of study, this is a crucial task. Rising to the challenges the online environment provides may foster the development of new teaching and evaluation methods to reach learners who are committed to learning and who might otherwise not be available for instruction were it not for the unique opportunities provided by online instruction.

References

Bennett, P. (1998). The heart of distance learning: A student's perspective. International journal of lifelong education, 17, 51-60.

Brookfield, S. D. (1995). Becoming a critically reflective teacher. San Francisco, CA: Jossey-Bass.

Bullen, M. (1998). Participation and critical thinking in online university distance education. Journal of distance education, 13, 1-32.

Dahlgren, E. (1998). Staff training and the development of distance education courses. Open learning, 13, 38-42.

McDonald, J. & Gibson, C. (1998). Interpersonal dynamics and group development in computer conferencing. American journal of distance education, 12, 7-25.

Pal, R. (1999). Media scripting and teacher training. Indian journal of open learning, 8, 157-61.

Saxena, A. & Jauhari, M. (1999). Quantifying interactive learning. Indian journal of open learning, 8, 163-69.

Taylor, J. (1998). Flexible delivery: The globalisation of lifelong learning. Indian journal of open learning, 7, 55-65.

Thomerson J. & Smith, C. (1996). Student perceptions of the affective experiences encountered in distance learning courses. American journal of distance education, 10, 37-48.

Waghid, Y. (1998). Improving teaching and learning at a distance through collegial partnerships. Open learning, 13, 45-51.

Nancy A. Piotrowski, Associate Scientist at the Alcohol Research Group, also teaches at Alliant International University, San Francisco Bay Campus, and at University of California, Berkeley. Mary Bemker also teaches in the School of Education at the University of Louisville. Bruce Fischer, Chair of the Addiction Psychology area at Capella University, also teaches at the Union Institute. Shelly Kleine also teaches at the University of Phoenix-Washington Campus. Steven V. Schneider also teaches at the Medical College of Wisconsin in the Department of Physical Medicine and Rehabilitation. Linda Whitley Reed is also teaching at the Wright State University School of Medicine and at the University of Dayton.