Perception of contradiction in verbal and nonverbal emotional expression and its impact on memory recall.
The face is regarded as the primary nonverbal channel in communication. The importance of the face in expressing emotion has been shown in me study of attitude, communication, and emotion, etc. (Argyle, 1988). Tomkins (1962, 1963, 1982) modified and extended Jame's theory of emotions. According to him, differentiated emotions derive from the feedback from inherently patterned facial expressions. This provided a strong theoretical support for studying the face as a source to learn about emotions. People can make many different faces but Osgood (1966) found that judges could discriminate only eight emotions. A number of studies conducted so tar have very good agreement over six basic emotion expressions which can be reliably identified, i.e., happiness, surprise, fear, sadness, anger, and disgust (Ekman, 1982, 1992a, 1992b, 1993).
Some early studies used photographs of faces taken from the newspapers and judges were asked to assess the emotion being shown. There was 70% agreement between the judges. Ekman (1982) reviewed nine studies for the accuracy with which emotions are judged from posed photographs; and Argyle (1988) has averaged the accuracy score; Happy (79%), Surprise (65%), Fear (62%), Anger (55%), Sad (57%), and Disgust (54%).
The accuracy of judging another person's emotional state depends on a number of factors. Exaggerated posed expressions are easiest, followed by posed, followed by spontaneous (Zuckerman, Larence, Spiegel, & Klorman, 1981). McLeod and Rosenthal (1983) found that accuracy increased if the best photographs are used, accuracy may go up to 80% or more for happiness and fear. So we know then that emotions can be reliably judged, at least from photographs when there is no conflicting or distracting information.
In conditions where for example the verbal content and the facial expression are sending different emotional signals they are known as mixed messages. If some one says "stop, you idiot!" with happy expression, the receiver may be confused by the inconsistency of the massage. Some people are very good at controlling their emotions while others are not. In our study the same effect could be noticed under incongruent conditions.
A number of studies have been conducted to see the effect of contradictory verbal and nonverbal content of the message on communication. The dominance of nonverbal aspect of communication was suggested by some researchers (e.g., Argyle, Alkema, & Gilmour, 1972; Argyle, Salter, Nicholson, Williams, & Bugress, 1970; Mehrabian & Ferris, 1967). Mehrabian and Ferris (1967) hypothesized that when there is inconsistency between verbally and implicitly expressed attitude, the implicit portion will dominate in determining the total message. For example, when there are inconsistencies between attitude communicated verbally and nonverbally, the nonverbal component will dominate in determining the total attitude that is inferred. The findings suggest that the combined effect of simultaneous verbal, vocal, and facial attitude communications is a weighted sum of their independent effects as follows:
[A.sub.total] = [0.07.sub.verbal] + [0.38.sub.vocal] + [0.55.sub.facial]
Argyle, Alkema, and Gilmour (1972) provided support for the hypothesis and experimental data showed that nonverbal communication cues were found to make greater contribution than verbal cues to the communication of a more dominant, or a more positive attitude.
Numerous other studies were published using similar methodology, and reaching the same conclusions (Bugental, Kaswan, & Love, 1970; DePaulo. Rosenthal, Eisenstat, 1978; Mehrabian & Wiener, 1967). In a typical experiment the judge would rate the affect conveyed by a message where verbal content and facial expression were incongruent. Thus the actor might say "I am so happy", with no trace of happiness on his face. The words and style are manipulated in such a way that they conveyed an oppositely toned message of similar strength. This method is called "cue opposition paradigm" (Walker, 1977). AH those studies using cue opposition paradigm to find out the effectiveness of emotional dimension, are in support of nonverbal dominance hypothesis (Burn & Beier, 1973; Mehrabian & Wiener, 1967). The results from these experiments show that nonverbal communication has significantly more effect than the verbal on the judgement of emotions. Subsequently, this methodological approach was greatly criticized (Beattie, 1983; Furnham, Treventhan, & Gaskell, 198 J) because content of the study were not natural and spontaneous. It is also very difficult to manipulate the same amount of nonverbal in two different messages but it is unlikely to use the same style. So a new approach based on Communication Context was developed (Walker & Trimboli, 1989). According to this Communication Context method the message has to be genuine and mixed. In order to see whether the verbal or nonverbal channel dominates, the valancy (the weighing on a scale from positive to negative) of the message component was compared with the affect conveyed in the total interaction. When the verbal and nonverbal components express opposite affect, the channel which is consistent with the communicational context, has the most impact on subjects judgement of emotion expressed. In this study a number of interfactional segments with mixed messages recorded from live, non-acted television programmes were used as stimulus material. The results of this study yet again demonstrate the nonverbal dominance in the judgement of affect.
Walker and Trimboli (1989), Russell (1994) criticized the methodologies in many early studies because they did not camouflage the purpose of the experiment They suggest that this directs attention to the nonverbal component of communication, and may thus bias judgements which favour the nonverbal dominance hypothesis. It has been tried to overcome this problem in the present study by incorporating an indirect measure of verbal/nonverbal effect, i.e., recall of the verbal statements as well as the direct judgement of emotion.
The main issues in this study were:
1. What happens to the judgement of emotion when verbal message and facial expression indicate different emotions-Which has more impact?
2. How does this influence the recall? Do indirect measure such as recall show the same pattern of effects as indirect measures such as judgement of emotions.
What happens if the verbal massage and the nonverbal cues are not simultaneous? Synchrony has been shown to be important in the process of communication. It has been confirmed that speaker and listener alike coordinate much of their nonverbal behavior to the vocalization of the speaker producing self synchrony for the speaker and interactional synchrony between the two communicators (Condon & Ogston, 1967; Dittman, 1972). Cognitive psychologist (e.g., Neisser, 1967) suggested that vocal rhythm is like a frame for verbal information, it integrates incoming information and expedite recall. This might suggest that when verbal and nonverbal signals are not synchronized recall would decrease.
Woodall and Burgeon (1981) studied the effects of nonverbal synchronization by a speaker on message processing and acceptance by a listener. For this purpose three versions of vedio message were shown to the subjects (i.e., high synchrony, minimal synchrony, and disynchrony between kinesic and verbal/vocal cues). Receivers of the high synchrony message showed higher comprehension and recall of the message as compared to receivers of disynchronous message. For recall a significant main effect was obtained, F(2,166)=5.76, p <.05, and means were in the predicted direction. In case of disynchorony, the receivers feel more distraction which will adversely effect the massage acceptance.
To establish that the verbal content of the messages to be used in the study, reliably elicited the intended emotion, a pilot test was conducted. A large list of sentences was constructed which conveyed one of the five emotions: Happiness, sadness, anger, fear, and surprise. These sentences were presented to volunteer subjects (N = 20) in written form, and they were asked to judge which emotion the sentence conveys. Only sentences for which 90% of subjects had chosen the target emotion, were retained for the experiment. Four different sentences for the emotions: Sad, anger, and happy met this criteria.
A second pilot study was conducted to ensure that the facial expressions used in this study also reliably indicated the target emotion. Short video clips were prepared with the help of two actors, one male and one female, showing three different emotions. These video clips were shown to 20 judges without any sound, and ratings of the extent to which each clip conveyed the intended emotion were obtained. For each item, subjects were required to rate how strongly they felt the clip was characterized by each of the three emotions under investigation, namely happiness, sadness, and anger. The ratings were performed on 7-point scale, ranging from the response "very much so" to "not at all". The criterion for the inclusion in the study was that 90% of subjects rate the target emotion either six or seven on the scale. In this way four clips which clearly showed happy, sad, and angry faces were selected, yielding to twelve facial expressions.
The sample (N = 46) was randomly selected from the under-graduate students of Psychology Department, University of Glasgow. Approximate age of the subjects included in the study was between 18 to 20 years.
Design and Procedure
In this study 2x2 factorial design was used. The two factors manipulated were congruence between verbal message and facial expression and synchrony between verbal message and facial expression on each emotion (i.e., happy, sad, and angry). From the pretested material, four different facial expressions which clearly indicated the intended emotion were selected. Each video clip had the sound track added for each of the four conditions of the experiment (congruent or incongruent verbal message, simultaneous with facial expression, or delayed by one second). Master tape was prepared in such a way that every happy, sad, and angry face was edited at four different levels, yielding to forty-eight video clips altogether. The clips were always arranged in the order as presented in the Table 2.
Four experimental video tapes were prepared from the master tape in such a way that clip no. I, 2,3, and 4 goes to tape A, B, C, and D, respectively. While in case of Face no. 2 the clip no.2 will go to tape A clip no.3 to tape B, clip 4 to tape C and clip no. 1 to tape D. This was done to keep the balance between the tapes. Finally the order of items (clips) was randomised for each tape. Each tape shown to subjects contained 12 clips with repeated measures of all the levels of the design, and across all four tapes each facial expression was repeated at each level of the design.
The experiment was run for every three students separately so that they should have excellent view of the video clips. Tape A, B, and C were viewed by 11 subjects while Tape D was seen by 13 subjects. Before the beginning of the experiment the subjects were given following instructions: "You will be shown a short tape with twelve short extracts. Please tick the box on your answer sheet after the person has spoken (but before the next clip appear). After this subject were given another trail, but without sound, and asked to recall the verbal statement of each video clip. The number of correct recalls of verbal statements and correct judgement of emotions over the two conditions served as dependent variable. The scoring technique was based on the subjects judgement of emotions. Correct judgements were given one point each. There were three video clips for each of the four conditions (congruent synchronous, congruent asynchronous, incongruent synchronous, and incongruent asynchronous). The maximum score for each subject over the four conditions was 12 points. The same scoring technique was applied to the recall of verbal statements. Again the maximum score for each subject was 12.
A 2x2 analysis of variance was conducted on the number of correctly recalled statements under four different conditions. Both main effects were statistically significant. The recall data showed a significant main effect of congruence (F(1,47)=32.203, p<.000)] with recall being significantly better where facial expression and verbal message are congruent. There was also a significant main effect of synchronouse [F(1,47)= 6.384, p<.015)]. When the two signals are not synchronized in time, recall drops significantly (20% less). There was a significant interaction between congruence and synchronous [F(1,47) = 5.565, p<.023)]. Analysis of simple effect shows an effect for congruence at both levels of synchronosity, i.e., for congruence synchronous [F(1,47) = 27.647, p<.000)], congruous asynchronous [F(1,47) = 4.972, p<.031)]. It also shows effect for synchrony at congruous synchronous level [F(1,47) = .018, p<.894)].
The average number of emotional words recalled is 55% when face and message display synchronized congruent signals (Table 3). This significantly reduces where the verbal massage is delayed by one second and only 37.3% of words are correctly recalled in asynchronous conditions. Where face and message display different incongruent emotions simultaneously and average recall of emotion words is 27.7%, significantly less than where the two signals are congruent. This figure does not differ significantly from the level of recall in congruent asynchronouse conditions.
t-test was also calculated. Differences between congruent synchronous vs. incongruous asynchronous, incongruent synchronous and congruent asynchronous were significant at p<.01. Where as the difference between incongruent asynchronous vs. congruous synchronous, and congruent asynchronous vs. congruent synchronous were significant (p<.05).
When facial expression and verbal massage are sending different, incongruous signals recall is significantly worse but there no effect of synchrony here. The relatively low level of recall when incongruous signals are present does not decrease significantly when there is a short time interval between seeing the facial expression and hearing the verbal message.
Judgement of facial expression
Another important factor studied in this research was the way judgements of emotion were affected when there is congruency/ incongruency between verbal message and facial expression and synchrony/asynchrony between the verbal message and facial expression. While incongruence and asynchrony used in the verbal statement creates misunderstanding and distraction.
The results obtained for the judgement of the emotion were assessed statistically by two-way analysis of variance. In this experiment the subjects were also asked to tick the emotion the person was feeling in each video extract. The correct judgement of emotion was scored and calculated in the same manner as the statements of recall. The judgement data showed a significant main effect of congruence [F(1,47) = 51.029, p<.000]. However there was no significant effect of synchrony [F(1.47) = .232, p<.633]. Interaction of congruency and synchrony was also not significant.
Subjects judgements of emotion displayed in the video clips was greatly affected by whether the facial expressions and the verbal messages were congruent. When both channels send the same signals 81.6% of judgement of the emotion expressed are as predicted from pilot study II, where judgements were based solely on the emotions shown on the actors faces. Where nonverbal expression and verbal message are incongruent, the number of judgment which are in accord with facial expression decrease significantly. Only in 52% of cases, subjects judge the emotion on the basis of the facial expression. The timing of the presentation of the two signals (synchronous or asynchronous) has no effect, nor does it interact with the effect of congruence between the two signals. Under the incongruent synchronous conditions there were 144 extracts shown to 48 subjects. The number of times the subjects have gone for the face is 79(54.86%)), 42 times for the words and 23 times were undecided. In incongruence synchronous condition 77 times for the face 30 times responded in the favour of words and 37 times were undecided.
The present research provides clear evidence of a difference in the number of correct recalls of verbal statement when there was congruency between the verbal and nonverbal content of emotional expression. Student's retention of the verbal message was consistently and substantially better than their retention of verbal statement under incongrous conditions. It was also noticed that whenever there was incongruence between the verbal and nonverbal content of the message, in most of the cases, the words conveying a particular emotion are interpreted differently according to the facial expression accompanying them, changing the semantic of the message. For example, in one of the clips portraying incongruence the model says: T am so glad', with sad face, most of the subjects recalled the statement as: I am so sad*. It seems that incongruence generate some kind of conflict and subject is likely to resolve it by changing the verbal statement in the favour of emotional expression. The recall task used here is a cued recall. Thus, it is uncertain whether the congruence effect in memory was due to encoding of congruent verbal messages or to effective retrieval of congruent verbal messages. Perhaps, another experiment in future, where faces are not provided as recall cue may clarify the issue Kitayama (1996) failed to find any congruence effect in recall when no retrieval cue is provided.
The simultaneous occurrence of verbal and nonverbal cues also affect the memory recall. If totality of the message is disintegrated, even by one second, then it looses its meaning which will adversely effect the retention and recall. The recall of verbal statement was better when there was synchrony between the verbal and nonverbal emotional cues. A support comes from Woodall and Burgoon (1981; receivers of high synchrony message showed higher recall of the message). There is such a strong effect of congruence and incongruence and so few cases of correct recall in incongruent conditions (both less than .9). This insignificant difference between synchrony and asynchrony in incongruent condition could be due to a floor effect.
Another important fact revealed in this study was that when verbal message and facial expression are congruent, 81.6% judgement corresponds to judgements based on the facial expression alone, compared to only 52% of emotional judgements in the incongruent conditions. The timing of the presentation of the two signals (synchrony and asynchrony) has no effect, nor does it interact with the effect of congruence between the two signals. The distracting influence of the semantic content of the verbal message seems to be a powerful one. These data show little evidence of nonverbal dominance. Where the verbal message is delayed (asynchronous condition) we might predict that the facial expression would be more dominant. No such effect was found as can be seen from Table 4.
Nonverbal dominance was demonstrated by Mehrabian and Ferris's (1967) in their study of combined effects of consistent and inconsistent facial, vocal, and verbal communication of attitudes. They used an equation to illustrate the influence of facial, vocal, and verbal communication components on inferred attitude [A.sub.totai] = 0.07 [A.sub.verbal] + 0.38 [A.sub.vocal] + 0.55 [A.sub.facial]. From this equation it can be inferred that 55% of the attitude is based on nonverbal component, i.e., a facial. This is comparable to findings in the present study which shows that 52% of emotion's judgement in incongruent conditions are based on the nonverbal component. This finding is supported by Mehrabian and Ferris's (1967) conclusion that "when there is inconsistencies between attitude communicated verbally and posturally, the postural component should dominate."
The previous research on synchrony has shown that multichannel synchrony, on the one hand, increases comprehension and recall of the message but lowers the degree of counter arguing. Whereas dissynchrony reduces comprehension and credibility, maximizes distraction but produces high counter argumentation and low message acceptance. (Burgoon, Cohen, Miller, & Montgomery, 1978). This study supports our findings. Dissynchrony inhibited the message acceptance and created doubts in the minds of the subjects which ultimately affected the recalls as well as judgement of emotions.
Argyle, M. (1988). Bodily communication. (2nd ed.) Rutlege London and New York.
Argyle, M., Alkema, F., & Gilmour, R. (1972). The communication of friendly and hostile attitude by verbal and nonverbal signals. European Journal of Social Psychology, 1, 385-400.
Argyle, M., Salter, V., Nicholson, H., Williams, M., & Bugress, P. (1970). The communication of inferior and superior attitude by verbal and nonverbal signals. British Journal of Social Psychology, 9, 222-231.
Beattie, G. (1983). Talk: An analysis of Speech and Nonverbal Behavior in Conversation. Milton Keynes: Open University Press.
Bugental, D. B., Kaswan, J. W., & Love, L. R. (1970). Perception of contradictory meanings conveyed by verbal and nonverbal channels. Journal of Personality and Social Psychology, 16, 647-655.
Burgoon, M., Cohen, M., Miller, M. D., & Montgomery, C. L. (1978). An empirical test of model of resistance to persuasion. Human Communication Research, 5, 27-39.
Burn, K. L., & Beier, E. G. (1973). Significance of vocal and visual channels in the decoding of emotional meaning. Journal of Communications, 23, 118-130.
Condon, W. S., & Ogston, W. D. (1967). A segmentation of behaviour. Journal of Psychiatric Research, 5, 221-236.
DePaulo, M. B., Rosenthal, R., & Eisenstat, R. (1978). Decoding discrepant nonverbal cues. Journal of Personality and Social Psychology, 36, 313-323.
Dittman, A. T. (1972). The body movement-speech rhythm relationship as a cue to speech encoding. In A. Seigman & B. Pope (Eds.), Studies in dyadic communication. New York: Pergaman Press.
Ekman, P. (1982). Emotions in the human face. (2nd. ed), Cambridge University Press.
Ekman, P. (1992a). An argument for basic emotions. Cognitions and Emotions., 6, 169-200.
Ekman, P. (1992b). Are there basic emotions? Psychological Review, 99, 550-553.
Ekman, P. (1993). Facial expressions and emotions. American Psychologists, 48, 384-392.
Furnham, A., Treventhan, R., & Gaskell, G. (1981). The relative contribution of verbal vocal, and visual channels to person perception: Experiment and critique. Semiotica, 37, 39-57.
Kitayama, S. (1996). Remembrance of emotional speech: Improvement and impairment of incidental verbal memory by emotional voice. Journal of European Social Psychology, 32, 289-308.
McLeod, P. L., Rosenthal, R. (1983). 'Micromomomentary Movement and the Decoding of Face and Body Cues', Journal of Non-verbal Behavior, 8, 83-90.
Mehrabian, A. & Ferris, S. R. (1967). Interfarence of attitudes from nonverbal communications. Journal of Consulting Psychology, 31, 248-252.
Mehrabian, A., & Wiener, M. (1967). Decoding of inconsistent communications. Journal of Personality and Social Psychology, 6, 109-114.
Neisser, U. (1967). Cognitive Psychology. New York: Appelton-Century-Crofts.
Osgood. C. E. (1966). Dimensionality of the semantic space for communication via facial expression. Scandinavian Journal of Psychology, 7, 1-30.
Russell, J. A. (1994). Is there recognition of emotion from facial expression? A review of cross-cultural studies. Psychological Bulletin, 7/5(1), 102-141.
Tomkins, S. S. (1962). Affect, imagery, consciousness: Vol. 1. The position affects. New York: Springer-Verlag.
Tomkins, S. S. (1963). Affect, imagery, consciousness: Vol. 2. The position affects. New York: Springer-Verlag.
Tomkins* S. S. (1982). Affect, imagery, consciousness: Vol. 3. Cognition and affects. New York: Springer-Verlag.
Walker, M. B. (1977). The relative importance of verbal and non-verbal cues in the expression of confidence. Australian Journal of Psychology, 29, 45-57.
Walker, M. B., & Trimboli. (1989). Communicating effect: The role of verbal and nonverbal content. Journal of Language and Social Psychology', 8(3-4).
Woodall, W. G., & Burgoon. (1981). The effects of nonverbal synchrony on message comprehension and persuasivenes. Journal of Non-Verbal Behaviour, 5(4).
Zuckerman, M., Larence, D. T., Spiegel, N.H., & Klorman, R. (1981). Controlling nonverbal cues: Facial expressions and tone of voice. Journal of Experimental Social Psychology, 17.
Received: March 25, 1998
Department of Psychology
University of Peshawar
(#) Correspondence concerning this article should be addressed to Farhana Jahangir, Department of Psychology, University of Peshawar, Peshawar, Pakistan.
The research reported in this paper was supported by the Department of Psychology, University of Glasgow.
Table 1 Examples of verbal statements Emotions Examples Sad It's so depressing. I am very unhappy. I am really down. I am very upset. Anger I will kill you. I am so angry. You are driving me crazy. I am getting mad. Happy I am so happy. It's wonderful. I am so glad. I am so relaxed. Table 2 Examples of video extract (happy faces) from the master tape Clip Happy Face Number No. 1 (Male) 1 congruous + synchronous 2 congruous + asynchronous 3 incongruous + synchronous 4 incongruous + asynchronous No. 2 (Female) 1 congruous + synchronous 2 congruous + asynchronous 3 incongruous + synchronous 4 incongruous + asynchronous No. 3 (Male) 1 congruous + synchronous 2 congruous + asynchronous 3 incongruous + synchronous 4 incongruous + asynchronous No. 4 (Female) 1 congruous + synchronous 2 congruous + asynchronous 3 incongruous + synchronous 4 incongruous + asynchronous Table 3 Means of total number of items recalled correctly on congruent and synchronous conditions (% of total possible) Synchronous Asynchronous Total Congruous 1.6667 (55.30%) 1.1250 (37.33%) 1.3958 (46.30%) Incongruous 0.8333 (27.66%) 0.8125 (27.00%) 0.8229 (27.33%) Total 1.2500 .9687 - Total number of statements were 48. Table 4 Means of total number of correct judgement of emotions under four different experimental conditions Synchronous Asynchronus Total Congruous 2.4375 (81.0%) 2.4792 (82.3%) 2.4583 (81.6%) Incongruous 1.5417 (51.3%) 1.5833 (52.6%) 1.5625 (52.0%) Total 1.9896 2.0312 2.0099
|Printer friendly Cite/link Email Feedback|
|Publication:||Pakistan Journal of Psychological Research|
|Date:||Dec 22, 1998|
|Previous Article:||Perceptions of obesity and diabetes in societies in transition.|
|Next Article:||Teacher efficacy: in terms of teachers' conceptions of the nature of ability and their motivational styles.|