In search of a standardised, comprehensive assessment of functioning.

Abstract

As part of a national programme to provide mechanisms for quality improvement in the mental health sector, New Zealand has established the MH-Smart Programme. The programme aims to introduce the routine use of standardised assessments that, with repeated use, can reflect consumer changes, outcomes or results. This paper describes the process and rationale for assessing the suitability of measures of functioning. No ideal measure was found, but a provisional recommendation has emerged.

Key words

Functional skills, assessment, mental health, outcome measures

Lutchman, R., Thompson, A., Tait, H., Savage, A., Aitchison, R., Ruru, R. & Mellsop, G., (2007). In search of a standardised, comprehensive assessment of functioning. New Zealand Journal of Occupational Therapy 54(1) 33-38

**********

The New Zealand Ministry of Health piloted the routine use of outcome measures in the Casemix and Outcomes Study (CAOS) in July 2002 (Gaines et al. 2003; Eagar, Trauer & Mellsop, 2005), carried out under the Mental Health Research and Development Strategy. This has been succeeded by a programme MH-Smart, which seeks to nationally implement the routine use of standardised assessment measures. In common language these measures are frequently referred to as outcome scales (Mellsop & Wilson, 2006). Increasingly, outcome measurement is being recognised as an important area for service delivery and a part of Government policy. Using such measures can facilitate clinician/consumer discussion or partnership and consumer recovery planning, as well as team and service level quality improvement activities and national benchmarking activities. MH-Smart seeks ultimately to have five standardised measures of assessment (outcome scales).

These are likely to be:

* The Health Of Nations Outcome Scale (HoNOS) Family (CRU, 1996; Wing et al, 1998), reflecting particularly clinical presentation issues.

* A measure of functioning.

* A consumer rated outcome measure.

* A culturally informed (Maori) outcome measure

* A standardised outcome measure for drug and alcohol services.

The term 'functionality' is defined by the World Health Organisation (WHO, ICIDH-2, 2001) as an 'umbrella' term to cover an individuals' activities and participation. The MH-Smart Programme sought through tender, recommendation for measure(s) of functionality that could be routinely used in the New Zealand context of adult mental health services. The Waikato DHB team won the tender and provided the ministry with the recommendation. This brief report describes the process of reviewing the available measures.

In addition, the Ministry of Health required the recommendation process to examine the cultural acceptability of the instruments, their consistency with the recovery philosophy and compatibility with the HoNOS family at the same time as determining any overlap. The selected measures had to address wellness, medication management, personal cares, family/social involvement and activity. Each of the instruments were rated on whether they measured performance (which describes what an individual does in his/her environment) or capacity/potential (which describes an individual's ability to execute a task or action), or both.

Method

A national and international literature search on outcome measures and measures of adult functionality was conducted. Sources for the literature review were obtained from key word searches conducted on MEDLINE, PSYCHINFO, CINAHL, AMED and other relevant databases, as well as from citation lists, texts and other pertinent policy documents. Key informants from the MH Smart Team were consulted along with other mental health service providers.

Starting with scales already known to be used in New Zealand, a template of content areas, potential outcome domains and indicators which the concept of functionality should cover was developed in a systematic manner. This was then modified and developed by occupational therapy, consumer and Maori cultural input. The completion of measures of functionality needed to incorporate the context of a clinician/consumer partnership. In the end, six major domains were identified, with a number of specific indicators in each (see Table 1). The content area and criteria template were further modified to allow for consideration of whether the scale was designed to assess performance or capacity/potential.

A second template based on the work of Andrews, Peter & Teeson (1994), was developed to assess the psychometric credibility of each instrument. The psychometric properties that were considered include: applicability, acceptability, practicality, validity and reliability.

Outcomes of consultation with key informants, coupled with findings from the global health internet search engines identified 25 instruments/scales for examination against these two sets of criteria (domains and psychometric credibility). Most of the scales were unknown in New Zealand and the few that were known were not widely used. Three members from the MH-Smart team viewed each instrument and its 'justification' literature and rated them against the set criteria. One potentially relevant scale, the Life Skills Profile (LSP16) (Rosen et al, 1989) was specifically excluded by the MH-SmartTeam because of negative feedback from consumers and clinicians during the CAOS study.

The rating panel was made up of a multidisciplinary team of clinicians. These included an occupational therapist, a psychiatrist, a social worker, a registered nurse and a clinical psychologist. All 25 instruments were then evaluated and scored by at least three of the clinicians individually using the two templates (Table 1). The individual results were then collated and tabulated (see Table 2). In that process five instruments were found to be superior to the other 20. They were subjected to a more careful qualitative examination, particularly against consumer and Maori cultural perspectives. The five final chosen instruments were then sent to the Senior Consumer advisor and the Maori Manager at the Waikato DHB for their opinion. Their evaluations assessed the utility, the responsiveness to Maori consumers and the cultural appropriateness of the functioning measures for consumer outcomes in Mental Health Services. A look at the instrument's compatibility with and complementarily to HoNOS was also considered.

Team representatives visited four North Island and two South Island psychiatric services to present the results of the review to MH-Smart stakeholders, and to seek their views on the short listed instruments. Their feedback, along with the qualitative examination of the short listed instruments informed, or underpinned, the ultimate recommendations.

Results

The initial screening ratings as presented in Table 2 indicate:

Yes                           the instrument met all the set
                              criteria

Moderately satisfactory       the instrument met the criteria at
an above average level

Satisfactory                  some criteria were met and some
                              were not

No                            the instrument did not meet the
                              set criteria

Five of the scales appeared to measure up much better than the others. As outlined below each of those still fell well short of meeting all of the criteria that had been set for the rating exercise but equally there were some significant strengths. The raters found it virtually impossible to make decisions about capacity vs. performance for many of the questions in most of the scales (ICIDH-2, WHO, 2001).

Table 3 outlines the five short listed instruments with strengths and weaknesses in relation to the functional criteria. They include:

* The Camberwell Assessment of Need (CANSAS) (Phelan et al, 1995; Slade et al, 1999). The title of 'need' is perhaps misleading, as most of the items could be seen to reflect functionality. A small number imitated the clinical psychopathology and so could be seen as overlapping with HoNOS. The CANSAS may be more time consuming to administer than some of the other scales. Provision exists for it to be separately rated by consumers, clinicians, or carers but this project was in search of a staff rated tool. The Camberwell had already been criticized in an earlier publication on consumer rated outcome measures (Gordon et al, 2003).

* Social Functioning Scale (Birchwood et a1,1990). This scale performed so well on most of the measures that we kept it within the potential vision of MH-Smart, despite it being essentially a consumer rated instrument.

* Multnomah Community Ability Scale (Barker et al, 1994). Covered the domains well but is rather long and with some superfluous areas.

* Personal and Social Performance Scale (PSP) (Morosini et al, 2000). This scale owes some of its derivation to the SOFAS, which in turn was a cleaned up Global Assessment of Functioning Scale (GAF of DSM 4) (Hilsenroth et al, 2000). Its particular advance on that comes from having excluded the clinical variables which otherwice contaminate the functionality of a GAF score.

* Canadian Occupational Performance Measure (COPM) (Law et al, 1991). This has similar qualities to the Social Functioning Scale.

Cultural Ratings

The second phase of the study required a more detailed examination of the cultural compatibility of the five short listed measures. The cultural criteria had been separately identified for this project, and were consistent with a Maori model of health (Durie, 2001).

The key concepts for consideration in the review and rating of the functionality measures are interdependence, relatability, and value or belief based motivations, evident in the approach and execution of daily living activities/tasks. Durie (2001) suggests these three concepts are implicit in every culture.

In this project considerations are not limited to culture being narrowly defined as ethnicity. Rather cultural considerations may include religion, music, communal living etc. Ethnicity provides a starting point for cultural considerations. The ability to consider wider cultural views may also be required and may depend on the person and the culture providing the dominant influence in an individual's current life.

The tool of favour when measuring interdependence and relatability (the two key characteristics) was the PSP. Interdependence relates to the active relationships supporting the individual. Key relationships recognize the significance of Atua (God), Tangata (family, whanau, iwi, hapu), Whenua (the land), and Tupuna (ancestors) in the life of the individual. These relationships and a commitment to participating in whanau events, local marae, community or whanau oriented activities are fundamental to healthy functioning in Maori culture.

Relatability--Identifies the way the individual relates to others, their motivation, willingness to participate and their contribution to relationships. For example roles and responsibilities may include parents, kaumatua, kuia, tohunga, older and younger siblings. In addition, the types of activities undertaken to support levels of functioning indicate the value of the relationship.

This holistic view of health is important to Maori and so all things Tinana (physical), Hinengaro (mental), Whanau (emotional) and Wairua (spiritual) are incorporated into the functionality assessment. Evidence of Maori cultural values at work, can be found in the types of activities undertaken. For instance, the presence of wairuatanga (spirituality) is assessed through meaningful activities that might include attendance to regular karakia (prayer). Care of self and whanau (family) incorporates the value placed on the presence and support of extended whanau while whenua relates to the importance placed on the home environment, the value of surroundings and the person's sense of purpose in relation to Maori values.

None of the tools were perfect and the PSP would benefit from some adaptation, however it does offer a basis for development by adding a list of options into the four main areas. These areas could then be individualised to the needs of the person.

Consumer view

While all the short listed instruments had met the predetermined recovery related dimensions, more specific exposure to the consumer perspective left them all falling short of meeting with clear approval. Many consumers stated they would prefer measures that focused on strengths as well as weaknesses. The consumer representatives on the project mostly favoured those assessments that were primarily consumer scored i.e. the Social Functioning Scale and the Canadian Occupational Performance Measure.

Examples of the comments received include:

* The language used in the Social Functioning Scale was quite simple and could potentially be offensive to some consumers.

* The style in which the questions are asked in the Camberwell Scale could be confrontational or distressing to consumers. They appear to dwell on the negative aspects of their life.

* The Multnomah Scale is clinical in approach and potentially judgmental towards the consumer. There does not appear to be much room for individuality or reasons for behaviour etc.

* The process of engaging the consumer in the Personal and Social Performance Scale is not clear and needs to be established.

Site visits

Approximately 150 people attended the various site visits and contributed their opinion. All clinical disciplines, consumers, carers, NGOs and Maori providers were represented. The single most important feedback from the site visits was that any instrument had to be relatively brief and easy to use to be acceptable to the sector. That led to particularly negative comments about two of the most comprehensive instruments (CANSAS and Multnomah). COPM was regarded by occupational therapists as unacceptable for general use, as they considered it a discipline specific instrument. In general, the other site feedback was consistent with the initial, structured, assessments (above), which had led to the development of the two instrument rating template systems.

Project discussion

During the course of this project the assessing team were made very aware of the complexities and pitfalls in the task of deciding on a measure of functionality for the diverse groups of adult psychiatric service users. Several issues require comment:

Who does the rating?

For several reasons, including the previous review of consumer rated outcome measures by the Wellington group(Gordon et al, 2003), and the present tender also to that group to develop a consumer rated measure of outcome, this functionality tender was aimed at having a clinician rated tool (Gordon et al. 2003). It is very clear that all clinician ratings of a consumer need to be well informed by the consumer's input. This in turn will be enhanced by the consumer's commitment to the assessment process and the rapport with the assessor. As functionality is further removed from pure measures of psychopathology, the 'judgement' of the consumer on their assessment parameters becomes increasingly central. We have therefore done our scale assessments bearing in mind that the underpinning concept must be partnership. Above all the completion of measures of functionality needs to be done in the context of a clinician/ consumer partnership. The short listed scales all embody that philosophy, but with some variation on precision. This was clear from the write-up of the procedure and usage of the instruments.

Nomothetic versus idiographic.

The MH-Smart Project, and most credible psychometrically based assessments, are conceptualised as nomothetic assessments. In other words they seek a standardised format, to assess people on the same parameters and on the same scales. This is a necessary approach for benchmarking and for some quality assurance feedback activities. However, the gains from such standardisation at times are in conflict with individual validity. Increased validity, in some situations, can only come from a more idiographic approach where each person is assessed against the parameters most relevant to their own circumstances. This approach is the most useful in the individual consumer/clinician informed discussions. Our assessments of the various instruments were done within a relatively nomothetic framework.

Disability versus functionality

'A rose by any other name....' is a dictum believed by many. However, although disability measures and functionality measures often seek to measure the same thing, the implications of the two terms appear to be very different, especially from a consumer perspective (Gordon et al, 2003; Teeson & Gallagher, 2000; Clark et al, 2004). None of the measures short-listed are called disability scales.

Performance versus capacity

In general, none of the measures gave clear guidance on wether they were most interested in performance or capacity. Most often it was the implied that performance was under consideration as people were being assessed in their own, personal environment. Performance is more relevant when seeking to measure change, which can be bought about by 'services' or contextual modifications.

Cultural and recovery principles

As these measures were all developed outside New Zealand none incorporated the specific cultural considerations that New Zealand Mental Health Services need to take into account. Also they were clearly not designed within the espoused recovery framework. However many of them did make points, or seek to assess some items, which would be relevant to both those principles. The project team found itself having to consider not so much how good the scales were in relation to recovery and culture, but rather to seek to eliminate those which did not address those two principles.

Conclusion

Functionality has been considered sufficiently important to be one of the five axes of DSM IV It has also been noted to be very relevant to both care seeking behaviour (MaGPi, 2003) and to individual recovery planning. This exercise clarified the domains which underpin functioning, and sought the most appropriate instrument for the New Zealand context consistent with the key principles adopted by the Ministry of Health. One single instrument, the Personal and Social Performance Scale, came closest to the designated criteria. It would be wise to undertake New Zealand based research on its reliability, validity and utility before considering its mandated use.

References

Andrews, G., Peters L., & Teeson, M. (1994). The measurement of consumer outcome in mental health: A report to the national mental health information strategy committee. Sydney: Clinical Research Unit for Anxiety Disorders.

Barker, S., Barron, N., McFarland, B & Bigelow, D. (1994) Multnomah Community Ability Scale. Community Mental Health journal, 39, 131-137.

Birchwood, M., Smith, J., Cochoran, R., Wetton, S., & Copestake, S. (1990). The Social Functioning Scale: The development and validation of a new scale of social adjustment for use in family intervention programmes with schizophrenic patients. British Journal of Psychiatry, 157, 853-859.

Clark, M. S., Bond, M. J., Prior, K. N., & Cotton, A. C. (2004). Measuring disability with parsimony: Evidence for the utility of a single item. Disability and Rehabilitation 26, 72-279.

Durie, M. (2001). Mauri Ora. The dynamics of Maori health. Melbourne: Oxford University Press.

Eagar, K., Trauer, T, & Mellsop, G. (2005). Performance of routine outcome measures in adult mental health care. Australian & New Zealand Journal of Psychiatry, 39, 713-719.

Gaines, P., Bower, A., Buckingham, W, Eagar, K., Burgess, P., & Green, J. (2003). New Zealand mental health classification and outcomes study. Final report. Auckland: Health Research Council of New Zealand.

Gordon, S., Ellis, P., Haggerty, C., Pere, L., Platz, G., & McLaren, K. et al. (2003) Preliminary work towards the development of a self-assessed measure of consumer outcome. Auckland: MHR & DS. Health Research Council of New Zealand.

Hilsenroth, M. J., Ackerman, S. J., Blagys, M. D., Baumann, B. D., Baity, N. R., Smith et al (2000). Reliability and validity of DSM 4 Axis 5. American Journal of Psychiatry, 157,185 8-1863.

Kingi, Te K. & Durie, M. (2004). Hua Oranga. Service application and management. Wellington: Te Pumanawa Hauora Massey University.

Kingi, Te K. & Durie, M. (2002). Hua Oranga. A Maori measure of mental health outcome. Auckland: Health Research Council of New Zealand.

Law, M., Baptiste, S., Carswell, A., McColl, M. A., Polatajko, H., & Pollack, N. (1991). Canadian occupational performance measure. Toronto: CAOT Publications ACE.

MaGPi Research Group. (2003). The nature and prevalence of psychological problems in New Zealand primary healthcare: A report on mental health and general practice investigation (MaGPi). New Zealand Medical Journal 116,1171-1185.

Mellsop, G. W & Wilson, J. (2006). Outcome measures in mental health services: Humpty Dumpty is alive and well. Australian Psychiatry, 14, 137-140.

MH-Smart website, www.moh.govt.nz.

Ministry of Health. (2005). Te Tahuhu. Improving mental health. 2005-201. The second New Zealand mental health and addiction plan. Wellington: Author.

Morosini, P. L., Magliano, L., Brambilla, L., Ugolini, S., & Pioli, R. (2000). Development, reliability and acceptability of a new version of the DSM4 Social and Occupational Functioning Assessment Scale (SOFAS) to assess routine social functioning. Acta Psychiatrica Scandinavia 101, 323-329.

Phelan, N., Slade, M., Thornicroft, G., Dunn, G., Holloway, F, Wykes, T. (1995). The Camberwell Assessment of Need: The validity and reliability of an instrument to assess the needs of people with severe mental illness. British Journal of Psychiatry, 167, 589-595.

Rosen, A., Hadzi-Pavlovic, D., & Parker, G. (1989). The Life Skills Profile: A measure of assessing function and disability in schizophrenia. Schizophrenia Bulletin, 15, 325-337.

Royal College of Psychiatrists Research Unit. (1996). Health of the nation outcome scales. Report of research. London: Author

Slade, M., Beck, A., Bindman, J., Thornicroft, G., & Wright, S. (1999). Routine clinical outcome measures for patients with severe mental illness: Cansas and HoNOS. British Journal of Psychiatry, 174, 404-408.

Teeson, M., & Gallagher, J. (2000). Measuring disability, need and outcome in Australian community mental health services. Australian & New Zealand Journal of Psychiatry, 34, 850-855.

Wing, J. K., Beevor, A. S., Curtis, R. H., Park, S., Hadden, S., & Burns, A. (1998). Health of Nation Outcome Scales (HoNOS). British Journal Psychiatry, 172, 11-18.

World Health Organisation, (2001). International classification of functioning, disability and health--2. Geneva: Author.

Raksha Lutchman, Andrew Thompson, Hemaima Tait, Anne-Louise Savage, Rachael Aitchison, Ripene Ruru and *Graham Mellsop

Raksha Lutchman--Clinical Psychologist, Andrew Thompson--Occupational Therapist, Hemaima Tait--Maori Cultural Advisor, Anne-Louise Savage--Consumer Advisor, Rachael Aitchison--Clinical Nurse Leader, Ripene Ruru--Social Worker, * Graham Mellsop--Professor of Psychiatry

Mental Health Division

Waikato Hospital 

Private Bag 3200

Hamilton

* Correspondence to:

Graham Mellsop

Professor of Psychiatry

Waikato Clinical School

University of Auckland 

[e-mail]

Table 1

Set of functional domains developed to help choose and classify
functioning outcome instruments

Functional domains       Functional indicators

Self care                Wellness management, medication management,
                         personal hygiene, diet/nutrition/eating,
                         Sleep, Exercise, weight management

Domestic                 Living conditions/accommodation, meal
                         preparation, budgeting/resource
                         management, Transport/Travel,
                         shopping, housework/cleaning

Work/productivity        Paid employment, study/education,
                         voluntary work

Leisure                  Sport and recreation, interests/hobbies,
                         holidays

Social                   Community involvement, family/whanau
                         relationships significant interpersonal
                         relationships communication--general,
                         read, write, listen, speak, comprehend

Cultural beliefs,        Te Atua (spiritual well being)--Does this
values and practices     measure question the consumer's awareness of
                         cultural identity and his/her links to
Exploring what is        culture? Does this measure allow
important to the         the consumer to identify their culture?
consumer that will
add to his/her          Te Tangata (family/people)--Does this measure
overall functioning     question the consumer's involvement/
                        contact with family and identify his/her
                        home/local community?

                        Te Whenua (land, home, community) Does this
                        measure question the consumer's language,
                        where he/she comes from?

                        Tupuna (history, knowledge, skill,
                        motivation, legacies)--Does this measure
                        question the consumer's knowledge of
                        his/her genealogy/family history?

                        Tikanga (protocols and the manner in which
                        it is done)--Does this measure question the
                        consumers awareness of the manner in which
                        things occur within the Iwi--Hapu?

Table 2
A summary of all instruments measured using the functional
dimensions criteria

Name of instruments rated            Does it meet      Does it meet
                                     the functional    the psychometric
                                     dimensions        criteria
                                     criteria

1. Global Assessment of              No                Satisfactory
   Functioning (GAF)

2. World Health Organisation         No                No
   Disability assessment Schedule
   II (WHO DAS II)

3. Manchester Short assessment of    No                No
   Quality of Life (MANSA)

4. Functional Rehabilitation         Satisfactory      No
   Education and
   Development (FRED)

5. Camberwell Assessment of          Yes               Yes
   Need (CAN SAS)

6. Liverpool University              No                No
   Neuroleptic Side Effects
   rating Scale (LUNSERS)

7. Rehab Readiness Scale             No                No

8. Social Functioning Scale (SFS)    Yes               Yes

9. Functional Impairment Measure     No                No

10. Social Occupational              No                No
    Functioning assessment
    Scale (SOFAS)

11. Social Adjustment Scale (SAS)    No                No

12. Functional Status Index (FSI)    No                No

13. Specific Level of                Moderately        Satisfactory
    Functioning (SLOF)               satisfactory

14. Multnomah Community              Yes               Yes
    Ability Scale (MCAS)

15. Short Form 36 (SF36)             No                Yes

16. Rehabilitation Evaluation        No                No
    Hall and Baker (REHAB)

17. Assessment of Motor and          No                No
    Process Skills (AMPS)

18. Occupational Case Analysis       No                No
    Interview and Rating
    scale (0CAIRS)

19. Clinical Outcomes in routine     No                No
    Evaluation--Outcome
    Measures (CORE-OM)

20. Personal and Social              Yes               Yes
    performance Scale (PSP)

21 Canadian Occupational             Yes               Yes
   Performance Measure (COPM)

22. Work and Social                  No                No
    Adjustment Scale

23. Scales of Independent
    Behaviour--Revised (SIB-R)

24. Kenny self-care evaluation       No                 No

25. Adaptive Behaviour               Yes                Yes
    Assessment Systems (ABAS II)

26. Life Skills Profile              Not rated          Not rated
    16 (LSP 16)

Table 3

An outline of the five short listed instruments with their strengths
and weaknesses in relation to the functional criteria.

1    Canadian Occupational         Has been rated good tool--but
     Performance Measure (COPM)    self-rated, it met the criteria but
                                   had to be excluded due to it being
                                   a self rated tool

2    Personal and Social           This the new version of the sofas;
     performance Scale (PSP)       good validity and psychometric
                                   properties, raters found this tool
                                   to be best suited to measure
                                   functionality and would work well
                                   with the honos. met all criteria
                                   well

3    Social Functioning            A good tool but we found that it
     Scale (SFS)                   lacked cultural bits, query whether
                                   it is self rated

4    Camberwell Assessment         This tool has been rated good on the
     of Need (CAN SAS)             communication bit, some clinical
                                   phenomena, rated by consumer, staff
                                   & caregiver--but it was too
                                   time consuming

5    Multnomah Community           This was a good tool but we found
     Ability Scale (MCAS)          that the self care dimensions were
                                   Missing