Competency assessment in a team-based laboratory.

Three years ago, this large, private laboratory had to comply with emerging accreditation requirements and institute a program to assess the competency of approximately 140 FTEs. Working within a team-based structure, they used some assessment tools already in place as well as created new ones. Here's how.

Three years ago, during our CAP inspection, the inspectors asked us how we knew that our staff members were competent. At the time, this concept was fairly new as it had not yet become one of CAP's requirements. I confidentially answered that we knew this because our interviewing and hiring process is very thorough. We only recruit those candidates with the necessary credentials; in addition, we check references, ask technical questions during the interview, and conduct performance appraisals. After an employee is hired, the company also provides many opportunities for in-house and external education.

The qualifications of the staff were not the issue, however. Competency refers to learning specific procedures and protocols and performing duties that can be dynamic. The inspector was looking for proof of competency through observation and documentation. How did I know that the technologists working on the benches were really competent to do that work? Had they been observed while working or had they taken written exams to prove that they were competent?

The inspector went on to tell me that in order to assess the competency of her staff, she spent one day a year with each technologist observing their work and asking them questions about the manual. This started me thinking about how we could introduce a competency assessment program into our laboratory. This thought wouldn't remain hypothetical for long because next year's CAP inspection would include a mandatory checklist asking if the competency of each person to do his/her job had been assessed. To fulfill this requirement, we were then faced with the task of setting up a competency assessment program.

Laboratory structure

MDS Metro Laboratory Services is a private facility that serves approximately 50 outpatient service centers in the area in Burnaby, British Columbia. These centers send all their patient samples to our main laboratory, which employs approximately 140 FTEs and has a volume of approximately 6 million tests per year. Just over 3 years ago, our laboratory moved away from the traditional structure to a team-based structure where staff members are assigned to teams based on the type of work they perform. Each team has a leader who deals directly with the lab manager and a medical staff member (a pathologist, MD, or PhD) who oversees the team. Every team member is responsible for the work done on that team. Sizes of the teams vary from 12 to 30 people.

For example, most of the high-volume testing is performed on automated analyzers and these are gathered together on one team called the autoline. The remaining testing is considered mid- to low-volume; these teams are appropriately called mid/low 1-4: the low volume, special chemistry, and immunology procedures team (1); manual and low-volume hematology testing team (2); and 2 microbiology teams (3 and 4). The analytical support team handles and sorts the arriving specimens.

I am a member of a small team called quality resources; one of our responsibilities is to ensure that the laboratory meets all accreditation and regulatory guidelines and policies. To set up a competency assessment program within this team-based structure, I first approached each team. I explained that we were setting up a new team to implement a competency assessment program and asked for volunteers within each individual team to be responsible for that team's assessment. I would oversee the competency assessment team to ensure that the lab was prepared for the next inspection.

Competency assessment team

At the first competency team meeting, I went over with the volunteers the basic definitions and rationale for this new program to avoid any misunderstanding or suspicion. First, I defined competency assessment as documenting the ability of the laboratory staff to perform their assigned duties. This definition then, of course, led to explaining the necessity of implementing such a program now and answering the question: Hasn't our staff always been competent?

I stressed that our staff members have always been competent, but this program has become important for several reasons. First, in the competitive marketplace of today, we need to demonstrate that all of our processes, from specimen pick up, to phlebotomy, to test performance, are monitored for quality and that competent personnel perform these services. Another reason is that the scope of laboratory testing is constantly changing. One result of widening laboratory responsibilities is considerable cross training; but despite training employees in multiple areas, there are few tools available for continual self-evaluation and therefore improvement. In addition, CAP accreditation requires that laboratories evaluate and ensure the competency of all personnel who perform laboratory testing

Benefits of competency assessment.

After assuring fellow team members that the purpose of the new assessment program was education and improvement rather than accusation and punishment, I listed the additional benefits of the program as follows:

* provides performance feedback to employees

* identifies performance problems early before they develop into major problems

* promotes performance consistency among the technologists

* encourages employees to carefully and critically read manuals

* pinpoints systems or processes needing improvement

* reminds staff of important information that may have been forgotten or missed during orientation

* demonstrates to employers, employees, inspectors, and clients that all personnel are qualified to perform assigned tasks, and

* ensures quality patient care.

Methods for evaluating competency.

Next, I presented different methods we could use to assess competency. The various methods chosen would allow us to evaluate employee knowledge from many angles to gain a comprehensive idea of staff members' aptitude and understanding. The volunteer for each team would be expected to select one of the following and implement it within his or her team:

1. Directly observe routine test performance, including patient preparation (if applicable), specimen handling, processing, and testing.

2. Monitor the recording and reporting of test results.

3. Review of intermediate laboratory test results or worksheets, quality control records, proficiency test results, and preventive maintenance records.

4. Directly observe the individual checking functions on instruments and maintaining instrumentation.

5. Assess test performance through testing of previously analyzed specimens, internal blind unknowns, and proficiency testing samples.

6. Evaluate problem-solving skills. [1]

Creating a plan

In addition to implementation, each volunteer would be expected to develop these objectives in a manner relevant to their team's area. Copies of any articles that I had managed to collect on competency assessment were handed out to help with this assignment. Each team member was asked to come up with a plan that included frequency, method of assessment, and grading, which included suggesting a passing mark. The importance of documentation was stressed. Although CAP did not give a lot of details with regard to how to measure competency, we were sure that documentation would be required. At that point, we had 4 months to go until our CAP inspection date, but we had a mock inspection planned in 2 months, which was the deadline for the plan to be written and for testing to be underway but not necessarily completed.

In discussing program implementation with team volunteers, we discovered that we already had some good internal assessment programs in place that could be used.

Hematology quality assurance slide program. This program assessed the technologists' ability to read blood films and encouraged the standardization of reporting among the various technologists working in our laboratories. As interesting cases appeared in the laboratory, the technologists were encouraged to send 50 unstained slides plus the hematology results, histogram, and any history or diagnosis to the quality resources team. From these cases, 3 would be selected, and a critique of the patient's condition would be written by a member of the medical staff. The critique and slides were then distributed to all the technologists reading blood films. Each employee would evaluate the slide set by doing a differential, red blood cell morphology, platelet estimation, and morphology. Staff members were given 2 months in which to complete the work and were encouraged to do them independently. This was done 3 times per year.

The results for the differential would be graded using the 95% confidence limits, and the morphology allowable limits are +1- 1 unit of the average results of the marking technologists. Serious deficiencies such as a lack of identification of clear-cut blasts or malaria would result in remedial education. Summaries of performance would be prepared and signed off by the team leader, quality resources team, and the medical scientific staff member.

Urinalysis microscopic QA program. Weekly, all staff members reading urinalysis microscopics read the same sample and their answers are compared to ensure that all results were consistent. Allowable limits are +1- 1 unit of the reported results. If discrepancies occurred, the slides would be reviewed and discussed with the pathologist.

Review of proficiency tests in parasitology. We decided we could also use proficiency testing results to assess competency. After the test results were submitted to the testing agency, each employee would independently read the slides and submit the results to a competency assessment team member. When the results were returned from the agency, each team member's results would then be compared with the correct answers and results would be documented.

The team realized that we had a good start with these programs. Some needed to include more documentation, which could easily be added.

Implementation

Presenting the program. Once the members of the competency assessment team were familiar with the concepts involved, and we had a general outline of our plan, our next step was to get the support of the teams. Our biggest fear was that the staff would feel intimidated and threatened. Our goal was to improve performance. We felt that corrective action should be taken in the event that someone failed, but felt our focus should be educational and not punitive.

At each team's regularly scheduled meeting, the competency assessment representative introduced the new program and defined the role of the competency team. To further introduce the idea, I gave a short presentation about our competency assessment program, which basically reiterated what we had discussed in our first meeting, including a definition of the program, its purpose, methods of assessment; current programs in place, and the team itself.

We also made clear that employee input into the assessment process was not only welcome but encouraged. We felt that if the team members had some input into how it was done, they would be more receptive to the idea. It was the responsibility of the competency assessment team volunteers to involve the rest of their individual lab teams in developing a program so that all staff members would be more committed to it.

I made sure to leave time for the staff to ask questions and express concerns. The biggest concern seemed to be confidentiality. Employees didn't want their fellow coworkers to know how they had done. This was easily remedied by assigning each person a number to use instead of his or her name. Their team leader and the medical staff member responsible for that team would be the only ones to see the results.

Program development The competency assessment team met a few times as a group to discuss any problems that anyone was having and to share information. If one team had a good idea, frequently other teams used it too. We also tried to standardize some of the forms used for documentation.

Although we recognized that direct observation of staff performance can be an important part of competency assessment, we felt that observing and documenting staff performance would not only be intimidating but costly. There were also concerns about who would be responsible for observation. Many volunteers felt that it could jeopardize their relationship with their co-workers. In a traditional lab setting, this task would fall to a supervisor, but our laboratory no longer had supervisors. As previously mentioned, each lab team did have a team leader, but few of these individuals had volunteered to be on the competency assessment team. They also expected the competency assessment team to implement the program.

Written quizzes were developed for different areas of the lab, especially in the highly automated areas of the laboratory. In some cases, the competency assessment team member developed the questions for that team. In other instances, each team member submitted questions, and a medical staff member then selected the questions for the final quiz. The questions were bench specific and designed to test an individual's knowledge of that bench (see Figure 1).

Each quiz usually contained 4 or 5 questions. The answers could all be found in the procedure manual and employees were required to list the manual page number in addition to the correct answer. This format ensured that employees actually read the manual even if they knew the answer. In addition, staff members were encouraged to indicate any areas of the procedure manual that were unclear, out of date, or missing necessary information. The process assisted in keeping manuals current. To pass the quiz, employees needed to have at least 80% correct.

Instead of a written quiz, one of the microbiology teams chose to assess the results of practical bench work. Each technologist was given stained smears that all technologists were required to read. Results were compared and an acceptable result was considered to be within 1 unit of the reported result. For example, if moderate white blood cells were reported, a reading of few or many would be acceptable. The same idea was used with testing culture readings. An organism was given on a swab to each staff member for identification and sensitivity. Staff members were also required to read and record the results of the same trichomonas broths and KOH slides. Assessing employee ability in this manner helped to promote standardization in the reporting of Gram stains from technologist to technologist.

Competency assessment of technicians

In addition to our technologists, we felt that it was important to know that our technical assistants, who give out patient results and sort, accession, and prepare specimens for testing, were also competent. The technical assistants who were responsible for setting up and receiving the microbiology specimens and who worked in client service helped develop comprehensive quizzes in those areas.

Documentation

After the competency testing, each staff member receives their marked quiz and marks are recorded for each staff member. These marks are reviewed and signed off by the team leader, the medical staff member assigned to that team, and me. In addition, results of all other methods of competency assessment, such as the slide program mentioned previously, were also documented and reviewed.

Improving competency

We did have concerns about how to deal with the event of an unsatisfactory performance. So, I worked with our human resources department to come up with a process. As soon as it is recognized that an employee has not achieved the minimum requirements, that person's team leader decides if it is necessary to remove the employee from testing on the bench(es) until competency requirements have been met. The team leader then meets with the employee to discuss improvement goals, including:

* the importance of competency assessment

* his or her understanding of job duties and expectations

* the reasons why he or she failed to meet the minimum requirements and what assistance may be required (ie, more training).

Together, the employee and the team leader come up with a plan and set timelines and a date for another competency assessment review. They also document the situation, action taken for improvement, and results (see Figure 2).

Employee feedback

Because this was going to be an annual occurrence, I was interested in getting feedback from our employees. I wanted to know what changes we should make for the next year and was also interested in introducing the program to some of our other laboratories. We thought that the program had gone fairly well but felt that any feedback could only improve the program. A few months after competency was finished, we sent out a survey asking the following:

1. Which competency assessment activity did you take part in? Which team?

2. Did you feel intimidated by the process? Feel free to elaborate.

3. Did you learn anything as a result of this? What did you learn?

4. Do you have any suggestions for improvements?

The results from the survey were very encouraging. Twenty-six responses were received out of 66. Only 1 employee stated that he or she was intimidated. Two employees said they were intimidated "a little" until it was discussed as to why this was being done. Most employees said that they had learned something, usually about reading the manual. A few had recognized areas in the manuals that needed updating. The suggestions for improvement mainly centered around the questions on the quizzes and varied from team to team. Some felt that the questions should be more exact and deal with "real life" situations while others felt that the questions should be more theoretical. All this feedback was shared with the competency assessment team members the next year.

Results

Our program has been running for 2 years. I was impressed with how quickly the team members implemented the programs the second year. One team went ahead on their own and developed quizzes, wrote them, and marked them before the other teams had decided what they were going to do. The second year we only had to meet as a group once, but I asked for progress reports every 2 weeks so that I could monitor progress and provide assistance where required. At our meeting, we went over the results of the feedback survey and made changes according to employee suggestions.

One big benefit has been that finding an answer to a question on a quiz helps ensure that staff members read the manual. In addition, more errors were found and more changes to the manual were made after competency than after the staff simply reviewed the manual. Our competency team volunteers also got the opportunity to take part on a different team and get to know others working in other areas of the laboratory. The enthusiasm spread and during Safety Awareness Week, a quiz was sent out testing the staffs' knowledge of the Safety Manual.

Problems. The major problem encountered the first year was convincing staff members to take competency seriously. Those that didn't take it seriously quickly jotted down an answer without checking the manual; they subsequently lost points, which resulted in an unsatisfactory grade. The second year, everyone had a much better understanding of competency and took it very seriously. Another problem was checking the competency of the person who made up the test. This problem could be solved either by having all staff members submit questions and answers, with the medical scientific staff member choosing which questions will appear on the final quiz or by having a different team member make up the quiz every year.

Another problem was that competency was treated as only an annual requirement. We also needed to test employees competency after training--when new methods, tests or equipment is introduced, and when an employee is retraining--not just once a year. To make sure that a new employee gets tested for competency shortly after their training, we have put competency assessment on our new employee training and orientation checklists. Because we also have checklists for when new methods and instrumentation are introduced, competency assessment was also added to this checklist.

Future

Now that the program has been successfully introduced and implemented in one of our laboratories, we plan to introduce it at our next largest laboratory. We may be able to use the same quizzes in both laboratories, and this will be determined by the appropriate team in that laboratory. It would also be advantageous to computerize all the quizzes and place the results on our Intranet site, currently in development. Access to results would be restricted to certain individuals.

Competency testing can be introduced in a team-based (or any) laboratory without intimidating employees and on a limited budget, by involving the staff. We have found that involvement usually means a greater commitment to the process. Communicating your expectations to staff members is essential for a successful competency program. In a constantly expanding environment, we can not assume the competency of staff members based on education and credentials alone. At our laboratory, we can now demonstrate to our clients, management, and inspectors that the lab has competent staff.

Sharon Gerbasi is quality assurance specialist at MDS Metro Laboratory Services, Burnaby, BC, Canada

Reference

(1.) Medicare, Medicaid, and CLIA Programs; Regulations Implementing the Clinical Laboratory Improvement Amendments of 1988(CLIA), Department of Health and Human Services, Health Care Financing Administration, Federal Register, Feb. 28, 1992; 57(40):7,180-7,181.