Review sessions and results: competency testing in the capstone business class.

Capstone is defined in the dictionary as "putting the final stone in place". Business Policy or Strategic Management is typically the capstone class in a business management degree program. As such, there are prerequisite requirements in terms of courses and specific knowledge. We have implemented a competency test for the capstone class and tested various methods of administration. We used logistic regression to test the various methods for first-time pass and ultimate pass rates of students. Our results indicate that while there was a difference in first time pass with and without review sessions, there was no difference in the ultimate pass rate when review sessions were held. We also found that infinite opportunities for taking the exam made no significant difference in the ultimate pass rate.

**********

Bloom's learning for mastery (1968; 1974) was introduced in the 1960's and early 70's and formed the foundation for educational research in the area of learning. Bloom's research was built on earlier work by Carroll (1963). In the 1990's, several papers in a stream by JR Gentile (Gentile, Voelkl, Pleasant, & Monaco, 1995; Livingston & Gentile, 1996) refined some of the earlier ideas by Bloom and later enhanced by Block (1974; 1987) into a practical approach to mastery learning that included competency grading.

The idea of mastery learning is quite common in primary and secondary education. Mastery learning focuses more on process than on content and is founded on the idea that students learn at different rates, but still master the subject matter. Thus, all students will achieve the same level of competency, given enough time and the opportunity to learn (Carroll, 1963). Mastery learning is defined as "defining an objective, teaching that objective, testing specifically what was taught, and finally, enriching that objective and/or re-teaching it if mastery wasn't accomplished" (Walker, 1998, p. 15). By extrapolation, competency testing is the assessment component noted in the definition. Thus, students must demonstrate "competency" on a test in order to have achieved mastery of the topic.

While mastery learning theory and competency testing are routinely applied in a variety of areas such as criminology (Kessler & Swatt, 2001), science (Camacho, 1995; Lazarowitz, Baird, & Bowlden, 1996; Lazarowitz, Hertzlazarowitz, & Baird, 1994; Walker, 1998), health care (Decker, 1990; Goodwin, 1997), and licensing situations or other areas in which content is cumulative (Kessler & Swart, 2001), it does not appear to have gained popularity in the teaching of business administration. However, there are some recent citations on the use of comprehensive testing in the information systems arena (Corbett & Anderson, 1992; Montazemi & Wang, 1995; Mukherjee & Cox, 1998).

While the cumulative nature of learning in a business curriculum is obvious, what is not obvious is that the cumulative building process needs to occur between and among the sub-disciplines of business (e.g., accounting, marketing, operations). In addition, many business programs routinely place the onus of integration and synthesis on the capstone class (strategic management or business policy). However, it may not be until well into the capstone class that we find out that, for some students, the foundation is weak.

The term "capstone" describes the final stone or finishing touch, just as the capstone was the final stone put in place before a tomb was closed in ancient Egypt. The capstone was placed after the foundation was solid. In a business program, the fundamental functional areas are mastered first before moving to the capstone course (Stephan, Parente, & Brown, 2002). Thus, what better place to test student competency of the foundation course material than at the beginning of the capstone class?

Two of the authors began using competency testing in the capstone course nearly eight years ago, while the third began more recently. During the early weeks of the class, all students are required to take an exam on the fundamental principles of the core areas of business. The core courses in business typically include management, marketing, statistics, economics, operations, social issues, accounting, finance, and management information systems. The areas and topics for the exam include, but are not limited to, finance (net present value, weighted average cost of capital, valuation), statistics (probability, expected values), and operations (forecasting, scheduling). (Please see the topic review sheet in Appendix 1.) The test includes problems and definitions as appropriate for the topics. We expect students to be able to apply some of the previously learned topics directly in the capstone class (e.g., NPV, forecasting, and decision trees). We can categorize these topics as tools or techniques. Other topics are indirectly applied (e.g., business ethics, marketing principles, and supply/demand concepts). These topics may be categorized as concepts and are used in the analysis of cases and discussions of current management practice.

One objective of the test is to identify areas in which student preparation is strong or weak. Instructors will commonly require students to shore up their knowledge in specific areas either through independent review or sessions with the instructors. A second objective is to set clear expectations for the students that they must have specific basic skills to be successful in the course and to succeed in business. We often say that the competency exam makes the students "sit up and take notice". We let them know that we are not going to re-teach the material and they are expected to know it. Likewise, if they do not have the tools to enter into a capstone class, perhaps they are not ready to complete the business degree.

The Competency Exam

Although the material was not difficult (the problems were well within the basics of each of the disciplines) and students had access to sample exams (see, for example, Appendix 2), students were typically apprehensive. We understand this was due in part to the performance penalty associated with not passing the test and that the students may have last covered the material several years prior to the capstone class. Other anxiety was not well-founded: students complained that the penalty for failing the exam in some sections was much heavier than in others; but we had coordinated the penalty so that the effect was identical across sections, raw points notwithstanding.

The exam was approximately 75% problems and 25% definitions. The instructors of the course agreed on the topics to be tested across all sections based on the likelihood of using the topic within the course. Instructors prepared similar questions for 4-6 versions of the test. As an example, one instructor would prepare the NPV questions for all sections of the course and grade all students for that question, while another instructor would prepare and grade a break-even problem. This approach insured consistency across sections and reduced potential instructor bias. Students were required to pass the test or accept heavy penalties in their final grade. In the spirit of mastery learning, students could re-take another version of the test and could theoretically take the test until they "pass" or achieve mastery.

As our objective was to use competency testing to make the class experience more rewarding, we experimented with several approaches to the process. In the autumn semester 2001, we administered the test having provided students with only sample exams as a guide to what we expected. We followed up the initial exam with voluntary review sessions, covering finance and accounting in one session, and operations and statistics in another. The reviews were conducted by faculty from the appropriate discipline and were held outside of normal class hours. Students were permitted to take the follow-up exams up to four times to demonstrate competence, with the tests again given outside normal class hours. We note, in this approach, that students often viewed the first exam as a way to find out what we "really" wanted. In fact, there were still a number of students taking the exam for the fourth time in the last week of the semester.

In the spring semester 2002, we changed our approach to emphasize early mastery. Since many of the tools we covered in the examination were used early in the semester, we argued that providing competency testing opportunities into the end of the semester was not achieving what we wanted in terms of student learning. Consequently, for the spring semester, we informed students that they would have only two opportunities to pass the exam. However, we also switched to providing the review sessions in advance of the first test. We hypothesized:

H1: The ratio of success on the first exam would be significantly higher with pre-test reviews.

H2: The overall ratio of student success (overall pass) would be unaffected by review timing.

Given the exam topics discussed above and the weight on problems versus definitions, we anticipated that students in certain disciplines might fare better than others. While the problems were typical of material all students covered in the core courses, students from the more problem-oriented majors (e.g., accounting, finance, and economics) would have had the opportunity to attack and solve problems more frequently over their academic careers. Thus, we hypothesized:

H3a: Students with problem-oriented majors would have a higher initial success rate.

H3b: Students with problem-oriented majors would have a higher ultimate success rate.

Methods and results

We tested these beliefs using data from all classes for autumn 2001 and spring 2002 with a total of 206 students. Because the outcome is dichotomous (pass/fail in the first test and ultimately pass/fail), we estimate a logistic model where the probability of success is:

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]

(Hosmer & Lemeshow, 1989)

The independent variables included in the study were: semester, pre-test review sessions, and student academic major. In the latter case, we constructed a dummy variable for each major. Because the number of students in both international management and business/liberal arts majors was low, we included them in the Management major based on similarity in curriculum. We also included controls for the professor teaching the class and the student's GPA entering the class. Results for the regression are presented in Table 1, where the coefficients represent the logit transformation of the logistic regression. That is, the coefficients are those in g(x) so that the marginal change in the odds ratio of a unit increase in any variable is [MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]. Model A uses first time pass as the dependent variable while Model B uses ultimate pass. The goodness of fit of the models were assessed as recommended by Hosmer and Lemeshow (1989). Observed outcomes were compared to predicted outcomes based on deciles of risk, generating a summary measure of the differences. Goodness of fit can be assessed by a chi-square statistic where a value greater than p >0.05 is considered acceptable (Lipsitz, Fitzmaurice, & Molenberghs, 1996). In this case, the Hosmer-Lemeshow statistics are p>0.65 and p>0.47, respectively, suggesting adequate fit for each model.

Model A shows that the review sessions prior to the first test did improve the first time pass rate significantly (p<0.000) but the results from Model B show that, ultimately, one approach was just as effective as the other. These results support Hypotheses 1 and 2.

Results of the various control variables are also shown in Table 1. The professor variable was not significant in either model, indicating no special effect on the part of the instructors. Grade point average was significant in both models: better students were significantly more likely to pass the first time and to pass ultimately. The effect of major was as predicted. The logit analysis used indicator variables for majors and assessed the likelihood of first time and ultimate success versus the base category of accounting majors. We found that management and MIS majors were significantly less likely to pass the first time through the examination (p<0.012 and p<0.015, respectively). This difference became much more pronounced when the dependent variable was ultimate pass. The number of usable records dropped from 200 to 163 because majoring in finance or economics accounted for ultimate success perfectly--that is, none of these majors experienced an ultimate failure. Again, students who ultimately failed the exam and were subjected to a performance penalty were predominantly from management, MIS and marketing--majors that use discrete problem solving pedagogical techniques less often than other majors (p<0.001 in all cases).

Discussion

The authors used competency testing in their capstone Strategic Management course to assess the mastery of both "tools/techniques" learned in common core courses, as well as conceptual knowledge from core courses. During autumn 2001, the competency exams were offered throughout the course, and review sessions were conducted after the first exam was given. In spring 2002, students were given only two chances to pass the exam, but review sessions were conducted prior to the administration of the first exam.

The format and content of the exams, themselves, were the same for each semester. The content included only material from common core courses--those courses required of all School of Business students. regardless of major within the school. The exams included problems from the areas of finance, statistics, and operations, which required computational solutions. Topics from other core courses were tested by requiring the students to define terms and/or provide examples of terms/concepts.

Consistent with our hypotheses, we found that by providing the review sessions prior to the first exam, the passing rate on the first exam was significantly higher than with no prior review sessions. However, the rate of ultimate success (ultimately passing the competency exam) was not affected by the timing of the review sessions. We also found that the rate of ultimate pass was not dependent on the number of times the competency test was offered.

There were significant differences among majors with respect to rate of ultimate pass on the competency exam. As hypothesized, students in majors that emphasized relatively more problem-solving (e.g., finance, accounting, economics) had higher ultimate pass rates than students from other majors. An additional explanation for these results may be that MIS and Management majors have not yet come to recognize the "need" for mastery of computational techniques such as NPV or decision analysis. Indeed, one of the goals of the capstone course is to help the students appreciate the value of all the functional areas--and their related "tools"--and recognize the need for integrating this knowledge.

Lessons Learned

One of the important lessons we learned was that "infinite re-testing" to establish mastery was not appropriate in our setting. First of all, since we wanted all students to demonstrate competency over core course topics early in our course, providing multiple opportunities to pass a competency exam throughout the course was inconsistent with our "early mastery" goal. In addition, we found that the rate of ultimate pass on the competency exam was the same when we offered several attempts as when we offered only two opportunities to pass.

We also found that conducting review sessions before the first exam improves the first pass rate. It appears that combining these early review sessions, along with only offering two chances to pass the exam, provided incentive for students to prepare for the first exam, rather than taking the first exam--with little preparation--to simply "test the waters".

Future Research Questions

While we reduced the number of chances to pass the competency exam, and also changed the timing of the review sessions, future research should examine different incentives or rewards and their impact on outcomes. Such incentives as bonus points for early pass and/or penalties for late pass or failure should be examined. Likewise, it would be interesting to see whether attendance at review sessions actually impacts pass rates.

Different formats for the competency exam should also be examined. Our current exam format features several computational problems along with definition of terms. The potential bias in our current format and content may also introduce bias in who passes--both early and ultimately. Examining student performance, by major, on exams with different formats and/or content should be informative.

Future research should also include additional control variables, such as age and work experience, to examine their relationship with pass rates.

Appendix 1

Competency Topics

Capstone Class Competency Requirements

(What you need to know before coming to this course)

1. Working knowledge of Accounting including:

a) Debits & Credits

b) Financial Statements including Income Statement, Balance Sheet and Cash Flow Statement

2. Working knowledge of Finance including:

a) Return on Investments (e.g., ROA, ROE) and how to calculate them

b) Discounted cash flow (NPV) and how to calculate it

c) Expected value

d) Descriptive Statistics (mean, standard deviation, frequency distribution, e.g.)

e) Two sample statistical test

f) The identity and importance of shareholders and creditors

g) How a company creates value for its shareholders

h) WACC

3. Understanding of basic supply and demand Economics (e.g., effects on prices and competition when supply > demand)

4. Understanding of aggregate production planning including:

a) Production, shipments and inventory analysis

b) Forecasting tools

c) Capacity planning issues

5. Understanding of basic concepts in Marketing including:

a) Components of the Marketing Mix

b) Break-even analysis (x)

c) Pricing strategies

d) Forecasting methods (including the use of regression, wgtd. average, etc.) *

6. Understanding of Organization Structure

a) Basic departmental functions in an organization (What are they? What tasks do they perform? An example would be the accounting departmental function.)

b) Individual managerial functions (e.g., What are the typical responsibilities of a CEO?)

7. Ability to write a good business letter

a) 100% accurate spelling

b) Proper punctuation

c) Appropriate word usage

d) Correct form (e.g., salutations, date. closing)

8. Ability to function effectively in a team environment

9. Familiarity with PCs and application software

a) Graphics (e.g., Powerpoint) and Spreadsheets (e.g., Excel)

b) You should be able to produce a professional document on a PC

(x) May have been covered in Finance

(*) May have been covered in Operations Management

Appendix 2

Sample Competency Exam

1.) Define the following terms (2% each):

a. Gross Margin

b. Internal rate of return

c. Rate-of-return Pricing

d. Return on sales

e. Market share

f. Marketing channels

g. Capacity

h. Job Shop

i. MAD

j. Economic barrier to trade

k. EDI

l. Chain of command

2.) Using a weighted moving average with weights of 20-20-60, provide a unit sales forecast for your company's Quarter 7 sales based on the data below:

Quarter   Units Sold

1           1,100        --   --   --   --
2           1,300        --   --   --   --
3           1,250        --   --   --   --
4           1,400        --   --   --   --
5           1,700        --   --   --   --
6           1,650        --   --   --   --
7

Your sales manager has asked you to evaluate whether your company's weighted moving average forecast methodology is really any more accurate than a 3-period simple moving average. Use MAD to assess which forecasting methodology is better, given your company's sales data. (12%)

3.) RBT (Really Big TVs) manufactures four sorts of screens. They have the following cost structure (12%):

Product    FC per year   VC per unit   Price per unit   Annual
                                                        Production
                                                        volume

Plasma        $100,000        $1,200           $1,500        1,200
CRT            $50,000          $225             $325        1,000
LCD            $75,000          $950           $1,000        3,000
NKT           $125,000        $2,500           $3,000          400

1) What is the breakeven number of units for each product?

2) Which process will breakeven first and in which month?

4.) (18%) Yinztel, a Pittsburgh-based computer peripheral device company, has developed a radically new approach to solving the human-machine interface problem: virtual mousing. This could take the industry by storm and Yinztel has secured a patent on the device. On the down side, success might be blocked by alternative solutions (of which there are several) or a general slump in the computer industry. The company is considering several options, Production of the VM device will require a complete re-vamping of production processes at a cost of $10 million. If the market accepts the product, Yinztel expects profits to ultimately reach $50 million. However, if the market does not develop, the company expects no sales at all. The firm could also just continue with its current product line (with an expected $20 million in profits) but management expects that other firms will develop alternatives to mousing and Yinztel could be shut out of the market. If so, Yinztel's profits would be reduced to $4MM. Finally, a much larger firm has indicated an interest in buying Yinztel for cash. Management expects that if they enter with the VM, there is a 60% chance of success. If they stay with current technology, there is only a 40% chance they will survive.

1) Which is better for Yinztel: invest in the new technology or stay with the old?

2) What is the minimum price Yinztel should ask if they decide to sell the company to the larger firm?

5.) Hayes International is a global retail clothing company. Use the following information to calculate Hayes' WACC. (15%)

Common Stock         10 million shares outstanding
                     Market price = $23
                     Book value per share = $16
                     Cost of equity = 15%
                     Most recent dividend = $1.80
                     Expected Divident Growth rate = 6%

Debt                 Par (Book) Value = $150 million
                     Market Value = 70% of par value
                     Maturity = 20 years
                     Cost of Debt = 11%
                     Annual coupon rate - 10% (with sem-annual
                     coupon payments)

Marginal Corporate
tax rate:            50%

6.) The Hokie Company is considering a new expansion project. It will require an initial outlay of $10,000 and will operate for 5 years. The project is expected to generate after-tax cash flows of $3000 per year for years 1 through 5. The project is expected to have zero salvage value. Hokie Corporation has decided to apply a required rate of return of 12 percent to this project.

Calculate the NPV for this project. Should Hokie make the investment? Explain why or why not. (15%)

References

Block, J. H. (Ed.). (1974). Schools, Society and Mastery Learning. New York: Holt, Rinehart and Winston.

Block, J. H. (1987). Improving Student-Achievement through Mastery Learning-Programs--Levine, Du. American Journal of Education, 95(4), 622-625.

Bloom, B. S. (Ed.). (1968). Mastery learning. New York: Holt, Rinehart & Winston.

Bloom, B. S. (1974). Time and learning. American Psychologist, 29, 682-688.

Camacho, M. (1995). The Effect of a Mastery Learning-Strategy on the Achievement of Science Students in College Chemistry. Abstracts of Papers of the American Chemical Society, 209, 125-CHED.

Carroll, J. (1963). A model of school learning. Teachers College Record, 64, 723-733.

Corbett, A. T., & Anderson, J. R. (1992), Student Modeling and Mastery Learning in a Computer-Based Programming Tutor. Lecture Notes in Computer Science, 608, 413-420.

Decker, B. (1990). Implementation of the Mastery Learning Modular Curriculum in Nurse-Midwifery Education. Journal of Nnrse-Midwifery, 35(1), 3-9.

Gentile, J. R., Voelkl, K. E., Pleasant, J. M., & Monaco, N. M. (1995). Recall after relearning by fast and slow learners. Journal of Experimental Education, 63(3), 185-197.

Goodwin, L. (1997). Performance support concepts for Web-based informatics instruction. Journal of the American Medical Informatics Association, 698-702.

Hosmer, D. W., & Lemeshow, S. (1989). Applied Logistic Regression. New York: John Wiley & Sons, Inc.

Kessler, D. A., & Swatt, M. (2001). Mastery learning, rewriting assignments and student learning of criminal justice research methods. Journal of Criminal Justice Education, 12(1), 127-146.

Lazarowitz, R., Baird, J. H., & Bowlden, V. (1996). Teaching biology in a group mastery learning mode: High school students' academic achievement and affective outcomes. International Journal of Science Education, 18(4), 447-462.

Lazarowitz, R., Hertzlazarowitz, R., & Baird, J. H. (1994). Learning Science in a Cooperative Setting--Academic- Achievement and Affective Outcomes. Journal of Research in Science Teaching, 31(10), 1121-1131.

Lipsitz, S. R., Fitzmaurice, G. M., & Molenberghs, G. (1996). Goodness-of-fit tests for ordinal response regression models. Applied Statistics, 45(2), 175-190.

Livingston, J. A., & Gentile, J. R. (1996). Mastery learning and the decreasing variability hypothesis. The Journal of Educational Research, 90(2), 67-74.

Montazemi, A. R., & Wang, F. (1995). An empirical investigation of CBI in support of mastery learning. Journal of Educational Computing Research, 13(2), 185-205.

Mukherjee, A., & Cox, J. L. (1998). Effective use of mastery based experiential learning in a project course to improve skills in systems analysis and design. Journal of Computer Information Systems, 38(4), 46-51.

Stephan, J. D., Parente, D. H., & Brown, R. C. (2002). Seeing the forest and the trees: Balancing knowledge using large scale simulations in capstone business strategy classes. Journal of Management Education, 26(2), 164-193.

Walker, M. H. (1998). 3 basics for better student output. The Education Digest, 15-18.

Diane H. Parente, Associate Professor of Management; Randy C. Brown. Lecturer in Management & Finance; Alfred G. Warner, Assistant Professor of Management, Penn State, Erie.

Correspondence concerning this article should be addressed to Diane H. Parente, Penn State Erie, School of Business, Station Road, Erie, PA 16563; Email: [e-mail].

Table 1
Results of Logistic Regression Analysis

                      Model A      Model B

Review Sessions       3.68         -.38
                      (.62) ***    (.53)

Professor             -.57         -.32
                      (.44)        (.53)

GPA                   2.57         1.07
                      (.59) ***    (.62) *

Economics             -.22         --
                      (1.29)

Finance               .78          --
                      (1.09)

Management            -2.1         -15.96
                      (.87) **     (1.88) ***

MIS                   -1.85        -17.48
                      (.81) **     (1.82) ***

Marketing             -1.32        -16.21
                      (.92)        (2.10) ***

Constant              -9.37        16.45
                      (2.17)       (1.50)

N                     203          174
LR Chi Square         125.52       12.10
P> chi square         .000         .06
Pseudo R square       .48          .11
H-L chi square        7.82         5.90
p>H-L chi square      .45          .66

* p<.10

** p<.05

*** p<.01