Printer Friendly

A stakeholder analysis of one institution's assuring learning experience - is the promise and the practice worth the price?

ABSTRACT

Universities around the world are rushing to implement assurance of learning policies and practices with varying degrees of success. One School investigated its own policy and practice development through the eyes of its key stakeholders to identify whether the practice was worth the price. Findings indicate that although the key stakeholders considered different needs and viewed their experiences differently, value did abound and was in the eye of the beholder.

Keywords: Assurance of learning; assessment; accreditation; HERN (higher education researcher's network)

Introduction

The process and practice of assuring learning in business schools in Australia has been under some debate for 10 years. Indeed, it is one of the most frequently discussed topics in tertiary education today (Martell and Caldron, 2009). Most Australian universities are undertaking the practice is some fashion and with various reasons from accreditation to quality outcomes (Lawson, Taylor, French and Ors, 2014). Defined as the process by which educational institutions measure the learning outcomes of students against a specific set of program goals (Hall and Kro, 2006), the results are most often used to demonstrate the attainment of learning outcomes to interested stakeholders. Lawson et al. (2014) found two different approaches of implementation in Australian universities: the top down approach driven by management who employ experts to develop rubrics and assessments and also potentially to standardise testing, and the bottom up approach which enables program coordinators to identify specific graduate attributes to map to their programs, and encourages academics to develop assessment rubrics and mark assessment for subject specific and generic attributes.

While the promises are high for a well-designed and well implemented process, the design of the process and its price continue to be debated. The promise is documented to include demonstrating accountability and educational quality; satisfying requirements of accreditation; facilitating continuous improvement; improving student learning experiences; enhancing curricula; gathering information for program development and teaching improvement (Zhu and McFarland, 2005) and addressing the business community requirements of competency in graduates (Haskell and Beliveau, 2010). Yet the price is more difficult to pin down and may require a new way of thinking and working and structuring programs with new developments in knowledge and skills. Where assurance of learning involves choosing, creating and innovating effective measures for assessing student accomplishments, Zhu and McFarland (2005) suggest the measures selected need to fit with the goals determined for the program and the pedagogues used as well as the circumstances of the institution. This requires leadership commitment and faculty support (Zocco, 2011). Hollister and Koppel (2007) identified that the development of an assurance of learning assessment process requires the broadening of thinking by those involved, from one of being unit- or subject-centred to one of focusing on the program as a whole, to build program goals, measure program outcomes and make adjustments to the program curriculum. However, academics have traditionally been the experts in their own fields with little requiring them to work across disciplines within programs (French, 2011). When assurance of learning is not aligned to assessment experiences, both students and academics can struggle to see the value of the outcomes relative to the graduate attributes, and therefore may not engage with it from a teaching, learning or quality development perspective (Taylor et al., 2009; Oliver, 2011).

Our objectives in this study were to investigate the views and the experiences of the designers, the implementers and the end users in one institution to identify the challenges, the learning, and the outcomes of a new assuring learning process, with a view to understanding the issues of engagement and sustainability and to identify if the promise of AoL was realised, and if the price of its development was worth the outcomes. Our approach was to survey key stakeholders for their experiences and perceptions. While the stakeholders of a degree program and its assurance of learning process include numerous groups including University governance boards, the public and society generally, we chose to involve the key stakeholders. The 'joined up thinking' of organisation enquiry through the use of a stakeholder analysis provided different research philosophies and methods that offered a robust and pragmatic investigation (Simmons and Lovegrove, 2005). Six different stakeholder groups were surveyed over two years, 2011 to 2013, within one undergraduate business degree program involved in a newly designed and implemented AoL process that had been recently reviewed and accepted by two accrediting bodies, namely Equis and AACSB.

The AoL Process

Different accrediting bodies identify a range of process frameworks for assuring learning, but key components include the design of program learning goals and the measures of learning accomplishments (Zhu and McFarland, 2005). Program learning goals are a reflection of the institution's graduate attributes which are the orienting statement of education outcomes used to inform curriculum design and the provision of learning experiences at a university (Barrie, Hughes and Smith, 2009). They commonly reflect the professional nature of work in that discipline, which helps students to put their academic learning into a professional context, making the educational experience more authentic (Boud and Falchikov, 2007). While universities claim that they design curricula to address such graduate attributes, according to Barrie, Hughes and Smith (2009), few can offer any convincing evidence that such curricula does comprehensively and systematically develop such attributes. Graduate attributes are usually measured through learning goals determined at the program level, which are aligned to assessments within each of the courses/units/subjects within a program. 'Learning goals express what we want our students to be, while learning objectives express what we want our students to do' (Bajada and Traylor, 2013).

Learning objectives within individual assessments that are well aligned to program learning goals and show development over time allow students to progress in their learning through regular teacher feedback and self-assessment. Establishing clear learning goals for a program aids the student in understanding the standards of skills required for successful completion.

The promise of well-developed graduate attributes addressed through strong program design where the learning goals are measured through appropriate course-embedded assessment continues to be supported in the literature. Yet the price may be high as the challenges and the limitations can be difficult. It is difficult to ensure that all stakeholders are engaged and committed to the process. The B Factor Project (Radloff et al., 2009) concluded that institutional strategies for integrating graduate attributes need to acknowledge the importance of academic staff beliefs. Any lack of confidence or willingness to teach and assess graduate learning or attributes may be a key obstacle in institutional initiatives to successfully integrate graduate attributes across the disciplines. Without dealing directly with the beliefs and perceptions of academics, attempts to drive the development of graduate attributes as part of a quality agenda will likely fail (Radloff et al., 2009). A process to engage academics is therefore required to facilitate the inclusion required to embed assurance of learning practice. However, The National Graduate Attributes Project and the B factor reports suggest that there is limited engagement by academic teaching staff in Australian universities (Barrie, Hughes and Smith, 2009, p. 14). In addition, the effective integration of graduate attributes into development approaches within the classroom as been somewhat intangible (Taylor et al., 2009).

While graduate attributes need to be specifically related to student learning in order to be valued and recognised (Taylor and Ors, 2009, p. 31), no amount of effort in this area will work unless it is perceived by the students to have actively engaged them in developing worthwhile attributes (Barrie, Hughes and Smith, 2009). Mills and Ors (2009) found experiential tasks helped students identify the importance of graduate attributes, and Radloff et al., (2009), p. 57) identified that engaging students early in their university experience through mentoring programs, and contextualising the graduate attributes, were critical to developed effective outcomes. Haskell and Beliveau (2010) concluded that students undertaking a simulation experience can clearly perceive improvement in their business skills as defined by program goals and that this improvement is progressive during their experience.

The second major factor in the AoL process is the development of measures of learning accomplishments (Zhu and McFarland, 2005). A key promise of success factors includes the development of an overall assessment strategy to allow students to demonstrate learning in different ways, encouraging a range of learning and creating a systematic and comprehensive assessment package (Owen, Davis, Coperf, Ford and McKeough, 2009). Course embedded assessment, which is the process of using artefacts generated through classroom activities to assess the achievement of student learning objectives (McConnell, Hoover and Miller, 2008), is most recommended as means of providing direct information that would help a faculty member improve student learning and in addition, assist student development through extensive feedback. Yet the price may be impossibly high as according to Bass and Geary (1997), drafting clear, measurable learning objectives is one of the most difficult tasks for many faculty members who often make errors in using unfocussed learning objectives. Further, these skills in writing learning objectives and the design of appropriate assessment is crucial to how students with diverse experiences and needs perform in that assessment (Jackson, Watty, Yu and Lowe, 2006).

Institution

One Australian Business School (The School) chose an international accreditations strategy more than 10 years ago as part of its competitive advantage. Recently it implemented an assurance of learning process to better assure quality in its learning and teaching. One of the reasons The School established the assurance of learning program in 2009 was that the accreditation process of two international accrediting bodies was imminent (Lawson et al., 2014). While other reasons such as effective assessment processes, measurable student outcomes and industry support were important, addressing the needs of accreditation was paramount. In establishing the process, The School developed a recursive process identified by Zocco (2011) as using direct assessment information to implement continuous and systematic curriculum improvement and documenting that improvement. The process involved the determination of learning goals at a program level and learning objectives in each unit to deliver on the graduate attributes, mapped across the curriculum to ensure that each was introduced, developed and assured identified by Bajada and Trayler (2013) as a graduate attribute framework designed to give the student the opportunity to successfully attain the attributes. Developing and measuring learning outcomes at a program level occurred in capstone units in nine (9) majors within the Undergraduate Program. Each unit coordinator worked within their team of eight (or more) other lecturers, delivering the units within that discipline major to identify unique goals, unit objectives, measurement techniques and ancillary processes, in order to analyse student outcomes and program outcomes to identify potential for continuous improvement. Subject Area Coordinators led each teaching team and reported to Teaching and Learning Chairs within each of the discipline Schools. These small communities of practice developed within a larger community of practice led by the program director.

The communities of practice met regularly to discuss the process of developing quality assurance as well as the outcome of the program which measured student achievement with program goals. These groups analysed the information and advised on changes to the undergraduate program. The School achieved its goal of accreditation with both international bodies. Indeed, an excerpt from one report on the assurance of learning program reads 'There is a very well developed and executed program for undergraduate programs that is widely known and understood by faculty and students' (French, 2011). It was the members of these communities of practice (as well as other stakeholders) whom we approached for interviews in order to answer our questions regarding the challenges, benefits and limitations of building an effective assurance of learning process.

Method

Our stakeholder approach offered a cross-sectional design for this survey research. A cross- sectional design allowed us to collect information on a number of measures from different stakeholders and compare the extent to which they differed (DeVaus, 2002). Our stakeholders included: the Assistant Dean Teaching and Learning (ADT&L) who answered to the Dean's Executive Group; the Course or Program Coordinators who answered to the ADT&L; the Unit or Subject Coordinators who answered to the Course or Program Coordinators; one Industry Advisory Board for one of the departments within The School; and undergraduate and postgraduate students as well as alumni students.

The six different groups were surveyed to identify their experiences in a new assurance of learning process linked to a set of graduate attributes in The School to understand their identification of the benefits and values, challenges and learnings. Figure 1 identifies each group and their relationships.

We used a multiple method approach in gathering the information using three different techniques: depth interview; focus groups; and an online survey to identify the challenges and the learnings of the groups. We used a depth interview with Assistant Dean Teaching and Learning from which we developed a list of questions to ask a random sample of academic program coordinators and a random sample of academic unit coordinators in two different focus groups, and finally, we used two online surveys to survey undergraduate and postgraduate students and alumni. The depth interview allowed us to develop a relatively unstructured, extensive interview. Such interviews are often used in the primary stages of the research process and differ from traditional interviews in that they encourage discussion on an undisguised subject area without influencing the direction of that discussion except through probe questions intended to encourage further elaboration (Zikmund, 2003).

Using focus groups allowed us to collect data from two different group interactions regarding their experiences with assurance of learning. The benefits of using both styles of interviewing meant that we received a balance of depth and breadth of information on the issues of assurance of learning. According to Stokes and Bergen (2006), focus groups provide more breadth and contextual information, with a consensus view that allows conclusions to be drawn more easily from the research, but with the potential pitfall that these may be invalid if the group effect has overridden key issues, while depth interviews offer more depth and clarity in the data collected and allow the researcher to perceive more attitudinal subtleties. The questions for the two academic groups were the same, with only a slight variation to acknowledge the management versus teaching roles of the participants. Each focus group was facilitated by an experienced facilitator who asked the questions to begin discussions and used probes when discussion faltered. The entire discussion for each group took between one and one and a half hours and was recorded and typed in full, and later evaluated using Nvivo to identify themes.

The two student surveys were designed in consultation with the faculty teaching and learning advisory committee, the ADT&L, and a sample of academics involved in the AOL process. Each was tested on a small group of students before use. Online surveys provide access to unique populations, who share special interests, characteristics, values or behaviours. We could access a greater number of people in a relatively short period of time. Moreover, the transfer of collected data into statistical analysis programs is quicker and tends to contain fewer errors. In this particular case, where information about the final year experience at university by alumni was sought, international students' opinions were also needed. This made an online survey the most suitable tool.

To address sampling validity we called for volunteer respondents from two specific academic groups responsible for the design, implementation and experience of assurance of learning, the program managers and the unit teachers. The academic managers were the program coordinators answerable to the Assistant Dean Teaching and Learning for their process development and program outcomes. The teachers were all coordinators of units that capped off specific programs and were responsible for collecting assessment data within their program and reported to their study area coordinators. Six (6) academic program coordinators participated in the management focus group and four (4) academic unit coordinators participated in the teaching focus group. Students (both current and alumni) were invited to participate in the online survey through an introductory email. From a total enrolment of 1,399 undergraduate students (in their third year), a sample of 387 usable questionnaires was collected (26% response rate). It is unknown how many alumni received the email but 48 usable questionnaires were collected.

Analysis

The transcript of each focus group was analysed using content analysis which involves breaking down the text into categories based on explicit rules of coding (Krippendorf, 2004). Nvivo was used to assist in the process of analysis to identify major groupings. Computer added text analysis allows for systematic comprehensive and exhaustive analysis (Gephart, 2004).

Regression analysis was used to identify students' overall level of satisfaction about their educational experience where graduate skills were used as independent variables (IV) and their perception of their experience was the dependent variable (DV).

Findings

Assistant Dean Teaching and Learning

The ADT&L discussed assurance of learning as a philosophy of continuous improvement driven by international accreditations but linked to the strategy of the business school and its positioning the market. 'Well, the AoL stuff started with the accreditation process. I think our approach has been a bit different to what other people have tried to do with it. I suppose it's really related to the philosophy about continuous improvement but it also was how I saw this and I saw this as being absolutely aligned to how we wanted to position the business school. ... Continuous improvement, it's the centre of everything we do here. It's a university philosophy.'

While the external accreditations were a recognised driver, the quality assurance and continuous improvement were an obvious motivation, but ongoing staff involvement in a cultural change was also discussed as part of the motivation. 'I wanted AOL to be integrated and embedded into the culture of the place and also into the daily practice. I didn't want it to be something that was out at the side - an add on. Because my belief is if you do that, it's not sustainable and it won't last. As soon as you've got somebody not driving it, it'll die.'

The ADT&L discussed the process of assuring learning as one where all programs (nine undergraduate and ten post graduate, an MBA and EMBA and Honours, Research Masters and PhD are mapped to five (5) major learning outcomes that incorporate university determined graduate attributes). These learning outcomes are slightly different in different programs and were designed by Dean, Heads of Schools and the senior management team, and further developed 'through an intensive consultative process that included pretty well everybody in the business school.' A generic matrix or rubric was designed for each learning goal. While these 'differ according to the program' and the level of development (undergraduate versus post graduate), all students are measured against these outcomes. 'If we were going to do this and make it explicit to students, I wanted all students to know how they were performing against those goals. Now we couldn't have done this without using a good data management system. ... It actually works out to be once a semester that each of the goals get addressed and measured in a unit ... usually those units at the end of the program such as capstone units. ... Really the intent when we started on this journey was that we would implement the review across the board and we would then be able to look at how [students] went at the end of the first year, end of the second year and the end of their degree. We are not there yet!'

In the area of challenges and learnings, the ADT&L discussed the hard work of getting staff buy-in, not just individually but as a disciplined team and embedding the process into the routine and as part of the culture. 'It's got to be a consistent message right across the board. When we agreed that that was the approach we'd use - those sorts of things I said before about engaging all staff and all students and all the rest of it, that automatically flows onto the sorts of things I've talked about.'

Academic Management

The management group consisted of the Course or Program Coordinators. This group discussed the philosophy of assurance of learning in terms of quality, consistence and validity of the content across the program. Their comments were consistent with these themes whether they discussed the program, the students or accreditations. They did not discuss external or international accreditations in terms of their philosophical underpinnings. '... Ensuring accreditation requirements are met by graduates during their program so they meet the professional body requirements initially led by the AACSB accreditation process however, [it] was soon recognised as part of good teaching practice to have a common understanding for all staff and students.'

The discussion on the motivators for AoL hinged around the employability of the graduates; student perceptions of employment outcome guarantees; enabling students to demonstrate their appropriate knowledge skills and attributes in their profession; and the transferability of achievements to other universities in others parts of the world - a network of professional and academic standards. In fact, any discussion regarding accreditations with external bodies was quickly downplayed to reveal that such bodies were seen as the providers of stimulus for which this institution had developed higher level outcomes than required. 'External bodies use a range of criteria to evaluate our courses, sometimes these align to our AOLs and at other times we have developed outcomes to a higher level than what's required.'

In their discussion of the process of assuring learning, the themes discussed by the management academics involved a discussion of the purpose of AoL for both the institution and the academics; the assessment of all AoL goals in capstones; rubrics; and comparisons between majors across the faculty to use the data for development of the majors and their units. 'It's a continuous process of curricular improvement at the program level.' 'It's based on rigorous data collection, evaluation and implementation.' 'We map AoLs across the major and build skills incrementally so that when students exit through a capstone unit, they have had a chance to practice and be assessed before final measurement in the capstone.'

In the area of engagement, the academic management staff reflected on the leadership and the resourcing for the development of the process in addition to the scaffold process of development over time. 'The critical thing for me is really the Dean. The Dean decided this was something we were going to do and we were going to do it well.' 'We invested in learning designers [in each school]. It's an expensive resource.' 'We started with the criteria sheet stuff anyway in 2004 so in 2004 we'd started with the CRA, [criterion referenced assessment] then we went to...' '...Yes, it was almost like a drip feed, wasn't it?' '...Yes, went to where we're designing our programs.' 'The CRAs came from the Vice Chancellor, thou shalt all have CRAs.'

The challenges and learnings the academic managers identified the work within the process that needed further development such as the teaching and practical aspects in the units. While the assessing and criteria for assessing are done quite well, the teaching and practical aspects may not always be well linked to the assessments and assessment process. In addition, the challenge of capturing data and reporting it is an ongoing challenge. 'Our next step now is to really bridge all those gaps that we've discovered between the gathering process and the reporting'. 'We've got two different structural things that this AoL process has given; a great whole of program feel to the major and to the core which the SACs probably have the most internal knowledge of; yet, all of our work as individual academics is assessed very much at the unit level.'

Academic Teachers

The teaching group consisted of Unit Coordinators collecting student learning outcome data in their units. These were the capstone units from each of the programs. This group discussed the philosophy for assuring graduate attributes in terms of comparisons. The main themes included the importance of benchmarking student achievements against expected standards for professional practice, and of identifying achievement outcomes against the planned standards, as well as comparing them against national and international business school competitors. 'It is a benchmark to assess how students meet expected standards for professional practice across the degree program.'

External motivations hinged around the concepts of quality and validation according to the Management Academics. 'Accrediting bodies want quality assurance against the programs that they validate through accreditation, as such evidence is required to prove that our degree programs are of a sufficiently high standard to warrant this "seal of approval".'

With regard to the process of development of assurance of learning within their specific programs, the teaching academics discussed three themes: the differences in approach between the different disciplines/schools; the opportunity for continuous improvement within each program provided by the reports showing student achievements against each goal each semester; and the importance of getting the sessional staff up to speed. 'Very pretty pie/bar charts ... shows you what percentage, like what weighting, you've contributed to a particular goal across a unit, or several units if there's more than one unit per measurement, and how students have performed against individual criterion.' 'The challenge of full-time academic staff members and sessional academic staff members . we work with a lot of sessional staff and how do we get them up to speed on "here's what this all means" and "here's how it fits together"?'

In the area of engagement, leadership was seen as the most important value in getting staff buy in for this new process. 'Leadership provided in the areas of teaching and learning quality assurance. A consultative process was used to establish the AoL goals, embed and evaluate the goals. An expectation for continued improvement: no pressure to get it "right" but rather an opportunity to improve over time. This has removed a lot of the fear factor that some accreditation processes can cause.'

In the area of challenges, the Teaching Academics identified a number of challenges. These included using the reported data to reflect on the needs of the programs for improvement and to implement changes that would bring about these changes: closing the loop in the continuous improvement process and not resting on past achievements; upskilling sessional staff; ensuring the measurement of graduate capabilities is authentic and meaningful, e.g. in topics such as ethics; motivation of all teachers and getting an equal understanding on what constitutes assessing a goal in an assessment piece; and finally, communicating with students, e.g. at the end of the program. 'Ensuring that reflection and closing the loop processes are embedded in business processes and not just an add-on when required for accreditation.' 'How open will academics be to revising the AoL goals as they might/should need to change?'

One learning from this process discuss by the teachers was the fact that the culture of individual unit 'silos' was breaking down and a program culture developing. 'In the initial stages, some academics found the AoL goals to interfere with their silo approach to units - this culture has changed over time.' 'Measurement points for the AoL goals do create more work for the academics involved - a more streamlined approach to data collecting and reporting would alleviate this.'

Industry Advisory Board

The Board discussed three features: their perceptions of the 'school' and its graduates in the market place; student integration with industry; and graduate capabilities in the market place and how these could be improved. This focus group was not taped and translated but notes were taken and transcribed later. The committee outlined some of their perceptions of The School. They were as follows:

Their perceptions were that The School was known for producing graduates with the knowledge and skills to 'hit the ground running'. The Board acknowledged that this was achieved through significant work through curriculum design. In particular, the committee acknowledged the effort placed in the development of assurance of learning goals in order to harvest the critical professional capabilities together with long-term capabilities. Employers have observed that graduates are provided with a good understanding of theoretical concepts with the ability to apply the theory in a practical way. The flexibility of the degree provides students with the ability to develop a combination of skill sets. The use of technology is another strong quality of our programs. In summary, The School reinforces the 'real world' strengths of 'experience, flexibility and technology'.

The issue of student integration with industry raised the question of how the Board could contribute to The School in a practical way and this was not identified as strong within the program. The Board determined that this was where they could assist with further development.

On the issues of graduate attributes, the committee discussed the graduate attributes identified in the Business Council of Australia Report 'Lifting the Quality of Teaching and Learning in Higher Education' (2011).

(*) A combination of in-depth knowledge and up-to date-technical skills in the discipline that they have studied;

(*) International capabilities, based on the ability to adapt to working in an international environment with people from different cultures as part of multinational and multi-disciplinary teams;

(*) The ability to think independently, to critically analyse issues and problems, and to adapt thinking and analytical capabilities to different contexts and new problems;

(*) Generic skills, including teamwork, problem solving, communication, and the ability to utilise technology and to engage in self-directed learning.

Overall, the Board was of the view that The School should continue to build and focus on its strengths and attributes. Whilst stretch targets are the norm and are important, a continued focus on The School's strengths should be the aim. It was felt that The School should continue working on building relationships with their students. Some discussion was held around what is meant by the concept of 'internationalisation' and the concept of a 'global mindset' and how to deal with the issue. It was noted that whilst 80% of students say that they intend to work overseas, the question may be whether students fully understand the issues associated with working internationally.

Students

A self-administered questionnaire was purposely developed to reflect the program learning goals and objectives for the bachelor degree program in Business. Questions included forced choice on a Likert scale, 1-5, from very well (5) to very poorly (1), on the professional skills linked to the graduate attributes, and the level of satisfaction students perceived, as well as one open ended question related to how their educational experience could be improved. Research participants were recruited via an introductory email sent to third year undergraduate students enrolled in a business program. Participants were provided with a link to the online questionnaire. From the Faculty of Business's total enrolment of 1,399 undergraduate students a sample of 387 usable questionnaires was collected, providing a 27.6% response rate. The survey data was imported into SPSS statistical analysis. Initial descriptive and more complex statistical analysis was performed on the data collected in order to provide a better understanding of the student experience. Two hundred and forty nine respondents (249) (64.3%) were female and 138 (35.7%) were male. Fifteen percent (15%) of the respondents were above 20 years of age, and 86% were full time students with 77.5% being domestic students.

The students do perceive their current studies prepare them well for a range of professional skills identified in the graduate attributes identified in The School's undergraduate programs.

Regression analysis was used in order to identify a possible relation between students' overall level of satisfaction regarding their educational experience and their perceived level of learning (the students' overall level of satisfaction was used as dependent variable and the different graduate skills (AOL) from table 1were used as independent variables.

In relation to the findings of the qualitative question: 'Please tell us how we can improve your educational experience?', three specific areas of comments were identified. Approximately 50% of the comments were related to the presentation issues of their education experiences such as the number of units presented on line or the amount of assessment or limitations to group work. Almost 40% of the comments were related to practice and identified the need for more opportunities for internships, real case studies and engagement with local firms. More than 10% discussed relationship issues such as more social activities, more activities for 1st year students.

Results indicate that 45% of undergraduate students' level of satisfaction regarding their educational experience is explained by their perceived learning in terms of the different graduate skills identified and assured throughout the learning experiences in their program.

Alumni

The response rate was unknown but 48 graduates responded to the survey from nine (9) different undergraduate program experiences. Because these alumni had not experienced AoL in its entirety, we sought information on three topic areas:

What sort of content, learning activities and/or assessment would you
include in a final year subject to help students prepare for the
workplace?


More than 20% of the respondents indicated their preference of a work internship experiences and in a similar vein, 15% of the respondents preferred activities based on practical requirements of the workplace including meeting client requirements; reports and presentations etc. A further 15% identified their preferences for practical administrative and support type activities such as cross- cultural training, technical expertise or presentation and interview activities. 17% of the respondents indicated a preference for simulations or case studies as preparation for the workplace. Only a few students identified the importance of group work as a help for preparation for the workplace.

Describe the most useful learning experience in your final year at
university (it could be a lecture, an assessment item, a whole
subject etc.). How did it help to prepare you for work in your field?


Overwhelmingly, students described their most useful learning experiences as authentic learning experiences. Activities that replicated real world expectations including designing Business Plans, addressing client briefs, an advertising campaign for a real client, etc.

f you have any further comments about how university subjects can
better prepare students for the workplace please write them below.


Students were overwhelmingly in two camps on this with one being practical work experience and internships and the other being authentic assessment pieces. Other comments were in line with more experiences to meet some of the graduate skills including Writing skills, Oral skills, Team Work and Ethical Practice.

Discussion

In addressing the question of whether the promise of an effective assurance of learning process was worth the price for The School, we found the answer to be in the affirmative but we did find the reconciliation of this to be different from that identified in the literature. Different groups did consider the price of such change differently with the students and alumni not considering any price at all.

The promises of an effective assurance of learning process were well discussed on the part of all the stakeholders we surveyed. While the ADT&L considered the meta-benefits of continuous program improvement, those involved in the program design and delivery identified the potential for program quality and teaching quality within the program and accreditation. In addition, these academics also identified the value of cross disciplinary collaboration at the program level and the breaking down of structural silos throughout the process of collaboration. The current students and alumni considered the promise of preparing for work and work readiness. It seems that all the groups saw value in the process as a potential means for developing excellence within the program and achieving excellence as a result of it. This is supported in the literature with Zhu and McFarland (2005) recognising the importance of program quality, continuous improvement and improved learning experiences, and Bajada and Trayler (2013) acknowledging the previous failure of curriculum to move away from the siloed approach of producing 'I' shaped graduates without the ability to think outside the silo towards a more 'interdisciplinary' focused approach.

The price of the change had a similar story. Zocco (2011) refers to the importance of leadership commitment and faculty support in developing a quality program with continuous improvement elements, and this was an important consideration for the academics. The difference lay in the perception of where the price may be paid and who was paying it. While the ADT&L discussed the engagement of staff and the means of doing this as including top management support and the development of communities of practice to encourage program teams to work together in their decision making, the management academics and the teaching academics discussed the difficulties of the process itself, including the need for better systems of data collection and storage to ensure good analysis for the decisions related to closing the loop in the continuous development process of the program. The management academics acknowledged the valuable input of resources into the program development in the form of learning advisors to both students and staff as well the continuing top level support for program change.

On the negative side of the cost ledger, the discussion turned to the changing nature of academic work which now involves team participation at a program level, but individual assessment of academic outcomes is still considered at the unit level. With the Australian Higher Education sector experiencing declines in funding, increases in student numbers and a heightened institutional emphasis on research output (McDonald and Star, 2008), a tension between traditional academic values and the corporatisation of higher education exists (Ryan and Guthrie, 2009). As the sector continues to grapple with the development of curricula designed to cater to government and industry, pressure for the production of graduates fit for a knowledge economy is increasing, but, with larger class sizes and reducing financial support, academics are increasingly resentful that resources have become scarce and the sense of being a 'socially altruistic and purposeful community of scholars' has been eroded (Duke, 2004, cited by Nagy and Burch, 2009, p. 239). In response to this pressure Dawson, Burnett and O'Donohue (2006) suggest higher education institutions are changing their organisational culture to implement new principles of community. These new principles of community were considered an important factor for the academics involved in this process but the changing values of individual recognition were of concern.

The industry advisory board and both groups of students were interested in the challenges of a good student experience and the value of high levels of generic skills for use in a day-to-day working system. Yet what was obvious was they did not discuss their own participations in such experiences or their own costs in such an exercise.

Reconciling the promise and the price resulted in four of the stakeholder groups discussing the value of the changes in the processes for assuring learning within a program of study as coming out ahead of the costs in the ledger. These groups included the ADT&L; both academic groups and the industry advisory board. The changes in the relationships between teams of academics and between academics and students were seen to be a high value outcome. This supports the notion that The School had built a sustainable process. Lejeune (2011) suggests that one of the factors required to build sustainability into the assurance of learning process is the internal pressure from school members to keep innovating and progressing both in terms of quantity and quality.

The ADT&L also discussed the benefits of feedback on learning for the students but also for academics in designing and redesigning the program, and that this was embedding program change within the culture of The School. While the ADT&L and the Industry Advisory Board believed that program change through the processes of designing and delivering good learning feedback to the students would encourage engagement between them in a new learning culture; neither the students nor teaching academics discussed this feature at all. The teaching academics valued the growing nature of the culture change within the program working environment including the breaking down of the discipline silos, with both academic groups discussing the positive value of the leadership support for the program and cultural change within The School. The students valued work skills development and authentic learning processes and asked for more of the same without any discussion of their own changing input requirements or the necessity for their involvement in the learning process to also continue to change and develop. Future research related to sustainability and engagement should involve the relationship between the students and the academics particularly in relation to the assurance of learning process and the pressure for continuous program development where both groups identify

different values and needs.

References

Barrie, S., Hughes, C. & Smith, C. (2009). The national graduate attributes project: integration and assessment of graduate attributes in curriculum. Retrieved from Australian Learning & Teaching Council website: http://www.altc.edu.au/resource-national-graduate-attributes-projectsydney-2009.

Bajada, C. & Trayler, R. (2013). Interdisciplinary business education: curriculum through collaboration, Education and Training, 55 (4), 385-402.

Bass, G.M. & Geary, W.T. (1997). Education research abstracts, Issues in Accounting Education, 12 (1), 248-252.

Boud, D. & Falchikov, N. (2007). Aligning assessment with long-term learning, Assessment and Evaluation in Higher Education, 31(4), 399-413.

Business Council of Australia Report (2011). 'Lifting the Quality of Teaching and Learning in Higher Education' Melbourne, Australia.

DeVaus, D.A. (2002). Surveys in social research 5th edition, Allen & Unwin, NSW Aus.

Dawson, S., Burnett, B. & O'Donohue, M. (2006). Learning communities - an untapped sustainable competitive advantage for higher education, International Journal of Educational Management, 20(2), 127-139.

Duke, C. (2004). Cited by Nagy, J. & Burch, T. (2009). Communities of practice in Academe (CoP-iA): understanding academic work practices to enable knowledge building capacities in corporate universities, Oxford Review of Education, 35(2), 227-247.

French, E. (2011). Communities of practice: building capability in an undergraduate assurance of learning program, e-Journal of Business, Education & Scholarship of Teaching, 5.

Gephart, R.P. (2004). Qualitative research and the academy of management journal. Academy of Management Journal, 47(4), 454-462.

Hall, O.P. & Kro, K. (2006). Learning assurance using business simulations applications to executive management education, Developments in Business Simulation and Experiential Learning, 33, 1-6.

Haskell, N. & Beliveau D. (2010). Measuring the soft side of the achievement of assurance of learning objectives, The Business Review, Cambridge, 16(2), 16-26.

Hollister, K.K. & Koppel, N.B. (2007). Framework for meeting AACSB international's assurance of learning requirements: application to information technology, Journal of Informatics Education Research, 8(3), 1-14.

Jackson, M., Watty, K. Yu, L. & Lowe, L. (2006). Inclusive assessment: improving learning for all: a manual for improving assessment in accounting education, ALTC, Melbourne, Vic.

Krippendorf, K. (2004). Content analysis: an introduction to its methodology 2nd edition, Sage, Thousand Oaks.

Lawson, R., Taylor, T., French, E., Fallshaw, E., Hall, C., Kinash, S. & Summers, J. (2014). Hunting and gathering: new imperatives in mapping and collecting student learning data to assure quality outcomes, Higher Education Research & Development, 1-15.

Lejeune C. (2011). Is continuous improvement through accreditation sustainable? A capability-based view, Management Decision, 49(9), 1535-1548.

McConnell, C., Hoover, G. & Miller G. (2008). Course embedded assessment and assurance of learning: examples in business disciplines, Academy of Educational Leadership Journal, 12 (3), 19-34.

McDonald, J. & Star, C. (2008). The challenges of building an academic community of practice: an Australian case study, in Engaging communities, Proceedings of the 31st HERDSA Annual Conference, Rotorua, 1-4 July 2008, 230-240.

Martell, K. & Caldron, T.G. (2009, July 6-7). Assessment in business school: what it is, where are we, where we need to go now. http://search.proquest.com/docview/199572794?accountid=13380.

Mills, J., Baguley, J., Coleman, G. & Meehan, M. 2009, Enhancing Communication and Life Skills in Veterinary Students: Curriculum Development and Assessment of Methods Final Report, Sydney: Australian Learning and Teaching Council. Retrieved from: http://www.olt.gov.au/resource-library?text=Veterinary+students

Oliver, B. (2011). Assuring graduate outcomes. Retrieved from http://api.ning.com/files/Stx00vzPr3OLB02As1go2kiUu-cHkBe-MYmmBh-MiH9rqqKJZOwyQB62n0WuJr7hLe3EKAanIdua9xac0OJSWzFiqvqrexBG/AssuringgraduateoutcomesALTCGoodpracticereport.pdf.

Owen , S., Davis, G., Coperf, M., Ford, W. & McKeough, J. (2009). Learning and teaching in the discipline of law: achieving and sustaining excellence in a changed and changing environment. Council of Australian Law Deans (CALD) <http://www.altc.edu.au/resource-learning-teaching-law-flinders-2009>.

Radloff, A., De la Harpe, B., Scoufis, M., Dalton, H., Thomas, J., Lawson, A., David, C. & Girardi, A. (2009). The B factor project: understanding academic staff beliefs about graduate attributes. Retrieved March 21, 2012, from: http://www.altc.edu.au/resource-b-factor-academic-beliefs-graduates-rmit-2009.

Ryan, S. & Guthrie, J. (2009). Collegial entrepreneurialism, Public management Review, 11(3), 317-343.

Simmons, J. & Lovegrove, I. (2005). Bridging the conceptual divide: lessons from stakeholder analysis, Journal of Organizational Change Management, 18(5), 495-513.

Stokes, D. & Bergin, R. (2006). Methodology or 'methodolatry'? an evaluation of focus groups and depth interviews, Qualitative Market Research, 9(1), 26-37. Retrieved from http://search.proquest.com/docview/213454178?accountid = 13380.

Taylor, T., Thompson, D., Clements, L., Simpson, L., Paltridge, A., Fletcher, M., Freeman, M., Treleaven, L., Lawson, R. & Rohde, F. (2009). Facilitating staff and student engagement with graduate attribute development, assessment and standards in business faculties. Retrieved March 21, 2012, from: http://www.olt.gov.au/system/files/resources/PP7322%20UTS%20Taylor%20Report%20Sept%202009%20edited.pdf.

Zhu, F.X. & McFarland, D. (2005). Towards assurance of learning in business programs: components and measurements', The Journal of American Academic of Business, 7(2), 69-72.

Zikmund, W.G. (2003). Business research methods, Thomson/South-Western, Cincinatti, OH.

Zocco, D. (2011). A recursive process model for AACSB assurance of learning, Academy of Educational Leadership Journal, 15(4), 67-91.

Erica French #

Queensland University of Technology Brisbane, Australia

Email: e.french@qut.edu.au

# Corresponding Author

Gian Luca Casali

Queensland University of Technology Brisbane, Australia

JEL Classification: I21

PsycINFO Classification: 3660

FoR Code: 1303; 1503

ERA Journal ID#: 35696

Table 1:
Undergraduate Third Year Students' Responses about Degree of
Preparation AOL

How well did your degree prepare you for   Very   Well   Neither  Poorly
a professional situation that requires:    well

Teamwork skills                            14.7%  62.0%  16.3%    4.7%
Problems solving skills                    17.15  66.4%  13.4%    2.1%
Leadership skills                          12.7%  45.5%  35.1%    5.2%
Time management skills                     29.7%  46.8%  19.1%    3.9%
Initiative and creativity skills           14.0%  44.4%  32.3%    7.2%
Communication skills -Oral                 17.6   57.4%  19.1     4.1%
Communication skills- Written              33.1%  50.1%  14.0%    1.3%
Ability to work independently              43.2%  45.2%  10.3%    0.8%
Practical experience                       12.4%  41.3%  32.3%    8.8%
Ethical practice in your profession        21.2%  51.7%  22.7%    2.3%
Professional knowledge in your field       24.8%  53.5%  16.3%    3.1%
(accounting, HR, etc.)
Intercultural awareness                    21.5%  49.9%  23.5%    2.8%

How well did your degree prepare you for   Very
a professional situation that requires:    poorly

Teamwork skills                            2.3%
Problems solving skills                    1.0%
Leadership skills                          1.6%
Time management skills                     0.5%
Initiative and creativity skills           2.1%
Communication skills -Oral                 1.8%
Communication skills- Written              1.6%
Ability to work independently              0.5%
Practical experience                       5.2%
Ethical practice in your profession        2.1%
Professional knowledge in your field       2.3%
(accounting, HR, etc.)
Intercultural awareness                    2.3%

Table 2:
Regression Analysis for UG Students

Sample    R    R Square  Ad R Square  STD Error  Anova Sig    F

 UG      .686   .471       .454        .631       .000      27.777
COPYRIGHT 2014 Australian Business Education Research Association
No portion of this article can be reproduced without the express written permission from the copyright holder.
Copyright 2014 Gale, Cengage Learning. All rights reserved.

Article Details
Printer friendly Cite/link Email Feedback
Author:French, Erica; Casali, Gian Luca
Publication:e-Journal of Business Education and Scholarship Teaching
Article Type:Report
Date:Jul 1, 2014
Words:7873
Previous Article:The impact of a professional selling course on student perceptions: A before and after look.
Next Article:Pivotal ways of improving leadership and the scholarship of learning and teaching in higher education.
Topics:

Terms of use | Privacy policy | Copyright © 2019 Farlex, Inc. | Feedback | For webmasters