IT governance measurement tools and its application in IT-business alignment.
The purpose of this exploratory research paper is to evaluate the deployment and assessment methodology of the information technology governance (ITG) measurement tools, with the purpose of gaining deeper insight into the ITG initiation process, the nature of tools employed, measurement processes, and the implementation methodology, using case studies. Analysis of the available academic and non-academic literature sources showed measurement issues being the most dominant and ironically the most neglected domain in ITG implementations. We view ITG measurement tools and it subsequent deployment through the two theoretical ITG models namely the Integrated IT Governance model, and the Structures, Processes, and Relational ITG model. To validate these findings and to get a deeper insight into the ITG measurement domain, we conducted four case studies of measurement tools usage and processes in commonly used ITG frameworks in four organisations in New Zealand and United Arab Emirates. The results indicate that the IT governance initiatives differ in the manner of positioning in the integrated ITG framework, and objectivity of measurement is more evident and emphasized in UAE than in New Zealand. The result of these findings provides practitioners with guidance on the contextual usage of ITG measurement practices
KEYWORDS: IT Governance measurement, IT business alignment, metrics
Assessing the measurement and value of IT is a complex challenge and a future research direction (De Haes, Van Grembergen, & Debreceny, 2013). Thus, there is an ever-increasing demand for accountability and objectivity in the measurement of information technology auditing, and IT processes performance (Maria, Fibriani, & Wijaya, 2012). Supplemented, with an ever-increasing demand for compliance in the information domain, organizations have witnessed an increase in the adoption of IT governance (ITG) frameworks. In a highly contextually different, but global organizational structure, ITG implementation however, remains an issue where the theory does not or cannot always deliver to the expectations of practitioners. Globally, IT governance is concerned with two things: that IT delivers value to the business and that IT risks are mitigated and both need measurement (Grembergen, Haes, & Guldentops, 2004), but contextually the subsequent practices may differ. This key issue of aligning IT goals with business goals, which overlap two domains namely IT and business is the primary goal of IT governance.
However, this continuous alignment of business and IT in a rapidly changing environment has also been the top concern (Kappelman, McLean, Johnson, & Torres, 2016) and a grand challenge for today's enterprises (Hinkelmann et al., 2016). In this respect, the objective of continuous measurement of IT processes/IT controls to ensure alignment, plays a critical role in IT business alignment success through higher-level measurement models (IT maturity model, balance scorecard); and process measurement tools namely heat map, key performance indicators, and key goal indicators. While organizations worldwide embark on adopting ITG frameworks, the subsequent need to select and integrate overlapping ITG frameworks has presented practitioners with challenges in terms of choice and integration of frameworks (Nicho & Muamaar, 2016). While the most prominent IT governance frameworks include ITIL, COBIT, ITCG & COSO (Benaroch & Chernobai, 2012), COBIT and ITIL are commonly used for IT governance implementations (Stevens, 2011). Hence, assessment of the IT processes/IT controls of these frameworks is not only a continuous process for audit and compliance, but also presents challenges in terms of consistency of audit, compliance, and/or measurement. With alignment of IT with the business being the highest management concern for organizations (Kappelman et al., 2016), IT governance become an important issue on the agenda for many enterprises (Simonsson, Johnson, & Wijkstrom, 2007). In this regard, evaluation of its success through objective measurement assumes great importance. Thus, we posit the main research question: How do organizations use ITG measurement tools to assess IT processes and IT controls of the ITG frameworks/standards and processes?
The paper is structured into four main sections. In the second section (following the introduction), the motivation and positioning of the study are provided, followed by section three, which details the research and analytical methodology. Section four provides the analysis of data based upon the findings, followed by discussion in section five.
ITG MEASUREMENT AND THEORETICAL PERSPECTIVE
The objective of this research is to evaluate the deployment and assessment methodology of the measurement tools and techniques used in the IT governance processes, and IT controls of commonly used ITG frameworks. Hence, focusing on the measurement aspect of the frameworks and standards used in ITG provides specific insight into the ITG measurement domain. The impetus of the research stems from three drivers. First, there is limited amount of literature on cases of ITG implementation with the result that practitioners have little guidance apart from the case studies given in white papers and the IT Governance Institute website. Secondly, researchers have emphasized the critical role of measurement in ITG domains namely IT assurance (Stockton, 1998), business IT alignment (Zhou & Cai, 2011), process maturity in COBIT (Walker, McBride, Basson, & Oakley, 2012), IT security governance (Baer & Dietrich, 2006), ITG process performance (Stevens, 2011), and IT strategy (Basili et al., 2010). Taking the commonly used ITG framework, COBIT into consideration, 'issues with measurement' was cited as the most frequent and challenging concern (Alfaraj & Qin, 2011; R. Debreceny & Gray, 2009; R. S. Debreceny, 2006; Ivanov, 2012; Simonsson et al., 2007; Walker et al., 2012). Thus, the researchers anticipated the need for a deep understanding of ITG measurement from a theoretical and empirical point of view, which would be of benefit to both academics and practitioners.
THEORETICAL POSITIONING OF THE STUDY IN ITG
The measurement of performance of IT processes/IT controls is a critical operational aspect of IT governance. From an integrated IT governance framework perspective (Dahlberg & Kivijarvi, 2006) measurement is viewed as one of the two operating functions of IT governance (Figure 1). From figure 1, it is evident that the IT governance process starts with business-IT alignment in the planning phase that had a guiding impact on the operating phase. In this phase, the monitoring of IT resources, risks, and management is affected by the selection of appropriate IT performance measurement tools, which ultimately affects the benefits, costs, opportunities, and risks. Hence, we view the research through the 'operating' phase of the framework.
[FIGURE 1 OMITTED]
ITG can be deployed using a mixture of structures, processes and relational mechanisms (SPR), where structures are devices and mechanisms for connecting business and IT; processes refer to IT monitoring procedure, while relational mechanisms relate to participation and collaboration between management (De Haes & Grembergen, 2005). Thus, integrating the two models into the 'model of ITG measurement assessment' (IMA model), enable us to view the structures, and processes of ITG measurement tools in the operating phase of integrated ITG framework (Figure 2). Thus, this paper looks at evaluating measurement only on the ITG 'structures', and ITG measurement 'processes' on IT resources, IT risks, IT management and IT performance measurement. Since, relational mechanisms relate to participation and collaboration among management and not entirely on measurement, this construct was not taken into account in the IMA model.
[FIGURE 2 OMITTED]
GLOBAL PERSPECTIVES OF ITG IN ITG MEASUREMENT
IT governance frameworks, being repositories of IT-effectiveness knowledge over time, organizations develop a shared culture of behaviours, values and expectations about their IT processes (Marshall, Curry, & Reitsma, 2010). Culturally different from their American and European counterpart, the Asian region presents new opportunities while facing different challenges in the ITG implementation. Asian region faces new challenges in ITG implementations in terms of the absence of documented strategy, communication of strategy, derivation of tactical plans, technology-driven IT plans, data classification, software documentation, project ownership by business, stage-wise sign-offs, configuration management and IT performance assessment (Ramanathan, 2007). Thus, there is lack of research on IT Governance adoption that look specifically within the context of an emerging yet still developing Asian country (Othman, Chan, Foo, Nelson, & Timbrell, 2011). According to them, national culture is a major factor affecting users to adopt IT governance practices. This was emphasised by Jacobson (2009) who stated that a dominant approach that describes effective governance views it as a matter of achieving fit with the environment, which has at its roots in contingency theory. Thus, good IT governance practices are known and applied, but not uniformly applied across the organizations (ISACA, 2011). Research in this domain is scant, as only a handful of empirical studies have investigated the utilization of IT governance frameworks from an Asian perspective (Lin, Guan, & Fang, 2010). With scant research in the Asian region, this study of exploring the implementation of ITG measurement from a western and an Asian context assumes great significance. Hence, organizations from countries representing the Asian and Oceania region can provide regional comparisons. This directs the researchers to the two sub questions: (1) What are the contextual differences or similarities in ITG frameworks implementation between the two regions? (2) What are the contextual differences or similarities in ITG measurement frameworks implementation between the two regions?
Research design being considered an action plan for answering an initial set of questions (Yin, 1994), this section assists in providing answers for some basic questions namely 'what', 'why' and 'how' of the study (Blaikie, 2000) through answering the research questions. Finding answers to the research question entails looking at the different modes of social research. Among the three approaches to social research namely quantitative, qualitative, and mixed approach (Cresswell, 2003), we follow qualitative research methodology as, it is deemed to be much more fluid and flexible than quantitative research in that it emphasizes discovering novel or unanticipated findings (Bryman, 1984). Since, the research questions are specified prior to the study by researchers who are observers/investigators rather than participant's case study research was deemed appropriate (Benbasat, Goldstein, & Mead, 2002). Thus in the proposed research we follow the qualitative approach using case studies, as the objective is to understand the phenomenon from the point of view of the participants and the particular context (Kaplan & Maxwell, 1994). The proposed study involves research into four organizations (two commercial banks and two government organizations) in New Zealand and in the United Arab Emirates. The first author has been a member of the Auckland (New Zealand) chapter of the Information Systems Audit and Control Association (ISACA), as well as the UAE ISACA chapter (ISACA is a worldwide organization with over 95000 members engaged in IT governance audit, assurance, and security). As a general rule, the number of replications is a matter of discretionary and judgmental choice, it depends upon the certainty a researcher wants to have about the multiple-case results (Yin 1994; Eisenhart 1989, cited in Pare, 2001,p. 14). Furthermore, there are no rules for sample size in qualitative inquiry, as it depends on the purpose of the research, and what can be done with available time and resources (Patton, 2002). Hence, we limit the study to four organizations in two countries. In qualitative research, researchers look for 'evidence' and 'theory' (Gillham, 2000) which comes in the form of interview responses. In this regard, this research employs in depth semi-structured interviews of respondents (See Appendix 1 for questionnaire schedule) who have taken a major role in the implementation and measurement aspect of ITG frameworks and standards. We aim to ensure construct validity through data triangulation in the form of interviews and measurement reports.
ANALYSIS OF DATA
The requirements for field research specified in section three were implemented with minor variations (part of the construct validity could not be ensured due to only one type of data being collected--interviews. Except in the case of the bank in UAE, reports of measurements were not shown to the researchers). The selection of the organizations for the cases has been sourced through the ISACA chapter network in Auckland (NZ), Wellington (NZ), and Dubai in United Arab Emirates based on two main criteria. (1) They should have implemented an ITG framework or are in the process of implementing it, and (2) should have a senior or middle management personnel solely responsible for the creation and/or evaluation of ITG measurement tools. In addition, it was decided to select one organization from the government sector and one from the private sector in each of the two countries to evaluate the similarities and differences in the measurement domain.
The collected qualitative data follows the five steps outlined by LeCompte (2000) namely tidying up, finding items, creating stable sets of items, creating patterns, and assembling structures. In this section, the obtained data (transcript) was tidied up, categorizing into different themes, thus creating stable sets of items using the qualitative analysis software NVIVO. The subsequent 'discussion' section outlines the issues from a measurement perspective based on patterns and assembling structures, using the simple influence diagram (Palvia, Midha, & Pinjani, 2006) to answer the research questions. For the purpose of anonymity (as requested by the respondents), the names of the respondents have been disguised as NZ bank and NZ government, UAE bank and UAE government. Care has been taken to select organizations similar in size and operations. Both the banks are based locally in New Zealand and the UAE with the main operations based in their home countries. Likewise the government organizations in both the countries are large organizations and among the top five employers in the government sector.
PROFILE OF THE NZ BANK:
This is one of the top three banks in NZ in terms of turnover and has a structured IT governance plan that is risk based, rather than based on COBIT, ITIL, ISO, or BASEL II. Being New Zealand based, it operates mainly in NZ and Australia, with limited multinational presence. Their motivation for ITG started with compliance requirement with BASEL II. The interview was conducted with the IT Governance Manger at their head office during May 2015.
Structures (frameworks) for monitoring of IT resources, IT risks, IT management, and IT performance measurement for NZ Bank (Table 1)
Monitoring is done through IT governance using tools namely ITIL, COBIT, and ISO 27 K series. When quizzed about the motivation to use COBIT and ITIL the IT Governance Manager stated, "They're all very good and mature frameworks, used widely in organizations, and the most effective industry standards." Their audit work is aligned with COBIT but does not follow a systematic process; rather they only use it as a guideline into their planning process. "So we use COBIT in the audit space. Our external audit work is aligned with COBIT but internally the organization is using ITIL in the operational area. Our IS security function is aligned with ISO 27001". In the case of ITIL, some modules like change management have been implemented in the IT operations domain, with ITIL aligned with ISO 27001 in the security domain. Hence, the bank has a hybrid model of homegrown IT governance framework based on the three models as is evident in the statement "we use our own policies and processes as drivers as accepted good practice". In the measurement aspect, they use KPI taken from COBIT apart from customized ones.
Processes (measurement tools and process) for monitoring of IT resources, IT risks, IT management and IT performance measurement for NZ Bank (Table 2)
Measurement Tools: The measurement tools used in NZ bank includes, the business IT goals alignment (B-IT) methodology, the balance scorecard (BSC), the heat map (HM), metrics, the maturity model (MM) and the risk matrix (RM). While the heat map and the risk matrix are the main tools used to measure IT risk, the balance scorecard is used to measure the performance of their IT assets, while the Capability Maturity Model Integration (CMMI) is done on an ad hoc basis. The first three are done in a comprehensive manner, internally by their own staff while, external auditors do the maturity level determination. The organization is not using any tool to measure the alignment of IS to business goals and metrics are sourced from COBIT apart from using their own customized ones.
The measurement tool (Business--IT goal alignment)
The role of IT in the bank is to support its strategies and objectives. Therefore, in terms of planning, the business units, the front line units, and support units, plan and design the key goals, strategies, objectives for the year from a business perspective. Subsequently, the technology units do the lower end planning. Thus, the business IT alignment that starts from the top is driven by the business with the IT plan as the support function, thus supporting the business goals. However, they do not have any tool to measure the strength of this alignment.
The measurement tool (Balance score cards): They use different score cards for measuring different aspects of the technology (IT performance) from a high level perspective, where some are done monthly, and others done quarterly. In this process, they use multiple key performance indicators that are tracked on a regular basis and reported.
Process: The methodology of the BSC has been described by the respondent as follows. Each technology unit will have their own reports, drivers, and metrics. For example, for operations they deploy metrics for systems uptime and systems availability whereas, in the development domain, the measures are completely different like the number of bugs, or lines of codes, while in the security space different metrics are used to measure the objectives. Therefore, each technology unit will have their own measures and at each level, where scores are fed to a higher level thus creating upward cascading effect.
The measurement tool (heat map): The heat map is a measurement tool used to measure risk and the ensuing process taken from the ITG frameworks. They employ a risk-based approach in their audit process. "So what we do is to specify the risks... and then we'll do our audit testing or come up with an audit program, we use ITIL, COBIT program etc. as input into designing the control objectives, the detailed control objectives etc., but the final report will measure against the original risks, but not against the control objectives". The tool used to measure is a heat map. "So we will get a traffic like rating like red, amber, green" and these are mapped against the original risks.
Process: The audit process starts with the specification of the risk rather than the control objectives. "Sure, as I have said we start with the risks; identify the risks, and then for each risk we decide the key processes". They make an assessment or formal opinion about how a specific control is being operated, to see its operational effectiveness. They not only evaluate to see whether the control (to mitigate the risk) was adequately designed but also oversee whether the control is operating well to mitigate the risk. If the answer is "yes" then they allocate a green rating. If the control is not working operationally or, if it has deficiencies in its design, then they may allocate an amber rating for the residual risk. Moreover, if a control is completely missing or if it is not operating at all, they allocate a red rating for the original risk. Since, there is no one to one relationship between risks and controls, one risk could be tested for a number of controls. Subsequently, one risk normally has a whole set of controls associated with it. Therefore, the optimal rating is dependent on the outcome of a comprehensive set of controls relating to that one risk.
The measurement tool (metrics): They use metrics in KPI sparingly. According to the respondent, the metrics in KPI are used "sparingly on a case by case basis ..., sometimes we use the KPIs in COBIT; sometimes we have our own customized organizational KPIs". These are considered as targets to achieve which they perceive as drivers.
The measurement tool (maturity model): This tool is not a commonly used measurement framework. According to the respondent they use it "sometimes, not every time, and it's mostly ad hoc". However, they did a one-time external audit exercise where they obtained a maturity rating on the COBIT areas. This was a "quick and courteous assessment" of the maturity for each specified COBIT area rather than a continuous formal assessment. Subsequently, they do not conduct this exercise on a regular basis.
The measurement tool (Risk matrix): Under their operational risk framework, they have a tool called the risk matrix where the risks are defined in terms of its likelihood on one axis and its impact on another axis, with a 1 to 5 rating for impact, and 1 to 5 rating for likelihood.
Process: Thus, it forms a 5 by 5 grid with detailed definitions of what a 1.1 impact is as compared to a 5.5 impact on different aspects. The co-ordinates are well defined, where each impact has a definition, and each likelihood of risk has a probability rating for it. The matrix is standard throughout the organization under the operational risk framework, and so everybody talks the same language. If one business unit calls the risk 'medium', another business unit will understand what a 'medium' risk means. Thus, the matrix values are consistent throughout the organization.
PROFILE OF NZ GOVERNMENT
Being a government department dealing with finance, they have appointed a person to oversee IT governance implementation and management. The interview was conducted with the IT Audit Manager during May 2015. Currently they are moving away from "mainframe technologies into commercial IT shop products". Since, managing a mainframe is different from the latter; they stated the need to establish an "organizational structure, an IT structure hardware, networking, and architecture" in the organization. They implemented IT governance concept based on a risk-based approach using a customized 'IT governance form' and a 'heat map' for measurement, whereas COBIT was implemented by an external entity in stages.
Structures (Frameworks) for monitoring of IT resources, IT risks, IT management, and IT performance measurement for NZ government (Table 3) The organization implemented selected domains of COBIT, ITIL, PRINCE II, and few areas of CMMI, but did not deploy 17799 or ISO 27 K series frameworks, except SAS 70, and an equivalent of Basel II. Information Technology is heavily outsourced and so the focus of governance is on the 'commercial contractual' space as "58 % of our running costs are in the outsourcing space". They have IT running cost of NZ $ 32 million. Hence, 32 % of this is outsourced which comes to NZ $ 18.56 million. COBIT maturity model was not used for assessing the maturity level, since they view IT governance through COBIT controls from a RACI perspective, while ITIL is deployed at the IT operational level.
Processes (measurement tools and process) for monitoring of IT resources, IT risks, IT management and IT performance measurement for NZ government (Table 4)
The measurement tools deployed are the BSC, HM, metrics, MM, and the risk matrix.
IT business goal alignment matrix: There is not much evidence of using a tool to measure the business IT alignment apart from stating that IT is used to support business strategies and objectives.
The measurement process: Currently they do not align organizational goals with the business goals, which according to the respondent is a "real gap at the moment", but their planned transformational exercise is a key enabler for this alignment. However, they do a similar exercise explained under the metrics section.
The measurement tool (Balance Score Card): They use the BSC as a key enabler to their audit.
The measurement process of the BSC: They use it on a monthly basis to measure IT performance. They link BSC with the best practices of Gartner. In this regard, they follow the principles of the BSC in terms of the cascading effect, but it is not linked to COBIT. First, they measure their strategy, followed by the business unit plan, finance, people, and performance. The lower end performance metrics are grouped, aggregated up, and visualized using heat map.
The measurement tool (heat map): The heat map is an operational tool that they implemented to report on risks like an outage or severity, the state of the system, and the email system in terms of its availability. It is graphical user interface of the risk register and the BSC, both of which are not linked to COBIT.
The measurement process of the heat map: In the process, they considered a few factors in terms of their core systems. The heat map showed the severity in terms of colour like green, amber, and red. If the colour is green or amber no steps are taken, but if it goes to green then they come up with a green plan'. "If something hits a red, we escalate it and the move to a green plan in order to manage the risk." These plans have been created either by the system owner or the person who is accountable for the availability and stability of the system for which the risk is reported. Hence, for each type of risk, there is a green plan listed on a form called the 'IT governance form". Even though this is called an IT governance form, this is not linked with COBIT.
The measurement tool (metrics): The organization use IT metrics provided annually Gartner Inc., to measure their IT investments from five critical perspectives namely IT enterprise, IT infrastructure, applications, information security, and IT outsourcing.
Process: The process starts from the top where they measure their strategic plan, cascading down to their business unit plan. Subsequently, they measure components finances, people performance, and change management. The values from the bottom are aggregated to each of the top layers, which are then visualized as a radar through a heat map.
The measurement tool (maturity model): They have a six sigma person in their organization who comes through and administer the six sigma maturity model process through a set of questions to a cross functional team of 25 people covering the entire organization.
The measurement process of the maturity model: The process is done through a series of questions individually done, where they measure results against the outcome of those questions. At the time of this interview, they were measured at 1.9.
The measurement tool (risk matrix): They use a risk register matrix with the likelihood of occurrence on one axis and the severity/consequence on the other axis. Therefore, this form of measurement enables them to come up with their risk profile.
The measurement process of the risk matrix: Regarding the assessment process, they consider risk as a core part of their governance. In this respect, they have set parameters within the systems, and once they move outside these parameters, the risk management process is activated where they use the IT governance form as the means to track the governance of the risk.
PROFILE OF UAE BANK:
Stared during the 1970s, it is one of the larg banks in the UAE. During the beginning of the year 2006, they started to implement best practices and standards in the IT department. At the turn of the last century, they built a new service architecture and changed the core banking system. In this regard, their first initiative was the implementation of incident management in ITIL followed by COBIT controls. A series of three interviews was conducted with the IT Strategy Manager from June 2013 to January 2014.
Structures for monitoring of IT resources, IT risks, IT management and IT performance measurement for UAE bank (Table 5)
They use COBIT, ITIL, PRINCE II, TOGAF, and Zackman framework for enterprise architecture. Regarding standards, they follow three standards namely ISO 9001 for quality management, ISO 20000 for ITIL, and ISO 27001 for security with PMBOK as the foundation for implementation. IT governance is viewed as a comprehensive overarching framework acting as an umbrella covering all other frameworks and standards.
Processes (measurement tools and process) for monitoring of IT resources, IT risks, IT management and IT performance measurement for UAE bank (Table 6)
The measurement tools used are the heat map, the BSC, the maturity model, the ITbusiness goal alignment matrix, and KPIs.
The measurement tool (IT-business goal alignment matrix): The bank uses the balance scorecard to link the IT goals with the business goals:
Process: The process starts with the high level strategic objectives, linked down to the corporate objectives, which is further linked to the IT objectives, the IT goals, IT goal initiative, and finally to the KPIs. They use a 0 to 5 value matrix to measure the alignment between business goals and IT goals. Towards the end of the year these are aggregated and measured upwards for an aggregated value.
The measurement tool (balance scorecard): They have the corporate balance scorecard covering the entire organization (including the branches) cascaded to the lowest level of KPI.
Process: The balance scorecard at the top level is cascaded to each division, and this is further cascaded to the department. In the department, they set up goals based on the balance scorecard target. The goals are transferred into projects and initiatives translated into KPI. In this measurement tool, ITG is only one part of the BSC domain. All the KPI have been linked to the BSC.
The measurement tool (heat map): The heat map is used by the risk department and the IT audit department. The risk management division employ this tool for the IT security rather than governance.
Process: Once the risk department conducts penetration testing and related IT security tests, they prepare a heat map from an IT security perspective. This is passed on to the IT audit division who increments the heat map periodically and submit it to the audit committee. It encompasses the entire audit observation and the audit risk, covering selected IT controls from COBIT, ITIL and the ISO standards. The automated heat map provides efficient and effective external audit.
The measurement tool (metrics): The bank use KPIs and metrics based on a variety of quantitative and qualitative scales, but mostly quantitative.
Process: The majority of the metrics are based on percentage. For example, a KPI will denote the targets they have to achieve and based on that, a percentage is given. Apart from that, they also use ratings scales from 0 to 5. Sometimes the metrics are derived through simple calculation.
The measurement tool (maturity model): The bank is already using the maturity model for ITIL for service management and currently moving towards PMBOK maturity model. The PMBOK maturity model was recommended by their consultant who stated that they should have it under the PMO. Regarding the ITIL MM they are already reached a maturity level of 2.0, and currently aiming for 3.0.
Process: They are using the enterprise monitoring systems and the robotics transaction systems, with system availability as the prime focus of ITIL. According to the respondent, the three requirements that makes the ITIL maturity goes up are incident management, program management and change management. They have outsourced the monitoring of the availability of their critical system to an external company. This system makes sure that the ITIL is proactive rather than reactive. In this respect, they have aimed for an ambitious 99.99 availability in the short term with a long-term goal of 99.9999% IT service availability.
PROFILE OF UAE GOVERNMENT
This is one of the five largest government organizations in UAE in terms of work force. The interview was conducted with their five member IT Governance team at their office during July 2013. Towards the end of 2005, they decided to implement IT controls. Hence, according to the respondent the idea of implementing IT governance developed because "in any dynamic environment with such rapid development and rapid changes happening, you will need to have some sort of control on what is happening mainly to know that you are doing the right things in a right way." Therefore, to implement IT governance they looked at what other organizations in similar sector are doing so that they "don't reinvent the wheel." Since the respondent had experience working with COBIT from his previous job and some of his colleagues in his department knew about COBIT, this is the first control framework they implemented along with ITIL. When they started to study COBIT to see which all controls need to be implemented they found out that most of the processes that they are doing are in line with COBIT processes.
Structures for monitoring of IT resources, IT risks, IT management, and IT performance measurement for UAE government (Table 7)
They integrate COBIT and ITIL aligned with ISO 27001 and ISO 20000 respectively, since majority of ITG activities things that they do as part of the COBIT, map with ITIL and vice versa. They are already ISO 20000 certified and working towards getting ISO 27001 certification. One of the reasons cited for choosing COBIT is that the UAE government audit department, which conducts audits, advises them to use COBIT including the list of controls to use.
Processes (measurement tools and process) for monitoring of IT resources, IT risks, IT management, and IT performance measurement for UAE government (Table 8).
The measurement tools used are the COBIT maturity model, IT business goal alignment, heat map in the project space, and metrics.
The measurement tool (Business--IT goal alignment): They use this tool to align their eight high level strategic objectives with IT goals up to the lowest level technical IT objectives.
Process: This process is illustrated by the respondent through an example. They have eight strategic objectives with sub objectives. For example, taking the high-level strategic objective #7, (Develop human resource, improve organization efficiency, and improve processes), there are sub objectives, and detailed sub objectives, followed by technical objectives that comes under IT (Ex. automate processes, and improve automation through deployment of the latest IT technology). Thus, this connects back to the strategic objective thus supporting the high-level strategic objective #7.
The measurement tool (balance scorecard): Apart from aligning and cascading the strategic objectives down to the KPI of IT, they do not use the BSC.
The measurement tool (heat map): The heat map is indeed used at the project management level (red, amber, and green). They use a dashboard approach for gaining information from the heat map tool.
Process: The CEO's office uses the heat map dashboard that shows the strategic objectives of the government, which are linked to organizational strategic objectives and how these are mapped to each project. Since it is automated, senior managers can drill deep into the three colours of the heat map to get granular results (from the aggregate). It illustrates the lower level objectives, display the problem with that objective, view the status of all initiatives associated with even the low-level objectives. Based on this, within a few minutes they can drill deep and ascertain whether a strategic objective is meeting the target or not, and can take appropriate decisions. This is "one of the system that will not bring any revenue, but helps in decision making."
The measurement tool (metrics): They use metrics and one of the challenge that they faced is the manner in which they measure the metrics for the maturity level, but for the IT processes, they use a rating scale of 1 to 5.
Process: Regarding the use of metrics for the different levels of the maturity model, the issue they faced was the challenge of defining the matrix, and the issue of putting weights for processes, as the respondent feel that these can be subjective. Spreadsheet was used for measurement of IT processes, where they use rating scale of 1 to 5 for most of the processes to ensure consistency and objectivity in the measurement results. There are a few areas where a rating scale was not appropriate like in the case of 'number of incidents'. They solved this issue by rounding it to a value in the rating scale. According to the respondent, the rating scale ensured consistency in tracking the progress of the IT processes using time series analysis over a period.
The measurement tool (maturity model): This is the foremost tool used for measurement in the ITG domain where they achieved a maturity of 2.6, the highest among all the UAE government departments.
Process: They have an external audit done regularly from the UAE government to audit them on their COBIT maturity level. According to the respondent, the government, "audit us based on COBIT. When they come to audit us, over a period of three to four months, they drill deep down into extreme details of all domains, processes culminating in a detailed report of the current standing, the maturity level along with recommendations to achieve the next level". This exercise helped them to see their gaps as well as the areas to focus on. The employees are given trainings in implementing the maturity level and eventually they started doing this exercise without the help of external consultants.
Evaluating the two sub questions necessitate viewing the ITG implementation and subsequent measurement in the two countries (in two regions) and the two different sectors to answer the main question: How do organizations use ITG measurement tools to assess IT processes and IT controls of the ITG frameworks/standards and processes?
(1) What are the contextual differences or similarities in ITG frameworks implementation between the two regions?
Organizations globally face challenges in terms of selection and integration of ITG frameworks, hence, differences in integrating relevant ITG frameworks in two regions (under study in this research) are evident and expected.. Empirical research indicate higher geo cultural differences than the sector wide differences on the ITG practices followed..
Regarding the ITG initiation process, from a New Zealand perspective, it has been observed that COBIT is not the starting point of an IT governance process, but a risk based approach is used to audit IT using traces of governance processes, and as stated by Merhout and Havelka (2008) most audits are conducted using a 'risk based' approach (Figure--3). This is true in the case of New Zealand only where neither of the organizations in NZ starts their ITG exercise with the IT goals or the control objectives, but relevant goals are taken from COBIT or ITIL to attach to the risk framework. Therefore, COBIT is consulted rather than deployed as an umbrella framework.
In this regard, we see that there are distinct differences in integration/mapping. Hence, while the two organizations in UAE, initiate ITG from COBIT, with the control objectives as starting point along with risk, other relevant frameworks and standards are integrated under the COBIT umbrella. The main reason for having COBIT as an umbrella framework is the directive from the UAE government to banks and government organizations regarding COBIT implementation. The bank has an integrated ITG framework in place with COBIT at the top and ITIL linked to ISO 20000, ISO 27000, ISO 9000, TOGAF along with Zackman, PMBOK, and the TSO frameworks forming as pillars to support the overall COBIT framework. Likewise, the government organization also use COBIT as an umbrella framework mapped with ITIL and ISO 27001. Thus, it is evident from the analysis of the empirical data that while the ITG initiation process are different in the two countries, ITG practices are universally applied.
[FIGURE 3 OMITTED]
(2) What are the contextual differences or similarities in ITG measurement frameworks implementation between the two regions?
Regarding the question of specificity or universal application of measurement frameworks in ITG implementation (Figure 4), the major difference noted was the absence of the two-dimensional risk matrix in UAE. Another difference is the focus of objectivity in measurement in UAE organizations as opposed to organizations in NZ where organization in UAE gives much priority to quantitative rather than the qualitative measures. The manner of applying heat map presents distinct variations. Whereas in NZ, it was applied to the ITG domain, in UAE it focused more on security and project management. In the maturity model also the emphasis and objectivity was evident where in UAE it is used to evaluate the maturity level of the ITG frameworks of ITIL and/or COBIT as well as related frameworks (PMBOK). Moreover, this objectivity was also observed in UAE organizations regarding metrics where either percentage or rating scale were used. Globally using a risk based approach to initiate ITG is universal, and so the choice and integration of ITG frameworks.
[FIGURE 4 OMITTED]
While figure 3 and 4 shows the overall differences in the ITG initiation processes and measurement framework implementations between UAE and NZ, they do not present specific sector wise details. Sector wise analysis was also performed to analyze the difference in depth. When individual organizations were compared between these two countries, it was observed that differences were substantial and specific between both banks and government organizations (Figure 5). Except for the use of balance scorecards, differences in the implementations of all other ITG measurement tools were clearly evident. However, in the case of the bank, significant differences were evident for risk matrix followed by metrics and the maturity model, while moderate differences were evident in the case of heat map, and business--IT alignment implementations.
[FIGURE 6 OMITTED]
In the case of government organizations, the differences were substantial and in all the measurement tools used as is the case with the commercial banks (Figure 6). Drastic differences are observed for risk matrix, the balance scorecard, and heat map, while moderate differences observed for metrics, maturity model and the heat map. While similarities were observed in the use of IT-business goals alignment tool, there still were differences in the way they are used for measurement.
This study focused on the ITG measurement tools and its deployment methodology through the lens of the integrated ITG framework of Dahlberg and Kivijarvi (2006) and the SPR model of De Haes and Grembergen (2005).. While research abounds in ITG and its application in organizations, the deployment and use of ITG measurement frameworks is a scant area of research despite the relevance of these measurement tools to evaluate the success or failures of ITG frameworks, standards and models.
It can be concluded that the ITG frameworks, standards and models are global in nature, however, successful deployment requires these to be customized to geographic contexts. Similarly, it was observed that while all ITG measurement tools are deployed irrespective of the geographical context or sector, the methodology of its application is determined by the distinct practices of the regions.
The study contributes to our understanding of the differences in the deployment methodology of ITG measurement tools to evaluate the success of ITG frameworks, standards and models. From a practitioner's perspective, understanding the subtle but distinct differences in their deployment promotes adoption of contextual variables in its deployment leading to successful implementation and subsequent evaluation of relevant ITG frameworks.
The study is not without its limitations. First, we did not go to the extent of finding out the appropriateness of the ITG measurement tools or scales/metrics. Second, the limitation to two countries and two sectors can limit its generalizability. Thus, from an academic perspective, a few areas of research need to be explored further. First, there is a need to understand the most appropriate ITG measurement tools for specific ITG frameworks; which ITG measurement tool/s works better with corresponding ITG frameworks, standards and models. Second, it would be of much interest to the academic community to ascertain appropriate scales/metrics for each of the ITG measurement tools, which would be of interest to the practitioners too. Third, extension of this study to a wider context and sector can generalize the findings as multiple case studies in diverse regions and sectors can elicit universal as well as regional practices in ITG measurement. The above three research domains could present a 'success factors matrix for ITG measurement' from multiple perspectives (global, regional, and sector wide for the ITG measurement tools mentioned in this study). Fourth, practitioners would want to know the impacts on organizations that quantitatively measured their IT effectiveness and alignment. In this regard, future researchers could not only evaluate the measurement tools deployed and the metrics used, but also evaluate if these measurements tracked over time, have provided them with greater audit control and enhanced IT-business alignment. Fifth, while the questions focused mainly on the application of these tools, future in depth interviews can elicit information on the role of IT and the IT infrastructure in ensuring effective ITG measurement.
Alfaraj, H. M., & Qin, S. (2011). Operationalising CMMI: Integrating CMMI and CoBIT Perspective. Journal of Engineering, Design and Technology, 9(3), 323-335.
Baer, D. R., & Dietrich, M. (2006). Validation of IT-Security Measurement Tools. Paper presented at the Proceedings of the First International Conference on Availability, Reliability and Security (ARES'06).
Basili, V., Lindwall, M., Regardie, M., Seaman, C., Heidrich, J., Munch, L.,... Trendwitz, A. (2010). Linking Software Development and Business Strategy Through Measurement. Computer, 43(4), 57-65.
Benaroch, M., & Chernobai, A. (2012). IT Operational Risk Events as COBIT Control Failures: A Conceptualization and Empirical Examination. Paper presented at the Proceedings of the 6th Israel Association for Information Systems (ILAIS) Conference, Haifa.
Benbasat, I., Goldstein, D. K., & Mead, M. (2002). The Case Research Strategy in Studies of Information Systems. In M. D. Myers & D. E. Avison (Eds.), Qualitative Research in Information Systems - A Reader (pp. 79 - 99). London: Sage Publications.
Blaikie, N. (2000). Designing Social Research. Malden: Blackwell Publishers Ltd.
Bryman, A. (1984). The Debate about Quantitative and Qualitative Research: A Question of Method or Epistemology? The British Journal of Sociology, 35(1), 75 - 92.
Cresswell, J. W. (2003). Research Design: Qualitative, Quantitative, and Mixed Methods Approaches Thousand Oaks: Sage Publications.
Dahlberg, T., & Kivijarvi, H. (2006). An Integrated Framework for IT Governance and the Development and Validation of an Assessment Instrument. Paper presented at the 39th Hawaii International Conference on Systems Sciences, Hawaii.
De Haes, S., & Grembergen, W. V. (2005). IT Governance Structures, Processes and Relational Mechanisms: Achieving IT/Business Alignment in a Major Belgian Financial Group. Paper presented at the 38th Hawaii International Conference on Systems Sciences, Hawaii.
De Haes, S., Van Grembergen, W., & Debreceny, R. S. (2013). COBIT 5 and enterprise governance of information technology: Building blocks and research opportunities. Journal of Information Systems, 27(1), 307-324.
Debreceny, R., & Gray, G. L. (2009). IT Governance and Process Maturity: A Field Study. Paper presented at the 42nd Hawaii International Conference on System Sciences - 2009, Hawaii.
Debreceny, R. S. (2006). Re-engineering IT Internal Controls: Applying Capability Maturity Models to the Evaluation of IT Controls. Paper presented at the 39th Hawaii International Conference on Systems Sciences, Hawaii.
Gillham, B. (2000). Case Study Research Methods. London: Continuum.
Grembergen, W. V., Haes, S. D., & Guldentops, E. (2004). Structures, Processes, and Relational Mechanisms for Information Technology Governance: Theories and Practices. In W. V. Grembergen (Ed.), Strategies for Information Technology (pp. 1-36). London: Idea Group Inc.
Hinkelmann, K., Gerber, A., Karagiannis, D., Thoenssen, B., Van der Merwe, A., & Woitsch, R. (2016). A new paradigm for the continuous alignment of business and IT: Combining enterprise architecture modelling and enterprise ontology. Computers in Industry, 79, 77-86.
ISACA, I. (2011). Global Status Report on the Governance of Enterprise IT (GEIT)--2011. Available on line at http://www. isaca. org/Knowledge-Center/Research/Documents/Global-Status-Report-GEIT-10Jan2011-Research.pdf.
Ivanov, M. (2012). Success in Information Technology Projects: A Comparative Review Based on the CobiT PO10 Maturity Model and Suggestions from Literature. Paper presented at the International Conference on Information Resources Management (CONF-IRM), Vienna.
Jacobson, D. D. (2009). Revisiting IT Governance in the Light of Institutional Theory. Paper presented at the 42nd Hawaii International Conference on System Sciences Hawaii.
Kaplan, B., & Maxwell, J. A. (1994). Qualitative Research Methods for Evaluating Computer Information Systems. In J. G. Anderson, C. E. Aydin, & S. J. Jay (Eds.), Qualitative Research Methods for Evaluating Computer Information Systems (pp. 45 - 68). Thousand Oaks, California: Sage Publications.
Kappelman, L., McLean, E., Johnson, V., & Torres, R. (2016). The 2015 SIM IT Issues and Trends Study. MIS Quarterly Executive, 15(1).
LeCompte, M. D. (2000). Analysing Qualitative Data. Theory into Practice, 39(3), 146 - 154.
Lin, F., Guan, L., & Fang, W. (2010). Critical Factors Affecting the Evaluation of Information Control Systems with the COBIT Framework: A Study of CPA Firms in Taiwan. Emerging Markets Finance & Trade, 46(1), 42 - 55.
Maria, E., Fibriani, C., & Wijaya, L. S. (2012). The Measurement of Information Technology Performance in Indonesian Higher Education Institutions in the Context of Achieving Institution Business Goals Using COBIT Framework Version 4.1. Journal of Arts, Science & Commerce, 3(3), 9-19.
Marshall, B., Curry, M., & Reitsma, R. (2010). IT Governance Norms and IT Success. Paper presented at the 2nd annual Pre ICIS Workshop on Accounting Information Systems, Saint Louis, MO, U.S.A.
Merhout, J. W., & Havelka, D. (2008). Information Technology Auditing: A Value Added IT Governance Partnership between IT Management and Audit. Communications of the AIS, 23(26), 463-482.
Nicho, M., & Muamaar, S. (2016). Towards a Taxonomy of Challenges in an Integrated IT Governance Framework Implementation. Journal of International Technology and Information Management, 25(2), 2.
Othman, M. F. I., Chan, T., Foo, E., Nelson, K., & Timbrell, G. (2011). Barriers to Information Technology Governance Adoption: A Preliminary Empirical Investigation. Paper presented at the 15th International Business Information Management Association Conference, Cairo.
Palvia, P., Midha, V., & Pinjani, P. (2006). Research Models in Information Systems. Communications of the Association for Information Systems, 17(47), 1041 - 1059.
Pare, G. (2001). Using a Positivist Case Study Methodology to Build and Test Theories in Information Systems: Illustrations from Four Exemplary Studies Retrieved from http://gresi.hec.ca/SHAPS/cp/gescah/formajout/ajout/test/uploaded/cahier0109.pdf.
Patton, M. (2002). Qualitative Research & Evaluation Methods (Thousands Oaks, Sage).
Ramanathan, S. (2007). IT Governance: IT Governance-Challenges in Implementation From an Asian Perspective. Information Systems Control Journal, 5, 26-27.
Simonsson, M., Johnson, P., & Wijkstrom, H. (2007). Model Based IT Governance Maturity Assessments With COBIT. Paper presented at the 15th European Conference on Information Systems, Switzerland.
Stevens, F. (2011). Frameworks for IT Governance Implementation. In N. S. Shi & G. Silvius (Eds.), Enterprise IT Governance, Business Value and Performance Measurement: IGI Global.
Stockton, J. L. (1998). Discussion: A Methodology for Developing Measurement Criteria for Assurance Services: An Application in Information Systems Assurance Auditing: A Journal of Practice & Theory, 17(Supplement), 99 - 102.
Walker, A., McBride, T., Basson, G., & Oakley, R. (2012). ISO/IEC 15504 Measurement Applied to COBIT Process Maturity. Benchmarking: An International Journal, 19(2), 159-176.
Yin, R. (1994). Case Study Research: Design and Methods (2nd ed.). Thousand Oaks: Sage Publications, Inc.
Zhou, X., & Cai, S. (2011). Research on the Measurement of IT-Business Alignment. Paper presented at the The International Conference on Management and Service Science (MASS), Wuhan, China.
Mathew Nicho (Robert Gordon University),
Shafaq Khan (University of Dubai),
Table 1: ITG frameworks used for NZ Bank Frameworks Emphasis Process COBIT Not used as a primary Do not start the ITG process tool, but serves only with the COBIT framework, but as a guideline for their use traces of COBIT, like overall governance. Used selecting a few KPIs of COBIT. by the external auditors The bank as such does not align but not internally. They the KPI in COBIT with the use a risk based approach processes. The final report in ITG rather than COBIT measures against the risk and approach not the control objectives. They use only those areas that is relevant to them. ITIL Primary tool, and used Implemented a few modules like internallyas a incident and change management comprehensive tool for ITG ISO 27001 Deployed in managing IT This is aligned with ITIL; do security not follow it step by step, but use only as a guideline BASEL II Deployed it in the area Not used as an ITG tool and, is of capital holding, but not aligned with any other not integrated with ITG frameworks or standards Table 2: ITG measurement tools used for case the NZ bank Framework Emphasis Process B-IT Align business goals to This process starts from the IT goals organizational strategies and objectives and cascades down to the IT level, but there is no measurement tool to measure the strength of alignment BSC This is used to track the Done on a monthly and quarterly KPIs of the various basis, the KPIs are tracked entities of IS technology regularly on a chart for and used for IT performance evaluation. Each performance measurement technology area is measured, aggregated, and reported to a higher level. Use specific metrics like 'systems uptime', 'system availability', etc. Does not use COBIT in this process HM Used as a tool to align There are different people with the risk matrix. reporting from different This is the outcome of departments on the heat map. The the report on risk matrix people who manage the risk matrix link the values ranging from 0.0 to 5.5 to the heat map which then provides an output in the form of green, amber and red Metrics KPIs from COBIT and They use the metrics for the KPIs customized that are borrowed from COBIT as well as use their own customized ones. MM Used rarely It is mostly done on an ad hoc basis and conducted by an external consultant who give them a maturity rating based on COBIT RM Called the Operational This is done by charting the Risk Matrix, this tool likelihood of occurrence on one measures risk axis and the impact of this risk on the organization on the other axis. Scale range from 1 to 5 for both the axis. The different bi values are well-defined, and highly consistent between departments such that each understand the language of the other in terms of this value Table 3: ITG frameworks used for NZ Govt. Frameworks Emphasis Process BASEL II The use an equivalent of The NZ audit team comes in in Basel II, but not in depth to review them in terms of their control objectives. Therefore, the motivation for NZ audit is to ensure that their financial statements are measured correctly. in is to, to ensure our financial statements ah, can be measured correctly. So our systems obviously have to be at managed to a state, that's the only the closest thing I can say is like a Basel II, but not at in depth COBIT Not used as a primary They are in the infancy stages tool, but used as a of COBIT implementation. At the support framework for time of this interview, they overall IT governance and used only fifteen controls of is used based on external COBIT which is externally recommendation. audited ISO 27001 Do not use this standard They use SS 70-008 standard focusing on the physical and logical security for IT resources ITIL ITIL is used primarily in They use it to implement the IT the operational governance service management. It is space aligned with the IT goals rather than the business goals. The focus is to make sure that they ensure basic incident management, problem management, change management, configuration management, and asset management PRINCE PRINCE 2 is used in the They use PRINCE 2 and SDLC for project management IT project delivery Table 4: ITG measurement tools used for the NZ government Framework Emphasis Process B-IT Not used Currently they don't align the organizational goals with the business goals BSC This is a key enabler to They use it on a monthly basis their organizational unit measuring the IT resources through a cascading process to show an overall picture of the measures in green, amber, and red thus linking it to the heat map HM This is the graphical user It is a tool that shows the interface of the risk product of the risk register, register and the BSC in terms of visualizing the outcomes in terms of green, amber, and red. If the color goes to red, they select a green plan from the database called the 'IT Governance form' and get approval to implement it. Green and amber is left as such. Metrics IT metrics of Gartner Inc. The company use the IT metrics published annually by Gartner Inc. MM Used as a benchmarking tool They have used this to benchmark their department against a similar department in the Australian government Risk Called the Risk Register, The organization use a risk Matrix this tool measures IT risk register matrix with the likelihood of occurrence on one axis and the severity of consequence on the other axis. Thus, they show their risk profile based on a value ranging from 0.0 to 9.0 Table 5: ITG frameworks used for the UAE bank. Frameworks Emphasis Process BASEL II Not mentioned N/A COBIT Used as an umbrella for The TTG process starts from COBIT other frameworks and thus look ITG as a whole and integrate the other frameworks into COBIT ISO 27001 Used for security They map the necessary COBIT controls with ISO 27001 ITIL Used for IT service ITIL is aligned with COBIT as well as with ISO 20000 comprising of incident management, problem management and change management PMBOK Used for managing IT According to the respondent "in projects this part of the world PMBOK is used". They have started the documentation for measuring the maturity level of PMBOK Table 6: ITG measurement tools used for UAE bank Framework Emphasis Process B-IT 0 to 5 value matrix tool The business goals are aligned used with the IT goals and are measured using a value from 0 to 5 BSC Corporate BSC The goals starts from the top and are cascaded down to the IT level HM Used by the risk and IT The heat map is used in the IT audit department security and IT audit domains. The heat map covers select controls from COBIT, ITIL and ISO standards in the above two domains Metrics Use a mix of The majority of them are in quantitative and percentages. Even if these metrics qualitative KPIs are in other units they convert these into percentages as far as possible MM MM is used primarily They have reached a maturity level for ITIL for 2.0 for ITIL and going for 3.0 Risk This is not used N/A matrix Table 7: ITG frameworks used for UAE govt. Frameworks Emphasis Process COBIT Used as an overall high They mainly use COBIT for the IT level framework governance as most of the people are familiar with it ITIL Implemented ITIL Certified with ISO 20000 ISO 27001 Used in the security Working towards this certification space BASEL II Not applicable N/A PRINCE 2/ Do not follow any N/A PMBOK standard Table 8: ITG measurement tools used for UAE govt. Framework Emphasis Process B-IT This is used to align the They do not any measurement business with IT Goals framework to measure the strength of the alignment BSC Not used N/A HM Used at the project There is no evidence of using management level this at the ITG domain Metrics They use a rating scale They use 1 to 5 rating scale for from 1 to 5 measuring the IT controls, but for the COBIT maturity level they have difficulty in defining the metrics MM They use COBIT MM They have achieved a maturity measurement tool level of 2.6. They are audited by the UAE government audit team on a regular basis with an advisory role Risk Not used N/A matrix
|Printer friendly Cite/link Email Feedback|
|Author:||Nicho, Mathew; Khan, Shafaq|
|Publication:||Journal of International Technology and Information Management|
|Article Type:||Case study|
|Date:||Jan 1, 2017|
|Previous Article:||Communication through social technologies: A study of Israeli women.|
|Next Article:||Cloud Computing Technology: Leveraging the power of the internet to improve business performance.|