Printer Friendly

Evaluation of Teacher Digital Competence Frameworks Through Expert Judgement: the Use of the Expert Competence Coefficient.


The European Union defines key competencies as "a combination of knowledge, capacities and attitudes adapted to the context" (Council of the European Union, 2018, p. 7). Being competent is related to everything that society requires to overcome the obstacles of the time in which it develops (Cabero-Almenara & Palacios-Rodriguez, 2019; Palacios-Rodriguez, 2020). In the so-called knowledge society, one of these fundamental competencies is digital competence, i.e., "that which involves the safe and critical use of information society technologies for work, leisure and communication" (Council of the European Union, 2018, p. 9). However, 44% of Europeans have not developed basic digital abilities yet, whereas 79% use the internet regularly, at least once per week (Williamson, Potter, & Eynon, 2019). At the same time, all predictions indicate that most jobs in the future will require digital abilities (Cruz-Diaz, Ordonez-Sierra, Garcia, & Rabasco, 2016; Olivares, Angulo, Prieto, & Torres, 2018). Therefore, the capacity to use technology to live, work and learn throughout life is addressed as a transversal topic for the development of any educational programme (Romero-Martin, Castejon-Oliva, Lopez-Pastor, & Fraile-Aranda, 2017). In this sense, teacher training is considered as highly important (Lazaro-Cantabrana, Usart-Rodriguez, & Gisbert-Cervera, 2019; Suarez-Rodriguez, Almerich, Orellana, & Diaz-Garcia, 2018). This article presents and evaluates the main reference frameworks to improve teacher digital competence, which combines professional, pedagogical, technological and organisational capacities (Corsi, Dominguez, & Rodriguez, 2019; Kullaslahti, Ruhalahti, & Brauerm, 2019; Mishra, Koehler, & Bragg, 2006).

1.1 Teacher Digital Competence Frameworks

Different organisations and institutions have identified different indicators or standards that describe teacher digital competence (Munoz-Repiso, Martin, & Gomez-Pablos, 2020). These classify the competencies that teachers must develop with different dimensions and descriptors. This work presents those which, according to different authors, are most frequently used in the international context (Cabero-Almenara & Gimeno, 2019; Cabero-Almenara & Palacios-Rodriguez, 2019; Lazaro-Cantabrana et al., 2019; Padilla-Hernandez, Gamiz-Sanchez, & Romero-Lopez, 2019; Prendes & Gutierrez, 2013; Rodriguez, Mendez, & Martin, 2018; Silva, Morales, Lazaro, & Gisbert, 2019)

1.2 European Framework of Digital Competence for Teachers DigCompEdu (M1)

DigCompEdu was published in late 2017 by the Joint Research Centre of the European Union (JRC) (Redecker & Punie, 2017). Its main objective is to align the European educational policies with such reference framework. Moreover, it is a synthesis of scientific studies at the local, national, European and international level (Ghomi & Redecker, 2018; Redecker & Punie, 2017). DigCompEdu is a digital competence model with 6 differentiated competence areas (Figure 1). Each area has a series of competencies that "teachers must have in order to promote effective, inclusive and innovative learning strategies, using digital tools" (Redecker y Punie, 2017, p. 4).

1. Professional commitment: Capacity to use digital technologies to improve the teaching process and interact professionally with colleagues, students, parents and different agents of the educational community. Furthermore, this communication through technology allows for individual professional development and collective and continuous innovation in the educational organisation.

2. Digital resources: Identifying quality educational resources. Teachers must also be able to modify, create and share these resources to adjust them to their objectives, students and teaching styles. Likewise, they must know how to use and administer the digital content responsibly, respecting the author rights and protecting personal data.

3. Digital pedagogy: Knowing how to design, plan and implement the use of digital technologies in all the phases of the teaching process, promoting student-centred approaches and methodologies.

4. Evaluation and feedback: Digital technologies can improve the existing evaluation strategies and pave the way for new and better evaluation methods. Moreover, after analysing the large amount of available data (digital) about the individual interactions of students, teachers can provide more specific comments and support.

5. Empowering the students: One of the key strengths of digital technologies in education is their potential to boost the collaboration of students in the teaching-learning process and their autonomy in it. Moreover, digital technologies can be used to provide learning activities adapted to the competence level, interests and learning needs of each student.

6. Facilitating the competence: The capacity to facilitate the digital competence to the students is an integral part of teacher competence in ICT and the main theme of this competence area.

Jointly, DigCompEdu proposes six progressive competence levels (Figure 3). In this way, the digital competence level of a teacher is identified, from the Novice level (A1), or those with very little experience and contact with educational technology, to Pioneer (C2), or those who innovate with ICT.

1.3 ISTE standards for Teachers (M2)

The International Society for Technology in Education develops this competence framework focusing on the needs of the students of the 21st century (Crompton, 2017). Its main objective is to delve into the teaching practice, promote student collaboration, rethink the traditional approaches and boost autonomous learning (Crompton, 2017; ISTE, 2018; Perez-Escoda, Garcia-Ruiz, & Aguaded, 2019). The general teacher profile is characterised by being active and innovative in the teaching-learning process (Gutierrez-Castillo, Cabero-Almenara, & Estrada-Vidal, 2017). Thus, the ISTE standards for teachers are divided into seven roles or profiles that an educator must develop along his/her professional carreer.

1. Learners: Teachers improve their practice continuously by learning in collaboration with other teachers and exploring practices that make use of technology to improve the teaching and learning process.

2. Leaders: The search for leadership opportunities to support the empowerment and success of the students, improving their teaching-learning practices.

3. Citizens: Teachers are an example to students, contributing in a positive way and participating responsibly in the knowledge society.

4. Collaborators: Collaboration with colleagues and students to improve their practice, as well as to share and discover ideas and resources, solving problems using ICT.

5. Designers: Design of activities and environments with ICT adapted to the needs of the students.

6. Facilitators: Facilitating learning with technology to assist students develop digital competences.

7. Analysts: Evaluation and use of data to improve teaching and help students to meet their learning objectives.

1.4 UNESCO ICT Competence Framework for Teachers (ICT-CFT) (M3)

This framework, developed by UNESCO, presents "a wide range of competencies that teachers need in order to integrate ICT in their professional practice" (Butcher, 2019, p. 2). It fosters the practical knowledge of the advantages that ICT provide in education systems. Moreover, it suggests that teachers, apart from acquiring competencies related to ICT, must be able to use these to help their students to become collaborative, creative, innovative, committed and decisive citizens (Rodriguez et al., 2018). This framework presents 6 fundamental areas or aspects of the professional teaching practice:

1. Understanding ICT in the educational policies: Using ICT to support the specific objectives identified in the curriculum and contributing to the evaluation process.

2. Curriculum and evaluation: Acquiring abilities with ICT to support effective teaching and learning methods.

3. Pedagogy: Using ICT to support effective teaching and learning methods.

4. Application of digital abilities: Acquiring abilities with ICT to integrate them in the teaching-learning processes.

5. Organisation and administration: Administering the digital devices of the education centre, protecting the people who use them.

6. Professional learning: Using digital competence in the professional environment.

1.5 Common Spanish Framework of Digital Competence for Teachers (M4)

The Spanish Ministry of Education, Culture and Sport launched a project in 2012 to define the Common Framework of Digital Competence for Teachers, updated four times (Instituto Nacional de Tecnologias Educativas y Formacion del Profesorado. INTEF., 2017a, 2017b). It is based on the DigComp Framework of Digital Competence for Citizens (Carretero, Vuorikari, & Punie, 2017; Vuorikari, Punie, Carretero, & Van-Den-Brande, 2016). It is a generic digital competence model for educators. The competence areas (5) and competences (21) are those of the DigComp framework. Such areas are specified in Figure 3.

It is important to mention that this framework has 6 competence levels with a structure similar to that of DigCompEdu. The teacher digital competence level is identified in a similar way as the Common European Framework of Reference for Languages (Instituto Cervantes., 2002). This progressive level of autonomy and development begins at the basic level (A1) and ends at the advanced levels (C2).

1.6 Brittish Framework of Digital Teaching (M5)

The Brittish Framework of Digital Teaching was created by the Education and Teaching Foundation (ETF) in association with the JISC company (Education and Training Foundation., 2019). Its main objective is to increase the understanding of teachers in the use of digital technologies to enrich their teaching practices and improve their professional development (Perez-Escoda et al., 2019). This framework consists of 7 key areas, with 3 levels for each of them: exploration, adaptation and leadership. The seven elements are:

1. Pedagogical Planning: using digital technology to help in the teaching-learning processes.

2. Pedagogical Approach: using digital resources to engage the students in face-to-face, mixed and online learning.

3. Employability of the Students: using digital technologies to improve the employment perspectives of the students.

4. Specific Teaching: developing the specific knowledge of the subject and entrepreneur-ship through technology.

5. Evaluation: using digital technology to improve the evaluation process and provide feedback.

6. Accesibility and Inclusion: guaranteeing that all students can benefit from digital technology.

7. Self-development: reflecting on professional practice, developing digital competence and promoting a safe digital identity.
Figure 3 Areas of the Common Framework of Digital Competence for
Teachers. Source: Instituto Nacional de Tecnologias Educativas y
Formacion del Profesorado. INTEF. (2017b)

Information and information litera

*Identify, locate, retrieve, store, organize and analyze digital
information, evaluating its purpose and relevance.

Communication and collaboration

*Communicate in digital environments, share resources through online
tools, connect and collaborate with others through digital tools,
interact and participate in communities and networks; intercultural

Digital content creation

*Create and edit new content (texts, images, videos...), integrate and
rework previous knowledge and content, make artistic productions,
multimedia content and computer programming, know how to apply
intellectual property rights and use licenses.


*Personal protection, data protection, digital identity protection,
security use. safe and sustainable use.

Problem solving

*Identify digital needs and resources, make decisions when choosing the
appropriate digital tool, according to the purpose or need, solve
conceptual problems through digital means, solve technical problems,
creative use of technology, update one's own competence and of others.

1.7 Competencies for the Professional Development of Colombian Teachers (M6)

The model proposed by the Colombian Ministry of Education aims to guide the professional development of teachers to improve educational innovation with ICT (Fernanda, Saavedra, Pilar, Barrios, & Zea, 2013) . It is targeted at both designers of traning programmes and teachers interested in generating ICT-enriched environments: relevant, practical, established, collaborative and inspirating (Hernandez-Suarez, 2016). This framework has 5 competences that teachers must develop:

1. Technological: Selection and use of ICT tools (principles, combination types and author rights).

2. Communicative: Types of communication through virtual environments (synchronous and asynchronous).

3. Pedagogical: Related to teaching-learning processes mediated by ICT.

4. Management: Planning, managing, organising and evaluating the teaching-learning and institutional processes.

5. Research: Transforming and generating new knowledge using and reflecting on Information and Communication Technologies.

There are 3 specific levels of educational innovation with ICT (Figure 4).

1.8 ICT Competencies and Standards for Chilean Teachers (M7)

The Education and Technology Centre of the Chilean Ministry of Education published this framework in the year 2011, as an updated version of a previous framework published in 2006 (Elliot, Gorichon, Irigoin, & Maurizi, 2011). It presents 5 dimensions (Figure 5) aligned with the UNESCO Framework of ICT Competencies for Teachers (Butcher, 2019).

All 5 dimensions work through descriptors, criteria and compe-tencies. Moreover, each standard allows teachers to recognise how to use and integrate ICTs, identify their training needs and define personalised training itineraries (Rios, Gomez, & Rojas, 2018).


This study is part of a larger research project entitled "Design, production and evaluation of t-MOOC for the teachers' acquisition of Teaching Digital Competencies (DIPROMOOC)". One of the aims of this investigation is to create and evaluate a training environment under the t-MOOC architecture for the training of non-university teachers in the acquisition of the teaching digital competence.

To this end, the first objective is to identify the framework of digital competencies through expert judgement.

The expert judgement consists of "requesting a series of people to judge a teaching object, instrument or material, or to give their opinion regarding a specific aspect" (Cabero & Llorente, 2013, p. 14). This strategy is expanding in educational research-evaluation (Galicia, Balderrama, & Edel, 2017; Robles & Rojas, 2015) and it is strongly associated with Delphi studies (Lopez-Gomez, 2018; Mengual-Andres, Roig-Vila, & Mira, 2016). However, the concept of expert is quite polysemic. There is no unambiguous conceptualisation of the term "expert" that can help to specify its defining characteristics and what it means to be an expert. Therefore, the obtained results depend directly upon the quality of the experts selected for the evaluation process. To this end, different procedures can be applied, which range from contemplating the profile of the selected expert to more complex procedures, such as the expert competence coefficient (ECC) (Cabero-Almenara & Barroso-Osuna, 2013; Lopez-Gomez, 2018).

The present study establishes two mechanisms for the selection of experts. Firstly, the experts must meet two or more of the following criteria:

* University teaching in subjects related to the educational use of digital technologies.

* Experience in the field of training and improvement of teachers in ICT.

* Publications about educational technology literacy, digital competencies or audiovisual literacy in Spanish or international journals (in the last five years).

* Member of a prestigious research group in the field of educational technology.

* Participation in Spanish and international conferences on technologies applied to education.

One of the issues associated with expert judgements is the number of experts required for their application. The proposals are in the ranges of 15-20 (Malla & Zabala, 1978), 1535 (Landeta, 2002) o 15-25 (Witkin & Altschuld, 1995).

As was already pointed out, their number is determined by different aspects: having different experts related to ICT, avoiding the reduction in the number of experts when several rounds are considered, the volume of work to analyse, the ease of access to information and the urgency at which the data is required (Cabero-Almenara & Barroso-Osuna, 2013). In this case, since problems emerge when working with a large database and only one round could be conducted, it was decided to work with the largest number of experts possible.

Considering these criteria, a total of 412 emails were sent. Finally, after the two weeks in which the questionnaire was open, 155 replies were received.

The interest to improve the identification of the final judges led to the application of the ECC (Cabero-Almenara & Barroso-Osuna, 2013; Lopez-Gomez, 2018; Martinez, Travieso, Sagaro, Urbina, & Martinez, 2018). This index is obtained from the self-perception of the expert about his/her level of knowledge regarding the analysed topic, as well as about the sources that allow him/her to argue his/her decision.

To obtain it, the following formula is used: K = [1/2] (Kc + Ka), where Kc is the "knowledge coefficient". It is obtained from the score given directly by the expert in the question:

A) Tick the box that corresponds to your degree of knowledge about the following topics: teacher training in ICT, digital competencies, digital literacy... Value yourself on a scale of 0 to 10 (considering 0 as having absolutely no knowledge and 10 as having full knowledge about the topic in the question).

Ka, the argumentation coefficient, is obtained by adding up the options specified by the expert in the table of the question:

B) Self-value the degree of influence (low, medium or high) that each of the following sources had on your knowledge and criteria about teacher training in ICT, digital competencies, digital literacy.. 1. Theoretical analyses performed by you. 2. Your experience obtained from practical activity. 3. Research on the topic, from Spanish authors. 4. Research on the topic, from international authors. 5. Your own knowledge about the state of the problem outside of the Spanish context. 6. Your intuition about the approached topic.

If the values used to determine the position of the expert suggest that the K score is between 0.8 and 1, his/her competence level is high; if K is between 0.5 and 0.8, his/her competence level is moderate or medium; lastly, if K is below 0.5, the competence level is low.

Once the ECC of all subjects was analysed, it was decided to work with those who obtained a high competence level (ECC [greater than or equal to] 0.8). Thus, the sample was composed of 119 experts: 76.66% of the people who completed the questionnaire.

It is important to mention that 73.88% (f=88) of the judges had a PhD and 17.6% (f=21) had a master's degree. Most of them worked in a university centre (f=86, 72.3%) and their professional activity was basically teaching (f=110, 9.4%), followed by administration (f=35, 29.4%).

Table 1, presents other characteristics of these experts.

As can be observed, most of the identified experts claimed to have teaching experience, publications and research on topics related to ICT and the digital competence and literacy of teachers.

The information gathering instrument contains two main sections. The first section gathers information about some characteristics of the expert (title, professional activity, workplace...) and incorporates the questions aimed at determining the ECC. The second section requests the valuation of the different frameworks of digital competencies for teachers that were previously described:

* European Framework of Digital Competence for Teachers (M1).

* Framework of the "International Society for Technology in Education" (ISTE) for teachers (M2).

* UNESCO ICT Competence Framework for Teachers (ICT-CFT) (M3).

* Common Spanish Framework of Digital Competence for Teachers of the "Spanish Institute of Educational Technology and Teacher Training" (INTEF) (M4).

* British Framework of Digital Teaching (M5).

* ICT Competencies for Teachers' Professional Development of Colombian Ministry of Education (M6).

* ICT Competencies and Standards for the Teaching profession ofthe Chilean Ministry of Education (M7).

This instrument is administered digitally and can be found in: Its application was conducted in January 2020.


Firstly, the values obtained by the experts in the knowledge (Kc) and argumentation (Ka) coefficients are presented. Table 2 shows the means of such scores; Kc was obtained from the mean of the scores given by the experts, and Ka was obtained by assigning 1 point to the "low" option, 2 to the "medium" option and 3 to the "high" option.

As can be observed, the self-valuations assigned by the experts could be considered high and very significant. This suggests that the initial criteria used to identify them are valid and significant for the pursued objective.

Next, the results of the values assigned for the different frameworks by those experts with ECC [greater than or equal to] 0.8 are presented, including the means obtained both in the total and in each of the standards of each framework. The aim was to select the framework with the highest mean score (Table 3).

After arranging the mean score obtained by each framework in descending order, the following ranking was obtained:

* 5.63 European Framework of Digital Competence for Teachers.

* 5.49 Common Spanish Framework of Digital Competence for Teachers of the "Spanish Institute of Educational Technology and Teacher Training".

* 5.45 UNESCO ICT Competence Framework for Teachers (ICT-CFT).

* 5.36 ICT Competencies for Teachers' Professional Development of Colombian Ministry of Education.

* 5.33 British Framework of Digital Teaching.

* 5.30 Framework of the "International Society for Technology in Education" (ISTE) for teachers.

* 5.30 ICT Competencies and Standards for the Teaching profession of the Chilean Ministry of Education.

With the aim of determining the validity of the ECC and verifying the existence of significant differences between the answers given by the experts with an ECC below 0.8 and the answers of those with an ECC equal to or higher than 0.8, the Mann-Whitney U-test was used (Siegel, 1976) as well as the calculation of Cohen's d (Cohen, 1988) to analyse the effect size.

Table 4 presents the mean scores and standard deviations reached by the experts based on having a score below or equal to/higher than 0.8.

To analyse the H0 (null hypothesis), which refers to the non-existence of differences between the scores assigned by the experts based on whether or not they had an ECC [greater than or equal to] 0.8, at a significance level of 0.05, the Mann-Whitney U-test was applied. Table 5 presents the average ranges and the Mann-Whitney U-test. Table 6 shows the U values obtained for the fulfillment or rejection of the H0.

Table 6 shows the values obtained in the Mann-Whitney U-test

The obtained values allow rejecting all the formulated H0 at a significance level of 0.05 or lower. Therefore, it can be concluded that the valuations of the different experts who participated in this study are different based on the score of 0.8 obtained in the ECC.

Such differences, when applying Cohen's d (Table 7), to analyse the effect size and according to the proposition of Cohen (1988), can be considered as high (M1, M3 and M4) and moderate (M2, M3, M4, M5 and M7).

After conducting this analysis, the next step was to verify the existence of significant differences between the different scores given by the judges to the different frameworks. To this end, two statistics were applied. Firstly, Friedman's test, which is a non-parametric alternative to the analysis of variance (Siegel, 1976), with the aim of identifying which of the different frameworks proposed had obtained the highest range. Then, the Wilcoxon signed-rank test for related samples was used (Siegel, 1976). Lastly, Cohen's d was applied to determine the effect size.

Table 8 presents the average range values, the most relevant framework and the arrangement of all the frameworks, as well as the significance of the obtained results.

The obtained results indicate that there are significant differences (p [less than or equal to] 0.000) at least between the first range obtained (M1) and the last (M7), that is, between the highest range obtained and the lowest.

Then, the Wilcoxon signed-rank test was used to analyse the differences between the highest range and the rest (Table 9).

The obtained valued reject the H0. Consequently, this suggests that there are significant differences between the highest range (M1) and the rest of the ranges at a significance level of p [less than or equal to] 0.05.

Lastly, to determine the size of such differences, Cohen's d was applied, obtaining the values presented in Table 10.

These values indicate that the obtained differences are low or moderate.


The conclusions of this study go in different directions. The first one refers to the significance of the ECC as a strategy to select experts. As was demonstrated, the scores given by the experts are very different, which supports the idea of conducting a preliminary screening, as in the present study.

After the round performed by the judges, an outstanding DCT framework was obtained, i.e., DigCompEdu, followed by INTEF. This is logical considering that the latter is inspired by the former. Such findings will be used in the research project DIPROMOOC to approach the teacher training plan from the perspective selected by the experts, as well as to provide our institutions with the proper guidelines to establish teacher training plans in DCT.

In any case, it is important to avoid missinterpreting the results and inferring that the other frameworks are not significant to acquire the DCT; on the contrary, the results are based on the preferences given by the judges, since there are considerable similarities between the different frameworks (Padilla-Hernandez et al., 2019). In our study, although the reasons are unknown, such differences can be due to the fact that the instrument was answered by Spanish judges.

The above-mentioned leads to suggest different research lines, such as replicating the present study for two or three rounds in which frameworks are progressively removed from the possible options. This would require a smaller number of experts and their previous commitment to participate in the investigation for a longer period of time. Moreover, it would also be necessary to request their nationality, in order to correlate it to the chosen framework, and to analyse whether this is a determining factor.


Funded by: Spanish Ministry of Science, Innovation and Universities, Spain

Funder Identifier:

Award: RTI2018-097214-B-C31

The present study is part of a larger research project entitled "Design, production and evaluation of t-MOOC for the teachers' acquisition of teaching digital competencies" (RTI2018-097214-B-C31), funded by the Spanish Ministry of Science, Innovation and Universities.


Butcher, N. (2019). Marco de competencias docentes en materia de TIC UNESCO. Paris: UNESCO.

Cabero, J., & Llorente, M. C. (2013). La aplicacion del juicio de experto como tecnica de evaluation de las tecnologias de la information (TIC). Eduweb. Revista de Tecnologia de Information Comunicacion en Education, 7(2), 11-22. Retrieved from

Cabero-Almenara, J., & Barroso-Osuna, J. (2013). La utilization del juicio de experto para la evaluation de TIC: el Coeficiente de competencia experta. Bordon. Revista de Pedagogia, 65(2), 25-38.

Cabero-Almenara, J., & Gimeno, A. M. (2019). Las TIC y la formation inicial de los docentes. Modelos y competencias digitales. Profesorado, Revista de Curriculum y Formaci[partial derivative]n del Profesorado, 23, 247-268.

Cabero-Almenara, J., & Palacios-Rodriguez, A. (2019). Marco Europeo de Competencia Digital Docente DigCompEdu. Traduction y adaptation del cuestionario DigCompEdu Check-In. EDMETIC, 9(1), 213-234.

Carretero, S., Vuorikari, R., & Punie, Y. (2017). The Digital Competence Framework for Citizens With eight proficiency levels and examples of use. Luxembourg: Publication Office of the European Union.

Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates, Publishers.

Corsi, D., Dominguez, F., & Rodriguez, M. (2019). Adquisicion de competencias emocionales mediante el desarrollo y uso de Serious Games en Education Superior. Pixel-Bit. Revista de Medios y Educacion, 56, 95-112.

Council of the European Union. (2018). Recomendacion del Consejo, de 22 de mayo de 2018, relativa a las competencias clave para el aprendizaje permanente. Brussels: Official Journal of the European Union.

Crompton, H. (2017). ISTE Standards for Educators: A Guide for Teachers and Other Professionals. Eugene: International Society for Technology in Education.

Cruz-Diaz, R., Ordonez-Sierra, R., Garcia, S. R., & Rabasco, F. P. (2016). Buenas practicas que desarrollan la competencia mediatica en entornos socioeducativos. Pixel-Bit. Revista de Medios y Educaci[partial derivative]n, 48, 97-113.

Education and Training Foundation. (2019). Digital Teaching Professional Framework. Taking Learning to the Next Level. London: JISC.

Elliot, J., Gorichon, S., Irigoin, M., & Maurizi, M. (2011). Competencias y Estdndares TIC para la Profesion Docente. Santiago: Ministerio de Educacion.

Fernanda, M., Saavedra, C., Pilar, P. D., Barrios, M., & Zea, C. (2013). Competencias TIC para el desarrollo Profesional Docente. Bogota: Ministerio de Educacion Nacional.

Galicia, L., Balderrama, J., &Edel, R. (2017). Validez de contenido por juicio de expertos: propuesta de una herramienta virtual. Apertura, 9(2), 42-53.

Ghomi, M., & Redecker, C. (2018). Digital Competence of Educators (DigCompEdu): Development and Evaluation of a Self-Assessment Instrument for Teachers Digital Competence. Berlin: Joint Research Center.

Gutierrez-Castillo, J., Cabero-Almenara, J., & Estrada-Vidal, L. I. (2017). Diseno y validation de un instrumento de evaluacion de la competencia digital del estudiante universitario. Espacios, 38(10), 1-27. Retrieved from

Hernandez-Suarez, C. A. (2016). Competencias TIC para el desarrollo profesional docente en educacion basica. Praxis & Saber, 7(14), 41-41.

Instituto Cervantes. (2002). Marco Comun Europeo de Referencia para las Lenguas: aprendizaje, ensenanza, evaluacion. Madrid: Instituto Cervantes.

Instituto Nacional de Tecnologias Educativas y Formacion del Profesorado. INTEF. (2017a). Marco Comun de Competencia Digital Docente. Enero 2017. Madrid: Instituto Nacional de Tecnologias Educativas y Formation del Profesorado.

Instituto Nacional de Tecnologias Educativas y Formation del Profesorado. INTEF. (2017b). Marco Comun de Competencia Digital Docente. Octubre 2017. Madrid: Instituto Nacional de Tecnologias Educativas y Formacion del Profesorado.

ISTE. (2018). Crosswalk: Future Ready Librarians Framework and ISTE Standards for Educators. Washington DC: International Association for Technology in Education.

Kullaslahti, J., Ruhalahti, S., & Brauerm, S. (2019). Professional Development of Digital Competences: Standardised Frameworks Supporting Evolving Digital Badging Practices. Journal of Siberian Federal University. Humanities & Social Sciences, 12(2), 175-186.

Landeta, J. (2002). El metodo Delphi: una tecnica de prevision del futuro. Barcelona: Ariel.

Lazaro-Cantabrana, J. L., Usart-Rodriguez, M., & Gisbert-Cervera, M. (2019). Assessing Teacher Digital Competence: the Construction of an Instrument for Measuring the Knowledge of Pre-Service Teachers. Journal of New Approaches in Educational Research, 8(1), 73-78.

Lopez-Gomez, E. (2018). El metodo Delphi en la investigation actual en education: una revision teoricaymetodologica. Education XX1, 1, 17-40.

Malla, F., &Zabala, I. (1978). La prevision del futuro en la empresa (III): el metodo Delphi. Estudios Empresariales, 39, 13-24.

Martinez, E., Travieso, N., Sagaro, N., Urbina, O., & Martinez, I. (2018). Identification de las competencias especificas de los profesionales de enfermeria en la atencion al neonato en estado grave. Medisan, 22(2), 184-192. Retrieved from

Mengual-Andres, S., Roig-Vila, R., & Mira, J. B. (2016). Delphi study for the design and validation of a questionnaire about digital competences in higher education. International Journal of Educational Technology in Higher Education, 13(1), 12-12.

Mishra, P., Koehler, M. J., & Bragg, W. H. (2006). Technological Pedagogical Content Knowledge: A Framework for Teacher Knowledge The advent of digital technology has dramatically changed routines and practices in most arenas of human work. Advocates of technology in education often envisage similar dramatic changes in the process of teaching. Teachers College Record, 108(6), 1017-1054.

Munoz-Repiso, A. G.-V., Martin, S. C., & Gomez-Pablos, V. M. B. (2020). Validation of an Indicator Model (INCODIES) for Assessing Student Digital Competence in Basic Education. Journal of New Approaches in Educational Research, 9(1), 110-125.

Olivares, K., Angulo, J., Prieto, M., & Torres, C. (2018). EDUCATIC: implementacion de una estrategia tecnoeducativa para la formation de la competencia digital universitaria. Pixel-Bit: Revista de Medios y Education, 53, 27-40.

Padilla-Hernandez, A. L., Gamiz-Sanchez, V. M., & Romero-Lopez, M. A. (2019). Niveles de desarrollo de la Competencia Digital Docente: una mirada a marcos recientes del ambito internacional. Innoeduca. International Journal of Technology and Educational Innovation, 5(2), 140-140.

Palacios-Rodriguez, A. (2020). Novas Literacias na Educacao Primaria: O Caso dos Videojogos Comerciais em Espanha. Revista Panoramica, 29(137), 2238-9210.

Perez-Escoda, A., Garcia-Ruiz, R., & Aguaded, I. (2019). Dimensions of digital literacy based on five models of development / Dimensiones de la alfabetizacion digital a partir de cinco modelos de desarrollo (Vol. 31). Informa UK Limited. Retrieved from 10.1080/11356405.2019.1603274

Prendes, M. P., & Gutierrez, I. (2013). Competencias tecnologicas del profesorado en las universidades espanolas. Revista de Educacion, 361,196-222.

Redecker, C., & Punie, Y. (2017). Digital Competence of Educators DigCompEdu. Luxembourg: Publications Office of the European Union.

Rios, J., Gomez, E., & Rojas, M. (2018). Valoracion de competencias TIC del profesorado universitario: un caso en Chile. Pixel-Bit. Revista de Medios y Educacion, 52, 55-65.

Robles, P., & Rojas, M. D. C. (2015). La validacion por juicio de expertos: dos investigaciones cualitativas en Linguistica aplicada. Revista Nebrija de Linguistica Aplicada, 18. Retrieved from

Rodriguez, M. D. M., Mendez, V. G., & Martin, A. M. R. M. R. (2018). Alfabetizacion Informacional Y Competencia Digital en Estudiantes de Magisterio. Profesorado, Revista de Curriculum y Formacion Del Profesorado, 22(3), 253-270. Retrieved from

Romero-Martin, R., Castejon-Oliva, F.-J., Lopez-Pastor, V.-M., & Fraile-Aranda, A. (2017). Formative assessment, communication skills and ICT in Initial teacher education. Comunicar, 25, 73-82.

Siegel, S. (1976). Estadistica no parametrica aplicada a las ciencias de la conducta. Mexico: Trillas. Silva, J., Morales, M. J., Lazaro, J. L., & Gisbert, M. (2019). La competencia digital docente en formation inicial: Estudio a partir de los casos de Chile y Uruguay. Archivos Analiticos de Politicas Educativas, 27(93), 27-27. Retrieved from

Suarez-Rodriguez, J., Almerich, G., Orellana, N., & Diaz-Garcia, I. (2018). A basic model of integration of ICT by teachers: competence and use. Educational Technology Research and Development, 66, 1165-1187. Retrieved from

Vuorikari, R., Punie, Y., Carretero, S., & Van-Den-Brande, G. (2016). DigComp 2.0: The Digital Competence Framework for Citizens. Luxembourg: Publication Office of the European Union.

Williamson, B., Potter, J., & Eynon, R. (2019). New research problems and agendas in learning, media and technology: the editors' wishlist. Learning, Media and Technology, 44(2), 87-91.

Witkin, B. R., & Altschuld, J. W. (1995). Planning and conducting needs assessment: A practical guide. Thousand Oaks: Sage.

Julio Cabero-Almenara (iD), Rosalia Romero-Tena (iD) and Antonio Palacios-Rodriguez (iD)

Departmento Didactica y Organization Educativa, Universidad de Sevilla, Spain

Corresponding Author Antonio Palacios-Rodriguez,

Received 2020-04-06

Revised 2020-04-06

Accepted 2020-04-14

Published 2020-07-15

Pages: 275-293

Funding: Spanish Ministry of Science, Innovation and Universities (Award: RTI2018-097214-B-C31)

How to cite this article (APA): Cabero-Almenara, J., Romero-Tena, R., & Palacios-Rodriguez, A. (2020). Evaluation of Teacher

Digital Competence Frameworks Through Expert Judgement: the Use of the Expert Competence Coefficient. Journal of New

Approaches in Educational Research, 9(2), 275-293. doi: 10.7821/naer.2020.7.578

Table 1 Table1. Some of thecharacteristics of the judges selected from
ECC [greater than or equal to] 0.8.

Dimension                                                    f   %

Throughout your professional                          Yes   113  95
life, have you taught
subjects / contents / subjects / training
actions related to the field
of Educational Technology?
                                                      No      6   0.05
Have you conducted or                                 Yes   112  94.1
participated in any research
related to aspects of virtual
training, distance learning, ICT
teacher training, digital skills
of teachers and students,
digital literacy...?
                                                      No      7   5.9
Have you carried out or                               Yes   104  87.4
participated in any
publication related to
aspects of virtual training,
distance learning, ICT
teacher training, digital
skills of teachers and
students, digital literacy...?
                                                      No     15   0.126
Throughout your professional                          Yes   114  95.8
life, have you taught
subjects / contents / subjects / talks / conferences
related to teacher training
in ICT, digital skills or
digital literacy?
                                                      No      5   4.2

Table 2 Means obtained in the knowledge (Kc) andargumentation (Ka)

Variable                                                    M     SD

Knowledge                                                   8.66  0.92
Theoretical analysis carried out by you.                    2.63  0.52
Your experience gained from your practical activity.        2.82  0.39
Working study on the subject of Spanish authors.            2.63  0.53
Working study on the subject of foreign authors.            2.54  0.58
Your own knowledge about the status of the problem abroad.  2.22  0.64
Your intuition on the topic addressed.                      2.38  0.55

Table 3 Means and standard deviations of the differentframeworks


Professional commitment                                   5.69  0.55
Digital resources                                         5.70  0.48
Digital pedagogy                                          5.67  0.57
Evaluation and feedback                                   5.52  0.72
Empower students                                          5.52  0.72
Facilitate students'                                      5.69  0.53
digital competence
TOTAL SCALE                                               5.63  0.39
ISTE STANDARDS FOR                                        M     SD
Apprentices                                               5.37  0.71
Leaders                                                   5.15  0.95
Citizens                                                  5.36  0.95
Collaborators                                             5.29  0.87
Designers                                                 5.33  0.76
Facilitators                                              5.51  0.70
Analysts                                                  5.11  0.97
TOTAL SCALE                                               5.30  0.63
ICT COMPETENTIAL                                          M     SD
Understanding ICT in                                      5.13  0.90
educational policy
Curriculum and evaluation                                 5.54  0.65
Pedagogy                                                  5.68  0.60
Digital skills application                                5.60  0.64
Organization and                                          5.19  0.89
Professional learning                                     5.57  0.72
TOTAL SCALE                                               5.45  0.54
COMMON FRAMEWORK FOR                                      M     SD
Information and                                           5.51  0.67
information literacy
Communication and                                         5.54  0.69
Digital content creation                                  5.40  0.87
Security                                                  5.48  0.80
Problem solving                                           5.50  0.79
TOTAL SCALE                                               5.49  0.57
DIGITAL EDUCATION                                         M     SD
Pedagogical planning                                      5.48  0.72
Pedagogical approach                                      5.34  0.96
Student employability                                     5.09  1.02
Specific teaching                                         5.02  1.10
Evaluation                                                5.34  0.86
Accessibility and                                         5.61  0.75
Self-development                                          5.41  0.86
TOTAL SCALE                                               5.33  0.68
ICT COMPETENCES FOR                                       M     SD
Technological                                             5.32  0.77
Communicative                                             5.32  0.77
Pedagogical                                               5.32  0.77
Management                                                5.32  0.77
Investigation                                             5.32  0.77
TOTAL SCALE                                               5.36  0.67
COMPETENCES AND ICT                                       M     SD
(CHILE) (F7)
Pedagogical                                               5.51  0.75
Technical or instrumental                                 5.16  0.85
Management                                                5.13  0.86
Social, ethical and                                       5.30  0.90
Development and                                           5.39  0.87
social responsibility
TOTAL SCALE                                               5.30  0.69

Table 4 Mean scores and standard deviations obtained bythe experts
based on the score equal to or higher than 0.8.

Marco         Expert [greater than or equal to] 0.8
       Yes         No
       M     SD    M     SD

M1     5.63  0.39  5.06  0.88
M2     5.3   0.63  4.9   0.77
M3     5.45  0.54  4.89  0.8
M4     5.49  0.57  4.9   0.73
M5     5.33  0.68  5.01  0.82
M6     5.36  0.67  4.92  0.81
M7     5.3   0.69  4.86  0.81

Table 5 Average ranges and Mann-Whitney U-test values

Framework  Expert [greater than or equal to]  Average  Sum of ranges
           0.8                                range

M1         Yes                                82.93     9868.5
           No                                 53.32     1759.5
M2         Yes                                81.8      9734
           No                                 57.39     1894
M3         Yes                                83.97     9993
           No                                 49.55     1635
M4         Yes                                84.54    10060.5
           No                                 47.5      1567.5
M5         Yes                                80.43     9571
           No                                 62.33     2057
M6         Yes                                82.15     9775.5
           No                                 56.14     1852.5
M7         Yes                                81.91     9747.5
           No                                 56.98     1880.5

Table 6 Mann-WhitneyU-testvalues

Framework  U de          W de Wilcoxon  Z       Sig.

M1         1198.5        1759.5         -3.488  .000 (**)
M2         1333          1894           -2.831  .005 (**)
M3         1074          1635           -4.008  .000 (**)
M4         1006.5        1567.5         -4.337  .000 (**)
M5         1496          2057           -2.103  .035 (*)
M6         1291.5        1852.5         -3.037  .002 (**)
M7         1319.5        1880.5         -2.906  .004 (**)

Note: (*)= significant at p [less than or equal to] 0.05, and (**) =
significant at p [less than or equal to] 0.01

Table 7 Cohen's values

Cohen statistic

M1  1.057
M2  0.599
M3  0.918
M4  0.963
M5  0.446
M6  0.622
M7  0.609

Note: *= significant at p [less than or equal to] 0.05, and ** =
significant at p [less than or equal to] 0.01

Table 8 Friedman's test

Average range        Friedman's test

M1             4.93  N            119
M2             3.46  Chi squared   50.624
M3             4.14  gl             6
M4             4.32  Sig.           0
M5             3.75
M6             3.91
M7             3.49

Table 9 Wilcoxon signed-rank test

Null hypothesis (H0) tested  Sig

M1-M2                        .000
M1-M3                        .000
M1-M4                        .000
M1-M5                        .000
M1-M6                        .000
M1-M7                        .000

Table 10 Cohen's d values

Cohen statistic

M1-M2  0.63
M1-M3  0.38
M1-M4  0.30
M1-M5  0.55
M1-M6  0.49
M1-M7  0.60
COPYRIGHT 2020 Universidad de Alicante
No portion of this article can be reproduced without the express written permission from the copyright holder.
Copyright 2020 Gale, Cengage Learning. All rights reserved.

Article Details
Printer friendly Cite/link Email Feedback
Title Annotation:ORIGINAL
Author:Cabero-Almenara, Julio; Romero-Tena, Rosalia; Palacios-Rodriguez, Antonio
Publication:NAER - Journal of New Approaches in Educational Research
Geographic Code:4EUSP
Date:Jul 1, 2020
Previous Article:Bibliometric and Collaborative Network Analysis on Active Methodologies in Education.
Next Article:Digital Competence: A Study from the Perspective of Pre-service Teachers in Turkey.

Terms of use | Privacy policy | Copyright © 2022 Farlex, Inc. | Feedback | For webmasters |