Printer Friendly

Cartoevaluation method for assessment of GIS software.


The comparing GIS software is not easy task. GIS software differs very often in native formats. It causes different processing of spatial data and set of GIS functions. The structure of data is often under the influence of the cartographic requests (Dobesova 2012). The amount of cartographic functionality is also at different levels.

The reason for cartography evaluation is to help future users of programs to select the most suitable and appropriate program focusing on cartographic outputs from GIS programs. Therefore, the user does not need to conduct its own survey and random testing of programs in order to determine an appropriate program. Evaluation is an early step in the decision making process. Insufficiencies and errors in software bring more manual work during map creation and at finishing of digital output map.

The quotation by Galileo Galilei is "What is not measurable make measurable". The CartoEvaluation method tries measure GIS software from the point of cartography.

1. Methodology for evaluation

The proposed CartoEvaluation method is part of the methodology, which consists of four steps:

1. step: Learning to work with selected GIS program by studying the documentation and practical activities in creating maps.

2. step: Creation of a series of test maps.

3. step: Application of CartoEvaluation method.

4. step: Evaluation of results obtained using CartoEvaluation method.

Expert who evaluates GIS software must be familiar with the software. The practical experiences with the creation of maps are necessary in that software. Important is the utilization of various cartographic methods (choropleth method, chart maps), labelling, creation of legend etc. The theoretical cartographical knowledge is precondition for evaluation (Vozenilek, Kanok et al. 2011; Brewer 2005, 2008).

2. The CartoEvaluation method

The new CartoEvaluation method is applied in the third step of the methodology. CartoEvaluation method is based on multi criteria evaluation and on Goal-Question-Metric method.

2.1. Multi criteria evaluation

The first decision was to choose the evaluation method. It is possible to use only a simple verbal evaluation or yes/no evaluation or multi criterion evaluation or fuzzy evaluation (Dobesova 2009b). Fuzzy evaluation is deeply described by J. Talasova (2003). Multi criteria evaluation is used very often in multi criteria decision making. Various criteria are considered to choose the best variant from the set of potential variants. In that case, the variant is the best software from the point of cartography.

The result of the overall evaluation is based on the weighted sum of the criteria (1). A key step in a multi criteria evaluation is to determine the weights of criteria. Different groups of assessors prefer different criteria with different weights, so it can be a significant conflict of opinions. Determination of weights is very difficult, and therefore there are methods based on simple subjective information from users to finally construct estimation of the weights.

All the different methods for estimating weights are based on the principle that the sum of the weights over all criteria is one (2).

S = [n.summation over (i=1)] [w.sub.i] * [c.sub.i], (1)

[n.summation over (i=1)] [w.sub.i] = 1, (2)

where w is weight, c is criterion, S is final sum.

2.2. Weights and Saaty matrix

Individual weighting factors of metrics were designed by expert estimation which arose from debates of cartographers. The weights of goals and weight of sub-goals arose from expert estimation of international group of cartographers. Previous discussions were performed among them. Finally, the method of quantitative pair wise comparison of criteria (Saaty matrix) was used. The set of weights is objective cartographer's consensus.

Method of quantitative pair wise comparisons (Saaty matrix) is based on a subjective evaluation which expresses preference between criteria. It is done by pair wise comparisons. There are no direct input weights and the weights are not set directly. The method is suitable in case when evaluators or experts can not afford direct determination of weights. Ratios are express the importance for every pair of criteria. The pair wise comparison of the use-weighing scales are set to 1 (equal criteria i and j), 3 (i criterion weakly prefers j criterion), ..., 9 (i criterion absolute prefers j criterion), and a reciprocal value. Ratings are stored in the Saatyho matrix S = ([s.sub.ij]) according to the following schedule:


The calculation of weights is a normalize geometrical average of rows in Saaty matrix (Brozova et al. 2007).

Cartographer pair wise compares the importance of the setting of coordinate system of software with the library of point, line and area symbols or with the possibilities to create choropleth map or an automatic function of generation of legend.

One form of multi criteria evaluation is goal-question metric method. The metric means the suggestion of weights for each criterion in that method.

2.3. Goal-Question-Metric method

The Goal-Question-Metric method (Basili et al. 1994) was chosen as the theoretical basis for the evaluation method. Goal-Question-Metric method (GQM) is designed by Victor Basili at University of Maryland, College Park and in Software Engineering Laboratory in Goddard Space Flight Centre NASA. This is a system of questions and simple answers for evaluation of properties.

Goal--Question--Metric method was developed for multi-purpose evaluation of software. Principle of the method consists of identifying the main goals that the program should perform (Dumke 2009). In this case, these goals reflect key tasks to be performed during the creation of map output. The main goals, which are the form the highest levels, are distributed to subgoals (the second level) and to partial goals (the third level). This method was also for evaluation open source GIS software under CASCADOSS project (Orlitova, Vobora 2008).

Division of main goals to the sub-goals and partial correspond to the Analytic Hierarchy Process (AHP). The AHP first decomposes the problem into a hierarchy of sub problems. Then the decision-maker evaluates the relative importance of its various elements by pairwise comparisons. The AHP converts these evaluations to numerical values (weights or priorities), which are used to calculate a score for each alternative.

GQM method defines a measurement model on three levels:

--Conceptual level (goal)--a goal is defined for an object for a variety of reasons, with respect to various models of quality, from various points of view and relative to a particular environment.

--Operational level (question)--a set of questions is used to define models of the object of study and then focuses on that object to characterize the assessment or achievement of a specific goal.

--Quantitative level (metric)--a set of metrics, based on the models, is associated with every question in order to answer it in a measurable way.

2.4. Goals of CartoEvaluation method

According to the GQM method the main goals were suggested. The main goals of CartoEvaluation methods are:

--A--Map construction;

--B--Map symbols;

--C--Cartographic methods--types of cartographic expression method;

--D--Map syntax--stratigraphical, componential and compositional.

Sets of questions and answers to these questions determine the fulfilment of partial goals. Resulting numerical rating is calculated using the metrics.

Maximum total sum is 100%. The contribution of main goals is in Table 1.

Contribution can be converted to multi criteria equation (4):

Score = 15 * A + 35 * B + 35 * C + 15 * D, (4)

where A, B, C, D are main goals.

All the main goals are divided to sub-goals. The number differs in each goal. The list of sub-goals is in Table 2. The sub-goals represent second level of goals. Each sub-goal consists of partial goals at the third level.

E.g. evaluation of library of point symbol belongs to the goal B--Maps symbol, sub-goal "2. Point symbols". The goal "B - 1. Basic methods" evaluates cartographic mapping methods as choropleth, areals, iso-lines methods.

2.5. Operational level of CartoEvaluation method

The set of question and possible answers is arranged in Microsoft Excel table (Figs 2, 3). A numeric score is obtained by answering questions concerning the accomplishment of individual cartographic goals. The cells contain predefined formulas for automatic calculation.

2.6. Implementation of CartoEvaluation method

The evaluation method is implemented in Microsoft Excel 2003 file because of a simple recording of answers and automatic calculation of the total numerical score.

This Excel file contains five sheets. The four sheets are for four main goals. The names of sheets correspond to the name of goals (A--Map construction, B--Map symbols, ...). The fifths sheet contains the table with the partial sum and calculates final score from the previous sheets. For automatic calculation of column D, all answers are set to the default maximum value (1 or 2). User can change the answer to 0 in the case of a negative response when the functionality is missing.

Any user can very easy perform an evaluation of any GIS program according to the CartoEvaluation method. CartoEvaluation method is freely available on the information project website ( (Fig. 4) as a downloadable Microsoft Excel file. It is possible to download Czech or English version.

Application of this methodology can be time-consuming as well as knowledge-demanding task. Second possible use is to explore the results of evaluations made by others earlier reviewers. The evaluations of some software are also downloadable. Therefore, it is not always necessary to perform new testing of the program and following evaluation according CartoEvaluation method.

3. Results of assessment

Method CartoEvaluation was practically used for assessment. Therefore, it is applied methodology. Several experienced users and producers of programs from the Czech and Slovak Republic were asked to work out evaluations. Evaluation of the same GIS program was usually carried out by multiple evaluators to eliminate minor inaccuracies in answers. In total, 13 GIS programs were evaluated, both commercial and Open Source Software, including free viewers. The evaluated programs and results are shown in Table 3.

The results of the evaluation confirm that most of the programs achieved good basic cartographic functions. Nine programs achieved more than 50% from the maximum possible score (100%). Tested programs were: ArcGIS, Maplnfo, Geomedia, GRASS, TopoL, AutoCAD Map, Kristyna GIS, MISYS and OCAD. Commercial programs are among the best because they have been developed for a long time, and thus have the chance to meet the requirements of expert cartographic outputs (Dobesova 2009a, b).

Programs GRASS, Kristyna GIS and MISYS also reached good ranking. When program reaches a score between 40% and 50%, it is appropriate to use the GIS program only for simple cartographic outputs. Such a program can be expected to miss some cartographic functions. Manual editing by graphic and text tools is likely to be necessary in finalisation of the map.

Evaluation of programs also revealed some weak or missing cartographic functions. Whereas the possibilities of setting a projections and reference coordinate system, including the choice of national coordinate systems are without problems, other functions were revealed to insufficient. The possibility of transformation of mapping face is missing in some programs. Programs often do not have the ability to generate metadata about the map construction and metadata about construction of symbols.

Some cartometric and geodetic tools are missing (cartographic distortion etc.). Included pattern books for point, line and areal symbols are numerous, but often one-sidedly oriented. Symbol editor is rarely available to let the user create and re-use new symbols, especially problematic are composite line symbols.

Functionality of setting colors is good. It is possible to interactively select the color from a palette in the different color models (RGB, HSV). Some color schemes (ramps) are, however, missing, in particular bipolar, gradation or hypsometric color schemes. Possibility to create, save and reuse custom color schemes is very rare. The advantage of the method is that is possible to compare software in sub-goal or partial goals. Example of comparing in partial goal "Color ramps libraries" is in Table 4.

The line vector method (line motion) and its variants are missing in the cartographic expression methods. Programs also have insufficiencies in creating point and area diagram map (chart diagrams). Multiparameters totalizing diagrams, comparative diagrams and dynamic diagrams are missing. Similar situation occurs with line diagrams (ribbon). Totalizing line diagram, compound structural line diagram and two-direction line diagram are missing. Diagram scales in the legend are incomplete. This insufficiency solves external custom extension "Diagram map creator" for ArcGIS (Dobesova, Valent 2011).

Cartograms methods (anamorphosis) are very seldom implemented. Only program MapInfo has cartograms method and an extension for ArcGIS exists. Programs have the lack of options in the definition of the functional scale for determining the parameters of symbols. On the contrary, options of classification (setting of interval scales of values) are well-implemented. Some insufficiencies are also in the automatic creation of a legend. It is not possible to create a hierarchy in a legend or divide legend to segments. Some symbols in map do not correspond to the symbols in the legend.

It is not possible to create animated symbols and animated maps (except for program ArcGIS). The only other frequent lack of programs in the map syntax is a possibility to create templates for repeated map composition, which is often used in compiling a series of maps with an identical composition. For detailed finding of implemented and the missing functions it is possible to look directly into the resulting evaluation tables at web pages (CartoEvaluation). Additional notes and comments in tables from evaluators are also interesting and helpful.

4. Discussions

The proposed methodology and CartoEvaluation method was designed for desktop GIS programs. The method could be modified and with certain changes and additions applied to applications that are used to create web map applications. Further utilization of method could also be used for evaluation of programs for mobile computing devices designed for field work. The newest research is finishing modification of method to the CartoEvaluationM method for assessment software for mobile computers (ArcPAD, QGIS etc.). Programs for mobile devices are generally characterized by a small number of functions in comparison to desktop programs, including restrictions of cartographic functions (limited pattern book of symbols etc.). The modification of method reduces the number of question.

There is also expectation to support the cartographic design by intelligent systems. These systems follow cartographic rules to avoid mistakes and errors by users (Brus et al. 2009, 2010; Brus, Dobesova 2011).

The utilisation of CartoEvaluation method is not limited. It is possible to test the new version of the software. The method predicts the evolution of cartographic function and the tables contain questions about new cartographic methods: animated maps with animated symbols, visualization of 3D surfaces, visualization of spatio-temporal data (Popelka, Vozenilek 2012) etc.


In conclusion, it is impossible to create the best simple single evaluation method. A simple evaluation method is not sufficient because software systems and cartography have many different properties. Depending on objectives, environment and processes, evaluations will vary from complex assessment method as CartoEvaluation. Just one single measurement is not sufficient to express software quality. Software quality has many aspects and needs different measurements, such as reliability, maintainability, error-rate, etc. There are also values that cannot be measure, such as user satisfaction.

The valuable effect of CartoEvaluation method is a wide list of aspect (goals) in the area cartographical function. The method concerns very deep and detailed cartographic functions and this complex evaluation has not been existed yet. The weight and final score is not the most important. The proposed CartoEvaluation method will help users to select from a wide variety of GIS programs and subsequently to use it in the maximum extent for the fast, comfortable and correct digital map output making.

doi: 10.3846/20296991.2013.859824

Caption: Fig. 1. The point symbol library in Quantum GIS

Caption: Fig. 2. Example of questions, answer and explanation in Excel sheet for goal B--map symbols, sub-goal 2. Point symbols

Caption: Fig. 3. Example of questions, answer and explanation in excel sheet for goal C--cartographic methods, sub-goal 2. Diagram map

Caption: Fig. 4. Internet web pages of CartoEvaluation method


The work was supported by International Visegrad Fund No. 20810129: "Evaluation of cartographic functionality in GIS software" and by the project CZ.1.07/2.3.00/20.0166.

Received 11 April 2013; accepted 09 December 2013


Basili, V.; Caldiera, G.; Rombach, H. D. 1994. Goal question metric paradigm, in Encyclopedia of software engineering. New York: John Wiley & Sons, 528-532. ISBN 0-471-54004-8.

Brewer, C. A. 2008. Designed maps. Redlands: ESRI Press. 184 p. ISBN 9781589481602.

Brewer, C. A. 2005. Designing better maps. Redlans: ESRI Press. 220 p. ISBN 9781589480896.

Brozova, H.; Subrt, T.; Houska, M. 2007. Modely pro fizeni znalosti a podporu rozhodovdni. Ceska zemedelska univerzita, Praha. 117 p. ISBN 978-80-213-1633-1(in Czech).

Brus, J.; Dobesova, Z.; Kanok, J.; Pechanec, V. 2010. Design of intelligent system in cartography, in 9th RoEduNet IEEE International Conference, June 24-26, 2010, University of Sibiu, Sibiu, Romania, 112-117. ISSN 2068-1038 (print), ISBN 978-1-4244-7335-9 (online).

Brus, J.; Dobesova, Z.; Kanok, J. 2009. Utilization of expert systems in thematic cartography, in International Conference on Intelligent Networking and Collaborative Systems, INCoS, November 04-06, 2009, Barcelona, Spain, 285-289. ISBN 978-076953858-7 (print).

Brus, J.; Dobesova, Z. 2011. Coping with cartographical ontology, in Conference Proceedings SGEM 2011, 11th International Multidisciplinary

Scientific GeoConfrence, June 20-25, 2011, Albena, Bulgaria, Sofia: STEF92 Technology Ltd., 377-384. ISSN 1314-2704.

Dobesova, Z. 2012. Database modelling in Cartography for the "Atlas of Election", Geodesy and Cartography 38(1): 20-26. Taylor & Francis Group. ISSN 2029-6991 (print), ISSN 2029-7009 (online).

Dobesova, Z.; Valent, T. 2011. Program extension for diagram maps, Geodesy and Cartography 37(1): 22-28. Taylor & Francis. ISSN 2029-6991 (print), ISSN 2029-7009 (online).

Dobesova, Z. 2009a. Evaluation of cartographic functionality in GIS software [online], [cited 15 March 2013]. Available from Internet:

Dobesova, Z. 2009b. Evaluation of cartographic functionality in geographic information systems. Olomouc: Publishing house of Palacky University. 131 p. ISBN 978-80-244-2353-1 (in Czech).

Dumke, R. 2009. GQM Method Application, Software Measure Laboratory SML@b [online], [citted 15 March 2013]. Available from Internet:

Orlitova, E.; Vobora, V. 2008. CASCADOSS--prehled Open Source geoinformacnich programu pro monitoring zivotniho prostfedi, in Proceedings GIS Ostrava 2008, 27-30 January, 2008, VSB--TU, Ostrava. ISSN 1213-2454.

Popelka, S.; Vozenilek, V. 2012. Specifying of requirements for spatio-temporal data in map by eye-tracking and space-time-cube, in Proceedings of International Conference on Graphic and Image Processing (ICGIP 2012), March 14, 2013, Singapore.

Talasova, J. 2003. Fuzzy methods, multi criteria evaluation and decision making. Olomouc: Publishing house of Palacky University. 179 p. ISBN 80-244-0614-4 (in Czech).

Vozenilek, V.; Kanok, et al. 2011. Methods of thematic cartography. Visualization of spatial data. Olomouc: Publishing house of Palacky University. 216 p. ISBN 978-80-244-27904 (in Czech).

Zdena Dobesova

Department of Geoinformatics, Faculty of Science, Palacky University,

17. listopadu 50, 771 46, Olomouc, Czech Republic


Zdena DOBESOVA. Ing, PhD Doctor. She holds a PhD (since 2007) degree from Technical University in Ostrava, Faculty of Mining and Geology. Research interest: GIS, spatial database, digital cartography, programming, scripting in Python for ArcGIS.

Table 1. The contribution of main goals

Goal                       Maximal score [%]

A--Map construction              15
B--Map symbols                   35
C--Cartographic methods          35
D--Map syntax                    15
Total score                       100

Table 2. The list of goals and sub-goals in CartoEvaluation

Main goal                             Sub-goal       Goal
Sub-goal                             weight [%]   weight [%]

A--Map construction                                   15

1. Base of the map                       3
2. Reference system of map face          4
3. Transformation of map outline         4
4. Metadata about map construction       2
5. Cartometry                            2

B--Map symbols                                        35

1. Pattern book of symbols              1,5
2. Point symbols                         8
3. Line symbols                          8
4. Area symbols                          8
5. Localization of map symbols           3
6. Color setting                         6
7. Metadata about symbols               0,5

C--Cartographic methods                               35

1. Basic methods                         8
2. Diagram methods                       8
3. Scale of values                       3
4. Label                                 10
5. Specific methods                      6

D--Map syntax                                         15

1. Intra-composition                    1,5
2. Map layers                            2
3. Extra-composition                     2
4. Basic elements: title, colophon      0,5
5. Legend                                5
6. Map scale                             1
7. Additional composition elements       2

Total score                             100          100

Table 3. Results of evaluated GIS programs

GIS software                A               B              C
                     Map construction   Map symbols   Cartographic
                          (15%)           (35%)       methods (35%)

JANITOR J/2                4,32            18,95          6,68
IDRISI Taiga               3,17            14,50          7,82
Kristyna GIS view.         2,16            17,70          12,31
Quantum GIS                9,4             18,95          6,95
Geomedia Viewer            9,32            25,45          3,88
uDig                      10,80            17,50          12,13
OCAD                       8,12            25,85          9,13
MISYS                     11,90            21,10          10,58
Kristyna GIS               7,32            19,55          16,53
AutoCAD Map 3D             9,30            20,35          16,21
TopoL xT DMT              10,80            17,65          16,00
GRASS                     10,60            17,05          21,03
Geomedia Professional     12,60            25,45          12,13
MapInfo Professional      10,10            25,80          22,59
ArcGIS--ArcMap            11,62            30,10          27,50

GIS software             D        Total score
                     Map syntax      100%

JANITOR J/2              1,60         31,75
IDRISI Taiga             7,48         33,50
Kristyna GIS view.       4,88         37,05
Quantum GIS              8,55         43,85
Geomedia Viewer          4,55         43,20
uDig                     7,7          48,13
OCAD                     7,88         50,98
MISYS                    7,47         51,05
Kristyna GIS             8,67         52,06
AutoCAD Map 3D          10,10         55,96
TopoL xT DMT            12,97         57,42
GRASS                   11,20         59,88
Geomedia Professional   11,73         63,58
MapInfo Professional    11,97         70,45
ArcGIS--ArcMap          13,03         82,25

Table 4. Comparison of results for the color ramps libraries

GIS Software                ArcGIS   AutoCAD   Geomedia   Geomedia V.
                                       Map      Prof.

Q1: Are there predefined      1         1         1           1
qualitative (categorical)
color schemes (various

Q2: Are there predefined      1         1         1           1
sequential color schemes?

Q3: Are there predefined      1         0         0           0
bipolar color schemes?

Q4: Are there predefined      1         1         0           0
color hypsometric ramps

Q5: Can you create your       1         0         1           1
own color schemes and
save them for reuse?

Total                        2,4       1,5       1,4         1,4

GIS Software                GRASS   IDRISI   JANITOR   Kristyna
                                    Taiga               GIS V.

Q1: Are there predefined      1       1         1         0
qualitative (categorical)
color schemes (various

Q2: Are there predefined      0       1         1         0
sequential color schemes?

Q3: Are there predefined      1       1         1         0
bipolar color schemes?

Q4: Are there predefined      1       1         1         0
color hypsometric ramps

Q5: Can you create your       1       1         0         0
own color schemes and
save them for reuse?

Total                        1,9     2,4       2.0        0

GIS Software                Kristyna   MapInfo   MISYS   OCAD
                              GIS       Prof.

Q1: Are there predefined       0          1        1      0
qualitative (categorical)
color schemes (various

Q2: Are there predefined       0          1        1      0
sequential color schemes?

Q3: Are there predefined       0          0        0      0
bipolar color schemes?

Q4: Are there predefined       0          0        0      0
color hypsometric ramps

Q5: Can you create your        0          1        1      0
own color schemes and
save them for reuse?

Total                          0         1,4      1,4     0

GIS Software                Quantum   TopoL   uDig
                              GIS      xT

Q1: Are there predefined       1        1      1
qualitative (categorical)
color schemes (various

Q2: Are there predefined       0        1      1
sequential color schemes?

Q3: Are there predefined       0        1      1
bipolar color schemes?

Q4: Are there predefined       0        1      0
color hypsometric ramps

Q5: Can you create your        1        1      1
own color schemes and
save them for reuse?

Total                         0,9      2,4    1,9


Please note: Illustration(s) are not available due to copyright restrictions.
COPYRIGHT 2013 Vilnius Gediminas Technical University
No portion of this article can be reproduced without the express written permission from the copyright holder.
Copyright 2013 Gale, Cengage Learning. All rights reserved.

Article Details
Printer friendly Cite/link Email Feedback
Author:Dobesova, Zdena
Publication:Geodesy and Cartography
Article Type:Report
Geographic Code:4EXCZ
Date:Dec 1, 2013
Previous Article:Photogrammetric mapping based on UAV imagery.
Next Article:A comparative analysis of different DEM interpolation methods.

Terms of use | Privacy policy | Copyright © 2022 Farlex, Inc. | Feedback | For webmasters |