Printer Friendly

The Civic Open Data and Crowdsourcing App Ecosystem: Actors, Materials, and Interventions.

INTRODUCTION

Opening up civic data has become part of the modern infrastructure of municipal governments. Opening data has several goals: To enhance transparency and accountability, increase government efficiency and service delivery, and promote economic development and business intelligence (Sieber and Johnson 2015). Most open data provisions, however, consist of little more than "throwing data over the wall" (ibid.). This may ensure open data data that can be "freely used, modified, and shared by anyone for any purpose" (Open Knowledge International 2012) begins to meet its goals, but can fail to describe all the actors involved in open data adoption, sustainability, and value. Regardless, government remains confident in the potential of open data apps to improve public services (Bates, 2014).

Concepts such as crowdsourcing (Brabham 2009), citizen science (Haklay 2013), and Volunteered Geographic Information (VGI) (Goodchild 2007) offer new sources of data for governments and introduce opportunities for citizen-side collaboration. Indeed, governments look to citizen contributions of data as a form of public engagement and a substitute for declining government resources.

Regardless of the source (government or citizen), civic data will require infomediaries. This has become the accepted norm in public use of open data, by which "entrepreneurial actors will create 'apps' that make data accessible to citizens" (Davies, Perini, and Alonso 2013, 14), which also can extend to government uses of data.

The term infomediary originally was defined by Hagel and Rayport (1997) as "custodians, agents, and brokers of customer information, marketing it to businesses on consumers' behalf." Infomediaries (sometimes referred to simply as intermediaries) now are generally understood as actors that help manage data and information between a data source and user (Janssen and Zuiderwijk 2014). They partake in converting data to information, such as creating databases that track activities of local politicians to increase citizen engagement with politics, or promoting data interoperability by combining local-level data as a form of community building (Worthy 2015). We use an Actor-Network Theory (ANT) definition of the actor, which is anything that "acts or to which activity is granted by others" (Latour 1996, 373). This allows us to use the "actor" to describe any human or non-human entity in the open data app ecosystem that exerts control over data. We define infomediaries as actors that transform or control data, but exist outside of government control. Not all actors are infomediaries; decision makers in government who exert their own control over data are not infomediaries.

Traditionally, government has outsourced much of its information technology (IT), some ofwhich is developed by traditional sources such as private-sector consultants and, more recently, created by emergent sources such as participants in hackathons (Chen and Gant 2001, Johnson and Robinson 2014). Infomediaries market their services with easy-to-use public interfaces and back-end analytics. The current Web 2.0 paradigm, defined by O'Reilly (2007) as a shift towards dynamic and user-generated web content, treats data and software as a dynamic service, making outsourcing of analytics and hosting to infomediaries more attractive. Government must trust that these infomediary services remain operational so that data continues to flow. Infomediaries may exert control in the form of data (re)formatting, aggregation, interface design, analytics, and mapping. Control includes access to data, as access can allow or limit the potential to manipulate data. Infomediaries may be human (e.g., software developer) or non-human, (e.g., data server). We are interested in identifying the significance of all actors because infomediaries presumably augment raw data with operational functionality and value.

In this article, we describe infomediaries found in five municipal applications (apps) that are built around civic data, using ANT to frame them within the app ecosystem. An ecosystem view is important as it does not restrict us to just individuals as a unit of analysis, but allows us to comment on the broader sociotechnical environment they inhabit. We focus on apps because civic data is likely to be offered in a mediated form, including crowd-sourcing sites and citizen dashboards (Sandoval-Almazan, Gil-Garcia, Luna-Reyes, Luna, and Rojas-Romero 2012). First, a review of government models of service provision afforded by Web 2.0 allows us to define the role of the infomediary. Then, we cover our methods of app selection and interviews with respondents with direct experience in appifying the raw civic data. We conclude with the increased significance of the infomediary as part of the government public service delivery process.

THE ROLE OF INFOMEDIARIES IN THE CIVIC DATA APP ECOSYSTEM

The rise of infomediaries is driven by two congruent forces, one from government and one from the information technology industry. For government, it is a matter of practicalities and paradigms. Governments engage external parties for numerous reasons, among them to draw upon global talent, leverage economies of scale, free up resources, enhance flexibility and responsiveness to new conditions, limit liability, and compensate for resource constraints (Grimshaw, Vincent, and Willmott 2002; Stroh and Treehuboff2003). Governance paradigms such as New Public Management (NPM) suggest a structural shift in the relationship between government and other sectors and civil society. Here, government becomes a manager of public service providers, where its functions and responsibilities are preferably delegated to external parties (Denhardt and Denhardt 2000, Hood 1995).

Historically, this outsourcing has predominantly been the domain of private-sector interests. Outsourcing has broadened to include citizen-sourcing (Nam 2012), where citizens-at-large actively contribute to coproduce public goods and services (Ostrom 1996). In the Government 2.0 paradigm, envisioned by O'Reilly (2011), the sector of the service provider does not matter so long as government operates as a platform upon which others can innovate. Cities, for example, should function solely to provide the resources, rules, and regulations to govern public services. External parties will do the rest.

O'Reilly (ibid.) views government data as the newest infrastructure to be outsourced. He and others (e.g., Klischewski 2010) argue that if cities simply open data then outside brokers will emerge to deliver services, whether filling potholes or constructing apps. This potential is enabled by Web. 2.0 technologies, which serve the long tail of users, provide a rich user experience, and enable software as a service (SaaS) of multiple interoperable components, and where citizen-generated information is as important or more important than official data (O'Reilly 2007).

Web 2.0 aids and complicates government functioning. Adaption to new innovations can be hindered by rules and regulations governing bureaucracy. Adoption of Web 2.0 tools, like harvesting of social media, results in higher data volumes, new skills and expertise, and higher labour and IT maintenance costs. Government employees may find themselves subject to information overload (Sivarajah, Irani, and Weerakkody 2015) simultaneously as new tools potentially disrupt organizational hierarchies (Bond 2015, Klischewski 2010). Relying on such Web 2.0 tools may prevent government from carrying out some of its basic required functions. For example, archiving communications is almost impossible when they sit on social media platforms outside of government control, designed to be ephemeral and subject to business models that may terminate a service at will (Bertot, Jaeger, and Hansen 2012). Government could see the potential "loss of ownership control and authenticity of the final products" of communications or data (Freeman and Loo 2009, 77) as they are unable to control outcomes.

With Web 2.0, government becomes dependent on a multiplicity of third parties, technologies, and standards to deliver services from data collection to data distribution. Web 2.0 shifts app development from large, customized, and single-sourced turnkey projects to modular platforms supported by many developers. Government dependence on proprietary standards and platforms has been a vulnerability (Evans and Reddy 2002); the shift to Web 2.0 and modular software promised freedom from singular platforms (Fishenden and Thompson 2013). However, a platform with many complex and interdependent software components, accompanied by high levels of expert labour, can expand reliance on external parties (Schneider and Sunyaev 2016). Yu (2014) describes this scenario as mutual dependence. The fewer the alternatives available to government, the higher its dependence on an infomediary. He notes that high mutual dependence can result in inflexibility and loss of control over data. Given sufficient trust, however, mutual dependence fosters mutual investment and can strengthen long-term public-private partnerships. The complications introduced by outsourcing suggests the notion of the "government as a platform" may itself be counterproductive to the aims of government.

METHODS

We tracked the flow of civic data (open data and citizen-sourced data) to identify the actors involved in the use of civic open data. Rather than choosing specific datasets or cities as units of analysis and examining the infomediaries of a government's open data portal, we chose to explore infomediaries by which open data is expressed to the public via an application. We selected apps that represented the best examples of app development at the municipal level in Canada. To identify the actors, some existing in government and some existing outside (i.e., infomediaries), we conducted qualitative interviews of app developers and government officials. Identification of apps and actors was assisted by examination of public information from government websites. Based on qualitative interview data and information from their respective government websites, we traced the data from source to user, and revealed actors along the way.

App Selection

Sampling of apps was based on a typology of citizen and market-oriented apps (Sangiambut and Sieber 2016). This typology helped us choose apps based on directionality of data flow. First, a scan of existing open data apps was performed in Canada, based on official app listings from Canada's major open data cities' websites: Edmonton, Montreal, Ottawa, Toronto, and Vancouver. The cities of Edmonton, Ottawa, Toronto, and Vancouver comprise the initial network of open data cities in Canada, known as the G4 (Carl 2012). As such, these cities were some of the leaders in Canadian open data production, had the longest sustained production of open data, and likely the best examples of local app development. Best cases were selected based on several criteria: Whether open data flowed to or from government (apps from both directions of data flow were chosen), popularity (number of users), and currency (how well the app was maintained). Apps that appeared to be "dead" (no patches in over six months) were not considered. Sampling was restricted to the local Canadian municipal level to remove the mixing of levels of government because municipal, regional, and federal governments can have differing operations and compositions as well as motivations for releasing datasets. Data flowing to government had to be used by government, otherwise there was no mediation. Apps outsourced by government were the best guarantee that government was involved in data production or collection. Ultimately, we chose five Canadian municipal-level open data apps: Citizen Dashboard, Ottawa Transit, Citizen Budget, Toronto Cycling App, and VanConnect.

Citizen Dashboard was chosen as it won a 2014 Public Sector Leadership Award for the (at the time) progressive nature of this app (Quigg 2014), and because it was a directly outsourced government app. Ottawa Transit was selected because it won an award at the Apps4Ottawa open data hackathon, had active development (bimonthly updates), and a download base of more than 20,000. Citizen Budget was chosen because it was the only public consultation app currently in use. Its developer, Open North, has demonstrated its use with clients in multiple cities across Canada and in the United States of America. Toronto Cycling App was chosen because it is an app outsourced to the private sector, its relatively high user base of more than 5,000, and its direct usage in city infrastructure planning. In particular, it was advertised as a tool being used to inform planners in their next cycling infrastructure plan, presenting a direct link to government. Finally, while VanConnect had strong developer support and a large install base of over 10,000 users, it also was included because it had both citizen reporting and push notification functionality, which exemplified hybrid or bidirectional data flow. This suggested the possibility of additional dependency relationships between government and developer to be observed.

Semi-structured Interviews

To ascertain the actors in the app ecosystem, semi-structured interviews were conducted with app developers and government officials in managerial or leadership positions. Semi-structured interviews could generate more nuanced responses on the dependencies between government and outsourced developers and any infomediaries they use (such as their own software platforms or third-party Application Programming Interfaces [APIs]). Semi-structured interviews also allowed us to ask probing questions on the reasoning behind decisions and their origins, which increased the likelihood of identifying additional actors that were unknown to the respondent. By selecting those who were in leadership positions related to the app, we were able to ensure respondents had a high level of historical and working knowledge on the app and data. At least two--one individual from each app's respective government and developer organizations--respondents were interviewed. If a respondent could not answer a question, we asked him or her to refer to a colleague in his or her organization. A total of 13 individuals were interviewed. A developer (PublicStuff) provided two respondents in the same interview, and the City of Toronto and City of Vancouver each provided two respondents in two separate interviews. Respondents were asked to describe the app and data, its development, and their interactions with their government or developer counterparts. They also were asked their views on outsourcing and public services.

Mapping of the Actors

To map the app ecosystem and find its obvious and not-so-obvious actors, we used two methods. First, we followed the data to determine the human actors. This was a snowball sampling method to identify interview respondents connected to the app and data flow, based on a similar implementation by Davies and Frank (2013) and Sands et al. (2012). We adapted this by selecting apps as a starting point, then looking forwards and backwards to identify who to interview at the source and the destination of the data flow. Interviewees then were asked to identify the next person down the data flow to interview.

Following the path through which data travels, from source to destination, captured the technical environment of the app. However, it did not capture the influence of non-humans or influences from entities outside of the data flow ecosystem. For instance, certain non-human actors may exert influence over the flow without the data physically passing through them, such as open data licences. To address this, we utilize Actor-Network Theory (ANT) to frame the interview data within an app ecosystem, with a view that an actor is any entity that "acts or to which activity is granted by others" (Latour 1996, 373), whether human or non-human. Callon states that examining power relationships entails "describing the way in which actors are defined, associated and simultaneously obliged to remain faithful to their alliances" (Callon 1986, 19). ANT therefore allows one to examine power relations and the conditions for power flows. Actors such as software platforms and APIs, and actors seemingly disassociated with data flow such as government legislation or directives, thus could be important in shaping the relationship between government and infomediaries.

The use of ANT in government IT projects is not new. Stanforth (2003) has examined actors' power over the development of public expenditure information systems in Sri Lanka, while Walsham and Sanjay (1999) examined GIS technology's embodiment of Western values in India. Heeks and Stanforth (2007) state it is common to consider ANT as a broad approach or perspective. They note that researchers have chosen specific aspects of ANT, such as the four moments of translation (Callon 1986), as a lens rather than a comprehensive analytical method. Carroll, Richardson, and Whelan (2012) documented some basic steps of adopting an ANT approach to include: Identifying actors, their relationships, tracing of actions (what activities led to these relationships), and which actors enable or inhibit certain actions within the network. Our semi-structured interviews were geared towards answering such questions.

Respondents problematized the ecosystem and identified actors by describing the origins of the app, key decision makers behind its origins, and current interactions between app developer and government officials. Their descriptions (e.g., who made decisions, what technologies they relied on) allowed us to see which actors had power over data and over other actors (via data). When individuals (such as corporate-level city officials) had enough power over data to differentiate their actions from that of their organizations (i.e., they expressed their own agency), they were considered unique actors within the ecosystem and became fixed. However, if an individual was beholden to their organization's goals, he or she was subsumed into a larger actor.

CASE OVERVIEW AND DATA FLOW

In the following sections, we build up the app ecosystem. First, we give an overview of each app ecosystem's actors. Then we present results describing the relationship between infomediaries and government, and the types of control over data they wield. We concentrate on the infomediaries in the ecosystem, but chose not to ignore government actors because we observed numerous possibilities for infomediaries to enter at different stages of data flow before data exits government control.

Citizen Dashboard

Citizen Dashboard is a browser-based interactive app from the City of Edmonton (see Figure 1). The app displays 64 (as of June 2016) performance measures that inform residents on progress towards city goals. Measures include event-specific measures such as snowploughing, monthly measures such as Disabled Adult Transit Service (DATS) on-time performance, and annual measures such as city operations greenhouse gas--emissions targets. Behind every performance measure is a continually updated dataset located in the City's open data catalogue. Citizen Dashboard's development, maintenance, and hosting were outsourced to a U.S.-based private-sector developer--Socrata. Socrata sells this dashboard to other customers under the name Open Performance.

Data flows from government to citizen primarily through Socrata. City officials in all branches of the administration are responsible for updating the data behind each performance measure. An employee in the Corporate Strategic Services department oversees all updates to Citizen Dashboard's performance measures to ensure they are conducted in a timely manner. Only employees in Corporate Strategic Services have edit permissions for the performance measures' descriptive texts and visualizations, which places them in a position of power. Performance measure datasets are manually updated and stored in the city's open data catalogue; all hosted by Socrata. The data visualization in Citizen Dashboard then is automatically updated by querying the open data catalogue via the Socrata Open Data Application Programming Interface (SODA).

In these cases, the data is not necessarily open throughout the ecosystem. Only when they are uploaded to Edmonton's open data catalogue does the data become open.

Ottawa Transit

Ottawa Transit allows users to plan trips across Ottawa and provides real-time notifications on bus arrivals (see Figure 2). Ottawa's transit agency, OC Transpo, uses a Computer Aided Dispatch and Automated Vehicle Location (CAD/AVL) software platform, called Hastus (developed by Giro), to create bus schedules. Hastus automates much of the data creation (employees select input variables but the software performs calculations) and transforms bus schedules into the General Transit Feed Specification (GTFS). In addition, transit schedule planning is dynamic, responding weekly or daily to changes in demand. Hastus also has an AVL component that can receive GPS sensor data and track bus locations in real time. This data is delivered to the public via an inhouse API called Live Next Bus, which is entirely automated.

Ottawa Transit was developed by 3lywa Solutions, a one-person firm based in Ottawa. App development was instigated by the developer's participation in a city hackathon. The app uses both the static bus schedule and the streaming bus locations to aid users in navigation. The developer downloads the entire bus schedule every season and transforms the GTFS-formatted data into a single relational database. Bus location data is accessed directly by the app in real time. OC Transpo provides guidance on the technical requirements for the data, such as how to access the API, but imposes no creative limits on developers.

A third-party API token management service (3Scale) manages API calls for Live Next Bus by sitting in between app developers and OC Transpo servers. It serves as a potential block for data as it forwards only authorized calls and establishes quotas on queries. The terms of use for both the GTFS and the Live Next Bus data are covered by an open data licence. Authentication was instituted because OC Transpo, prior to this app, had denied access to another infomediary that had been scraping schedule data from its website.

Citizen Budget

Citizen Budget is a browser-based app that collects citizen feedback on the Borough of Plateau-Mont-Royal annual budget (see Figure 3). The Borough of Plateau-Mont-Royal is part of the City of Montreal but possesses separate jurisdictional and data-collection authority. The app provides an interactive form within which citizens can act as finance officers and simulate budget allocation. Residents are asked whether they would like to increase or decrease spending and taxes on municipal operations, such as snowploughing, and whether they would like to initiate capital projects such as a new library. The app reflects the real funds available to the Borough and can constrain users to create a balanced budget. Budget data is available as a CSV file or in an automated PDF report with data charts.

Data in the Citizen Budget ecosystem flows through few actors. To create the questions in the app that guide users through the budget, political aides collect initial data from the City of Montreal's various departments and relays this to Open North via an electronic form. Open North hosts the online dashboard and the citizen-sourced data on its own servers. The data then passes to the Plateau-Mont-Royal administration. Political aides within the administration interpret both the automated report and the raw tabulated data and report their findings to the Borough Mayor and councillors for their budget planning. Aggregated data is revealed at the annual budget consultation to support budget decisions. Data also is only open once it reaches the public budget consultation, and is publicly available only in aggregated form.

The app is a product offered by Open North, a Montreal-based non-profit organization that promotes democratic participation through online tools. At the time of surveying (2015), the app was not in use because of a restructuring of the city's budget allocation.

Toronto Cycling App

Toronto Cycling App collects cyclists' travel routes to assist the City of Toronto's Cycling Infrastructure and Programs division plan Toronto's cycling network infrastructure (see Figure 4). App users contribute their cycling trips (GPS coordinates) and can complete an optional survey of demographic information and cycling habits. The app can display cycling-related information from Toronto's open data catalogue on a map.

The app was outsourced under a Request for Proposal bidding process to a private sector Ontario-based developer called Brisk Synergies. After individuals contribute information, the app sends the data to Brisk Synergies' data server. Here it is error-corrected, converted into line segments, and attached to the road network. Brisk Synergies hosts this data and provides an online portal for city officials to visualize the data and download for further analysis in their own systems.

Toronto planners input the data into their own geographic information system (GIS) (Esri's ArcGIS software) for spatial analysis. The output then is used in cycling infrastructure planning to map demand for routes. It was used to inform the 2015 Cycling Network Plan, an extensive five-year cycling infrastructure plan proposed to the council.

Planners in Cycling and Infrastructure Programs are the primary users of the data, although other employees from the city's Transportation Services division also can access the data. The City of Toronto respondent also mentioned a potential conversion of aggregated data into open data for Toronto's open data catalogue. When this is enacted, it will offer another point of data transmission.

VanConnect

VanConnect is an app used to file service requests akin to calling a 311 telephone hotline for non-emergencies (see Figure 5). Using either a mobile app or browser interface, users fill in a form to categorize the issue (e.g., pothole, broken streetlight, graffiti), map the location, and add any other information required. The objective of the app is to provide an alternate channel of communication for service requests and facilitate more automation of the process. VanConnect is developed by PublicStuff, a U.S.-based company and subsidiary of Accela.

The app sends the collected service requests to PublicStuff servers, which then forwards them to the City of Vancouver 311 Contact Centre's Customer Relationship Management (CRM) tool called LAGAN. The CRM automatically forwards the service request to a City of Vancouver department or an external organization such as Canada Post if the request lies outside the City of Vancouver's jurisdiction. The Director of Digital and Contact Services is a key decision maker who manages 311 Contact Centre operations; whereas the City's Digital Strategy, a development framework, dictates the Centre's motivation for automation and digitization.

PublicStuff's online dashboard is accessible by 311 Contact Centre employees and allows government to modify the app itself, such as adding "widgets" or modules that display information and maps and send push notifications. This means that data flows both to and from government. If users register their contact details, the city can send automated responses (via PublicStuff), notifying users of the status of their service requests. Finally, all 311 service requests (including telephone and e-mail) are aggregated and automatically published on Vancouver's open data catalogue.

PERCEPTIONS OF RELATIONSHIPS OF INFOMEDIARIES IN AN OPEN DATA ECOSYSTEM

This section covers responses to questions about the relationships between human infomediaries and government. Government respondents for all apps were unanimous in their view that external actors were essential to fill resource gaps in municipalities. The City of Toronto's Cycling and Infrastructure Programs is an analytical and planning unit, but reportedly lacked the resources to develop its own software. The City of Vancouver respondent stated that developing VanConnect "was not even on our radar to do it inhouse." Regarding their prior internal projects, "version control is very poor, it doesn't seem to grow very quickly, and usually the talent that built it, when they move on, we lose a whole bunch of knowledge." A third-party developer, therefore, is necessary to ensure a project will endure.

The Manager of Transit IT at OC Transpo's message was straightforward. It is infeasible and irresponsible for government to maintain the application because government cannot and should not keep pace with technological innovations. "The list is endless in terms of what [software] you have to support. The cost of that cannot be borne by the taxpayer; unfortunately, that's the reality." For this reason, OC Transpo only developed an app for the iOS mobile platform and leaves it to third parties to develop apps for other platforms.

Socrata's responses were contextualized in the ideology of open government. Socrata argued that "access to information is/ will change the way governments operate, both internally and externally." Socrata views its infomediary role as promoter of a larger open government movement in which government data needs to be freed for repurposing. It is business model reinforced by ideology:
   no it [data content] doesn't really matter [to Socrata]. But
   because we are also concerned with consumption ... we try
   to encourage the data to be more rich ... The more high
   value datasets individual customers publish, the more useful
   it is both to the citizens and to the developer community.


Socrata's survival depends on governments publishing quality open data, which suggests a symbiotic relationship with government. Other developers expressed much less codependency with government. PublicStuffs respondent expressed no opinion on data once it entered the government ecosystem. When asked how they hoped VanConnect's service request data would be used by the city, they separated their role from government, where "our hope is that the city will use the data to respond to citizens requests for citizens, but really that's not for us. We provide a service for clients, they can do whatever they want with the data."

Infomediary developers were driven by varied motivations. Open North is Canada's largest open data non-profit. Its mission is to promote democratic participation in more transparent and accountable government. Open North's respondent saw its role as guide or mentor to government in the implementation of Citizen Budget. An advisory role was necessary as it was the Borough's first attempt at an online tool. The respondent noted that, "usually they [Plateau-Mont-Royal] don't feel comfortable enough, so they always want for someone to say 'yea that's good'" and that the Borough did not always know what it wanted from Citizen Budget. Developers, such as Open North and Socrata, may view their roles in the ecosystem as greater than a single government contract. This differs from traditional project outsourcing. Infomediaries may be distinguished from traditional outsourcers in part because they integrate into outsourcing an intention to ensure civic data meets its objectives of citizen engagement and accountability.

Finally, the nature of civic data may reveal personal motivations driving app development. For the developer of Ottawa Transit (3lywa Solutions), "This whole application came from a personal struggle with busing, [which] started because of me as a user. I wanted to get data." The developer saw a gap in quality transit apps, particularly on the Android platform, and decided to create one for the developer and the community. The developer therefore positioned him or herself as both private-sector infomediary and representative of civil society. Our research finds the sectors of the developers start to blur, as do the boundaries between government and developer.

ROLES OF GOVERNMENT AND INFOMEDIARIES IN THE OPEN DATA ECOSYSTEM

In this section, we categorize the actors in each app ecosystem. Table 1 categorizes actors according to their roles, their attributes, and their types of control over the data or ecosystem. This allows us to separate the non-human and human infomediaries that influence data flow. Categorizing infomediaries allows us to explore the variations in actor roles. Table 2 defines the forms of control that the actors can exert. As seen in Table 1, all but one kind of actor exerts multiple forms of control over data, and government does not wield all the forms of control over data found in the app ecosystem. Instead, infomediaries express overlapping or completely different forms of control over data, particularly if they create their own technologies (such as APIs). Breaking down the forms of control over data exerted by actors (particularly infomediaries) allows us to map the varying interdependent relationships between government and infomediary.

City Departments

City departments act as dominant control points in the ecosystem. They exhibit control over data access, collection, modification, and decision making. While not infomediaries themselves, they hold major influence in the interdependent relations throughout an ecosystem.

The City of Edmonton's Corporate Strategic Services initiated Citizen Budget in partnership with Socrata and maintains administrative permissions for the app. All branches of administration are involved in data production but only one specific branch has oversight and the major interactions with the developer. Any changes, for example to data visualization or descriptive text, must obtain approval from Corporate Strategic Services. For Ottawa Transit, OC Transpo's Manager of Transit IT (who also was an interview respondent) is responsible for more than information management and software procurement. The Manager of Transit IT took a hands-on role in app ecosystem by educating (building data literacy) and regulating (setting guidelines for) developers. The manager hosted developer hackathons to explain the data schema and how it could be used and explained API call limits. In this way, the respondent demonstrated the ability to act both as a representative of his or her organization and with his or her own agency.

Developer

Outsourced developers are the traditional infomediaries represented in the literature. We found these actors adopting important roles of data structuration, representational transformations, data analysis, data literacy, and data advocacy. In many cities, Socrata has been given control of the entirety of the open data portal, from publishing (access) to analytics (value). Brisk Synergies' involvement in the collection and conversion of point data gives it control over the structure of data that is presented. Brisk Synergies also exerts control over the presentation and visualization of the route data via its online data dashboard. Skills such as processing point data, attaching line segments to the road network, and visualizing this data in an online interface were reported as outside the capacity of the City of Toronto's cycling planners.

Developers also may take on important roles beyond simple outsourcing contracts. Open North acted in an advisory capacity to government in teaching borough employees and aids the potential of online consultation tools. In this way, the non-profit acted to improve data literacy and also advocate for open data and crowdsourcing.

Data Directives

City reports, terms of service, and data laws can impact the ecosystem. Open data licences, such as the City of Ottawa's Open Data Licence, stipulate that the data may be freely used and modified, provided the source is acknowledged. For Citizen Dashboard, the City of Edmonton respondent cited The Way Ahead, a strategic plan that sets out development goals for the city for the year 2040, as an instigator of performance measures for the public and mandate for increased transparency. The City of Vancouver's Digital Strategy mandates the automation and digitization of public services. Legal jurisdiction over data is also a factor in the ecosystem. While Socrata does not claim ownership over the open data it hosts, its data servers are physically located in the U.S.A., which subjects City of Edmonton open data to foreign laws such as the U.S. Patriot Act. This concern has been raised for cloud-computing services in the past (Zhou, Zhang, Xie, Qian, and Zhou 2010) and represents significant obstacles to the engagement of U.S. infomediaries by Canadian cities, which operate under stricter privacy laws than does the U.S. (Scassa, 2013).

Open data Catalogue

Open data catalogues are the primary access portals by which governments control access to open data and its representation. Socrata's open data platform provides the user interface to the data, and displays the data on a map. The catalogue becomes the gateway for data downloads and the data representation (e.g. web mapping). When a data catalogue becomes the host for data displayed in an app, such as Citizen Dashboard, dependency on this actor increases. Citizen Dashboard's entire functionality relies on the City's Socrata-powered open data catalogue being functional.

Control over representation can be exerted through control over the type of data or data formats that the catalogue allows to be stored. Storing point data in a CSV file can reduce the potential for complex analyses compared to a Shapefile. Typical data portals display a list of datasets for the user to download wholesale. However, Socrata's open data catalogue does not store the dataset in discrete files. Instead, data is stored in a database, free of file associations. When a user downloads a specific dataset, it triggers an API call to the catalogue that queries the database and returns the specified file for download. This allows Socrata-powered data catalogues to export to a wide number of file formats, and allows users to make data queries straight into datasets without having to download them. This implementation does not mean that all data is available in all formats. For example, Edmonton's Tree Species dataset is displayed on a map in the open data catalogue, but is only downloadable in tabulated (CSV) and markup language (JSON, XML, RDF, RSS) formats, with no true spatial data format as an export option even though the dataset itself contains coordinate data. This limits the ability of users to perform complex analysis or mashups with other spatial datasets. Limiting the available formats also limits potential reuse to only those with the skills with that particular type of data.

Online Dashboard

Online dashboards exhibit control over representational transformation, data access, documentation, and augmentation. Socrata's platform allows the City of Edmonton to upload and modify data, and modify the content and visualization of performance measures. In cases such as Toronto Cycling App and Citizen Budget, an online dashboard is the only means for government to access the data; whereas others, such as Socrata, provide alternate access points such as an API and stand-alone software. PublicStuff's online dashboard also is used to add widgets to the VanConnect app, which can display additional information or notifications to users.

Stand-alone Software

Some government organizations relied on large software packages to perform tasks. These software packages exert control over data access, collection, modification, analysis, and decision making.

Several examples suggest the impact of software on the ecosystems of open data. The real-time bus-locations feed from OC Transpo's Live Next Bus API is completely automated and thus entirely reliant on the HASTUS system. The City of Vancouver's 311 Call Centre's LAGAN CRM also automates the rerouting of VanConnect service requests to service providers to negate the need for human mediation. This CRM is the central actor into which all 311 service requests flow. Because this software is responsible for categorizing requests, it dictates how app users structure their service requests. Stand-alone software, such as ArcGIS software used by the City of Toronto, are also entry points for the influence of their owners (in this case, ESRI) into the ecosystem.

Data Servers

Data servers act as important control points over data access and storage. Data for Citizen Dashboard and for the City of Edmonton's open data catalogue are remotely hosted by Socrata. The city's open data catalogue is stored abroad in the U.S., and thus is subject to U.S. data surveillance laws. The Citizen Dashboard's web pages also are served by Socrata's data centre, which extends the actor's power over a city's transparency policy. Interestingly, the city official at the City of Edmonton stated an incorrect belief that the City hosted its own open data catalogue.

APIs and Data Schema

APIs and schema control the structure, access, documentation, and representational transformation of data. Socrata's Open Data API provides an important example. The API mediates data by shaping the categories of data that are received by and sent to the Citizen Dashboard. The Socrata developer stated that the platform emphasised quantitative analytics. Qualitative data, particularly text, represents a secondary, and more difficult to aggregate and visualize, priority. This has resulted in certain visualization issues for the City of Edmonton, where qualitative metrics or event-based performance measures are difficult to depict. Even its credit rating measure, which is represented alphabetically (i.e., A++, A+, A, B, C, D) is insufficiently numerical and thus must be readapted for Socrata. The city also differs from Socrata in the temporal frequency of data updates. Edmonton produces performance measures called Lot Grading Inspections, which are conducted on an annual basis but cannot be displayed as a monthly trend as visualized by Socrata. Differences are not inherently negative but demonstrate the adjustments necessary for new IT and the control induced by any adaptation.

A subtler control point is GTFS, which is used in Ottawa Transit. GTFS remains the de facto standard for displaying transit data online. While it is an open specification where changes are agreed upon by the community, GTFS still is overseen by Google. Like the Google Maps API (used in Ottawa Transit, Toronto Cycling App, and VanConnect), GTFS has become an industry standard that serves as an entry point for Google into the app ecosystem and illustrates that technologies are not entirely under the control and ownership of infomediary developers.

Authentication

The Ottawa Transit case revealed the use of a token management service, 3Scale, as a control point over data access. By limiting access to data to those with authorized tokens, the token management platform allows OC Transpo to enforce quotas on API calls. OC Transpo has the ability to block an app should it contravene the terms of service and the open data licence. Control over data access is also a control over the potential for data transformation. For example, quotas on API calls can introduce issues of temporal accuracy and uncertainty in streaming data visualizations.

DISCUSSION

Infomediary interventions operate throughout the civic data production process. Interventions vary in size and extent, ranging from providing a schema to managing the entire data infrastructure. As expected, interview results confirmed support for outsourcing, and exploration of actors revealed the types of control that are being outsourced. Infomediary influence extended far into government as interview results confirm that sectoral boundaries are increasingly porous.

Infomediaries assumed roles traditionally held by government IT departments, a transferral welcomed by government. OC Transpo, for example, argued that government should not bear the responsibility of creating and maintaining apps, particularly due to the development costs that would be a needless burden on taxpayers. This extended to open data portals. Socrata assumed control of the entirety of Edmonton's open data portal. It should be noted that when we began a larger research project in 2011, Canadian cities including Edmonton expressed antipathy towards corporate management of open data portals. Cities, such as Toronto and Montreal (Plateau-Mont-Royal), now frankly admit their need to outsource, primarily due to their lack of app development capacity. Brisk Synergies and Open North therefore assumed government roles of data preprocessing (e.g., data aggregation and accuracy checking) and visualization. Potential complications, such as lack of control over the archiving or storage of data, were dismissed as government respondents were confident in the strength of contractual agreements.

Governments may delegate to infomediaries functions such as data publishing and distribution. The concern is that this will further a neoliberal delegation of power and responsibility (Leszczynski 2012). Coupled with cloud computing such as SaaS and storage services, government may be reduced or disassembled as it relinquishes control over more functions, a movement towards Government 2.0. Ceding control of functions, for example through the structuration of qualitative data, also suggests cities yield to a business model that effectively homogenizes cities. Contrary to the nuanced empowering potential of Web 2.0 (e.g., an enabler of multimedia narratives from citizens, ibid.), interdependency in the ecosystem can be quite limiting.

The level of dependence on infomediaries, particularly those embedded in the data production process, indicates a lack of substitutability for those services. If an outsourced service disappears, then government will need to adjust procedures that already were adapted to the prior service; services that may have been reified through policies and rules. Even if they chose a different vendor, government may require nontrivial reinvestment in hardware, software, and skills retraining. Efficiencies may be gained in shifts to new developers; however, this may be an irreversible process if government fails to retain the skilled labour and expertise required to rebuild data infrastructure. In such a case, government may lose the ability to evaluate outsourcing projects (Grimshaw et al. 2002).

This is not to say that O'Reilly's (2011) vision of "government as a platform" is inevitable. Should government become entirely reliant on third-party services, infomediaries themselves may become the new platforms that government plugs into, not the converse. Cloud platforms bring about the risk of being locked into proprietary APIs that limit a government's ability to outsource other components or substitute the platform itself (Paquette, Jaeger, and Wilson 2010). Infomediaries also are not equally substitutable. Sangiambut and Sieber (2016) found that developers came with differing agendas, particularly those with anti-corruption and transparency mandates built into their missions. As such, government may not easily replace a non-profit advocacy's open data service with that of a for-profit business.

Government also may experience a highly inelastic demand for services such as open data platforms, particularly when open data is mandated by data directives. Efforts towards government data that is "open by default" (Government of Canada 2012) may pull open data to the centre of mission-critical government functions. Should the price for portal services increase substantially, government may need to pay that cost. High dependency coupled with inflexible demand may induce monopolistic or anti-competitive behaviour by developers seeking to strengthen or maintain market share, especially if they are not easily substituted.

CONCLUSION

Through an exploration of open data app ecosystems, we identified infomediaries and categorized them by type and function. Sources of control over data and data flow are not necessarily between one government and one outsourced app developer but exist in a complex network of mutual interdependencies. We began by examining human actors but found that non-human actors exerted considerable, and sometimes unacknowledged, influence in the ecosystem. With sufficient trust, governments expressed low concern over the risks of government "downsizing" payments to the portal company due to austerity (Sieber and Johnson 2015) or private-sector firms terminating services due to lacklustre business models.

Infomediaries are increasingly crucial in an era of interconnected and cloud-based IT. Compared to traditional software outsourcing, current apps built on open data, crowdsourcing, and Web 2.0 require more complex interdependencies. These arrangements are further supported by existing governance paradigms such as citizen-sourcing and NPM. The more mission critical a service (e.g., an unofficial transit app upon which everyone now relies), the more government should be mindful of the infomediaries and the resultant interdependencies of the ecosystem.

Conflict of Interest

All cities mentioned in this paper are affiliated with a multi-researcher Social Sciences and Humanities Research Council (SSHRC) partnership grant (895-2012-1023), which funded this research. The respondent from Open North and one respondent from the City of Vancouver are the contacts for the grant. However, there was no obligation to participate in this study.

Suthee Sangiambut completed his M.A. in McGill University's Department of Geography. His graduate research was related to open government, open data, and civic apps, particularly their impacts on citizen-government interactions, government practices, and policy. He continues his work in open data and open government policies in the non-profit sector.

Renee Sieber is an associate professor of geographic information science, jointly appointed between the Department of Geography and the School of Environment at McGill University. She received a Ph.D. in planning and public policy from Rutgers University. Her research interests include public participation geographic information system, community engagement, governance, open data, and civic technology.

Corresponding Address:

Department of Geography

McGill University

Montreal, QC, Canada H3A 0B9

(514) 398-4941

renee.sieber@mcgill.ca

References

Bates, J. 2014. The strategic importance of information policy for the contemporary neoliberal state: The case of Open Government Data in the United Kingdom. Government Information Quarterly, 31(3): 388-95.

Bertot, J. C., P. T. Jaeger, and D. Hansen. 2012. The impact of policies on government social media usage: Issues, challenges, and recommendations. Government Information Quarterly 29(1): 30-40.

Bond, A. T. 2015. An App for That: Local Governments and the Rise of the Sharing Economy. Notre Dame Law Review 90(2): 77-96.

Brabham, D. C. 2009. Crowdsourcing the Public Participation Process for Planning Projects. Planning Theory 8(3): 242-62.

Callon, M. 1986. Some elements of a sociology of translation: domestication of the scallops and the fishermen of St. Brieuc Bay. In J. Law, Ed. Power, action, and belief: A new sociology of knowledge. London: Routledge, 196-223.

Carl, T. 2012. The G4: Setting city data free--Canadian Government Executive. Retrieved March 20, 2016, from http:// canadiangovernmentexecutive.ca/the-g4-setting-city-datafree/.

Chen, Y.-C., and J. Gant. 2001. Transforming local e-government services: the use of application service providers. Government Information Quarterly 18(4): 343-55.

Davies, T., and M. Frank. 2013. 'There's no such thing as raw data': exploring the socio-technical life of a government dataset. In Proceedings of the 5th Annual ACM Web Science Conference. New York: ACM Press, 75-78.

Davies, T., F. Perini, and J. M. Alonso. 2013. Researching the emerging impacts of open data. Open Data Research Network. Retrieved from http://www.opendataresearch.org/ sites/default/files/posts/Researching the emerging impacts of open data.pdf.

Denhardt, R. B., and J. V. Denhardt. 2000. The New Public Service: Serving Rather Than Steering. Public Administration Review 60(6): 549-59.

Evans, D. S., and B. J. Reddy. 2002. Government preferences for promoting open-source software: a solution in search of a problem. Michigan Telecommunications and Technology Law Review 9: 313.

Fishenden, J., and M. Thompson. 2013. Digital Government, Open Architecture, and Innovation: Why Public Sector IT Will Never Be the Same Again. Journal of Public Administration Research and Theory 23(4): 977-1,004.

Freeman, R. J., and P. Loo. 2009. Web 2.0 and E-Government at the Municipal Level. In Proceedings of the World Congress on Privacy, Security, Trust, and the Management of e-Business, 2009 (Congress 2009), 70-78.

Goodchild, M. F. 2007. Citizens as sensors: the world of volunteered geography. GeoJournal 69(4): 211-21.

Government of Canada. 2012. Canada's Action Plan on Open Government. Retrieved from http://open.canada.ca/en/ canadas-action-plan-open-government.

Grimshaw, D., S. Vincent, and H. Willmott. 2002. Going privately: partnership and outsourcing in UK public services. Public Administration, 80(3): 475-502.

Hagel, J., and J. F. Rayport. 1997. The Coming Battle for Customer Information. Harvard Business Review, 64-77.

Haklay, M. 2013. Citizen Science and Volunteered Geographic Information: Overview and Typology of Participation. In D. Sui, S. Elwood, and M. F. Goodchild, Eds. Crowdsourcing Geographic Knowledge. Springer Netherlands, 105-122.

Heeks, R., and C. Stanforth. 2007. Understanding e-government project trajectories from an actor-network perspective. European Journal of Information Systems 16(2): 165-77.

Hood, C. 1995. The "new public management" in the 1980s: Variations on a theme. Accounting, Organizations and Society 20(2-3): 93-109.

Janssen, M., and A. Zuiderwijk. 2014. Infomediary Business Models for Connecting Open Data Providers and Users. Social Science Computer Review 32(5): 694-711.

Johnson, P., and P. Robinson. 2014. Civic Hackathons: Innovation, Procurement, or Civic Engagement? The Review of Policy Research 31(4): 349-57.

Klischewski, R. 2010. Drift or Shift? Propositions for Changing Roles of Administrations in E-Government. In M. A. Wimmer, J.-L. Chappelet, M. Janssen, and H. J. Scholl, Eds. Electronic Government. Springer Berlin Heidelberg, 85-96.

Latour, B. 1996. On actor-network theory: A few clarifications. Soziale Welt 47(4): 369-81.

Leszczynski, A. 2012. Situating the geoweb in political economy. Progress in Human Geography 36(1): 72-89.

Nam, T. 2012. Suggesting frameworks of citizen-sourcing via Government 2.0. Government Information Quarterly 29(1): 12-20.

Open Knowledge International. 2012. OpenDefinition. Retrieved from http://opendefinition.org/.

O'Reilly, T. 2007. What is Web 2.0: Design Patterns and Business Models for the Next Generation of Software. Communications & Strategies 1. Retrieved from http://papers.ssrn.com/ abstract=1008839.

O'Reilly, T. 2011. Government as a Platform. Innovations: Technology, Governance, Globalization 6(1): 13-40.

Ostrom, E. 1996. Crossing the great divide: Coproduction, synergy, and development. World Development 24(6): 1,073-87.

Paquette, S., P. T. Jaeger, and S. C. Wilson. 2010. Identifying the security risks associated with governmental use of cloud computing. Government Information Quarterly 27(3): 245-53.

Quigg, Bridget. 2014. City of Edmonton Wins Silver for Citizen Dashboard. Retrieved July 22, 2017, from https://socrata. com/blog/city-edmonton-wins-silver-citizen-dashboard/

Sandoval-Almazan, R., J. R. Gil-Garcia, L. F. Luna-Reyes, D. E. Luna, and Y. Rojas-Romero. 2012. Open government 2.0: Citizen Empowerment through Open Data, Web and Mobile Apps. In Proceedings of the 6th International Conference on Theory and Practice of Electronic Governance--ICEGOV '12. New York: ACM Press, 30.

Sands, A., C. L. Borgman, L. Wynholds, and S. Traweek. 2012. Follow the data: How astronomers use and reuse data. Proceedings of the American Society for Information Science and Technology 49(1): 1-3.

Sangiambut, S., and R. Sieber. 2016. The V in VGI: Citizens or Civic Data Sources. Urban Planning 1(2).

Scassa, T. 2013. Legal issues with volunteered geographic information. The Canadian Geographer/Le Geographe Canadien 57(1): 1-10.

Schneider, S., and A. Sunyaev. 2016. Determinant factors of cloud-sourcing decisions: reflecting on the IT outsourcing literature in the era of cloud computing. Journal of Information Technology Impact 31(1): 1-31.

Sieber, R., and P. Johnson. 2015. Civic open data at a crossroads: Dominant models and current challenges. Government Information Quarterly 32(3): 308-15.

Sivarajah, U., Z. Irani, and V. Weerakkody. 2015. Evaluating the use and impact of Web 2.0 technologies in local government. Government Information Quarterly 32(4): 473-87.

Stanforth, C. 2003. Using Actor-Network Theory to Analyze E-Government Implementation in Developing Countries. Information Technologies & International Development 3(3). MIT Press.

Stroh, L. K., and D. Treehuboff. 2003. Outsourcing HR Functions: When--and When Not--to Go Outside. Journal of Leadership & Organizational Studies 10(1): 19-28.

Walsham, G., and S. Sahay. 1999. GIS for District-Level Administration in India: Problems and Opportunities. MIS Quarterly 23(1): 39.

Worthy, B. 2015. The Impact of Open Data in the UK: Complex, Unpredictable, and Political. Public Administration 93(3): 788-805.

Yu, T.-Y. 2014. An empirical study of collaborative partnering among enterprises and government organizations for information system outsourcing. Applied Economics 46(3): 312-22.

Zhou, M., R. Zhang, W Xie, W Qian, and A. Zhou. 2010. Security and Privacy in Cloud Computing: A Survey. In 2010 Sixth International Conference on Semantics, Knowledge and Grid (SKG), 105-12.

Caption: Figure 1. Citizen Dashboard app ecosystem

Caption: Figure 2. Ottawa Transit app ecosystem

Caption: Figure 3. Citizen Budget app ecosystem

Caption: Figure 4. Toronto Cycling App

Caption: Figure 5. VanConnect app ecosystem
Table 1. Categories of actors in the app ecosystem

Category         Example                     Description

City             Vancouver 311 Call          Units and individual
departments      Centre, employees e.g.,     employees within
                 OC Transpo Manager of       government
                 Transit IT

Developer        Brisk Synergies, Open       Organizations and
                 North                       individuals outside
                                             government who create
                                             software applications.
                                             May be government
                                             contracted or independent

Data             Open data licences, The     Texts that establish
directives       Way Ahead development       mission, legal mandate,
                 plan, U.S. Patriot Act      terms of service,
                                             jurisdiction,
                                             intellectual property
                                             over the data

Open data        Edmonton Open Data          Internet application that
catalogue        Catalogue                   facilitates the user in
                                             finding repurposable and
                                             downloadable information,
                                             through vocabularies,
                                             metadata, and user
                                             interfaces

Online data      Socrata, PublicStuff        Editing interfaces used
dashboard                                    by governments to access
                                             data and modify apps (not
                                             to be confused with the
                                             Citizen Dashboard app)

Stand-alone      HASTUS, ArcGIS, CRM         Traditional offline
software                                     computer software
                                             packages that do not rely
                                             on cloud services

Data server      OC Transpo, Socrata open    Hardware repository for
                 data server                 data

APIs and         Socrata Open Data API       Software-to-software
Data Schema      (SODA), Live Next Bus,      interfaces that manage
                 Google Maps API             access to data, who
                                             accesses the data, how
                                             the data is structured,
                                             and how fast it flows

Authentication   3Scale                      Software that vets access
                                             to a data server and
                                             enforces quotas on data
                                             downloads

Category         Points of Control

City             Data collection, data
departments      modification, data
                 access, data advocate

Developer        Data structuration,
                 representational
                 transformation, data
                 analysis, data literacy,
                 data advocate

Data             Data mandate
directives

Open data        Data access, data
catalogue        documentation,
                 representational
                 transformation

Online data      Representational
dashboard        transformation, data
                 access, data
                 documentation, data
                 augmentation

Stand-alone      Data collection, data
software         modification, data
                 analysis

Data server      Data storage, data access

APIs and         Data structuration,
Data Schema      representational
                 transformation, data
                 access, data
                 documentation

Authentication   Data access,
                 representational
                 transformation

Table 2. Definitions of types of control

Types of Control   Definition                  Category

Data collection    Data creation or            City departments, stand-
                   Acquisition                 alone software

Data access        Authority to download,      Data server, tokens and
                   stream, or view data and    authentication, online
                   at what volume/rate.        data dashboard, open
                   Includes tokens and other   data catalogue
                   forms of authentication,
                   as well as blocked
                   Internet Protocol (IP)
                   addresses.

Data storage       Container or custodian      Data server
                   for data

Data               Methods of categorization   Developer, APIs, and
structuration      and classification          data schema

Data               Changes (e.g., arithmetic   City departments
modification       operations, geometric
                   transformations,
                   conversions) of data or
                   data fields

Representational   Changes in how the data     Online data dashboard,
transformation     is represented or           APIs, and data schema
                   interacted with.
                   Includes user interface,
                   analytics, graphical
                   displays

Data mandate       Authority, ordinances to    Data directives
                   create, use data.
                   Includes licences,
                   legislation, mission
                   statements

Data               Data about data, which      APIs and data schema,
documentation      explains, e.g.,             online data dashboards
                   provenance, purpose,
                   time, location, and other
                   descriptions. Includes
                   metadata, technical
                   specifications

Data               Extensions of data.         Online data dashboard
augmentation       Includes alerts, comments
                   about the data

Data literacy      Material (physical,         Developer
                   virtual) to increase
                   written comprehension,
                   numeracy, and data
                   utility

Data analysis      Numerical, statistical,     Developer, stand-alone
                   or graphical treatment of   software
                   the data

Data advocate      Promotion of data           City departments,
                   sharing, open government,   developer
                   and open data
COPYRIGHT 2017 Urban and Regional Information Systems Association (URISA)
No portion of this article can be reproduced without the express written permission from the copyright holder.
Copyright 2017 Gale, Cengage Learning. All rights reserved.

Article Details
Printer friendly Cite/link Email Feedback
Author:Sangiambut, Suthee; Sieber, Renee
Publication:URISA Journal
Date:Dec 1, 2017
Words:9316
Previous Article:The Geospatial Contents of Municipal and Regional Open Data Catalogs in Canada.
Topics:

Terms of use | Privacy policy | Copyright © 2019 Farlex, Inc. | Feedback | For webmasters