The research practices and needs of non-profit organizations in an urban center.
Keywords: nonprofits, research methods, data management
Of all the capacity issues facing nonprofit organizations in the United States, perhaps none has been so neglected as their research data needs. There are numerous training and technical assistance resources for various programming areas, organizational development, and information technology. But, outside of evaluation research, the crucial area of collecting, managing, analyzing and using research data is neglected by the trainers and the researchers working with nonprofits. And that is a serious neglect. Grant proposals, evaluations, fiscal monitoring, resource allocation, and overall project planning are all dependent on tight research and solid information. The types of research data needed for those activities can range from census data to client case data to all varieties of specialty data for different nonprofits, making research data practice a highly complex and time-consuming part of nonprofit work.
What do I mean by "research data practice"? Basically, I am concerned with the systematic collection of information to inform nonprofit program development and evaluation. This includes typically-recognized research practices such as surveys. It also includes systematic collection of client or membership data that could be, though it is often not, analyzed for patterns or categories. Research data practice does not have to be informed by a theoretical or disciplinary-based question and, instead, may be informed by a practical need such as "what are the best practices in the area of youth programming" (Stoecker, 2005). This research focuses on what research data nonprofit organizations collect, how they use that data, and what challenges they face in collecting and using research data. It is important to understand that the concept of research data includes most of the information that nonprofits collect, including client-level data that may not be viewed as research data by the organization. Such information, however, can often be very informative in helping nonprofits understand trends in who they are serving and in supplying the sampling frame from which they can conduct evaluation research. That such data is often not seen as research data is part of the problem.
The lack of focus on nonprofits' research data practice needs has real consequences. This research project began as an outgrowth of my experience conducting participatory evaluation research with a local neighborhood organization. As we attempted to collect outcome data for the organization, we realized how little data there was, and how difficult it was to fill the data gaps. Consequently, a group of us concerned about this issue set out to determine the state of the art of nonprofit information management.
The Lack of Data on Nonprofit Research Needs and Practices
We know little about the research data needs, practices, and capacities of small and medium size nonprofit organizations. In fact, in the broad area of nonprofit research, research data practice is almost completely uncharted. That doesn't mean we lack knowledge about research and data management methods that nonprofits could use. Indeed, there are voluminous literatures on everything from needs assessments and asset assessments to evaluation and a wide variety of other data practices in between that are common to nonprofit organizations. But we simply have no knowledge of the extent to which nonprofits effectively deploy those methods to collect, manage, and use research data and other information. There is only one published case study of a nonprofit's information management strategies (Houchin, 2002), and one other published report on an attempt to build nonprofit research and data capacity (Dattalo, 1998).
We do have some research suggesting that nonprofits' needs in the area of research data practices are pronounced. Perception gaps between nonprofit service recipients and providers, for example, not only exist, but are often unrecognized, particularly for neighborhood organizations (Kissane & Gingerich, 2004). And it is in neighborhood organizations where the research data deficit may be most pronounced because most data in the nonprofit realm is not coded by neighborhood. Nonprofits engaged in policy battles also need to learn about and engage in policy research methods to have any hope of impacting social policy (Fox, 2001; Appleton, 2003). And nonprofit management experts (Letts, Ryan, & Grossman, 1999; Bryson, 1995) emphasize the importance of research for effective nonprofit mission accomplishment.
It is in the arena of evaluation where research and data management issues have been most discussed (Mancini, Marek, Byrne, & Huebner, 2004). Without developing their own data and research capacity, nonprofits are at the mercy of outsiders who can impose performance measures in support of increasingly popular "pay for performance" schemes (Theuvsen, 2004). The antagonism generated by most funder-directed evaluation schemes leads nonprofits and funders to play a cat and mouse game with research data, with nonprofits often reduced to selectively sharing information, enlisting experts to massage information, or generating information that is only symbolic (Ebrahim, 2002).
There is continuous argument over how outsiders can measure nonprofit organization effectiveness (Sowa, Selden, & Sandfort, 2004; Wing, 2004). Social accounting is expanding in popularity as a way to measure the social and economic value of nonprofit and community organization activities (Mook, Richmond & Quarter, 2003). More common are attempts to measure the economic impact of nonprofit activities, particularly by community development organizations (Woller and Parsons, 2002). There is pressure, however, to shift from output evaluation to outcome and impact evaluation (McNamara, 1999; United Way of America, 2005), vastly expanding the thorny methodological issues facing nonprofits.
There is also an expanding debate over who should be doing evaluation. Wadsworth (1991) urges organizations to integrate evaluation research into their daily activities in order to become more self-directed. Others have promoted empowerment evaluation and participatory evaluation, which better protect organizations against admitting failure because the model actually helps organizations achieve greater success (Fetterman, 2002; Patton, 1997; Millett, 1996). But while there is plenty of urging going on, there is only one published case of an evaluation requested by a nonprofit organization to address a concern it had about one of its programs (Farmer & Walsh, 1999).
The broader model of participatory research is beginning to produce some literature on nonprofit research data practice. There are recent articles written on using participatory research with nonprofit organizations (Castelloe, Watson, & White, 2002) across a variety of research activities such as needs assessments (Balaswamy & Dabelko, 2002) and policy research (Padilla, Lein, & Cruz, 1999). But the emphasis of this literature is more on the process of the research relationships than on the specific data collection and analysis methods.
This uncharted territory, then, leaves us with two crucial unanswered questions. How much research capacity do nonprofit organizations have? How much do they need?
This research project set out to begin answering those questions for one metropolitan area. Toledo is a medium-sized Midwestern rustbelt city, with declining population in the central city and little to no growth in the metropolitan area. It has yet to make the transition to a post-industrial economy. Industrial, commercial, and housing abandonment is common in many parts of the city and its tax base has been stretched by the out-migration of the middle class. There are, as a consequence, many social needs. The overall project, of which the survey reported in the paper was one part, was designed to first diagnose nonprofit research data needs, and then to develop programming around those needs. It included a survey of nonprofits (reported in this paper), in-depth interviews with funders, and then subsequent programming that involved a research training series and a pilot neighborhood indicators database.
To find out whether my experience with the neighborhood evaluation project described above was common to the nonprofit sector, I gathered a group of nonprofit organizations in fall of 2003 for a focus group to explore the question of nonprofit research needs. The focus group discussion, which was organized around very general questions of how organizations conducted research, for what reasons, and with what challenges, confirmed my suspicions. Organization staff felt as if they were at the mercy of funders' perceptions of needs, and funders' definitions of good evaluation. The staff had a lot of data, but didn't know what to do with it, in the case of city-wide nonprofits, or severely lacked data for smaller geographies such as neighborhoods, in the case of neighborhood organizations. Based on the results of this focus group, we organized a research project to gather detailed information on the depth and extent of Toledo area nonprofits' research and data needs.
This research was conducted using a modified participatory research process, following an initiator model where the researcher chooses the initial research idea and then uses the research process to build more and more control over the process by the target group (Stoecker, 2005). Ideally, in this model, the research participants will have increasing control over every stage of the research process: choosing the research question, designing the research methods, collecting the data, analyzing the data, and developing action plans based on the data. The first step was to establish a research core group representing a cross-section of Toledo community organizations. The focus group helped establish the outlines of the research project and provided its core group members.
This core group of seven members met monthly through the six months of the project. They participated at every stage of the research, shaping the questions we wanted to ask in the survey of nonprofits and the interviews of funders, recruiting nonprofits to complete the survey, going over the survey results and a rough draft of this report to contribute to the data analysis, and shaping the planning event growing out of this research. The core group members' ongoing discussions added to the existing data by providing an array of nonprofit interpretations of the data we were receiving through the surveys and interviews described below.
The core group decided early on to focus on small to medium size nonprofits--excluding large health and education nonprofits--believing that the smaller organizations would be most in need of resources to collect, manage, and analyze research data. Another goal of the overall project was to look for ways to better support the research data needs of organizations that serve distinct geographic areas such as neighborhoods--community development corporations or CDCs--since we had already learned that neighborhood-level research data is extremely difficult to collect and maintain. Neither government nor non-geographic social service agencies manage their data in such a way to make it easily analyzable for an individual neighborhood. In addition, neighborhoods are a crucial geographic unit. Healthy neighborhoods provide the immediate, necessary support for family systems, convenient services, and youth activities, and may be the single most important point of intervention for improving safety, security, and happiness (Iannotta & Ross, 2004). Particularly when the neighborhoods are as well-defined as they are in Toledo, they also become a source of identity and, when they are healthy, pride.
The main focus of the research, then, was a survey of small to medium-size nonprofit organizations in the three-county Toledo metropolitan area. With the core group, I created a fairly detailed two page booklet-style survey with 18 questions, but 164 possible response categories, which took our core group members about 15-20 minutes to answer. Hager, Wilson, Pollak, and Rooney (2003) found that questionnaire complexity and incentives had little impact on return rates, but that form of invitation did. In their case, using Federal Express increased returns. I then compiled a population list from two existing lists of NPOs (both of which attempted to find the entire population of NPOs in the metropolitan area). After eliminating large nonprofits with 100 or more employees (which was the standard used by one of the two lists), and combining nonprofit programs managed under a single sponsor, we had a list of 432 nonprofit organizations. Five requested to be removed from the study, and eight could not be contacted, leaving the final survey pool of 419 organizations.
We used multiple methods to attempt to encourage response to the survey. For those organizations with e-mail addresses, we sent an e-mail invitation to participate in the survey, with a Microsoft Word version of the survey attached, as well as brief instructions for how to complete a web version of the survey. 153 organizations had listed e-mail addresses, but 27 of those addresses bounced, and three organizations requested to be removed from the study, leaving an e-mail accessible pool of 123 organizations. We contacted another 83 organizations by fax (including 22 organizations whose e-mail addresses had bounced), of which 62 were successfully sent. Of the 21 organizations not accessible by either means, 13 could be contacted by phone and were added to the postal survey. We used postal mail to send paper surveys with postage-paid reply envelopes to 234 organizations, also offering them the option to complete the survey on the web. We sent two follow-up e-mails to the e-mail pool, and two follow-up postal mailings to both the fax and the postal mail pools.
We suspect that questionnaire complexity may have held down response rates, and we also noticed variation in response rates that paralleled the form of invitation. We received 33 surveys via the web form, 8 via fax, 12 via e-mail, and 27 via postal mail. We suspect, but cannot be certain, that most if not all of the web and e-mail surveys were returned from organizations contacted via e-mail. So, in our case, we believe the best response came via the e-mail invitation. And while the different media showed different response rates, past research has shown that different media do not seem to produce different response distributions except around questions concerning information technology, where variation in media may increase the overall response by appealing to different respondent preferences (Parackal, 2000; Yun, 2000).
The total response was 80 organizations--a response rate of 19%. Low response rates are increasingly common in the survey industry (Sheehan, 2001). And while such a response rate may provide suspect data, Hager, Wilson, Pollak and Rooney (2003) argue that response rates as low as 15% should still provide accurate data when the demographic characteristics of the respondents are roughly representative of the population in general (Hikmet & Chen, 2003). Our survey included questions to judge the representativeness of our respondents' organizations, such as class
level of clientele served and mission areas of the nonprofit. Table 1 shows the responses to the mission activities question. The core group members judged that, based on the mission area data, the responding organizations were proportionally representative of the industry overall in the Toledo area, with a slightly high response from youth organizations. We also appear to have obtained a 100% response from the community development organizations in the city, which we had hoped for.
Types of Data Collected
Toledo area nonprofits collect data across many levels of analysis. It is interesting to note, however, that relatively few organizations collect data at the neighborhood level. In fact, organizations are more likely to collect data at the city and county level than at the neighborhood level, as Table 2 shows. Furthermore, 9 of the 23 organizations collecting research data at the neighborhood level are engaged in neighborhood development and community building activities. Both the nonprofit focus group participants and funders noted that neighborhood development requires comprehensive data from a variety of sources, including social service agencies who track some of the social ills associated with community underdevelopment. It may seem natural for agencies that do not operate at the neighborhood level to not collect such data, but those are also the very organizations that neighborhood groups would most like to get data from to assess needs and evaluate outcomes. In many cases it is also difficult to "objectively" determine neighborhood boundaries, but most neighborhood organizations are more concerned with gathering data from their defined service areas. Given the lack of data collected at the neighborhood level, community development organizations and other neighborhood groups will suffer some of the greatest research data hardships.
The lack of data collection at the neighborhood level is probably due to two factors. One is that client intake forms or membership forms generally do not ask for people to identify their neighborhood. The second is that many organizations may not realize that address-level data could be coded by neighborhood. Even if they did realize that possibility, however, doing such coding is a time consuming task outside of the capacity of most nonprofits. Instead, even neighborhood groups use census tract and zip code-level data that poorly approximates Toledo neighborhood boundaries.
Toledo area nonprofits also collect data on a wide variety of topics. Table 3 shows just how wide the topics are. Approximately three quarters of the organizations collect basic demographic data. In addition, over half collect some neighborhood level data, though we suspect that most of the respondents interpreted this question as asking whether they collected address information from clients and participants. Beyond that, however, there is no standard data collection pattern. This makes it difficult for organizations to share data and, as we will see, data sharing among the groups is in fact limited. The lack of organizations that collect data on leadership skill points to another research data gap for CDCs, who need such information to build strong resident participation in neighborhood development activities. This is, of course challenging information for the average social service agency to collect. But if other organizations that routinely collect data on clients or members could also learn how to ask what leadership experiences individuals have had, they could refer skilled individuals to their respective neighborhood organizations.
There is also wide variation in the number of categories for which organizations collect data. Half of the organizations collected data in seven or fewer categories. The most common response was to collect data in only one category, which is true of 11 organizations.
Research Data Management
One of our main concerns in understanding nonprofit data and research needs is how they handle the data that they currently collect. The picture that emerged from the survey is that nonprofits spend enormous person-hours collecting data that is seldom used. The average nonprofit organization has 5 employees and 4 volunteers who have some involvement with data collection, entry, and storage on a day-to-day basis. Combined, those 9 people spend 56 hours per week, over six hours per person, on data management. That is more than a full-time position just for data management. And while it may seem like a lot of time, remember all the possible kinds of data collection and management involved in nonprofit management, from logging phone calls to managing budgets to tracking client contact hours. So, given that most of the staff in a nonprofit are collecting data on clients or program participants, the lack of a standard system for data management could create information chaos.
Toledo nonprofits have piles and piles of data. Seventy-one of the 80 organizations store data more than three years. On average, 61% of the data is saved in paper files, likely creating both space and data recovery issues for many nonprofits. Data in paper form cannot be easily databased or analyzed. On the other hand, it's a good thing all that paper is kept because only 30 percent of the organizations use any kind of backup for electronic data. Twenty-eight of the 80 responding organizations indicated they had lost data due to document misfiling or computer-related problems. That figure is not as high as we had feared, but higher than it should be.
One of the most interesting findings, reported in Table 4, is that Toledo area nonprofits do not make much use of all that data that they spend so much time collecting and managing. If we take the data presented earlier, in Table 3, and add a column showing how many organizations actually use the data they collect, we can see that, in most cases, less than two-thirds of the organizations use the data that they collect in any one category. It could be argued that two-thirds is actually pretty high usage, but if we go back to the finding that the organizations are spending an average of 56 hours a week managing data, then more than 18 hours a week is wasted effort, adding up to 970 wasted hours in a year. Given that the average organization has 5 staff members and 4 volunteers (and assuming that the volunteers are not full-time), that is a high proportion of wasted effort.
The original focus group that prompted this study, as well as the input of the nonprofit core group guiding the study, can help us understand why there is so much wasted effort. One of the most vocal complaints coming from nonprofits has to do with funder-mandated data collection. Many organizations find that the things they are required to report on do not help them actually do their own work. The focus group and core group also noted they collect a large amount of information to meet legal requirements. Technically, we could say that, because such data collection results in further funding, or maintains organizational legal status, it is used. But our organizations seem to define "use" as actually taking the information and analyzing it to improve their practice. And, in this sense, much of the information they collect is not used. Finally, as we will see below, much of the information is not used because organization staff and volunteers do not have the skill or capacity to use it.
As some of the earlier survey results have intimated, not only do a lot of data go unused by the organizations themselves, data are not widely shared among organizations. Using just a raw average, any single nonprofit shares research data with only seven other organizations. But that figure is inflated by the inclusion of two organizations who said they shared data with more than 100 other groups. If we remove them from the calculation, the average falls to just four other groups that each organization shares data with. This may also be overstated if the groups interpreted the question as asking about receiving data from others as well as providing it. Eighteen organizations, nearly a quarter of the total, share data with no one. Thirty-nine organizations, nearly half of the total, share data with two or fewer organizations. This may be partly a consequence of limited collaborations among Toledo nonprofits. But it is probably also likely due to a lack of standard data collection that would make data sharing easy. The focus group participants noted that variations in funding often cause their data collection processes to be inconsistent, and realize that makes the data unreliable and thus less worth sharing.
Most puzzling are the results for the question asking nonprofits to indicate the categories in Table 4 for which they needed research data. The core group suspected that the question was interpreted differently than we meant it, as a number of organizations who indicated that they collect data in a category also said they needed data in that category, perhaps indicating that they collected it because they needed it. But if that was a common interpretation of the question, the results are more troubling, as very few organizations indicated a need for any category of research data. Fewer than a quarter of the organizations indicated a need for data in any category, with the greatest need being expressed for funding data (by 19 organizations). This finding is consistent with the data on lack of use. While collecting data may be required by funders and the law, using it is often a higher-order activity available only to those groups not already completely overstretched just doing their work.
The question is whether the lack of perceived need for data indicates that it is not important to the work of nonprofits or reveals a lack of knowledge about how data might be used in nonprofit work. The core group suspected it to be the latter. As we will see next, the nonprofits do indicate a need for increasing their research data capacity.
Need for Increased Research Data Capacity
The survey also focused on the nonprofits' research data strategy needs. One of the ways to judge those needs is by the number of organizations who report that they are required to conduct evaluations for funders. Nearly half, 36, are required by funders to conduct evaluations of their programs to receive continued funding. But there is more going on here than simply meeting a funder requirement. In fact, 49 of the 80 respondents indicated that they conduct annual evaluations, and another 19 conduct semi-annual evaluations. However, 23 organizations indicated that they do not conduct formal evaluations, leading us to believe that, of the 68 organizations that say they evaluate their work, some likely do so in only a cursory fashion. On the other hand, 24 organizations use outside consultants in their evaluations--a surprisingly high number and an expensive proposition for a small to medium size nonprofit. We must keep in mind, however, that number may not reflect the Toledo area nonprofit industry in general, as we suspect that area nonprofits more attuned to responding to issues of data collection and analysis may have been more likely to respond to the survey. In fact, the funders interviewed as part of the larger project, both government and private, were generally dissatisfied with the research data that organizations provided both to justify grant proposals and to support evaluations.
If we move beyond evaluation to the overall research data skills of nonprofit staff, we see more clearly the capacity issues facing the organizations. We asked the organizations to tell us how many staff had training in research at the undergraduate, graduate, or professional level, as well as being self-taught. None of the averages even reached one staff person. Nearly hall 38 organizations, have no staff or volunteers with formal research methods training. Of the remainder, most indicate having one or two people with research training. And remember, the average nonprofit organization in this survey has 9 people spending a total of 56 hours per week on research data management. And a huge chunk of that time is for naught. One of the most important explanations for that wasted time may be that too few organizations have the in-house skill to make good decisions about what data to collect, how to manage it, and how to use it. Having such skill could allow nonprofits to make better choices about data collection and better use what they do collect.
The nonprofit focus group prepared us to find a strong need for research and data capacity enhancement, and the survey confirms what they told us. When asked about their research data needs wish list, at least one-third of the organizations responded affirmatively to every choice on the list. More than half indicated a need for training in program evaluation, funding, computer programs in general, spreadsheets, research methods, and a tracking database. Just under half indicated a need for training in statistical analysis, geographic information systems analysis, and accounting and budget management. There is important overlap between this list, and a list of research and training needs compiled through the nonprofit focus group and interviews with area funders.
This research has attempted to gain an initial understanding of the research data practices and needs of nonprofit organizations. We have seen that organizations collect voluminous data on a wide variety of topics. Yet, much of that data goes unused. Furthermore, nonprofit staff have a difficult time identifying research data needs. So meager is their research methods background that it is entirely possible they can't imagine what to do with their existing data, let alone imagine what other data they might use. The result is a highly inefficient research data management system in nonprofits that wastes time collecting research data that are never used and not collecting research data that might be used. Getting rid of the wasted time and using it to make the most of the existing data could help a lot in nonprofit's grant applications, evaluation practices, and program planning. There are four important implications of this research for funders, nonprofit managers, and researchers.
1) Providing better research methods training for nonprofit staff and volunteers. The nonprofits clearly expressed their interest in research methods training. And it also seems clear that such training would allow nonprofits to both collect more useful data and put more data to use. As a consequence of this research, the Toledo Community Foundation funded a pilot research methods training sequence for a group of nonprofit organizations in the summer of 2005. We designed this particular research training series through a nonprofit core group process similar to what we used in the initial diagnostic research. This process created four modules of 2.5 hours each focusing on developing good research questions, using qualitative research, writing a good survey, and managing data using spreadsheet software. The first module, on developing good research questions, was targeted at helping nonprofits collect data that would be more usable. The module on qualitative research came from nonprofits' concerns that funders were only interested in quantitative changes, and most nonprofit programs were too small, too short, or too focused on quality of life issues to produce reliable quantitative findings. So while the nonprofit was not able to say that "x percent of children in the summer enrichment program experienced an improved home environment" in numbers, they were accumulating stories from parents talking about how their child seemed calmer or happier, and they wanted to know how to collect and present those stories in a convincing way. The survey module was motivated by their desire to get better at collecting quantitative data, and the spreadsheet module came from their desire to better at managing such data. The initial evaluations of the training were positive, but we do not know the extent to which the training resulted in actual changes to the participants' research data practices.
2) Educating funders on the importance of supporting nonprofit research and data management capacity. Even if we are able to improve the skills of nonprofit staff and volunteers in research design and data management and analysis, we still face the problem that there are so many hours in the day, and most nonprofit staff and volunteers are already running over their capacity just producing programs. Having better skills will be of no consequence if they do not have the time to deploy those skills. And most funders still do not provide adequate funding for the data collection and management that is needed for effective needs and asset assessments on the front end of a project and evaluation on the back end of a project. Part of this is due to a lack of knowledge on the part of funders, as my interviews with them indicated that they also were not well-trained in research methods. They certainly were not satisfied with the data that nonprofits provided them at either the grant proposal stage or the end-of-project evaluation stage, but they did not have enough expertise to make any recommendations for how to improve the situation. The funders are actually quite sympathetic to the research and data challenges faced by the nonprofits. But in Toledo, most of the funders are themselves stretched thin, and increasing funding for the research portion of one program could mean not funding another program at all. Finally, given that so much of the current research practice of nonprofit organizations is funder-driven, a frank discussion needs to occur among funders and nonprofits about the importance of doing evaluation research, including who should control the scope and method of such research, the lack of resources for doing such research, and the fears among nonprofits of presenting research that may show weaknesses in their funded programs. In addition, there needs to be broader discussion about evaluation models, particularly empowerment evaluation and participatory evaluation as forms of research which puts program improvement ahead of du jour funder-driven fetishes such as logic models.
3) Provide better stock databases for nonprofits to easily use. Even if we educate nonprofits on how to do better data collection and management, and even if we educate funders on the need to better support those efforts, nonprofits are likely to still face capacity challenges in collecting and using good data. One suggestion from both funders and the nonprofit core group was the provision of databases that nonprofits could easily access and use. Part of the overall project of which this research was a part involved the creation of a pilot neighborhood data system that could be accesses by any nonprofit to see census statistics on an individual neighborhood. Prior to this, neighborhood-based organizations had to compile their own census statistics, and most neighborhood boundaries varied significantly from both zip code boundaries and census tract-level boundaries. Most of the neighborhood-based organizations applying for annual City of Toledo funding used the database and found it useful for their proposals. Our pilot database was primitive compared to some of the other neighborhood data systems in other cities. There are many ways to build such databases, and they can include hand-collected data as well as government data provided by the census, the police department, and other government agencies. The important thing is involving end-users in the process of creating the dataset so that it can be of maximum use to maximum users (Stoecker, 2006).
4) Engage higher education students and faculty in nonprofit research data collection and management. The final way that we can begin addressing the capacity issues faced by nonprofit organizations in collecting and managing data is by better engaging higher education institutions in providing research and data support. As the service learning movement in higher education begins to support community-based research (Strand et al., 2003), the possibilities grow for directing the work of faculty and students to serving the research data needs of nonprofit organizations. While faculty may be necessary for careful research design, students who are appropriately screened and trained can do original data collection and data analysis. They can also convert paper records to electronic records, with appropriate privacy protections in place. This does not absolve nonprofits from seeking out education on research design and data management, as they will still have to hold faculty and students accountable for all the work they do. But it can dramatically expand the organization's capacity to collect and manage data.
This project has focused on the first layer of issues surrounding nonprofits' research data needs and management. We have found challenges in collecting, managing, and using research data, along with specific training and capacity building needs, in one urban area. Thinking more deeply, we have found challenges in even conceptualizing the data that nonprofits collect as research data. How different it is to see all of the information a nonprofit collects as providing an information base that can help the organization better design and implement programming, then as drudgery that must be endured for funders and government bureaucrats. Ultimately, the change we are seeking involves as much a change in how nonprofit staff, volunteers, and supporters think about data in a nonprofit setting as a change in what nonprofits do.
But we need to know much more. How similar is this situation in Toledo to other places where there may be more nonprofit managers with graduate degrees and other kinds of advanced training? Furthermore, what are the consequences of inefficient and low skilled research data practices for program outcomes and funding?
If I am correct that good research practice makes a real difference in receiving grants and developing effective programs, then we also need to develop locally appropriate interventions. In each locale, we need to ask a set of questions. What databases would be frequently used across a variety of organizations? What needs and assets data would be most frequently used? What might be the role of universities in providing or leveraging resources to support the training, infrastructure building, database development, and other related activities necessary to meet the research data challenges and fill the research data needs of nonprofits? How in-depth should research methods training be? Should it be scheduled over a long or short period of time?
This research is only a suggestion of the possible research data capacity issues facing nonprofits. We are at the cusp of a minimally understood capacity issue for nonprofit organizations. Hopefully this research will help catalyze further research to help support this increasingly important area of nonprofit research data capacity.
This research was supported by a grant from the University of Toledo Urban Affairs Center. Many thanks to Carlin Abbott, Morlon Harris, Bob Krompak, Stephen MacDonald, and Kelly Spivey, who were intimately involved in guiding this research. Also thanks to Patrick McGuire and the editor and anonymous reviewers of the Journal of Sociology & Social Welfare for comments on an earlier draft.
Appleton, L. (2003). The contribution of NPOs to family policy formulation in EU member and applicant atates. Voluntas: International Journal of Voluntary and Nonprofit Organizations. 14, 79-103
Balaswamy S. & Dabelko, H. I. (2002). Using a stakeholder participatory model in a community-wide service needs assessment of elderly residents: a case study. Journal of Community Practice. 10, 55-70.
Brewer, J. & Hunter, A. (1989). Multimethod research: A synthesis of styles. Newbury Park: Sage Publications.
Bryson, J. M. (1995). Strategic planning for public and nonprofit organizations. San Francisco: Jossey-Bass.
Castelloe, P., Watson, T., & White, C. (2002). Participatory change: an integrative approach to community practice. Journal of Community Practice. 10, 7-31.
Dattalo, P. (1998). Time series analysis: concepts and techniques for community practitioners. Journal of Community Practice. 5, 67-85.
Ebrahim, A. (2002). Information struggles: the role of information in the reproduction of NGO-funder relationships. Nonprofit and Voluntary Sector Quarterly. 31, 84-114.
Farmer R. & Walsh, J. (1999). Living Room Assessment. Journal of Community Practice. 6, 79-94.
Fetterman, D. (2002). Collaborative, participatory, and empowerment evaluation, http://www.stanford.edu/~davidf/ empowermentevaluation.html.
Fox, J. (2001). Vertically Integrated policy monitoring: a tool for civil society policy advocacy. Nonprofit and Voluntary Sector Quarterly. 30, 616-627.
Hager, M. A., Wilson S., Pollak, T. H., & Rooney, P. M. (2003). Response rates for mail surveys of nonprofit organizations: a review and empirical test. Nonprofit and Voluntary Sector Quarterly. 32, 252-267.
Hikmet, N. & Chen, S. K. (2003). An investigation into low mail survey response rates of information technology users in health care organizations. International Journal of Medical Informatics. 72, 29-34.
Houchin, S. & Nicholson, H. J. (2002). Holding ourselves accountable: managing by outcomes in girls incorporated. Nonprofit and Voluntary Sector Quarterly. 31, 271-277.
Iannotta, J. G. & Ross, J. L. (2004). Equality of opportunity and the importance of place: Summary of a workshop. Washington, DC: National Academy of Sciences. http://www.nap.edu/ books/0309084679/html/.
Kissane, R. J. & Gingerich, J. (2004). Do you see what I see? nonprofit and resident perceptions of urban neighborhood problems. Nonprofit and Voluntary Sector Quarterly. 33, 311-333.
Letts, C. W., Ryan, W.P., & Grossman, A. (1999). High performance nonprofit organizations: Managing upstream for greater impact. New York: John Wiley.
MacNamara, C. (1999). Basic guide to outcomes-based evaluation for nonprofit organizations with very limited resources, http:// www.mapnp.org/library/evaluatn/outcomes.htm.
Mancini, J. A., Marek, L. I., Byrne, R. A. W., & Huebner, A. J. (2004). Community-based program research: context, program readiness and evaluation usefulness. Journal of Community Practice. 12, 7-21.
Millett, R. (1996). Empowerment evaluation and the W. K. Kellogg Foundation. In D. M. Fetterman, S. J. Kaftarian, and A. Wandersman (Eds.) Empowerment evaluation: Knowledge and tools for self-assessment and accountability (pp. 65-76). Thousand Oaks: Sage.
Mook, L., Richmond, B. J., & Quarter, J. (2003). Integrated social accounting for nonprofits: a case from Canada. Voluntas: International Journal of Voluntary and Nonprofit Organizations. Volume 14, Number 3, 283-297.
Padilla, Y. C., Lein, L. & Cruz, M. (1999). Community-based research in policy planning. Journal of Community Practice. 6, 1-22.
Parackal, M. (2000). Internet-based & mail survey: a hybrid probabilistic survey approach, http://ausweb.scu.edu.au/ aw03/papers/parackal/paper.html.
Patton, M. Q. (1997). Utilization-focused evaluation: The new century text, 3e. Thousand Oaks: Sage.
Sheehan, K. (2001). E-mail survey response rates: a review. Journal of Computer-Mediated Communication. 6, http://www.ascusc.org/ jcmc/vol6/issue2/sheehan.html.
Sowa, J. E., Selden, S. C., & Sandfort, J. R. (2004). No longer unmeasurable? a multidimensional integrated model of nonprofit organizational effectiveness. Nonprofit and Voluntary Sector Quarterly. 33, 711-728.
Stoecker, R. (2005). Research Methods for Community Change: A Project-Based Approach. Thousand Oaks, CA: Sage Publications.
Stoecker, R. (2006). Neighborhood data systems: A best practices analysis. Community Development: Journal of the Community Development Society. 37, 109-122.
Strand, K., Marullo, S., Cutforth, N., Stoecker, R., Donohue, P. (2003). Community-Based Research and Higher Education: Principles and Practices. San Francisco: Jossey-Bass.
Theuvsen, L. (2004). Doing better while doing good: motivational aspects of pay-for-performance effectiveness in nonprofit organizations. Voluntas: International Journal of Voluntary and Nonprofit Organizations. 15, 117-136.
United Way of America. (2005). Outcome Measurement Resource Network. http://national.unitedway.org/outcomes/.
W. K. Kellogg Foundation. (2001). Logic model development guide. http: //www.wkkf.org/Programming/ResourceOverview. aspx?CID=281&ID=3669.
Wadsworth, Y. (1991). Everyday evaluation on the run. Melbourne: Action Research issues Association.
Wing, K. T. (2004). Assessing the effectiveness of capacity-building initiatives: seven issues for the field. Nonprofit and Voluntary Sector Quarterly. 33, 153-160.
Woller, G., & Parsons, G. R. (2002). Assessing the community economic impact of nongovernmental development organizations. Nonprofit and Voluntary Sector Quarterly. 31, 419-428.
Yun, G. W. (2000). Comparative response to a survey executed by post, e-mail, & web form. Journal of Computer-Mediated Communication, 6, http://www.ascusc.org/jcmc/vol6/issue1/yun.html.
University of Wisconsin
Department of Rural Sociology
Center for Community Economic Development
Table 1. Types of Nonprofits Responding Type of Nonprofit Number Reporting Youth related (i.e. mentorship or skill development) 25 Neighborhood development or community 18 building Housing or homeless services 16 Education, including GED or ESL 15 Food storage or distribution 15 Mental health 12 Drug or alcohol addiction prevention and 12 recovery Social justice advocacy, political causes 12 Culture, race, or ethnicity specific 10 Arts, ballet, theater or music related 8 Family transitions (i.e. adoption, foster care or 7 divorce) Emergency relief services (i.e. crime victim 7 support) Seniors * 7 Disability services (i.e. home repairs or 6 accessibility) Legal services 6 Religious * 6 Medical or reproductive services 5 * indicates write-in responses not included in original categories Table 2. Level of Analysis at Which Nonprofits Collect Data Level of Analysis Number reporting Individual level, i.e. children, youth or adults 72 Family level 44 Neighborhood /community level 23 City level 29 County level 28 Regional level, i.e. Northwest Ohio 19 State level 14 National level 9 Table 3. Topics on which Nonprofits Collect Data Number Topic Reporting Age 61 Sex 59 Race/ethnicity 52 Street/neighborhood-level address data 45 Family characteristics 35 Previous program participant 35 Employment status 34 Physical health conditions or disabilities 33 Education level 32 Client contact with other organizations 28 Mental health conditions 26 Funding resources 25 Transportation needs 22 Native or non-native English speaker 21 Criminal record 19 School system for children 17 Religious affiliation 15 Drug/alcohol treatment 13 Leadership skill 6 Table 4. Topics for which Nonprofits Collect, Use, and Need Data Number Number Number Topic Collecting Using Needing data data data Age 61 41 10 Sex 59 39 8 Race/ethnicity 52 34 12 Street/neighborhood-level address data 45 30 10 Family characteristics 35 20 13 Previous program participant 35 23 9 Employment status 34 20 (26) * 9 Physical health conditions or 33 21 (24) * 9 disabilities Education level 32 19 7 Client contact with other 28 17 11 organizations Mental health conditions 26 14 (17) * 6 Funding resources 25 15 (18) * 19 Transportation needs 22 17 11 Native or non-native English 21 12 8 speaker Criminal record 19 10 (13) * 7 School system for children 17 14 10 Religious affiliation 15 10 3 Drug/alcohol treatment 13 7 10 Leadership skill 6 3 7 * numbers in parentheses indicate organizations that use data beyond those that also collect it
|Printer friendly Cite/link Email Feedback|
|Publication:||Journal of Sociology & Social Welfare|
|Date:||Dec 1, 2007|
|Previous Article:||A qualitative study of letters to President Kennedy from persons with mental illness and their families: using the research poem in policy oriented...|
|Next Article:||Public and private sources of assistance for low-income households.|