Chapter V: bridge program evaluation and continuous improvement.
Developing a bridge program ideally is itself an evaluative process that seeks to identify the requirements of success for low-skilled adults at the next levels of education and employment, highlight gaps in existing education and employment programs that prevent low-skilled adults from advancing, and modify existing programs and services and add new ones to help bridge the gaps identified.
To evaluate a bridge program, it is essential to continue this process by staying abreast of the requirements for advancement to the next levels of education and employment, tracking how readily participants are in fact advancing, and using this information to modify the programs in ways that improve participants' outcomes.
The following steps will help evaluate and continuously improve the bridge program. The first set of steps involves collecting data needed to gauge gauge
In manufacturing and engineering, a device used to determine whether a dimension is larger or smaller than a reference standard. A snap gauge, for example, is formed like the letter C, with outer “go” and inner “not go” jaws, and is used to the performance of students in a bridge program and track their outcomes when they leave. The second set, under "formative evaluation Formative evaluation is a type of evaluation which has the purpose of improving programmes. It goes under other names such as developmental evaluation and implementation evaluation. ," outlines steps for evaluating how well the program is being implemented and how to improve program operations. The steps under "summative Adj. 1. summative - of or relating to a summation or produced by summation
additive - characterized or produced by addition; "an additive process" evaluation" describe how to tell whether the program is effective. The final, and in many ways the most important, steps involve using the findings from formative formative /for·ma·tive/ (for´mah-tiv) concerned in the origination and development of an organism, part, or tissue. and summative evaluations to improve program operation and outcomes over time.
Data Collection for Tracking Student Program Performance and Outcomes
* Secure informed consent from every participant upon entry to participate in an evaluation of the program. Figure 15, p. 85, presents a sample informed consent form. **
* Collect data on the demographic characteristics of every participant upon enrollment in the program. The data elements listed in table 5, p. 86, are the minimum set necessary to conduct a summative evaluation of the project. Programs may want to collect other sorts of information on participants based on funding requirements and objectives.
* Collect data on the performance and immediate outcomes of every participant. Table 6, p. 87, lists a recommended set of data elements.
* Interview participants, instructors, and staff around the midpoint mid·point
1. Mathematics The point of a line segment or curvilinear arc that divides it into two parts of the same length.
2. A position midway between two extremes. and at the end of each initial program cycle to identify what is working with the program, what is not, and how it can be improved. Every effort should be made to interview participants who drop out to find out why they are leaving and what can be done to prevent attrition Attrition
The reduction in staff and employees in a company through normal means, such as retirement and resignation. This is natural in any business and industry.
Notes: in the future.
* Regularly interview participants who have completed the bridge program, employers that have hired them, and faculty and support staff at the next level of education about the strengths and weaknesses of the program and ways it can be improved.
* Examine the performance of the program over time benchmarked against its own historical performance (or that of similar pre-existing programs) in terms of participant retention and completion rates, tested basic skills gains, job-placement rates and wage levels, and rates of advancement to higher levels of education and training.
* Document the main components of the program being evaluated, including program content, duration, and support services support services Psychology Non-health care-related ancillary services–eg, transportation, financial aid, support groups, homemaker services, respite services, and other services . It is important to accurately describe the eligibility criteria criteria (krītēr´ē),
n. for program entry and the process by which participants are recruited and selected. Describe any major changes in program design made during the course of each cycle. Interview program faculty and staff to better understand the reasons for these changes. (This will make it possible to more accurately compare and contrast outcomes across programs and over time.)
* Analyze an·a·lyze
1. To examine methodically by separating into parts and studying their interrelations.
2. To separate a chemical substance into its constituent elements to determine their nature or proportions.
3. the full costs of the program to all partners, including costs for staff, materials and supplies, equipment, and administrative overhead. ("Bridge Program Costs and Funding," pp. 66-74, discusses costs in detail.)
* Track the further education participation outcomes of leavers (completers and non-completers) from each program level using data from partner institutions, the state, or, if data are not available from those sources, from the National Student Clearinghouse, which collects data on enrollment in undergraduate programs across the country.
* Collect data on employment and earnings of program leavers (completers and non-completers) for at least eight quarters prior to and eight quarters following program participation using unemployment insurance wage records. ***
* Secure data on an appropriate comparison group with which to compare labor-market outcomes of program participants. Use a quasi-experimental design to compare the labor market labor market A place where labor is exchanged for wages; an LM is defined by geography, education and technical expertise, occupation, licensure or certification requirements, and job experience (and, if possible, further education) outcomes of pairs of program participants and comparison group members matched statistically based on demographic characteristics (age, gender, race, education, and prior earnings). ([dagger])
* Bring together program staff, instructors, and partners to study the data from the formative and summative evaluation activities, discuss what is working and what is not, diagnose diagnose /di·ag·nose/ (di´ag-nos) to identify or recognize a disease.
1. To distinguish or identify a disease by diagnosis.
2. the causes of the barriers that prevent participants from succeeding and progressing to the next level, and decide on ways to modify the program to promote participant advancement.
* Track the effectiveness of program modifications in improving outcomes.
* Shift resources (people, money, facilities) to support program strategies that prove effective in supporting participant advancement.
* Start a network of other bridge programs to share lessons learned, best practices, resources and tools, approaches to team development, and so on.
Figure 5: Sample Informed Consent Form
PERMISSION TO PARTICIPATE IN AN EVALUATION OF THIS TRAINING PROGRAM
The training program you are in is provided by [NAME OF PROVIDER] with funding from [NAME OF FUNDER(S)]. Both [SHORT NAME OF PROVIDER] and the funding agency(ies) would like to see how effective this program is in helping you and other participants get better jobs and pursue further education and training.
We would like to have your permission to have access to information on your employment, wages, and participation in education and training for use in evaluating this training program. This information is available from state agencies such as the [UNEMPLOYMENT DEPARTMENT AND HIGHER EDUCATION BOARD]. All of the information we collect on you from these agencies will not be shared with anyone else, and no one will ever be able to connect this information with you personally.
You do not have to participate in this evaluation. If you choose not to participate, [NAME OF PROVIDER] cannot use information on you to evaluate the program.
Please read the following statement and then sign and date below if you agree to give [NAME OF PROVIDER] access to information needed to evaluate the program.
I hereby give [NAME OF PROVIDER] permission to use the information I have provided and information collected by state agencies on my employment and further education once I leave this training program. I understand that this information will be used to improve the quality of this program for future students, and that I will not benefit directly. All information about me and my job and education outcomes will be kept strictly confidential and will be used for evaluation purposes only. I understand that I do not have to give this information if I do not want to.
PRINT FULL NAME
** "Informed consent" is a process in which the risks, benefits, and requirements of a research study are explained to persons invited to take part in a study. Before entering the study (in this case, a program evaluation Program evaluation is a formalized approach to studying and assessing projects, policies and program and determining if they 'work'. Program evaluation is used in government and the private sector and it's taught in numerous universities. ) a participant should sign an informed consent form, which should contain a description of the goals and methods of the study, the risks involved, and the steps that will be taken to protect participants' confidentiality. Participants can elect whether or not to take part in the study. If they opt out opt
intr.v. opt·ed, opt·ing, opts
To make a choice or decision: opted for early retirement; opted not to go. , information about them cannot be used.
*** By law, all employers with employees eligible for unemployment insurance (UI) are required to report to the state the quarterly earnings of every employee they employ in that state. States use this information to calculate unemployment insurance benefits. In most states, the employment security agency is responsible for issuing UI wage data. In most states, providers of education and training (or the state agencies that fund them) can gain access to UI wage data for purposes of evaluating and improving programs if they have the necessary consents from participants and sign a data-sharing agreement with the state agency that collects UI data.
([dagger]) Hollenbeck and Huang Huang (Chinese: 黃) is a Chinese surname. While Huang is the pinyin romanisation of the word, it may also be romanised as Wong, Vong, Bong, Ng, Uy, Wee, Oi, Oei or Ooi, Ong, Hwang, or Ung due to pronunciations of the word in (2003) used this method and data on Employment Service participants as a comparison group to assess the net impact of several workforce training programs in the Washington State workforce system.
Toni Henle Hen·le , Friedrich Gustav Jacob 1809-1885.
German anatomist whose works, including Handbuch der Rationellen Pathologie (1846-1852), integrated the study of pathology and physiology. , Women Employed Institute
Davis Jenkins Jen´kins
n. 1. A name of contempt for a flatterer of persons high in social or official life; as, the Jenkins employed by a newspaper s>. , University of Illinois at Chicago This article is about the University of Illinois at Chicago. For other uses, see University of Illinois at Chicago (disambiguation).
UIC participates in NCAA Division I Horizon League competition as the UIC Flames in several sports, most notably Basketball. Great Cities Institute
Whitney Smith Whitney Smith is a professional vexillologist, i.e., scholar of flags. The term vexillology, coined by him in 1958 in the article Flags of the Arab World, refers to the scholarly analysis of all aspects of flags. , Chicago Chicago, city, United States
Chicago (shĭkä`gō, shĭkô`gō), city (1990 pop. 2,783,726), seat of Cook co., NE Ill., on Lake Michigan; inc. 1837. Jobs Council
Table 5: Data Elements to Collect on Bridge Participants Upon Program Enrollment MEASURE DATA ELEMENTS Personal * Social security number Identifier Age * Date of birth Gender * Male or female Race/Ethnicity * White, African-American, Hispanic/Latino, Asian, or other Education * Earned high school diploma? (y/n) * Earned GED? (y/n) * Previously enrolled in job-training program? (y/n) --If yes, name of provider, program name, and date enrolled * Previously enrolled in at least one college-level class (y/n) --If yes, college name, program (or course) name, and date enrolled Recent Work * Currently employed? (y/n) History --If yes, name of current employer, job title, and one-sentence job description --If yes,hourly wage --If yes, hours per week currently working --Receive health benefits from employer? (y/n) * Number of months employed in the past 12 months Native Language * Native language is English? (y/n) Disability * Disability that would require special support during the program (y/n) --If yes, nature of disability Education and * Main reason for enrolling in program Career Goals * Main goal for employment in the next 12 months * Main goal for further education (beyond this program) in the next 12 months Tested Basic * Test reading and math levels Skills at Entry Table 6: Data Elements to Collect on Bridge Participant Performance and Initial Outcomes MEASURE DATA ELEMENTS Personal * Social security number Identifier Start Date * Date participant started in program Retention * Participant successfully completed the program (y/n) --If no, reason for leaving the program Tested Basic * Test reading and math levels, using the same Skills at instrument used upon enrollment Completion Job Placement * Employed for at least 30 days within 12 months of program completion? (y/n) --If yes, new job held during the program? (y/n) --If yes, start date --If yes, name and address of employer --If yes, job title and one-sentence job description --If yes, hourly wage --If yes, hours worked per week --If yes, receive health benefits? (y/n) GED Completion * Completed GED? (y/n/NA) Certification(s) * Earned a certification recognized by employers and/or educational institutions? (y/n/NA) --If yes, name of certification and issuing agency Further * Enrolled in further education and training within Education and 12 months of program completion? (y/n) Training --If yes, date enrolled --If yes, name of college or school --If yes, name of program and one-sentence description --If yes, participant's goal for the program