Outcome-Based Evaluation Training Plan
The New York State Library (NYSL) proposes to develop a training package to help the staff of the State Library and library systems to build their capacity for using results-oriented evaluation in their State and Federal programs. Over the long term, the NYSL expects the training to spread to the systems' member libraries as well, and this plan includes a "train-the-trainer" component to assist the systems with that long-term goal.
During the process of evaluating the first five years of the LSTA program, the NYSL learned through its evaluation consultant and evaluation facilitator that there is a great need for outcome-based evaluation throughout the library community. Current data being collected is insufficient for measurement of the impact of LSTA on the library services of the State as these data are focused more on the activities of those providing the services than on user outcomes. More and more the numerous funding agencies (State government, Federal government, local government, private corporations) are asking library service providers to show the impact of their services. At the same time, the library professionals are not trained in how to do this. Even people with educational backgrounds find the demands of results-oriented evaluation confusing and difficult.
New York State's libraries and library systems are facing some difficult times over the next two to three years, as are many libraries in other states. The NYSL believes that it is even more important in hard times than good ones for librarians to be able to show evidence that libraries have value for their users and that programs libraries offer them affect their lives. This evidence can be presented to funders in justifying budgets. It can also be presented to the users themselves to help organize users as advocates for libraries.
Finally, the NYSL, looking ahead to the next five-year evaluation of LSTA, believes that the process of training librarians in outcome-based evaluation will improve that next major evaluation. Linking the results viewpoint to advocacy will also help the NYSL in implementing its new advocacy plan.
As a result of identifying weaknesses in its evaluation of programs in the first five-year evaluation of LSTA, the NYSL affirmed its intent to develop a comprehensive results-based approach in its new Five-Year Plan. It expects to adopt OBE methodology broadly for its work, not just for LSTA programs. The NYSL proposes a multi-stage project to train key participants in OBE as described in the following pages. There are eight stages. The timeline for the whole plan depends on setting the date for the workshop in Stage 1. Once the training in Stage 1 is complete, the NYSL expects to implement Stages 2-6 within a year to eighteen months. The remaining stages will probably take another two to three years.
Overall goal: To increase NYSL staff capacity to use an Outcome Based Evaluation model to measure outcomes of all New York's library programs, State and Federal.
Target audience: NYSL staff who profess an interest in OBE and are willing to apply their OBE knowledge to the programs they administer. An evaluation consultant working with the NYSL on the training package will also be included in this stage.
- NYSL staff will understand the components of OBE and be able to build a logic model.
- NYSL staff will be able to write good outcomes and indicators for at least one program they administer.
NYSL staff person: Sara McCain
Dates: Sometime in June, 2003 would be best as the rest of the summer will be taken up with New York's FY 2004 LSTA grant applications. Staff could be available from May 26 through June, except for June 2-4 when a number of staff will be attending a public library system conference.
The NYSL requests the following from the Federal Institute of Museum and Library Services:
- Trainers to conduct a two-day introduction to OBE--in Albany--for selected staff from the Division of Library Development (LD) and the Research Library (RL) and the evaluation consultant. The three LD staff who have already attended training in Washington will also participate in this training because they will be part of the LD project team that will implement Stages 2-8.
- Training materials with individual copies for each attendee.
- Any prerequisite homework assignments (readings, exercises, etc.) that attendees would have to complete or read before the training.
- Any follow-up activities that attendees would have to complete.
- Follow-up by either email or telephone to review attendees' completed assignments and advise changes and revisions.
Objective: NYSL OBE project team will develop a comprehensive OBE training package.
The elements of this package include:
- Rationale for training including how OBE provides a stronger basis for advocacy and helps make tough decisions in hard times.
- Pre-requisite component that participants will be expected to arrive at training having completed (advance homework).
- Instructor/trainer manual.
- Learner toolkit (Project team will review available toolkits or materials available from other states before developing something new.)
- Participants in the training in Stage 1 will serve as reviewers of all prototype materials.
- NYSL will ask IMLS to review the prototype training materials.
Objective: Test training package
Use the comprehensive training package to train library system staff with a focus on technology projects, such as technology training. This type of project was the focus of the Outcomes Logic Model prepared by LD staff for their two IMLS training events. The trainers for this stage may include the evaluation consultant, some NYSL staff, and contract trainers. The project team will provide some follow-up assistance to the library system staff to help them complete any assignments from the training workshop. They will also help them as the system staff begin implementation of OBE.
Objective: Conduct review of training
During the training events and follow-up calls, etc., the project team will:
- Capture applications issues that arise during training.
- Analyze obstacles and barriers to implementation.
- Capture ideas for revising training materials.
Objective: Provide advanced training and technical assistance
The NYSL will request from IMLS a one-day workshop in Albany for hands-on problem-solving of issues identified during Stage 4 for a small number of selected system and LD staff who can represent the range of issues. The NYSL may also request some follow-up technical assistance after this workshop by IMLS by email or telephone.
Objective: Revise training package based on testing experience and hands-on assistance from Stage 5.
Use the results of the problem-solving workshop and feedback on initial training to revise trainer and learner materials. Publish comprehensive training package after this stage is concluded.
Objective: Implement OBE throughout NYSL and library systems.
NYSL staff will use OBE for all guidelines for plans, applications and reports. Plans and reports will reflect the training provided to library system staff and will yield a rich resource for planning, decision-making, and advocacy.
Objective: Train trainers for member libraries' training.
Conduct training workshops of one trainer for each system and give each trainer a training template for conducting training for member libraries. Systems would now have trainers and tested training materials for training member libraries and other system staff.
Objective: Library systems will develop training plans for implementing OBE in their member libraries.
The systems will submit a training plan to the NYSL that outlines how they will implement the OBE training in their system. Library Development staff will provide technical assistance in refining the plans.
Objective: System staff will train member libraries' staff.
Library Development staff will provide technical assistance to the systems in carrying out this responsibility. They will also assist the system staff in evaluating the effectiveness of their training.
Go to Library Development home page