obÌåÓý

The IRS, through the joint efforts of the learning and education community, develops and delivers a variety of training courses designed to meet organizational goals and contribute to mission accomplishment. Effectively evaluating training helps personnel determine course improvement opportunities and training¡¯s overall contribution to business results.

The IRS follows established procedures outlined in the instructional design model known as the Training Development Quality Assurance System (TDQAS) to administer training evaluations. Specific procedures address the evaluation of learner reaction, learner achievement, job performance, and organizational impact. Adherence to TDQAS processes ensures procedural consistency in the overall quality of our training systems and products. IRM 6.410.1, Learning and Education Policy, and its subsections address the actions associated with the evaluation phase of TDQAS.

Laws: .

Executive Orders:

Regulations: at .

The IRS Human Capital Officer is the executive responsible for this IRM and overall Servicewide policy.

The HCO, OHRST, Transformation, Policy and Engagement (TPE), Policy Office (PO) is responsible for developing and publishing content in this IRM.

The HCO, OHRS, IRS University Office is responsible for the operation and administration for this IRM.

The Servicewide Training Evaluation Program (STEP) within IRS University is responsible for:

L&E is responsible for:

The VLM or CLS office is responsible for:

The program office gauges the effectiveness of this program based on feedback from customers and stakeholders and considers any statutory or regulatory changes. During review and publishing, in partnership with the policy office, IRM sections are revised, added or deleted based in part on this process.

Annual Review: An annual review is conducted to determine program effectiveness and feasibility.

The policy office is responsible for reviewing policies to ensure conformance with applicable laws and regulations.

The program office is responsible for implementing, monitoring and improving internal controls which are programs and procedures that ensure:

Annual program review requirements include the program office:

The following activities ensure program success:

The following Table provides a list of terms and definitions specific to this IRM.

This table lists commonly used acronyms and their definitions:

.

Internal Revenue Service (IRS) and National Treasury Employees Union (NTEU) National Agreement (Document 11678 ), Article 30.

Servicewide Training and Event Management (STEM) SharePoint site.

Level 1 training evaluation is the degree to which participants ?nd training favorable, engaging and relevant to their jobs. Its purpose is to assess the immediate reaction of learners to the training.

New World Kirkpatrick Model Level 1 dimensions include:

The Level 1 evaluation process enables business units to gather and interpret participants¡¯ reactions to the training they received.

Level 2 training evaluation is the degree to which participants acquire the intended knowledge, skills, attitude, con?dence and commitment based on their participation in training. Level 2 training evaluation can be either knowledge-based or performance-based.

Level 3 training evaluation is the degree to which participants apply what they learned during training when they are back on the job.

New World Kirkpatrick Model Level 3 training evaluation dimensions include:

The Level 3 training evaluation process enables business units to gather and interpret the level of impact the training had on participants behavior.

Level 4 training evaluation is the degree to which targeted outcomes occur as a result of training. It assesses the impact of the improvements in the trainees¡¯ performance on the mission of the organization. Business units must be able to identify and provide data on measures that have organizational impact.

New World Kirkpatrick Model Level 4 training evaluation dimensions include leading indicators which are short-term observations and measurements suggesting that critical behaviors are on track to create a positive impact on desired results. Examples of leading indicators include balanced measure ratings, compliance ratings, customer response ratings, etc.

The Level 4 evaluation process enables business units to gather and interpret the results the training had on the organization.

Level 1 training evaluations are automatically issued through the LMS when a course with an associated standardized online Level 1 training evaluation is recorded in an employee¡¯s learning history. The LMS automatically captures the responses.

Standardized paper Level 1 training evaluations are available for employees with restricted access to computers or as a reasonable accommodation.

The two standardized paper Level 1 training evaluation forms currently available are

The VLM or CLS office must manually administer the standardized paper Level 1 evaluations

The VLM or CLS office must send the results to the EMS vendor for input into the EMS

L&E, through discussion with their business unit customer(s), will annually determine the priority of training events included on their training plans. Refer to IRM 6.410.10 , Event Planning and Approval, for additional information.

L&E must obtain Level 1 training evaluation results from the LMS or the EMS.

L&E must analyze the results of required Level 1 training evaluation results for all Priority 1 and Priority 2 training courses in their purview within one year of initial delivery. Analysis must be performed at least every three years thereafter, while the training remains active. Analyze other Level 1 training evaluation results (e.g., for Priority 3 and Priority 4 training events) as resources allow.

L&E must compile and document Level 1 training evaluation results analysis in a consolidated report within the course file.

Documenting Level 1 training evaluation results analysis must include:

Additional documentation may include, but is not limited to:

Since Level 2 training evaluations measure the knowledge obtained from the subject matter of each individual course, there are no standardized forms or questions required to be used in all Level 2 training evaluations.

If a course has a Level 2 training evaluation requirement, L&E must develop a Level 2 evaluation and include in the course. Refer to IRM 6.410.9.4.2 for types of Level 2 evaluations.

L&E must develop and include in the course:

If using Level 2 training evaluation results for employee development or coaching purposes, methods or procedures for capturing the employee¡¯s individual Level 2 training evaluation results must be developed and included in the course.

Level 2 training evaluations may be administered online or by instructors.

If a training event has a Level 2 training evaluation requirement, the evaluation must be administered. Typically, Level 2 training evaluations are administered during course completion; however, there are instances when they might be administered after the course is completed.

Certain Level 2 evaluation data must be captured in the LMS for Servicewide analysis and reporting purposes. For instructor-led or blended training, the course instructor or business unit designee must:

Data captured on the IDC Form includes (as applicable):

To capture comprehensive Level 2 training evaluation results for more extensive analysis, L&E must communicate any additional reporting requirements to instructors.

If a Level 2 evaluation is required but not developed or administered, the instructor or business unit designee must complete the IDC Form. The form must include the reason the Level 2 was not administered. Upon completion, the instructor or designee must submit the form to VLM or CLS via the Event Support Request System (ESRS) for input into the LMS.

Limited Level 2 training evaluation results will be available from the LMS or EMS. Access additional Level 2 training evaluation results from data housed in alternative systems, in accordance with procedures established by the business unit. Use comprehensive results for analysis when available.

L&E organization must analyze results from Level 2 training evaluations for all training events with a Level 2 training evaluation requirement in their purview within one year of initial delivery and annually thereafter, while the training remains active.

If Level 2 training evaluation results are available for courses without a Level 2 training evaluation requirement, analyze results as resources allow.

L&E must compile and document Level 2 training evaluation results analysis in a consolidated report and maintain in the course file.

Documentation of Level 2 training evaluation results analysis must include:

Additional documentation may include, but is not limited to:

If a training event has a Level 3 training evaluation requirement, Level 3 training evaluations must be developed. This should be completed during the design and development phase of TDQAS, but may be completed later, if necessary.

Level 3 training evaluations must be developed in the EMS.

A corresponding participant and manager Level 3 training evaluations, referred to as an evaluation pair, must be developed simultaneously.

All Level 3 training evaluations must include the mandatory training transfer questions to provide results for Servicewide reporting. The mandatory training transfer questions are:

Level 3 evaluation instruments must be reviewed and approved by the STEP team prior to delivery.

Level 3 training evaluations must be conducted on all training events with a Level 3 training evaluation requirement.

Administer Level 3 training evaluation after completion of the training event and after employees have been given an opportunity to apply course content while back on the job. They are typically delivered one to six months after training but must be administered within nine months of the end of the training event.

If a required Level 3 training evaluation has not been administered within the typical delivery time frame, L&E must document the consolidated report within the course file with a scheduled administration date.

Once a Level 3 training evaluation pair is reviewed and approved in the EMS, evaluation links will be sent to L&E, who will then issue them to participants and their managers. Electronic Level 3 training evaluation responses are automatically captured in the EMS.

If a paper Level 3 training evaluation is used, a PDF version will be created and sent to L&E, along with a batch sheet. L&E office is responsible for:

L&E may delegate administering, compiling and submitting Level 3 paper evaluations; however, they must communicate the requirements to their designee and confirm completion.

Waiving the use of the EMS in its entirety for Level 3 training evaluations requires STEP written approval; complete and submit a Training Evaluation Waiver Request to detail the business need for an exception.

L&E must obtain Level 3 evaluation results from the EMS.

L&E must analyze results from Level 3 training evaluations for all training events with a Level 3 training evaluation requirement in their purview within one year of initial delivery and annually thereafter, while the training remains active.

If Level 3 evaluation results are available for courses without a Level 3 training evaluation requirement, analyze results as resources allow.

L&E must compile and document Level 3 training evaluation results analysis consolidated report within the course file.

Documentation of Level 3 training evaluation results analysis must include:

Additional documentation may include, but is not limited to: