Changes in the Examination Process: October 2022
What is the rationale underlying ACLAM’s reconfiguration of the certifying examination over the past several years?
ACLAM has continued to review the certification examination process regularly to ensure that the examination is fair, standardized, legal, defensible, valid, and effective in certifying laboratory animal veterinarians who have sufficient knowledge and skills to conduct the safe and effective practice of laboratory animal medicine. In 2005, a professional examination consultant firm evaluated our examination processes and presented 12 recommendations to the ACLAM Board of Directors (BOD). A committee was appointed to provide an implementation plan for the recommended changes to the examination process. Since that time, the process has continued and a series of changes have occurred based on recommendations of our consultants.
What have been the major steps taken to implement these changes?
The most significant items in reconfiguration of the examination since 2005 were (1) the need to closely align the certifying examination with the Role Delineation Document (RDD), in order to base the examination on the accepted list of skills and knowledge areas used by a certified laboratory animal veterinarian, and (2) the need to base the examination pass point on criterion-referenced standards through the use of a Standard Setting Study (SSS). This eliminates the previously used fixed 66% pass point using norm-referencing, which the consultant advised was not legally defensible.
When was the examination aligned with the RDD?
During 2006 the alignment of test items with the RDD was accomplished for the first time. The alignment of test items was accomplished with no more than 1% variance from the RDD task distribution.
Why do we need the role delineation document for the examination, and how is information on the RDD used in designing the examination?
Our consultant stated that ACLAM was vulnerable to criticism and perhaps litigation by not having a direct linkage between our examination and our task analysis (the RDD). The consultant further recommended that ACLAM should use the percentages expressed in the RDD without varying within the limits indicated in order to limit content sampling bias even further. The examination committee accomplishes this by using the Test Template, which is derived from the ACLAM Role Delineation Document. The proportion of questions relating to each domain in the Test Template is carefully assigned by the Examination Committee, and verified by the Examination Review Committee, prior to administration of the exam. The proportion of questions relating to specific animal species is similarly controlled and verified. Each year, the examination committee and examination review committee spend considerable time and effort properly assigning tasks from the Test Template to every test item in the ACLAM database, and ensuring that each examination is balanced to the RDD as required.
Are both the written and practical examinations individually balanced across the RDD?
Yes. Distribution of questions over both written and practical examinations based on the RDD was also recommended by our examination consultant. To accomplish this, a single examination containing both item types, fully aligned with specifications based on the RDD, and having a single, overall passing standard was recommended. This recommendation was approved by the BOD in 2006, and the previous allowance for a candidate to pass only one part of the examination was eliminated. The recommendations became fully effective in 2010 when written and practical questions became part of a single examination.
How does ACLAM ensure that the RDD remains relevant?
Since the first implementation of the RDD in 1997, the Certification Oversight Committee (COC) has been charged with reviewing the document annually, and convening an appropriately constituted review group every 5 years for a comprehensive review. Beginning in 2007, the process was improved by a role delineation survey that was developed with the assistance of an external consultant and administered to all ACLAM members in an electronic format for response. Surveys of the membership have been part of subsequent RDD comprehensive reviews, and yearly reevaluation by the COC has continued. Our specialty will continue to change over the years. A periodic review and update of the RDD ensures that our specialty is defined accurately and appropriately in the future. The most current update was completed in August of 2019.
Why the emphasis on the RDD?
The RDD defines the knowledge, skills, and abilities required to be considered an ACLAM-certified laboratory animal specialist. Originally, the RDD became the standardizing template for designing ACLAM-recognized training programs. Since that time, the RDD has evolved into an examination template against which the number and categories of questions included on the ACLAM certification examination must be aligned. The RDD defines our specialty, and will continue to be used in the future to balance our certification examination by tasks, knowledge areas, and species. The RDD also continues to provide the standard for our recognized training programs, and has been used for other purposes, such as formulating Continuing Education efforts and development of programs for the ACLAM Forum.
What is criterion referencing, and how does it apply to the ACLAM certification examination?
Criterion referencing means that determining the passing score on an examination is based on specific knowledge criteria relevant to our specialty. Criterion referencing cannot be based upon the comparative performance of the candidates. This is referred to as norm referencing. The use of norm referencing in examination processes is contrary to recommendations from the American Board of Veterinary Specialties and to recommendations of all members of the community of certification examination experts. General veterinary licensure examinations (such as NAVLE) eliminated norm referencing over 30 years ago as this process is simply not acceptable for high-stakes certification examinations in any field.
How is criterion referencing accomplished?
Criterion referencing is accomplished by performing a Standard Setting Study. The first ACLAM Standard Setting Study panel was convened in 2006. Multiple criteria were used to ensure that the panel represented all demographic groups in ACLAM. This panel met one week after the administration of the examination, in order to set a criterion-referenced passing score (cut score) for that examination. The consultant reviewed the scoring of the examination, conducted a statistical analysis of the resulting data, and made statistical recommendations for the passing score to the BOD. ACLAM has continued to perform standard setting studies at intervals as recommended by our consultants. Following the first standard setting session in 2006, standard setting studies have been repeated in 2010 and 2014.
What is a cut score (passing point)? Why was the old cut score of 66% not good enough?
A cut score is the score that determines pass or fail on the certifying examination. The historic standard of 66% was based on the judgment of diplomates based on their assessment of the performance of a minimally competent, but board-certifiable laboratory animal veterinarian. The problem with this method is that it was an entirely arbitrary standard that was reset each year and adjusted on given examinations, based on the performance of that year’s candidate group. ACLAM was not adequately controlling the difficulty level of the examination, and was relying on an unacceptable norm referencing procedure. The high-stakes certification decision of pass or fail must be based on specific criteria, using a standard that does not vary from year to year.
How is the cut score determined?
The Angoff Modified Technique is the method ACLAM has used to determine the passing standard. This procedure is the most widely employed and one of the most thoroughly researched in the field of licensure and certification testing. Using the Angoff Modified Technique, a group of ACLAM Diplomates, chosen to represent the spectrum of activities in our specialty serves as a subject matter expert group in a peer review session to evaluate the examination. The panel evaluates each item on the examination by estimating the percentage of the minimally acceptable candidate group that would answer the question correctly. These estimates, which the panelists make independently, are then combined across all panelists and questions, the result of which represented the minimally acceptable score.
How does ACLAM ensure that there are no major differences in examination difficulty and fairness from one year to another?
ACLAM uses a process called linear equating. This process uses accepted statistical methods to ensure that candidates are nether penalized nor rewarded when one version of a test is more or less difficult than another version.
In equating, the examination committee works with our consultants to review item analysis statistics and identify a set of high-quality questions for both the written and practical from the anchor versions of the test to be used as the equating pool for subsequent examinations. These questions are referred to as embedded equator items. In addition, each new version of the examination has questions that are unique to it because these items were not included in the previous examinations.
The next step in performing equating is to conduct an equating study, consistent with the Tucker model for linear equating for nonrandom groups and an embedded group of equator items. The calculation employs a complex linear regression formula to identify and control variance due to the differences in the ability of two groups taking the examination in different years, and to isolate the differences in item difficulty between the two versions of the examination. Based on these calculations, the consultant determines the score that candidates taking a new version would have achieved if they had taken the anchor version of the examinations.
What other changes have been made in the examination?
The number of items on the examination have been reduced and all questions are now written in multiple choice format. In order to use more questions with known performance characteristics, there are a number of exact duplicate questions used from one year to the next to use for linear equating statistics. Beginning in 2007, color photographs were printed in the practical test booklet rather than shown as projected images, and the written and practical components of the examination were combined into a single examination in 2010.
Why did ACLAM eliminate the examination review sessions which used to provide continuing education to Diplomates?
Our consultants advised us that the dual purpose of certification and continuing education is not appropriate for a high-stakes certification examination. This examination should serve a single purpose, to distinguish board-certified specialists from those who do not qualify. Failure to adhere to adequate security measures can result in sharing, intentionally or unintentionally, of test context, materials covered, and potentially even test items, making the examination unfair for others, especially those seeking certification by the experience route. In addition, the requirement for ACLAM members to produce large numbers of new test items every year strains the abilities of the examination committee to maintain a standard of high-quality test items, threatening the quality of the examination. Most importantly, the consultant advised that we were not in compliance with accepted (and legally defensible) psychometric standards if we continue to use the examination for multiple purposes. While releasing sample questions may be appropriate, proper examination security demands that only members of committees involved with preparing and reviewing the examinations for content validity be given access.
Why can’t training program directors review the examination each year? If they can’t see the examination, how do they know what to teach?
The examination should be used only for certification purposes, not as continuing education or formulation of training program curricula. Since the examination is directly related to the RDD, Diplomates do not need to see the entire examination to know what is covered. Training program directors, who are among the most experienced ACLAM Diplomates and have significant experience in preparing candidates, should continue to prepare those candidates in the ways that have a history of success, using source materials that closely align with the RDD.
Do changes in the examination lower or raise the bar to entry into ACLAM?
The changes in the examination and examination process were not implemented with a goal of either lowering or raising the bar for ACLAM certification; they were intended to ensure the examination is fair, consistent, and legally defensible. The consistency of future examinations is ensured with the process of equating, which provides a level playing field for all candidates.
Why has ACLAM selected a new examination consultant?
ACLAM used the services of Castle Worldwide for over ten years for overall programmatic advice and for yearly analysis of the examination and scoring. Beginning in 2014, the five-year contract with Castle was coming to a close and the ACLAM Board of Directors decided to begin an evaluation and competitive bidding process to select an examination consultant for the next several years. Castle Worldwide and several other examination consultant vendors were invited to submit bids, which were evaluated based on cost and alignment with the goals and philosophy of ACLAM regarding our examination processes. One important criterion for the future examination consultant was the consultant’s ability to manage the database of questions for ACLAM and effectively assist us in using the database to manage the assembly of our examination form each year. ACLAM eventually selected Schroeder Measurement Technologies (SMT) as our examination consultant vendor beginning in 2015, and the 2016 examination will be the first examination developed and scored with the assistance of SMT.
Questions about the Examination
Does the examination still contain questions covering recent journal articles?
Yes. The examination committee will continue to use items that are drawn from current literature and recent publications. New questions will also be added from standard laboratory animal resources (such as textbooks and regulatory documents) with the goal to focus all questions to the important topics of laboratory animal medicine as highlighted in the ACLAM Role Delineation Document. Depending on the significance of new publications in the field, new questions will certainly be included in future examinations to cover material appropriately linked to the RDD. Therefore, candidates should continue to study new publications with a critical eye on the RDD. However, there should not be questions based solely on new research published in a journal unless it is known that the new information is related to a knowledge area of the RDD.
What are the recommended study references?
The recommended list of references is available and is based on the major references used to make up 80% of the examination. Candidates should concentrate on traditional study materials, including the primary materials on the reference list. However, candidates should be aware that there is a large list of supplemental materials with topics that are very broad, but represent important sources of material for the laboratory animal veterinarian.
Is it true that every question must have two references?
In recent years, the Examination Committee has ensured that two references exist for answers to questions, unless the information contained within the first reference is widely accepted as important, confirmed knowledge within the field. The Examination Committee has worked to eliminate any questions covering material that is not yet widely accepted in the field.
Which journals are important to review for the examination?
The list of journals is included in the recommended list of references. The examination typically covers only recent volumes of journals that are published by December 31 of the year prior to the examination. Important topics or information contained in journals that are more than 5 years old usually will have been incorporated into newer texts or other references that appear on the primary materials list.
How does the examination reflect the information and skills needed to be a laboratory animal veterinarian?
ACLAM has worked very hard in recent years to ensure the fairness and relevance of the certifying examination by making certain that the RDD lists the tasks, knowledge, and skills required in the field and that the test items are carefully linked to the RDD. Diplomates should participate in the review and revision of the RDD to ensure that this document reflects the knowledge and skills required to be a laboratory animal veterinarian.
Could current ACLAM Diplomates pass the examination?
Yes. The person would have to study in order to pass the examination, since this a high-stakes examination designed to qualify entry-level specialists into the field. It is intentionally difficult in order to ensure the capabilities of any who hold the title of Diplomate. Also, a Diplomate may have concentrated on any of the many subspecialty areas in the field and might need to re-familiarize themselves with all topic areas in the RDD. However, every Diplomate has demonstrated the necessary entry-level knowledge and skills once in their career, and each continues to meet ACLAM’s approved standards for recertification in order to maintain Diplomate status.
Is the tested information necessary to be a successful laboratory animal veterinarian?
Yes. The RDD states: "The board certified laboratory animal medicine specialist is a veterinarian with specialized training and experience whose mission is to advance the humane care and responsible use of laboratory animals. This mission is fulfilled through the application of specific knowledge to safely and effectively perform the tasks listed in the RDD.
Can candidates get additional information regarding how they scored?
The Examination Committee sends reports of performance on groups of tasks from the RDD to failed candidates following the examination scoring. Detailed numeric score reports are not issued because the examination is used only to make pass/fail decisions. However, candidates who fail to attain certification receive information on performance based on RDD categories in order to help guide future studying.
What are the recent pass rates for the exam?
2023-63%
2022-47%
2021-58%
2020-52%
2019-54%
2018-46%
2017-50%
2016-47%
The passing point established via equating is determined post-administration for every examination. Actual candidate performance influences the passing point and thus it changes year to year.