Our Most Crucial Task
The development and maintenance of certification exams is one of the most important tasks undertaken by the ACAC. Several important stages are involved in the creation of a credible certification exam, including skills development, item development and assessment, exam construction and review and cut score specification. In order to ensure that Council exams accurately represent their rapidly changing fields, our certification boards guide each exam through all the stages of the process.
Skills Development:
The certification board authorizes and undertakes a job analysis study to define the job-related activities, knowledge and skills required of a competent field professional. The board solicits the assistance of industry experts in order to ensure that a broad range of stakeholders is represented in the process. The board then generates (and publishes on the website) formal documentation of the knowledge and skills required for its certification. The board periodically solicits the assistance of a random sample of industry practitioners to validate the exam topic documentation to ensure that exam specifications are consistent with current industry practice.
Item Development:
The certification board oversees the development of all examination items. The board employs or retains experienced item writers or solicits the assistance of certified personnel. Item writers are given specific instructions as to the format and content requirements for each item:
§ Items must be coded to specific skills or knowledge base documents.
§ Items must be relevant to the skill or knowledge base.
§ Items must be classified according to level of difficulty.
Exams (and items) are designed to separate candidates into two distinct groups: candidates whose knowledge and skill levels are deemed acceptable for certification and candidates whose level of competence falls below the minimum required for certification.
Functioning as an item review committee, the board uses a formal content validity scale to evaluate each item’s relation to the content area. Items deemed acceptable by the board at this point are re-checked for grammar and style.
Item Assessment:
The certification board conducts an annual item assessment survey, in which statistical information is gathered and analyzed for the items in each examination. Test items with the following undesirable statistical characteristics are flagged for replacement:
§ Items whose difficulty indices (expressed as a proportion of candidates answering the item incorrectly) are too high.
§ Items whose discrimination indices are either too low or negative. Item discrimination index is expressed as a point biserial correlation coefficient, which measures the relationship between a candidate’s answer on a particular item and his score on the whole test. It tells how well an item distinguishes between high and low scoring candidates.
Examination Construction and Review:
The certification board oversees the construction of each exam and annually reviews each exam for reliability. This process includes a number of important steps:
§ The board authorizes the creation of item banks arranged by content area.
§ The board determines exam specifications based on Skills and Knowledge Base documentation. Such specifications include the number of topics covered by the exam, the number of items required for each topic, the number of dummy items included in each exam for purposes of pretest analysis and the rotation of items on and off of examinations based on statistical item assessment.
§ The board authorizes the assembly of a draft exam and reviews it for content coverage, item redundancy and accuracy of the answer key.
§ The board conducts annual reviews of the reliability of each examination to ensure mistake-free processing and internal consistency. An ongoing relationship with Kryterion is maintained for computer-based exam delivery and scoring. Randomization of items eliminates the need for several forms of the same examination and ensures consistency of reliability indices for each exam administration. The board employs the Kuder Richardson Formula #20 (KR-20) to establish an internal consistency coefficient for each examination. This coefficient measures the homogeneity of the content area tested by the exam. Examinations whose KR-20 coefficient is below 0.8 are flagged for further review and modification.
Examination Cut Score Specification:
The certification board uses the Modified Angoff Method to determine a criterion-referenced passing score for each examination. According to this method, certification board members discuss the characteristics which distinguish a minimally qualified certificant from an individual who should not be certified and develop a profile of the “borderline” candidate. Independently, each board member asks the Angoff question of each item on the examination: “What percentage of borderline candidates WILL answer this question correctly?” Board members’ answers for each item are averaged. The average of the Angoff scores for each item on the exam is the passing score.