結合作答反應和反應時間對CAT情境中外洩試題的偵測效果

Abstract

電腦化適性測驗（computerized adaptive testing, CAT）能根據受試者能力選擇相應難度的試題，有效提升測驗效率。然而，隨著試題反覆施測，試題外洩的風險也隨之增加，進而威脅到測驗的信度、效度與公平性。雖然過去已有文獻利用序列序識別外洩試題，但主要依賴作答反應，較少結合反應時間資訊，對外洩試題的偵測效果仍有限。此外，現有研究聚焦在二元計分試題，並無多元計分的外洩試題偵測方法。本研究旨在發展一種結合作答反應和反應時間資料之加權統計指標，並納入序列程序中，以提升外洩試題的偵測效能。此外，我們亦將所提的方法拓展至多元計分試題，以便未來用在多元計分的電腦化適性測驗中，進行外洩試題偵測。本研究設計了兩項模擬研究與一項實徵研究。模擬研究藉由操弄多種模擬情境，比較不同偵測外洩試題的方法。結果顯示，相較於僅使用作答反應、僅使用反應時間，或是採用多變量分析方法，本研究所提出的加權統計指標，在多數模擬條件下具有較高的正確檢測率與較低的延滯人數，特別在試題預知考生比例較高時的效果更加顯著。實徵研究亦證實本方法具備實務可行性，能有辨識真實測驗中的疑似外洩試題。Computerized adaptive testing (CAT) enhances testing efficiency by selecting items that match an examinee’s ability. However, as items are repeatedly administered, the risk of item compromise increases, threatening the reliability, validity, and fairness of the test. Although previous studies have proposed sequential procedures to detect compromised items, most of them were based on response accuracy without incorporating additional information, such as response time data, thereby reducing detection performance. In addition, previous research has focused primarily on dichotomous items, leaving a lack of detection methods for compromised polytomous items.This study aims to develop a novel weighted statistical index that combines both response accuracy and response time within a sequential detection framework to improve the identification of compromised items. Furthermore, the proposed method is extended to polytomous items to enable its application in CAT with polytomous items.The research includes two simulation studies and one empirical study. The simulation studies manipulate various conditions to compare the performance of different detection methods. Results show that, compared with using response accuracy alone, response time alone, or traditional multivariate approaches, the proposed index achieves higher true positive rate and shorter lag under most simulation conditions, especially when the proportion of examinees with item preknowledge is high. The empirical study further demonstrates the practical utility of the method in identifying suspected compromised items in real CAT data.