以常態混組模型討論書籤標準設定法對英語聽讀基本能力標準設定有效性之輻合證據

Abstract

本研究旨在探討書籤標準設定法（簡稱書籤標定法）應用於2005年台灣學生學習成就資料庫（TASA, Taiwan Assessment of Student Achievement）中之「英語文學習成就評量」的英語聽讀基本能力標準設定（Standard Setting）的判斷歷程，以及所判斷結果之輻合證據的有效性。研究樣本有10101名小六學生，分層抽樣自全台各地，施測方式採取平衡不完全區塊設計 （Balanced Incomplete Block Design），將70個聽與讀的題目設計成6個40題組成的題本。書籤標定法標準設定會議由十三位專家（七位學科內容教授、三位測驗教授、以及三位英語 專家教師）組成，會中共識設一切分點（Ө=-0.57），通過組學生佔全體72.9%。本研究利用常態混組模型（normal mixture model）之計量模式結果作為書籤標定法有效程度的輻合證據（convergent evidence），其估計的切分點（Ө=-0.40）與專家設定的分類結果達到 .87之Kappa一致性。文末研究者提出實務使用上以及理論上的討論議題。

This study investigated convergent validity of the bookmark standard setting method used for English reading and listening ability. The data set was obtained from 2005 Taiwan Assessment of Student Achievement (TASA) data bank. A total of 10101 sixth graders from different areas of Taiwan were cluster sampled and tested by a 40-item scale. The scale was developed through balanced incomplete block design out of 70 items. Thirteen experts formed bookmark standard setting seminar. Among them, 7 were university professors in the English-as-a-Foreign-Language (EFL) field, 3 were professors in measurement, and 3 were elementary school English master teachers. They attained the consensus of cut score Ө=-.57 with 72.7% of students were classified as passed. The result from normal mixture model (Ө=-.40) was consistent with the result from the bookmark standard setting method with classification consistency Kappa=.87, indicating convergent validity evidence. In line with this finding, issues on how to implement bookmark standard setting approach were further explored and discussed.