[영어논문] 수능 영어 시험에 대한 평가 - The evaluation of an English test

[ Caution ]본 Essay를 통해 여러분들이 수행해야 할 과제에 큰 도움이 되길 바라지만, 지나친 문장 도용과 표현은 논문 필터에 걸릴 수 있으니 주의하세요~~ ^^

 

 

1. Introduction

 

In the 1970’s, the Korean College Scholastic Ability Test was initiated to assess the students’ English ability as a foreign language by the Korean Institute for Curriculum Evaluation (KICE). Hence, many students who are preparing to take the test have studied in both public and private English schools with their objectives being to do well on this test. The English Tests in CSAT originated from the requirement to assess the students’ ability to study at university and to diagnose what they have learned for educational purposes. However, even though CSAT has crucial goals  in relation to assessing students’ foreign language achievements, CSAT has some difficulties that have beenidentified, such as low validity and some problems in its application.

 

Therefore the purpose of this paper is to analyse the overall description and characteristics of the English tests in the Korean College Scholastic Ability Test in 2007, and to suggest theoretical approaches to improve the test. This evaluation consists of an analysis of several items on the listening and reading test. Although the College Scholastic Ability Test has four main sections (Korean language, mathematics, foreign language, social studies and science), I will highlight and analyse only the English test, in three parts; the test of general information, the framework, and theoretical approach to the KCSAT

 

2. What is the CSAT?

 

 The CSAT is a test designed to assess the level of college scholastic ability. All students who want to enter a university must take the test at the same time. They all take the CSAT at the one time each year, in the middle of December. The English test in the College Scholastic Ability Test contains a variety of contents ranging from English conversation to cross-cultural communication, in order to assess the candidate’s understanding of linguistic knowledge and communicative skills. The Korean Institute of Curriculum & Evaluation (KICE) supervises the College Scholastic Ability Test (CSAT)’s development and implementation procedures. CSAT scoring and data analysis are also managed by the KICE. However most people cannot access the information about who designs this test every year because CSAT is a nationally significant test. The point of this is one of secrecy. The main purpose of the CSAT is to measure the candidates’ abilities as to how well they can understand English and communicate in terms of the macro-skills, including the testing of higher-order thinking abilities together with the students’ understanding of cross-disciplinary materials in the English test. According to the Korean Institute of Curriculum & Evaluation, clarification of the crucial aims is about the linguistic knowledge and English skills of the KCSAT as set out below.  

 

1.      Measuring ability to write English in the complete reading comprehension.

2.      Measuring speaking language skills rather than linguistic knowledge

3.      Measuring ability to understand daily English conversation

4.      Measuring reading ability to summarise and infer the main points

5.      Measuring ability to understand facts, to make inferences, and to apply language resources to problem-solving.

 

As follows from this list, CSAT has been involved in implementing these goals in public English instructions. Moreover, KICE has modified and promoted the development of CSAT in this form as being crucial for assessment.

 

2.1 Clientele

 

Year 12 students in Korea do not have enough free time as they are always concentrating on the Korean College Scholastic Ability Test (KCSAT). They are always struggling to improve their scores as this test is seen as the last means of upgrading their social position as it is a high-stakes test. Among the various tests - the Korean language, mathematics, social studies, sciences and vocational education in KCSAT- the English test section needs more student effort in the developing of their abilities. Korean students learn English language from primary school in the public curriculum. The English curriculum is changed and modified by Korean educators every 5 years. The current Korean, 7^th English Curriculum which was initiated 2003, is intended to educate the students in the curriculum objectives as follows:

 

To help communicate naturally about daily life and topics of general interest

To develop their communicative competence

To understand and utilize a variety of information from other countries

 

As I mentioned above, the objectives of the Korean English curriculum is to facilitate the students by gaining their interest and extending their English competence as well as linguistic knowledge. However the English test in the KCSAT, as the foreign language test, still has not been sufficiently linked to the English curriculum. Such a problematic point needs to be addressed by Korean educators.

 

2.2 Context for which test has been developed

 

English education in Korea has experienced educational policy changes over a long period, towards more effective and communicative language teaching. Therefore the English test in the context of KCSAT has been shaped and modified gradually according to these changes.  The context in which this test has been developed clearly bases the public education English instruction on its purposes. For example, the extension of English listening instruction in the English curriculum has caused listening items to be added to the KCSAT test, with an increase from 8 items to 17 items. This change of content can positively affect the candidates’ improvement incommunicative skills, which will be further mentioned in a discussion on wash back. 

 

2.3 Frame of reference

 

There is no information available for me to be able answer this question. Why not??

 

2.3 Scoring procedure

 

KCSAT which is an objective form of assessment consists of all multiple-choice questions so that this test only needs the candidates to provide simple answers. The candidates have to mark their answers on the Optical Mark Reader cards so it is easy to score and mark their results. Such a system has a high reliability for the scoring procedure. The English test in the KCSAT has 50 multiple-choice items, each item being scored at between 1 and 3 points, depending on the degree of difficulty. The English test in the KCSAT in 2007, has 100 points as the maximum raw score. When I analysed the KCSAT, item scores were distributed as shown below,

 

         1 point Questions x 3 items = 3 points

         2 point Questions x 44 items = 88 points

         3 point Questions x 3 items = 9 points

                                                                          Sum: 100 points

 

Finally, the candidates receive their scores at their schools around a month after the test.

 

3. Tests, Design, Content, Format and Procedure

 

3.1 Administrative and scoring requirements

 

According to the Korean Institute of Curriculum and Education, around 800,000 students per year, take this CSAT test in 950 locations in Korea. Hence, the Korean Institute of Curriculum & Evaluation, as the SCAT administration, needs to set up the testing centers and prepare many trained personnel for the test’s implementation every year. On the examination day, most public schools have been changed from common schools to test centers in order to provide sufficient test places. Almost all of the trained personnel consist of incumbent middle school teachers. In addition, the police officials control the traffic and any situational problems in order to ensure a completely satisfactory test environment. The CSAT test does not need much equipment for test performance but students should bring their OMR (Optical Marking Reader) pens as a test instrument. In the procedure of the test, for listening an audio system is required, hence, each place has to have installed a radio system. On the test day, the candidates are offered the best environmental conditions, which are under the control of the Korean government and the KICE, in terms of traffic, equipment and environmental conditions.

 

3.2 Test design

 

As the English test is a kind of proficiency test, it assesses the students’ English listening and reading ability by multiple-choice items. This item format is such that it is to mark the test for a huge number of test candidates, and it also has good reliability. The English test has a disadvantage in that it does not properly assessing English ability in a diverse way for the reason that the test consists of only multiple choice items. The foreign language (English) test in the CSAT consists of 50 items including 17 listening items and students sit this test for 70 minutes. The English listening test starts from 13:20 for about 20 minutes, and the reading test starts after the listening test. Although all the scheduling, data, and scoring of CSAT are administered by the KICE, I could not find the information about who designed this test. Because this test is also one of the nationally crucial tests, the KICE keeps these details confidential.

 

3.3 Instruction/Rubric

 

The KCSAT test takers are assessed by the multiple-choice items by following each rubric. The rubric shows the candidates the way to answer questions clearly in the Korean language. Through the concise and clear Korean instructions, the candidates do not have any problems answering the questions in both listening and reading section. The rubric for the listening and reading questions ensures that they make good sense of what the questions require and provides valuable information about the questions.

 

3.4 Item content & Format

 

Listening Items

Listening, which is a complicated process, is different from the other macro skills. Therefore to measure the listening skills of the candidates, diverse approaches are needed in the test. The listening test of the KCSAT is aimed at measuring the test-takers’ micro and macro listening abilities such as the range of the vocabulary, grammatical knowledge, and understanding of the overall content and comprehension of a conversation. Hence the listening part of KCSAT involves various topics and content such as the analysis of a graph, descriptions, inference questions and so on. For the listening test,  the emphasis is on the measuring of listening comprehension ability and the global listening comprehension skills. This test therefore requires the candidates to find the proper information and to solve the situational problems for the various topics being used.

 

Reading Comprehension Items

The 33 reading question items in the KSCAT include a variety of topics covering natural science, practical passages, tasks and cultural content. Moreover, there are many different types of genre such as reports, discourse, and narrative. The reading test is presented in many test item formats including cloze, true and false, matching information and short answer. However, in the same way as for the listening test, all the answers should be provided through multiple-choice items. Generally, the length of each text in the reading test ranges from one paragraph to three paragraphs. There are 33 reading questions which should be answered by the test takers in 40 minutes, which is not enough time to complete their test. Hence this reading task requires the ability to understand each task as quickly as possible.

 

4. Overall characteristics

 

4.1 Validity

 

Weir (1993) notes that validity is the first factor for consideration in a test. Validity can show the degree to which a test measures what it is supposed to measure. Therefore validity should be analysed and considered to decide whether a test is valid or not.

 

Hughes (1989) defined each characteristic of validity as follows;

Face validity: if it looks as if it measures what it is supposed to measure, this test can be understood as having high face validity. Therefore the KCSAT goals should be taken into account for the demonstration of validity. KCSAT has as its purpose, to measure the students’ linguistic ability and communicative skills. According to KICE, the goals of the KCSAT test are as below,

 

Measuring ability to understand facts, to make inferences, and to apply language resources to problem-solving

Measuring ability to understand daily English conversation

Measuring reading ability to summarise and infer the main points

 

KCSAT, which involves 17 listening items and 33 reading comprehension items can be shown to have high face validity. This test measures the students’ linguistic knowledge such as their range of vocabulary, pronunciation and grammar as well as general understanding of reading passages relating to genre and situational information. In the listening task, students are given 17 situational listening passages, so they are required to find the correct answers with their linguistic ability. For the reading task, which consists of 33 reading comprehension questions, it is especially expected that students should use their linguistic knowledge and ability in reading comprehension in a diversity of functional genres. Such characteristics of the test require students to demonstrate their overall linguistic knowledge according to the test’s objectives.

 

The various types of genre and functional passages in both the listening and reading tests can be seen to represent the relevant test objectives. The listening and reading questions, which are based on linguistic knowledge for diverse topics, also afford to assess the students’ knowledge of the general structure of English. Also these test constructs elicit not only candidates’ linguistic knowledge but also prior knowledge. Therefore this test is likely to measure test specifications and structures appropriately.        

 

 In the reading task, most tasks require the candidates to understand the whole content and the concepts in the reading sub-test. The candidates need to be able to understand of the overall passage and structures from their linguistic knowledge. However, some reading questions just require the students’ grammar knowledge as we can see in questions number 22 and 23. These questions do not require students to demonstrate their reading comprehension ability but just see if students can infer the answers from their grammar knowledge only. These problematic questions show low construct validity. Hence these problematic questions should be of concern for test designers. Also the imbalance of questions between 17 listening and 33 reading items cannot be understood as there being a good test to measure the students’ overall foreign language ability.  

 

4.2 Authenticity & Interactiveness

Bachman (1996), who defined authenticity as being the quality of the relationship between features of the test and real life, states that authenticity is to be analysed from two aspects; the real life and situational. In the case of the English test in KCSAT, there are only some questions showing authenticity. Question 11 needs the candidates to understand an authentic table showing data. This task shows a proper approach to real life performance. Furthermore, questions 5 and 8 require the understanding of situational information. These tasks attempt to provide authentic interaction between the test takers and the test performance. In the listening part, most questions require students to understand the communicative situation in the target world. These questions can be seen to correspond to a real ability. Moreover, the diverse situational and functional topics in the reading test prompt the candidates to be familiar with the characteristics of the test and to be making decisions on real tasks.

 

As I have already mentioned above, authenticity provides an implication for illustrating the relationship between the test and target language performance. Such authenticity also can be associated with Interactiveness. According to Bachman and Palmer (1996, p25), “Interactiveness is the extent and type of involvement of the test taker’s individual characteristics in accomplishing a test task”. In this test, candidates would not use their special knowledge or judgement to find the correct answers in the listening and reading tests, but they use their linguistic knowledge.    

 

4.3 Reliability

 

 For both the listening and reading tests, which consist of multiple choice items, examiners can give reliable marks to the candidates in the KCSAT. The scoring process does not require the rater’s judgement as to how to interpret the candidates’performances. The KCSAT, which can be understood as an objective test, marks the candidates’ responses according to precise answers. Each question requires the candidates to mark only one answer so that they only choose one item from several choices.

 

However, each question is supported by3 different points ranging from 1 to 3. The different points rely on the degree of how difficult the question is. The candidates can be marked by their answers from among 3 different levels of questions. Lastly I could not access the information about the determination how the classification of the questions is assigned according to degree of difficulty.   

 

4.4 Practicality

 

Weir (1993) points out that practicality is the consideration of the amount of time spent, scoring and processing the result as well as the cost and resources of test administration. For the KCSAT, as a national test, there has been consideration about the administration of the test in terms of such diverse aspects as the test places, time, and trained personnel. However the KICE (Korean Institute of Curriculum and Evaluation), which is authorized by the Korean government to design and administer the KCSAT has administered the financial or test conditions for making the test, practical. Likewise the test resources are always developed and designed by educators every year so that the test can be performed according to their schedule. Hence KCSAT has been carried out every year without any problematic practical aspects. 

 

4.5 Impact / Wash-back Effect

 

The effect of tests on teaching and learning is known as a test’s positive/negative wachback. As has been mentioned above, KCSAT as a national test has strongly affected a change in teaching and student learning in the public curriculum inform both positive and negative aspects. From the negative aspect, it is well know that the KCSAT leads many students to ignore their studies in the public curriculum because of the low interrelation between the KCSAT and the current English curriculum. Furthermore, such a situation can also affect the teaching and learning styles. Although the purpose of the KCSAT is to measure the candidates’ integrated linguistic knowledge and communicative English skills, teachers and students have been placing emphasis on the test skills, for example, of how to find the key terms or catch onto hints in the test without improvement of their communicative skills and linguistic knowledge. On the one hand, the first KCSAT test had had only 8 listening questions up until 1994. But as the number of listening questions has been increased from 8 to 17 items in the KCSAT, English teachers have started to teach the listening comprehension as a communicative language skill in their classes and students have tried to improve their listening ability. With slight changes in the KCSAT, students could have had many opportunities to engage in communicative instruction.

 

 5. Conclusion

 

The KCSAT has been administered as a high stakes test for 13 years in Korea. This test has been carried out with some problematic aspects under the administration of KICE, so that it has been modified toward being an appropriate test by providing a high degree of practicality and by developing the content. Moreover this test has brought about a positive impact on teaching and learning in achieving its objectives in the changed form of the test. Lastly this test can be shown to have high reliability in grading the candidates’ results.

 

However, although this reliable test has had functional or conceptual changes so the test still has a few problematic points as I have mentioned previously. The low construct and content validity cannot appropriately measure students’ language ability within the scope of the test objectives. Furthermore, as oral speaking does not exist in this test, it cannot be said to measure the students’ communicative skills. As many educators have suggested, the speaking and writing abilities, as productive skills should be assessed for more precise measurement in foreign language proficiency and achievement. 

 

References

 

Bachman, L. & Palmer, A. (1996) Language testing in practice Oxford: Oxford University Press

 

Davies, A. (1990) Principles of Language Testing. Oxford: Basil Blackwell

 

Hughes, A. (1989) Testing for Language Teachers Cambridge: Cambridge University Press

 

McNamara, T. (2000). Language Testing. New York, Oxford University.

 

 

Weir, C. (1993). Issues in Language testing in “Understanding and developing language tests”, Prentice Hall, P1-29

 

 

Web sites

 

The Korean Institute Curriculum and Evaluation http://www.kice.re.kr/kice/index