1. General information
1.1 Topic and Target Group: This topic of assessment is based on National Information on Tourism as a main topic. This topic can provide diverse information to the students: cultural and national information as well as for travel.
The students, aged 15, are in an Asian middle school, EFL (English as a Foreign Language) class. The students are studying at an intermediate level. They study English four times a week in the public curriculum. I am assuming that they have conversational communicative skills and grammar knowledge.
1.2 Scope and Sequence Chart
|
Overview of the Unit |
Link/s to other curriculum areas |
General Communicative Goal/s |
Cultural Focus |
General description of tasks |
|
In this unit, we will prepare a series of classes to let students know how to go travelling overseas, independently. Students will gain abasic knowledge of travelling in this unit. |
Tourism
|
Reading, listening, speaking, and writing |
Cross-cultural background |
(On the next page) |
|
Lesson |
Aims |
Language Focus |
Multimedia |
Materials/Resources |
|
1 |
World facts |
Reading and context comprehension |
A projector and computers |
World map, handout, pictures |
|
2 |
Travel |
Listening and communication |
A CD/Tapes player and a projector |
Authentic-related materials, pictures |
|
3 |
Shopping |
Speaking and communication |
Computers and a projector |
Authentic materials (money, goods, etc.) and pictures of authentic materials |
1.3 Descriptions of procedure (class and assessment): The whole integrated task will be completed in three stages, and the grades of the first and the third stages will be counted in the final grades.
Stage1:
|
1 |
England |
6 |
China |
|
2 |
China |
7 |
America |
|
3 |
China |
8 |
Australian |
|
4 |
Australia |
9 |
Australia |
|
5 |
Australia |
10 |
China |
Each
student will have 15 minutes to read two items of tourism reading materials* (Appendix 1) that were given out by the teacher. After reading the materials, each student will have another 15minutes to complete a gap-filling assessment* (Appendix 2) which aims to help students to understand the content of the reading materials and to assess their reading comprehension ability. Students have to do the assessment individually, and discussion is not permitted during the assessment procedure. There are total 10 questions on this test (10 points/question), and the grades of Stage1 weight is 30 % of the final grades. This task is an objective test for which the following list indicates the correct answers used in marking the students’ responses.
Stage 2:
More topic-related reading materials will be delivered by the teacher after the assessment. Students, who have the same reading materials, form a group and a period of 20 minutes preparation time is permitted. Each group has 15 minutes to do a brief group presentation, and note-taking is compulsory for each student during the process of presentation. The presentation process and the note-taking content will not be marked.
Stage 3:
Instructions: Pretend you are a journalist working for a travel magazine. Your boss has asked you to write an information sheet or handout about travelling to a particular country. You should also provide background information of the chosen travel destination. Reading material sheets about different countries will be provided. You are required to write the information based on the reading material given and your notes will be taken up from the in-class presentation activity. Further research of the selected country is acceptable but copying directly from reading materials and web pages is forbidden. (The grades for stage 3 are worth 70% of the final grades.)
· Make sure your targeted readers will be the general public that who read the magazine.
· Make sure that your report involves at least 5 pieces of information about the chosen country.
· Length: 250 to 300 words
Before you start:
What sort of information do you need to focus on?
Who are you writing for?
1.4 Test specifications: This writing test will be performed during the ongoing class process. More than 15 students will take this assessment in the same location. This writing assessment takes about one hour including administration time. For this writing assessment, which is designed as a report which is based on tourism, students should follow the writing format in the specific genre for general writing and description. They can not only use a piece of paper and a pen as the test instrument but also their notes taken from in the class presentation activity. For the structure of the assessment, this writing assessment requires students to perform a situational formal writing task. Lastly they will get their grades for their performance after the scoring procedure which will be explained in the next section.
1.5 Scoring scales/criteria
This writing assessment as a form of productive skill assessment is to be judged against rating scales in order to make for a more reliable test and to provide sufficient guidelines to the students. Hence the task rubrics should be designed precisely before students are assessed on their performances. The rubric of this assessment focuses on measuring a stated objective performance and uses a range to rate performance from 0 to 10. Moreover it contains specific performance characteristics arranged in levels indicating the degree to which a standard has been met, as follows below.
|
Score |
Task rubrics description |
|
10 |
1. Successfully selects the important information from the reading
2. Accurately presents this information in relation to the relevant information presented in the reading.
3. Well organized and developed
4. Displays progression and unity
5. Occasional structural errors do not influence the correct understanding of the text. |
|
8 |
1. Good selection of the important information from the reading
2. Presenting this information in relation to the relevant information in the reading, but it may have minor inaccuracies.
3. Well organized and developed
4. Displays progression and unity, though it may contain occasional redundancies.
5. Occasional noticeable minor errors in structure. |
|
6 |
1. The overall response is definitely oriented to the task, but it conveys vague, unclear, or a somewhat imprecise connection of the points made in the reading.
2. The response may omit one major point.
3. Some key points may be incomplete, inaccurate, or imprecise.
4. May demonstrate inconsistent facility in sentence formation and word choice that may result in lack of clarity.
5. Error of grammar usage that causes misunderstanding of the text and meanings. |
|
4 |
1. The response significantly misrepresents the reading.
2. Inadequate organization or connection of ideas.
3. A noticeably inappropriate choice of words or forms.
4. Obvious errors in sentence structure or usage. |
|
2 |
1. The response just provides little or no meaningful content from the reading
2. Serious and frequent errors in sentence structure or usage
3. The response completely omits the overall connection between the reading and writing. |
|
0 |
1. The response at this level only copies sentences from the reading
2. The content is not connected with the reading
3. Wrong grammar structure usage |
This table is based on Hughes’s book (1993, p.95)
2. Theoretical approach to the characteristics of writing assessment
It is well known that reading and grammar translation have been the basis of the main curriculum of the traditional foreign language classes in EFL countries. As to these characteristics of EFL class, many students have faced many problematic points like a lack of ability to write and speak English as the productive skills. Therefore, many EFL classes have emphasised writing instruction in the Communicative Language Teaching (CLT). The writing instruction or task for EFL students does not just bring about the benefit of linguistic improvement but also helps them to extend their cultural or social knowledge by dealing with diverse topics in the context of specific objectives. Actually a writing task demands students to have a variety of linguistic knowledge: a range of vocabulary, grammar and a prior knowledge about writing styles including the difference between a formal and informal writing format. Moreover the writing can provide skills in functional type of writing and, a wide range of academic fields such as business, culture, history and science while adopting a step-by-step approach to teaching how to write in diverse genres. Hence the writing instruction should be involved in a foreign language class.
Along with these characteristics of writing instruction, the writing tasks, which can be associated with the class content, also should be performed for achieving other specific goals. As they look at the writing assessment above, for example, students can judge their linguistic ability and general skills such as creating ideas, writing structure and using prior knowledge by using the feedback from writing assessment as a peer assessment. Moreover the writing assessment can provide a reason to read and understand the specific target language used in their writing. Such characteristics of writing assessment can help students and instructors in their teaching and learning of the target language and has its own objectives.
2.1 Reliability
“To be valid a test must provide consistently accurate measurements. It must therefore be reliable… if the scoring of a test is not reliable, then the test results cannot be reliable either.” (Hughes, 1993, p.42)
This test will be administered to the same group of individuals on the one occasion so that the rating score should be considered based on clarification provided by the criteria. The reliability of this writing assessment requires a teacher’s judgement as a rater when they do the scoring compared with the scoring of short answers. Hence the teacher has to consider how reliable they are in making judgements to measure students’ writing tasks and how to process the scoring of the tasks in order to have a reliable test. In this writing task, teachers should be trained in how to interpret each criterion. I believe that a better method of or the scoring procedure for a high reliability of results in this writing task is for two raters to do the scoring of the same paper rather than one rater, but I assume that only one teacher would be measuring the students’ result on the writing task in this test. Hence, as I mentioned above, students might expect good results only from their understanding of what they are doing based on the criteria and following the rubric. However, as the scoring of this test relies on only one teacher’s judgement apparently, it will not be easy to achieve high reliability for this writing test. Hence each teacher should understand of the purpose of the test well and form more specific criteria.
2.2 Validity
According to Hughes (1993, p.27), “a test is said to have face validity if it looks as if it measures what it is supposed to measure”. Based on this definition of face validity, this assessment could be understood as having high face validity..
This assessment consists of one formal writing task. The objectives of this assessment are to measure students’ writing ability in terms of spelling accuracy, grammar, and fluency so that students should be able to use their general writing skills adequately. For these objectives relating to linguistic production, this assessment required students to demonstrate their writing ability, grammar knowledge, a suitable range of vocabulary, and fluency and an understanding of writing form. Therefore, their individual reports about the national information which students have chosen, will be assessed in relation to its objectives.
This writing task provides the appropriate contents, structure and genre, which can be see as relevant test objectives. Also it would show content validity as students write the report on national information as part of a travel guide relative to the goals. A report is one form of writing among diverse types of genre. Students might therefore have the opportunity to understand and learn the characteristics of a report through prior activity as a sample of the relevant contents. And also it would be representative of students’ general writing ability. Moreover, before the writing assessment, students can recognize what they would write through the previous reading materials and filling the gap items. Such prior activities or materials can be the test of the blueprint to help guide students in their writing assessment.
As many have noted, in a writing assessment, construct validity is considered to be the test’s ability to represent the students’ general writing ability. The aim of this assessment is to write a formal report for each student by involving the use of their language skills. This assessment emphasises the overall constructs of writing of structure, grammar and vocabulary. This assessment assesses the writing ability directly so that it seems to be an appropriate assessment to measure students’ writing ability rather than other indirect measurements. Moreover students should understand what they write and the purpose of writing for this genre. Therefore such characteristics of this writing assessment can provide an approach to a proper relationship with the measurement of language writing ability needed in construct validity.
2.3 Authenticity
A simulated task can be devised, which may enable students to perform by adapting writing skills to real life tasks. All these skills - planning, researching, writing and editing are called into play when students write reports. Hence, there is an integral demand on students’ ability to complete the specific form of writing in a genre by using their target language. Likewise, students demonstrate their linguistic ability to write by using such skills as a range of vocabulary and grammar knowledge. The he topic about national facts might elicit the student’s prior knowledge or familiarity with the language. These characteristics of the task can be interpreted as the relationship between the target language situation and the specific purpose of language use.
2.4 Practicality
I can say that the general administration of this writing assessment seems to be quite easy because of not too much need for test administration. A teacher who has the role of test administrator does not need to take into account many considerations but the they will essentially set up the scoring criteria, the process of timing for the procedure and the test instrument. Students will be given enough time and a piece of assessment paper to perform their writing tasks properly. In the process of the assessment, the clear test description will not cause students to be confused. Lastly each explicit criteria can also help students perform their tasks. Therefore the practicality of this assessment can be appropriately demonstrated.
2.5 Subjective and Performance assessment
This writing test has implications for the characteristics of subjective assessment. This test would use the teacher’s judgment, in contrast with an objective test. This characteristic can be associated with performance assessment. In this writing task, performance assessment can be interpreted as if the assessment is designed to have students demonstrate their understanding by applying their knowledge and linguistic skills to a target language situation. Students are given some written materials containing diverse national information and they are asked to write a report about one chosen national fact by constructing and using their information and linguistic knowledge. This assessment does not require a simple answer from each student but elicits student’s productive skills.
2.6 Criterion-referenced assessment
According to McNamara (2000), the implication of criterion-referenced assessment is that it occurs when a test is designed to provide a measure of performance that is interpretable in terms of a clearly defined and delimited domain of learning tasks. With his explanation, this writing assessment can be understood as criterion-referenced because this test defines the performance criteria of a writing task in terms of it achieving its objectives and then also measures the student’s performance in individual achievement. Hence students are not given a grade in comparison with other forms of performance but are just given a rating from the specific description of performance at a given rating scale. For this L2 writing assessment, primarily designed to elicit a performance of the students’ writing ability, the scoring criteria should clearly articulate the definitions of the construct of English writing ability and raters should be trained to interpret these criteria. Having received the criteria with the assignment, students are able to write toward their specific goals.
3. Glossary of 10 relevant assessment terms
1. Reliability (Bachman, L, E. & Palmer, A, S., 1996, P.19): Not only to certain topic, every test should have/show a consistent result. It means that a test should give a very similar result when it is marked by the same group at a different time or when the test is marked by a different group. For example, there should be guidelines for the marking process and teachers should mark the test in a similar environment/atmosphere.
2. Content Validity (McNamara, T., 1996, P.16): It is the degree in which, a test measures what it is supposed to measure. The items on the test have to reflect what has been taught. It should cover a series of language skills, vocabulary and knowledge that are related to the chosen topic and perhaps, related to student learning plan.
3. Construct Validity (Messick, S., 1996, p.11): It is an evidence of validity gained by showing the relationship between a theoretical construct and tests that propose to measure the construct. In other words, it is the degree to which a test measures an intended hypothetical construct.
4. Face Validity (Hughes, A., 1989, p.27): It is a property of a test intended to measure something. The test is said to have face validity if it “looks like” it is going to measure what it is supposed to measure.
5. Authenticity (Douglas, D. 2000, P.16): The materials that the test designers design to apply in the assessment sourced from the test-takers’ related real life, their learning goals and the purposes that most applicants want to take the test for.
6. Interactiveness (Bachman, L, E. & Palmer, A, S., 1996, P.25): The range and types of the knowledge and skills that are involved in an assessment are the test-takers’ areas of language knowledge, strategic competence, or meta-cognitive strategies.
7. Integrative tests (Hughes, A., 1989, p.16): The assessments that have combined the various domains of language knowledge and skills with the test-takers’ language interpreting and production ability of the language knowledge background.
8. Formative assessment (Bachman, L, E. & Palmer, A, S., 1996, P.98): Formative assessment is a part of instructional process. The purpose is to improve student learning and to determine success or to what extent the program/project/course has met its goals. For example, an In-Classroom test is one of the most common formative assessment techniques.
9. Subjective assessment (Hughes, A., 1989, p.19): This is more difficult because rater has to use their own judgment. This assessment is a form of questioning which may have more than one current answer. E.g.] extended-response questions and essay.
10. Performance assessment (McNamara, T., 2000, P.5): This is a test that requires students to demonstrate their knowledge and skills by using their productive skills rather than choosing from several multiple-choice options.
4. Reflection on the whole planning of assessment process
Difficulties in the procedure of designing process
This assignment is not just to design the specific assessment but also requires an overall understanding of testing knowledge. I think that designing assessment seemed to be quite easy except for the application of the theoretical background in the assessment that has to be considered. The vague conception of the each bit of terminology about assessment sometimes led to many problematic points in this assessment. Such aspects have sometimes influenced the group discussion to complete the pair work in this assignment. Moreover some glossaries did not give a good definition of the precise characteristics used in this assignment so that we spent much time on conceptualising each term in the glossary and discussing about how they can be interpreted. These difficulties however, provided us with an opportunity to have a critical approach to this assessment and to understand what we need for completing the assignment.
Consideration of the application of the writing assessment in a Korean English class
Because there are the different points of view about English writing tests in Korea , I have considered whether this writing assessment can be performed in a Korean English class. Although the English pedagogies have shifted from traditional instruction to Communicative Language Teaching (CLT), thus far, foreign language instruction involving writing has not been performed much in the English curriculum. There are many problematic points when the writing instruction or tests are delivered in a Korean English class. Because grammar or reading comprehension has mainly occupied English instruction, students do not have the writing skills and they are not familiar with writing instruction. As a result, therefore it is not easy for students to perform a writing test. However the writing instruction is necessary to improve the target language skills even though that is a big consideration. Hence if such writing instruction was modified within the Korean English curriculum, it would be able to be carried out in CLT instruction.
Unit evaluation
Lastly, I would like to talk about the effects of this class on my studying. I have not always been able to conceptualize the quality of overall language testing so that I have sometimes had a few difficulties to understand what something has meant or potential aspects of testing. Although now, I cannot say that all notions or the rationale of language testing can be explained by me, I recognize that I have learned the necessary knowledge for teaching English. Lastly to be a good teacher needs not only the appropriate teaching method or teacher’s linguistic knowledge but also the ability to create the ideal test to be able to measure students’ performance from the point of view of the diverse theoretical aspects.
References
Bachman, L. & Palmer, A. (1996) Language testing in practice Oxford: Oxford University Press
Davies, A. (1990) Principles of Language Testing Oxford: Basil Blackwell
Douglas, D. (2000) Assessing languages for specific purposes Cambridge: Cambridge University Press
Hughes, A. (1989) Testing for Language Teachers Cambridge: Cambridge University Press
Messick, S. (1996). Validity and Washback in Language Testing, language Testing, 13(3)
McNamara, T. (1996) Measuring second language performance London; New York: Longman.
McNamara, T. (2000) Language Testing London; New York: Longman.
Weir, C. (1993) Understanding and developing language tests Prentice Hal
Appendix 2
Blanked Gap-Filling
*Fill in the blank to complete the national information
1. London is the capital of _______________
2. ________ has this electric plug
3. __________________ has over 1,000,000,000 of population.
4. 61 is the _______________ country dialling code.
5. Kangaroo is an ___________ animal.
6. 70% of the population speak Mandarin in __________
7. _______________ is a multi-cultural country, so it’s not unusual to walk down a city street and hear people speaking Italian, Greek, Lebanese, Vietnamese or Arabic as their first language.
8. This is a map of_______________
9. Canberra is the capital of ____________________
10. Which country is the biggest among Australia, China, and England?
______________
