View
124
Download
10
Category
Preview:
Citation preview
bifie – Bundesinstitut für Bildungsforschung, Innovation & Entwicklungdes österreichischen Schulwesens
External Assessment in Austria
. © Gabi FRIEDL- LUCYSHYN (2009)
bifie – Bundesinstitut für Bildungsforschung, Innovation & Entwicklungdes österreichischen Schulwesens
Educational context:
bifie – Bundesinstitut für Bildungsforschung, Innovation & Entwicklungdes österreichischen Schulwesens
Standards on primary and lower secondary level
describe essential competences a student is supposed to sustainably have acquired in certain subjects at decisive stages in his/her school career
formulate the desired outcome of the learning process
aim at evidence-based quality assurance on classroom, school and system level
bifie – Bundesinstitut für Bildungsforschung, Innovation & Entwicklungdes österreichischen Schulwesens
3 Functions of Standards
1. Accountability for outcome of teaching and learning
2. Individual coaching of students (diagnostic feedback)
3. System monitoring
bifie – Bundesinstitut für Bildungsforschung, Innovation & Entwicklungdes österreichischen Schulwesens
Clear focus on teaching, not on testing No influence on assessment of individual student
Aims: gauging the degree to which expected goals are
achieved data for quality assurance teachers must assure the acquisition of defined
competences and coach – as best they can - students individually by continually comparing defined goals with actual performance
Support: diagnostic tools
bifie – Bundesinstitut für Bildungsforschung, Innovation & Entwicklungdes österreichischen Schulwesens
The Assessments (regular monitoring) Psychometrically validated tests Subjects, test cycles Structure of feedback Design of feedack:
Context variables (questionnaires) allow for fair feedback (performance in relation to classes and schools in comparable contexts)
Feedback given in as differentiated a way as possible Trained experts can be called upon to support schools in interpreting data
and taking action
bifie – Bundesinstitut für Bildungsforschung, Innovation & Entwicklungdes österreichischen Schulwesens
Challenges
bifie – Bundesinstitut für Bildungsforschung, Innovation & Entwicklungdes österreichischen Schulwesens
School leaving exam
bifie – Bundesinstitut für Bildungsforschung, Innovation & Entwicklungdes österreichischen Schulwesens
Context Basis: BIFIE has the mandate to design, develop, and
implement scientifically validated, competence-oriented written exams for all types of upper secondary schools
Subjects: Language of instruction, Modern Languages (CEF Levels B1 or B2), Latin/Greek, Maths
Aim: Validity of exam and assessment of competences
Key data: nationwide introduction 2013/14 (general secondary), 2014/15 (vocational colleges on secondary level)
© FRIEDL-LUCYSHYN 2009
bifie – Bundesinstitut für Bildungsforschung, Innovation & Entwicklungdes österreichischen Schulwesens
Architecture: Basic concept:
Combination of individual student and school profile as well as standardisation in essential areas
3 phases: „pre-scientific“ paper (internal) 3 or 4 written examinations (external) 2 or 3 orals (internal)
Certificate: Exam results in each part + study programme of previous
4 years (subjects, number of hours, grades)
© FRIEDL-LUCYSHYN 2009
bifie – Bundesinstitut für Bildungsforschung, Innovation & Entwicklungdes österreichischen Schulwesens
Written exam (external)
No external correction, but keys and justifications, assessment criteria und cut scores are binding
Various models of quality assurance under discussion
Online help desk and hotlines
bifie – Bundesinstitut für Bildungsforschung, Innovation & Entwicklungdes österreichischen Schulwesens
Piloting in all subjects until 2013/14
Voluntary participation Modern languages: since 2008 (reading, listening,
language in use), 300 out of 350 schools on board; writing in 2011/12
Maths, language of instruction, Latin, Greek: from 2010/11
bifie – Bundesinstitut für Bildungsforschung, Innovation & Entwicklungdes österreichischen Schulwesens
Test development by Austrian universities
Cooperation with/review by (inter)national tertiary institutions
Active and essential role of teachers as practitioners at all stages
Integration of learners (as target group) in validation process
Open-ended and close-ended item formats
© FRIEDL-LUCYSHYN 2009
Development of tasks, items and test papers – general procedure
bifie – Bundesinstitut für Bildungsforschung, Innovation & Entwicklungdes österreichischen Schulwesens
How tests come into being(foreign languages) Designed and developed by teachers and methodologists (university
academics): Regional, professional and institutional diversity Training
Validation: Trialing of tasks and items (+ questionaires)
Processing, analysis and interpretation of stats (test-psychologists + psychometricians) Item-moderation Trialing ….
Standard setting (stakeholders)
© FRIEDL-LUCYSHYN 2009
bifie – Bundesinstitut für Bildungsforschung, Innovation & Entwicklungdes österreichischen Schulwesens
How tests come into being (2)
Aim of validation process: Appropriateness of level of difficulty for student population Validity and quality of items Clarity of instructions Motivation
University of Innsbruck Cooperation/Review:
University of Lancaster (Prof. Charles Alderson, Dr. Rita Green): Training of item writers, item moderation, standard setting
© FRIEDL-LUCYSHYN 2009
bifie – Bundesinstitut für Bildungsforschung, Innovation & Entwicklungdes österreichischen Schulwesens
The test paper in the foreign languages
© FRIEDL-LUCYSHYN 2009
Skills Listening Reading Language in Use
Number of tasks
4 -5 tasks 4 - 5 tasks 4 – 5 tasks
Test formate
•Multiple Choice•Multiple Matching•True/False•Short answer questions
•Multiple Choice•Multiple Matching•True/False/Not Given •Short answer questions
•Editing/error correction•Multiple choice cloze •Word formation
bifie – Bundesinstitut für Bildungsforschung, Innovation & Entwicklungdes österreichischen Schulwesens
The test paper in the foreign languages Writing:
Part 1: argumentative text (essay); compulsory
Part 2: 2 tasks, student chooses one
Text types: letter, report, article, composition
Writings skills: argumentative, discursive, descriptive and narrative
Assessment scales B1/B2
Recommendations for orals
© FRIEDL-LUCYSHYN 2009
Thank you!
www.bifie.at
© FRIEDL-LUCYSHYN 2009
bifie – Bundesinstitut für Bildungsforschung, Innovation & Entwicklungdes österreichischen Schulwesens
Results compared to internal tests in final year 2009
listening
11%7%
30%52%
comparable
better
worse
no comment
Results compared to internal tests in final year 2008
listening
13%4%
61%22%comparable
better
worse
no comment
Evaluation of exam results 1 (N = 444)
bifie – Bundesinstitut für Bildungsforschung, Innovation & Entwicklungdes österreichischen Schulwesens
Evaluation of exam results 2
Results compared to internal tests in final year 2009
reading
2%
9%17%
72%comparable
better
worse
no comment
Results compared to internal tests in final year 2008
reading
15%11% 72%
2%
comparable
better
worse
no comment
bifie – Bundesinstitut für Bildungsforschung, Innovation & Entwicklungdes österreichischen Schulwesens
Evaluation 3 (N-444)
020406080
percentage of teachers reporting changes in the amount of instruction
time caused by introduction of the new final assessment
hotline
7,9%
3,0%
19,8%
9,9%
59,4%listening
reading
language in use
reading, hearing
reading, hearing, LiU
bifie – Bundesinstitut für Bildungsforschung, Innovation & Entwicklungdes österreichischen Schulwesens
bifie – Bundesinstitut für Bildungsforschung, Innovation & Entwicklungdes österreichischen Schulwesens
bifie – Bundesinstitut für Bildungsforschung, Innovation & Entwicklungdes österreichischen Schulwesens
bifie – Bundesinstitut für Bildungsforschung, Innovation & Entwicklungdes österreichischen Schulwesens
Recommended