Day 2: Break Out Sessions

Positive Washback: Authentic English Tests Foster Successful Teaching and Learning

Susan Jones, Lead Academic, English Language (Americas), Trinity College London

Across Latin America, youth unemployment tops 18%, with consequences for years in the future if students do not acquire the necessary skills in school. Students need more than just theoretical knowledge, they want real world skills (such as English language proficiency) that they can apply when they enter the workforce. Education systems must move towards a constructive alignment that enhances deep learning and motivation.

This presentation will show how English language tests can help to achieve these imperatives, particularly as part of a comprehensive program starting with learning outcomes that articulate the skills; curricula that offer opportunities to develop the skills; teachers with the training to support development of the skills; and finally, assessment that requires demonstration of real skills, not just memorization of information. The presentation will demonstrate how an authentic test will have a positive impact on how English is taught and practiced in the classroom as well as provide language learners with transferable and marketable language skills.

Test Specifications for Assessing English Proficiency in Cuban Higher Education

Marisol de la Caridad Patterson Peña, Associate Professor, Universidad de las Ciencias Informáticas. Co-presenter: Ivonne de la Caridad Collado, Director of Language Centre, Universidad de las Ciencias Informáticas, Cuba

This paper presents the development process of a system of test specifications that resulted from a project carried out in a collaborative approach by an external expert from the University of Bremen and a network of developers from all Cuban universities. Partners in this endeavour are the Cuban Ministry of Higher Education, the University of Informatics Sciences, the University of Bremen, Germany, and the British Council Cuba and UK. We will present the process and steps of the development of the series of test specifications for assessing reading, listening, writing and speaking skills at both A2 and B1 levels of the Common European Framework of Reference (CEFR).

International Expertise and Local Knowledge: Striking a Balance

Jill Fortune, Language Dept. Head / Academic Consultant, Universidad Externado de Colombia / British Council

This presentation highlights the complexities of integrating international assessment models with what could or should be more contextualized and relevant EFL education models in Latin America, and poses the question as to how far such imported expertise and models can and should be adopted in Latin America, and how much they might be used more for inspiration and possible baselines, heading more towards the inclusion of more local knowledge and input.

Curriculum Review for Secretaría de Educación Pública (SEP) Mexico

Rosalia Valero Strategic Development Head - Americas, Cambridge Assessment English. Co-presenter: Dr. Alejandro Velázquez Elizalde Lecturer, Universidad Nacional Autónoma de México

This presentation will explore the approach adopted by Cambridge Assessment English when it supported SEP in the revision of its English language teaching curriculum in pre-schools, primary and secondary schools in Mexico. A focus will be on highlighting the principles of a quality English curriculum and will provide participants with a detailed run-through of the stages involved in undertaking curricula review, including a comparison of aspects of

English language use described in curricula against the CEFR, looking across skills and across levels of proficiency. Consideration will also be given to the extent to which curricula embrace pedagogical approaches conducive to communicative learning, as well as the factors which need to be taken into account to ensure the successful implementation of curricula. The outcome will be to provide education and testing practitioners as well as policy makers with an overview of English language curricula development and successful implementation.

Language Teacher Certification in the Implementation of a Foreign Language Policy at the BUAP

Roberto Criollo, Lecturer & Researcher, Benemérita Universidad Autónoma de Puebla

This paper will present the experience of implementing a Foreign Language Policy at the Benemérita Universidad Autónoma de Puebla.  The project included a language needs analysis in all different areas and departments of this institution, and the adoption of a number of measures to effectively promote its internationalization. 

Measuring the Effectiveness of Teaching Knowledge Test (TKT) in Mexico

Hugh Moss, Senior Education Advisor, Cambridge Assessment Englis. Co-presenter: Fátima Vidal, Director of Special Projects, Ceneval, A.C.

During the course of the presentation, participants will learn more about data gathering and how it formed the basis of a multi-stakeholder research study, assessing the effectiveness and impact of the Teaching Knowledge Test (TKT) within the Mexican educational context. Practical advice will be offered on the adoption of a ‘mixed methods’ approach to research, involving the use of instruments such as surveys, structured interviews and test takers’ score data. 

Teacher Talk in EFL: Language for Communicative and Pedagogical Purposes

Douglas Altamiro Consolo, Associate Professor, UNESP - State University of Sao Paulo

This research study aims at offering a better comprehension of the concept of linguistic-communicative-pedagogic proficiency of teachers of English as a foreign language. We aim at verifying the existence of differences (i) between foreign language teachers’ ability to use the language for general and for pedagogical purposes, and (ii) between the characteristics of teachers’ aforementioned abilities performance and their proficiency grading in a specific Brazilian examination, the EPPLE (Proficiency Examination for Foreign Language Teachers). Initially, we describe and analyze teachers’ oral performances in tasks of general and specific language use in the examination. Later, we analyze such performance samples in relation to the concepts of assessment holistically attributed to candidates in the examination. Results point to evidence of higher levels of pedagogical language use abilities associated with higher levels of language use abilities for general purposes. However, the opposite does not prove to be true, that is, candidates who manage communicative language use do not necessarily know how to use the language for teaching purposes. Thus, the contrast provided by this research study endorses the thesis that questions the sufficiency of communicative-language ability to work as an English teacher, reinforcing the need for proper teacher training to enable teachers-to-be to use classroom language. Results from this study are expected to contribute towards the assessment of language teachers’ proficiency and in teacher education, in Brazilian and other Latin American contexts.

Multi-region Language Assessment of Brazilian Teachers for a Bilingual Programme

Dr Cristiane Corsetti, Training Coordinator, Edify

This PhD thesis project presented in this session aims to design a language test for a group of Brazilian high-performance military pilots – who represent Brazil and its Air Force abroad, while airborne and on the ground, as diplomatic representatives – based on a needs analysis for the group. The main question that guides the study focuses on the characteristics of an English proficiency test for a group of pilots with such special needs. By means of a qualitative study, data has been generated with the aim to determine the content, the construct and the means of operationalization of such a test which, hopefully, will provide reliable information for these pilot’s safety on international flights. By the end of this study we hope to contribute to the unexplored area of studies on language assessment for military pilots in Latin America.

The EXAVER Proficiency Test: Customers’ Experiences for Test Improvement

Janeth Tello Carrillo, EXAVER test coordinator, Universidad Veracruzana

EXAVER exams are currently offered to university students and the general public in eight examination centers, spread all over the state of Veracruz, twice a year. Approximately 2,500 candidates take one of our tests every year. As providers of a service, it is vital for us to understand the customer experience. During the September – October 2018 test administrations, an online survey was designed to collect data regarding customers’ opinions of the test enrollment process and the administration of the test. Our findings suggest three main areas of improvement in relation to test validity and quality control: materials, procedures and examiner training. 

Policy in Action: Promoting Localised Proficiency Standards through Stakeholder Engagement

Flora Wang, Project Manager, British Council. Co-presenter: John Dylan Burton, Assessment Solutions Consultant, British Council

This presentation will report on China’s Standards of English (CSE) – a localised English language proficiency framework – as a case study for how one region has sought to set standards for its own educational context. It will first consider the rationale and objectives of the CSE as well as recent developments in how these standards have been used in the region, highlighting the advantages of localisation over the use of international standards. The presentation will then consider the vital role of stakeholder engagement activities when promoting these standards amongst educators, in particular efforts by the British Council and the ongoing impact to this day. Key regional activities will be described as well as preliminary data supporting the positive promotion, understanding, and use of the CSE.

Examiner Engagement and Candidate Satisfaction: The Case of EXAVER

Marion Alain Meunier, Universidad Veracruzana

The EXAVER programme consists of a suite of three English Language Certification Exams belonging to the Universidad Veracruzana (UV), Mexico. The examinations were designed and developed with the support of the British Council, the University of Cambridge, and the University of Roehampton, England. With this set of three exams, the Universidad Veracruzana certifies the English proficiency of its own staff and students, and the general public, based on the Common European Framework of Reference for Languages (CEFR) (Hernández Alarcón, 2010). This study aims to explore whether there is a connection between the level of work engagement of 73 EXAVER examiners and 1,400 candidates’ level of satisfaction in the service provided while taking any of the tests in the February - March 2019 administration. 

A Test Development Model for Developing a Localised Test

Judith Fairbairn, Manager, Assessment Development Team, British Council. Co-presenter: Richard Spiby, Test Development Researcher, British Council

This talk is aimed at test development practitioners as a practical example of a test development project for clients in China. The British Council’s circular-flow model of test development (Fairbairn and Dunlea, 2018) is illustrated. The development also employed a theoretical model of localisation (O’Sullivan and Dunlea, 2015) to adapt an existing test, Aptis for Teens, for a particular group of test takers and for a clearly defined test use. From the initial stakeholder engagement activities undertaken to identify the test development context through to the final product, challenges that required balance and compromise are explored. The project was successful because it included an assessment research team, an assessment development team and a team working with the client on the ground. 

Exam Preparation – Positive or Negative Backwash Effect?

Jose Luis de Paz, User engagement manager, Tyndale Education

Washback in testing has been defined as the influence a test has on the way candidates perform in any classroom.  This concept is key in assessment since examinations prevail in most programmes in the world.

This presentation will focus on the results of teachers concentrating excessively on the exam format..  Most authors in this discipline have suggested a balanced view towards testing and concentrating on the learner needs, not on the exam (Weimer, 2002).

In the last few years, different ELT professionals have been very much interested in proficiency or teaching knowledge exams.  The resul has been an important increase in exams promotion and their relevance to achieve promotion and prestige in the career.

During this talk, participants will be asked to share their experiences in this process and some ideas will be proposed to improve results.

Four-Dimensional Curriculum for Adult ESL Learners and its Assessment: Benefits and Challenges

Sotiria Koui, Assistant Professor of English, Community College of Philadelphia

International learners of English at ESP programmes in community colleges in the U.S. do not need just language instruction to integrate into American society; they also need employment-transferrable skills, and a sense that they are valued members of a community. The literature on curriculum design and pedagogy (C. Tassone, 2017, Healey, M. 2005) supports the need for curricula that equip our future citizens with a range of skills that can be constructively applied to academic life, the workplace, and civic engagement activities. Building competencies that enable collaboration within the community and the workforce provides our future citizens with opportunities to take on new and fulfilling challenges and enjoy accomplishments; these competencies can also prepare these individuals for a volatile work environment while teaching them civic responsibility for sustainable humanity – key points explicitly stated by the Center for Curriculum Redesign, United Nations, and Council of Europe.

Fortunately, such curricula are already being designed and implemented; however, assessment standards that measure employment-transferrable skills are still in their nascent stages. Thus, there is a clear need for such assessment standards.

In this session, the presenter describes a “four-dimensional curriculum” used in a community college English for Business course, and the accompanying assessment rubrics that she developed; these rubrics assess knowledge, employable skills and growth mindset.

Assessment Literacy Enacted? Unearthing Spontaneous Assessment Events in Student-teacher Conferencing

Colin Barnett, Teaching Fellow Columbia University

This paper presents how techniques from Conversation Analysis (Sacks, Schegloff, & Jefferson, 1974) and dimensions of Learning Oriented Assessment (Turner & Purpura, 2016) were used to identify and dissect what was happening in on-the-fly assessment events occurring in the ‘collaborative interactional frame’ (Poehner & van Compernolle, 2011) of a student-teacher conference. The (non-)verbal resources identified this single-case analysis are shared along with reflections on how such processes support assessment literacy.

Does Writing Assessment Training Really Work? The Mexican EFL Perspective

Elsa Fernandez González, Universidad Autónoma de Tamaulipas

The present study analyzed the impact that two sessions of WAT had on eleven Mexican EFL university teachers who were tracked for a period of 12 months. Data was collected with a background questionnaire, teacher focus group sessions, and two semi-structured interviews to the eleven teachers. Data obtained suggested that training had positive impact in three main areas: a) classroom teaching of writing, b) classroom assessment of writing and c) teachers´ self-awareness of assessment. However, the impact of the training on teachers´ actual classroom assessment was quite shallow. Greater impact was found in teachers’ reflection processes and self-awareness of themselves as EFL teachers and assessors. The Writing Assessment Training Impact Categorization is proposed in an attempt to classify the innovations training may have encouraged in EFL teachers. The presentation finalizes with a discussion of possible implications for EFL classroom assessment, teacher trainers and language program managers.

Supporting the Development of Assessment Capacity in Colombia

Hugh Moss, Senior Education Advisor, Cambridge Assessment English. Co-presenter: Moravia Elizabeth González, Test Manager, ICFES

The presenters take a historical look at how assessment was utilised in Colombia’s national bilingual project. Instituto Colombiano para la Evaluación de la Educación (ICFES) partnered with Cambridge Assessment English to support the development of English tests aligned to international standards as part of Colombia’s Ministry of Education’s National Bilingual Project aim of equipping the country’s student population with English skills for work and higher study. Participants will be provided with an overview of the approach to benchmarking students’ and teachers’ level of English according to the Common European Framework of Reference and how the data produced acted as an essential element in the setting of English language targets and design of assessments appropriate to the level of test takers. 

Assessment Literacy: Local Voices, Global Needs

López Gopar, Mario E.

Debunking a Myth: Colombian Higher Education Examinations and Virtual Language Learning

Leopoldo Pinzón, Director of the Language Institute, Universidad Santo Tomás

Based on the outcomes of the Colombian compulsory higher-education examinations in the last seven years, this presentation aims to analyse the suitability of E-learning for language learning. Covering a population of 1.300.000 students, 400 universities and 500 academic programs, this study presents a robust and comprehensive analysis of the actual implication of implementing an E-learning approach, offering a detailed landscape on the achievements this methodology has attained and what can be expected as language goals from these programs.

Maximising the Utility of English Assessment Scores

Jason Turner, Senior Vice President, Global Services, MetaMetrics

As increasing numbers of learners study English and take examinations designed to monitor language growth and determine readiness for entrance into the next stages of education, it is important to maximise test score utility by incorporating information from supplemental metrics to identify appropriate reading materials for learners.

During the pilot stages of the Aptis Teens Higher (ATH) development, a study was conducted to link The Lexile Framework reading scale to the ATH scale. Now, scores on ATH may be reported as Lexile measures as well as and CEFR levels.  Participants were Chinese learners preparing for IGSE exams. The average learner reading ability (N=1358) was 1033L, within the B2 range on the CEFR scale. Previous linking studies with Aptis General and Aptis Advanced show strong alignment of CEFR levels and Lexile measures reported from the three assessments.

In a related study of IGSE textbooks and exam papers administered in Chinese international schools, text complexity, in Lexile measures, ranged from 780L to 1180L. When a learner’s English reading ability (e.g. 1033L) is lower than the text demand of a text/exam (e.g. 1180L), he/she will encounter reading comprehension challenges. Knowing the Lexile measure of a learner and text provides educators insight into when a learner will need reading comprehension support.  

Aligning Needs Analysis and Test Design for Brazilian Military Pilots

Ana Ligia Barbosa, University of Campinas

This PhD thesis project presented in this session aims to design a language test for a group of Brazilian high-performance military pilots – who represent Brazil and its Air Force abroad, while airborne and on the ground, as diplomatic representatives – based on a needs analysis for the group. The main question that guides the study focuses on the characteristics of an English proficiency test for a group of pilots with such special needs. By means of a qualitative study, data has been generated with the aim to determine the content, the construct and the means of operationalization of such a test which, hopefully, will provide reliable information for these pilot’s safety on international flights. By the end of this study we hope to contribute to the unexplored area of studies on language assessment for military pilots in Latin America.

CEFR Companion Volume: Re-aligning Pronunciation-Focused Descriptors in a Holistic Way

Philip Horne, Test Quality Assurance Manager, British Council

 This presentation specifically reports on a study of the updated Phonological Control Scale in the CV.  This has clear implications for the CEFR-linked pronunciation descriptors in the holistic rating scale utilized in the case study.  There are also broader issues concerning pronunciation descriptors within holistic rating more generally.  Research questions focused, therefore, on the nature of pronunciation-related decision-making in holistic rating, and whether a more detailed operationalisation of pronunciation descriptions (based upon the CV) would refine scoring.

The study took a mixed-methods approach.  Six raters rated a number of speaking tasks (n=50), using both the existing rating scales and the updated Phonological Control Scale.  This was followed by a series of paired interviews, which indicated that the current pronunciation descriptors lack sufficient detail.  However, there was high correlation between the two sessions (rs = .91), indicating that pronunciation can factor more greatly into decision-making in holistic rating.  Furthermore, a chi-squared test revealed that pronunciation-based decisions occur less frequently at higher levels in particular, again indicating the descriptors in their current format are less relevant to decision-making.  The session concludes by suggesting modifications and potential revised descriptors.