ICAO Language Proficiency Requirements — Rated Speech Samples
The ICAO Language Proficiency Requirements were developed to address the safety concern raised by accidents and incidents in which communication played a key role.
The first edition of the ICAO Language Proficiency Requirements — Rated Speech Samples training aid is the latest tool developed by ICAO to support the implementation of language proficiency requirements. The purpose of the training aid is to provide examples of speech rated according to the levels on the ICAO Language Proficiency Rating Scale.
While the training aid can be used on its own, familiarity with Doc 9835 — Manual on the Implementation of ICAO Language Proficiency Requirements is advisable.
|
||||||||||||||||||||||||||||||||||||||||
Since the adoption of the ICAO Language Proficiency Requirements, a need for examples of speech rated in accordance with the ICAO Rating Scale has become clear. A call for speech samples was made to the industry in November 2004 with a view to the creation of a training aid. The PRICESG Linguistic Sub-Group convened in Montreal for a week in September 2005 to analyse and rate the speech samples. The meeting brought together a group of operational and linguistic experts from France, Japan, Poland, Russia, Ukraine, United Kingdom and United States. The present training aid is the result of the work of this group.
The group listened to numerous samples to ensure that they were of good audio quality, appropriate, that they included regional accents, and examples of all the language proficiency skills of the rating scale. The speech samples were generously provided from institutions around the world (see credits).
The group listened at length to each of the selected samples and analysed them in detail. The ratings that accompany each sample are the product of this analysis. These samples provide the basis and examples of the various levels and language skills found in this training aid.
This first edition of the training aid contains speech samples for ICAO Levels 3, 4 and 5. It was decided to focus on these three levels initially for two reasons. First, ICAO Levels 3, 4 and 5 are considered the most critical in determining whether a test taker can be allowed to be involved in international operations. Second, paragraph 1.2.9.7 in
The speech samples have been rated according to the ICAO Language Proficiency Rating Scale. The tests from which these samples are taken use a variety of formats. Their inclusion in the training aid does not imply in any way that ICAO endorses the content, format, features or interview techniques of these samples. On the rating form of each rated sample you will find some comments on the impact the particular test format may have on the accuracy, scope and reliability of the test and rating itself.
The training aid can play a key role for personnel directly involved with language proficiency testing or training such as language raters, operational raters, interlocutors (i.e. personnel who administer the conversational part of a test), teachers, course material developers, testing system designers. The training aid can provide support in calibrating the scoring of raters, and establishing benchmarks for training materials and testing systems. Other personnel are involved in the management of the implementation of language proficiency requirements. These personnel belong to air navigation service providers, management of operations, training management, licensing , etc. The training aid can assist them in becoming familiar with the ICAO language proficiency levels and the process involved in testing and rating. Finally, the training aid can be very useful to the key stakeholders in the implementation of language proficiency requirements – pilots and air traffic controllers. The training aid provides authentic examples of speakers at different levels, with various regional accents, and what is expected of them at ICAO Operational Level 4.
The purpose of the training aid is to assist and support personnel involved in implementing the ICAO Language Proficiency Requirements consistently and reliably worldwide. The training aid is a tool that can be used for several purposes:
- familiarise personnel with what an ‘ICAO Operational Level 4’ or ‘ICAO
Pre-Operational Level 3’ is; - become more aware of the criteria by which a test taker's speech is evaluated;
- find a justification and specific examples that illustrate the test taker's ability in each of the six language proficiency skills on the ICAO Language Proficiency Rating Scale;
- access specific examples of speech illustrating the various skills at different levels of the ICAO Language Proficiency Rating Scale; and
- realise the way in which test content and format affect the validity and reliability of the test results.
As the paragraph above indicates, personnel using this training aid may exercise a variety of functions related to the implementation of language proficiency requirements. For example, personnel may fulfil the function of linguistic rater, operational rater, interlocutor, or testing system designer. For each of these functions, minimum qualifications in terms of education, training and experience should be met. Depending on the function they carry out, personnel will need to demonstrate a set of competencies. In some instances, personnel may meet the competencies of several functions while in others a team approach may be necessary. For example, one of the competencies a linguistic rater would have to demonstrate is the ability to characterize the performance of the test taker for each language proficiency skill at one of the six ICAO Levels. The training aid can be used in a variety of ways to support and enhance the competencies required in these different functions.
The training aid provides personnel involved in the implementation of language proficiency requirements with:
- examples of thirteen long speech samples rated at ICAO Levels 3, 4 and 5 from different regions of the world rated by an international team of operational and linguistic experts;
- ratings of these speech samples with a detailed rationale for each of the six ICAO language proficiency skills (Pronunciation, Structure, Vocabulary, Fluency, Comprehension, Interactions);
- examples of the six ICAO language proficiency skills for ICAO Levels 3, 4 and 5;
- a reminder of the operational rating principles specific to the implementation of the rating scale;
- reflections about the design and content of aviation English testing processes; and
- links to additional resources.
Use of this training aid requires the following hardware and software components:
- Pentium-based computer
- Minimum 64 MB RAM
- Windows 98/2000/XP
- CD-ROM drive
- Sound card (speakers/headphones recommended)
- Audio Player for MP3 files such as Windows Media Player, RealPlayer, QuickTime, etc.
- Adobe Acrobat Reader, Version 7 or higher
- Internet browser
Since its publication in March 2003, a number of attempts have been made to establish a correlation between the ICAO Language Proficiency Rating Scale and other widely used English language rating systems (e.g. Test Of English as a Foreign Language (TOEFL), Test Of English for International Communication (TOEIC), Association of Language Testers in Europe (ALTE), International English Language Testing System (IELTS), Common European Framework etc.).
Although some qualified general correlations may be made in certain areas of language use, it is not possible to make an overall correlation. The scope and focus of the ICAO Language Proficiency Rating Scale are quite specific and unique in several important ways. The scale:
- only addresses spoken language (speaking and listening); unlike nearly all other scales, it does not address reading and writing skills;
- has a distinct aviation focus; it addresses the use of language in a
work-related, aviation context; and - does not target grammatical correctness or mother tongue pronunciation; grammar, syntax, vocabulary and pronunciation are judged primarily to the extent they do not interfere with effective oral communication.
The final rating is not the average or aggregate of the ratings in each of the six ICAO language proficiency skills but the lowest of these six ratings.
This final rating is due to the fact that although linguistic considerations obviously contribute to assessing a test taker's proficiency, it is operational and safety considerations that are paramount. For example, a person who has been rated at Level 5 in Structure, Vocabulary, Fluency, Comprehension and Interactions but whose ‘pronunciation, stress, rhythm and intonation frequently interfere with ease of understanding’ would be rated Level 3 in Pronunciation. His/her overall ICAO Language Proficiency Level would consequently be rated at Level 3.
In other words, in the final analysis, the question that has to be answered is "Would the pilot or air traffic controller be able to communicate successfully and with relative ease in case of complications or unexpected turn of events?"
The ICAO Language Proficiency Rating Scale and Part III of Appendix A of Doc 9835 provide clarification of the six language proficiency skills and the criteria for each level.
Click here to access the ICAO Language Proficiency Rating Scale with examples in the form of brief audio files for the language proficiency criteria for Levels 3, 4 and 5.
The training aid uses speech samples generously provided from around the world. They have been rated according to the ICAO Language Proficiency Rating Scale. The tests from which these samples are taken use a variety of formats. Their inclusion in the training aid does not imply in any way that ICAO endorses the content, format, features or interview techniques of these samples. On the rating form of each speech sample you will find some comments on the impact the particular test format may have on the accuracy, scope and reliability of the test itself and the rating.
This first edition of the training aid contains speech samples for ICAO Levels 3, 4 and 5. It was decided to focus on these three levels initially for two reasons. First, ICAO Levels 3, 4 and 5 are considered the most critical in determining whether a test taker can be allowed to be involved in international operations. Second, paragraph 1.2.9.7 in
ICAO does not endorse the information, views and opinions expressed by the interlocutors and test takers recorded in the speech samples.
Through the Rated Speech Samples — Audio Files and Ratings, users can access speech samples and their corresponding ratings. Within each rating, examples for each of the six ICAO language proficiency skills have a time code
Users may feel that a transcription is more or less accurate depending to a large extent on their familiarity with a dialect and/or accent. The transcription has been provided as a marker in the speech sample to support the evidence of the rating provided.
Text highlighted in blue can be found in the transcription. The blue highlight indicates the part of the speech sample on which the comments are focused.
Some parts of the speech samples have been deleted. The areas where this has been done are indicated by the following text: [deleted]. This was done in order to eliminate any references to a company or a person.
The examples listed in the ratings should not be considered as exhaustive.
Through its work on this training aid, the PRICESG Linguistic Sub-Group identified a number of factors that impact the effectiveness of test formats in demonstrating the test-takers' language proficiency. A brief discussion of these factors is provided below. This list should not be considered exhaustive and Doc 9835 should be consulted for more information on test design and format.
One factor that often affected the scope of the rating, and which was repeatedly mentioned on the rating sheets, was the fact that in a test format using a single interlocutor, it was impossible to assess the test taker's ability "to comprehend a range of speech varieties (dialect and/or accents) or registers."
As a result, in several cases, test takers who showed all the other characteristics of Level 5 could only be rated as a Level 4 in Comprehension, which may not adequately reflect the test taker's ability if he or she were measured in a more complete testing situation.
The PRICESG Linguistic Sub-Group also observed that in many interview-type tests, the interlocutor tended to modify his speech delivery and language content with weaker test takers in order to ensure that they understood the questions. Interviewers need to control such modifications to ensure that a test taker's comprehension is adequately assessed. Furthermore, variations in delivery and language complexity by the interlocutor from one test taker to another can seriously jeopardise the impartiality, accuracy and fairness of the testing system.
Another way in which the interlocutor may affect the accuracy of the testing is by intervening too soon, providing prompts or paraphrasing. This may either mask the test taker's inability to interact or, on the contrary, prevent him or her from demonstrating his/her interactive skills.
Supportiveness, although a natural human reaction, can result in the accuracy of the test results being distorted and blurring the real distinctions between the levels.
Because of the high stakes involved, pilots and controllers deserve to be tested in a context similar to that in which they work, and test content should be relevant to their roles in the work-place. While seeking to make test content relevant, designers should ensure that no material of a strictly operational nature is introduced. This restriction on test content is essential to ensure that the focus of the test remains on the use of plain language, and avoids any implication that evaluation of specialised operational language (such as formulaic phraseology) is part of the test. Training and evaluation of specialised operational language can only be conducted by appropriately trained and experienced operational personnel.
The Appendix to Annex 1 includes a note that the descriptors of the Language Proficiency Rating Scale and the Holistic Descriptors have an application to both phraseologies and plain language. Phraseologies may be of a number of types. Phraseologies are, of course, a component of plain language. Within languages for special purposes, including aviation language, particular phraseologies come into common use and take on more specific significance. Within the operational domain, specialized phraseologies are carefully constructed that convey specific operational meaning and are used as operational tools by qualified controllers and pilots. These phraseologies are known as ‘standardized phraseology’ or ‘formulaic phraseology’. Formulaic phraseology is constructed in consideration of operational factors and not according to the rules that govern plain language. Formulaic phraseology uses words that, although common to plain language, have a distinct operational significance that can only be fully recognized and properly responded to by trained controllers and pilots. Controllers' and pilots' application of formulaic phraseology is tested by operational examiners as a part of States' regulatory programmes. Many of the descriptors of the Language Proficiency Rating Scale and the Holistic Descriptors have no relevance to formulaic phraseology (i.e. comprehension, interaction, fluency, structure) or limited relevance (i.e. pronunciation and vocabulary). The phraseologies to be tested in compliance with the ICAO language provisions are those phraseologies that are used in plain language and work-related aviation language that are devoid of operational significance.
In the event that States choose to combine a test in plain language (in accordance with the language provisions) with formulaic phraseology (as a part of regular operational testing), it is necessary that the test design, delivery, assessment and rating be undertaken by linguistic and operational experts as a team.
While fully computerized tests have the advantage of being able to provide a wide variety of accents and types of delivery to evaluate a test taker's comprehension at Level 5, they do not test sufficiently the test taker's ability to interact. In the present state of technology, live interlocutors seem to be indispensable to administer a test that encompasses all ICAO Language Proficiency Skills. A further issue in computer or tape-administered tests, is the amount of time allotted for each response. If the time is too short, the test taker may be put under excessive stress, which may affect his or her ability to fully demonstrate an accurate range of proficiency.
Although the interlocutors do not have to be aviation specialists, in several cases the interlocutor's lack of basic aviation knowledge resulted in the pertinence of the test taker's responses not being appreciated and the fluency of the interaction being affected. This type of insufficiency also affects the test's face validity.
For practical reasons, it is often not possible to have more than one interlocutor. However, in all the cases where there were two interlocutors in the interview situation, the test gained in objectivity, scope, consistency and interactivity.
Finally, the interlocutor should have a polite attitude towards the test taker and be aware of the cultural and professional background of the person he/she is addressing. A lack of sensitivity in this regard could discredit the test and should be addressed during interlocutor training.
ICAO is grateful to the following institutions that provided speech samples for this training aid:
Association of Air Transport Engineering and Research (ATEC), Japan
Civil Aviation Safety Authority (CASA), Republic of Korea
Delta Connection Academy, United States
Department of Civil Aviation, Brazil
Dirección General de Aeronáutica Civil (DGAC), Peru
École Africaine de la Météorolgie et de l'Aviation Civile (EAMAC), ASECNA, Niamey, Niger
École Nationale de l'Aviation Civile (ENAC), France
Eurofly S.p.A, Italy
FAA Academy, United States
HungaroControl, Hungary
International Air Transport Association (IATA), Montreal, Canada
Mayflower College, United Kingdom
Moscow Air Traffic Control Center, Russian Federation
Oklahoma State University, United States
Polish Airports State Enterprise - Polish Air Traffic Agency, Poland
Queen Noor Civil Aviation Technical College, Jordan
RMIT English Worldwide, Australia
Rudloe College, Astac Ltd, United Kingdom
Sophia University, Japan
Tamagawa University, Japan
ICAO is grateful to the members of the Proficiency Requirements in Common English Study Group (PRICESG) and its Linguistic Sub-group (LSG) who participated in the preparation of this training aid. ICAO appreciates very much the support that the employers of these members provided.