Examining the difficulty pathways of can-do statements from a localized version of the CEFR

Introduction
In  Japan,  there  is  presently  a  lack  of
consistency across the systems employed by
Japanese  primary,  secondary  and  tertiary
educational institutions for the measurement
of  proficiency  and  progress  of  English
language  learners.  Negishi  (2011)  suggests
that  introducing  a  common  language
framework  in  Japan  would  allow  for
standardization  in  the  field  of  foreign
language learning and teaching. O’Dwyer
and  Nagai  (2011)  recommend  the  Common
European  Framework  of  Reference  (CEFR,
Council of Europe, 2001) given the previous
success  of  its  usage  in  Europe  (North,
Ortega,  &  Sheehan,  2010)  and  growing
interest  in  the  system  outside  of  Europe
(Figueras, 2012). One of the goals of such a
system  is  to  provide  learners  and  educators
with  a  set  of  learner-centered  performance
scales  that  allow  for  standardized
assessment  of  level  (North,  2007).  The
CEFR  measures  learner  proficiency  and
progress  via  illustrative  descriptors  that
describe  communicative  competencies  in
five  skills:  listening,  reading,  spoken
production,  spoken  interaction  and  writing
(North,  2007).  The  descriptors  progress
from easy to more difficult over six levels of
proficiency  (Council  of  Europe,  2001)  and
each  descriptor  provides  a  self-sufficient
criterion  of  achievement  (Skehan,  1984).
While this progression of difficulty has been
continually  validated  in  a  European  context
                       

for  the  CEFR,  regarding  the  inherent
difficulty  hierarchy  of  localized  versions  of
the  system,  comparatively  little  research
exists.  Given  the  increasing  interest  in
applying  the  CEFR  outside  of  Europe,  the
process of developing  alternate versions  “to
suit local needs and yet still relate back to a
common system” (Council of Europe, 2001,
p. 32) requires further study.   
 
Research  on  the  implementation  of  the
CEFR in Japan began in 2008 at the Tokyo
University  of  Foreign  Studies  (Tono  &
Negishi,  2012;  Negishi,  Takada  &  Tono,
2011).  Illustrative  descriptors,  known  as
can-do  statements,  from  DIALANG
(Council  of  Europe,  2001,  pp.  231-234)
were  administered  to  360  Japanese
university students. The purpose was to test
if the rank ordering of difficulty by Japanese
students, target users of the system, matched
what  was  predicted  by  the  CEFR.  The
statements  were  indeed  found  to  order
consistently.  A  further  study  by  Negishi
(2011)  showed  that  over  80%  of  English
language learners in Japan fell within  the A
level of the CEFR (also known as the Basic
User  level):  the CEFR’s can-do  statements
did  not  appear  to  provide  specific  enough
criteria  for  distinguishing  effectively
between the population’s span of language
learners  and  development  of  an  alternate
version thus began (Negishi, 2011).   
 
The  Japanese  adaptation  of  the  CEFR
(known  as  the  CEFR-Japan  or  CEFR-J),
increased  the  number  of  levels  from  the
CEFR’s original six to twelve (by breaking
down  the  four  A  and  B  levels  into  nine).
Furthermore,  all  of  the  can-do  statements
were  contextualized  for  Japanese  learners
(Tono & Negishi, 2012) and tested to ensure
that  the  rank  ordering  of  difficulty  matched
the  predictions  of  the  system  (Negishi,
2011). However, the development of a scale
is  only  the  first  step  in  implementing  a
system  (North  &  Schneider,  1998)  and  due
to the new divisions and statements, further
research,  such  as  ensuring  that  target  users
of  the  system  behave  similarly  to  the
participants  of  the  initial  development
studies, is required. In terms of ensuring the
difficulty  hierarchy  of  the  CEFR-J,  little
beyond  describing  the  development  process
has  been  published  (see  Tono  &  Negishi,
2012;  Negishi,  Takada  &  Tono,  2011;
Negishi, 2011).     
 
A  preliminary  study  by  Runnels  (2013)
measured  the  rank  ordering  of  difficulty  by
almost  600  university  students  on  the
CEFR-J’s  A1  and  A2  sub-levels.  While
there was no disordering in the levels found
(with  A1.1  being  ranked  the  easiest  and
A2.2  being  ranked  the  most  difficult),  the
mean  difficulty  ratings  frequently  exhibited
no  significant  differences  from  adjacent
sub-levels.  It  was  suggested  that  perhaps
this  was  due  to  the  sub-divisions  being  too
great  in  number:  splitting  the  A1  level  into
three  sub-levels  and  the  A2  level  into  two
may limit the ability of users or assessors to
be  able  to  reliably  distinguish  features  of
language learners at each of those sub-levels
(Runnels,  2013).  On  this,  the  Council  of
Europe (2001, p. 21) notes that “the number
of  levels  adopted  should  be  adequate  to
show  progression…but  should  not  exceed
the  number  of  levels  between  which  people
are capable of making reasonably consistent
distinctions”.  However,  the  lack  of
significant  differences  between  levels  in
Runnels’  (2013)  study  may  have  been
related  to  how  the  difficulties  of  each  skill
were  being  rated  by  participants  in  that
perhaps  one  skill  skewed  the  results  of  the
entire  level.  Thus,  the  progression  of
difficulty  should  also  be  examined  for  each
of the skills.
 
The current study was therefore designed to
explore  the  difficulty  pathways  formed  by
difficulty  ratings  on  can-do  statements
within  each  skill.  Specifically,  the  inherent
hierarchy  of  the  CEFR  (and  the  CEFR-J)
requires  that  there  be  a  gradual  progression
of  easy  to  more  difficult  as  a  learner
progresses up through the levels, and if this
requirement  is  not  met,  the  system’s
intended  function  is  lost.  It  is  subsequently
 
expected  that,  like  the  levels,  the  skills
should  also  order  as  predicted  by  the
CEFR-J,  with  the  A1.1  writing  can-do
statement, for example, being rated as more
difficult  than  A1.2  writing  and  so  on.  It  is
not  hypothesized  that  every  skill  will  order
perfectly,  but  a  general  tendency  of
increasing difficulty ratings across the levels
for  each  skill  is  certainly  expected.
Furthermore,  an  ideal  system  might  be  one
where  the  difficulty  of  A1.1  writing  is
comparable to A1.1 listening, with linear or
exponential  increases  in  difficulty  between
the  levels,  but  the  underpinning  theory  of
the  CEFR-J  does  not  require  this.  What  it
does  require,  however,  is  that  there  are
distinctions  between  the  skills  at  each  level
(Council  of  Europe,  2001)  and  therefore,  it
is  also  hypothesized  that  significant
differences  in  difficulty  ratings  between
each  level  should  exist.  Ensuring  this  kind
of  a  pathway  means  that  the  system  is
functioning as intended, and that the process
of local contextualization of the system was
successful.   
 
Methods
Participants
590  first  and  second  year  students  from  a
private  university  in  Japan  participated  in
this  study.  The  survey  was  administered
following  completion  of  either  one  or  three
semesters  of  twice  weekly  90  minute
English classes. Participation was voluntary.
 
Instrument
The  survey  was  administered  on
www.surveymonkey.com©  (SurveyMonkey,
2012).  Participants  used  a  5  point  scale  to
indicate  their  perceived  difficulty  of  the  50
randomly  ordered,  Japanese  can-do
statements from levels A1.1 to A2.2.   
 
Procedure
For  each CEFR-J level,  there  are 10  can-do
statements  (two  for  each  of  the  five  skills).
The  mean  difficulty  for  each  skill  at  each
level  (in  logits)  was  calculated  using  Rasch
measurement  software  Winsteps®  (Linacre,
2010;  for  a  full  explanation  of  Rasch
analysis see Bond & Fox, 2007; Baghaei  &
Amrahi, 2011). To measure difficulty across
levels within each skill, a logit difference of
0.3  is  required  for  a  significant  main  effect
for  difficulty  (Miller,  Rotou,  &  Twing,
2004; Lange, Greyson, & Houran, 2004).   
 
Results
The  following  five  figures  illustrate  the
Rasch bubble pathways for each of the skills
(Bond  &  Fox,  2007).  Each  level  within  the
skill is represented with a circle, whose size
is  proportional  to  the  standard  deviation  of
the  measure.  The  infit  mean  squares  are
shown  on  the  x-axis  where  it  can  be  seen
that no items exhibit any misfiting infit (see
Wright  &  Linacre,  1994). A larger value on
the  y-axis  is  associated  with  increased
difficulty ratings.  

From Figure 1, it is evident that the ordering
for  the  listening  can-do  statements  for  the
A1  sub-levels  was  consistent  with
predictions  of  the  CEFR-J,  but  that  A2.2
falls below A2.1. The overall range of logits
for  all  levels  is  1.76.  In  terms  of  the  logit
difference  required  for  a  main  effect  of
difficulty,  the  logit  difference  exceeds  the
required  0.3  difference  for  all  adjacent
 
The  difficulty  pathway  for  the  reading
can-do  statements  is  shown  in  Figure  2.
Some  disordering  is  evident:  the  sub-levels
from  both  A1  and  A2  rated  in  the  reverse
direction of difficulty from what is predicted
by  the  CEFR-J.  Specifically,  A1.3  is  rated
as less difficult than A1.2, and A2.2 as less
difficult  than  A2.1.  The  span  of  logits  is
0.91 and the required logit difference of 0.3
for  significance  exists  between  none  of  the
adjacent categories except for between A1.3
and  A2.1  although  on  this  scale,  these  two
levels do not fall adjacent to each other.

The spoken interaction pathway of difficulty
ordered  exactly as predicted by the CEFR-J
(Figure  3).  However,  it  is  evident  that  the
A1 sub-levels and A2.1 all fall very close to
one  another.  Indeed,  the  range  between  all
five  levels  spans  only  1.04  logits.  The  only
categories  with  a  difference  of  greater  than
0.3  logits  are  between  categories  A1.1  and
A1.2 as well as A2.1 and A2.2.

Figure  4  illustrates  some  major  disordering
of  categories  along  the  spoken  production
pathway of difficulty. Specifically, A2.1 has
fallen  below  the  difficulty  ratings  for  A1.2
and  A1.3  while  A2.2  was  rated  as  the  most
difficult.  The  span  across  all    logit  scores
reaches 1.3.

For the writing pathway  shown in Figure 5,
the can-do statements from both the A1 and
A2 sub-levels grouped very closely together.
The  range  of  difficulty  is  only  0.97  logits
and  the  0.3  logit  difference  required  for
significance  only  exists  between  A1.3  and
A2.1,  or  in  other  words,  between  the  two
higher  order  levels  but  not  for  any  adjacent
sub-levels.     
 
To  summarize  the  results  of  the  rank
ordering,  the  listening  can-do  statements
performed  reasonably  well,  with  only  the
A2  sub-levels  exhibiting  disorder.  Both  the
reading  and  spoken  production  can-do
statements  showed  disordering  at  both  the
A1  and  A2  levels  whereas  spoken
interaction  can-do  statements  ordered
exactly  as  predicted.  For  writing,  only  the
A2  sub-levels  rank  ordered  as  expected
although  the  difference  in  difficulty  ratings
between  the  sub-levels  at  both  the  A1  and
A2 levels is negligible.   
 
In terms of the significant differences found
between  the  levels  for  each  skill,  the
listening  can-do  statements  exhibited
significant  differences  between  all  adjacent
A1  categories,  but  not  for  A2.  For  reading,
the  required  significance  level  was  found
between  only  A1.3  and  A2.1  (although  due
to  disordering,  these  categories  were  not
adjacent).  Spoken  production  can-do
statements  behaved  similarly,  with  no
significant differences between any adjacent
categories.  While  the  spoken  interaction
can-do  statements  ordered  as  expected  in
terms of the CEFR-J, only the A2 sub-levels
exhibited significant differences. Finally, for
writing,  differences  between  the
higher-order A1 and A2 levels were evident,
but not among the sub-levels.
 
Discussion
Overall,  the  difficulty  judgments  made  by
target  users  of  the  CEFR-J  (Japanese
university  students)  on  can-do  statements
from  A1.1  to  A2.2  did  not  match  entirely
with  the  predictions  of  the  CEFR-J.
Moreover,  most  skills  exhibited  disordering
and a lack of significant differences between
adjacent categories was found for each skill.
This  relates  to  the  preliminary  findings  by
Runnels  (2013)  who  found  very  little
disordering overall, but a lack of significant
differences  between  adjacent  categories.  It
may be the case that performing this kind of
an analysis on an individual skill’s basis
does  not  support  the  underpinnings  of  the
CEFR-J  which  if  language  is  seen  as  a
uni-dimensional  construct  it  should  not  be
analysed  modularly,  according  to  skill.
Nonetheless,  the  results  herein  suggest  that
the  division  of  A1  and  A2  into  five
sub-levels  might  be  too  great  a  number  for
users  of  the  system  to  adequately  and
consistently  distinguish  features  that  are
characteristic of learners at each level.
 
In  fact,  one  of  the  major  criticisms  of  the
CEFR  is  that  there  is  little  empirical
evidence  to  support  the  inherent  hierarchy
of  increasing  difficulty  beyond  the
perception of language educators (Westhoff,
2007;  Fulcher,  2003;  2004;  2010;  Hulstijn,
2007)  and  it  seems  as  if  the  participants  in
the  current  study  perhaps  do  not  share  the
same  views  as  those  of  language  educators.
 
In some cases, the contrasts between can-do
statements  across  levels  are  quite  subtle,  as
can  be  seen  in  the  spoken  interaction  A1.2
(1)  and  A1.3  (2)  statements  where  the
primary  difference  is  that  the  higher  level
A1.3  statement  does  not  contain  “using  a
limited repertoire of expressions”:
 
(1) “I  can  exchange  simple  opinions  about
very  familiar  topics  such  as  likes  and
dislikes  for  sports,  foods,  etc.,  using  a
limited  repertoire  of  expressions,
provided that people speak clearly.”
(2) “I  can  ask  and  answer  simple  questions
about  familiar  topics  such  as  hobbies,
club  activities,  provided  people  speak
clearly.”
 
It  may  simply  be  that  students  do  not
associate  an  increase  in  difficulty  between
the  requirements  to  complete  such  tasks  in
the  same  way  that  a  language  educator
might.  In  fact,  this  highlights  one  of  the
major  limitations  of  the  current  study  and
perhaps  even  of  how  the  system  was
developed:  the  difficulty  data  is  not
comprised  of  scores  on  task  performance.
Rather,  the  analysis  is  based  on  difficulty
judgments  or  self-assessment  by  learners.
While  the  can-do  statements  are  indeed
designed  to  function  as  progress  or
proficiency  markers  when  used  by
individual learners, the learners that did not
associate less difficulty with the term “using
a limited repertoire of expressions” may not
behave  the  same  way  on  a  self-assessment,
as they might during a  more formal kind of
performance-based assessment.
 
Nevertheless,  the  results  also  suggest  that
replications  of  the  current  study  with  other
samples  of  student  populations  and  at  other
CEFR-J  levels  might  be  useful  in  order  to
determine  whether  refinement  or
modification  of  the  CEFR-J’s  can-do
statements  and  their  level  divisions  is
required.  Alternatively,  further
contextualization  of  the  existing  can-do
statements  for  use  with  the  specific
population  of  students,  to  ensure  increasing
difficulty  through  the  levels  might  also  be
necessary.   
 
In  either  case,  the  CEFR-J  is  neither
designed nor guaranteed to behave perfectly
among  every  group  of  students  or  learners
that  is  ever  administered  its  can-do
statements.  In  the  current  study,  the
hierarchy  of  difficulty  was  not  consistently
found  which  has  implications  for  CEFR-J’s
users: the scale of increasing difficulty is not
always  empirically  supported  (Westhoff,
2007;  Fulcher,  2003;  Hulstijn,  2007)  and
progression  may  proceed  at  differing  rates
or even in different directions for individual
learners.   
 
Conclusion
Ultimately,  the  results  described  herein
highlight  that  the  process  of
contextualization of a  generalized European
framework  for  local  purposes  outside  of
Europe is feasible and that the initial version
of  the  CEFR-J’s  levels  and  their  illustrative
descriptors was relatively successful. Indeed,
developing  and  testing  the  CEFR  is  an
on-going  process  involving  both
quantitative  and  qualitative  methods,
supplemented  by  replication  studies  (North,
2002;  North,  2000;  North  &  Schneider,
1998).  Updates  and  modifications  are
continually  being  made.  Although  these
processes  are  underway  for  the  CEFR-J,
additional empirical support is still required
so  that  the  CEFR-J  can  be  used  in  the
construction  of  curricula,  materials  and
assessments for improving foreign language
learning  in  the  tertiary  institutions  of  Japan
or  as  a  model  for  any  organization  looking
to  localize  a  general  framework  of
reference.