## Research Papers from the Shell Centre for Mathematical Education

This represents a small selection of papers and other research materials from the Shell Centre team. We plan to develop this section further to include some of the papers and theses previously available from Shell Centre Publications in print form, permissions from the authors permitting.

A better indexing system will be introduced shortly. Currently, papers are in alphabetic order by title.

### A population of assessment tasks

#### Phil Daro and Hugh Burkhardt (2012)

**Daro, P. and Burkhardt, H. (2012).** *A
population of assessment tasks,* Journal of
Mathematics Education at Teachers College 3 Spring-Summer
2012

We propose the development of a “population” of
high-quality assessment tasks that cover the performance
goals set out in the *Common Core State Standards for
Mathematics*. The population will be published.
Tests are drawn from this population as a structured
random sample guided by a “balancing algorithm”.

### Applications and Modelling in Mathematics Education

#### Eric Muller and Hugh Burkhardt (2007)

**Burkhardt, H., Muller. E. R. et al. (2007).**
*Applications and Modelling in Mathematics Education.*
in W. Blum, P. Galbraith, H-W. Henn, & M. Niss (Eds.). *Modelling
and Applications in Mathematics Education.*
Heidelberg: Springer Academics.

These five chapters address important issues on mathematics teaching and learning. They include, amongst others, how Applications and Modelling help students learn mathematics in ways that result in a deep and holistic under- standing; are central to the development of Mathematical Literacy, and; are en- riched by the creative use of technology.

### Assessing Mathematical Proficiency: What is important?

#### Hugh Burkhardt (2007)

**Burkhardt, H. (2007).** *Assessing
Mathematical Proficiency: What is important?* In
A. H. Schoenfeld (Ed.), *Assessing Students'
Mathematics Learning: Issues, Costs and Benefits.*
Volume XXX. Mathematical Sciences Research Institute
Publications. Cambridge: Cambridge University Press.

My brief is to look at those aspects of performance in mathematics that are important, and to illustrate how they may be 'measured' in practical assessments of students K-12, both high-stakes and in the classroom. To this end, this paper addresses the following questions:

- Who is assessment for? Students? Teachers? Employers? Universities? Governments?
- What is it for? To monitor progress? To guide instruction? To aid, or to justify, selection?
- What mathematics values should assessment reflect in its tasks and scoring?
- When should it happen to achieve these goals? day-by-day? monthly? yearly? once?
- What will the consequences be – for students, teachers, schools, parents, politicians?

### The Assessment of Problem Solving Skills

#### Hugh Burkhardt (2001)

**Burkhardt, H. (2001b)** *The Assessment
of Problem Solving Skills* in *Assessing Gifted
and Talented Children,* Richardson C (ed), London:
Qualifications and Curriculum Authority.

Problem solving has long been recognised as a key element in performance, in most school subjects as in life itself. One is not educated without the ability to adapt one’s knowledge and skills to tasks and situations that are significantly different from those one has studied in school. The best teachers of gifted and talented students have always challenged them with non-routine tasks that require the student to construct, not just to remember, long chains of reasoning involving connections that are new to the student. Over the last half century such work has become part of the intended curriculum in many subjects in many countries – the UK, the US, Australia, the Netherlands have been among the pioneers while Japan, Taiwan and other Far Eastern systems are increasingly moving in this direction. Enquiry-based approaches to learning science, investigative work in mathematics, and the emphasis on design in the technology curriculum are all examples of this.

### Assessing Problem Solving: Characteristics Of Student Performance On Paper-Based And Computer-Based Tasks

#### Malcolm Swan, Alan Bell

Given the wide-ranging set of tasks described in the paper Domain Frameworks in Mathematics and Problem solving, it is of interest to identify what general student competencies are evoked by these tasks. This paper describes some characteristics of student performance on the Problem-solving element of the World Class Tests of Mathematics and ‘Problem Solving in Mathematics, Science and Technology.

### Assessment Tools for Implementing the Standards

#### Hugh Burkhardt, 2010

Hugh Burkhardt's talk on Assessment Tools for Implementing the Standards, with experiences from the Shell Centre at the conference on Curriculum Design, Development, and Implementation in an Era of Common Core State Standards Washington, DC, August 1-3 2010

### Computer-based Assessment: a platform for better tests?

#### Hugh Burkhardt, Daniel Pead (2002)

**Burkhardt H and Pead D, (2002).** *Computer-based
Assessment: a platform for better tests?* in *Whither
assessment?,* Richardson C (ed), London:
Qualifications and Curriculum Authority.

The potential of the computer as an aid to better assessment has long been thought exciting but has not yet yielded much that is impressive in practice. Once you look beyond simple short items with multiple-choice or other correct–incorrect response modes, there are difficult and well-understood challenges for assessment designers. Nonetheless, the future looks promising. This chapter will explore, and illustrate with some examples, the opportunities and challenges for the computer as a medium for the four key aspects of assessment: task presentation, student working, student response and evaluating student responses. We shall focus on the domain of problem solving in mathematics, science and design technology.

### Computers in Mathematics Assessment

#### Daniel Pead (2007,2010)

Computers have the potential to present a wide range of mathematics assessment tasks to students, but the complications of responding to tasks via a keyboard and mouse present many difficulties, as does the analysis of the responses obtained. Furthermore, few projects will have free reign to reinvent the curriculum or make drastic changes in the style of tasks used. This work details recent research and development undertaken at the Mathematics Assessment Resource Service, University of Nottingham, and focusses on three different computer-based assessment projects.

Summary paper (2007) - PDF (840K)

D. Pead PhD thesis
*On Computer-Based Assessment of Mathematics*
(2010) and live examples

### Curriculum Design and Systemic Change

#### Hugh Burkhardt (2013)

**Burkhardt, H. (2013b).** *Curriculum
design and systemic change.* In Y. Li & G. Lappan
(Eds.), *Mathematics curriculum in school education.*
Heidelberg: Springer.

This chapter describes and comments on the large qualitative differences between curriculum intentions and outcomes, within and across countries. It is not a meta-analysis of research on international comparisons; rather the focus is the relationship between what a government intends to happen in its society’s mathematics classrooms and what actually does. Is there a mismatch? In most countries there is. Why? This leads us into the dynamics of school systems, in a steady state and when change is intended – and, finally, to what might be done to bring classroom outcomes closer to policy intentions. Two areas are discussed in more detail: problem solving and modeling, and the roles of computer technology in mathematics classrooms.

### Domain Frameworks in Mathematics and Problem Solving

#### Alan Bell, Hugh Burkhardt

In many fields there is an essential complementarity between the analytic and the holistic – for example in music, between the rules of melody and harmony and musical compositions. In assessment, the holistic aspect is represented by the assessment tasks themselves, which provide students with “the opportunity to show what they know, understand and can do”; the complementary analytic framework is provided by the specification of the domain of performance to be assessed. While MARS sees the richness of the task set as the key factor in the quality of an assessment, the domain framework is essential for the explanation of the assessment, and for the balancing of the tests. The challenges in designing such a framework are substantial, when you move beyond short technical exercises assessing knowledge and skills to the assessment of substantial reasoning involving higher level skills.

### Draft Content Specifications for the Summative Assessment of the Common Core State Standards for Mathematics

#### Alan Schoenfeld, Hugh Burkhard et. al. (2012)

**SBAC (2012)** *Draft Content
Specifications for the Summative Assessment of the
Common Core State Standards for Mathematics,*
SMARTER Balanced Assessment Consortium, Alan Schoenfeld
and Hugh Burkhardt, principal authors, http://www.k12.wa.us/SMARTER/ContentSpecs/MathContentSpecifications.pdf

### Firmer Foundations for Policy Making

#### Hugh Burkhardt (2008)

This paper outlines a development in evidence-based policy making that will yield outcomes closer to intentions in education and, perhaps, some other policy areas. For known or predictable challenges, the approach offers ministers a choice of well- developed solutions that have been shown to work well; these can replace often-hurried responses that are inevitably speculative and thus unreliable. The key new weapon is a programme of inexpensive, small-scale developments using the kind of “engineering research” methodology that is standard in successful research-based fields.

### From design research to large-scale impact: Engineering research in education

#### Hugh Burkhardt (2006)

**Burkhardt, H. (2006a).** *From design
research to large-scale impact: Engineering research
in education.* in J. Van den Akker, K.
Gravemeijer, S. McKenney, & N. Nieveen (Eds.), *Educational
design research.* London: Routledge.

### High–stakes Examinations that Support Student Learning

####
Paul Black, Hugh Burkhardt, Phil Daro, Glenda Lappan,
Daniel Pead, and Max Stephens

for the ISDDE
Working Group on Examinations and Policy

The recommendations in this paper arise from meetings of this Working Group of ISDDE, the International Society for Design and Development in Education. The group brought together high-level international expertise in assessment design. It tackled issues that are central to policy makers looking for tests that, at reasonable cost, deliver valid, reliable assessments of students’ performance in mathematics – with results that inform students, teachers, and school systems. This paper describes the analysis and recommendations from the group, with references that provide further detail. It is designed to contribute to the conversation on “how to do better”.

This paper is now published in *Educational
Designer* .

### Improving Educational Research: towards a more useful, more influential and better funded enterprise

#### Hugh Burkhardt, Alan Schoenfeld (2003)

**Burkhardt, H., & Schoenfeld, A. H. (2003).**
*Improving Educational Research: towards a more
useful, more influential and better funded enterprise.*
Educational Researcher 32, 3-14.

Educational research is not very influential, useful, or well funded. This article explores why and suggests ways that the situation could be im- proved. Our focus is on the processes that link the development of good ideas and insights, the development of tools and structures for implementation, and the enabling of robust implementation in realistic practice. We suggest that educational research and development should be restructured so as to be more useful to practitioners and to policymakers, allowing the latter to make better-informed, less- speculative decisions that will improve practice more reliably.

### Letter to the Chair of the UK Research Assessment Exercise.

#### Hugh Burkhardt (2000)

**Burkhardt H, (2000)** Letter to the Chair
of the UK Research Assessment Exercise.

I am writing to try to clarify the rules of the game for the RAE in Education. There seems to be a conflict between the admirable general RAE principles and the more specific guidance on Education. This is distorting the pattern of innovative academic work in schools of education in ways that seem incompatible with the innovative skills of many fine academics, the needs of the education system, and with Government policy...

### Making Functional Mathematics Happen

#### Hugh Burkhardt, Alan Bell, Daniel Pead and Malcolm Swan, 2006

These comments are in response to the following recent QCA documents:

- Functional skill subject definitions and On Developing Functional Skills Qualifications;
- Functional Mathematics Standards, along with those for ICT and English;
- Ken Boston's recent speech on Mathematics to ACME.

We will focus on Functional Mathematics (FM), with which we have been creatively involved for more than two decades of research, development and classroom practice.

### Making mathematical literacy a reality in classrooms

#### Hugh Burkhardt (2006)

**Burkhardt, H. (2006b).** *Making
mathematical literacy a reality in classrooms.* in
C. Haines, P. Galbraith, W. Blum, W. and S. Khan (Eds.) *Mathematical
Modelling (ICTMA 12): Education, Engineering and
Economics.* Chichester: Horwood Publishing ISBN:
1-904275-20-6

Modelling of new problems is at the heart of mathematical literacy, because many situations that arise in adult life and work cannot be predicted, let alone taught at school. There are now plenty of examples of the successful teaching of modelling at all levels – yet it is to be found in few classrooms. How can every mathematics teacher be brought to teaching modelling reasonably effectively? This paper discusses how progress may be made, illustrating it with examples of „thinking with mathematics” about everyday life problems of concern to most citizens. It discusses the role that curriculum materials, professional development and various kinds of assessment may play, together with the challenges at system level. There are some reasons to be optimistic.

### Methodological Issues in Research and Development

#### Hugh Burkhardt, 2013

**Burkhardt, H. (2013a).** *Methodological
issues in research and development.* In Y. Li & J.
N. Moschkovich (Eds.), *Proficiency and beliefs in
learning and teaching mathematics - Learning from Alan
Schoenfeld and Günter Törner.* Rotterdam: Sense
Publishers.

This chapter builds on Alan Schoenfeld’s seminal contributions on methodological issues and on our discussions over many years of collaboration and complementary thinking: Alan with the priorities of a cognitive and social scientist with a concern for practice; I with those of an educational engineer who recognizes the importance of insight- focused research for guiding good design. Alan has primarily aimed to bring rigor to research in mathematics education – to move it toward being an “evidence- based” field with high methodological standards. The Shell Centre team has an approach to research that gives high priority to impact on practice in classrooms. The analysis here reflects the challenges that we have faced, individually and together, and their wider implications for research methods in education.

Download PDF (2MB)### Modelling Examples and Modelling Projects

#### Hugh Burkhardt (2012)

**Burkhardt, H. (2012)** *Modelling
Examples and Modelling Projects: Overview* in G.
Kaiser et al. (eds.), *Trends in Teaching and Learning
of Mathematical Modelling:* ICTMA14, DOI
10.1007/978-94-007-0910-2_50

### Modelling in Mathematics Classrooms: reflections on past developments and the future

#### Hugh Burkhardt (2006) with contributions by H. Pollak

**Burkhardt, H., with contributions by Pollak H.O.
(2006c)** *Modelling in Mathematics
Classrooms: reflections on past developments and the
future.* Zeitschrift fur Didaktik der Mathematik.
38 (2)

This paper describes the development of mathematical modelling as an element in school mathematics curricula and assessments. After an account of what has been achieved over the last forty years, illustrated by the experiences of two mathematician-modellers who were involved, I discuss the implications for the future – for what remains to be done to enable modelling to make its essential contribution to the "functional mathematics", the mathematical literacy, of future citizens and professionals. What changes in curriculum are likely to be needed? What do we know about achieving these changes, and what more do we need to know? What resources will be needed? How far have they already been developed? How can mathematics teachers be enabled to handle this challenge which, scandalously, is new to most of them? These are the overall questions addressed.

The lessons from past experience on the challenges of large-scale of implementation of profound changes, such as teaching modelling in school mathematics, are discussed. Though there are major obstacles still to overcome, the situation is encouraging.

### Quantitative Literacy for All

#### Hugh Burkhardt

**Burkhardt, H. (2008).** *Quantitative
Literacy for All* in Madison, B. L. and Steen, L.
A. *Calculation vs Context: Quantitative Literacy and
its implications for Teacher Education.*
Washington, DC: Mathematical Association of America

This paper traces the essential elements of QL—from performance goals, through student learning activities, to their teaching implications and those for teacher education. It takes an engineering research perspective, pointing out that the power of situated learning depends crucially on how well designed and developed the situations are. It sees QL primarily as an end in itself, and a ma- jor justification for the large slice of curriculum time that mathematics occu- pies. It also points out that QL can be a powerful aid to learning mathematical concepts and skills, particularly for those who are not already high achievers.

Download PDF (400K)### Specifying a national curriculum

#### Hugh Burkhardt (1990)

**Burkhardt, H. (1990).** *Specifying a
national curriculum.* In I. Wirszup, & R.Streit
(Eds.), *Developments in school mathematics around the
world 2.* 98-11. Reston, VA: National Council of
Teachers of Mathematics.

The general issues of the dynamics of curriculum change have been discussed elsewhere in this volume.1 Here I want to consider a problem of current concern in both the U.S. and England – that of specifying a national curriculum.2 In the U.S. the work of the National Council of Teachers of Mathematics to define Standards and that of the Mathematical Sciences Education Board to establish a Curriculum Framework come into that category; in England, to the surprise of many of us, the Government has decided to institute a National Curriculum for students from age 5—16, with Mathematics as one of the three “core subjects”. I have been taking part as a member of the Working Group whose task is to make recommendations within a framework defined in its terms of reference. (One year has been allowed for this enterprise, with no full-time staff support!)

In looking at the model we were given, I was naturally led to consider the general problem of specifying a curriculum in ways that will lead to reasonably faithful implementation of the designers’ intentions in most classrooms of the educational system. As far as I can see, in cases where the changes sought are substantial, this central problem does not seem to have been solved anywhere worldwide; in terms of the patterns of classroom learning activity and student performance, a qualitative mismatch between stated intentions and outcomes is the norm. Could we do better? What models are available, and what seem to be their strengths and weaknesses?

This paper is but a commentary on these matters, with a particular focus on the model put forward by the Department of Education and Science (DES) in Britain. Serious empirical research and development needs to be done, though there is little sign of it yet – or even of active recognition of the need for it. This arises partly because it is not yet routine practice to observe classroom activity and performance in detail and, as a result, the picture is unclear.

### Styles of Research: Insight and Impact

#### Hugh Burkhardt

In this paper I look at the roles of different approaches to research in improving the performance of education systems. I compare the approaches characteristic of different traditions – the humanities, the sciences, engineering and the arts – all of which are recognisable in Education. I suggest that, if impact on the quality of education provided to most children, rather than just insight into it, is to become a primary research goal, the engineering approach needs greater emphasis in the balance of research effort, and research credit. Such a shift is likely to have other positive effects. The different characteristics of research, and the roles of strong and weak ‘theory’ are discussed.

### Systematic Classroom Analysis Notation (SCAN)

#### Terry Beeby, Hugh Burkhardt, Rosemary Fraser (1979)

SCAN is a shorthand notation that allows an observer to record live the essence of the dialogue in a mathematics lesson and to relate it to content, teacher objectives, pupil work and the use of resources.

### Towards Research-based Education

#### Hugh Burkhardt (2018)

The goals for STEM education are largely agreed, nationally and internationally. Work over the last 30 years in the research and development community has shown how to develop tools and processes for teaching, assessment and professional development that enable typical teachers to teach much better mathematics and science much more effectively. Why is this not reflected in most classrooms, and what could be done about it?

### World Class Assessment: principles, practice and problem solving

#### Hugh Burkhardt (2001)

**Burkhardt, H. (2001a)** *World Class
Assessment: principles, practice and problem solving*
in *Assessing Gifted and Talented Children,*
Richardson C (ed), London: Qualifications and Curriculum
Authority.

The assessment of problem solving epitomises all the problems of designing and developing high quality assessment. Indeed, any test that goes beyond the routine, covering more than learned facts and procedures in familiar contexts, assesses problem solving – in some sense of that rather-too-widely used phrase. Most assessment makes that claim, though often the tasks the students are asked to do and the aspects of performance rewarded in the scoring schemes do not match those claims. Similarly, the assessment of gifted and talented children simply highlights more general problems. Clearly mundane and narrow assessment tasks are not good enough for them; neither should they be for any child.

Page updated 25 July 2013 - Contact publications@mathshell.com.

This
page is published and maintained by Shell Centre for Mathematical
Publications Ltd.