L`exercice britannique 2008 d`évaluation de la recherche, RAE 2008

L’enjeu
Le “RAE”
Le modèle empirique
Conclusions
L’exercice britannique 2008 d’évaluation de la
recherche, RAE 2008
Ralph Kenna 1 and BB 2
1
Coventry University, England
Nancy Université, France
2
R Kenna & BB, arXiv:1004.3155
Groups
L’enjeu
Le “RAE”
Le modèle empirique
Conclusions
Outline
1
L’enjeu
2
Le “RAE”
3
Le modèle empirique
4
Conclusions
Groups
L’enjeu
Le “RAE”
Le modèle empirique
Conclusions
Concentration ou dispersion des moyens ?
The head of the Russell Group “said that the country’s most successful
institutions should get 90% of the $ billion in research funding that is
allocated to higher education every year. They currently receive around 80%.”
[Telegraph, 23 Oct 2009]
The UK must abandon the idea that excellent research should be funded
wherever it is found and concentrate resources on just 25 to 30 universities,
the chair of the Russell Group of large research-intensive institutions has
argued.
[Times Higher Education, Oct 2009]
But the Director of University Alliance said “It is a principle of ... funding
research, based on excellence, that has driven the improvement in the UK
research base and will continue to do so - not an explicit policy to concentrate
research in particular institutions... It is only in ... a meritocratic system that
we can continue to drive up the efficiency and quality of UK research.”
[Press release, Nov 2009]
Groups ?
So which is best : concentration or competition
L’enjeu
Le “RAE”
Le modèle empirique
Conclusions
The Research Assessment Exercice
Subject areas were scrutinised to assess the proportion of research work
submitted which fell into categories
4* : Quality that is world-leading in terms of originality, significance and
rigour
3* : Quality that is internationally excellent in terms of originality,
significance and rigour but which nonetheless falls short of the highest
standards of excellence
2* : Quality that is recognised internationally in terms of originality,
significance and rigour
1* : Quality that is recognised nationally in terms of originality,
significance and rigour
Unclassified : Quality that falls below the standard of nationally
recognised work
Groups
L’enjeu
Le “RAE”
Le modèle empirique
Conclusions
The Research Assessment Exercice
An example of a UOA quality profile is as follows :
Quality level
pn∗ = % of research
activity
4*
3*
2*
1*
u/c
p4∗ = 15
p3∗ = 25
p2∗ = 30
p1∗ = 20
p0∗ = 10
Based upon quality profiles, a formula is used to determine how research
funding is distributed. In England in 2009/2010 the funding formula is
s = p4∗ +
1
3
p3∗ + p2∗ .
7
7
The funding allocated to a groups of size N is then proportional to S = sN.
Groups
L’enjeu
Le “RAE”
Le modèle empirique
Conclusions
Limites de l’analyse (qui va suivre) et précautions
There are obvious limits to what our statistical analysis can achieve. We
assume RAE scores are reliable and robust
cannot account for links with industry
cannot account for managerial tactics
cannot account for individual calibre
These factors amount to (sometimes considerable) noise. It is remarkable
that despite them we can make quantitative statements.
Groups
L’enjeu
Le “RAE”
Le modèle empirique
Conclusions
L’évaluation des groupes de recherche
100
applied mathematics
s
75
Cambridge
Oxford
Liverpool
50
25
Warwick
QMUL
Coventry
Brighton
UWE
Plymouth
0
0
50
institution
clearly : s = constant ± noise.
Quality is randomly distributed about a mean, with older, prestigious
universities tending to be above and newer ones below.
Groups
L’enjeu
Le “RAE”
Le modèle empirique
Conclusions
L’évaluation des groupes de recherche
One may expect that, on average, a group of size N = 10 is twice as strong
as a group of size N = 5 : The strength of the group g is the sum of the
strengths ag i of its N individuals :
Sg =
N
X
ag i = N āg
i=1
so that the quality of the group is the average strength per head of its
individuals sg = āg .
This is also the philosophy behind RAE rankings.
Ex : if S = strength of a group of size N, one may expect S ∝ N then we
“explain” that
s = constant ± noise.
But this is not the full picture....
Groups
L’enjeu
Le “RAE”
Le modèle empirique
Conclusions
Corrélation taille-qualité
100
applied mathematics
s
75
Warwick Oxford
Liverpool
50
Cambridge
QMUL
25
Coventry
Brighton
UWE
0
0
10 20 30 40 50 60 70 80 90 100
N
Quality s is clearly correlated with groups size N : s = s(N).
What is the nature of this correlation ?
Groups
L’enjeu
Le “RAE”
Le modèle empirique
Conclusions
Les relations entre individus comme moteur de la
qualité
Instead, we have to treat (research) groups as collaborating systems. With
ai
=
strength of individual i,
bi,j
=
strength of interaction between individuals i
and j
the strength of the (small) group is
S = N ā +
N(N − 1)
b̄.
2
However, two-way communication cannot be carried out between every pair
of nodes if N is too large. If N > Nc , N/Nc subgroups form and interact with
strength β,
„
«
N
b̄
β N
−1 .
S = āN + NNc +
Nc
|
{z2
} |2 Nc {z
}
intra-subgroup
inter-subgroup interaction
I.e., we have a different linear dependency of quality s on size N.
Groups
L’enjeu
Le “RAE”
Le modèle empirique
Conclusions
Les relations entre individus. . .
We therefore expect
s=
(
a1 + b1 N
a2 + b2 N
if N ≤ Nc
if N ≥ Nc
b1 ≃ 21 b
b1 ≃ 2Nβ 2 .
c
If N > Nc , we call the group large.
If N < Nc , we call the group small or medium.
Since b2 ∼ 1/Nc2 , the slope to the right is small for large Nc .
100
applied mathematics
We have now explained
s
75
Warwick Oxford
Liverpool
50
Cambridge
QMUL
25
Coventry
Brighton
UWE
0
0
10 20 30 40 50 60 70 80 90 100
N
Groups
L’enjeu
Le “RAE”
Le modèle empirique
Conclusions
Natural Sciences
We start the analysis with the natural sciences
100
100
physics
geography and environment
75
s
s
75
50
25
50
25
0
0
0
50
100
150
0
30
60
N
N
We measure Nk = 13 ± 3
(with R 2 = 53%)
and Nk = 15 ± 2
(with R 2 = 66%).
Compare to the result for applied mathematics where we find
Nk = 6 ± 1
(with R 2 = 74%)
Groups
90
L’enjeu
Le “RAE”
Le modèle empirique
Conclusions
Pure mathematics
100
pure mathematics
75
s
In pure mathematics, no crossover
point was found :
50
25
0
0
20
40
60
N
A linear fit has relatively small slope and high intercept implying all groups
are large.
The smallest group submitted had N = 4. So we may estimate Nc
Nk
<
∼
<
∼
2.
I.e., pure mathematicians work alone or in pairs (size does not matter !).
Groups
4 or
L’enjeu
Le “RAE”
Le modèle empirique
Conclusions
Other subject areas
100
100
French, German, Dutch
and Scandinavian languages
English
75
s
s
75
50
25
50
25
0
0
0
15
30
45
0
30
60
N
N
Nk = 3 ± 1
Nk = 16 ± 2
Groups
90
L’enjeu
Le “RAE”
Le modèle empirique
Conclusions
Stability
This year (2010) the HEFCE funding formula will change to s = p4∗ +
3
9
p3∗ +
1
9
p3∗ .
The effect on the above analysis is
100
100
applied mathematics
(2009 funding formula)
applied mathematics
(2010 funding formula)
75
s
s
75
50
25
50
25
0
0
0
25
50
75
100
0
25
50
75
100
N
N
Nk = 6.2 ± 0.9 (R 2 = 74.3)
Nk = 6.3 ± 1.0 (R 2 = 73.5)
Similarly for physics
Nk = 12.7 ± 2.4 (R 2 = 53.0) → Nk = 12.6 ± 2.3 (R 2 = 54.4)
Groups
L’enjeu
Le “RAE”
Le modèle empirique
Conclusions
University ranking
For applied mathematics, we plot s − s̄ and s − hsi against N, highlighting the
various university groupings
50
50
applied mathematics
applied mathematics
s - s
|
s - <s>
Warwick
QMUL
25
0
-25
Coventry
-50
25
Russell Group
1994 Group
Million+
University Alliance
Unaffiliated
Warwick
QMUL
0
Coventry
-25
-50
0
10
20
30
40
institution
50
0
10
20
30
40
50
institution
Left plot represents deviation of data from global average. St. dev. = 12.6,
range = 52.9
Right plot represents deviation of data from local average. St. dev. = 6.4,
range = 28.8
Groups
L’enjeu
Le “RAE”
Le modèle empirique
Conclusions
University ranking
Same plots for physics :
50
50
physics
s - s
Warwick
|
0
QMUL
-25
-50
institution
s - <s>
physics
25
Russell Group
1994 Group
Million+
University Alliance
Unaffiliated
25
0
QMUL
Warwick
-25
-50
institution
Left plot (global average) : St. dev. = 7.8, range = 32.1
Right plot (local average) : St. dev. = 5.3, range = 25.1
Left line separates groupings ; right line goes through groupings
Groups
L’enjeu
Le “RAE”
Le modèle empirique
Conclusions
The French System (AERES)
Cumulative effect :
80
Paris 7
Exact Sciences
Sciences of Life
Strsbg 1
60
Paris 6
Paris 13
Strsbg 1
Paris 12
Nancy 1
40
Paris 6
Metz
Paris 7
Paris 10
Nancy 1
Paris 12
20
Mulhouse
Paris 10
Paris 8
Metz
0
0
1000
500
Groups
1500
L’enjeu
Le “RAE”
Le modèle empirique
Conclusions
The French System (AERES)
We plot the standardised data for the hard and life sciences in the UK (red)
and France (13 universities of 2008 assessement, blue), where
σ=
ν=
2
1
1
0
0
-1
-1
-2
-2
-3
-1.5
0
N − N̄
σN
3 (b) life sciences
(a) hard sciences
2
σ
σ
3
s − s̄
σs
ν
1.5
3
-3
-1.5
0
Groups
ν
1.5
3
L’enjeu
Le “RAE”
Le modèle empirique
Conclusions
Saturation of Russell Group
Times Higher Education (THE), [29 Nov 2009]
“Analysis questions call to concentrate funding on Russell Group
universities”
100
applied mathematics
75
s
“Research produced by the bulk
of the elite Russell Group of ... is
no better than that generated by
[the 1994 Group], according to a
new analysis. Thomson Reuters ...
compared the percentages of all cited research papers published by
the two groups between 2002 and
2006...”
50
25
0
0
10 20 30 40 50 60 70 80 90 100
N
Clearly, both the Russell Group and the 1994 Group are in fact large and
have hit saturation point.
Groups
L’enjeu
Le “RAE”
Le modèle empirique
Conclusions
Saturation of Russell Group
The same plot for physics
100
Russell Group
1994 Group
Million+
University Alliance
Unaffiliated
physics
s
75
50
25
0
0
30
60
90
120
N
Saturation beyond Nc ≈ 25 explains the Thomson Reuters findings.
Groups
L’enjeu
Le “RAE”
Le modèle empirique
Conclusions
Critical Mass
Critical mass is the minimum size a research team must achieve for it to be
viable in the longer term.
The total strength of a group is
S(N) = ai N + bi N 2
where i = 1 or 2 if N < Nc or N > Nc .
If M new nodes become available where should they be allocated ?
S(N + M) = ai (N + M) + bi (N + M)2 .
The average increase in societal strength is
depends on whether N < Nc or N > Nc .
∆S
M
= ai + bi (2N + M) and
Next we take the limit M → 0. (i.e., we simply maximize the gradient of S(N)).
Groups
L’enjeu
Le “RAE”
Le modèle empirique
Conclusions
Critical Mass
It turns out it is best to support the small/medium group if N > Nk where
Nk =
Nc
.
2
If N < Nk , we call the group small.
If Nk < N < Nc , we call the group medium.
It is best to support the medium group (N > Nk ). Then to support the large
group (N > Nc ) and then the small one.
I.e.,
Nk = critical mass.
Groups
L’enjeu
Le “RAE”
Le modèle empirique
Conclusions
Critical Mass
750
applied mathematics
S
500
250
0
0
5
10
15
20
25
N
Without the crossover point, the steepest gradient condition would lead to
supporting only the biggest among the large groups so that only one group
remains in the end.
Groups
L’enjeu
Le “RAE”
Le modèle empirique
Conclusions
Critical masses
The estimates for the critical masses of 24 (combined) subjects are
Subject
Applied mathematics
Physics
Earth sciences
Biology‡
Chemistry
Agriculture, vet. etc‡
Law†
Architecture & planning
Non-English languages
English
Pure mathematics∗
Medical sciences
Nk = Nc /2
Subject
Nk = Nc /2
6±1
13 ± 3
15 ± 2
10 ± 2
18 ± 7
5±2
15 ± 2
7±2
3±1
16 ± 2
≤2
20 ± 4
Nursing, etc.
Computer science♯
Archaeology♯
Economics
Business
Politics
Sociology
Education
History‡
Philosophy & theology
Art & design‡
Other arts
9±3
25 ± 5
8±2
5±2
24 ± 4
13 ± 3
7±2
15 ± 3
12 ± 3
9±2
12 ± 4
4±1
Groups
L’enjeu
Le “RAE”
Le modèle empirique
Conclusions
Bibliometry vs peer review
Two line fit of Evidence data gives half value for Nc and worse R 2 . One
should prefer peers review than bibliometry.
Groups
L’enjeu
Le “RAE”
Le modèle empirique
Conclusions
Conclusions
Community is greater than sum of it parts ...
... an effect which saturates beyond Nc
It is unwise to judge a group solely on quality profile : small/medium
groups should not be expected to yield same strength as large ones
It is unwise to judge individuals in a group on the basis of the group
strength as this neglects interactions
Need to take size into account to determine which groups are punching
above or below their weight
other factors (e.g. individual excellence) seem to contribute
subdominantly (i.e. noise here !) compared to individual interactions
Groups
L’enjeu
Le “RAE”
Le modèle empirique
Conclusions
Conclusions
Strength is primarily ascribed to two-way communication links.
Universities should facilitate this
Hotdesking bad !
Distance working is bad !
Collaboration is good !
Medium-sized groups should be supported
BUT small groups must endeavour to achieve critical mass
Groups
L’enjeu
Le “RAE”
Le modèle empirique
Conclusions
Natural Sciences
100
100
biology
chemistry
75
s
s
75
50
25
50
25
0
0
0
50
100
150
200
250
0
20
40
60
N
N
We measure Nk = 10 ± 2
(with R 2 = 54%)
and Nk = 18 ± 7
(with R 2 = 62%).
80
Note the large errors for chemistry. The smallest chemistry group had N = 10
(cf N = 1 for physics and N = 3 for biology). So chemistry in the UK is
dominated by large/medium groups, the small ones having petered out prior
to 2008.
Groups
L’enjeu
Le “RAE”
Le modèle empirique
Conclusions
Other subject areas
100
100
philosophy and theology
history
75
s
s
75
50
25
50
25
0
0
0
20
40
60
80
100
0
30
60
90
N
N
Nk = 9 ± 2
Nk = 12 ± 2
Groups
120