I. General frame of the research subject and hypotheses

ENIDScienceandTechnologyIndicatorsConference2011
onActorsandnetworksinEuropeanScience
Title:
Academicqualityasaresultofindividualperformancesandcollectiveachievement.
Insidetheblackboxofhighstandingacademicdepartments.
Nameandaffiliationoftheauthor(s)
CatherineParadeise:UniversitéParis‐Est,LATTS,UPEMLV,IFRIS/FR
AntoineSchoen:UniversitéParis‐Est,LATTS,ESIEEParis,IFRIS/FR
AurélieDelemarle:UniversitéParis‐Est,LATTS,ESIEEParis,IFRIS/FR
Conferencetrack
PitfallsandPerilsofInternationallyComparativeHigherEducationIndicators:
experiencesandresultsfrominternationalresearchprojects
I. General frame of the research subject and hypotheses
Excellencehasbecomethemottoofacademicevaluation.Theformalizedmeasurement
ofexcellenceislargelybasedonanalytical,impersonalandpresumablynon‐contextual
performanceindicatorsthatconcentratemostlyonresearch.Itcontrastssharplywith
substantialjudgmentsofprestigethatdonotopentheblackboxofacademicproduction
andarebasedonsocialcirculationofidiosyncraticandsyntheticjudgmentsmadeby
socialnetworks.
Researchpolicyhasanalyzedextensivelytherationalesforpolicymaking,askingwhat,
why,howandwithwhatresultspolicymakersbuildincentives,withthepurposeto
enhancequalityofacademicinstitutions.Yet,ithasrarelyaddressedthetopictheother
wayaround:howdoacademiccollectiveactors,attheirownorganizationallevel,use
resourcesandconstraintsprovidedbytheirvariousenvironments(national,regional,
academicorganization,departmentalandindividuallevelstoenhancetheirown
reputationofquality.
Thisstudyaimsatunderstandingacademicperformanceofhighstandingdepartments
inrelationwiththeirprocessesofresourcebuildingandmaintenance.Itdescribesand
comparesorganizational“localorders”patternsofinputsandanalyzestheirimpacton
qualityasmeasuredbyvariousoutputs.
Theresearchworkisbasedonatwofoldcomparativestudythatcombinesdisciplinary
variety(chemistry,management,history)andnational(France,Italy,Switzerland,the
USA,thePeople’sRepublicofChina)variety.
Thisstudy,thatispartofthePrestEnceresearchproject,willpresentthefirst
intermediaryresultspertainingto3departmentsthathavebeenalreadyinvestigated.
Theanalysisisbasedonthecross‐characterizationofdepartmentsaccordingto
excellenceasmeasuredbythesumofindividualcontributionsandtheavailable(human,
material,social)resources,bothintermsofstructuralcapital(positionsinvarious
disciplinaryorinstitutionalnetworks)andoperationaltools(ofinternalandexternal
coordination)gatheredbyin‐depthinvestigation.
1
II. Methodology
Therisingpracticeandcultureofevaluationhasnourishedalargestreamofresearch
thataimatcharacterisingresearchgroups’(ordepartments’)production,evolutionover
time,etc.Buildingonthislargebodyofknowledgeonresearchgroups’features–e.g.
researchgrouplifecyclesproposedbyBraamR.andvandenBesselaarP.(2010)–this
researchaimsatdiggingonestepdeeperbyanalysingtheinternalavailability,
distribution,useandorganisationofresourceswithinthedepartment.
ForansweringthesociologicalissuesthatdrivethewholePrestEnceproject,this
researchhascollecteddataatthelevelofindividualacademicsbefore“consolidating”it
atdepartmentlevel.
Theresearchers’individualactivityisanalysedalongvariousdimensions,signalling
differenttypesofresourcesmobilisation:
 Academicoutreach:publications(academicandnot,WoSandnotWos),medals
andawards,patents(ifrelevant)
 Otherscientificoutreach:awards,PhDsupervision,regularteachingduties
outsideone’sregularinstitution,researchprojects
 Institutionaloutreach:participationinjournals’editorialboards(akascientific
gatekeeping),involvementinexternaladvisorybodies
Datarelatestoanyactivityperformedafter2000‐exceptdataonpublicationsthatis
withouttimelimit.Thedatacollected(atindividuallevel)withinthedepartmentisfirst
organisedwithinatemplate(seeannex)beforebeingfedinahybriddatatreatment
processthatcombinesautomaticandsemi‐automaticsteps.
2
Otheractivities
Publications
WoSonly
NoWoS
WoS&no
WoS
PresTence
template
Gexftables
(nodes,
edges)
Gexftables
Semi‐automatic
(nodes,
processing
edges)
Gexftables
Semi‐automatic
(nodes,
processing
edges)
Automatic
processing
Semi‐automatic Gexftables
processing
(nodes,edges
FurtherexploitationofSNAdata
table
ResultsofSNA
SNA(SocialNetworkAnalysis)
Outputofdatatreatment
Datatreatment
InputFormat
Typesofactivities
DatatreatmentwithinthePrestEnceproject
Graphs
SNA
SNAdata
table
Profiles,
radars
stat.
analysis.
Graphs
SNA
SNAdata
table
Profiles,
radars,
stat.
analysis.
Gephi
Gephi
III. Data collection and treatment
Despitetheuseofvariousoriginalsemi‐automatedstrategiesfordatatreatment,this
processinvolvesalotofwork.Itcombinesfieldworkanddatamining:information
collectedduringinterviewsiscompletedandcheckedwithdataextractedfrom
databasesandweb‐basedsearches,e.g.intheresearchers’personalwebpages.This
compilationisthenintegratedinastandardresearchertemplate.
Thisfinelygraineddatacollectionproducesahigh‐resolutionpicture.Forinstance,itis
wellknownthatidentificationofacademicproductionatdepartmentlevelisunlikelyto
producerobustresults(evenmoresoinsocialsciences).Withinacademicdatabases,be
itWoSorScopus,publicationsaremostgenerallyattributedtouniversitiesratherthan
departments,andthevariousspellingsusedfordesignatingadepartmentmakesthe
identificationevenmoredifficult.Thesameinadequacyholdstrueforpatents,inwhich
3
case,moreover,theinventionacademicoriginfrequentlydisappears,duetothefactthat
acorporationiscitedastheapplicant.Thelimitationisevenmoreseverefortheother
dimensionstakenintoaccount:informationonexternalinvolvementineditorialboards
oradvisorybodies,PhD,orteachingdutiesarehardlytobefoundatdepartmentlevel.
IV. Social network analysis
Sixtypesofsocialnetworkhavebeenconsidered.Theyrelateto:
 Co‐authoring–distinguishingdepartmentmembers,stafffromthesame
institution(e.g.university)andexternalcollaboratorsforassessingthetypesof
linksthatarebeingactivatedforproducingresearch.
 Accesstojournalsforstudyingtheaccesspatternstoacademicandnonacademic
publicationspace–fortheformerbeyondtheWoSscope‐thatwillbe,bythe
way,assessedtowardsthemoreexhaustivereferenceslistscompiledinthis
research.
 Co‐authoringinstitutionsforidentifyingintegrationinprestigiousinternational
scientificnetwork
 Participationinjournals’editorialboards,akascientificgatekeeping,fortaking
intoaccountthefieldshapingpotentialforactivecognitivespacebuilders.
 Externalteachingformonitoring,concerningteachingduties,integrationin
prestigiousinternationalscientificnetwork
 Involvementinadvisorybodiesusedasamarkerformappingexpertiseinthe
field
Forallthesenetworks,usualcomputationhasbeenperformed,e.g.percentageof
internalvs.externalcollaborations,distributionofbeetweennesscentralitywithinthe
department‐centrednetworks…
V. Activity profiles and statistical analysis
Datacompiledforbuildingthepreviousnetworkshasbeenusedaswellforestablishing
activityprofilesandradarscharacterisingthevariousactors–beitresearchersor
departments.Intheexamplepresentedasannex,individualcontributionsofHEC
strategydepartmentresearchers’(accordingtovarioustypesofactivitiesandcriteria)
havebeenplotted,orderedindecreasingranks.Seeexamplesofactivityprofilesin
annex.
Datacollectedforthispieceofresearch,thatpertainstothreebusinessdepartments(in
France,Italy,USA,willbeanalysedstatisticallyforidentifyingarchetypicalpatterns.
VI. Selected references
 BleiklieI.2008.Excellenceandthediversityofhighereducationsystems,Paper
presentedatthe21thCHERAnnualConference,Excellenceanddiversityin
Highereducation.Meanings,goalsandinstruments,Universitàdeglistudidi
Pavia,Sept.11‐132008.
4
 MusselinCh.&ParadeiseC.2009.Frenchuniversitiesintransition.InParadeise
C.,FerlieE.,BleiklieI.,RealeE.Universitygovernance:WesternEuropean
comparativeperspectives,Dortrecht,Springer.
 SchoenA.etal(2006),ObservatoryoftheEuropeanUniversityMethodological
Guide,StrategicmanagementofUniversityresearchactivities,PRIMEproject,
237p.http://www.prime‐
noe.org/Local/prime/dir/Projects/OEU/OEU%20methodological%20guide%20
v2007.pdf
 BraamRandvandenBesselaarP.,2010,Lifecyclesofresearchgroups:thecase
ofCWTS,ResearchEvaluation,19(3),September2010
 vanderWeijdenIandal.,Implicationsofmanagerialcontrolonperformanceof
Dutchacademic(bio)medicalandhealthresearchgroups,ResearchPolicy37
(2008)1616–1629
 MorrisN.,Thedevelopingroleofdepartments,ResearchPolicy31(2002)817–
833
 SchmochU.etal.,Howtouseindicatorstomeasurescientificperformance:a
balancedapproach,ResearchEvaluation,19(1),March2010,pages2–18
5
VII. Annexes
PrestEncetemplatefordatacollectionatindividuallevel–PrestEnceJune2011
scientific papers
Nom
initiales
prénom
prénom
type publi
article wos ou
scopus = 1
autres articles = 2
livres comme auteur
principal = 3
livres comme
contributeur = 4
rapport = 5
titre article
cle ISI UT
journal
année de
publicatio
n
nom coauteur 1
applicant1
pays
applicant
1
….
pays
institution institution
1
…
1
…
nom coinventeur
1
si
disponible
patents
Nom
initiales
prénom
prénom
numéro d'application titre brevet authority
application
date
prénom
coinventeur
1
pays coinventeur
1
projects
Nom
initiales
prénom
financeur
budget
total du
projet
période du
projet
prénom
nom du projet
statut
coordinate
ur = 1
participan
t=2
initiales
prénom
prénom
Rang ou
metal (1er
nom de la distinction prix, or)
institution
attributair
e
année
pays de
l'institution d'attributi
attributaire on
external involvment
Nom
initiales
prénom
nom de l'institution
type
période de
pays
pays
participan participan participan participan participan
t1
t1
t2
t2
t3
awards
Nom
instance
6
prénom
d'engagement
de
l'institution
d'institutio l'engagem
n
ent
firm = 1
PRO = 2
agence de
fiancemen
t=3
govt
assignmen
t and
political
bodies = 4
instances
univ hors
departeme
nt = 5
scientific gatekeeping = participation à editorial board
d'une revue
Nom
initiales
prénom
prénom
nom de la revue
période de
l'engagem
ent
teaching = hors de l'institution de rattachement
Nom
initiales
prénom
prénom
nom de l'université
d'accueil
pays de
l'université
d'accueil
type
d'institutio
n
période de
l'engagem
ent
nom du candidat
PhD suivi
nationalité
du
candidat
PhD suivi
année de
soutenanc
e du PhD
pays
institution1 institution
1
post PhD
PhD
Nom
initiales
prénom
prénom
période
dans
type
institution institution
1
1
durée
institution pays
dans
institution
institution 2 post
2
PhD
1
7
IndividualtotalcollaborationsfortheHECbusinessdepartment(staff
of15people)–PrestEnceJune2011
8
ActivityprofilesofHECbusinessdepartmentforpublications–
PrestEnceJune2011
Numberofpublications
Degreecentrality
PageRankcentrality
BeetweennessCentrality
9