Evaluation of research systems

Evaluation of Research Systems

Rigas Arvanitis and Roland Waast

ESTIME Coordinating team

The best promotion of research relies on enhancing quality

  Morocco has multiplied its scientific production by four in 10 years....
  Many South American countries have improved the efficiency of their research....
  In all cases, research has been evaluated at the national level and quality has been enhanced.

Example: Morocco

Tools for the evaluation:
  Bibliometrics : general and detailed
  Surveys by questionnaires
  Field work
  Evaluation by foreign experts

Evaluation is a process...

  Phase 1 :Preparation : bibliometrics / Information gathering / budgets, personnel, higher education
  Phase 2: Preliminary analysis : specific topics/fieldwork/questionnaires
  Phase 3: Expert selection
  Phase 4: Expert evaluation
  Phase 5: Restitution at local and national level

...that needs independent and professional advice

• The team needs to be independent from political interference and coordinated with national and local authorities
• The process needs to be transparent
• Results need to be fully discussed
  • Undesirable results need also to be discussed

Evaluation needs some basic conditions to be fulfilled

• Political willingness
• Openness to foreign eyes
• A clearly defined budget
• A small operational team
• Coordination, not imposition

Expected outputs -1

  A quasi exhaustive review of the research system at the national and regional level
— By disciplines
— By institutions/at the level of departments and laboratories
— By geographical location

Expected outputs -2

  A description of institutional capabilities
— Infrastructures
— Budgets and funding
— Size and organization
— International collaborations
— Capabilities to produce and circulate knowledge
— Capabilities for use of research

Expected outputs -3

  A quality assessment by scientific experts
— Internationally recognized experts
— National experts as associates
— Experts with a view on uses of research
— With no previous linkages with the national teams
— Accepted by University authorities and research heads
— Assessment made by on site visits to labs

Expected outputs -4

  A wide discussion of the quality assessment by scientific experts and other results
— As open as possible
— Including as many experts as possible
— With no previous linkages with the national teams
— Assessment made by on site visits to labs and meetings

Benefits for research

  The quality of the evaluation will reveal strong points and shortcomings
  The implication of all parts of the research system will permit to reveal ways to enhance research
  The large discussions will validate results and suggest actions

Benefits for further action

  The policy process will benefit from first class information, socially accepted
  Accepted national programmes
  A certification process for high achieving labs and teams
  An accepted identification of priority actions