6 Preparing for Committee

6.1 Comparative judgement

If your office has established and operated custom sets for particular types of application, all staff who have managed assessment of each set should meet to undertake comparative judgement once individual applications have been assessed.

The group's experience and deliberations should help them to establish a sound grasp of the quality and value for money offered by each application, and so to contribute to positive recommendations during overview assessment about which applications should have priority for funding.

At the same time, comparative judgement can help the team as a whole to develop more a more detailed understanding about expressed local funding needs, which in turn may be reported through overview assessment to RAPs and/or Committees.

Process steps

1. Managing the set

Your office will have determined any custom sets to be used. As part of this process managers will nominate individual members of staff who will work on a set together. You should meet once the set has been established to consider such matters as:

• definitions of beneficiary need and value for money in the context of the set
• how characteristics of the applications in the set might meet local funding priorities
• the qualitative aspects of applications you will examine in comparative judgement; and
• an overall timetable for completing comparative judgement.

This approach will help to ensure consistency during the assessment process. You may also find it useful to involve your country policy officer or regional development officer in these initial meetings.

Comparative judgement should normally take place when you have completed individual judgement on each application in the set, but before you sign off the application.

2. Moderating the set

You should begin by undertaking an overview of the set with your colleague(s), perhaps by reviewing each other. s work. This step might entail changes to your assessment on AMI. You should consider the following:

• Have you and your colleague been consistent in your interpretation of policy?
• Can your assessments stand side by side without contradicting one another?
• Have you been consistent in your allocation of codes?
• Have you been consistent in the application of criteria and in the scores awarded?

3. Preparing for judgement

• Complete the first part of the comparative judgement report, listing each application. s URN, organisation name, amount recommended, location, target group and project work.
• Use that information to identify possible areas of duplication by location, target group and project work. Make a note of these.

4. Establishing priorities for funding

You should refer to local guidance to discuss any established funding priorities. You should list applications in rank order for priority against any criteria used. You might like to consider your set against the following:

• Identified level of need

What need is each project meeting? Are the beneficiaries distinct by location and/or need? Which project is meeting most need? Would funding all applications lead to duplication of provision? If so, which applications should take priority?

• Best fit against local priorities

How far does each application meet stated local funding priorities? Which meet more of them?

• Value for money

How do costs compare? Is there a comparable unit cost?

• Quality of the project

Which projects will deliver the most significant benefits and to what extent do these meet programme priorities? Which are most innovative in approach and delivery? Which best demonstrate good practice across all aspects of delivery?

5. Completing the process

You and your colleague(s) should now be in a position to determine which applications now score most strongly against these and/or any other criteria that you have determined. You should now record a summary of your conclusions on the comparative judgement report, and file a copy of this with each application, and give another copy to your manager.

You may also wish to write up a brief narrative report about your findings, which will help inform overview assessment, RAP and/or Committee decisions, and contribute to your local repository of information.

6.2 Overview assessment

Overview assessment is a process that takes place after full assessment has been completed. Its main functions are to:

• examine consistency of policy interpretation across the team
• ensure that scoring is consistent across the team and in line with guidance provided for scoring
• review the whole portfolio in the light of the funding priorities in order to build a portfolio of recommendations for the regional advisory panel and/or Committee.

Effective overview assessment will bring together the diverse experience and knowledge of the members of the grants team analyse and refine the portfolio of applications. It provides an opportunity for every member of the team to contribute to the development of a portfolio that reflects as far as possible the programme priorities and the range of high-quality applications received. Staff who have worked on sets of applications will in particular have the opportunity to apply their more detailed knowledge of particular types of application.

Overview assessment meetings will review all applications received and assessed in the round, whether they have been recommended for funding or not. The team should take a strategic view of the portfolio, reviewing such matters as the number, size, range, location and aims of the applications received.

It's important to note that the regional manager, operations manager or country director will present at least two versions of the portfolio to the RAP or Committee, one being the "raw" portfolio of original recommendations, and another being the portfolio revised in the light of overview assessment. These committees will also receive a report on the main issues emerging from overview assessment.

Process steps

1. Moderating your own caseload

You should prepare for wider overview assessment by developing an overview of your own caseload. Review the consistency and range of your assessments by considering the following areas and questions:

a) Interpretation

• Have you interpreted policy consistently throughout assessment?
• Can your judgements stand side by side without contradicting one another?
• Do the reject reasons entered match the assessment summary and the feedback that will ultimately be provided to unsuccessful applicants?
• Do you need to revise any of your assessment records in the light of this review?

Please note that you will not be able to change records on AMI or Portfolio Manager once you have signed them off. Your manager will have to do this for you.

b) Coding

• Remembering that the coding you apply is important for identifying this application among others and for wider statistical review, are you satisfied that coding has been applied as accurately as possible?

c) Scoring

• Have you been consistent in the application of the criteria?
• Have you been fair?
• Do you need to review any records to create more accurate assessment records?

2. Reviewing your caseload

You should now consider your caseload as a whole and develop an understanding of it by asking the following questions:

• How many applications have you recommended for funding?
• What proportion of your caseload does this represent?
• What is your average score?
• What is the average size of the bids in your portfolio?

3. Overview assessment

Before the meeting begins, ensure that you have obtained information from Portfolio Manager that summarises the type of information you considered in step 2 across the entire portfolio.

The meeting is likely to consider the portfolio as a whole from the following perspectives:

Scoring

• What are the scoring patterns of each staff member?
• What are their patterns of recommendation and rejection?

You may have to alter scores, text and/or recommendations if your interpretation of policy or scoring differs markedly from that of the team as a whole. Once again your manager will have to alter the records in Portfolio Manager.

Programme priorities

• How well does the distribution of applications recommended for funding reflect published programme priorities?
• Is there an acceptable range of recommendations within each of those? Local funding priorities
• How far does the range of recommended applications reflect any local funding priorities.

Changes made to scores in the light of overview assessment will be recorded in Portfolio Manager rather than AMI.

Please also note that a record of the discussion, decisions and changes arising from overview assessment must be made and retained for audit purposes.

Once any alterations arising from the meeting have been entered into Portfolio Manager by your manager, the assessment process is essentially complete.

Contents - Introduction - 1 - 2 - 3 - 4 - 5 - 6 - 7 - 8 - Appendix: 1 - 2