Difference between revisions of "Google Summer of Code Evaluation"

From BZFlagWiki
Jump to: navigation, search
(don't evaluate based on others)
(Completely replace the evaluation criteria from 2007. There were too many points of confusion and the criteria were poorly defined. This year their are 6 specific categories.)
 
Line 1: Line 1:
The Google mentor evaluation form allows for three positive and two negative evaluation levels (in addition to unevaluated/zero): '''+4 +2 +1 -1 -2'''
+
In order to help consistently, effectively, and efficiently evaluate GSoC student applications, specific evaluation criteria should be used by project mentors. GSoC applications are to be scored on a scale of '''0 to 6''' where 0 is a ''poor candidate'' that meets no evaluation criteria and 6 would indicate a ''perfect candidate'' that fulfills all criteria.
  
In order to help set an initial sorting of applications, the following guidelines should be used for ranking evaluations.
+
In order to score applications, each mentor needs to indicate the project(s) they would be willing to mentor by selecting the "I am willing to Mentor" button on the mentor dashboard.  For scoring an application, apply the below 6 evaluation criteria and evaluate a judgement scoring between 0.0 and 1.0 for each.  Round the cumulative score to the closest integer when ranking.   '''''Only rank applications you are willing to mentor.'''''
  
= Guidelines =
+
= Evaluation Criteria =
* Do not evaluate if you are ambivalent or have ''absolutely no intention or interest in interacting with the student on the task'' being proposed.  That is, '''if you don't care, then don't evaluate'''.
+
* If you do care, especially if you want to be listed as the mentor for a given idea (i.e. you'll willing to help with their final review and evaluation), say so in the comments and '''use the "I'm willing to mentor" button'''.
+
* A '''negative evaluation implies that you don't want the proposal to go forward''' completely regardless of any other submissions.  Negative means not even minimally desirable or insufficiently proposed to be known.
+
* Base rankings on the assertion that there are unlimited slots.  That is, '''rank completely independently''' without a consideration that ranking positively might somehow be "stealing" a slot from another proposal.
+
* Don't rank negative simply because you don't like an idea as much as others.
+
* The numbers will go through a final sort evaluation before being submitted to match up with mentor interests.
+
* Evaluations should '''take overall potential impact into (strong) account'''.  How many people would the proposal prospectively benefit?  Does it have academic/research merit outside of gaming?  Academic applicability was one of our primary application acceptance criteria.
+
* Proposals that don't actually propose ''any'' work, even general maintenance or work in a vague area, are considered inappropriate and not eligible to be evaluated.  If it does propose general maintenance or indicates a vague area of work, then it's merely a negative evaluation.
+
* '''Only evaluate once'''.  Unless you're changing your evaluation to a different value (like from +2 to +4 or +1 to -1), you shouldn't evaluate multiple times per application.  Never evaluate to values other than those predefined (+4, +2, +1, -1, -2).
+
* Do not base your evaluation on the evaluations and comments of others.  Base the application on its own merit as you understand it and with the responses of the student.  Use the criteria contained here.
+
  
= Evaluation Levels =
+
* '''Significance:''' Application Impact and Relevance
* '''+4 => EXCEPTIONAL'''
+
** How many people would the proposal prospectively benefit?
** Exceptionally interesting and/or very appropriate and/or very well thought out proposal with detail.
+
** Does it have benefits outside of our community?
** Should show an understanding of BZFlag development constraints and/or have compelling immediate end-user impact potential.
+
* '''Approach:''' Application Quality and Feasibility
* '''+2 => GOOD'''
+
** How well is their application written?
** Interesting and/or appropriate and/or detailed.
+
** Does what they propose seem reasonable?
** Should show an understanding and be well proposed.
+
* '''Visibility:''' Responsive and Engaging
* '''+1 => MINIMAL'''
+
** Have they been active in the channel?
** Minimally interesting (i.e. "not undesirable")
+
** Do they respond to questions and engage in discussions?
* '''-1 => POOR'''
+
* '''Charisma:''' Personality and Ease-of-Interaction
** Mildly inappropriate or is very poorly defined.
+
** Are they easy to talk to?
** Submission maybe lacks detail on the proposed work.
+
** How well do they interact with others?
* '''-2 => UNDESIRABLE'''
+
* '''Potential:''' Long Term Developer Karma
** The proposal should not even be considered.
+
** Are they excited about working on BZFlag?
** This should mostly be the case for ideas that are unacceptable to the gameplay even with unlimited developer resources.
+
** Perceived chance that they will stick around and keep coding?
 +
* '''Ability:''' Technical Coding Aptitude
 +
** How impressive is their patch?
 +
** Can they actually write code?
 +
 
 +
= Notes =
 +
 
 +
The Google mentor evaluation form allows for three positive and two negative evaluation levels (in addition to unevaluated/zero):  '''+4 +2 +1 -1 -2'''.  The sum of all your rank selections should equal more than 0 and less than 7.  Additionally:
 +
 
 +
* Do not use negative scores except to correct or adjust your own evaluation.
 +
* Do not adjust or compensate for other mentor scores (up or down) even to correct an obvious mistake.
 +
* Do not evaluate them if you don't care for their proposal and/or don't really intend to mentor that student.
 +
* Do not base your evaluation on the evaluations and comments of others.
 +
 
 +
Scores for each application will be '''averaged''' and ''weighted by mentor assignments'' after evaluations are complete and plugged into a spreadsheet for group review before the final ordering.

Latest revision as of 05:23, 9 April 2008

In order to help consistently, effectively, and efficiently evaluate GSoC student applications, specific evaluation criteria should be used by project mentors. GSoC applications are to be scored on a scale of 0 to 6 where 0 is a poor candidate that meets no evaluation criteria and 6 would indicate a perfect candidate that fulfills all criteria.

In order to score applications, each mentor needs to indicate the project(s) they would be willing to mentor by selecting the "I am willing to Mentor" button on the mentor dashboard. For scoring an application, apply the below 6 evaluation criteria and evaluate a judgement scoring between 0.0 and 1.0 for each. Round the cumulative score to the closest integer when ranking. Only rank applications you are willing to mentor.

Evaluation Criteria

  • Significance: Application Impact and Relevance
    • How many people would the proposal prospectively benefit?
    • Does it have benefits outside of our community?
  • Approach: Application Quality and Feasibility
    • How well is their application written?
    • Does what they propose seem reasonable?
  • Visibility: Responsive and Engaging
    • Have they been active in the channel?
    • Do they respond to questions and engage in discussions?
  • Charisma: Personality and Ease-of-Interaction
    • Are they easy to talk to?
    • How well do they interact with others?
  • Potential: Long Term Developer Karma
    • Are they excited about working on BZFlag?
    • Perceived chance that they will stick around and keep coding?
  • Ability: Technical Coding Aptitude
    • How impressive is their patch?
    • Can they actually write code?

Notes

The Google mentor evaluation form allows for three positive and two negative evaluation levels (in addition to unevaluated/zero): +4 +2 +1 -1 -2. The sum of all your rank selections should equal more than 0 and less than 7. Additionally:

  • Do not use negative scores except to correct or adjust your own evaluation.
  • Do not adjust or compensate for other mentor scores (up or down) even to correct an obvious mistake.
  • Do not evaluate them if you don't care for their proposal and/or don't really intend to mentor that student.
  • Do not base your evaluation on the evaluations and comments of others.

Scores for each application will be averaged and weighted by mentor assignments after evaluations are complete and plugged into a spreadsheet for group review before the final ordering.