[IGFmaglist] Suggested workshop grading and selection process
Zeina BOUHARB
zbouharb at yahoo.com
Tue Jun 14 05:23:31 EDT 2016
Hello,
I still think it is better to develop the grades section in the evaluation sheet and to assign weight to each criteria based on importance
I propose this order of importance
|
1 - Relevance (IG problem/question/challenge to be addressed)
2 - Diversity (gender, geography, stakeholder group, perspective)
3- Completeness (list of speakers, participating individuals and organizations, or a description of how different stakeholder perspectives will be represented, rapporteur)
4- New Format (Break-out Group Discussions, Debates, Flash Sessions, Birds of a Feather, Roundtables and Other formats-online participation)5- New proposers might be encouraged by an additional 1 point to their score. |
By this we can avoid equal grading and whenever the number of workshops accommodated is defined, the top ranked will be accepted.
Attached is a draft of evaluation sheet. we can add the weights and develop it to get less subjective results.
Zeina BOUHARB
OGERO Telecom/ MoT
Beirut -Lebanon Mobile: + 961 3876436
From: Flávio Rech Wagner <flavio at inf.ufrgs.br>
To: Dr. Rasha Abdulla <rasha at aucegypt.edu>; "Chalmers, Susan" <schalmers at ntia.doc.gov>
Cc: MAG-public <igfmaglist at intgovforum.org>
Sent: Tuesday, 14 June 2016, 4:24
Subject: Re: [IGFmaglist] Suggested workshop grading and selection process
Dear Rasha I share your concern. But I don't see how we can promote balance, in any dimension we may consider relevant (stakeholder group, theme, diversity, etc.), without approving proposals A, B, C, ... with average scores that are lower than the average scores of other proposals X, Y, Z, ... that will have to be rejected, so that we open room for A, B, C, ... in the final selection. And yes, this will be a subjective process. We will have to decide that, for instance, a proposal A with average score equal to 3.2 will be accepted, while a proposal X with average score equal to 3.6 will not be accepted, because proposal A brings a diversity that is missing in the other already accepted proposals, while proposal X covers the same subject as other better ranked proposals. This may be subjective, but it is not necessarily unfair if we consider the overall goal of the IGF, which calls for diversity.
And remember that, if we consider that proposal X is indeed good, it will be a good candidate for a merger with other better ranked proposals on the same subject. Best regards Flavio
Em 13/06/2016 17:53, Dr. Rasha Abdulla escreveu:
Dear Flavio,
Thank you for your proposal. I trust that each MAG member inadvertently adopts a similar ranking in mind when assigning one score to each proposal, because the ranking is from 1 to 5, so naturally 5 is excellent (must accept), 1 is bad (must reject), and the in between scores are ranked from very good to acceptable to poor, etc.
However, the question arises in the between categories, particularly the 3s and the 4s, since I would assume most of the workshops (after initial screening by the secretariat) will fall into this range. That's why it's very difficult to arrive at a good ranking system with only one overall score assigned to each workshop without breaking that score down. The question will be: if 150 proposals that score 4 lack diversity, and if most proposals that has some diversity score 3 (I would suppose more experienced proposers could submit better proposals), are we going to choose a good number of 3s over 4s to ensure diversity? And what would justify doing that in these instances? i.e., how many of the proposals which have scored 3 and have diversity would we then accept, and at the expense of which proposals that scored 4s? It's a subjective and unfair process, and I hope the MAG would move to discuss/test the other system which we set aside for lack of time this time around to modify it as we see fit and adopt it for next year to bring about some kind of objectivity. For this year, we'll just have to stick to the process as it is and do what we can.
Best regards.
Rasha
Rasha A. Abdulla, Ph.D.
Associate Professor and Past Chair
Journalism and Mass Communication
The American University in Cairo
www.rashaabdulla.com
Twitter: @RashaAbdulla
<http://twitter.com/rashaabdulla>
On Mon, Jun 13, 2016 at 10:33 PM, Chalmers, Susan <schalmers at ntia.doc.gov> wrote:
Dear Douglas, colleagues,
Flavio has sent out excellent advice for the individual grading process. Unless I am missing something, it does not necessarily conflict with your concern about thematic balance.
I would suggest that thematic balance be addressed during the plenary conversation on July 14th, or otherwise that you propose an approach that can easily be incorporated into Flavio's advice.
Sincerely,
Susan
-----Original Message-----
From: Igfmaglist [mailto:igfmaglist-bounces at intgovforum.org] On Behalf Of Douglas Onyango
Sent: Monday, June 13, 2016 3:40 PM
To: flavio at inf.ufrgs.br
Cc: MAG-public
Subject: Re: [IGFmaglist] Suggested workshop grading and selection process
Hi Flavio,
Your proposal looks reasonable, however my worry is that it reduces the priority/weight of diversity of proposals.
If implemented as-is, your proposal could result into us completely excluding certain topics/themes and stakeholder groups because IPv6 proposals, for examples, didn't stake up well compared to SDGs.
I don't know how diversity has been maintained in the past, but I would like to see a grading mechanism that produces a diverse array themes/topics by maintaining the balance between them.
Regards,
On 13 June 2016 at 21:40, Flávio Rech Wagner <flavio at inf.ufrgs.br> wrote:
> Dear MAG colleagues
>
> [ Note: I sent the same message below last year. I just adapted the
> numbers for 2016.]
>
> We can accept only around 100 of 273 proposals. Therefore, we need to
> give scores that clearly make a difference between the proposals. If
> we give a score of 4 or 5 to all proposals, we will end up with no
> objective criteria to decide which 100 proposals are to be accepted.
>
> Each of us must remember that only around 36% of the proposals can be
> accepted. I would thus strongly suggest that each of us does not
> assign a score of 4 or 5 to more than 36% of the proposals s/he is
> evaluating, although MAG members evaluating only a small subset of the
> proposals may find more than 36% of very good proposals in her/his sample.
>
> For the same reason, the scores must have clear different meanings.
> For the final decision in our meeting in New York, we need to
> differentiate the proposals in order to have a clear, objective
> procedure to decide on the final selection.
>
> I thus suggest the following correspondence between scores and
> selection
> criteria:
>
> 5) proposals that MUST be accepted, because they are excellent and
> cannot be rejected,
> independently of any other conditions;
>
> 4) proposals that COULD be accepted, because they are good, but
> maybe some of them
> are not kept in the final selection if there are no more time
> slots available or because
> we want to achieve a better balance among stakeholder groups
> and among relevant themes,
> or give priority to first-time proposers and to developing
> countries, so that unfortunately
> we have to select other proposals;
>
> 3) proposals that are not so good as the ones in groups 4 and 5
> above, but some of
> them can still be included in the final selection because we
> want to achieve a better balance
> among stakeholder groups and among relevant themes, or give
> priority to first-time proposers
> or to developing countries;
>
> 2) proposals that are interesting but not so good as the ones in
> groups 3, 4 and 5 above,
> so that unfortunately we cannot keep them in the final
> selection;
>
> 1) proposals that cannot be accepted because they either have
> problems in their
> presentation or do not fit well into IGF.
>
> I thus suggest that MAG members give their scores based on the above
> rationale, trying to remember that only around 36% of the proposals
> can be accepted and that the scores given by each of us must reflect
> this "acceptance rate".
>
> I, personally, would try to classifiy into bins #4 and #5 at most 36%
> of the proposals I will evaluate. I would try to assign more than 40%
> of the proposals to bins #1 and #2. I would classify the remaining 20%
> - 25% of the proposals into bin #3, to be possibly traded off against
> proposals in bin #2 during our meeting in New York.
>
> If each of us adopts the same rationale, then the procedure to be
> followed in the meeting in New Work could be as follows, after
> calculating the average score of all proposals (and again supposing we
> can accept 100 proposals, or around 36% of all proposals):
> - identify 20% to 30% of the proposals that fit in bin #5 (those with
> average scores possibly above 4.0) and accept them;
> - identify 30 to 40% of the proposals that fit in bins #1 and #2
> (those with average scores well below 3.0) and do not accept them;
> - complete the acceptance of 100 proposals by assessing proposals in
> bins #4 and #3, giving priority to proposals with higher average
> scores, but seeking a balance among stakeholder groups and among
> relevant themes and giving priority, whenever possible, to first-time
> proposers and to developing countries.
>
> Best
>
> Flavio
>
>
> --
> Prof. Flávio Rech Wagner Tel: +55-51-3308 9494
> Universidade Federal do Rio Grande do Sul Fax: +55-51-3308 7308
> Instituto de Informática E-mail: flavio at inf.ufrgs.br
> Porto Alegre, Brasil URL: www.inf.ufrgs.br/~flavio
>
>
> _______________________________________________
> Igfmaglist mailing list
> Igfmaglist at intgovforum.org
> http://intgovforum.org/mailman/listinfo/igfmaglist_intgovforum.org
>
--
Douglas Onyango, PRINCE 2, ITILv3
UG: +256 776 716 138
_______________________________________________
Igfmaglist mailing list
Igfmaglist at intgovforum.org
http://intgovforum.org/mailman/listinfo/igfmaglist_intgovforum.org
_______________________________________________
Igfmaglist mailing list
Igfmaglist at intgovforum.org
http://intgovforum.org/mailman/listinfo/igfmaglist_intgovforum.org
_______________________________________________
Igfmaglist mailing list
Igfmaglist at intgovforum.org
http://intgovforum.org/mailman/listinfo/igfmaglist_intgovforum.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://intgovforum.org/pipermail/igfmaglist_intgovforum.org/attachments/20160614/ef5a6a10/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: EvaluationSheet.xlsx
Type: application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
Size: 13699 bytes
Desc: not available
URL: <http://intgovforum.org/pipermail/igfmaglist_intgovforum.org/attachments/20160614/ef5a6a10/attachment.xlsx>
More information about the Igfmaglist
mailing list