[IGFmaglist] Initial comments on the Workshop Evaluation Process

[Dr. Rasha Abdulla]

Dear  MAG colleagues,

I stayed up really (really) late last night, and managed to finish all my
evaluations a couple of hours after the initial midnight deadline (4 am in
Cairo). All except one, that is, WS #418, which was second to last on my
list, and which I realized was submitted in French.

Now that the deadline has been extended, I'm not sure if the Secretariat
wants to route that workshop to a colleague. I did not feel that my French
was efficient enough to allow me to evaluate a workshop properly (although
with my basic level of the language, I don't think it would make the cut
anyway). However, I'm also reluctant to call for having it re-routed
because we did not allow that opportunity to submit in a different language
to everybody, and it would set a precedent of the MAG having evaluated a
workshop in French. I don't think that would be fair.

I'm a third year MAG member, and therefore will not be on the MAG next
year, so I wanted to share with you a few comments so that the process
continues to be improved under new leadership next year.

First of all, as Sala pointed out, we did come a long way from two years
ago, when every MAG member had to evaluate every proposal and give it one
single score. (Imagine having to evaluate 450 proposals this year! How much
time and attention could you afford to each one?) The grading form is very
helpful, and the addition of the options to edit a workshop you already
evaluated, suggest mergers, and include internal comments are all very
useful. Thanks Luis for the great effort you did there!

A few things need to be fine tuned: For example, flash sessions are not
evaluated on diversity, so we need to be able to skip that score without
the form insisting that we fill it out. On a related point, a few of my
sessions were Birds of a Feather, which I thought were supposed to form on
the spot at the IGF. Perhaps next year we need to take that option out of
the session proposal format pull down menu.

My experience was also that it took me considerably more time to grade a
workshop that I was not an expert on than a workshop I was an expert on,
and honestly, I'm not sure it's in the best interest of the proposer to
have the session evaluated by a non-expert. This point needs to be
re-visited for next year.

Quite a few proposers wrote statements to the effect of "Here's the
proposal, here are a few names, I'll add the rest when it's approved." I
think it's imperative that we communicate to the community that ALL
speakers need to be contacted, confirmed, and added to the proposal before
submitting it. It doesn't make sense to evaluate highly on diversity based
on a "promise" that more speakers will be added later. And it doesn't make
sense to put people's names on a proposal when they have not necessarily
been contacted for that proposal. We are then evaluating a very different
proposal than the session that will actually take place. For next year, I
suggest that proposers would have to tick a box next to each speaker's name
that says "Confirmed." A speaker would not be added to the proposal unless
that box is checked, and proposers need to know that the promise to add
more speakers later is not sufficient anymore.

Finally, as the proposals keep increasing in number (which is great,
obviously!), each MAG member's share of evaluations increases. This year I
had 70 proposals. I would say that's still too much given the time and
effort I (and I'm sure all MAG members) take to evaluate each individual
proposal. I would suggest that next year, the number of members evaluating
each proposal be re-visited. I believe now it's at 11 or 12. I would think
9 would be more than sufficient, that would be three of each stake holder
group, so it's still plenty of votes.

Finally, I'm very proud of the work we did in this regards. And I would
once again like to thank the members of the WG on Workshop Evaluations for
their diligence and thoroughness. I would also like to thank Lynn for her
leadership, support, and input, and the MAG at large for the continuous
feedback and the support as we continue to fine tune this process.

All the best.
Rasha

Rasha A. Abdulla, Ph.D.
Professor
Journalism and Mass Communication
The American University in Cairo
www.rashaabdulla.com
Twitter: @RashaAbdulla
<http://twitter.com/rashaabdulla>