[IGFmaglist] [Wgwseval] Initial comments on the Workshop Evaluation Process

Thu Jun 28 11:34:46 EDT 2018

Dear Rasha, dear Lynn, 

I do think a review of this years process would be very beneficial. I'd be happy to support this with a lot of ideas from the evaluation fresh in my mind. Would it be possible to schedule a meeting either on Wednesday or Thursday evening when we are in Geneva together?

Kind regards
Jutta

> Am 28.06.2018 um 17:23 schrieb Lynn St.Amour <Lynn at Internet-Matters.org>:
> 
> Thank you Rasha and Ben,
> 
> many very helpful comments.  Rasha, could I ask you to perhaps hold a WG-PREP meeting on the margins of the MAG meeting in Geneva, or a conference call shortly thereafter?   The sole purpose would be to review this years process and provide an updated recommended process to the MAG.   Doing it then would ensure it was done while still fresh in everyone’s mind.   
> 
> Providing an updated recommendation would also significantly improve the handover process from one year to the next for the MAG and for the WGs.
> 
> Thanks in advance for considering this.
> 
> Lynn
> 
> 
> 
>> On Jun 28, 2018, at 10:39 AM, Ben Wallis (CELA) <bewallis at microsoft.com <mailto:bewallis at microsoft.com>> wrote:
>> 
>> Thanks Rasha, I think these are very helpful observations and suggestions. 
>> 
>> I wanted to add one of my own. Assigning us workshops on a particular theme makes it viable for us to look for potential mergers. And I think it is good to already get us to think about possible mergers as we evaluate the workshops assigned to us, particularly given how many proposals there are relative to available slots. 
>> 
>> To do so effectively, I felt like I needed to try and group them together, e.g. to find and read consequentially all the IoT proposals, to do the same with all the GDPR-related proposals, all the CLOUD Act proposals, all the cybersecurity capacity-building proposals etc etc. I did that both by looking through those sharing the same sub-theme and by looking for key words such as GDPR, CLOUD Act, WHOIS. 
>> 
>> That way, I would be reading proposals which could potentially be merged within 30 minutes of each other rather than randomly coming across them over hours or days, when my memory of potentially mergeable proposals will be much less fresh. 
>> 
>> (I hope I managed to explain that clearly...). 
>> 
>> The thing is that it took me quite a bit of time to manually group them in this way. So I wonder if there's anything that can be done next year to help us group them in this way. That might be something clever that Luis can do, or it might be something that can be achieved by each of us using functionality in widely-available software. 
>> 
>> Maybe I can just leave that idea with the Secretariat and WG on Workshop Evaluation to see if it's something that could be addressed for next year.
>> 
>> Ben
>> 
>> 
>> 
>> -----Original Message-----
>> From: Dr. Rasha Abdulla [mailto:rasha at aucegypt.edu] 
>> Sent: Thursday, June 28, 2018 5:04 AM
>> To: IGF Maglist <igfmaglist at intgovforum.org>; wgwseval at intgovforum.org
>> Subject: [IGFmaglist] Initial comments on the Workshop Evaluation Process
>> 
>> Dear  MAG colleagues,
>> 
>> I stayed up really (really) late last night, and managed to finish all my evaluations a couple of hours after the initial midnight deadline (4 am in Cairo). All except one, that is, WS #418, which was second to last on my list, and which I realized was submitted in French.
>> 
>> Now that the deadline has been extended, I'm not sure if the Secretariat wants to route that workshop to a colleague. I did not feel that my French was efficient enough to allow me to evaluate a workshop properly (although with my basic level of the language, I don't think it would make the cut anyway). However, I'm also reluctant to call for having it re-routed because we did not allow that opportunity to submit in a different language to everybody, and it would set a precedent of the MAG having evaluated a workshop in French. I don't think that would be fair.
>> 
>> I'm a third year MAG member, and therefore will not be on the MAG next year, so I wanted to share with you a few comments so that the process continues to be improved under new leadership next year.
>> 
>> First of all, as Sala pointed out, we did come a long way from two years ago, when every MAG member had to evaluate every proposal and give it one single score. (Imagine having to evaluate 450 proposals this year! How much time and attention could you afford to each one?) The grading form is very helpful, and the addition of the options to edit a workshop you already evaluated, suggest mergers, and include internal comments are all very useful. Thanks Luis for the great effort you did there!
>> 
>> A few things need to be fine tuned: For example, flash sessions are not evaluated on diversity, so we need to be able to skip that score without the form insisting that we fill it out. On a related point, a few of my sessions were Birds of a Feather, which I thought were supposed to form on the spot at the IGF. Perhaps next year we need to take that option out of the session proposal format pull down menu.
>> 
>> My experience was also that it took me considerably more time to grade a workshop that I was not an expert on than a workshop I was an expert on, and honestly, I'm not sure it's in the best interest of the proposer to have the session evaluated by a non-expert. This point needs to be re-visited for next year.
>> 
>> Quite a few proposers wrote statements to the effect of "Here's the proposal, here are a few names, I'll add the rest when it's approved." I think it's imperative that we communicate to the community that ALL speakers need to be contacted, confirmed, and added to the proposal before submitting it. It doesn't make sense to evaluate highly on diversity based on a "promise" that more speakers will be added later. And it doesn't make sense to put people's names on a proposal when they have not necessarily been contacted for that proposal. We are then evaluating a very different proposal than the session that will actually take place. For next year, I suggest that proposers would have to tick a box next to each speaker's name that says "Confirmed." A speaker would not be added to the proposal unless that box is checked, and proposers need to know that the promise to add more speakers later is not sufficient anymore.
>> 
>> Finally, as the proposals keep increasing in number (which is great, obviously!), each MAG member's share of evaluations increases. This year I had 70 proposals. I would say that's still too much given the time and effort I (and I'm sure all MAG members) take to evaluate each individual proposal. I would suggest that next year, the number of members evaluating each proposal be re-visited. I believe now it's at 11 or 12. I would think
>> 9 would be more than sufficient, that would be three of each stake holder group, so it's still plenty of votes.
>> 
>> Finally, I'm very proud of the work we did in this regards. And I would once again like to thank the members of the WG on Workshop Evaluations for their diligence and thoroughness. I would also like to thank Lynn for her leadership, support, and input, and the MAG at large for the continuous feedback and the support as we continue to fine tune this process.
>> 
>> All the best.
>> Rasha
>> 
>> Rasha A. Abdulla, Ph.D.
>> Professor
>> Journalism and Mass Communication
>> The American University in Cairo
>> www.rashaabdulla.com
>> Twitter: @RashaAbdulla
>> <http://twitter.com/rashaabdulla>
>> _______________________________________________
>> Igfmaglist mailing list
>> Igfmaglist at intgovforum.org
>> To unsubscribe or manage your options please go to http://intgovforum.org/mailman/listinfo/igfmaglist_intgovforum.org
>> 
>> _______________________________________________
>> wgwseval mailing list
>> wgwseval at intgovforum.org <mailto:wgwseval at intgovforum.org>
>> To unsubscribe and other options please go to http://intgovforum.org/mailman/listinfo/wgwseval_intgovforum.org <http://intgovforum.org/mailman/listinfo/wgwseval_intgovforum.org>
> 
> 
> _______________________________________________
> Igfmaglist mailing list
> Igfmaglist at intgovforum.org <mailto:Igfmaglist at intgovforum.org>
> To unsubscribe or manage your options please go to http://intgovforum.org/mailman/listinfo/igfmaglist_intgovforum.org <http://intgovforum.org/mailman/listinfo/igfmaglist_intgovforum.org>