[IGFmaglist] May evaluation procedure (Was Online registration for May Open Consultations and MAG meeting)
William Drake
wjdrake at gmail.com
Fri May 2 10:27:41 EDT 2014
In previous years we had Excel spreadsheets. Workshop proposals were grouped so one could easily see how many there were per theme and think about the overall program balance. One also could see how many there were per specific topic under each theme, which made it easier to comparatively evaluate and to think about possible mergers; and how many proposals there were from any given organizer. This would have been a nice complement to the web form.
Bill
On May 2, 2014, at 2:49 PM, Marilyn Cade <marilynscade at hotmail.com> wrote:
> i want to agree that we do need some 'guidance' for those whom we ask to merge. Last year, a few folks got sent to merge, and found 'nobody home' at the workshops they were directed to.
> If merging is a standard operating practice, then we have to make sure that it is actually feasible and that the 'receiving workshop group understands what the expectations are that they are open to merging.
>
> M
>
> Sent from my iPad
>
> On May 2, 2014, at 12:04 PM, "Subi Chaturvedi" <subichaturvedi at gmail.com> wrote:
>
>> Dear Janis,Susan and all,
>>
>> Many thanks for your very valuable contributions Susan. Also thank you for addressing the concerns that I had raised on the last call about the new evaluation guidelines and the familiarity that all of us must have with them as MAG members-old and new. Your email was most helpful. And we have made fantastic progress from last years. It is indeed most unfortunate that you won't be able to join us in person Susan. I'd be happy to assist the process by taking down detailed notes so that we can continue to improve the evaluation process and for institutional memory and share them with you and the general MAG list.
>>
>> I am however most interested in how we will will respond to the following issues raised by you before we go into the May meeting :
>>
>>
>> "
>> I believe that development of the Procedure should be guided by what the MAG has agreed upon and adopted. (I have attached that document to this email - there was interest expressed in seeing the document during the last call). In relevant part, during Stage 3:
>> MAG members look at the results to ensure an overall balance of the themes/topics covered;
>> MAG members discuss 5-10 proposals just below the threshold of space availability*** to determine if improvements can be made to overcome proposal deficiencies;
>> Proposers contacted by May 26th and asked to submit revised proposal to address deficiencies by June 16th. If the proposer responds the expectation is they will get a workshop slot;"
>>
>> Before we are ready to publish the final list of accepted proposals on 30th June and even before we get into the May meetings, we as MAG must have clarity on these issues, so as to avoid any arbitrariness in the process and be completely transparent about the decision making process.
>>
>> Janis if you could please address these questions it would be most helpful:
>>
>> 1. Will we have breakout groups this time as well, according to themes?
>> 2. Are we hoping to achieve a thematic balance of workshops or will we follow a straight ratings based approach depending upon the numerical scores achieved by different workshops?
>>
>> Both these parameters significantly impact the final outcome of the successful workshop proposals and the resultant agenda.
>>
>> Personally my suggestion would be to do this as MAG collectively, without breakout groups, though we compromise on ease and working convenience but we minimize ambiguity and subjectivity. This is further reinforced by the fact that remote participation if often difficult and unavailable in breakout rooms. This would also allow the entire community present in the room to participate, without having to choose which thematic group they can attend, since it is a parallel process. There was some concern about these issues in the larger community last year.
>>
>> 3. The MAG members had also discussed a mentoring role for the workshop proposals? Is that limited to the 5-10 proposals, falling just below the threshold level and their review/retrieval? or do we see ourselves taking responsibility for some proposals each and following through with consistent engagement which translates into capacity building and improvements in the overall workshop quality? Also this can't be a forced engagement and should be on a voluntary basis for both parties involved.
>>
>> 4. What is our stand on mergers?
>>
>> How will these new guidelines be operationalised?
>>
>> Since these were excellent steps forward which came out in the form of revised guidelines after the MAG meeting in Feb, we shouldn't loose them in translation.
>>
>> Looking forward to hearing more on the above.
>>
>>
>> warmest
>>
>> Subi
>> ----
>> Subi Chaturvedi
>>
>> Member MAG, UN-IGF (Media & Civil Society)
>> Member MAG, India-IGF (Civil Society)
>>
>> Convenor WG-India IGF
>>
>> Co-Chair, Netmundial, Brazil (Civil Society)
>>
>>
>>
>> Assistant Prof. Journalism & Comm,
>>
>> Lady Shri Ram College for Women (LSR),
>>
>> Delhi University, India
>>
>> Twitter:@subichaturvedi
>>
>>
>>
>> Founder & Hon. Managing Trustee,
>>
>> Media For Change
>>
>>
>>
>> Founder, Chief Mentor & Editor
>>
>> The Saltlist
>>
>> www.thesaltlist.org
>>
>>
>>
>> Independent Documentary Filmmaker, Photographer,
>>
>> Curator, Media Critic & Scholar
>>
>>
>>
>> PhD. Scholar,
>>
>> Indian Institute of Technology (IIT-D), New Delhi
>>
>>
>>
>>
>> On 2 May 2014 14:37, <karklinsj at gmail.com> wrote:
>> Dear Susan,
>> Thank you very much for your suggestions.
>> It is unfortunate that you will not be able to join us in Paris. You presence would be extremely useful. Thank you for the commitment to contribute to the debates remotely.
>> As far as I understand we will have streaming from room IX (plenary). It may not be available from the break-out rooms. Chengetai will check with UNESCO hosts.
>> Hope we ill discuss our Paris meeting modalities during the next conference call on 7 May
>> Best regards
>> JK
>>
>> Sent from Surface
>>
>> From: susan at susanchalmers.com
>> Sent: Thursday, May 1, 2014 1:41 AM
>> To: igfmaglist at intgovforum.org
>>
>> Dear Bill, all,
>>
>> Bill, thank you for raising these questions, and you are not the only MAG member without funding to come to Paris -- I will be joining you as a fellow remote participant.
>>
>> Below some thoughts on the May meeting procedure:
>> This is of course up to our Chair, but I'd suggest that the procedure for the May evaluation meeting ("Procedure") should be developed, agreed upon, and understood before the meeting takes place.
>> In February the MAG adopted a 3-stage process. Stage 1 - Initial Screening* / Stage 2 - Evaluation process to be completed by MAG members by May 2nd** / Stage 3 - Discussion and Finalisation of Programme.
>> I believe that development of the Procedure should be guided by what the MAG has agreed upon and adopted. (I have attached that document to this email - there was interest expressed in seeing the document during the last call). In relevant part, during Stage 3:
>> MAG members look at the results to ensure an overall balance of the themes/topics covered;
>> MAG members discuss 5-10 proposals just below the threshold of space availability*** to determine if improvements can be made to overcome proposal deficiencies;
>> Proposers contacted by May 26th and asked to submit revised proposal to address deficiencies by June 16th. If the proposer responds the expectation is they will get a workshop slot;
>> Final schedule published by June 30th.
>> Based upon these guidelines I do not see any reason for MAG to break into groups, especially since we decided against assigning subsets of proposals to evaluate, to subgroups within the MAG. In other words, it's expected that all MAG members commit to grading all 200-odd workshop proposals. Also, as a matter of logistics, plenary would be much easier for remote participants.
>> The score for each proposal is tabulated by the Secretariat. Per what was agreed in February, proposals are ranked in order to scale to space availability, which means that we need from the Secretariat a blank template for the IGF programme, with a predetermined number of slots. The "5-10 proposals just below the threshold of space availability" are then discussed by the MAG in concert.
>> Perhaps then it would be most appropriate to take an overall look at the programme and determine subject matter balance.
>> As part of the team that kicked off the workshop evaluation process, I had hoped to be present in May to observe how we all worked through it together, to take notes on the strengths and weaknesses of the process, so that the MAG could re-examine and improve the process for next year. Unfortunately I won't be able to, so might I ask if all MAG members wouldn't mind jotting down a few notes as they go through the process? I'd be willing to work with some others in digesting these and suggesting improvements following the IGF in September.
>>
>> I'd really like to hear everyone's thoughts on the above, and would love to work with others on implementation of what we agreed upon. What have I missed?
>>
>> I believe that if we work together we can improve by leaps and bounds the process that so many found disappointing and frustrating last year!
>>
>> Warm regards,
>> Susan
>>
>> * Re Initial Screening - Chengetai, were any proposals rejected during the initial screening?
>> ** Re Grading - Has everyone graded their proposals? :-)
>> *** Re Space Availability - May we have an update on the schedule template?
>>
>>
>>
>> On Tue, Apr 29, 2014 at 1:37 PM, William Drake <wjdrake at gmail.com> wrote:
>> Hi
>>
>> There was some discussion on the last call about the Paris meeting breaking down into groups to consider proposals according to themes. I would have some concerns about that procedure if it were to lead to outcomes like last year’s, in which some groups were very strict and didn’t approve workshops that didn’t surpass the numerical cut-off point, while other groups were more permissive and for various reasons approved workshops that hadn’t made the numerical cut-off. There was also some lack of clarity as to whether we were trying to adjust to achieve some measure of comparability across themes, e.g. since some themes attracted @ 50-60 proposals while others got @ 10, did we need to ‘prune’ more in the former case, etc.
>>
>> So I’m wondering what the thinking is regarding break out sessions by themes, e.g. dis/advantages relative to doing everything together in ‘plenary.’
>>
>> If this approach is to be followed, it’d also be good to know how remote participation would be accommodated. I don’t imagine I am the only MAG member without funding to come to Paris….Actually, it might be worth knowing, when making decisions on how to organize the work, how many of us will in fact be remote?
>>
>> Best,
>>
>> Bill
>>
>>
>> On Apr 28, 2014, at 2:33 PM, Chengetai Masango <CMASANGO at unog.ch> wrote:
>>
>> Dear All,
>>
>> Online registration for the May Open consultations and MAG meeting is now open until the 15 of May.
>>
>> http://www.intgovforum.org/cms/registrations-for-open-consultations-and-mag-meeting-paris
>>
>> I will send out the draft agenda soon.
>>
>> Best regards,
>>
>> Chengetai
>>
>> _______________________________________________
>> Igfmaglist mailing list
>> Igfmaglist at intgovforum.org
>> http://mail.intgovforum.org/mailman/listinfo/igfmaglist_intgovforum.org
>>
>> ***********************************************
>> William J. Drake
>> International Fellow & Lecturer
>> Media Change & Innovation Division, IPMZ
>> University of Zurich, Switzerland
>> Chair, Noncommercial Users Constituency,
>> ICANN, www.ncuc.org
>> william.drake at uzh.ch (direct), wjdrake at gmail.com (lists),
>> www.williamdrake.org
>> ***********************************************
>>
>>
>> _______________________________________________
>> Igfmaglist mailing list
>> Igfmaglist at intgovforum.org
>> http://mail.intgovforum.org/mailman/listinfo/igfmaglist_intgovforum.org
>>
>>
>>
>>
>> --
>> Susan Chalmers
>> Consultant, Internet Policy
>>
>> +1 269 324 4101
>> www.susanchalmers.com
>>
>> _______________________________________________
>> Igfmaglist mailing list
>> Igfmaglist at intgovforum.org
>> http://mail.intgovforum.org/mailman/listinfo/igfmaglist_intgovforum.org
>>
>>
>> _______________________________________________
>> Igfmaglist mailing list
>> Igfmaglist at intgovforum.org
>> http://mail.intgovforum.org/mailman/listinfo/igfmaglist_intgovforum.org
> <Mail Attachment.txt>_______________________________________________
> Igfmaglist mailing list
> Igfmaglist at intgovforum.org
> http://mail.intgovforum.org/mailman/listinfo/igfmaglist_intgovforum.org
***********************************************
William J. Drake
International Fellow & Lecturer
Media Change & Innovation Division, IPMZ
University of Zurich, Switzerland
Chair, Noncommercial Users Constituency,
ICANN, www.ncuc.org
william.drake at uzh.ch (direct), wjdrake at gmail.com (lists),
www.williamdrake.org
***********************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://intgovforum.org/pipermail/igfmaglist_intgovforum.org/attachments/20140502/2636512c/attachment.htm>
More information about the Igfmaglist
mailing list