[IGFmaglist] Proposal for Improving the Evaluation System

Tue Jul 5 19:22:59 EDT 2016

Colleagues
Let me add a few more observations about issues arising from evaluating the Workshop proposals, in addition to those raised by others - which I agree with. Please forgive any repetition of points made better by others.

Format
There seems to have been a high level of failure around the rules for each format. Panels  required the filing of a paper - many people said they wanted to hold a panel, but provided no paper.
Others described their sessions by another title ( I suspect to avoid having to write a paper) then described what seems like a panel session.

Others in several categories filed a paper, but these were sometimes not in English, or were an industry paper on the whole field, not on the topics to be covered in the proposed workshop.

Many people said they were going to hold a debate, but described something quite different.

Other applicants described the format for their session as “other” and had a descriptive name for it - but failed to describe how their new concept would work ( or was different from a panel) .

Sessions described as “break outs” are supposed to build on other work - with no description of that “other work” and no linking to past results and outcomes, assessment of the contribution a session will make is problematic. ( see note about not writing a paper)

Others have commented on the difficulty of grading a BoF vs a Flash session - or something else.

Suggestion - every applicant for each format should have to write a position paper, describing in more detail than the application form the background to their work, where it fits in, how the sessions will help, etc. It may be that we can refine that over time, so that the formats required can be tailored to fit the type of session. More is required about a debate than a Flash session...

Themes and SDGs
Many were not linked to SDGs at all.This marks a major problem - because we don’t set themes with practical limits any topic under the sun can usually be brought under one of our themes or sub-themes without too much semantic gymnastics. The SDGs at least provide a published list to relate ones proposal to. To not link with even an SDG seems weak.

Previous Reports
Another area of major misunderstanding - perhaps on my part.
The report that is sought is the applicant’s report, required to be submitted after running an event at a previous IGF.
This is supposed to detail how the event went- who showed up, by geography, and gender, what was achieved.
I don’t know if it does, but it should also show the session was received by its audience.
Feedback is the breakfast of champions.

What many people linked to was the announcement of a session they had hosted in the program of a previous IGF. All that shows is that they had persuaded a previous MAG to allow them program time. What is required is an assessment of the usefulness, success etc of the event. 

Failure to file a report after an event should be close to an automatic refusal for a slot this time around. Its a known requirement, and a useful one.

The topic of post-event assessment and review has been discussed in the MAG but I suggest we spend some time developing some action plans around it for this year’s IGF.
One of the features of such systems is that people need to be told in advance of any criteria that are going to be used for review - so we should adopt those and publish them well before the sessions themselves. People can also define for themselves the grounds against which they should be reviewed, as there is no one ground that fits all sessions. 

Good ideas going to waste

There is a risk that some good ideas will not be accepted because the authors were unskilled in preparing applications. The obverse is true - some mundane ideas have been well-dressed and will go through because they are promoted by “old hands”.  I hope there is a way to take these and work with the proposers to see if something can be done.

Ditto new applicants - some form of outreach and response might be helpful in bringing on new talent and perspectives that would otherwise get burned off.

Finally, I wonder whether the secretariat could group the sessions together around at least broad themes.
That makes them easier to compare, and to suggest mergers. I suspect some of that has happened, and its helpful.

My regards

Peter

> On 5/07/2016, at 11:10 pm, Marina Sokolova <sokolova at data.minsk.by> wrote:
> 
> Thanks, Douglas,
> you have just formulated the challenges I encountered while evaluating. If we could come to general approaches to these three issues, there will be some time to re-evaluate workshops.
> 
> Best regards,
> Marina
> 
> 2016-07-05 14:02 GMT+03:00 Douglas Onyango <ondouglas at gmail.com <mailto:ondouglas at gmail.com>>:
> Dear all, 
> Thanks for opening this up for discussion. my additions are below:
> 
> 1. Advertisement: I saw several proposal that were entirely revolving around a product/service/project. I think it would be nice for us to put in place some rules around this marketing type proposals -- my take is that we shouldn't allow them as they set a very dangerous precedent.
> 
> 2. The one-size-fits all evaluation criteria: As has already been pointed out in the case of BOFs, not all criteria is applicable to every proposal. To illustrate,  BOFs don't necessarily need a speaker panel per se -- let alone a representative one. However proposals that lack this aspect are supposed rank low :-) We need to figure out a more nuanced evaluation criteria -- or else drop these formats for which we can't objectively evaluate (not my preference)
> 
> 3. Non-global Proposals: I have seen some proposals that were good, but not global in geography -- naturally the panel would also lack the geographical diversity. Most of them came from South America and Asia. So the question is how do we evaluation these proposals if all?
> 
> 
> 
> Regards,
> 
> 
> On 5 July 2016 at 12:42, Hossam (Personal) <hossam at elgamal.com <mailto:hossam at elgamal.com>> wrote:
> +1 Elizabeth, Shita and Omar;
> 
>  
> 
> especially the raised point of diversity for BoF and similar, and also the fact that some workshops may end up being open forum..
> 
>  
> 
> also another observation that some proposers are submitting quite large number of proposals where they highlight themselves as organizers and speakers/moderators which will not be feasible in case many of those workshops are accepted!!
> 
>  
> 
> Finally just few organization proposed flash session not to showcase a project and discuss it but to market their product!! I don’t know if that is acceptable!
> 
>  
> 
> Best;
> 
> Hossam
> 
>  
> 
> From: Igfmaglist <igfmaglist-bounces at intgovforum.org <mailto:igfmaglist-bounces at intgovforum.org>> on behalf of THOMAS-RAYNAUD Elizabeth <elizabeth.thomas-raynaud at iccwbo.org <mailto:elizabeth.thomas-raynaud at iccwbo.org>>
> Date: Tuesday, July 5, 2016 at 11:22 AM
> To: Shita Laksmi <slaksmi at hivos.org <mailto:slaksmi at hivos.org>>, Omar Mansoor Ansari <omar.m.ansari at gmail.com <mailto:omar.m.ansari at gmail.com>>
> Cc: IGF Maglist <Igfmaglist at intgovforum.org <mailto:Igfmaglist at intgovforum.org>>
> 
> 
> Subject: Re: [IGFmaglist] Proposal for Improving the Evaluation System
> 
>  
> 
> Agree with the helpful input from Omar and Shita. A few reflections to add to the mix: <>
>  
> 
> 1)      The new formats – particularly flash sessions and BoF result in content suggestions that may be interesting and of value but if graded against IGF workshop criteria they don’t meet the requirements, particularly for diversity, which are important to the IGF programme overall.
> 
>  
> 
> If we offer these formats as options, however they need to be allowed into the programme without disrupting the balance and quality we seek to arbitrate in this evaluation exercise. I suggest these examples may need to be filtered and considered differently – perhaps we identify how many we could have, we could agree up to that number if those put forward meet an adapted set of criteria? We could also select for diversity among them as opposed to within them, which as we’ve all seen doesn’t work.
> 
>  
> 
> 2)      Some proposals ended up in the workshop line up that appeared to be open forums so it would be great to filter those out ahead of the process also.
> 
>  
> 
> 3)      While considering the future, it would be valuable to have a sense for people’s views on including submissions put forward in other languages (at least UN official) but we need a mechanism to ensure they could be reviewed by a critical mass. Knowing how many MAG members could review Spanish, French etc. could help us assess this. My personal view would be favourable towards seeking to accommodate this as it may be more inclusive for those who communicate in verbal English comfortably but feel more daunted by the rigour of a proposal in written English that will be evaluated.
> 
>  
> 
> Cheers, Elizabeth
> 
>  
> 
> Elizabeth THOMAS-RAYNAUD
> 
> International Chamber of Commerce (ICC)
> 
> Project Director, ICC Business Action to Support the Information Society (BASIS)
> 
> Senior Policy Executive, Digital Economy
> 
>  
> 
> From: Igfmaglist [mailto:igfmaglist-bounces at intgovforum.org <mailto:igfmaglist-bounces at intgovforum.org>] On Behalf Of Shita Laksmi
> Sent: 05 July 2016 00:54
> To: Omar Mansoor Ansari
> Cc: IGF Maglist
> Subject: Re: [IGFmaglist] Proposal for Improving the Evaluation System
> 
>  
> 
>  
> 
> +1 for this. Thank you, Omar.
> 
>  
> 
> It would be also useful to know:
> 
> 1. The flexibility of Host Country on how they accepting many types of workshop format in the sitting/room setting. We gave proposers a lot of format choices (flash, panel, debate, BoF, arm-chair, fish bowl, mix between all types), therefore , flexibility of room setting is also important to know. At least during our meeting in New York.
> 
> 2. There are many flash sessions requested, therefore the number of workshop can be more than usual.
> 
>  
> 
> Thank you.
> 
> Shita
> 
>  
> 
> --
> Shita Laksmi
> Program Development Manager
> OPEN | Transparency and Accountability - Freedom of Expression
> 
> Hivos Regional Office Southeast Asia
> Jl. Kemang Selatan XII/No. 1 Jakarta Selatan 12560, Indonesia, Tel. +62-21 78837577 <tel:%2B62-21%2078837577>/ 7892489, Fax. +62-21 7808115 <tel:%2B62-21%207808115> 
> E-mail: slaksmi at hivos.org <mailto:slaksmi at hivos.org>, twitter: @slaksmi, skype: slaksmi, www.hivos.org <http://www.hivos.org/>
>  
> 
> From: "Omar Mansoor Ansari" <omar.m.ansari at gmail.com <mailto:omar.m.ansari at gmail.com>>
> To: "IGF Maglist" <Igfmaglist at intgovforum.org <mailto:Igfmaglist at intgovforum.org>>
> Sent: Tuesday, July 5, 2016 3:23:47 AM
> Subject: [IGFmaglist] Proposal for Improving the Evaluation System
> 
>  
> 
> Dear all, 
> 
> While evaluating the workshop proposals I came across following issues:
> 
>  
> 
> 1. Proposer is not from a developing world, but flagged as a "developing country."
> 
> 2. Proposer is not a first timer, but flagged as "new proposer."
> 
> 3. As Renata pointed out as well, there was no gender balance in some panels, yet the proposal was submitted for a MAG review. If this is mandatory, then it should be sorted out before a proposal is submitted to MAG for review.  Same goes with diversity in terms of stakeholder and geography. 
> 
>  
> 
> I'd like to propose following solutions for the future: 
> 
>  
> 
> If we can improve the system to be able to validate if proposers are first timer, from a developing country, or if they have submitted their report for previous workshop or not-- this can be done through linking databases. Or through having the secretariat check proposals, for these, first to make sure the information is correct. 
> 
>  
> 
> For the gender, stakeholder, geography, we also need the system to be able to validate entries. Also, if the speaker list on the main proposal page can display participant name, gender, stockholder group, region etc, with the bio being displayed as a popup or dropdown when clicked, would make it convenient for the evaluators to see details. I had to read individual bios to find out gender, stakeholder groups and other details for some of the proposals. 
> 
>  
> 
> Regards, 
> 
> Omar
> 
>  
> 
> 
> _______________________________________________
> Igfmaglist mailing list
> Igfmaglist at intgovforum.org <mailto:Igfmaglist at intgovforum.org>
> http://intgovforum.org/mailman/listinfo/igfmaglist_intgovforum.org <http://intgovforum.org/mailman/listinfo/igfmaglist_intgovforum.org>
> _______________________________________________ Igfmaglist mailing list Igfmaglist at intgovforum.org <mailto:Igfmaglist at intgovforum.org> http://intgovforum.org/mailman/listinfo/igfmaglist_intgovforum.org <http://intgovforum.org/mailman/listinfo/igfmaglist_intgovforum.org>
> _______________________________________________
> Igfmaglist mailing list
> Igfmaglist at intgovforum.org <mailto:Igfmaglist at intgovforum.org>
> http://intgovforum.org/mailman/listinfo/igfmaglist_intgovforum.org <http://intgovforum.org/mailman/listinfo/igfmaglist_intgovforum.org>
> 
> 
> 
> 
> -- 
> Douglas Onyango, PRINCE 2, ITILv3
> UG: +256 776 716 138
> 
> _______________________________________________
> Igfmaglist mailing list
> Igfmaglist at intgovforum.org <mailto:Igfmaglist at intgovforum.org>
> http://intgovforum.org/mailman/listinfo/igfmaglist_intgovforum.org <http://intgovforum.org/mailman/listinfo/igfmaglist_intgovforum.org>
> 
> 
> _______________________________________________
> Igfmaglist mailing list
> Igfmaglist at intgovforum.org
> http://intgovforum.org/mailman/listinfo/igfmaglist_intgovforum.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://intgovforum.org/pipermail/igfmaglist_intgovforum.org/attachments/20160706/e36791a5/attachment.htm>