[IGFmaglist] Proposals for tomorrow: workshop evaluation process & content organisation

[Shahram Soboutipour]

Dear Susan, Dear Colleagues

 

First of all, thank you for the information shared till now. 

I would like to add some more points, trying to keep them realistic and tangible, while I agree with most of the points mentioned by colleagues:

 

1.       I personally had difficulties in evaluating the “diversity of perspectives”, when trying to score last years’ workshop proposals. I think we need better definitions on the evaluation criteria, by choosing betters criteria or choosing better words and descriptions for them.

2.       I do support the idea of adding more points for:

a.       Is the topic new (freshness)? (+1 and even more, because a repetitive topic may not add any added value to IGF)

b.      Is it proposed from a developing country? (+1)

3.       I support the idea of limitation on how many proposals one can make, plus more limitation for MAG members.

4.       I also support the idea that some criteria should have more weight in the scoring system. My opinion: the relevancy to the main themes is more important to the others.

5.       I think we can use better technical formats on our scorings. I mean for example we can use online services like google docs (forms) with automatic outcomes for scoring platform, instead of distributing an excel sheet which at the end there is a need for hours of human work to summarize them, with the risk of human mistakes. This will bring more flexibility for us.

6.       The data provided to the evaluator (at a glance) at the time of scoring is also important. Usually it is not possible to evaluate a proposal based on its title. On the other hand, it is also very hard to open another page for each proposal, or to include the full description of the proposal on the same page. The idea is to ask the workshop proposer to answer directly on “why does he/she think his/her proposal meets a specific criteria” in 1-2 sentences. And the evaluator must evaluate the proposers answer on this question, rather than the main proposal. I think it will be much easier to evaluate. This can work for some criteria (not all), and on the other hand helps more transparency on MAG evaluation process. 

7.       And finally an idea which would take courage to be accepted:
why not providing the opportunity for public scoring on the received proposals with a lower weight than MAG members (e.g. 20/80)?
technically it is easily possible through the use of online services (as described in #5), and I think it is also possible to do this public evaluation in a way that makes less bios (I mean fake answers) through the same systems. It will also help the transparency of the evaluation process.

 

Regards,

Shahram Soboutipour | IT Business Advisor | cell: +98 913 141 6626 | fax: +98 21 8978 7875 |  <mailto:shahram at soboutipour.com> shahram at soboutipour.com

 

From: Igfmaglist [mailto:igfmaglist-bounces at intgovforum.org] On Behalf Of Michael R. Nelson
Sent: Wednesday, February 19, 2014 7:09 PM
To: Susan Chalmers; igfmaglist at intgovforum.org
Subject: Re: [IGFmaglist] Proposals for tomorrow: workshop evaluation process & content organisation

 

Susan, 
I want to thank you for an incredibly useful e-mail.

A few reactions.
1) When the only choices are 0 or 1, there is  too much randomness added. 
2) We should give more points for more important criteria (e.g. Quality of proposal and speakers)
3) Having different grades for different types of diversity will lead to perverse incentives. With the current criteria, you can get twice as many points for diversity as for quality. We MUST have diversity of perspectives. But we cannot expect geographic, gender, stakeholder diversity in each panel (and if we do, we end up with panels with 9 panelists).  So, provide 0, 1, or 2 points for diversity of perspectives and 0, 1, or 2 for other types of diversity (not all of which were listed)
4) I would let people earn a point for "freshness". Is the topic new and timely?
5) point for track record. Is people have proposed panelists who didn't show, they get a 0.

Mike Nelson

Sent from my Windows Phone

  _____  

From: Susan Chalmers <mailto:susan at susanchalmers.com> 
Sent: ‎2/‎19/‎2014 11:35 AM
To: igfmaglist at intgovforum.org
Subject: [IGFmaglist] Proposals for tomorrow: workshop evaluation process & content organisation

Dear colleagues, 

 

Before we enter into the second day of our meeting, I’d like to offer some thoughts on two items: 1) content organization, and 2) workshop proposals & selection. My commentary is built in very large part upon the contributions and ideas of others.

 

 

1.         Organising IGF Content Online

 

For years IGF has been webcast, videoed, transcribed, tweeted, blogged and generally recorded - building up a rich corpus of material about Internet governance. Yet, much of this material is scattered across the web, and is under-utilised, leading to sessions year-on-year at the IGF that fail to build on work from previous years. 

 

The Friends of the IGF project (“FoIGF”) was launched last year with the aim of bringing all that content together, to archive, organise and transform it into a resource that can improve future Internet Governance debates.  

 

The usefulness of the site depends on how the content is organized and enriched with metadata. There is an opportunity to facilitate this process during the IGF workshop submission process.

 

PROPOSAL: To require workshop proposers to provide tags associated with their workshops on their proposals.

 

(Note that the tag options will depend on the outcome of the MAG conversation on streams or tracks)

 

 

2.         Workshop Proposals & Selection

 

What should submitters include within their proposal? Under which transparent set of evaluation criteria will the MAG accept or reject the proposal?

 

Proposal Criteria

An important touchstone for this discussion is found here: http://www.intgovforum.org/cms/component/content/article/1308. 

 

In sum, workshop organisers should prioritise diversity in speaker selection – in gender, perspective, and geography. Whilst early confirmation of speakers is encouraged, it is difficult if not unrealistic to ask someone to confirm their potential availability months in advance. What’s more important is that the organizer, in his or her proposal, undertake that the speaker, whoever they may be, will represent a certain perspective or speak to a certain topic. 

 

Proposals must be well prepared. The workshop must focus on a relevant Internet Governance issue or issues. Workshop organisers would do well to plan for remote participation with equal care and attention as in-person participation, including a description of remote outreach plans within their proposal. Workshop format and desired room set-up must be specified, and the proposal should set clear expectations as to the planned degree of attendee engagement, on the spectrum from a little (e.g. a lecture) to a lot (e.g. a free-for-all discussion). In my opinion, the IGF programme should offer attendees different choices for engagement; the meeting is culturally heterogeneous – some people are comfortable taking the floor, others less so. 

 

What is not clearly communicated at present is that certain sessions will not be evaluated against the workshop criteria – the Open Forum, for example. 

 

Establishing appropriate criteria is a relatively simple task. Establishing a fair and transparent system for assessing workshop proposals against said criteria is a greater challenge.

 

Selection Process

List discussion has centered around two approaches. 

 

PROPOSAL: That MAG members review, as a starting point, the two general approaches outlined below in advance of tomorrow’s discussion, as a starting point, and then decide upon the general criteria for the workshop selection process tomorrow. The process document should be drafted immediately following the adjournment of the meeting and well-settled prior to the next in-person meeting.

 

 

Proposal  No 1

Last year’s workshop streamlining working group, led by Fiona Alexander, advanced a proposal.[1] Terrific discussion on the list ensued. (CF 2 discussion threads, attached. Thank you to Veronica Cretu, who undertook to summarise discussion on the proposal, also attached). 

 

Under this proposal, each MAG member would score workshops according to the following system:

 

 


Criteria

Rating


Well written, thought out and complete proposal

Yes = 2, No=0


Template complete including speakers identified and confirmed

Yes = 1, No= 0


Relevance of topic to Internet governance

Yes = 1, No= 0


First time workshop proposer or new workshop topic (i.e., substantially different from previous IGF workshops)

Yes = 1, No= 0


Gender diversity

Yes = 1, No= 0


Geographic diversity

Yes = 1, No= 0


Stakeholder diversity

Yes = 1, No= 0


Perspective diversity

Yes = 1, No= 0


Plan for active remote participation

Yes = 1, No= 0


Plan for active audience participation

Yes = 1, No= 0

 

Proposals that score an 8 or higher will be accepted.  In cases where topics are substantially similar workshops may be merged, in particular if there are space limitations due to the venue.  Proposals that score between 6-7 will be rejected for workshops, but may qualify for other types of sessions such flash mobs.  Proposals that score 5 and below will be rejected.

 

 

Proposal No 2

Alternatively, in lieu of a numerical system, the MAG could employ a “qualitative assessment along the guidelines commonly agreed on.” (Cf Tero Mustala’s email in Thread 1, attached). Every MAG member would have a duty to select his or her 50 or 80 (or whichever) number of preferred proposals, according to the criteria provided. The resulting lists would be merged and the top 100 or so – or however many workshops will be accommodated – would form the programme. 

 

Additional criteria

Under either proposal, MAG members who have submitted workshop proposals would be precluded from voting upon their proposals. 

 

There should be a limit to how many proposals one can make, whether an individual or organization. 

 

Perhaps the limit for MAG members should be lower than for non-MAG members.

 

Many thanks.

 

Cheers,

Susan

 

 




  _____  

[1] Cf Meeting Minutes July 2013, at 8. <http://www.intgovforum.org/cms/Draft%20Virtual%20MAG%20Meeting%20Summary%2026%20July%20and%2031%20July%202013.v2.final.pdf>  

 

-- 

Susan Chalmers 

Consultant, Internet Policy

 

+1 269 324 4101

www.susanchalmers.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://intgovforum.org/pipermail/igfmaglist_intgovforum.org/attachments/20140220/0e26e832/attachment.htm>