Photo by israel palacio on Unsplash

Call For Paper The CFP process

The selection process

During previous editions, we received many questions regarding the selection process of the Toulouse DevFest: Who chooses the sessions? What are their criteria? What makes a good or bad proposal? And so on. Whether you made a proposal to the CFP or not, these questions are totally legitimate, and we want to be as transparent as possible about this.

This post explains the main steps of our selection process.

Let’s begin with some figures about the proposals received for the 2019 edition:

Thanks again for all these proposals! We had only 28 full conference slots and 8 lightning talk slots, which leads to an acceptance rate of 8.75% and 8.4% respectively. As you might guess, it’s not easy at all to select the “best” 28 conferences among 320 for an event about general software development.

In our first editions, when the event was much smaller, the selection process was pretty much ad hoc, leading to endless arguments between organizers, as we had no way to settle differences among necessarily subjective evaluations. We have since established a process that makes the selection as straightforward and objective as it can be. Let’s see what it looks like.

The Communities Call

In Toulouse, there are many active communities, with various expertise in several domains of software development. Some of these communities were already represented among DevFest Toulouse team members (GDG, Toulouse Data Science, Duchess, DevOps, etc.)

Last year, as an experiment, we invited some of the other communities to join the selection committee, specifically:

By diversifying the selection committee members, we hoped to find the right balance for proposals’ ratings, and a final selection with higher quality for all attendees.

Generale rules

Here are the general rules regarding the Call For Paper and selection committee and process:

The selection process (aka. “the Bitardo method”)

We went along the following steps during the last event (2019):

Step #1 – Collect proposals

The CFP is opened for 2 months. During this period, speakers submit their proposal(s) on the https://conference-hall.io/ platform. All members of the selection committee review and rate (from 1 to 5 stars) all the proposals. The platform lets voters and speakers interact with each other in order to clarify the speaker’s proposal or ask/answer questions if needed, all along the selection process.

Step #2 – Selecting the Early Speakers

At mid-CFP point, the team meets to choose “early” speakers among the proposals which have been submitted during the first half of the CFP opening. During this meeting, each team member nominates two talks: a first choice with a ponderation of 2, and a second choice with a ponderation of 1. We order all talks according to score, then remove duplicates if two proposals by the same speaker are highly ranked. During previous edition, we selected 4 early speakers.

The goal of this early selection is to promote the event and encourage a maximum of people to submit proposals by starting announcement of speakers.

All speakers who are not selected during this phase are treated equally in the next one.

Step #3 – CFP Closing and Shortlisting

For each format (conference / lightning talks) we preselect the highest ranked 1.5N talks, where N is the number of remaining available slots for that format. For example, last year we had a total of 28 slots available for conferences (7x4 rooms) and 8 for lightning talks (2x4). As we selected 4 speakers during “early” selection, we thus selected the first 36 conferences and 12 lightning talks ordered by their ratings on conference-hall.

At this point, each voter can optionally recover one proposal (and only one) which wouldn’t have been included in this preselection, and add it to the shortlist.

Step #4 – Refinement using Condorcet method

From each shortlist (one per format), each committee member votes using the Condorcet method, i.e. ranks all the proposals in the shortlist according to her own preference. At the end, this results in an ordered and ponderated list of proposals per format. The first 28 conferences and 8 lightning talks make up the final selection.

While this refinement by Condorcet method requires more work for committee members, it seems to give a final result that more closely aligns with the committee’s overall preferences. We have been using it for the last 2 events and have been very satisfied with the end result.

Step #5 – Final deliberation

We then meet for a final deliberation, to ensure that: All selected proposals are well balanced: no similar proposals, no overrepresented category The only criteria (25% of talks in English) is fulfilled If any of these criteria is violated, we look for the smallest possible selection modification needed to meet the criteria. The idea is that the current ordering is, by construction, the closest possible to the overall committee’s consensus, hence big changes are to be avoided if at all possible.

We thus obtain the final selection. We then reach out to all speakers and ask them to confirm their availability. At this point, a small number of speakers need to rescind their proposal, in which case we replace them with the highest ranked remaining ones from the shortlist.

We followed this selection method during the last two events and noted (either on organization and voters side) that despite the fact that it requires more work for all voters (particularly regarding refinement by Condorcet method), the resulting selection has always been balanced and was generally reflecting the submitted proposals (categories and expected audience level).

Last year, we were also very pleased with the participation from Toulouse communities sharing their opinion and expertise for proposals rating. Many thanks! We’ll reiterate for next edition, by inviting even more communities to join the selection committee.

Like the previous year, we got a lot of positive feedback about the selection, which makes us feel very proud and happy! Thank you for that.

Of course, nothing is ever perfect, and we will continuously refine this selection process over the coming years. Despite the fact that selection is getting tougher and tougher, our objective remains super simple: offer our attendees the best selection possible!

If you have any question, don’t hesitate to reach out, we’ll be pleased to discuss this with you in more details.