Hi everyone! John here. After holding Punctuation we emailed our members to ask what they’d thought of the convention, and I’ve collated the results. In the post below, I’ll go through relevant pieces of feedback and pull out quotes from returned evaluations, but first, let’s talk methodology.
We posed several questions which we asked people to answer on a scale of 1–5, where 1 was ghastly and 5 was brilliant. We got 41 responses, which is ~20% of the total membership of 200. From the responses I can calculate the average score for each question on a scale of 1–5 (which is the blue bars, below) and I can also calculate the standard error on the mean (which is the orange line on each bar, and which can be ignored if you’re not particularly statistically inclined).
I’ll go through the different aspects of the convention below, outline what worked, and what we would look at changing next time. A note before I dive in: I’m making the assumption, here, that the returns we receive are representative of the convention. This assumption may be faulty, but unfortunately, I have no way to test this, and so it’s the best assumption I have!
Joining in the fun.
We asked three questions about joining the convention:
- How easy did you find it to join Punctuation on the website? (4.0)
- How easy did you find it to pay for a membership? (4.3)
- How much did you feel that the convention was value for money? (4.9)
Encouragingly, the response to all three of these questions was, on average, high, and there were no key issues highlighted in the quantitative feedback from people who answered them with a rating of 3 or less, so we can’t take much in terms of notes for next time from the feedback here. One thing we are aware of from committee discussions is that the WordPress payment portal is designed on the assumption that you’re supporting a blog, not joining an event, so we are conscious that there is room for improvement with how we do this in future.
Of the 41 returned surveys, 39 people said they used Discord during the convention. We asked six questions about Discord:
- How easy did you find it to join the Discord? (4.2)
- How well did we communicate that pre-con planning was occurring on Discord? (3.6)
- How easy did you find Discord to use after a week or so of using it? (4.1)
- How useful did you find “tables” in Discord? (2.6)
- How did you find #help-desk, #newsletter, and other help channels? (4.2)
- What did you think of our moderators and listener facilities? (4.0)
Questions 1, 3, and 5 have high ratings, which is encouraging as it is those questions that relate to how easy Discord was to use and how well we did at putting people at their ease with it. There were a minority of members for whom this wasn’t true; looking at the quantitative feedback from those who responded with a 3 or less to these questions, two quotes jump out. One person said “I didn’t find the Discord interface even slightly intuitive, even after you’d given me some help to sign up…” and another said “Not a fan of Discord, so I did feel it was over used.” However, there was generally little complaint about Discord’s ease of use.
In terms of our communication about Discord, we score less well (3.6). This is a definite area for improvement; we realised in the run-up to Punctuation that not everyone had realised that they could interact on Discord straight away. One respondent criticised our programming, saying “…[panels] felt exactly like a bunch of friends who decided that x was a cool thing to talk about. Which is fine but led to insular, unstructured panels, and one that often didn’t contain any diversity of approach that stretched the topics…”; one area for improvement will be making sure that more members are aware that they can impact programme by suggesting items on Discord before the convention. Another person wrote, “In retrospect I really wanted something like a PR#1 as soon as I’d joined, with all sorts of detail about what you were doing and how I could/should now interact with that as a member.” This is a good point, and something that we can take on board to help make it easier to get involved. However, those members who did take part in Discord pre-con found it useful; one respondent said “The advance planning and communication on Discord worked very well.”
The “tables” we set up in Discord were the least popular part of the convention (2.6) according to our 41 respondents, and as such, we would probably not bring them back at the next convention.
Finally, on the topic of moderators and listeners, we scored highly (4.0). The only comment we received on this was from a moderator, who said, “I felt Discord Moderator was a bit of a non-job, although it did at least mean I got involved in Discord conversations in rooms I wouldn’t otherwise have visited. Combine it with first-line #help-desk in future?” This is an excellent suggestion, and reworking the volunteer roles that require people to be on-shift so that volunteers don’t feel extraneous will be something we look at in future.
We asked questions about the interactive aspects of the convention, which were:
- How much did you enjoy your virtual con badge? (3.3)
- How much did you enjoy the treasure hunt? (2.8)
- How easy did you find it to participate in interactive programme items on Zoom? (4.0)
- How interactive did you find the convention? (4.2)
Questions 1 and 2 both receive medium scores. Only one person mentioned con badges in their response, and said they were “more effort than they are worth”. On the treasure hunt, at least one member specifically called it out as one of the highlights of the convention, and one person who gave it a low rating wrote, “I scored the treasure hunt low because I don’t know what I was doing wrong but I failed to find any of it”, indicating that there might be technological barriers we hadn’t considered. In general, though, both aspects of the convention were well-received.
In terms of how easy it was to take part in interactive programme items on Zoom, and how interactive people felt the convention was overall, we score highly (4.0 and 4.2). This is gratifying as interactivity was one of the key aims of the convention, and the results show that we do very well here overall. One respondent wrote that it was “[the b]est effort so far at making the on line interactive.”
- How easy did you find it to use the programme? (4.2)
- How easy did you find it to find links to programme items? (4.3)
- How much did you enjoy the panel items? (4.1)
- How easy did you find it to chat on YouTube or Discord during panel items? (3.6)
In general, the programme was well-received, with high ratings. There was one criticism of the panels (quoted above), but generally relatively little criticism.
We ranked less well in terms of the split across platforms. One person wrote, “…having two streams for questions/discussion on the YouTube events was very disorienting. PICK ONE for goodness sake, so the conversation is happening in one place.” Another person wrote, “Spreading it across 3 services was… awkward. I realise the limitations of any single service possibly preclude anything much else, though.” This latter point is a key limitation, and one which is difficult to circumvent. Additionally, there were a number of people who agreed that the chat should either be solely on YouTube or solely Discord, but those people did not agree on which service should be abandoned. A key takeaway is that if possible, we need to implement technology that allows comments to be crossposted between the two, but this may be wishful thinking.
37 respondents indicated they used Zoom over the weekend, and we asked three questions about social spaces:
- How easy did you find it to find links to social spaces (including the Space Bar)? (3.7)
- How much did you enjoy the social spaces? (3.8)
- How much did you like the Zoom-breakout-rooms model we adopted for the Space Bar? (3.9)
We received medium rankings on these questions. We were aware from at-con feedback that the Space Bar was not sufficiently well-advertised, and this is something that we need to handle better at the next Punctuation. We got several pieces of constructive criticism on Zoom. Some people mentioned that the Space Bar was occasionally too empty. One member wrote, “For a convention that was supposed to be focusing on the social space, it felt particularly un-social to me. Possibly because the bar was supposedly open all the time, but actually not inhabited” – the last point was also echoed by other members. This may be something we can address at the next convention by holding interactive programme items in the Space Bar, rather than in their own Zoom accounts.
Another criticism of Zoom concerned the breakout room model. One member mentioned the “The eternal question of how to manage the size of breakout rooms in Zoom – I’d have liked to join a room with fewer peopl[e] on occasion, but there usually wasn’t one.” Another wrote, “If you are going to have party rooms entered from a central room, th[e]n that room has to have one host who explains what to do, not four people chatting.” We may need to look at how we structure breakout rooms, and how we handle the host role, at the next convention.
- How did you find Streamyard as a panellist? (4.5)
- How did you find Streamyard as a moderator/wrangler? (4.5)
- How did you find Zoom as a panellist? (4.8)
- How did you find Zoom as a moderator/wrangler? (4.0)
These returns are from much smaller numbers of respondents (3–6), but they indicate that people in general found the technology we used relatively easy to use. Zoom was more popular with panellists than Streamyard, which might be because more people had used Zoom than had used Streamyard prior to the convention. Zoom was less popular with moderators and wranglers than Streamyard, which is probably because Streamyard is built for this and Zoom is sometimes a little obtuse when it comes to being the event coordinator.
- In general, people liked the convention!
- We need to communicate better before the con, especially to encourage people to join the planning efforts beforehand.
- Volunteer roles could be streamlined.
- Con badges were not popular enough to offset the large amount of effort they represented.
- If possible, having panel comments crosspost to Discord and YouTube would address criticism around the ease of following discussions about panels.