I recently finished a paper that presents a novel social computing system called the Wikipedia Adventure. The system was a gamified tutorial for new Wikipedia editors. Working with the tutorial creators, we conducted both a survey of its users and a randomized field experiment testing its effectiveness in encouraging subsequent contributions. We found that although users loved it, it did not affect subsequent participation rates.
A major concern that many online communities face is how to attract and retain new contributors. Despite it’s success, Wikipedia is no different. In fact, researchers have shown that after experiencing a massive initial surge in activity, the number of active editors on Wikipedia has been in slow decline since 2007.
Research has attributed a large part of this decline to the hostile environment that newcomers experience when begin contributing. New editors often attempt to make contributions which are subsequently reverted by more experienced editors for not following Wikipedia’s increasingly long list of rules and guidelines for effective participation.
This problem has led many researchers and Wikipedians to wonder how to more effectively onboard newcomers to the community. How do you ensure that new editors Wikipedia quickly gain the knowledge they need in order to make contributions that are in line with community norms?
To this end, Jake Orlowitz and Jonathan Morgan from the Wikimedia Foundation worked with a team of Wikipedians to create a structured, interactive tutorial called The Wikipedia Adventure. The idea behind this system was that new editors would be invited to use it shortly after creating a new account on Wikipedia, and it would provide a step-by-step overview of the basics of editing.
The Wikipedia Adventure was designed to address issues that new editors frequently encountered while learning how to contribute to Wikipedia. It is structured into different ‘missions’ that guide users through various aspects of participation on Wikipedia, including how to communicate with other editors, how to cite sources, and how to ensure that edits present a neutral point of view. The sequence of the missions gives newbies an overview of what they need to know instead of having to figure everything out themselves. Additionally, the theme and tone of the tutorial sought to engage new users, rather than just redirecting them to the troves of policy pages.
Those who play the tutorial receive automated badges on their user page for every mission they complete. This signals to veteran editors that the user is acting in good-faith by attempting to learn the norms of Wikipedia.
Once the system was built, we were interested in knowing whether people enjoyed using it and found it helpful. So we conducted a survey asking editors who played the Wikipedia Adventure a number of questions about its design and educational effectiveness. Overall, we found that users had a very favorable opinion of the system and found it useful.
We were heartened by these results. We’d sought to build an orientation system that was engaging and educational, and our survey responses suggested that we succeeded on that front. This led us to ask the question – could an intervention like the Wikipedia Adventure help reverse the trend of a declining editor base on Wikipedia? In particular, would exposing new editors to the Wikipedia Adventure lead them to make more contributions to the community?
To find out, we conducted a field experiment on a population of new editors on Wikipedia. We identified 1,967 newly created accounts that passed a basic test of making good-faith edits. We then randomly invited 1,751 of these users via their talk page to play the Wikipedia Adventure. The rest were sent no invitation. Out of those who were invited, 386 completed at least some portion of the tutorial.
We were interested in knowing whether those we invited to play the tutorial (our treatment group) and those we didn’t (our control group) contributed differently in the first six months after they created accounts on Wikipedia. Specifically, we wanted to know whether there was a difference in the total number of edits they made to Wikipedia, the number of edits they made to talk pages, and the average quality of their edits as measured by content persistence.
We conducted two kinds of analyses on our dataset. First, we estimated the effect of inviting users to play the Wikipedia Adventure on our three outcomes of interest. Second, we estimated the effect of playing the Wikipedia Adventure, conditional on having been invited to do so, on those same outcomes.
To our surprise, we found that in both cases there were no significant effects on any of the outcomes of interest. Being invited to play the Wikipedia Adventure therefore had no effect on new users’ volume of participation either on Wikipedia in general, or on talk pages specifically, nor did it have any effect on the average quality of edits made by the users in our study. Despite the very positive feedback that the system received in the survey evaluation stage, it did not produce a significant change in newcomer contribution behavior. We concluded that the system by itself could not reverse the trend of newcomer attrition on Wikipedia.
Why would a system that was received so positively ultimately produce no aggregate effect on newcomer participation? We’ve identified a few possible reasons. One is that perhaps a tutorial by itself would not be sufficient to counter hostile behavior that newcomers might experience from experienced editors. Indeed, the friendly, welcoming tone of the Wikipedia Adventure might contrast with strongly worded messages that new editors receive from veteran editors or bots. Another explanation might be that users enjoyed playing the Wikipedia Adventure, but did not enjoy editing Wikipedia. After all, the two activities draw on different kinds of motivations. Finally, the system required new users to choose to play the tutorial. Maybe people who chose to play would have gone on to edit in similar ways without the tutorial.
Ultimately, this work shows us the importance of testing systems outside of lab studies. The Wikipedia Adventure was built by community members to address known gaps in the onboarding process, and our survey showed that users responded well to its design.
While it would have been easy to declare victory at that stage, the field deployment study painted a different picture. Systems like the Wikipedia Adventure may inform the design of future orientation systems. That said, more profound changes to the interface or modes of interaction between editors might also be needed to increase contributions from newcomers.
This blog post, and the open access paper that it describes, is a collaborative project with Sneha Narayan, Jake Orlowitz, Jonathan Morgan, and Aaron Shaw. Financial support came from the US National Science Foundation (grants IIS-1617129 and IIS-1617468), Northwestern University, and the University of Washington. We also published all the data and code necessary to reproduce our analysis in a repository in the Harvard Dataverse. Sneha posted the material in this blog post over on the Community Data Science Collective Blog.
Your study about the usefulness of the tutorial itself is really interesting.
Gamifying a tutorial is an interesting way to make learning basics more accessible or attractive to newcomers. But if the real issue is as you mentioned a hostile tone in comments from more veteran contributors then it must be that toxicity that should get addressed in order to retain more people around.
Toxicity in communication is always going to either drive new people away, or drag them in to copy that toxicity in order to shell themselves from hostility.
This is very interesting.
Since Wikipedia started I have contributed a few edits a month, started a new page or two, and even challenged the newsworthiness of an entry and got it removed.
I’ve encountered the occasional hostile behavior of experienced editors that has driven many others away. What was so incredibly frustrating was that even though I took the time to read up and follow the rules, occasionally an experienced editor would revert my edit on a whim, citing no Wikipedia rules at all. After many years of occasional editing, the last straw for me was a few months ago when I finished watching an old film from the 1940s, and went to read about it on Wikipedia. The article’s description of the plot was mostly correct, but there was a critical error near the end that was just flat out wrong. It was also missing what I thought was a major plot point entirely. I corrected the minor error and explained it was a correction, and I added a phrase to include the plot point. Within minutes, a much experienced editor reverted both changes because “he thought the article was better without the changes”.
What the experienced editor did violated a few Wikipedia rules, including making arbitrary reversions. It was so frustrating that I thought about challenging his decision, but then you run up against the entire Wikipedia process and rules of doing that, and the knowledge that the more experienced editor has all of his friends all lined up to side against anyone that challenges him. Was I really going to invest literally hours of my time to go through all that for a site that allows other users (more experienced editors) to treat me like that and to specifically decide on a whim that an article was better if it was left with incorrect and incomplete information? No.
I clicked on the option to challenge the revision and said it was arbitrary, but decided not to put up a huge fight and jump through all the hoops. Instead I blocked myself from Wikipedia entirely, until eventually I coded something that allows me to read Wikipedia but hides all of the editing functions from me entirely so I’m not tempted to ever edit again.
I am reminded of the challenge of increasing diversity in software engineering and that we’re finding that all the active recruiting in the world to increase diversity doesn’t help in the end because when the new people come in they are almost all driven out within a few years by the existing cultural toxicity in the industry. Even if we create brilliant technological solutions they won’t help if the existing culture they are going up against is toxic.
Those who made it to the game were tough enough to get that far and would already have won editorial battles. Unless someone is prepared to challenge the editors at the point of rule deviation they will be unable to contribute to the end with joy. It is good to learn through the game process and I think this is a brilliant initiative but it does not fix the hostility of capricious editors.
I agree with Conor and Amy here: you try and make a “good faith” edit and end up dealing with people exercising their own dubious subjective judgements. I can understand editors getting upset about, say, the quality of sources, but I’ve seen people trying to correct erroneous garbage on Wikipedia having their edits reverted and being told that the relatively authoritative stuff they’re referencing is somehow inferior to stuff that people have basically made up.
Indeed, fairly recently, I told one contributor who was risking a ban for editing things to which he had a personal connection to just demand that the page in question be deleted. Clearly, the page was just misrepresenting his work and, if Wikipedia’s bureaucracy can be mobilised to do anything, it is to deal with misrepresentation because that can have legal consequences. I see that the page concerned has now been deleted. Of course, such shenanigans do not help to improve the quality or breadth of Wikipedia.
Apart from the editors’ behaviour, it doesn’t help that the editing interface keeps gaining extra “bolt-ons” rather than actually being improved, that everyone still gets punted over to verbose documentation pages, and that there are mountains of vague (and archaic) guidelines that don’t actually help people do the right thing. Recently, I wondered whether I should link to a video which would demonstrate something referenced in an article, but the evasive guidelines didn’t give me any confidence that some citation snob wouldn’t remove the link and insist that the video first appear in some mainstream broadcast medium (or in some fantasy Encyclopaedia Britannica DVD edition) first.
After a while, one has to wonder what the objective is with Wikipedia. When you can spend an hour making even a modest edit whose persistence is guaranteed more by a lack of editorial supervision than any kind of coherent editorial process, no amount of enthusiasm-dependent, process-deficient “onboarding” is going to tackle the decline in contributions.