Home > City of Heroes > 3 months of user created content – a view of Mission Architect

3 months of user created content – a view of Mission Architect

July 8, 2009

Three months ago Paragon Studios released Issue 14 for City of Heroes/Villains, Mission Architect. The major feature in this update were the Mission Architect system which allowed players to created their own story arcs. A story arc consisted of a number of instanced missions, with a large part of the maps in the game available, most of the enemy groups and enemies were available and players could also create their own enemy groups.

And Mission Architect provided regular XP also for the missions and had its own reward system (architect tickets), similar to the merit rewards in the regular game. There has been a fair share of controversy and issues around the system, in particular due to the rampant powerlevel/farming that some players were able to create and (ab)use.

So now 3 months after the release I decided to do a follow-up on the statistics I collected a few days after the release of Issue 14. What is the state of the Mission Architect system now? The numbers will not tell the whole truth, but there may be some interesting observations.

First of all, there are over 50000 published story arcs in the system today. At the low end it would mean 16000-17000 players that have created story arcs and the high end around 50000 players. The actual number of players will likely be somewhere in between.

First piece here is to look at ratings for the story arcs.

Story Arc Ratings

Story Arc Ratings

This picture above divides the arcs into average ratings of 1 to 5 stars (5 stars being the best) and also include the arcs with no ratings. What also is included here is the number of ratings each arc has received.  What can be seen here is that a majority of all story arcs have received very few or no ratings. It is only for the arcs with an average 4-star rating that there is a somewhat large piece of arcs with a larger number of ratings. Around 40% of all atory arcs have received a 4-star average rating. So for the arcs that have been played and rated, the quality seems to be pretty good. It is certainly far from “90% of everything is junk”. But even if a story arc is good, the numbers also indicate that it is not likely that many players will experience it.

If we ignore the arcs which do not have a large number of ratings (all below 100 ratings) we see this picture instead:

Story arc ratings with higher number of ratings

Story arc ratings with higher number of ratings

In this case the total number of arcs can be counted in the 100s and the vast majority have 4-star average rating. It seems that the 4-star range is where good arcs which also are popular end up. Is it a snowball effect that causes these arcs to get high number of ratings, or have various ways to promote and market the arcs worked? This the numbers cannot say anything about, unfortunately.

When Mission Architect was released a few special categories were created also

  • Developer’s Choice
    Paragon Studios choices for good examples of using Mission Architect
  • Hall of Fame
    Story arcs that have received a large number of votes (999+) with a high average rating (4.5+ I think, but may be different).
  • Guest Author
    Perhaps intended for other game designers, comic book authors and other celebrities is my guess
Special arc categories

Special arc categories

There are currently 17 in Developer’s Choice and 5 in Hall of Fame. No Guest Author arc has appeared yet. Worth noting here is that around release of Issue 14 there were 12 Developer’s Choice. There has not been a huge amount of additions in that category in the past 3 months – 5 more. And with the latest Issue 15 it was also changed so that an arc could be both a Developer’s Choice and in Hall of Fame. I did not check if all of the Hall of Fame ones were also Developer’s Choice now, but in either case I expected a bit more from Paragon Studios here. I guess it has not worked out so well in practice to do this.

Based on these number there has been a whopping 0.03% chance so far to get an arc in either category and be able to get another story arc slot. Still a bit better than a lottery though.

Moving to some other categories…

Story arc language

Story arc language

There is no surprise that English is the dominating language used here. If oen wants to reach a large audience there is not much choice really.

Morality for story arcs

Morality for story arcs

The morality is what role the player of an arc will have, villain or hero or either. The Neutral choice is the default and most arc creators seem to stick with that. Whether that is because they do not care of have made a conscious choice for neutral is unknown of course.

Story Arc lengths

Story Arc lengths

The length of the story arc is the next category and the proportions are quite similar to what it was near the Issue 14 release. Most likely half of the story arcs are just one mission – very short typically means a 1 mission with a small map and short typically means either 1 mission with medium-to-large map or 2 missions with small maps.

A new feature which was included with Issue 15 (release a bit over a week ago) was the ability to set a status on the published story arc. The arc can now have one of three status values:

  • Work In Progress
  • Looking For Feedback
  • Final

By default all published story arcs received Work in Progress as their status. So the story arc author has to make an explicit update to change the status. So how many have changed the status after a bit over a week then?

Current status for story arcs

Current status for story arcs

Around 95% of the arcs have not changed status. Either the authors do not care or are bothered to change, or they are actually works in progress. Worth noting is that all Developer’s Choice arcs seem to be in Work In Progress status when this was checked. Some Hall of Fame choices have changed status though.

But Work In Progress may mean precisely that here also; it is a work in progress. So another check was made to see when story arcs with Work In Progress status was updated last time.

Last update for Work In Progress arcs

Last update for Work In Progress arcs

Roughly 80% of the story arcs which are Work In Progress have not been updated in the past month. My guess is that for most of those are arcs that the author is not really bothered with or interested in anymore. The other 20% may potentially still be stuff that people are working on. Once the novelty of the new shiny toy has worn out I think what we may start to see here is more accurate numbers of people interested in this.
And even if 80-90% would be discarded here we are still talking about 1000s or story arcs. That is nothing to sneeze at.

So if we go back to the 5% of the arcs that have changed status since Issue 15, how are they holding up from a rating perspective?

Story arc ratings for Looking For Feedback + Final

Story arc ratings for Looking For Feedback + Final

They seem to be pretty good in terms of ratings they have received. And the total number of arcs are certainly more manageable, with less than 3000 in total here – which is still quite a large number.

Again, the numbers does not say whether you would actually like those missions with high ratings.  But combined with some of the new search/browse features and mission creation features introduced with Issue 15 I do think that people will be able to find decent story arcs easier. And there are other things they could change which would make Mission Architect more attractive to play in (for non-farming), if they need/want that. But that is a topic for another post.

Categories: City of Heroes
  1. Tesh
    July 8, 2009 at 17:26

    How are you gathering this data? It’s an interesting collection, and I’m definitely curious to see where it goes over time.

    One thing that stood out to me, perhaps erroneously, is the curious potential correlation of the glut of four star ratings and the glut of 70ish percentile ratings in the game industry at large. It’s not a prototypical bell curve of score distribution, which would suggest a glut around 50%, it’s shifted. It’s also a bit like grades; it’s like a C average (75% or so) rather than an F average (50%).

    It’s not a huge effect, but it does suggest to me that what might be viewed as four star (80% “good”) is really just “average”.

  2. July 8, 2009 at 19:03

    It is collected through the browser interface of Mission Architect. The part that takes a bit of work is to get the number of ratings in each interval, since it does not allow to sort data exactly like that – it does randomize things to a certain extent, bu there is a pattern to that.

    I think you are correct in that people may set slightly high scores, although I think it varies a bit. It certainly has changed a bit over time how I rate arcs I play.

    But I do think that many people will rate at least a three unless they really dislike it or get annoyed or really bored by it – if they rate at all then.

  3. Tesh
    July 8, 2009 at 20:51

    Interesting. It’s nice to see that this sort of data is readily available in the game, more or less. I love data mining, and some of the most fascinating game design articles I’ve read are those with hard numbers from the devs. That data all too often isn’t available.

    You’re probably right in that people don’t bother with one or two star ratings. I guess my point is just that it’s an incomplete picture, and that a 4 star rating doesn’t really mean “great”, it means “average”. To a degree, that undermines the point of having a five star scale, if only the top three fifths of it is being used.

    It’s not a big deal either way, just something that stood out to me. 🙂 Thanks for the analysis!

  4. July 8, 2009 at 22:47

    All five of the stars are used, but since rating is optional I think that people may be less likely to bother rating if they do not like it or are indifferent to it.

    If they really hate/dislike it, they will rate it. And then there is the group that rate low just out of malice and the people who rate high just because something is made by their friends or family.

  5. July 9, 2009 at 12:15

    Numbers make my head hurt! 😀

    Kidding – interesting stuff there. I can’t remember what the case is for unsubbed accounts, and how that might affect the numbers. Is my mission still in there, I wonder? Or are there mechanics to remove arcs from people who no longer play? If not, then there may be a serious glut of stuff whose status will never change from “work in progress” since the players aren’t there anymore to make the change.

  6. July 9, 2009 at 17:44

    Ysh, your mission is still there (15 ratings, average 4 star).

    I do not think they remove any story arcs, besides perhaps some banning of excessive farming missions.

    They really have not had any good means to determine which arcs to remove though, if they decide to do that. At least not until they included the status. If they decide to remove old arcs at some point I think they will do that on Work In Progress missions that has not been updated in X amount of time.

    But that is probably a decision for marketing – will it be better to keep it and have a high aroy arc count, or would it be better to have a lower count but more arcs that may be finalized/active?

  1. No trackbacks yet.
Comments are closed.
%d bloggers like this: