AG - Introduction and Discussion

[NuckElBerg]

Week 7 has ended! Welcome Week 8

17 builds for Octan, 15 for Kawashita, 10 for MANTIS and 1 for the Aliens Civilizations.

I think I posted my build after you wrote this message. It was posted before the 10:00 GMT deadline though, so I hope you will accept it... (I had some trouble uploading my pictures.. I know you hate last minute postings, so I'm really, really sorry Bob! )

[Dardanel]

Hello all, Dardanel again, here to complain about the judging :D

Today’s judging results sparked a little discussion in our private chat and it was suggested that it might be for the best that it continues in public, since there were some valid points. While I will try to take as many opinions that were voiced as possible please consider this little rant as my own and nobody else's, as well as all the opinions stated within.

Unfortunately, in order to carry my point across, I will have to use examples. I do not wish to inconvenience anyone (, but those in charge,) so I will try to make this as anonymous as possible, by posting only pictures, and not links to threads. I know that this is a poor attempt at hiding identities, but it the best I could come up with. Again, I am very sorry if your build was used as an example and know that I do not say that there is anything wrong with either you or your build, in fact I like it for what it is! An expression of yourself and your cherished creation. My beef is with the judging, and only with the judging.

I would like to kick off the actual rant by paraphrasing myself, from the internal chat. If you consider that we know just how much effort, and ingenuity, on our part, the builders, as well as the previous judging results (of our own builds) you will understand why we highly anticipate scores that we are “sure” to get this week. Then, in turn, you will understand why I am of such heavy heart almost every week when I see a score I feel I (or others!) do not deserve.

It is difficult to stay motivated in this game when the judging seems much more like a rolling of a dice, than something deterministic. I do not consider a game with such (lax) judging to be worth the effort. Ultimately any game should be about enjoying oneself (and for me part of the enjoyment is getting a fair and just reward, in this case a score) and not figuring out what the judges like/dislike. Judging should not be based on intangible aspects like friendship, taste and similar.

So far I’ve been all talk and no evidence, I must be making this up, right? Let’s consider these examples then. First I will show only my work:

http://www.eurobricks.com/forum/index.php?showuser=132931&tab=topics

All these builds received the same score! Do they all deserve the same score? I will let you decide.

Next, we have this build which, if we consider previous judging results, is a 3 for build and probably a 4 because of the very nice story, but it received a 5:

What other builds have received 5 or less?

But we also have some examples that have neither remarkable craftsmanship, nor an amazing story, nor outstanding photography. In other words, they’re solid, good looking builds, but nothing that stands out about them. This means that they should get average scores, right?

All of these examples received high scores.

Again, I am terribly, terribly sorry if I used your build as an example. I really did not want show any, but I feared that, like last time, the upper echelon would waive it off as groundless accusation. Should it bother you too much, please tell me to take down picture of your build(s).

What I am hoping to do with this post is spark an argument that will lead to either us all better understanding just why the scores are so random, or, a change in the judging process so that we can all have a fair(er) game.

I will also offer what I consider possible solutions to this, so that there is something constructive in this, as well (consider these alternatives, that may be used together or separately):

1. Clear judging guidelines
   
   • Per judge, or
     
   • Global
     

[*]More judges

[*]Larger scale

• Dismiss judging if difference is more than ?% and try to align scores
  

[*]Qualitative score together with quantitative score

[*]Shadow judges with no affiliation to AG

[*]Make all scores/comments public

Sorry for the long post and sorry for sounding like a broken record.

[Commander Turtle]

Dardanel, to clarify, are you planning on staying if there are changes made? I know you have said in the company PM that you intend to leave once your company debt is repaid.

Personally, it does seem some weeks are more severe than others. Last week there were a lot of 6s and a 7, this week there's only a couple. I was expecting the two large vehicles from myself and Bob to be surefire 6s, as well as some of the landscapes from other builders. Is this random variation, or are there ongoing discussions between judges as to how strict scoring should be?

I don't think anything this week needs taking down score-wise, but I'm surprised that some weren't higher. I quite liked the slightly inflated scoring from the past couple of weeks, it made each build feel more significant.

Final thought: different people obviously value different things. I think a well-knit story is very worthwhile, and I don't think photography skills should be relevant at all. We all come to this with different beliefs about what makes a build cool or interesting, and part of playing this game is recognising that the scores aren't the most important part. Heck, the team sizes are imbalanced, the scores differing by a point or two are minor compared to that!

Anyway, thanks to all 3 judges for being so prompt each week. It's still very fun playing as far as I'm concerned.

[Kai NRG]

I have to say that I do kind of agree with Dardanel about the scores. I don't look at everyone's scores, I just mark the forum as read - partly to save me time, but also partly because I don't want to get mad.

But it does seem like - and I know this is really, really hard to balance, I ran across what seemed to me as the same thing in MOCathalon several times - that while each build's score is reasonable when just looking at the build - in other words, on a scale of 1-7, this build may seem like it deserves a 3 (though I must say I have my doubts about that and was quite disappointed with my score that particular week):

when compared to this:

- you've got to be joking if you're trying to say that both those builds deserve the same exact score! One took me a good three or maybe even four or five days, the other one was probably barely three hours!

It's not really incredibly motivating when a build whipped out in all of one day gets the exact same score as a build that took at least three times as long.

Now of course I know that you guys don't know how long I spend on my builds (which is a good thing ) but still in some cases it's pretty obvious that certain builds represent a far larger time investment than others.

So basically, what I'm saying is that I think there needs to be more consistency within the whole scoring thing. More than just, what score does this build deserve, let's make sure it's comparable to all the other builds that have been placed in that same score bracket.

Like I said, I try not to compare what scores my builds receive to what other people are receiving, but when I do happen to run across someone else's score, I'm often rather unpleasantly surprised! Which is not to say that they don't deserve it, just that I do too.

Keeping all scores proportionately reasonable and making sure each three (for instance) represents approximately the same quality level is of course very difficult, so I'm not trying to complain about the judging or anything - you guys are definitely doing a good job on what must be a ton of work! - but just wanted to mention that so hopefully things can come across as a bit more "fair" in the future!

[Kodan Black]

It is hard because we don't know what the criteria is. And of course my fear in knowing the criteria is people building to the score instead of to the spirit of the game. I'm an average builder and my scores have been average. I don't have the parts or time really to be a 6 or 7 builder. Probably not the skill either.

And there are times I see someone get a point higher and I feel like our builds were similar, but its possible theirs was a 3.55 that got rounded up and mine was a 3.45 that got rounded down. I don't see a lot of people earning well above their build grade. For instance your first example of a 5 is pretty unfair. You took the first pic and said it was a 3 with a good story. But there were so many other pictures that contributed to the build that you seem to be skipping over to say it wasn't a 5.

While I get that the scoring can seem off at times we don't know what exactly is going on behind the scenes. And I think generally you are talking a point here or there. Maybe a 3 gets a 4. Or a 5 gets a 4. That is down to personal bias and rounding at times. For instance, I know that there are a lot of people who want all builds to be totally smooth with no exposed studs, personally I think that studs are the iconic element that defines Lego. But I'm not going to be able to successfully lobby everyone to see it from my viewpoint. I just accept the auto deduction and move on or find another outlet for creativity.

At the end of the day its an online game with strangers using a plastic children's toy. There is no money on the line. I'm here to make stuff, hopefully get some feedback and criticism, and enjoy the stuff others make. As long as I'm having fun I'll keep at it.

[Commander Turtle]

There is no money on the line.

Oh my god, I've been playing it wrong the whole time!

[rodiziorobs]

I totally agree with Kodan's comment above.

Edited July 7, 2015 by rodiziorobs

[Dannylonglegs]

I agree with Commander Turtle's comment above. About money. I ducked up.

~Insectoid Aristocrat

[mediumsnowman]

I agree, sometimes the scores seem a little off, but at the end of the day does it really matter? I applaud the judges for the sheer amount of work they have put into this to make it enjoyable for all. Is whining about a minor imperfection any way to pay them back?

[MKJoshA]

I have to say that I agree with Dardanel. While the scores should not be the main reason we build, it is one of the main reasons many of us are building for AG. Back when SoNE first started, the judges gave reasons why they scored each build they way they did so that if a player asked about their score there was a definitive answer. I think Dardanel has laid out some excellent ideas, explanations for why each score was given being one of them. I know first-hand how easy it can be to give fast scores because you are trying to just get the job done. But I think it's only fair that each player be able to know why they got the score they did. Most of us just want to improve our skills and a 4 when we think we deserved a 5 (like my last build!) doesn't help us get better.

[kabel]

C'mon guys, scoring really is a pain in the butt! And with no objective guidlines as to what scoring is based on some people will always end up frustrated. The thing is, 40+ mocs each week is just an insane amount of mocs to grade. In GoH it's sometimes really overpowering to grade that many mocs entered over a time of 2-3 months!

Do I build for the score? Well, its nice to know where you're at when comparing to others but seriously, it usually takes me several weeks and several trips to the Lego store to come up with something really great. So how much is there to expect when you only have a week and do not use LDD?

So everybody all ease up a little!

[Kodan Black]

Shouldn't the community as a whole provide feedback to help people get better instead of the opinions of a very small subset of builders? I'm by no means knocking the skills of our judges, but some of the builders who aren't judges have at least the same skill level if not better.

In our internal chat we've been discussing that constructive feedback is lacking in AG. Part of it is that there are so many builds. Part of it is not knowing how people will take feedback. But overall if I got scored a 3 and the judges told me why that isn't necessarily going to help me get better. But if there are 5 or 6 comments from other builders on techniques to try or examples of other people doing things that I was trying I am likely to learn.

You also have to factor in that with SoNE the judging component tends to be much more lenient on time. If it takes 3 weeks to get a score up, that isn't a big deal. I think with things already being a 1-2 day lag it will be tough for the judges to post comments on 45+ builds in a timely manner. What you'll end up with will be very brief like "small, mostly studs up" instead of the paragraphs BEaVER posts in SoNE.

Right now this is a weekly build game and the more administrative tasks you add, the more burden on the judges. Already Bob is maintaining the planet map, the timeline, etc. I personally don't want the judging to take until Thursday and then have to scramble to figure out where to build as a reaction to the past week. And the more you tie a judge to a comment/score the more likely people are to take their issue to that person. Which inevitably leads to judges not wanting to deal with all the complaints.

The system isn't perfect, but it is a good compromise between efficiency and quality. I've yet to see a build be 2 or more points off what I think it should score. 1 point comes down to rounding. I find it interesting to see people complain about the scoring while saying they want to get better. The score doesn't help you get better, the comments do. If anything I wish there was more constructive criticism, but I know that can be tricky as one man's constructive is another man's destructive.

[kabel]

I think what some people here sometimes forget is that all staff put in their work for FREE. None of us has any advantage of spending all these hours on keeping EB running other than the occasional "hey you guys are doing a good job" comment. All of us still do this though because of our enthusiasm for LEGO, and that's pretty much all that needs to be said about it!

[Dragonfire]

I must say that Dardanel puts across some very good points. I'd like to know exactly what separates, say, a 3 from a 4, or a 4 from a 5...

One other question: Why didn't I get my job bonus for this build? I included my character exploring new terrain and included flora of the planet as well.

[Bob De Quatre]

I'd like to know exactly what separates, say, a 3 from a 4, or a 4 from a 5...

one point...

One other question: Why didn't I get my job bonus for this build? I included my character exploring new terrain and included flora of the planet as well.

I must have missed the "exploration" tag as it wasn't reported on the scoring spreadsheet, sorry, I'll correct that

[Dannylonglegs]

one point...

I must have missed the "exploration" tag as it wasn't reported on the scoring spreadsheet, sorry, I'll correct that

Also hate to be a bother, but is that the reason my Land vehicle from "Shadow over Drigo" was not counted as such? It said I got 6 DP on the timeline, but if that was taken into account, that would have been 7. I understand, though if you don't think it counts as it was not important to the build, or featured in a proper way. Thanks!

~Insectoid Aristocrat

[Bob De Quatre]

You post 6-7 pictures, and in the corner of one of them is shown a part of what could be a vehicle.....

[Dannylonglegs]

You post 6-7 pictures, and in the corner of one of them is shown a part of what could be a vehicle.....

Ok! Just want to make sure it was intentional, and not an error. I got rushed at the end, and needed to finish to catch the last of the natural light, and I didn't make a big enough backdrop to feature it more prominently. I'll make sure to amend that for future weeks.

~Insectoid Aristocrat

[aeralure]

Oh my god, I've been playing it wrong the whole time!

Let's just say I know I have spent some.

[Dragonfire]

one point...

That's not what I meant. What I meant was: what criteria separates a 3-point build from a 4-point build (for instance)

I must have missed the "exploration" tag as it wasn't reported on the scoring spreadsheet, sorry, I'll correct that

In fact, to make it easier for you, can you please give the 3 credits directly to NuckElBerg instead of giving them to me?

[Commander Turtle]

Bob said the signatures aren't updated yet but the credits are there. We'll either see them soon, or when they're all refreshed next week.

[Donnie Bricko]

I am finally able to sign up for AG...

Engineer Donnie Bricko joining M.A.N.T.I.S

I will post my sigfig later today

EDIT: Sigfig Posted

-D

Edited July 8, 2015 by Donnie Bricko

[NuckElBerg]

For what it's worth, I'd like to say that I agree with Kodan Black and kabel. Sure, some judgings may be a little questionable from the outside, some practices may be improved (I for one would like to see an unrounded scale from 1 to 70), but over time, I think most things even out and become "fair". Some weeks you get a higher score than you expect, others you get a lower (which I guess is happening more than the first-mentioned for most of us (myself included) since we're all self-centered bastards ), but in the end we're all just playing for fun (or have I missed something? ).

I do sincerely agree with Kodan that AG could use more constructive criticism. It feels like words such as "wonderful" and "amazing" gets thrown around too easily and lose their value. I, for one, only comment on builds that I really like, in order to not promote this kind of circlejerking (even if it can be nice when you are the center of attention )... One way of encouraging constructive criticism could be to ask all players who want it to add that to their builds (C&C is appreciated or something like that).

Some things I have noticed (solely based on my builds and the scores they receive) regarding the judging:

- Judges like more advanced techniques. One might dismiss this by saying "why use an advanced technique when standard 'studs-on-top'-bottom-up work?", but the truth is that most builds actually benefit from using from using more advanced techniques... it simply makes them look better and more well thought-out (if you know how to use them correctly). Look at a lot of builds, both real ones and LDD ones, and build a lot, in order to get ideas for advanced techniques!

- Judges like creative techniques. This is really a sub point to my previous one, but what I'm saying is; try coming up with new ways to use bricks and not just throw on "standard" advanced techniques and hope that you will get a good score. New, creative uses of bricks always shine through!

- Judges like stories. A good story is always great, and it can really enhance a build. However, what this really ties into is:

- Judges like a concept. Sure, you might've created a cool vehicle, and written a fun little story about it, but turning it into a concept, is what really makes it stand out and not just being "this is my thing this week, and this is the story". So, what turns a build from just being a build to a concept? Well, start by creating thinking up a backstory... what's going on outside of the build and the written story? After this, create an environment based on the backstory. A great environment which ties into the story and the "main attraction" can really make all the difference. Longer story-arcs, some mystery, collaborations, etc. are also great ways to turn a build from a build to a concept.

- Judges need a good presentation. I'm sorry, but in order for a judge to make a good call, the pictures submitted have to be good. This means good lightning and good angles, zoomed in shots, etc. so that the judges can accurately judge your build. If you only include a couple of zoomed-out shots with so-so lightning, then it's impossible for the judges to make a good call on how good it actually is.

As I said above, all these points are my own observations, so I can't guarantee that following them will net you a higher score (except maybe the presentation one), but I think they could be good guidelines (both in order to do better builds and to net you higher scores).

In the end, I would like to end this comment with a shoutout to all the involved staff: You are all doing a great job in creating and keeping Andromeda's Gate afloat, especially you Bob (I can't believe you have the time for all this...)!

[pombe]

The three judge system is actually quite good. If you think about the math, if you scored a 4, most likely that means one of three situations:

a) 2 judges gave you a 4 (3, 4, 4 give you an average of 3.66 or 4, 4, 5 gives you an average of 4.33)

b) you got individual scores of 3, 4, and 5 which also averages to 4

c) you got individual scores of a 4, 4, and 4, duh

So unless something is wildly crazy about the individual scores, the judges are consistent regarding the build scores. I'm sure Bob could post the overall standard deviation to back this up.

And I think there will always be some sort of subjectivity involved in the scoring, because, let's face it, LEGO is as much about art as it is about engineering and puzzle solving. Therefore, composition and presentation are important components, as well, which are aspects of LEGO that defy quantification.

Edited July 7, 2015 by pombe

[robuko]

Everyone makes good points here. Dardanel's examples are spot on with regard to the variability of scoring, and the judges do an incredible amount of work in a very short time. It would be helpful if the judges can give guidelines or transparency on what they are scoring on. Nobody likes to see mediocre Mocs scoring more than good ones, but that is what's happening at the moment.

Edited July 7, 2015 by robuko