There should be a limit as to how many rating criteria one can add. I have encountered decidely unwitty pranksters adding seventeen criteria to their work, haiku writers with twelve “get me published” criteria stacked above one another and people who misunderstand that the same criteria worded differently is not a stand-alone criteria.

I propose a maximum of five… that should be enough. For those of us who do not take a vested interest in the rankings systems (for it is unfair and credit-judged rather than talent-judged), this would be most helpful.

Agreed. Some of the categories are ridiculous.

I too find some of the catagories to be a little odd and am yet to figure out how i am to rate “need to find agent” and such like as a catagory on someones work. But then again I don’t really get the whole ranking and rating thing either to be honest. I guess it appeals to some more than others.

If we are to continue to be baffled by how to rate someones need to find an agent and such like it seems right that there should be a limit. Maybe 4 or 5?

Agreed. 3 would be enough IMO.

Glad someone brought this up. I think a limit would be helpful. Some of the ratings aren’t really rateable (if that’s a word) and yet we have to give them all a number in order for the review to be accepted. I think 5 would be a maximum.

Agree as well. We will put a limit.

I moved this to feature requests.

I officially cast my vote for 4.

I would love to see it down to 3 or 4. Also, the rating criteria list needs to be cleaned up. We don’t need 10 criteria that all mean “Publishable” or 20 that mean “Get/Attract/Find/Need/Kidnap an Agent”.

The funny ones are another issue. I say let someone do this. Why not?

To answer thefarmerswife’s questions above: You are rating the item’s attractiveness to an agent. You have to ask yourself Would an agent be interested in this? And you’re right, this is nonsense. My items are at the tops of the “Attract an Agent” lists, and not one agent has shown interest. Hmmm. Maybe we need a “Bore an Agent” rating criterion.

I too have a few items at the tops of the lists. But it’s my opinion that…

The Ratings & Rankings page has really become nothing other than “Texas Sharpshooting”. What is a Texas Sharpshooter you ask?

It’s shooting at a barn door ten or fifteen times and then taking a piece of chalk and drawing a circle around the (usually five) hits which are most closely grouped together.

I’m totally speculating here, but if an agent or editor looks at a top ranked piece and then sees that it’s only received five reviews, I imagine this is what they think: The true mark of a Texas Sharpshooter. Yep, the best in the West.

I hope no one takes this as a dig. Just making an observation.

That’s why I think the minimum number of reviews should be raised to 10 before an item is rated. I revise a lot, so there are times that I revise before the item is rated. If my item is number two or three, it’s only because it’s been reviewed by 15 people and I have taken their advice. Still, raising the number of reviews to rate the item would have lots of benefits: 1) It would cut down on abuse, 2) It would encourage users to write more reviews (to get more reviews), and 3) I agree with Curt that 5 reviews just isn’t enough.

Exactly! 10 was what I was thinking too… All good points DC.

It’s funny how a few of the items on the top of the list have a 5th review by a “person” whose only review is on that piece before they disappear forever.

I think that to implement this, the pieces in the Ratings and Rankings page would look much more attractive to prospective agents and editors. Especially you know, now that Urbis and Steve have graced the glossy pages of Esquire.

We will limit the number of rating criteria to 3, but maybe let PRO members have more. We may also give x credits for each rating (so the creator loses credits for each rating of each review, and the reviewer gains).

As for the ranking, we can increase this number as well. 10 may be too much. 7?

7 or 8 would be fair… That’s also a great idea for the credits and rating.

I don’t think we should charge for ratings, especially if we’re limiting them to 3 (which I think is a fair number). I for one would probably stop using them altogether since they don’t seem to be particularly useful, really. I think others might feel the same. If less people are using the ratings, then the value of those ratings decreases.

I do like the limit though, as well as changing it to 10 reviews for rankings. :)

It would only be 2 or 3 credits per criteria. I actually prefer people to not use them if they don’t find them useful – why waste reviewer’s time? Charging a couple of credits might help weed those out.

Having now got into a total mess, we come to the problem of sorting it all out.

Given the abundance of criteria and ratings already in existence, changing the ground rules effectively removes whatever value those that exist had.
Leaving a mix of old and new also prevents any real value going forward.
(See the ongoing dissatisfaction with review ratings and the effect of the period they were switched off).

So, what to do with the existing stuff?

I suggest this decision is made first before implementing any changes, including a retro-fit solution.

As an, now ex, IT project Manager, seeing the ‘taste and try’ approach so often used on Urbis development has grated for quite some time. Many site designers fall into the trap of thinking RAD means no formal design.

I think Urbis is now old enough and large enough to take a more professional approach.
Experience has proved often enough in the last x decades that the professional approach actually costs less in the long run.

We’ve been seeing a large amount of the ‘gotchas’ with recent urbis changes. I suspect this is because the developers often have to do a spot of reverse engineering, with the corresponding risk of mistakes.

Hey, Avedis, could you boil these thoughts down to a concrete suggestion? I see where the irritations lie, but what can the developers do now?

Steve, I would like us to really consider having 10 reviews before the item is rated/ranked. This would “encourage” writers to write more reviews so that they get more reviews to see where their writing is ranked overall. Sorry, everyone: I do believe most people are interested in seeing where their work falls in the overall rankings. It would also make it more difficult for writers to boost their rankings by having their friends log on once to give them a 10. These one-timers don’t do anything for Urbis, except maybe inflate the number of users artificially.

I don’t think we should get rid of all the rating criteria, but I do think we should consolidate those that are obviously repetitive. Publishable – Overall and Publication, for example. But NOT Publishable – Overall and Published in an Anthology.

DCAllen:

Without seeing the ‘internals’, it’s hard to say.

I very much suspect the stage has been reached where the existing ‘Urbis’ is used solely to extract requirements and a set of ‘rules’.

Design and build a new system, then first cleanse, then migrate the data across – using several conversion routines.

I doubt if they have the finances for this.

Data cleansing and conversion would be less expensive, if the existing design and code can take it.

Changing the ground rules, like changing data entry screens with new or different fields for data, always leads to a mess.
Without cleansing and conversion, whatever you do, you end up either excluding data you wanted, or including data you didn’t.
And Urbis is essentially a database with an unusual front end.

So, keeping it simplistic with ratings:

Two parallel activities:
a. Data
1. Define new ratings data, format and values.
2. Analyze existing ratings data, decide on conversion rules.
3. Run conversion and migrate data into new database.
b. Front end.
1. Design front end changes.
2. Write and implement underlying code rules.
3. Point front end to new database.

Sorry, this has to remain at the simplistic level because I have no idea of how Urbis is implemented.

Note to all: I agree about limiting those dumb rating criteria.
Note to AstridM: Every serious writer should have an unabridged dictionary. Because I do, I checked it for you: ratable is indeed a word; rateable is an alternate spelling. Cheers.

Agree with DCAllen. Someone needs to take a hatchet to all the “publish me” files. What does that mean anyway? Does it relate to the draft or the concept – some concepts are good but poorly written – or to the quality of the draft (others are sloppy or posting rough dradts)?

Perhaps a way around this would be to allow reviews the opportunity to leave a ranking blank, or (where there are multiple criteria) to answer a minimum number (eg 3, 4, 6, whatever). Rather than the message that we have to go back and fill in the criteria, it could be a warning which you can bypass.

I agree about the ‘published’ criteria. I’ve been giving 1’s if they have that on there, because it’s up to you to submit it for publication. Same goes for ‘attract an agent.’ How are you supposed to grade that? Do I think this will attract an agent? I dunno. I can’t get one myself, so how can I grade someone else on it?

As far as the criteria, have a fixed number. Instead of skipping it, give the reviewer the option of perhaps leaving a message as to WHY they chose not to fill that particular criteria. You could get a general idea of how feasible a criteria is from this and not just have a really, really, really low score.

Hey guys,

A few points here.

1. Limiting the number of rating criteria one is allowed to add to each item doesn’t mean deleting existing criteria. It just means that, of the exiting criteria, members could only add x number of them (not sure if there was confusion there or not).
2. There is an additional problem with duplicate criteria (example: publish me, I want to be published, publishable?). We have a way to merge these criteria in admin (that doesn’t require data migrations) and we do it every six months or so.
3. We might move to a model that disallows custom criteria. Or at least limits who may create custom criteria. This would solve the duplication problem.

“As an, now ex, IT project Manager,”

Interesting. What frameworks did you work with?

I vote for a maximum of 4 selectable- if that doesn’t cover it, buggered if I can think why.
I also vote for a maximum of 10 available criteria – again, if that doesn’t cover it, buggered if I can think why.

“3. We might move to a model that disallows custom criteria.”
I like this, but with a modification.
Restrict the number of criteria available, do not let anyone add to this indiscriminately.
Instead, and sorry to suggest an admin overhead here, authors have to request a new criteria if they want to have one added – with their reasons. The effort involved should discourage the frivolous.

Steve:

I started in 1982 and continued through to 2005, working for both commercial companies and software houses (including IBM amongst others). Only a couple of short periods as a permanent employee – the rest as a contractor/consultant.
Contracts ranged from a few months to several years.
So, everything from Fortran/Cobol through to .net, plus a lot of specialist tools.
Methodologies from supposedly waterfall (which never existed in a pure form) to V-Model to RAD (again, a generic term for a variety of methodologies, including ‘iterative’ and ‘agile’).

I belonged to many professional forums and have ‘discussed’ ideas with most of the major players.
I was with IBM when they ‘merged’ with Rational, took part in the negotiations, and ‘tested’ the concepts in their ‘Unified process’.
I was one of the early supporters of ‘Test Driven” development and have had input to defining exactly what that means.
My full CV is about 20 pages long, for the last few years I had to restrict myself to using ‘extracts’. lol.

This is the Internet, I can hide behind an anonymous ID and make any claim I like. I want to keep that anonymity for personal reasons, yet will state that I was a major player in my professional life before retiring.

“I vote for a maximum of 4 selectable- if that doesn’t cover it, buggered if I can think why.
I also vote for a maximum of 10 available criteria – again, if that doesn’t cover it, buggered if I can think why.”

4 selectable is good. Not sure about 10 available. Our original business requirements (talking to you like a project manager now, ha) separated criteria into user generated and admin generated. There would have been around 10 admin generated (Overall, language, plot, character development) and unlimited user generated, like it is now. The two types would have been separated into tabs, defaulting to the more desirable admin generated. We moved away from this model for fear it would confuse users.

“Instead, and sorry to suggest an admin overhead here, authors have to request a new criteria if they want to have one added – with their reasons. The effort involved should discourage the frivolous.”

Yes, this was actually the original thought with user generated criteria. The problem, as you mention, is admin overhead.

Based on all your thoughts, here is what I am thinking:
1. Combine the duplicate criteria to clean things up.
2. Limit new criteria to 4 per item (but keep existing items as is – if they have tons, so be it – too much madness to get people to remove).
3. Disallow new criteria creation, yet add a “Suggest a Criterion” link. Allow the RG’s to approve/disapprove new suggestions.
4. Add a flag for “ridiculous criteria.” If a criterion is flagged enough, it will be reviewed by admin and not appear in search results for future additions (but also will stay on the items that have already added it).