CAT 2009 was a nightmare in terms of how it was administered and everything that people experienced while taking up the test (server crashes, re-taking the test, etc.).
While everyone anticipated the changes in CAT2010, the team of IIM-Prometric-Meritrac-Everonn combined did a better job at administering the test.
When I thought all was fine, came the troubles with results. Not only is there widespread chaos (wherein people who were scoring in the higher percentiles in their mock-CAT got damning final scores), there is also the 'explanation' (the official one) to all these that has got my bowels moving.
I am not going to explain all the pain and hardships people go through to prepare for this very test, and all the heartbreaks that are caused when results are announced that are way beyond anyone's nightmares. I am not sure if I have ever heard of suicides post CAT results, but I guess that may change too.
My point is very simple. According to my understanding and expertise of test design, all thats explained as reasons to how a person got his or her percentile is shocking. I almost couldn't believe what I read and the more I dug into the statistics and concepts that they have used, the more terrible it became to understand what really went on in the name of this test this year.
Lets demystify:
Any scientific test (the Final test - like CAT 2010), when designed, involves a huge amount of pre-test work that involves -
Raw Test design -> Administering the test on a carefully selected Sample (related to the final test takers) -> Get Raw scores -> Use statistical tools (may include Item Response Theory or "IRT") to determine Item weight/Item characteristic score/etc -> Normalization of the scores -> Create an error-free "Final Test"
This way, certain things are taken care of, like -
a) the distribution of the difficulty levels in every question paper that may be unique from the other set
b) a proper test paper set wherein one person taking Test 1 would probably be scoring in the same percentile if he takes Test 2.
c) final scores, percentiles, etc.
What I understand from reading the official statements mentioning the process is:
- They have done the process upside-down
Beginning from: Final Test -> Raw score -> IRT -> Normalization -> Percentiles and percentages
Therefore the premise seems to be flawed.
The content on IRT and Rasch Model of conducting Psychometry analysis and developing test is available on Wikipedia - http://en.wikipedia.org/wiki/Item_response_theory; http://en.wikipedia.org/wiki/Rasch_model
If you read carefully, each of these are used for developing the Final tests, and not developing results.
So, what was CAT 2010? Was it a data gathering exercise for CAT 2011? Is it worth giving One year in one's career to face such a situation wherein there are now doubts on whether the whole psychometry and normalization process was mindless banter to cover up for a shoddy process?
Who will answer these questions? Some answers like "questioning the psychometry is like questioning maths" etc show the disdainful attitude to get things right. My heart pains for those aspirants who believe in this system and may today believe that they are really not good enough.
I am open for a debate on this. We can probably get the best guys from GMAT or GRE teams to talk on this on a open forum and tell everyone whats the actual process.
Filing RTI will never let us know if the process followed was right or wrong. We need global experts to make a point here. Else, the whole attitude of "we are the best in India and we can get away with anything and everything" will soon doom our management education system.
