Inside the Educational Testing Sweatshop

City Pages a newspaper in Minneapolis wrote recently about the educational testing business and in the process gave both credibility and publicity to a book written by an insider. Here are several excepts from the article (City Pages):

Today, tens of thousands of temporary scorers are employed to correct essay questions. This year, Maple Grove-based Data Recognition Corporation will take on 4,000 temporary scorers, Questar Assessment will hire 1,000, and Pearson will take on thousands more. From March through May, hundreds of thousands of standardized test essays will pour into the Twin Cities to be scored by summer.

The boom in testing has come with several notable catastrophes. The most famous happened in 2000, when NCS Pearson incorrectly failed 8,000 Minnesota students on a math test. Pearson shelled out a $7 million settlement to the students, and Gov. Jesse Ventura participated in a makeup graduation for students who were wrongly denied their diplomas. In 2010, Pearson again miss-scored two questions on Minnesota's fifth- and eighth-grade tests. Delays in its Florida scoring resulted in a $3 million fine and glitches in Wyoming led the company to offer a $5.8 million settlement.

But while a mistake on a bubble form is a black-and-white problem, few scandals have broken on the essay side of the test-scoring business.

"It requires human judgment," says Michael Rodriguez, of the University of Minnesota's educational psychology department. "There is no way to standardize that."

Now scorers from local companies are drawing back the curtain on the clandestine business of grading student essays, a process they say goes too fast; relies on cheap, inexperienced labor; and does not accurately assess student learning.

There are several examples (all negative) that show the application of the grading 'rubric' to scoring essays:

Her first project was from Arkansas, an essay written by eighth-graders on the topic, "A fun thing to do in my town."

And that's where the troubles began.

Suddenly, she was being asked to crank through 200 real essays in a day. The scanned papers popped up on the screen and her eyes flitted as fast as they could down the lines. The difference between "excellent" and "good" and "adequate" was decided in a matter of seconds, to say nothing of the responses that were simply off the reservation. How do you score a kid who rails that his town sucks? What about an exceptionally well-written essay on why the student was refusing to answer the question?

All over the room, the teachers were raising their hands and disputing the rubric. Indovino preferred to keep her head down and just score the way she was told to.

The article peters out at the end but in the comments section was the following more balanced comment from JWHyperion (maybe he works in accounting):

When it comes to essay testing, there will always be controversy, just as there will always be differing opinions in the classroom when a teacher gives you a "C" on an essay and you KNOW you should have gotten an "A". There are things called human error and human subjectivity. I agree that essay testing for a standardized test has many inherent problems. Again, however, know that in most states the teachers write these questions and create the rubrics for how they should be scored. Also, most importantly people should know that the essay portion of the state assessment does not dictate whethere a student passes or fails. A student's overall score is determined by both MC questions and the essay question with the emphasis in scoring on the MC questions. Not all states include essay questions as part of their assessment. Many use a short answer response that asks students to elaborate upon information gained from the text while bringing in their own experiences and opinions. In this case, if a student writes on a completely different topic they receive a zero just as they would on a math test if the added two numbers incorrectly.

There are of course inherent flaws in the system just as there are inherent flaws in most every system. Instead of just complaining, though, it would be nice if the detractors could use some of their effort to help come up with a solution. As my principal always said, "Don't come to me with a problem unless you have a proposed solution."

MediaWeek (Vol 4, No12): Hay Festival, Reviewers, Heart of Darkness, Alice in NYC,

Ahead of the Hay Festival in the UK, The Telegraph has travel tips. PND hasn't been to the festival but the town and area boast very pretty English country side. Worth a drive. (Telegraph)

So nobody was really surprised when, 24 years ago, a recent graduate called Peter Florence decided to put on a literature festival. The first Hay Festival sold tickets from a caravan under the clock tower for events that took place in the British Legion and the back rooms of various pubs. The public proved enthusiastic. The festival moved to the school, expanded into various tents, and (still under the directorship of Florence) migrated to purpose-built sites to the west of the town, expanding as it went.

Jimmy Carter appeared and so did Bill Clinton. Authors and their agents began to lobby passionately for slots. The Italian province of Lombardy, twinned, bizarrely, with Powys, asked for Florence's advice in setting up a festival in Italy. The idea caught fire. This year, and with the support of the Telegraph, there will be Hay Festivals not only in Wales but in countries including Mexico, Spain, India and Kenya.

The original festival has continued to grow and the organisers are confident that they will comfortably top the 200,000 tickets sold last year, with events featuring Nigella Lawson, V S Naipaul, Rowan Williams, West Wing actor Rob Lowe, Philip Pullman, Paul Merton and a host of other luminaries.

Hay Festival: writers' tips and recommendations Some of the key figures appearing in Hay this year, including the comedian Marcus Brigstocke, offer their tips for what to do and see in Hay-on-Wye (Telegraph)

Are women reviewers that scarce? The Independent looks into it:

A few weeks ago, Vida, an organisation for women in the literary arts, caused a stir when it surveyed a year's reviews coverage from major US publications (as well as the UK's London Review of Books and the Times Literary Supplement) and found that the reviewers and the authors reviewed were predominantly male. The LRB was a fairly typical case: in 2010, men wrote 78 per cent of the reviews and 74 per cent of the books reviewed. And a brief (and admittedly unscientific) survey of UK papers reveals a similar result.

Looking at the past week's books coverage in 10 major newspapers (excluding this one – for now), we can see that 71 books by men and 37 books by women were reviewed. Of the reviewers, 68 were men and 36 women. (One paper carried 17/20 reviews of male authors, and 18/20 male reviewers.)

You can do your own maths on the books pages here, but bear in mind that all of the reviews were commissioned before this article was (so we're not cheating), and that the results are skewed by the fact that this week's paperbacks reviewer is a woman. This week it's seven female authors/subjects to five men, and seven female reviewers to four. Last week, with a male paperbacks reviewer, and including the interview (of a woman, by a man), we could count authors at six men to seven women; reviewers at nine to four.

About the lasting power of Conrad's Heart of Darkness (Intelligent Life):

Published in 1902, “Heart of Darkness” had an immediate political impact—it was widely cited by the Congo Reform Movement—but it wasn’t an instant classic. Twenty years later, T.S. Eliot wanted to choose a quotation from it for “The Waste Land”, but his colleague Ezra Pound dissuaded him: “I doubt if Conrad is weighty enough to stand the citation.” Three years after that, Eliot chose another line from the novella as one of the epigraphs for “The Hollow Men”. (“Mistah Kurtz—he dead.”) “Heart of Darkness” had become part of the cultural landscape. In 1938 Orson Welles adapted it for a Mercury Theatre live radio production. He went on to write a screenplay in which he planned—with Wellesian gusto—to play both Marlow and Kurtz. He couldn’t get the movie made and was forced to move on to a project about a media mogul called Charles Foster Kane. In the late 1970s, Francis Ford Coppola took “Heart of Darkness” and transplanted it to Vietnam as “Apocalypse Now”, with Martin Sheen as Captain Willard, the Marlow character, a special-operations officer, sent on a mission to “terminate…with extreme prejudice” the life of the deranged Captain Kurtz, played by Marlon Brando. In this version the Mekong River becomes the Congo. The irony was hard to miss: just as Britain, once colonised by the Romans (“1,900 years ago—the other day”, writes Conrad), had become an imperial power, so had the United States, once colonised by the British, now become one too. With the success of the movie, the novella’s place on the campus syllabus was assured. “Apocalypse Now” launched a thousand sophomore essays comparing and contrasting the book and the movie.

A visual dictionary (TheAtlantic)

Alice is alive and well and apparently visiting New York (The Atlantic):

Sin City Fables looks intriguing, I'll have to hunt out a copy. The reference materials we used for the project were photos of the city from around the 1920s and '30s. I now have a wall here plastered with old photographs of Manhattan. But we weren't religious about representing any particular era, and the New York that Alice explores in the book is more of a dreamy amalgam of a timeless New York. The reason I picked New York is that (aside from my love of the city) I was looking out across the island from the Empire State Building observation deck last summer, and had a sudden epiphany that Manhattan could be retrofitted onto the original Lewis Carroll book with supernatural accuracy. The chessboard world in Through the Looking Glass found an exact equivalent in the grid-system of the New York City streets. The Red Queen became the Statue of Liberty with very few changes to her character, and locations like Central Park and the subway system matched other key scenes. It was almost like Lewis Carroll had planned it that way.

From the twitter (@personanondata):

CNN Sending 400 To Cover Royal Wedding, 50 To Japan OutsideBW

Valerie Plame Wilson to Write Series of Spy Novels - NYT

Are mid-list authors an endangered species? - Globe&Mail

David Carr Mocks Idea of 'Journalistic Independence' at Murdoch's News Corp. MRC

Houghton Mifflin CEO O'Callaghan Steps Down - WSJOnline

Kyoto 1972: Temple of the Golden Pavilion (Kinkaku-ji)

Kyoto 1972: Temple of the Golden Pavilion (Kinkaku-ji)

A weekly image from my archive. Click on the image to make it larger.

This image is from the same roll as one of the first I posted in this series. I wasn't on this trip and I've not gone beyond the hallways of Narita on my infrequent trips to Japan.

Looks very peaceful here.

Join me on Flickr

I Steal Stuff

I remember a conversation early on in my tenure at my last company that ended along the lines of ‘well, you know, consultants just take credit for the ideas they hear from the employees’. There is some truth to this and, having been on the employee side of that observation/accusation, I can attest it can be demotivating to hear consultants spouting your ideas and concepts to a room full of people. Consultants (me, for example) might counter that, if they’re doing their job well, they are able weave together the implications of discrete ideas and potential strategies across the organization. That, of course, isn’t always within the purview of individual employees.

Every one of the operations jobs I have undertaken in my career has either been a new position or I’ve replaced someone who was fired. I’ve never come into a job where the person who held my new position was still the with company. What I’ve found consistently is that good ideas and strategies for building or expanding the business were all in plain sight and that, with a little diligence and execution, success can come quickly. Another interesting aspect of my career is that I’ve been hired into positions where I had little to no specific experience. I have also found that skillful ‘adaptation’ and execution of the ideas already known or present in the organization can aid in building a deeper and more rapid understanding of the dynamics of the business you’ve been tasked with operating.

Early in my career at Berlitz, I was given responsibility for their direct-mail business, selling language learning materials in airline and travel magazines. I had no experience in direct mail and the manager in charge of the unit had neglected the business and was no longer employed. Direct mail is all about testing and math, and the prior manager had done a test which called for placing a four-color tri-fold insert in the American Airlines Advantage mileage statement received each month by members. The test had been done six months before I took responsibility and had been successful (it actually made money) but, through inattention, he had not expanded the program to the full membership list. My staff told me about this and we decided to go all in with a full mailing to over 12million recipients. When I inherited the business mid-year we were already running well short of budget but, thanks to the strength of this program and other improvements, we made our budget that year with room to spare.

That American Airlines program was a great start and in the following year we placed the insert four more times. We also explored other, similar mileage programs but none performed as well. With this strong performance as a foundation, we began looking at other aspects of the business for opportunities. I had a small telemarking team that took all the incoming phone orders but this team had never been properly trained (nor really paid any attention). As I began working with them it became clear they could do much more to expand sales. One aspect of language self-learning is that very few people finish the course. We even used to joke that no one ever got past the first of the twelve cassettes. (Cynical I know, but the customer generally blames themself not the product – just like health clubs). Berlitz offered a second-level product for the same price but, unsurprisingly, less than 10% of buyers purchased the second (more advanced) level course.

We now had a tremendous number of calls coming in to our little telemarketing operation due to bigger, more frequent mailings to American frequent flyers and all the other direct response ads we continued to place. As a team, we determined that if we could hook the callers into purchasing both level 1 and level 2 at the same time, we could potentially generate far more revenues per call. As we implemented this upselling activity we also put in place the first sales targets and rewards program the team had ever had. Starved of attention and recognition by prior managers, my little team of telemarketers – who worked in an office park in Delran, NJ – took to the program like ducks to water. What, in retrospect, seem like quite basic operating improvements were implemented and ultimately very successful because of a fresh perspective on and an open attitude to the potential of the business and the employees. Some of the improvements occurred simply because someone listened more attentively to the idea.

Obviously, business improvement opportunities are not always so easily identified but they can be found by anyone with an open mind about the business and, perhaps, a different view of how the ideas could be implemented. In this context, anyone in the business can be a “consultant” and “steal” ideas that, in retrospect, might appear to be in plain sight. In the second year of my tenure, our direct mail business doubled profit and contributed twice the profit per employee than any other Berlitz business unit and we did it all by working with what was already in the business.