How could P&J be improved?

You're asking Chuck Eddy & others at the Voice to do alot(as well as the contributors to the poll). Rather than (or in addition to?) trying to read the Voice and other publications and listen to music wherever throughout the year so that you can figure out which cds are dance and which are metal, you want the Pazz & Jop to be a handy-dandy guide to all....Hmmm. For now you can get a country list by going to the Nashville Scene; Latin Beat has a mostly Latin-jazz and salsa list; The Beat covers Afropop and Caribbean and some South American and Asian;Living Blues has a list I think; there are various lists covering mostly indie-rock...

Christgau and Eddy are generalists. Whether they can take the time to tally votes in specialized genre fields as well(some in your view invented by voters), well Chuck to thread...

"I'm not a regular Voice reader. I don't write for it, I don't live in New York, I don't happen to share Christgau's tastes or appreciate his reviewing style. I come across individual reviews from the Voice sometimes, but not enough to care about the paper for that reason."

I hardly ever watch pro football. I don't think it's very interesting. I don't read about it in the paper. Yet I think the Super Bowl would be way cooler if it was more like MTV's Rock 'n' Jock. For the sake of football fans everywhere, I'm hoping the NFL doesn't care what I think.

Also, Steve's right: there are specialist lists everywhere. Besides, specialist lists are generally less useful to me than generalist lists anyway. I'm not really a metal fan, so I'm more likely to enjoy the one metal record a bunch of other non-fans got off on this year than the ten records that diehards like.

I didn't say I didn't like music, I said I didn't read the Voice. The football analogy would be that you don't watch normally watch that network, yet do care about football and have some ideas about how to broadcast the Pro Bowl to take advantage of how it's different from regular football games.

But hey, if everybody but me thinks the poll is perfect as is, fine.

what do you do 51 weeks of the year, dare I ask?

I mean, you don't have to pay close close close attention to music writing those 51 weeks or anything, but you make it sound like you've never heard of ANYTHING on the lists, and if you're posting on ILM I have to imagine that's not the case. besides which, isn't it just easier to look at the reviews on Amazon or All Music Guide to figure out genres and such than to wade through P&J when it's obviously not set up for that purpose?

Oh, I spend the other 51 weeks obsessing about music, and I didn't mean to imply otherwise. There were three albums in the top 40 that I didn't happen to know at all (Drive-By Truckers, Fiery Furnaces, Four Tet), and maybe a dozen more in 41-120, so I think I'm fairly well informed. I feel like I'm exactly the kind of listener/reader the poll is aimed at, which is why my feedback might be helpful. But maybe that's my mistake, and in your mind the poll is aimed at someone who knows even more than I do, or far less. Which is why I asked what you feel the goal is.

And maybe I should have double-emphasized at the beginning that I'm trying to be productive, not contentious. I'm a system- and information-designer in my professional life, and it seems to me that the P&J process and presentation aren't taking best advantage of their resources. But that's "best" according to my own perspective, because that's all I have to go on so far.

So I'm really, genuinely, non-rhetorically asking: What do you feel is the goal of the poll? We've had threads discussing the makeup of the electorate, and the Voice appears to have made concerted efforts to change that. Yet the mechanics of the poll haven't been altered at all in a while. There might be good changes to be made to the process, too. But if nobody else suggests any, and this just becomes a debate about my sample suggestion, then maybe I'm just wrong.

From a market researchers POV the only thing wrong with the poll is that its competitive set is based on unpromted recall, which means that familiarity (i.e. how many people have heard/heard of a particular record) has a massive influence on the results. But that's fine - that's part of how pop music works, so the poll is reflecting actual market conditions.

Prompted-recall polls (like Radio On and the FT Focus Group) don't have this issue but have the much bigger problem of relatively limited scope.

In an ideal world where Eddy, Christgau and their interns were robots who didn't need sleep or fun what I'd like to happen is for a prompted follow-up to the previous year's ballot to be sent out to critics with the ballots - a simple mark-out-of-x list of the last year's Top 40, which would show how tastes have changed/the records have aged, how wide each record's appeal is and how controversial each pick might be.

I'd also like to see much MORE number-crunching because I'm a stats geek, along the lines of Nate and Matos - ppl who voted for the #1, women voters, young voters, old voters, the overlap between each of the top 10, etc etc. Again this would be lots of work (though a lot less work than the other ideas here, depending on what database software the VV is using).

I said it once before, but I'd love it if the critics were forced to provide a list of every album that they considered before choosing their top 10. Knowing the ratio of critics who listened to album x/critics who liked album x would be pretty valuable.

I'd love it if next year's P&J only had one thread.

A way of getting at the exposure idea in Tico Tico and dlp's comments, obviously, would be doing the poll in two passes. The first pass would be to collect nominations, the second to vote on those nominations. This doesn't have to be a lot more complicated than the current poll. In the first pass, everybody just submits ten nominations. Anything that gets more than X nominations is placed on the ballot, and the same people then vote for their ten picks from the ones on the ballot.

If we treat this year's poll as a sample nomination round, cutting the field off at albums with ten or more votes would give us 100+ nominations. Cutting it off at five would give us more like 250. Running the poll this way would involve more work communicating with voters, but less in processing their input, so it doesn't seem obviously impractical to me. And it might produce some very interesting results, as people whose nominations didn't make the ballot weigh in on the records that did.

xpost: the goal of the poll seems simple enough: to measure (American) critical tastes for the year. if it's not a reliable indicator of them it's more to do with the low turnout of certain types of critics (hip-hop, country, techno) than anything, which is a stray observation and not really what you asked. the "51 weeks" comment was a genuine question and I'm glad you took it as such, I come across as far more sarcastic than I intend to sometimes.

thing is, the minipolls idea is a good one--it just seems wrong for P&J, not in essence, given P&J's generalist stance (and the Voice's, and Christgau and Eddy's). Christgau has written/stated many times that he'd like to cut waaaay back on the number of contributors, and so would lots of people, but it would be a matter of picking out who to cut back, which becomes as much a problem as anything. (part-timers with great stuff to say vs. loyal full-timers with zilch to say tends to be the biggest headache, in a fantasy who-to-cut-from-the-rolls game.)

And here are a couple ideas that only involve changing the presentation, not the polling mechanics:

1. In the online version, link each entry to its original review in the Voice, if there is one.

2. Better yet, or also, include a short blurb about each of the top 40 or 120 or something entries. You're already doing the work to cull through contributors comments on the stuff at the top of the list, so dig a little more and find one choice snippet for each thing a little ways further down the list. This is farily standard practice for top-X lists, and makes for significantly more interesting reading than a straight list.

I don't think the two-passes idea is a very good one, partly because tabulating P&J is already a tedious, lengthy process (I've helped in the past and can assure you that ballots sometimes get triple-checked), partly because removing nine-mentions-or-less albums makes it essentially the Grammys, which it sort of exists in opposition to, at least philosophically. Obviously the tastes are different and whatnot, but the weird math of it (the varying point totals etc.) are a large part of its appeal to longtime watchers. Also, what about the fact that a good many of the participants don't even hear what end up with ten or more nominations? And why would I want someone like Banning Eyre, an African-music who almost never votes for anything that makes the top 200 much less 40, weighing in on Justin or Broken Social Scene when he obviously isn't interested? You know?

xpost: I think 1. is a good idea, actually, though again, no everything in the poll gets reviewed in the Voice. 2. though is impossible, because they don't even get usable comments about all of the top 10 most of the time! (Christgau regularly asks people to weigh in on one or two top 10 albums because he doesn't have anything worth reprinting about them in his comments file.)

djp otm

The other big difference between P&J and something like the Grammys or Oscars is that there is no such thing as a big promotional push for the Pazz & Jop Critics Poll. In theory there might be--some publicists send critics a checklist of the albums they worked that year as a reminder for year-end listmaking--but you're assuming all the writers/critics have access to all the records, which just isn't the case, and wouldn't be even if it were 1999 and Napster were still going ballistic. LOTS of people don't download, myself included, from free public file-sharers like Kazaa or Soulseek. I don't have access. And I'm not about to pay $10-$20 a pop to check out every single album in the top 40, much less 120, if I haven't gotten it or unless it's already piqued my interest.

I'd love it if next year's P&J threads weren't full of people bitching about P&J threads

then don't have any

that's telling 'em!

nature of the beast

Matos: Sure, I understand the generalist point, although looking through the results it's easy to see that lots of the current voters aren't generalists individually. I still think the poll is effectively throwing away a huge body of valuable knowledge about what the music in question is, as opposed to how many points it's worth. But adding blurbs (like I suggested above) and poking around in the numbers (as Tico Tico suggests) might start to get at this knowledge a little. I was just trying to think of a way to solicit more-structured descriptive information than you currently get in comments.

xpost: Yes, the Grammys use a nomination pass too, but a 250-album ballot nominated by critics is pretty different from a 5-album ballot nominated by Quincy Jones, yes? As for the fringe critics, the fact that someone's top ten doesn't have much from the top 200 doesn't mean that they have no input at all on anything in the top 200. (And if they literally have no input on anything that more than ten other people nominated, maybe they shouldn't be voting in this poll.)

Also, you'd get a lot more blurbs if people knew the best ones for the top 120 albums were going to be printed right in the list. You could help by putting fields for blurbs right in the poll, rather than soliciting them separately.

xpost2: Yeah, I don't think the nomination pass would prompt remedial listening much, but it might compensate for some amount of balkanization. Like, we know that more people put Outkast in their top ten than Basement Jaxx, but we don't actually know from that whether more voters prefer Outkast to Basement Jaxx. I wouldn't have voted for Lucinda Williams, the Postal Service or Belle & Sebastian, myself, but I have a very strong opinion about which of them is best, and I'd rank all three of them above everything in the final top ten.

Just to be clear, I'm not saying that my personal preference for B&S over FoW means the current poll is flawed. I'm just using myself as an example of how the two-pass approach would yield different votes from the one-pass approach, and thus different results. Would they be significantly and interestingly different? Hard to say.

The P&J List is just that. A list. There are never any shortages of critical discussion on the top-ranked albums in the collective music press.
I like that there are specialists and generalists all up in the mix. I'm a generalist who tends to write for specialized mags and it drives me a little kooky sometimes.
But the P&J threads are almost always more enlightening, cynical whinging included, than the list itself.

I agree with Huck--it's a lot of fun to argue about, a lot more so than the list itself. and ara, I do think your suggestions are interesting and potentially a lot of fun in a what-if way, which is probably how you intend them. But enough people vote for albums or singles that had made the top 5 the previous year (not to mention not ranking their lists in any fashion) that it's hard to imagine something like those ideas working even if adopted.

Since all the voting data is open to the public anyway (unless there's some opt-out on the ballot I've always missed) one very cool thing the VV could do is make the data available as a downloadable file, in SPSS or some similar program. Imagine the stats pleasure! And the threads!!

and the moany protestations!

Let's make ILM one big thread.

Yes!! I mean can you imagine how many people would get annoyed by a bit of segmentation analysis on the results!

It would keep Glen McD. going for months too!

like he needs the help!

Bring on I Love Statistics!

Add a "B".

P&BJ = insert Dave Chappelle R. Kelly parody/ies here

in response to a couple of things Matos and Tico said upthread: If my memory serves, at least one year the Voice published top 10 breakdowns based on different voter criteria (can't remember which, but I think race, age, and gender were all taken into account), similar to what Matos, Nate, and I did on ILM. Like Tico, I'm a stat geek who finds these sorts of questions interesting. There's a limit to what enterprising ILMers can do with this (see me trying to guess gender by voter name on that thread), so I'd love to see the Voice do those sorts of breakdowns again.

It was done in at least one poll, right Chuck? (this was before I was a voter)

>>Christgau and Eddy are generalists. Whether they can take the time to tally votes in specialized genre fields as well(some in your view invented by voters), well Chuck to thread...<<

Well, if people don't mind the poll coming out in June, I guess we might be able to pull this off. Though do you really want ME to decide which records count as "metal" or "country" or "hip hop"? If so, you have no idea what you're getting yourself into, dude....

>>in response to a couple of things Matos and Tico said upthread: If my memory serves, at least one year the Voice published top 10 breakdowns based on different voter criteria (can't remember which, but I think race, age, and gender were all taken into account),<<

It was sometime in the mid '80s (1987 or so); I think Bob did top 20 lists of albums among women, black voters, voters over 40, and voters under 30. Something like that. He says it was a logistical nightmare, though, and now that I've run this poll for four years, I totally understand. In the past there have also been separate polls for EPs, reissues, compilations, and local bands. All of which seemed fun to me--back when I wasn't the main person spending every waking minute from late December through early February doing the tabulating!

I sort of miss the reissue category, and thought compilation was an interesting idea, though it didn't really develop.

And oh yeah, Bob and I have both long agreed never to let people pick from any kind of list of "nominated" albums (which the new media techies here have suggested more than once, to make their lives easier). I have no idea what good it would do. It would make the results less reflective of what critics actually liked, not more.

And we're also not interested in printing some hack's half-assed comment about the 120th highest finishing album in the poll. As Matos said above, it's hard enough to get interesting comments about albums in the TOP TEN. Most of the comments we're sent are about specific albums on people's lists, and most of them are really boring. But Bob does pick his favorite comments about non top 10 records as well, either for publication or for the website. The best comments sent are almost inevitably NOT about a specific record; in my mind, the "Top Ten" comments page is ALWAYS the dullest of the comments pages in the print edition. If you want to know more about, I don't know, the Unicorns record, it's really not that hard to do a google search. If you want to see what the Voice wrote about it, search the website!

Just to be clear, the point of my idea about minipolls or categorization or whatever was to let the voters do the categorizing. I think that idea could actually be implemented without hugely increasing the complexity of administering the poll.

Also, the nominations-first idea could reduce the amount of work required to tabulate results. In the first pass all you'd care about was whether an album got ten nominations or not, so instead of having to reconcile every variation on that OutKast title in order to get an exact count, for example, after the first ten mentions of it you'd just skip the rest. Sorting into two piles is always much easier than counting everything. Not to mention the fact that you wouldn't have to keep track of who nominated what.

But Chuck, since you're here, are there any ways in which you wish the poll could be improved? There's an audience of geeks here that might be able to think of ways to pull off things you're imagining are impractical.

xpost: OK, sounds like somebody has already make the practicality case for the nomination approach to you, although I agree with you that having you pick the "eligible" stuff would miss the point.

If I read ara correctly, what she's suggesting is that voters submit two (or be able to submit two) ballots: a general one and a genre-specific one in the genre of their choice, and for those second ballots to be tallied for each genre. This is probably untenable and will never happen, but is an interesting idea.

A couple of problems: it sort of assumes that most voters are generalists, and I'm not sure if that's true, so the two different ballots might be mostly the same for a lot of critics. Another problem is that the genre ballots would be more prone to voter mischief than the general ballots (because of the smaller sample).

the mere thought of this is probably giving Chuck a heart attack.

these suggestions are ass

But what would be the point of having ANYBODY nominate the albums, ara? It would be more work, not less. (Where our work comes in as at the TOP of the poll, not in records beyond the top 200. Those lower placements are figured almost entirely by computer.) And again, the results would be LESS ACCURATE. And less interesting. People would be voting for albums they like LESS than the ones the ones they vote for now. That seems obvious to me, by definition. It might also aggravate the bandwagon bullshit where people try to second guess which albums would draw a concensus. It's a useless and impossible, if well-meaning, suggestion. You also seem to think 750 voters would WANT to vote twice. Do you have any idea how hard it is to get them to vote ONCE, and to follow even the simple directions now in place? What happens if 200 sit out the second round, which is very likely? You're completely deluded about the complexity of the poll, and the hours of work it takes to pull it off, ara. Voting by categories, too, would make the procedure MORE complicated, by definition. It would leave more room for error, too. There's just no fucking way around it.

I do kind of like the idea of linking album titles to Voice reviews, actually; I may even mention that one to the new media people. Though my guess is that they don't have the people-power to pull it off.

Eliminating the points system is ridiculous, too. Point-to-voter ratios say a lot. And if I'm interested in a particular critic, I WANT to know which album he or she thought was worth 25 points. How would dumbing lists down to ten points per album IMPROVE anything? You'd get less information, not more. But as I said on the other thread, if you can't learn more about music than what you already know from what's already in Pazz & Jop, you're not trying. Period. The problem isn't the poll; it's your own laziness about using the information that's there-- sometimes in conjunction with OTHER information at your fingertips, sure. But what's WRONG with that? And how hard is it?

Well, if it's critical consensus you're after, then throwing out the bottom 1500 albums and getting people to revote on the top 200 makes excellent conceptual sense, whether it's practical or not. Which is why I asked what your goal for the poll is. Imagine you had magic powers, for a second, and forget all the logistical details. What do you wish you could do?

As for "accuracy", right now the poll claims to report a ranking, but because the voting mechanism doesn't distinguish between ignorance, disapproval and eleventh-favorite, the results could be very inaccurate by various measures. It's possible that half the people who heard OutKast hated them, for example, while every single person who heard Stellastarr* voted for them. Hell, it's possible that everybody who didn't hear Stellastarr* would have ranked them above OutKast if they had. Those are senses of "accuracy". Interesting? To me and djp and maura above, yes. To you?

And no offense, but you're kind of not getting into the spirit of brainstorming. I'm not lazy, and I do find out interesting things by putting some energy into the poll. Yet I think the poll could tell me more interesting things. I bet there are things you wish it could tell you, too, or things you wish were easier to find out. What are they?

Actually, what somebody amusingly recommended once which might accomplish some of what ara seems to be proposing above is that critics also be allowed to cast NEGATIVE votes as well as positive ones -- say, maybe 30 points (or whatever) that could be distributed among as many as five albums (or whatever), and would thereby be deducted from the totals for those albums. That way, critics who, say, hated Cat Power or Wilco would affect their results as much as critics who love those acts. Still a logistical nightmare no doubt (rock critics SUCK AT MATH, trust me), and Bob would never go for it (I asked once!), but it might add SOMETHING to the poll, in a much easier way than the double-ballot modest proposal. (Actually, in the early '80s, the Voice used to have a mini monthly poll, featuring a dozen or so critics, and they WERE allowed to cast negative votes.)

The monthly thing was called the Pazz and Jop product report, by the way.

And ara apparently doesn't understand that I've probably spent more time whining about Pazz and Jop results than any other critic over the past couple decades. The big problems, though, seem to be with TASTES, not procedure. And sorry, but it's a waste of time to divorce procedural suggestions from their workability; I don't get that at all. I wish the Pazz and Jop poll told me more about hard rock, teen pop, Nashville country, lots of things. I wish it told me less about white indie pop bands with boring singers and no rhythm sections. But I've felt that way forever, basically. The problem is undoubtedly the composition of the electorate, though, not the voting methods; or really, the problem is with the kind of people who decide to write about music when they grow up, which seems to be somewhat limited. I am not sure how to change this. But yes, I'd love suggestions. (And suggestions about weeding down the electorate would also be helpful.)

Ara, when you keep suggesting "re-voting" it tells me that you are not listening to what Chuck is saying about the work involved in conducting the poll. He's the one who has to actually run the poll. Jeez. And your notion of throwing out votes for 1,500 albums to somehow better understand the the top 10 doesn't seem very democratic to me or designed to provide more information. Now, as I've suggested to Chuck elsewhere, I would love to see a handful more of specialist genre critics added(contributors to the polls of the Beat, Latin-rock zines, Latin Beat...) but that won't affect the top 10 very likely and who knows if those folks will even respond and submit ballots.

The winners always seem to attract detractors--Wilco, Outkast...

actually the top-ten-overall and then genre-specific lists is almost exactly how The Wire do their annual lists, but they only use about 30 writers.

Ara -- Why are you not INTERESTED in those bottom 1500 albums? I don't get that at all. Weren't you the guy on the other thread who was complaining about how Pazz and Jop told you about too FEW records? Or was that the other guy? Either way, your fewer records = more information brainstorm strikes me as completely contradictory.

No, no, I totally understand the questions and complaints about the electorate, and acknowledged those discussions explicitly in an earlier comment. I started this thread to see if there were also things that could be improved about the mechanics, even given the current electorate.

The point of entertaining impractical ideas is that sometimes recognizing a particular abstract value in an impractical idea can help you get to a practical idea that accomplishes it. The negative-points idea, for example, is still kind of impractical, but it's a whole lot more practical than having every critic list all the other albums they heard, and gets at the same idea. There might be an even more practical idea lurking behind that, although I can't think of it myself offhand.

Or take your desire to find out more about teen pop. If you think the current voters don't know shit about it, then it's totally an electorate problem. If you think they actually do have useful opinions about it, but opt to fill their limited ballots with Pavement-alikes out of complacency, then a separate teen-pop ballot might accomplish something.

xpost: Yes, personally I'm not very interested in consensus at all. But that's why I asked what your goals for the poll are, since those matter way more than mine. Throwing out the bottom 1500 would be worse for me, but if it made the poll accomplish your goals better, then it's a good suggestion for you, no?

So who told you "more consensus" was their "goal", ara? Where the hell did that COME from? I think the poll figures out consensus just fine, but that's never been ALL I want the poll to do. I want it to reflect as much of the year in music as possible. I want it to be a rollercoaster and a boxing match and a really fucking good Golden Earring song, which is what I said on the other thread, and which is one reason why so many comments and essays AREN'T limited to "which record was better than which other record." Which has always been only PART of Pazz and Jop. As should be pretty fucking obvious.

I just want to read my friends' ballots in a semi-ritualistic manner

I hear there are laws against that.

shhhh

Oh, and genre lists might be a way to cull the electorate, actually, if you thought you could hand-pick some genre voters. You would then effectively be rebuilding the electorate from zero with more focused criteria, which might be easier than just looking at everybody on the voting list and trying to figure out who to eliminate.

xpost: Matos said, above, "the goal of the poll seems simple enough: to measure (American) critical tastes for the year." I was using "consensus" as a shorthand for that, which I guess is debatable, but that's all I meant. The more people vote for fewer albums, the more "measurement" you get to do, since at the other extreme no consensus means a 7200-way tie for last.

And yes, obviously the poll tabulation is only part of the effort. But saying it's only part of the whole poll doesn't mean it too can't be improved somehow.

>>Matos said, above, "the goal of the poll seems simple enough: to measure (American) critical tastes for the year." I was using "consensus" as a shorthand for that,<<

Why??? Were you taking a Rohrshach test or something? (Those bottom 1500 albums represent American critical tastes too, don't they?)

OK, never mind. I think there's a perfectly reasonable conversation to be had here, but we aren't having it. Your poll is fine.

The Pazz and Jop Product Report was great. You should bring that back.

I've actually *thought* of bringing it back! I LOVED that thing! And it might be pretty easy to pull off, though it would take up space, and I don't have a whole lot. Who should be the dozen select critics voting in it, though? That might be the hard part....Also, there are about 500 times as many releases now as there were back in the old days or something, right? And the product report ONLY works if there's a consensus. Which might be extrmely tough, but who knows?

You really think it needs a consensus? I guess the running total could get a little weird ("this week Sterolab leads again with 8 points...") but I always thought the individual lists were where the action was.

Well, there was always that overall top ten chart at the top above all the individual lists, right? (I've also thought about filling space in the section with random "Ten things I'm listening to lately" lists like they used to use for fillers in *New York Rocker* in the early '80s, which might actually work better when you get down to it. I dunno. I wish I had more ROOM to do this sort of thing!) (And time!)

The points made it more fun, though. And I loved how Greil Marcus was always deducting points from things everybody else was voting for (Talking Heads, say).

I wish you had more room, too. Especially since the section is the best its been in years (and its always been good).