I’m going to let you in on a little secret: most of the reviews you read online are performed in a manner that you, as an intelligent consumer, would find abhorrent. I’m not naming names nor am I pointing fingers, but aside from a few very specific cases, your vision of a highly-experienced tech journalist sitting down at a workbench next to a Faraday cage and a drop test station is pretty much fiction.

This is a little bit of inside baseball, so bear with me or skip reading this.

First, I want to talk a little bit about the reviews cycle. This is the plan PR people have when sending out items for review. For years, that plan was simple: you fly to New York, drop off a few devices, fly back. All the print media there would futz with things and the go to press. This gave reviewers a month lead time, if not longer. I used to write for Laptop magazine and we had lead times of three months. Now, with the always-on Internet, reviews go up as quickly as possible. In fact, when you see a bunch of reviews go up at exactly the same time its because the company set an embargo for that date. Rather than risk looking slow, all the major sites pop up their reviews in unison. But almost everyone gets a few days before the review embargo is up.

If you’re a MAJOR MEDIA TECHNOLOGY WRITER at any MAJOR OLD AND NEW MEDIA PROPERTY you’re beholden to this for a few reasons. The primary reason is because it’s a holdover from the old days of embargoed news that had to be physically sent by mail out to the frontier lands by Pony Express. The second reason is that it lets the oldsters have plenty of time with a device before they cough up a review. I’m only being partially tongue-in-cheek about this.

So I’ll use the iPhone 5 as an example, although almost any major device follows this pattern. First, the announcement is made. In this case, the announcement was two weeks ago but announcements can happen at CES and devices can take months to appear or they can appear without warning – although some tech press still gets them early.

In the case of the iPhone 5, the cream of the tech press (MG, Mossberg, oddly not Topolsky UPDATE: Topolsky held his review to create frisson! How novel!) got early review units with express instructions not to show the device off to anyone. If there hadn’t been an announcement/handout event, the cream of the tech press would get the device a week or so early anyway, via FedEx or a “deskside meeting” with express instructions not to publish until (and this is increasingly not the case) either Thursday and/or the day of the official unveiling boozeathon that they usually hold in a major city. Why Thursday? Because that’s when Pogue and Mossberg publish their columns and in the world of PR having the NYT or WSJ to slip casually into your client’s clips file is like printing money. With the advent of the Internet, Pogue and Mossberg can now publish whenever but, like some sort of weekly Feast Day, Thursday was traditionally the Day Of Reviews.

But there’s a problem. One person spending one week with a device is a pretty small sample size. Whereas the proud men and women of the tech press pride themselves on working quickly, succinctly, and with a fervor for the facts that would make Mr. Murrow proud, they still only have a week to mess with this stuff. So you miss a lot. And I mean a lot. You miss Maps sucking, purple flaring, scratches, static. Considering how many iPhones were shipped and how many eyeballs ended up inspecting every cranny of the new device, it’s not surprising that these problems cropped up.

I would also posit that every other phone out there has similar problems. However, because this is the iPhone and everyone is staring at their iPhones at dinner, the problems are writ large. The early reviewers miss the problems because they’re enamored with the device. They don’t have it long enough to really see the problems (if any) or nit-pick on perceived problems. Now imagine this is for a less popular phone. The reviewers for those are far less thorough, which is why we stopped reviewing incidental Android phones: the temptation to give these phones a 6 out of 10 and call it a day is too great. The reader receives no value.

Readers will also yell that the writers just want to suck up to Apple/Google/Microsoft and so they won’t give anything a bad review. This is false. Most writers won’t write about bad stuff. I’ve seen so much garbage roll through my attic office that I could build my own little mini landfill. I’ve seen phones and tablets that were about as exciting as a block of concrete and devices with no earthly purpose. If we reviewed them all – like CNET does – we’d probably all go crazy. I’m happy to let CNET have the Google juice for a four year old HP inkjet printer. I have my pride.

More to the point, however, is that we can’t really trust early reviews. I always recommend caution when it comes to buying products that have just launched and I rarely take my own advice. Many devices only begin to exhibit problems after lots of use and many faulty devices pop up only after the first batch of highly scrutinized devices runs out.

So now you know a little bit about how the reviews process works and why you shouldn’t (or should) be mad at tech writers for singing encomiums about the latest and greatest: the problems you’re facing aren’t the problems they faced. They didn’t sit with the device for very long. Their review, while presumably thorough, was written over a few days and aims to paint a picture of the particular device in a way that offers a minimum of consternation. After all, they’re the lucky ducks who got this stuff early.

In the end, the real reviews are the ones that percolate up out of the forums and blogosphere. Devin wrote about this earlier, as well. In short, in the great drama of tech journalism, the players in the play mean well, and they are often right. But the plebeian chorus, in the end, always has the last laugh.

Original post: The Problem With Early Reviews