how i test products now: like a user, not like a reviewer, and why it changed what i write

product · 2024-02-09 · 5 min

for a long time i reviewed things the way reviewers are supposed to: open the box, run through the features, compare to the category, write the verdicts. it produced clean copy and weak recommendations.

the shift did not come from a better template. it came from noticing that the things i remembered after a week were never the things i had scored on day one.

the old way and why it misled me

the old way asked a product to prove itself in a session. set it up, exercise the main features, take notes, form an opinion. that sounds fair. it is also a weird way to learn how something behaves when it belongs to a person.

reviews built from a single session tend to overvalue first impressions and undervalue friction. the thing that feels impressive for twenty minutes can become annoying for twenty days. the thing that feels clumsy at first can disappear into routine. a review written from a demo frequently answers a different question than the one the buyer is actually asking.

the old way also trained me to write about specs and surface. megapixels, battery milliamp-hours, paper output, thread count, storage tiers. those numbers are easy to gather and easy to compare. they are also easy to narrate around while missing the thing that actually decides whether the product earns a spot in someone's life.

how i test now

now i start from use, not from spec sheets. i pick one ordinary job the product is meant to do, and i give it that job for long enough to stop being new.

not forever. not a month of tenure. long enough that novelty stops masking behavior. long enough that the first awkward workaround reveals itself as a workaround. long enough that i can tell the difference between a product that is good at being demonstrated and a product that is good at being used.

the test is usually built around a small set of real tasks, not a tour of features. if it is a phone, i do not spend the week trying to trigger every camera mode. i take the kinds of pictures i would actually take. if it is a notebook, i do not cycle through every app. i write, i move windows around, i leave it on a table and pick it up later. if it is a service, i do not click through the marketing tour. i try to accomplish the thing i actually want and notice where the service helps, where it gets in the way, and where it quietly assumes something about me that is not true.

what i pay attention to

i pay attention to friction that repeats. one-off annoyances matter, but repeated annoyances are the story. a product can survive a bad first minute. it cannot survive a bad first week if the bad minute keeps coming back.

i pay attention to the gap between the happy path and the real path. the happy path is what marketing shows. the real path is what happens when the thing is dusty, rushed, half-configured, interrupted, or used by someone who does not care about the product nearly as much as the reviewer does.

i pay attention to what the product asks me to forgive. every product asks for some forgiveness early on. the question is whether the forgiveness is temporary and narrow, or structural and broad. a setup step i do once is not the same as a workaround i am expected to live with.

i pay attention to what disappears. the best products are often the ones whose best feature is that i stop noticing them. a camera that gets out of the way. a keyboard that stops being a topic. a browser that becomes the place i read instead of the thing i am fiddling with. invisibility is not nothing. it is often the whole point.

i pay attention to the last thing i would have tested. the battery case, the edge case, the accessory situation, the way the product behaves when the network is bad, when the room is noisy, when i am not in a demo mood. the carefully staged test is useful. the unplanned use is where the product gets honest.

what changes in what i write

the biggest change is that i trust lived behavior more than first impressions. if something feels great on arrival and annoying by day four, i say so, even if the demo was beautiful. if something felt clumsy at first and then became easy, i say so, even if the opening paragraph would have been weaker.

i also write less like a spec translator. i still mention numbers when they matter. but i do not let a list of features become a substitute for a judgment about use. the reader is usually not buying a spec sheet. they are trying to decide whether this thing will make their day better or worse in the ways they actually live their day.

and i write more about the conditions i tested under. not in a defensive way. in a useful way. if i tested a phone mainly for documents and not for photos, i say that. if i tested a tool on one kind of project and not another, i say that. the reader can then decide whether my use is close enough to theirs to trust.

the bigger method behind this

the method is not review-specific. it is the same muscle i use when i choose any tool: pick the real job, use the thing for that job long enough to stop being a stranger to it, and judge the thing by what it did to the work, not by what it did during the presentation.

that method is older than any single category of product. it is just easier to see in a review, because reviews are where products get talked about as if familiarity were optional.

it is not optional. it is the whole thing.

for the device that prompted the spreadsheet: product testing one device. for the writing shift behind trusting lived behavior over first impressions: my first real product.