one year of chatgpt: what i use it for, what i don't, and what i got wrong

ai · 2024-07-22 · 7 min

i used chatgpt daily for a year. not as a toy, and not as a replacement for thinking. as a tool in the middle of real work: drafting, debugging, rewriting, and a lot of "explain this to me like i have ten minutes".

a year is long enough for the novelty to wear off and for the actual pattern to show. some uses stuck. some disappeared. and the biggest thing i got wrong at the start was not about the model at all.

what stuck

the first draft. a rough outline, a bad first paragraph, a list of options. the model is good at producing something to react to, and reacting is easier than staring at a blank page.

rewriting. take a paragraph i wrote and ask for three versions: shorter, clearer, less annoyed. i almost never use a version as-is, but seeing three makes the edit obvious.

code i already understand. a regex, a shell one-liner, a small function, a type definition. the model is fast and usually right, and when it is wrong the compiler tells me immediately. the feedback loop is tight, which is why this use survived.

explaining unfamiliar code. paste a function and ask what it does. it is not always right, but it is a useful first pass before i read the source properly.

rubber ducking. describe the bug out loud in a chat. half the time the answer is in my own description. the model does not need to be smart for that. it needs to be there.

translation drafts. a first pass into another language, then a human edit. the grammar is fine. the tone needs work. as a draft, it saves time.

what did not stick

facts. anything with a date, a number, a citation, or a name. the model is confident and sometimes wrong, and checking the answer takes as long as finding it myself.

math. not arithmetic, which it can do, but multi-step reasoning with numbers. it makes a plausible mistake in the middle and then reasons confidently from the mistake.

long documents. summarizing a forty-page report produced a summary that missed the one paragraph that mattered. the summary was fluent. that was the problem.

legal, medical, and financial specifics. not because the model is useless there, but because the cost of a wrong answer is high and the cost of checking is low. i check, so i skip the middleman.

"write this for me" in my own voice. the output is grammatically clean and tonally generic. i spend more time removing the polish than i would spend writing the thing.

the biggest mistake

the biggest mistake was trusting the confident summary. in the first month i treated a fluent answer as a finished answer. it is not. fluency is a property of the writing, not evidence about the facts. the model is optimized to sound right. that is not the same as being right, and the gap is invisible when the topic is unfamiliar.

the second mistake was using it for the wrong part of the task. i asked it to decide, when i should have asked it to list options. i asked it to conclude, when i should have asked it to summarize. the model is better at generating possibilities than at choosing between them.

the workflow i kept

ask for options, not answers. three approaches, five titles, two explanations. options are cheap to evaluate and hard to be wrong about.

ask for the source. "quote the sentence you are summarizing" or "which part of this document says that". a model that has to point at the source is easier to check.

verify the cheap things, skip the expensive ones. a regex is cheap to test. a legal claim is not. put the model where verification is fast.

edit in my own voice. the last pass is always mine. the model can draft, but it cannot be the last reader.

the year in one sentence

the model got better, the tools changed, and the rule stayed the same: it is a draft machine, not an oracle. the people who get value from it are the ones who keep the verification loop short and the judgment on their side.

for a concrete example of the failure mode, see when ai gets it wrong. for the browser side, see the year of the ai browser.