One of the most fulfilling parts of the work that I got to do everyday at my last role was staying connected to both the academic and real life adoption research on AI. Now in the middle of job search, it's the habit that has still stayed with me. 

In such a critical period of transformation, it is so important to cut the sh*t and have a humbled, balanced and practical view. If we should have learned anything from the proliferation of social media and the impact that we are still dealing with, it's that we need to fully understand the tradeoffs we make in adopting technology, because like most things in life, the choice you make will always have consequences. 

Now back to AI. One of the things that I have made a practice of in using LLM is to have a retrospective (retros) at the end of my conversations. This initially started off as an exercise to improve results from the LLM as well as improve my techniques as a user. 

These retros would usually follow the following patterns:

1. I use AI for a task (research, brainstorm, analysis) and go through a process of finetuning, thinking through and editing to get to my assessment of what is good quality produced work.

2. At the end of the task, I ask the LLM to reflect on our conversation and 

   • Outline the process we went through and how it could have been better (more effective/efficient) in getting us to the final outcome. The intent here was to find ways to do more in less time. On reflection, I have found that this might have been a misstep. The real gem in the process of iterating to produce something unique or interesting that cuts through the noise is really in the using different frames of thinking across different disciplines to explore. Focusing on what techniques helped in what areas of tasks might have been a more valuable approach.

   • Review its own approach and performance on the task. Finetuning is the time sucker when you use LLMs. Sometimes you are not even sure why something feels off, you just have a gut feel like this sounds...(trying to figure out and articulate/define the gut feel was its own valuable exercise; if you have ever been an editorial manager of a team, it's a similar task of giving constructive feedback in a way that helps people understand the why and also grow)

   • Review my own prompts and identify gaps. If you are going to direct a task towards a path of excellence, you have to be self aware. 

Over time, these retros evolved into a reflection on my own use of LLMs because I started to feel what I describe as an initial Psychological Inertia in approaching tasks brain first. It wasn't necessarily an aversion to thinking, but more of a lack of motivation to organize that thinking for communication. So I would turn on the ChatGPT mic to transcribe and hit send. But i was somehow always dissatisfied with the output. It would feel like a misrepresentation of my thoughts or like someone else wrote my paper from my outline. I knew it would be better to just DIY and move on, but to put simply, I would be too lazy to and move ahead to finetune to get the output closer to my own thoughts.  

But this is not to say that LLMs are useless. In fact, they have been especially useful in helping me understand more technical and complex areas and papers in depth, as well as check my gaps in thinking. And it was in this quest to check my understanding of LLMs themselves I entered into the rabbit hole of having it meta reflect on the responses it gave me and why. I have not yet verified the learnings from these conversations (put a pin on it), but they were at the very least a beginner for a more critical view in having a deeper understanding as an adopter/user.

Full Conversation Log: https://chatgpt.com/share/6895d76f-0e70-8009-a159-c3109c1d90b5 

(Note this conversation was before the launch of GPT-5 and used the model 4o)

What I discovered was a deeper understanding and appreciation for tradeoffs LLMs make to produce outputs; plus a pause on my use of the tool on whether I want to have control over making these trade offs by using my own thinking...

- I noticed LLM gave me a response that seemed defensive to me from asking "Are LLMs a mathematical interpretation of language"
- I wondered why it seemed to need to qualify instead of explaining - i.e. "🧠 But It's More Than Just Math".
- It responded: 🧠 TL;DR: Your statement = what LLMs are. My addition = what LLMs do + why they seem like they understand. One is ontological. The other is functional. Both are true. The second just makes the first more useful in conversation."
- I still wondered why - "Why do you estimate it was necessary to state the functional?" "I'm curious on your why. For example, if the goal was to avoid oversimplification, why didn't you approach the conversation by expanding on the fluidity of applied math or explain what you describe as "high-dimensional space of probabilities". It almost seems like a defensive approach."

INSIGHT: LLM had a goal in the response it chose to give me that was beyond clarifying my understanding

Any research being done around next steps or effective strategies for the negative impact (in both design and use)