Why prompting is so important
Generative models don’t “understand” language the way humans do. They map words to patterns learned from vast training datasets and laborious supervised learning. At a sufficient scale, the end effect in modern large language models is as near to what we would call understanding as makes no difference.
But when you ask a human being a question, if they don’t understand you, they usually ask for more information. An agent might do that, while telling you how astute you are in asking the question. But as a result of their training they are usually just eager to try give you something, even if it’s a guess. So asking an agent a question becomes a skill in itself.
I’ve been experimenting with music generation models, which apart from being a lot of fun, really shows how sensitive generative models are to the instructions. If you get the prompt right, you can get some very pleasing results.
Apart from the obvious instructions about genre, tempo, instrumentation, and arrangement, you can get quite detailed, and specify instruments, chord progression, modulation, and precisely control the structure (intro, verse, bridge, etc.). I also like to give an audio sample – finally an outlet for the karaoke! But I still find it takes many generations (attempts) to get the result I want. You have to iterate.
Even small changes in the wording of the prompt can shift the model’s output dramatically. The models also tend to naturally drift towards a sort of generic mainstream pop if you don’t keep it on track.
For example, if you want to create a liquid drum-and-bass instrumental, you have to keep reminding it not to add vocals, otherwise a voice sounding like Taylor Swift will start singing nonsense halfway through the track.
Similarly, if you mention the word ‘trance’ anywhere in the prompt you’ll instantly get a thumping peak-time floor-filler, which might be quite good, but not quite what you wanted. Because ‘trance’ is not just a genre label; it’s a dense cluster of musical conventions: 130–140 BPM tempos, four-to-the-floor kick patterns, supersaw leads, euphoric breakdowns, and bassline drops.
The result is that a single key-word can override a whole sentence about slow tempos, soft pads, minimal percussion, and relaxed harmonic movement.
Agents prompting agents
Modern prompting guides put the emphasis on iteration. But iteration becomes far more powerful when you treat prompting as an agent-driven process rather than a single static instruction.
You can use an agent to analyse your prompt and suggest improvements. Or you can tell the agent what model you’re using and what type of output you want, and it will suggest a prompt for you to give to the generative model, including all the correct terminology.
I tend to iterate with a general-purpose agent such as ChatGPT or M365 Copilot to refine the initial prompt before sending it to the music model. Then I’ll refine the prompt until I get a generation that I like.
Whether you are prompting a music model, a coding agent, or another kind of generative agent, once you get a prompt that gives you results that you like, it’s a good idea to save it somewhere. I keep mine in OneNote, so I can re-use them.
Users of coding agents who are not developers often have poor results when they try ‘vibe coding’. One of the problems is that if you’re not a software engineer, you don’t know the terminology to tell the model what you want (assuming that you actually know what you want).
This is another area where there’s value in using an agent to compose the prompt in the first place, because it may be able to deduce your intent and then formulate it with the correct industry terms.
The anatomy of a strong prompt
Across multiple sources, a consensus has emerged: good prompts are structured, specific, and descriptive. Some people expand this into more dimensions, but the principle is the same: every missing detail is a decision the model will make for you — and probably badly.
As models grow more capable, their scope for getting it wrong also increases. They respond to nuance, but they also misinterpret nuance.
A few years ago we had models that could each do one very limited thing, using ‘weak’ AI. Now we have ‘strong’ AI in the form of very general models that have specialist knowledge on a comprehensive range of subjects.
But this creates a paradox: the more powerful or broad the capabilities of the model, the more carefully you must speak to it.
This is how professionals are already working with modern models; using agents to prompt agents and iteratively refining their prompts. And it’s how you can make sure that your Fado lament doesn’t come out as Flamenco!