Prompt engineering and the model parameters

Four parameters, one instruction block, and a rule about which lever to reach for first. Most of the marks here come from knowing precisely what each parameter does — not roughly.

What each parameter actually controls

Temperature and top-p both control randomness. The important operational rule: tune one or the other, never both at once. Adjusting both makes the effect of either impossible to reason about. Low values give deterministic, repeatable output.

max_tokens caps the length of the reply only. It is not a budget for the whole exchange. The context window is shared between your prompt and the response, so a long prompt already reduces the room available for an answer. When a reply is cut short by this cap, it shows up as finish_reason: length — worth checking for, since a truncated answer otherwise looks like a complete one.

frequency_penalty reduces repetition of the same tokens.

presence_penalty pushes the model toward new topics.

Those last two sound interchangeable and are not. One discourages saying the same thing again; the other encourages moving on to something else.

The system message

Sometimes called the metaprompt. It is the standing instruction that shapes behaviour across the whole conversation: persona, scope, tone, output format, and what the model should and should not do.

Techniques that reliably lift quality:

• Few-shot examples — showing two or three worked examples in the prompt rather than describing what you want.
• Chain-of-thought — asking for the reasoning steps rather than only the conclusion.
• Breaking the task into sub-tasks instead of asking for everything at once.
• Stating the output format explicitly, rather than hoping it is inferred.

Reflection, and why it is not chain-of-thought

Reflection or self-critique is a loop: the model generates, critiques its own draft against stated criteria, then revises. Chain-of-thought is a single pass that happens to show its reasoning. The exam distinguishes them, and so should you — a loop and a single pass have very different cost profiles.

Which leads to the rule that matters in production: always cap the iterations. An uncapped critique loop has no natural stopping point, and both cost and latency are unbounded. Two or three rounds, then stop.

Reach for the levers in this order

When the output is wrong, there is a correct order of escalation, cheapest first:

1. Prompt engineering. Cheapest and fastest. Always start here.
2. Retrieval-augmented generation. When the model lacks knowledge it needs. You are giving it the facts, not changing it.
3. Fine-tuning. Only when the behaviour is wrong — the style, the format, the way it approaches the task.

The distinction between two and three is the one people get wrong in real projects. Fine-tuning does not teach a model facts reliably; it teaches it how to behave. If the problem is that it does not know something, retrieval is the answer and fine-tuning is an expensive detour.

The trap

"The output must be deterministic and repeatable" points straight at temperature 0, and that is usually right.

But check whether the question names a reasoning model. Reasoning models ignore both temperature and top-p entirely. If one is named, temperature 0 is not the answer no matter how well the phrasing matches — the answer is a different lever, because that one is not connected to anything.