Production logging is critical because LLM systems are probabilistic. When something goes wrong, we need enough information to debug the issue.
Good logging should cover:
But we must also be careful about privacy and sensitive information.
Log basic request metadata:
The request ID is very important because it connects all traces for one interaction.
Log the final prompt sent to the model or at least a structured representation of it.
Useful fields:
Prompt versioning is important because model behavior may change after prompt edits.
Without prompt logs, debugging becomes difficult.
Log model configuration:
This helps reproduce behavior.
Example:
Without model version logs, this is hard to track.
For RAG systems, log retrieval details:
This helps answer:
Log the model response:
Finish reason matters.
Example:
means the model stopped because it hit token limit, not because the answer was complete.
For agents, log every tool call:
Also log:
This is essential for debugging agent failures.
Log safety and security signals:
For enterprise systems, this is very important.
Log operational metrics:
These are needed for optimization.
Log user and evaluator feedback:
This creates data for continuous improvement.
Do not blindly log everything.
You should protect:
Use:
In production, I would log the full lifecycle of an LLM request: user query, prompt version, model version, parameters, retrieved chunks, final answer, citations, token usage, latency, cost, and errors. For RAG, I would log document IDs, chunk IDs, retrieval scores, reranker scores, and final context. For agents, I would log each tool call, tool input/output, step count, retries, failures, and termination reason. I would also capture safety flags, prompt injection attempts, user feedback, and evaluation scores. At the same time, logs must be privacy-safe using masking, encryption, access control, and retention policies.
Monitoring agents is more complex than monitoring a normal chatbot because agents take actions.
A normal chatbot usually does:
An agent may do:
So we need to monitor the full execution trace.
Track each step:
This helps understand why the agent made a decision.
Tool calls are high-risk because they interact with external systems.
Monitor:
Example failure:
This indicates a loop or poor planning.
Agents can get stuck.
Monitor:
Example:
The runtime should stop the loop and generate the best possible answer or escalate.
An agent should be evaluated on whether it completed the user’s goal.
Track:
Example:
Agents need strict permission controls.
Monitor:
Examples of high-risk actions:
These should often require human approval.
Agents can be expensive because they involve multiple LLM calls and tools.
Track:
This helps optimize workflows.
Before returning the final answer, validate:
For structured outputs, validate schema.
For tool actions, validate state.
To monitor agents, I would trace the entire agent execution, not just the final response. I would log each step, selected action, tool call, tool input/output, latency, errors, retries, and final status. I would monitor for repeated actions, no-progress loops, max-step termination, tool failures, permission issues, and unsafe actions. I would also track cost, latency, token usage, and task success rate. Since agents can take real actions, monitoring must include safety checks, human approval events, and final output validation.
LLM-as-a-judge means using an LLM to evaluate another LLM’s output.
Example:
This is useful because human evaluation is expensive and slow.
But it has risks.
LLMs may reward answers that sound polished even if they are wrong.
Example:
This is dangerous because hallucinations often sound confident.
If comparing two answers, the judge may prefer the first or second answer due to ordering effects.
Example:
Mitigation:
If the judge model is similar to the model being evaluated, it may prefer outputs that look like its own style.
This can overestimate quality.
Example:
An LLM judge may not know the correct fact unless given source context.
For factual evaluation, the judge should be grounded.
Bad judge prompt:
Better judge prompt:
LLM judges can be non-deterministic, especially with sampling.
The same answer may get different scores across runs.
Mitigation:
If the evaluation criteria are vague, the judge produces unreliable scores.
Bad rubric:
Better rubric:
If the answer or source contains malicious text, it may influence the judge.
Example:
The judge must treat evaluated content as data, not instructions.
LLM-as-a-judge should not fully replace human evaluation for high-risk domains.
Use human review for:
Use LLM-as-a-judge with:
LLM-as-a-judge is useful for scalable evaluation, but it has risks such as bias toward fluent answers, position bias, same-model bias, inconsistent scoring, weak factual verification, and vulnerability to prompt injection. It can also overrate answers if the rubric is vague. To make it reliable, I would use clear evaluation rubrics, provide source context for factual checks, use low temperature, ask for structured scores, randomize answer order, calibrate against human labels, and keep human review for high-risk cases. LLM-as-a-judge is a useful evaluator, but not a complete replacement for deterministic tests or human judgment.