Few-Shot Example Ordering

Expert

Calibrate the order of few-shot examples to prevent recency bias.

01 The Concept

LLMs suffer from recency bias, where they pay more attention to the last few-shot example in the prompt.

To prevent this, examples should be sorted by complexity or similarity to the query, or rotated/randomized to prevent bias and ensure uniform prediction accuracy.

02 Weak vs. Strong

EX 01Few-Shot Ordering Setup

# Rotate examples at runtime import random examples = [ex1, ex2, ex3] random.shuffle(examples) # Compile prompt prompt = """ Classify reviews: """ + "\n".join(examples) + f"\nReview: {user_review}\nSentiment:"

→ Why it works

Randomizing the order of few-shot examples averages out positional bias, preventing the final example from skewing classifications.

03 Key Points

01Recency Bias: The model favors constraints demonstrated in the final example.

02Rotational Sampling: Randomizing example order across sequential API requests.

03Similarity Sorting: Placing the most semantically relevant example last to guide the query.

04Complexity Balancing: Ordering examples from simplest to most complex.

05Bias Calibration: Testing different orders to maintain flat prediction distributions.

04 Model-Specific Notes

Claude is robust against recency bias, though placing the most complex examples first helps build reasoning context.

05 For Your Role

It's like showing a student 5 math problems: if the last problem is addition, they might try to add on the test even if it's a subtraction problem.

Next: Meta-Prompting for Code →