docs/advanced-pipeline-examples.md

# Working with Advanced Pipeline Examples

This guide demonstrates how to load, modify, and run an existing advanced pipeline example, focusing on the two-step justified confidence model for tossup questions.

## Loading the Two-Step Justified Confidence Example

1. Navigate to the "Tossup Agents" tab at the top of the interface.

2. Click the "Select Pipeline to Import..." dropdown and choose "two-step-justified-confidence.yaml".

3. Click "Import Pipeline" to load the example into the interface.

## Understanding the Two-Step Pipeline Structure

The loaded pipeline has two distinct steps:

1. **Step A: Answer Generator**
   - Uses OpenAI/gpt-4o-mini
   - Takes question text as input
   - Generates an answer candidate
   - Uses a focused system prompt for answer generation only

2. **Step B: Confidence Evaluator**
   - Uses Cohere/command-r-plus
   - Takes the question text AND the generated answer from Step A
   - Evaluates confidence and provides justification
   - Uses a specialized system prompt for confidence evaluation

This separation of concerns allows each model to focus on a specific task:
- The first model concentrates solely on generating the most accurate answer
- The second model evaluates how confident we should be in that answer

## Modifying the Pipeline for Better Performance

Here are some ways to enhance the pipeline:

1. **Upgrade the Answer Generator**:
   - Click on Step A in the interface
   - Change the model from gpt-4o-mini to a more powerful model like gpt-4o
   - Modify the system prompt to include more specific instructions about quizbowl answer formatting

2. **Improve the Confidence Evaluator**:
   - Click on Step B
   - Add specific domain knowledge to the system prompt
   - For example, add: "Consider question length when evaluating confidence. Shorter, incomplete questions with less information revealed typically result in lower confidence scores."
   - Change the order of input variables so that model produces justification before confidence score, and hence conditions its confidence score on the justification.

## Running and Testing Your Modified Pipeline

1. After making your modifications, scroll down to adjust the buzzer settings:
   - Consider changing the confidence threshold based on the performance of your enhanced model
   - You might want to lower it slightly if you've improved the confidence evaluator

2. Test your modified pipeline:
   - Select a Question ID or use the provided sample question
   - Click "Run on Tossup Question"
   - Observe the answer, confidence score, and justification

3. Check the "Buzz Confidence" chart to see how confidence evolved during question processing

## Advantages of Multi-Step Pipelines

Multi-step pipelines offer several benefits:

1. **Specialized Models**: Use different models for different tasks (e.g., GPT for general knowledge, Claude for reasoning)

2. **Focused Prompting**: Each step can have a targeted system prompt optimized for its specific task

3. **Chain of Thought**: Build sophisticated reasoning by connecting steps in a logical sequence

4. **Better Confidence Calibration**: Dedicated confidence evaluation typically results in more reliable buzzing

5. **Transparency**: The justification output helps you understand why the model made certain decisions