[P1] Discovery questions are static instead of Socratic/dynamic

[frankbria]

Problem

Discovery questions are being generated as static templates rather than being dynamically generated by an AI agent that reacts to and learns from previous answers. The questions should follow a Socratic methodology where each question builds on the context of the previous answer.

Current Behavior

Discovery questions appear to be pre-defined templates that don't change based on user input:

• Q1: "Who are the primary users..." (problem)
• Q2: "Who are the primary users..." (users) - redundant repeat of Q1
• Q3: "What are the core features..." (features)
• Q4: "Are there any technical constraints..." (constraints)
• Q5: "Do you have a preferred tech stack..." (tech_stack)

Each time a user starts discovery, they get the same set of questions regardless of their specific answers.

Expected Behavior

Discovery should be driven by an AI agent that:

1. Reads and understands each user answer
2. Generates follow-up questions based on the specific context of previous answers
3. Adapts the question flow - different projects get different questions
4. Builds on answers - Socratic method where each question probes deeper into relevant areas

Example of expected dynamic flow:

• User answers: "Building a weather app for sailors"
• AI asks: "What offshore weather data sources do you need access to?" or "Should the app include maritime weather alerts?"

Impact

• Users feel the discovery process is generic/unintelligent
• PRDs may miss critical project-specific requirements
• Reduces trust in AI capabilities of the platform
• Not a blocker, but degrades user experience significantly

Suggested Approach

1. Use the LeadAgent or similar AI agent to dynamically generate each next question
2. Pass conversation history to the agent for context
3. Allow the agent to determine when sufficient information has been gathered
4. Implement fallback to static questions if AI generation fails

Acceptance Criteria

• Discovery questions vary based on user answers
• AI agent reads previous answers before generating next question
• Questions build on context (Socratic method)
• Different projects produce different question flows
• Error handling falls back to reasonable defaults

Priority

P1 - This should be fixed before beta for best experience. It's a core part of the "AI-powered" value proposition.

Environment

• Repo: frankbria/codeframe
• Test URL: https://dev.codeframeapp.com
• Tested Project: test-weather-app

---

Reported by: Rocket 🦊 via workflow testing

[traycerai]

Plan

Observations

The discovery system currently uses a hybrid approach: AI-generated first question followed by static questions from DiscoveryQuestionFramework. The static questions are hardcoded in file:codeframe/discovery/questions.py with fixed text and categories. The LeadAgent already has AI capabilities via the Anthropic provider and conversation history management. The issue reports redundant questions (Q1 and Q2 both asking about users) and lack of context-awareness in follow-up questions. The architecture already supports AI-powered interactions through LeadAgent.chat() and provider.send_message(), making this enhancement feasible without major refactoring.

Approach

Transform the discovery flow from static template-based to fully AI-driven Socratic questioning while maintaining the existing category framework as validation guardrails. Modify LeadAgent.process_discovery_answer() to generate each subsequent question dynamically using Claude API with full conversation history context. Keep DiscoveryQuestionFramework as a fallback mechanism and category tracker to ensure all required topics are covered. This approach preserves the existing completion criteria and database schema while enabling adaptive, context-aware questioning that builds on previous answers.

Implementation Steps

1. Enhance LeadAgent Discovery State Management

File: file:codeframe/agents/lead_agent.py

Add conversation history tracking for discovery:

• Add _discovery_conversation_history list attribute to LeadAgent.__init__()
• Store each question-answer pair as conversation turns
• Load/save conversation history in _load_discovery_state() and _save_discovery_state()
• Use database memory category discovery_conversation to persist history
• Track which framework categories have been covered to ensure completeness

2. Create AI-Powered Question Generation Method

File: file:codeframe/agents/lead_agent.py

Add new method _generate_next_discovery_question():

• Accept parameters: previous answers dict, conversation history, uncovered categories
• Build prompt that includes:
  • All previous Q&A pairs for context
  • List of categories still needing coverage (problem, users, features, constraints, tech_stack)
  • Instruction to ask ONE focused Socratic follow-up question
  • Guidance to probe deeper into interesting areas or cover missing categories
• Call self.provider.send_message() with conversation context
• Return generated question text
• Log token usage for monitoring
• Include error handling with fallback to framework questions

3. Modify process_discovery_answer() for Dynamic Questions

File: file:codeframe/agents/lead_agent.py

Refactor process_discovery_answer() method:

• After saving answer, append Q&A pair to _discovery_conversation_history
• Determine which framework categories have been adequately covered
• Check if discovery is complete using discovery_framework.is_discovery_complete()
• If not complete, call _generate_next_discovery_question() with full context
• Store generated question in _current_question_text and use special ID like ai_generated_N
• Map AI-generated answers to appropriate framework question IDs for completion tracking
• Persist conversation history to database after each turn
• On AI generation failure, fall back to discovery_framework.get_next_question()

4. Update Discovery Prompt Engineering

File: file:codeframe/agents/lead_agent.py

Create _build_discovery_question_prompt() method:

• Format conversation history as numbered Q&A pairs
• List categories covered vs. remaining
• Include Socratic questioning guidelines:
  • Build on previous answers
  • Ask clarifying questions when answers are vague
  • Probe deeper into interesting technical details
  • Connect related concepts across categories
  • Keep questions conversational and focused
• Add examples of good Socratic follow-ups
• Specify output format: single question, no preamble

5. Enhance Category Coverage Tracking

File: file:codeframe/agents/lead_agent.py

Add method _get_category_coverage():

• Analyze conversation history to determine which categories have been addressed
• Use simple keyword matching or AI classification
• Return dict mapping categories to coverage status (covered/partial/uncovered)
• Use this to guide next question generation
• Ensure all required categories get at least one substantive answer
• Store coverage status in discovery state for persistence

6. Update Completion Logic

File: file:codeframe/agents/lead_agent.py

Modify completion detection in process_discovery_answer():

• Check both framework completion AND category coverage
• Require minimum number of questions answered (e.g., 5-10)
• Validate that all required categories have substantive answers
• Allow AI to determine when sufficient information is gathered
• Add optional "Are we ready to generate the PRD?" confirmation question
• Transition to completed state only when both criteria met

7. Add Fallback Mechanisms

File: file:codeframe/agents/lead_agent.py

Implement robust error handling:

• If AI question generation fails, use discovery_framework.get_next_question()
• If AI is unavailable, fall back to static questions entirely
• Log all fallback events for monitoring
• Add retry logic with exponential backoff for transient API errors
• Ensure discovery can always progress even with AI failures
• Store fallback usage metrics in database

8. Update Tests for Dynamic Questions

File: file:tests/discovery/test_discovery_questions.py

Modify existing tests:

• Update tests to expect AI-generated questions instead of static ones
• Mock provider.send_message() to return test questions
• Test conversation history building
• Test category coverage tracking
• Test fallback to static questions on AI failure
• Add integration tests for full Socratic discovery flow
• Verify completion logic with dynamic questions

File: file:tests/agents/test_lead_agent.py

Add new test cases:

• Test _generate_next_discovery_question() with various contexts
• Test conversation history persistence
• Test category coverage detection
• Test Socratic follow-up logic (questions build on answers)
• Test different project types get different question flows
• Mock Anthropic API responses for deterministic testing

9. Update Discovery Router Documentation

File: file:codeframe/ui/routers/discovery.py

Add comments explaining:

• Discovery now uses AI-generated questions
• Static framework serves as fallback and validation
• Conversation history maintained for context
• No API contract changes needed
• Existing endpoints work unchanged

10. Add Configuration Options

File: file:codeframe/agents/lead_agent.py

Add configuration for discovery behavior:

• Environment variable DISCOVERY_MODE (dynamic/static/hybrid)
• Minimum questions threshold (default: 5)
• Maximum questions threshold (default: 20)
• AI model selection for discovery (default: claude-3-5-sonnet)
• Fallback behavior configuration
• Token budget limits for discovery phase

11. Implement Monitoring and Logging

File: file:codeframe/agents/lead_agent.py

Add comprehensive logging:

• Log each AI-generated question with token usage
• Log category coverage progression
• Log fallback events and reasons
• Log discovery completion metrics
• Track average questions per project
• Monitor AI generation latency
• Store metrics in database for analytics

12. Update Documentation

File: file:codeframe/discovery/questions.py

Update module docstring:

• Explain that framework now serves as fallback/validation
• Document category coverage requirements
• Note that AI generates questions dynamically
• Preserve existing static questions for reference

File: Create file:docs/discovery-socratic-methodology.md

Document the Socratic discovery approach:

• How AI generates context-aware questions
• Category coverage requirements
• Fallback mechanisms
• Example conversation flows
• Best practices for prompt engineering
• Troubleshooting guide

Architecture Diagram

sequenceDiagram
    participant User
    participant API as Discovery API
    participant LA as LeadAgent
    participant AI as Claude API
    participant FW as DiscoveryFramework
    participant DB as Database

    User->>API: POST /discovery/answer
    API->>LA: process_discovery_answer(answer)
    LA->>DB: Save answer
    LA->>LA: Append to conversation_history
    LA->>LA: Update category_coverage
    LA->>FW: is_discovery_complete()?
    
    alt Discovery Complete
        FW-->>LA: True
        LA->>DB: Set state=completed
        LA-->>API: "Discovery complete!"
        API-->>User: is_complete=true
    else More Questions Needed
        FW-->>LA: False
        LA->>LA: Build Socratic prompt with history
        LA->>AI: Generate next question
        
        alt AI Success
            AI-->>LA: Context-aware question
            LA->>DB: Save question + history
            LA-->>API: Next question
            API-->>User: Display question
        else AI Failure
            AI-->>LA: Error
            LA->>FW: get_next_question() [fallback]
            FW-->>LA: Static question
            LA->>DB: Save question + log fallback
            LA-->>API: Next question
            API-->>User: Display question
        end
    end

Loading

Testing Strategy

Unit Tests

• Test _generate_next_discovery_question() with mocked AI responses
• Test conversation history building and persistence
• Test category coverage tracking logic
• Test completion detection with dynamic questions
• Test fallback mechanisms trigger correctly
• Test prompt building with various conversation states

Integration Tests

• Test full discovery flow with AI-generated questions
• Test that different project descriptions lead to different questions
• Test that questions build on previous answers (Socratic method)
• Test graceful degradation when AI unavailable
• Test discovery completion with minimum required coverage

Manual Testing

• Create project with description "weather app for sailors"
• Verify first question is relevant to maritime/weather domain
• Answer with specific details (e.g., "offshore weather alerts")
• Verify next question probes deeper into that specific area
• Complete full discovery and verify PRD captures context
• Test with different project types to verify question variation

Rollout Plan

Phase 1: Foundation (2-3 hours)

• Add conversation history tracking to LeadAgent
• Implement _generate_next_discovery_question() method
• Add category coverage tracking
• Write unit tests for new methods

Phase 2: Integration (2-3 hours)

• Modify process_discovery_answer() to use AI generation
• Implement fallback mechanisms
• Update completion logic
• Add comprehensive error handling

Phase 3: Testing & Refinement (2-3 hours)

• Write integration tests
• Perform manual testing with various project types
• Refine prompts based on question quality
• Tune category coverage detection
• Add monitoring and logging

Phase 4: Documentation (1 hour)

• Update code documentation
• Create discovery methodology guide
• Document configuration options
• Add troubleshooting guide

Risk Mitigation

Risk: AI-generated questions may be too generic or off-topic

• Mitigation: Strong prompt engineering with category guidance, fallback to static questions, manual testing with diverse project types

Risk: Increased API costs from multiple AI calls per discovery session

• Mitigation: Token budget limits, efficient prompt design, caching of common patterns, monitoring and alerting

Risk: Discovery may take longer with AI generation latency

• Mitigation: Async processing, progress indicators, timeout handling, fallback to static questions on timeout

Risk: Breaking existing discovery flows or tests

• Mitigation: Maintain backward compatibility, comprehensive test coverage, feature flag for gradual rollout, preserve static framework as fallback

Import In IDE

🤖 Prompt for AI Agents

## Observations

The discovery system currently uses a hybrid approach: AI-generated first question followed by static questions from `DiscoveryQuestionFramework`. The static questions are hardcoded in `file:codeframe/discovery/questions.py` with fixed text and categories. The `LeadAgent` already has AI capabilities via the Anthropic provider and conversation history management. The issue reports redundant questions (Q1 and Q2 both asking about users) and lack of context-awareness in follow-up questions. The architecture already supports AI-powered interactions through `LeadAgent.chat()` and `provider.send_message()`, making this enhancement feasible without major refactoring.

## Approach

Transform the discovery flow from static template-based to fully AI-driven Socratic questioning while maintaining the existing category framework as validation guardrails. Modify `LeadAgent.process_discovery_answer()` to generate each subsequent question dynamically using Claude API with full conversation history context. Keep `DiscoveryQuestionFramework` as a fallback mechanism and category tracker to ensure all required topics are covered. This approach preserves the existing completion criteria and database schema while enabling adaptive, context-aware questioning that builds on previous answers.

## Implementation Steps

### 1. Enhance LeadAgent Discovery State Management

**File:** `file:codeframe/agents/lead_agent.py`

Add conversation history tracking for discovery:
- Add `_discovery_conversation_history` list attribute to `LeadAgent.__init__()`
- Store each question-answer pair as conversation turns
- Load/save conversation history in `_load_discovery_state()` and `_save_discovery_state()`
- Use database memory category `discovery_conversation` to persist history
- Track which framework categories have been covered to ensure completeness

### 2. Create AI-Powered Question Generation Method

**File:** `file:codeframe/agents/lead_agent.py`

Add new method `_generate_next_discovery_question()`:
- Accept parameters: previous answers dict, conversation history, uncovered categories
- Build prompt that includes:
  - All previous Q&A pairs for context
  - List of categories still needing coverage (problem, users, features, constraints, tech_stack)
  - Instruction to ask ONE focused Socratic follow-up question
  - Guidance to probe deeper into interesting areas or cover missing categories
- Call `self.provider.send_message()` with conversation context
- Return generated question text
- Log token usage for monitoring
- Include error handling with fallback to framework questions

### 3. Modify process_discovery_answer() for Dynamic Questions

**File:** `file:codeframe/agents/lead_agent.py`

Refactor `process_discovery_answer()` method:
- After saving answer, append Q&A pair to `_discovery_conversation_history`
- Determine which framework categories have been adequately covered
- Check if discovery is complete using `discovery_framework.is_discovery_complete()`
- If not complete, call `_generate_next_discovery_question()` with full context
- Store generated question in `_current_question_text` and use special ID like `ai_generated_N`
- Map AI-generated answers to appropriate framework question IDs for completion tracking
- Persist conversation history to database after each turn
- On AI generation failure, fall back to `discovery_framework.get_next_question()`

### 4. Update Discovery Prompt Engineering

**File:** `file:codeframe/agents/lead_agent.py`

Create `_build_discovery_question_prompt()` method:
- Format conversation history as numbered Q&A pairs
- List categories covered vs. remaining
- Include Socratic questioning guidelines:
  - Build on previous answers
  - Ask clarifying questions when answers are vague
  - Probe deeper into interesting technical details
  - Connect related concepts across categories
  - Keep questions conversational and focused
- Add examples of good Socratic follow-ups
- Specify output format: single question, no preamble

### 5. Enhance Category Coverage Tracking

**File:** `file:codeframe/agents/lead_agent.py`

Add method `_get_category_coverage()`:
- Analyze conversation history to determine which categories have been addressed
- Use simple keyword matching or AI classification
- Return dict mapping categories to coverage status (covered/partial/uncovered)
- Use this to guide next question generation
- Ensure all required categories get at least one substantive answer
- Store coverage status in discovery state for persistence

### 6. Update Completion Logic

**File:** `file:codeframe/agents/lead_agent.py`

Modify completion detection in `process_discovery_answer()`:
- Check both framework completion AND category coverage
- Require minimum number of questions answered (e.g., 5-10)
- Validate that all required categories have substantive answers
- Allow AI to determine when sufficient information is gathered
- Add optional "Are we ready to generate the PRD?" confirmation question
- Transition to completed state only when both criteria met

### 7. Add Fallback Mechanisms

**File:** `file:codeframe/agents/lead_agent.py`

Implement robust error handling:
- If AI question generation fails, use `discovery_framework.get_next_question()`
- If AI is unavailable, fall back to static questions entirely
- Log all fallback events for monitoring
- Add retry logic with exponential backoff for transient API errors
- Ensure discovery can always progress even with AI failures
- Store fallback usage metrics in database

### 8. Update Tests for Dynamic Questions

**File:** `file:tests/discovery/test_discovery_questions.py`

Modify existing tests:
- Update tests to expect AI-generated questions instead of static ones
- Mock `provider.send_message()` to return test questions
- Test conversation history building
- Test category coverage tracking
- Test fallback to static questions on AI failure
- Add integration tests for full Socratic discovery flow
- Verify completion logic with dynamic questions

**File:** `file:tests/agents/test_lead_agent.py`

Add new test cases:
- Test `_generate_next_discovery_question()` with various contexts
- Test conversation history persistence
- Test category coverage detection
- Test Socratic follow-up logic (questions build on answers)
- Test different project types get different question flows
- Mock Anthropic API responses for deterministic testing

### 9. Update Discovery Router Documentation

**File:** `file:codeframe/ui/routers/discovery.py`

Add comments explaining:
- Discovery now uses AI-generated questions
- Static framework serves as fallback and validation
- Conversation history maintained for context
- No API contract changes needed
- Existing endpoints work unchanged

### 10. Add Configuration Options

**File:** `file:codeframe/agents/lead_agent.py`

Add configuration for discovery behavior:
- Environment variable `DISCOVERY_MODE` (dynamic/static/hybrid)
- Minimum questions threshold (default: 5)
- Maximum questions threshold (default: 20)
- AI model selection for discovery (default: claude-3-5-sonnet)
- Fallback behavior configuration
- Token budget limits for discovery phase

### 11. Implement Monitoring and Logging

**File:** `file:codeframe/agents/lead_agent.py`

Add comprehensive logging:
- Log each AI-generated question with token usage
- Log category coverage progression
- Log fallback events and reasons
- Log discovery completion metrics
- Track average questions per project
- Monitor AI generation latency
- Store metrics in database for analytics

### 12. Update Documentation

**File:** `file:codeframe/discovery/questions.py`

Update module docstring:
- Explain that framework now serves as fallback/validation
- Document category coverage requirements
- Note that AI generates questions dynamically
- Preserve existing static questions for reference

**File:** Create `file:docs/discovery-socratic-methodology.md`

Document the Socratic discovery approach:
- How AI generates context-aware questions
- Category coverage requirements
- Fallback mechanisms
- Example conversation flows
- Best practices for prompt engineering
- Troubleshooting guide

## Architecture Diagram

```mermaid
sequenceDiagram
    participant User
    participant API as Discovery API
    participant LA as LeadAgent
    participant AI as Claude API
    participant FW as DiscoveryFramework
    participant DB as Database

    User->>API: POST /discovery/answer
    API->>LA: process_discovery_answer(answer)
    LA->>DB: Save answer
    LA->>LA: Append to conversation_history
    LA->>LA: Update category_coverage
    LA->>FW: is_discovery_complete()?
    
    alt Discovery Complete
        FW-->>LA: True
        LA->>DB: Set state=completed
        LA-->>API: "Discovery complete!"
        API-->>User: is_complete=true
    else More Questions Needed
        FW-->>LA: False
        LA->>LA: Build Socratic prompt with history
        LA->>AI: Generate next question
        
        alt AI Success
            AI-->>LA: Context-aware question
            LA->>DB: Save question + history
            LA-->>API: Next question
            API-->>User: Display question
        else AI Failure
            AI-->>LA: Error
            LA->>FW: get_next_question() [fallback]
            FW-->>LA: Static question
            LA->>DB: Save question + log fallback
            LA-->>API: Next question
            API-->>User: Display question
        end
    end
```

## Testing Strategy

### Unit Tests
- Test `_generate_next_discovery_question()` with mocked AI responses
- Test conversation history building and persistence
- Test category coverage tracking logic
- Test completion detection with dynamic questions
- Test fallback mechanisms trigger correctly
- Test prompt building with various conversation states

### Integration Tests
- Test full discovery flow with AI-generated questions
- Test that different project descriptions lead to different questions
- Test that questions build on previous answers (Socratic method)
- Test graceful degradation when AI unavailable
- Test discovery completion with minimum required coverage

### Manual Testing
- Create project with description "weather app for sailors"
- Verify first question is relevant to maritime/weather domain
- Answer with specific details (e.g., "offshore weather alerts")
- Verify next question probes deeper into that specific area
- Complete full discovery and verify PRD captures context
- Test with different project types to verify question variation

## Rollout Plan

### Phase 1: Foundation (2-3 hours)
- Add conversation history tracking to LeadAgent
- Implement `_generate_next_discovery_question()` method
- Add category coverage tracking
- Write unit tests for new methods

### Phase 2: Integration (2-3 hours)
- Modify `process_discovery_answer()` to use AI generation
- Implement fallback mechanisms
- Update completion logic
- Add comprehensive error handling

### Phase 3: Testing & Refinement (2-3 hours)
- Write integration tests
- Perform manual testing with various project types
- Refine prompts based on question quality
- Tune category coverage detection
- Add monitoring and logging

### Phase 4: Documentation (1 hour)
- Update code documentation
- Create discovery methodology guide
- Document configuration options
- Add troubleshooting guide

## Risk Mitigation

**Risk:** AI-generated questions may be too generic or off-topic
- **Mitigation:** Strong prompt engineering with category guidance, fallback to static questions, manual testing with diverse project types

**Risk:** Increased API costs from multiple AI calls per discovery session
- **Mitigation:** Token budget limits, efficient prompt design, caching of common patterns, monitoring and alerting

**Risk:** Discovery may take longer with AI generation latency
- **Mitigation:** Async processing, progress indicators, timeout handling, fallback to static questions on timeout

**Risk:** Breaking existing discovery flows or tests
- **Mitigation:** Maintain backward compatibility, comprehensive test coverage, feature flag for gradual rollout, preserve static framework as fallback

---

Execution Information

Branch: main
Commit: 0306a4e

---

💡 Tips

Supported Commands (Inside Comments)

• Use @traycerai generate to iterate on the previous version of the implementation plan.

Supported Commands (Inside Description)

• Add @traycerai ignore anywhere in the ticket description to prevent this ticket from being processed.
• Add @traycerai branch:<branch-name> anywhere in the ticket description to specify the target branch for the implementation plan.

Community

• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[coderabbitai]

📝 CodeRabbit Plan Mode

Generate an implementation plan and prompts that you can use with your favorite coding agent.

• Create Plan

Examples

• Example 1
• Example 2

---

🔗 Related PRs

#181 - fix(discovery): Allow starting discovery when project is running but discovery idle [closed]
#191 - feat(e2e): Enable conditional tests and fix discovery startup [merged]
#193 - Fix discovery questions repeating in staging [merged]

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!