Skip to content

Commit 6fc4da7

Browse files
huangmingxiahuangmingxia
committed
Migrate the test-generation-assistant into the .ai_testgen folder.
1 parent a175f00 commit 6fc4da7

33 files changed

+3959
-19
lines changed

‎.ai_testgen/CLAUDE.md‎

Lines changed: 168 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,168 @@
1+
# CLAUDE.md
2+
3+
## MANDATORY Protocol
4+
5+
**CRITICAL: Each user request must be treated as a fresh execution: always re-read the YAML config and execute every step in full, regardless of conversation context.**
6+
**CRITICAL: NEVER use Task tool for agents**
7+
8+
Before executing ANY user request, Claude MUST:
9+
1. Read prompts/STARTUP_CHECKLIST.md
10+
2. Identify the correct agent YAML config file
11+
3. Read the agent YAML config directly using Read tool
12+
4. Execute EACH step in the agent configuration manually using available tools
13+
5. Verify each step's output before proceeding to next step
14+
15+
**MANDATORY** = Required, cannot be skipped
16+
17+
---
18+
19+
## Fresh Start Enforcement
20+
21+
**CRITICAL RULE: Every user request is a fresh execution, even if it happens in the SAME conversation.**
22+
23+
- MUST always re-read the correct agent YAML config before executing any agent.
24+
- MUST NOT reuse execution context or skip steps based on conversation history.
25+
- MUST verify that the "Read agent YAML" step has been executed before proceeding.
26+
- If the YAML has not been read in the current request → STOP and re-execute mandatory steps.
27+
28+
### Implementation Notes
29+
- Introduce a request-level hook: each user request = enforce mandatory startup protocol.
30+
- NEVER rely on Task Tool (causes step skipping and incomplete execution).
31+
- Log confirmation before agent execution:
32+
`Agent config loaded: config/agents/<agent>.yaml`
33+
34+
---
35+
36+
## Why NOT to use Task Tool
37+
- Task tool uses general-purpose subagent that doesn't follow YAML configs exactly
38+
- Steps get skipped, simplified, or executed incorrectly
39+
- Results in incomplete or wrong outputs
40+
- Agent configurations are designed for direct execution only
41+
42+
## Supported Agents
43+
44+
### Agent Execution Order
45+
The agents can be executed independently or in sequence:
46+
47+
**Complete End-to-End Flow:**
48+
1. Generate Test Cases → 2. Generate E2E Test Code → 2.1 Execute E2E Tests → 3. Submit E2E Pull Request → 4. Update JIRA QE Comment
49+
50+
**CRITICAL: When user requests complete test generation and execution tests:**
51+
- MUST execute test_case_generation → e2e_test_generation_openshift_private → test-executor in sequence
52+
- Each agent uses its own configuration
53+
- Each agent executes ALL steps defined in its config
54+
- NO shortcuts or step skipping allowed
55+
56+
**Dependencies:**
57+
- e2e_test_generation_openshift_private requires completion of test_case_generation (test case must exist before E2E generation)
58+
- test-executor requires completion of e2e_test_generation_openshift_private
59+
- PR submission requires completion of e2e_test_generation_openshift_private (and optionally test-executor)
60+
- JIRA QE comment update can be executed after any agent completion
61+
62+
### 1. Generate Test Cases
63+
- **Input**: Create test case for JIRA issue key (e.g., "Create test case for HIVE-2883")
64+
- **Action**: NEVER use Task tool
65+
- **PREREQUISITE CHECK**: Check if test case already exists:
66+
1. Check if test case file exists in `test_artifacts/{COMPONENT}/{JIRA_KEY}/test_cases/{JIRA_KEY}_test_case.md`
67+
2. If test case exists: Report existing test case location and skip generation
68+
3. If test case does not exist: Proceed with test case generation
69+
- **Execution Steps:**
70+
1. Read `config/agents/test_case_generation.yaml`
71+
2. Execute each step in `task.steps` section manually
72+
3. Use tools: jira-mcp-snowflake MCP, WebFetch, Read, Write
73+
4. Verify outputs match `output.files` specifications
74+
- **Agent config**: `config/agents/test_case_generation.yaml`
75+
- **Output**: Test cases in markdown format for Polarion integration
76+
77+
### 2. Generate E2E Test Code
78+
- **Input**: Request to generate E2E code for JIRA issue key (e.g., "generate E2E test case for HIVE-2883")
79+
- **Action**: NEVER use Task tool
80+
- **PREREQUISITE CHECK**: Before generating E2E test code, MUST verify test case exists:
81+
1. Only check if test case file exists in `test_artifacts/{COMPONENT}/{JIRA_KEY}/test_cases/{JIRA_KEY}_test_case.md`
82+
2. If test case does NOT exist, FIRST execute test_case_generation agent
83+
3. Only proceed with E2E generation after test case is confirmed to exist
84+
- **Execution Steps:**
85+
1. Read `config/agents/e2e_test_generation_openshift_private.yaml`
86+
2. Read and analyze the generated test case from `test_artifacts/{COMPONENT}/{JIRA_KEY}/test_cases/{JIRA_KEY}_test_case.md`
87+
3. Execute each subtask in `task.subtasks` section sequentially
88+
4. Each subtask has its own steps and success_criteria
89+
5. Use tools: Bash, Read, Write, Edit, WebFetch
90+
6. Verify each subtask's success_criteria before proceeding
91+
- **Agent config**: `config/agents/e2e_test_generation_openshift_private.yaml`
92+
- **Output**: E2E test code integrated into openshift-tests-private repository
93+
94+
### 2.1 Execute E2E Tests
95+
- **Input**: Run the generated E2E test cases and capture their outcomes
96+
- **Action**: NEVER use Task tool
97+
- **Execution Steps:**
98+
1. Read `config/agents/test-executor.yaml`
99+
2. Execute each subtask in `task.subtasks` section sequentially
100+
3. Each subtask has its own steps and success_criteria
101+
4. Use tools: Bash, Read, Write, Edit, WebFetch
102+
5. Capture test results and logs
103+
6. **Generate comprehensive test report** including test case coverage analysis and E2E execution results after successful test completion
104+
- **Agent config**: `config/agents/test-executor.yaml`
105+
- **Output**: Test execution results and comprehensive test reports
106+
107+
### 3. Submit E2E Pull Request
108+
- **Input**: Request to create PR for generated E2E tests
109+
- **Action**: NEVER use Task tool
110+
- **Execution Steps:**
111+
1. Read `config/agents/e2e_test_generation_openshift_private.yaml` to identify where E2E test code was generated
112+
2. Navigate to the E2E test code location and verify test files exist
113+
3. Load and apply PR submission rules from `config/rules/pr_submission_rules.yaml`
114+
4. Use GitHub CLI (gh) to create pull request with hold status
115+
- **Tools**: Bash (git, gh CLI), Read, Write
116+
- **Output**: Created pull request with E2E test code and JIRA updated
117+
118+
### 4. Update JIRA QE Comment [In Progress]
119+
- **Input**: Request to add QE comment to update QE test status
120+
121+
## Agent Usage Examples
122+
123+
**CRITICAL: When a user requests to "generate test cases and run them", Claude MUST execute test_case_generation → e2e_test_generation_openshift_private → test-executor sequentially, performing all steps and verifying outputs at each stage, even if this request occurs in the same conversation. DO NOT STOP ANY STEPS.**
124+
125+
### MANDATORY STEP-BY-STEP EXECUTION PROTOCOL
126+
127+
**BEFORE STARTING ANY AGENT:**
128+
1. **Read agent YAML config completely** - Use Read tool to load entire config file
129+
2. **Execute steps in EXACT sequence** - Must complete step N before starting step N+1
130+
3. **Verify each step's output** - Confirm step completion before proceeding to next step
131+
4. **Use EXACT commands from config** - Copy commands exactly as written in agent YAML
132+
5. **NEVER skip, simplify, or combine steps** - Each step must be executed individually
133+
134+
**VIOLATION PREVENTION RULES:**
135+
- If ANY step is skipped → STOP immediately and restart from skipped step
136+
- Each subtask completion MUST verify ALL steps executed before proceeding to next subtask
137+
- If using different commands than specified in config → VIOLATION, must use exact config commands
138+
- If combining multiple steps into one action → VIOLATION, must execute separately
139+
140+
141+
### Example: Complete Flow for HIVE-2883
142+
```
143+
User: "Create test case for HIVE-2883"
144+
→ Execute test_case_generation agent
145+
146+
User: "Generate E2E code for HIVE-2883"
147+
→ Execute e2e_test_generation_openshift_private agent
148+
149+
User: "Run the E2E tests for HIVE-2883"
150+
→ Execute test-executor agent
151+
152+
User: "Generate test cases and run them for HIVE-2883"
153+
→ Execute test_case_generation → e2e_test_generation_openshift_private → test-executor
154+
155+
User: "Create a Pull Request for HIVE-2883 E2E tests"
156+
→ Execute PR submission process
157+
158+
User: "Add QE comment to HIVE-2883"
159+
→ Execute JIRA QE comment update process
160+
```
161+
162+
163+
## Key Rules
164+
- Always read agent YAML configs directly
165+
- Execute each step manually using available tools
166+
- Verify each step before proceeding
167+
- Follow agent configurations exactly
168+

‎.ai_testgen/GEMINI.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
#

‎.ai_testgen/README.md‎

Lines changed: 184 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -1,25 +1,190 @@
1-
# AI-Assisted Test Generation
1+
# 🧪 Test Generation Assistant
22

3-
This proposal aims to integrate an AI-assisted test generation system into the Hive upstream repository.
4-
To achieve this, we suggest creating a dedicated directory `.ai_testgen` within the repository to store prompt files and related resources.
5-
This ensures that contributors who already have the Hive repository can directly leverage these assets without the need to clone or maintain a separate repository.
3+
## 🎯 Overview
64

7-
## Two possible implementation options are described below:
5+
AI-assisted test generation and execution system that automates the complete testing workflow:
86

9-
**1. Clone external repository**
10-
- Migrate the contents of `[email protected]:huangmingxia/test-generation-assistant.git` into the `.ai_testgen` folder.
11-
```bash
12-
git clone [email protected]:huangmingxia/test-generation-assistant.git
13-
```
14-
- Populate `.ai_testgen/config/rules` with upstream-specific rules to be used for generating test cases and end-to-end (e2e) tests.
7+
- **Test Case Generation**: Generate comprehensive test cases from JIRA tickets
8+
- **E2E Test Code Generation**: Create executable E2E test code
9+
- **Test Execution**: Run E2E tests and capture results
10+
- **PR Submission**: Automatically create pull requests for generated test code
11+
- **JIRA Integration**: Update JIRA tickets with test status and results
1512

13+
This system uses multiple specialized agents to handle each phase of the testing process, ensuring consistent and thorough test coverage for OpenShift components.
1614

17-
**2. Rules-only integration**
18-
- Only add the `rules/` directory inside `.ai_testgen/rules`.
19-
- Configure test case or e2e generation to load these rules by default.
20-
**Limitation**:
21-
- For pre-merge testing, `jira-mcp-snowflake` currently lacks support for extracting associated PR links from Jira issues.
22-
- Additionally, DeepWiki is updated on a weekly basis, which means that code changes within this interval cannot be retrieved.
15+
## 📋 Prerequisites
2316

24-
### Recommended approach [TBD]
25-
Currently recommend approach 1 (cloning the external repository), as it ensures all code changes, including pre-merge changes, are available during test generation.
17+
Before you can use this test generation system, you need to set up the following integrations:
18+
19+
### 1. Install Claude Code
20+
- **Claude Code access**: Follow [Claude Code Install Instructions](https://docs.google.com/document/d/1eNARy9CI28o09E7Foq01e5WD5MvEj3LSBnXqFcprxjo/edit?usp=drivesdk)
21+
22+
### 2. JIRA MCP Access
23+
- **Apply for jira-mcp-snowflake token**: Refer to [Jira-MCP-Snowflake-Token-Guide](https://docs.google.com/document/d/1pg6TkwezhIahppp5k0md0Zx0CC-4f_RWQHaH9cTl1Mo/edit?tab=t.0#heading=h.xyjdx8nsdjql)
24+
25+
- **Config jira-mcp-snowflake mcp server**:
26+
```bash
27+
claude mcp add jira-mcp-snowflake https://jira-mcp-snowflake.mcp-playground-poc.devshift.net/sse --transport sse -H "X-Snowflake-Token: your_token_here"
28+
```
29+
### 3. DeepWiki MCP Connection
30+
```bash
31+
claude mcp add -s user -t http deepwiki https://mcp.deepwiki.com/mcp
32+
```
33+
### 4. Prepare Test Generation Rules
34+
- **Purpose**: Configure rules to guide AI in generating realistic and executable test cases
35+
- **Location**: `config/rules/`
36+
- **Content**: Component-specific testing guidelines, validation criteria, and best practices
37+
38+
### 5. Configure GitHub CLI (gh)
39+
- **Purpose**: Enable PR creation and GitHub repository operations
40+
- **Installation**:
41+
```bash
42+
# macOS
43+
brew install gh
44+
45+
# Linux/Windows
46+
# Follow: https://cli.github.com/manual/installation
47+
```
48+
49+
### 6. Configure E2E Repository Fork
50+
**Purpose**: E2E test generation requires write access to openshift-tests-private repository
51+
**Requirement**: Update the repository URL in the agent configuration to use your own fork
52+
53+
**Steps to configure:**
54+
```
55+
Edit `config/agents/e2e_test_generation_openshift_private.yaml`:
56+
"If not exists: Clone https://github.com/YOUR_USERNAME/openshift-tests-private.git to temp_repos/openshift-tests-private/"
57+
```
58+
**Verify access**: Ensure you have write permissions to your fork for PR creation
59+
60+
### 6. Execution Method
61+
62+
**CRITICAL**: All agents must be executed by reading agent YAML configurations directly.
63+
64+
**Execution Pattern**:
65+
1. Read `CLAUDE.md` for agent instructions
66+
2. Read specific agent YAML config (e.g., `config/agents/test_case_generation.yaml`)
67+
3. Execute each step manually using available tools
68+
4. Verify each step before proceeding
69+
70+
## 🚀 Agents
71+
72+
### 1. Generate Test Cases
73+
**Purpose**: Generate manual test cases from JIRA tickets
74+
**Agent Config**: `config/agents/test_case_generation.yaml`
75+
**Input**: JIRA issue key (e.g., "HIVE-2883")
76+
**Output**:
77+
- `test_requirements_output.yaml` - Detailed requirements analysis
78+
- `test_strategy.yaml` - Test strategy based on requirements
79+
- `{JIRA_KEY}_test_case.md` - Executable test cases in markdown format
80+
81+
**Example Usage**:
82+
```
83+
"Create test case for HIVE-2883"
84+
"Generate test case for JIRA issue HIVE-2883"
85+
"Generate test cases and run them for HIVE-2883" # Full flow
86+
```
87+
88+
**Features**:
89+
- Systematic thinking framework (4 mandatory phases)
90+
- JIRA data extraction and analysis
91+
- Architecture analysis with DeepWiki integration
92+
- User-reality driven test scenarios
93+
- Quantitative validation methods
94+
95+
### 2. Generate E2E Test Code
96+
**Purpose**: Generate E2E test code based on existing test cases
97+
**Agent Config**: `config/agents/e2e_test_generation_openshift_private.yaml`
98+
**Output**: E2E test code integrated into openshift-tests-private repository
99+
100+
### 2.1 Execute E2E Tests
101+
**Purpose**: Execute generated E2E tests and capture results
102+
**Agent Config**: `config/agents/test-executor.yaml`
103+
**Output**: Test execution results and reports
104+
105+
### 3. Submit E2E Pull Request
106+
**Purpose**: Create PR for generated E2E tests
107+
**Tools**: Git, GitHub CLI (gh)
108+
**Output**: Created pull request with E2E test code
109+
110+
### 4. Update JIRA QE Comment [In Progress]
111+
112+
113+
### Agent Dependencies
114+
- test-executor requires completion of e2e_test_generation_openshift_private
115+
- PR submission requires completion of e2e_test_generation_openshift_private (and optionally test-executor)
116+
- JIRA QE comment update can be executed after any agent completion
117+
118+
## 🚀 Claude Code Usage
119+
120+
### Quick Start
121+
122+
1. **Setup Prerequisites** (see above sections):
123+
- Install Claude Code
124+
- Configure JIRA MCP access
125+
- Configure DeepWiki MCP
126+
- Configure GitHub CLI (gh)
127+
128+
2. **Open Claude Code**:
129+
```bash
130+
claude
131+
```
132+
133+
3. **Navigate to project directory**:
134+
```bash
135+
cd /path/to/.ai_testgen
136+
```
137+
138+
4. **Execute agents directly**:
139+
```
140+
# Generate test cases
141+
"Create test case for HIVE-2883"
142+
143+
# Generate E2E code
144+
"Generate E2E case for HIVE-2883"
145+
146+
# Execute E2E tests
147+
"Run E2E tests for HIVE-2883"
148+
149+
# Complete end-to-end flow (executes all 3 agents in sequence)
150+
"Generate test cases and run them for HIVE-2883"
151+
# This automatically executes: test_case_generation → e2e_test_generation_openshift_private → test-executor
152+
153+
# Create PR
154+
"Create PR for HIVE-2883 E2E tests"
155+
156+
# Add QE comment
157+
"Add QE comment to HIVE-2883"
158+
```
159+
160+
### How It Works
161+
162+
1. **Claude reads CLAUDE.md** - Gets agent instructions
163+
2. **Reads agent YAML configs** - Gets specific execution steps
164+
3. **Executes steps** - Uses available tools (JIRA MCP, WebFetch, Bash, etc.)
165+
4. **Verifies outputs** - Ensures each step completes successfully
166+
5. **Generates artifacts** - Creates test cases, E2E code, PRs, etc.
167+
168+
### No API Server Needed
169+
- Everything runs through Claude Code directly
170+
- No need for separate microservice or Docker containers
171+
- All agents executed via natural language commands
172+
- Real-time interaction and feedback
173+
174+
## 🔗 Integrations
175+
176+
### JIRA Integration
177+
- **MCP Endpoint**: `jira-mcp-snowflake`
178+
- **Purpose**: Extract ticket requirements and details
179+
180+
### DeepWiki Integration
181+
- **MCP Endpoint**: `DeepWiki MCP`
182+
- **Purpose**: Code change analysis and testing implications
183+
184+
### GitHub Integration
185+
- **Repository Management**: Automatic branch creation
186+
- **File Operations**: Direct file creation in E2E repositories
187+
188+
## 📚 Documentation
189+
190+
- **[CLAUDE.md](CLAUDE.md)** - Complete agent instructions and execution guide

0 commit comments

Comments
 (0)