add bmad techniques to the project
This commit is contained in:
74
_bmad/tea/workflows/testarch/README.md
Normal file
74
_bmad/tea/workflows/testarch/README.md
Normal file
@@ -0,0 +1,74 @@
|
||||
# TEA Workflow Step Files
|
||||
|
||||
This folder contains the Test Architect (TEA) workflows converted to step-file architecture for strict LLM compliance. Each workflow is tri-modal (create, edit, validate) and uses small, ordered step files instead of a single monolithic instruction file.
|
||||
|
||||
## Why Step Files
|
||||
|
||||
- Enforces sequential execution and prevents improvisation
|
||||
- Keeps context small and focused per step
|
||||
- Makes validation and edits deterministic
|
||||
|
||||
## Standard Layout (per workflow)
|
||||
|
||||
```
|
||||
<workflow>/
|
||||
├── workflow.md # Mode routing (create / edit / validate)
|
||||
├── workflow-plan.md # Design reference for step order and intent
|
||||
├── workflow.yaml # Installer metadata
|
||||
├── instructions.md # Short entrypoint / summary
|
||||
├── checklist.md # Validation criteria for outputs
|
||||
├── steps-c/ # Create mode steps
|
||||
├── steps-e/ # Edit mode steps
|
||||
├── steps-v/ # Validate mode steps
|
||||
├── templates/ # Output templates (if applicable)
|
||||
└── validation-report-*.md # Validator outputs (latest run)
|
||||
```
|
||||
|
||||
## Modes
|
||||
|
||||
- **Create (steps-c/):** Primary execution flow to generate outputs
|
||||
- **Edit (steps-e/):** Structured edits to existing outputs
|
||||
- **Validate (steps-v/):** Checklist-based validation of outputs
|
||||
|
||||
## Execution Rules (Summary)
|
||||
|
||||
- Load **one step at a time**. Do not preload future steps.
|
||||
- Follow the **MANDATORY SEQUENCE** exactly in each step.
|
||||
- Do not skip steps, reorder, or improvise.
|
||||
- If a step writes outputs, do so **before** loading the next step.
|
||||
|
||||
## Step Naming Conventions
|
||||
|
||||
- `step-01-*.md` is the init step (no menus unless explicitly required).
|
||||
- `step-01b-*.md` is a continuation/resume step if the workflow is continuable.
|
||||
- `step-0X-*.md` are sequential create-mode steps.
|
||||
- `steps-v/step-01-validate.md` is the validate mode entrypoint.
|
||||
- `steps-e/step-01-assess.md` is the edit mode entrypoint.
|
||||
|
||||
## Validation
|
||||
|
||||
- Each workflow has a latest `validation-report-*.md` in its folder.
|
||||
- Validation uses the BMad Builder workflow validator (workflow-builder).
|
||||
- The goal is 100% compliance with no warnings.
|
||||
|
||||
## References
|
||||
|
||||
- Step-file architecture: `docs/explanation/step-file-architecture.md`
|
||||
- Subagent patterns: `docs/explanation/subagent-architecture.md`
|
||||
|
||||
## TEA Workflows
|
||||
|
||||
- test-design
|
||||
- automate
|
||||
- atdd
|
||||
- test-review
|
||||
- trace
|
||||
- framework
|
||||
- ci
|
||||
- nfr-assess
|
||||
|
||||
## Notes
|
||||
|
||||
- `workflow.md` is the canonical entrypoint. `instructions.md` is a short summary for quick context.
|
||||
- Output files typically use `{test_artifacts}` or `{project-root}` variables.
|
||||
- If a workflow produces multiple artifacts (e.g., system-level vs epic-level), the step file will specify which templates and output paths to use.
|
||||
371
_bmad/tea/workflows/testarch/atdd/atdd-checklist-template.md
Normal file
371
_bmad/tea/workflows/testarch/atdd/atdd-checklist-template.md
Normal file
@@ -0,0 +1,371 @@
|
||||
---
|
||||
stepsCompleted: []
|
||||
lastStep: ''
|
||||
lastSaved: ''
|
||||
workflowType: 'testarch-atdd'
|
||||
inputDocuments: []
|
||||
---
|
||||
|
||||
# ATDD Checklist - Epic {epic_num}, Story {story_num}: {story_title}
|
||||
|
||||
**Date:** {date}
|
||||
**Author:** {user_name}
|
||||
**Primary Test Level:** {primary_level}
|
||||
|
||||
---
|
||||
|
||||
## Story Summary
|
||||
|
||||
{Brief 2-3 sentence summary of the user story}
|
||||
|
||||
**As a** {user_role}
|
||||
**I want** {feature_description}
|
||||
**So that** {business_value}
|
||||
|
||||
---
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
{List all testable acceptance criteria from the story}
|
||||
|
||||
1. {Acceptance criterion 1}
|
||||
2. {Acceptance criterion 2}
|
||||
3. {Acceptance criterion 3}
|
||||
|
||||
---
|
||||
|
||||
## Failing Tests Created (RED Phase)
|
||||
|
||||
### E2E Tests ({e2e_test_count} tests)
|
||||
|
||||
**File:** `{e2e_test_file_path}` ({line_count} lines)
|
||||
|
||||
{List each E2E test with its current status and expected failure reason}
|
||||
|
||||
- ✅ **Test:** {test_name}
|
||||
- **Status:** RED - {failure_reason}
|
||||
- **Verifies:** {what_this_test_validates}
|
||||
|
||||
### API Tests ({api_test_count} tests)
|
||||
|
||||
**File:** `{api_test_file_path}` ({line_count} lines)
|
||||
|
||||
{List each API test with its current status and expected failure reason}
|
||||
|
||||
- ✅ **Test:** {test_name}
|
||||
- **Status:** RED - {failure_reason}
|
||||
- **Verifies:** {what_this_test_validates}
|
||||
|
||||
### Component Tests ({component_test_count} tests)
|
||||
|
||||
**File:** `{component_test_file_path}` ({line_count} lines)
|
||||
|
||||
{List each component test with its current status and expected failure reason}
|
||||
|
||||
- ✅ **Test:** {test_name}
|
||||
- **Status:** RED - {failure_reason}
|
||||
- **Verifies:** {what_this_test_validates}
|
||||
|
||||
---
|
||||
|
||||
## Data Factories Created
|
||||
|
||||
{List all data factory files created with their exports}
|
||||
|
||||
### {Entity} Factory
|
||||
|
||||
**File:** `tests/support/factories/{entity}.factory.ts`
|
||||
|
||||
**Exports:**
|
||||
|
||||
- `create{Entity}(overrides?)` - Create single entity with optional overrides
|
||||
- `create{Entity}s(count)` - Create array of entities
|
||||
|
||||
**Example Usage:**
|
||||
|
||||
```typescript
|
||||
const user = createUser({ email: 'specific@example.com' });
|
||||
const users = createUsers(5); // Generate 5 random users
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Fixtures Created
|
||||
|
||||
{List all test fixture files created with their fixture names and descriptions}
|
||||
|
||||
### {Feature} Fixtures
|
||||
|
||||
**File:** `tests/support/fixtures/{feature}.fixture.ts`
|
||||
|
||||
**Fixtures:**
|
||||
|
||||
- `{fixtureName}` - {description_of_what_fixture_provides}
|
||||
- **Setup:** {what_setup_does}
|
||||
- **Provides:** {what_test_receives}
|
||||
- **Cleanup:** {what_cleanup_does}
|
||||
|
||||
**Example Usage:**
|
||||
|
||||
```typescript
|
||||
import { test } from './fixtures/{feature}.fixture';
|
||||
|
||||
test('should do something', async ({ {fixtureName} }) => {
|
||||
// {fixtureName} is ready to use with auto-cleanup
|
||||
});
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Mock Requirements
|
||||
|
||||
{Document external services that need mocking and their requirements}
|
||||
|
||||
### {Service Name} Mock
|
||||
|
||||
**Endpoint:** `{HTTP_METHOD} {endpoint_url}`
|
||||
|
||||
**Success Response:**
|
||||
|
||||
```json
|
||||
{
|
||||
{success_response_example}
|
||||
}
|
||||
```
|
||||
|
||||
**Failure Response:**
|
||||
|
||||
```json
|
||||
{
|
||||
{failure_response_example}
|
||||
}
|
||||
```
|
||||
|
||||
**Notes:** {any_special_mock_requirements}
|
||||
|
||||
---
|
||||
|
||||
## Required data-testid Attributes
|
||||
|
||||
{List all data-testid attributes required in UI implementation for test stability}
|
||||
|
||||
### {Page or Component Name}
|
||||
|
||||
- `{data-testid-name}` - {description_of_element}
|
||||
- `{data-testid-name}` - {description_of_element}
|
||||
|
||||
**Implementation Example:**
|
||||
|
||||
```tsx
|
||||
<button data-testid="login-button">Log In</button>
|
||||
<input data-testid="email-input" type="email" />
|
||||
<div data-testid="error-message">{errorText}</div>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
{Map each failing test to concrete implementation tasks that will make it pass}
|
||||
|
||||
### Test: {test_name_1}
|
||||
|
||||
**File:** `{test_file_path}`
|
||||
|
||||
**Tasks to make this test pass:**
|
||||
|
||||
- [ ] {Implementation task 1}
|
||||
- [ ] {Implementation task 2}
|
||||
- [ ] {Implementation task 3}
|
||||
- [ ] Add required data-testid attributes: {list_of_testids}
|
||||
- [ ] Run test: `{test_execution_command}`
|
||||
- [ ] ✅ Test passes (green phase)
|
||||
|
||||
**Estimated Effort:** {effort_estimate} hours
|
||||
|
||||
---
|
||||
|
||||
### Test: {test_name_2}
|
||||
|
||||
**File:** `{test_file_path}`
|
||||
|
||||
**Tasks to make this test pass:**
|
||||
|
||||
- [ ] {Implementation task 1}
|
||||
- [ ] {Implementation task 2}
|
||||
- [ ] {Implementation task 3}
|
||||
- [ ] Add required data-testid attributes: {list_of_testids}
|
||||
- [ ] Run test: `{test_execution_command}`
|
||||
- [ ] ✅ Test passes (green phase)
|
||||
|
||||
**Estimated Effort:** {effort_estimate} hours
|
||||
|
||||
---
|
||||
|
||||
## Running Tests
|
||||
|
||||
```bash
|
||||
# Run all failing tests for this story
|
||||
{test_command_all}
|
||||
|
||||
# Run specific test file
|
||||
{test_command_specific_file}
|
||||
|
||||
# Run tests in headed mode (see browser)
|
||||
{test_command_headed}
|
||||
|
||||
# Debug specific test
|
||||
{test_command_debug}
|
||||
|
||||
# Run tests with coverage
|
||||
{test_command_coverage}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Red-Green-Refactor Workflow
|
||||
|
||||
### RED Phase (Complete) ✅
|
||||
|
||||
**TEA Agent Responsibilities:**
|
||||
|
||||
- ✅ All tests written and failing
|
||||
- ✅ Fixtures and factories created with auto-cleanup
|
||||
- ✅ Mock requirements documented
|
||||
- ✅ data-testid requirements listed
|
||||
- ✅ Implementation checklist created
|
||||
|
||||
**Verification:**
|
||||
|
||||
- All tests run and fail as expected
|
||||
- Failure messages are clear and actionable
|
||||
- Tests fail due to missing implementation, not test bugs
|
||||
|
||||
---
|
||||
|
||||
### GREEN Phase (DEV Team - Next Steps)
|
||||
|
||||
**DEV Agent Responsibilities:**
|
||||
|
||||
1. **Pick one failing test** from implementation checklist (start with highest priority)
|
||||
2. **Read the test** to understand expected behavior
|
||||
3. **Implement minimal code** to make that specific test pass
|
||||
4. **Run the test** to verify it now passes (green)
|
||||
5. **Check off the task** in implementation checklist
|
||||
6. **Move to next test** and repeat
|
||||
|
||||
**Key Principles:**
|
||||
|
||||
- One test at a time (don't try to fix all at once)
|
||||
- Minimal implementation (don't over-engineer)
|
||||
- Run tests frequently (immediate feedback)
|
||||
- Use implementation checklist as roadmap
|
||||
|
||||
**Progress Tracking:**
|
||||
|
||||
- Check off tasks as you complete them
|
||||
- Share progress in daily standup
|
||||
|
||||
---
|
||||
|
||||
### REFACTOR Phase (DEV Team - After All Tests Pass)
|
||||
|
||||
**DEV Agent Responsibilities:**
|
||||
|
||||
1. **Verify all tests pass** (green phase complete)
|
||||
2. **Review code for quality** (readability, maintainability, performance)
|
||||
3. **Extract duplications** (DRY principle)
|
||||
4. **Optimize performance** (if needed)
|
||||
5. **Ensure tests still pass** after each refactor
|
||||
6. **Update documentation** (if API contracts change)
|
||||
|
||||
**Key Principles:**
|
||||
|
||||
- Tests provide safety net (refactor with confidence)
|
||||
- Make small refactors (easier to debug if tests fail)
|
||||
- Run tests after each change
|
||||
- Don't change test behavior (only implementation)
|
||||
|
||||
**Completion:**
|
||||
|
||||
- All tests pass
|
||||
- Code quality meets team standards
|
||||
- No duplications or code smells
|
||||
- Ready for code review and story approval
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Share this checklist and failing tests** with the dev workflow (manual handoff)
|
||||
2. **Review this checklist** with team in standup or planning
|
||||
3. **Run failing tests** to confirm RED phase: `{test_command_all}`
|
||||
4. **Begin implementation** using implementation checklist as guide
|
||||
5. **Work one test at a time** (red → green for each)
|
||||
6. **Share progress** in daily standup
|
||||
7. **When all tests pass**, refactor code for quality
|
||||
8. **When refactoring complete**, manually update story status to 'done' in sprint-status.yaml
|
||||
|
||||
---
|
||||
|
||||
## Knowledge Base References Applied
|
||||
|
||||
This ATDD workflow consulted the following knowledge fragments:
|
||||
|
||||
- **fixture-architecture.md** - Test fixture patterns with setup/teardown and auto-cleanup using Playwright's `test.extend()`
|
||||
- **data-factories.md** - Factory patterns using `@faker-js/faker` for random test data generation with overrides support
|
||||
- **component-tdd.md** - Component test strategies using Playwright Component Testing
|
||||
- **network-first.md** - Route interception patterns (intercept BEFORE navigation to prevent race conditions)
|
||||
- **test-quality.md** - Test design principles (Given-When-Then, one assertion per test, determinism, isolation)
|
||||
- **test-levels-framework.md** - Test level selection framework (E2E vs API vs Component vs Unit)
|
||||
|
||||
See `tea-index.csv` for complete knowledge fragment mapping.
|
||||
|
||||
---
|
||||
|
||||
## Test Execution Evidence
|
||||
|
||||
### Initial Test Run (RED Phase Verification)
|
||||
|
||||
**Command:** `{test_command_all}`
|
||||
|
||||
**Results:**
|
||||
|
||||
```
|
||||
{paste_test_run_output_showing_all_tests_failing}
|
||||
```
|
||||
|
||||
**Summary:**
|
||||
|
||||
- Total tests: {total_test_count}
|
||||
- Passing: 0 (expected)
|
||||
- Failing: {total_test_count} (expected)
|
||||
- Status: ✅ RED phase verified
|
||||
|
||||
**Expected Failure Messages:**
|
||||
{list_expected_failure_messages_for_each_test}
|
||||
|
||||
---
|
||||
|
||||
## Notes
|
||||
|
||||
{Any additional notes, context, or special considerations for this story}
|
||||
|
||||
- {Note 1}
|
||||
- {Note 2}
|
||||
- {Note 3}
|
||||
|
||||
---
|
||||
|
||||
## Contact
|
||||
|
||||
**Questions or Issues?**
|
||||
|
||||
- Ask in team standup
|
||||
- Tag @{tea_agent_username} in Slack/Discord
|
||||
- Refer to `./bmm/docs/tea-README.md` for workflow documentation
|
||||
- Consult `./bmm/testarch/knowledge` for testing best practices
|
||||
|
||||
---
|
||||
|
||||
**Generated by BMad TEA Agent** - {date}
|
||||
374
_bmad/tea/workflows/testarch/atdd/checklist.md
Normal file
374
_bmad/tea/workflows/testarch/atdd/checklist.md
Normal file
@@ -0,0 +1,374 @@
|
||||
# ATDD Workflow Validation Checklist
|
||||
|
||||
Use this checklist to validate that the ATDD workflow has been executed correctly and all deliverables meet quality standards.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Before starting this workflow, verify:
|
||||
|
||||
- [ ] Story approved with clear acceptance criteria (AC must be testable)
|
||||
- [ ] Development sandbox/environment ready
|
||||
- [ ] Framework scaffolding exists (run `framework` workflow if missing)
|
||||
- [ ] Test framework configuration available (playwright.config.ts or cypress.config.ts)
|
||||
- [ ] Package.json has test dependencies installed (Playwright or Cypress)
|
||||
|
||||
**Halt if missing:** Framework scaffolding or story acceptance criteria
|
||||
|
||||
---
|
||||
|
||||
## Step 1: Story Context and Requirements
|
||||
|
||||
- [ ] Story markdown file loaded and parsed successfully
|
||||
- [ ] All acceptance criteria identified and extracted
|
||||
- [ ] Affected systems and components identified
|
||||
- [ ] Technical constraints documented
|
||||
- [ ] Framework configuration loaded (playwright.config.ts or cypress.config.ts)
|
||||
- [ ] Test directory structure identified from config
|
||||
- [ ] Existing fixture patterns reviewed for consistency
|
||||
- [ ] Similar test patterns searched and found in `{test_dir}`
|
||||
- [ ] Knowledge base fragments loaded:
|
||||
- [ ] `fixture-architecture.md`
|
||||
- [ ] `data-factories.md`
|
||||
- [ ] `component-tdd.md`
|
||||
- [ ] `network-first.md`
|
||||
- [ ] `test-quality.md`
|
||||
|
||||
---
|
||||
|
||||
## Step 2: Test Level Selection and Strategy
|
||||
|
||||
- [ ] Each acceptance criterion analyzed for appropriate test level
|
||||
- [ ] Test level selection framework applied (E2E vs API vs Component vs Unit)
|
||||
- [ ] E2E tests: Critical user journeys and multi-system integration identified
|
||||
- [ ] API tests: Business logic and service contracts identified
|
||||
- [ ] Component tests: UI component behavior and interactions identified
|
||||
- [ ] Unit tests: Pure logic and edge cases identified (if applicable)
|
||||
- [ ] Duplicate coverage avoided (same behavior not tested at multiple levels unnecessarily)
|
||||
- [ ] Tests prioritized using P0-P3 framework (if test-design document exists)
|
||||
- [ ] Primary test level set in `primary_level` variable (typically E2E or API)
|
||||
- [ ] Test levels documented in ATDD checklist
|
||||
|
||||
---
|
||||
|
||||
## Step 3: Failing Tests Generated
|
||||
|
||||
### Test File Structure Created
|
||||
|
||||
- [ ] Test files organized in appropriate directories:
|
||||
- [ ] `tests/e2e/` for end-to-end tests
|
||||
- [ ] `tests/api/` for API tests
|
||||
- [ ] `tests/component/` for component tests
|
||||
- [ ] `tests/support/` for infrastructure (fixtures, factories, helpers)
|
||||
|
||||
### E2E Tests (If Applicable)
|
||||
|
||||
- [ ] E2E test files created in `tests/e2e/`
|
||||
- [ ] All tests follow Given-When-Then format
|
||||
- [ ] Tests use `data-testid` selectors (not CSS classes or fragile selectors)
|
||||
- [ ] One assertion per test (atomic test design)
|
||||
- [ ] No hard waits or sleeps (explicit waits only)
|
||||
- [ ] Network-first pattern applied (route interception BEFORE navigation)
|
||||
- [ ] Tests fail initially (RED phase verified by local test run)
|
||||
- [ ] Failure messages are clear and actionable
|
||||
|
||||
### API Tests (If Applicable)
|
||||
|
||||
- [ ] API test files created in `tests/api/`
|
||||
- [ ] Tests follow Given-When-Then format
|
||||
- [ ] API contracts validated (request/response structure)
|
||||
- [ ] HTTP status codes verified
|
||||
- [ ] Response body validation includes all required fields
|
||||
- [ ] Error cases tested (400, 401, 403, 404, 500)
|
||||
- [ ] Tests fail initially (RED phase verified)
|
||||
|
||||
### Component Tests (If Applicable)
|
||||
|
||||
- [ ] Component test files created in `tests/component/`
|
||||
- [ ] Tests follow Given-When-Then format
|
||||
- [ ] Component mounting works correctly
|
||||
- [ ] Interaction testing covers user actions (click, hover, keyboard)
|
||||
- [ ] State management within component validated
|
||||
- [ ] Props and events tested
|
||||
- [ ] Tests fail initially (RED phase verified)
|
||||
|
||||
### Test Quality Validation
|
||||
|
||||
- [ ] All tests use Given-When-Then structure with clear comments
|
||||
- [ ] All tests have descriptive names explaining what they test
|
||||
- [ ] No duplicate tests (same behavior tested multiple times)
|
||||
- [ ] No flaky patterns (race conditions, timing issues)
|
||||
- [ ] No test interdependencies (tests can run in any order)
|
||||
- [ ] Tests are deterministic (same input always produces same result)
|
||||
|
||||
---
|
||||
|
||||
## Step 4: Data Infrastructure Built
|
||||
|
||||
### Data Factories Created
|
||||
|
||||
- [ ] Factory files created in `tests/support/factories/`
|
||||
- [ ] All factories use `@faker-js/faker` for random data generation (no hardcoded values)
|
||||
- [ ] Factories support overrides for specific test scenarios
|
||||
- [ ] Factories generate complete valid objects matching API contracts
|
||||
- [ ] Helper functions for bulk creation provided (e.g., `createUsers(count)`)
|
||||
- [ ] Factory exports are properly typed (TypeScript)
|
||||
|
||||
### Test Fixtures Created
|
||||
|
||||
- [ ] Fixture files created in `tests/support/fixtures/`
|
||||
- [ ] All fixtures use Playwright's `test.extend()` pattern
|
||||
- [ ] Fixtures have setup phase (arrange test preconditions)
|
||||
- [ ] Fixtures provide data to tests via `await use(data)`
|
||||
- [ ] Fixtures have teardown phase with auto-cleanup (delete created data)
|
||||
- [ ] Fixtures are composable (can use other fixtures if needed)
|
||||
- [ ] Fixtures are isolated (each test gets fresh data)
|
||||
- [ ] Fixtures are type-safe (TypeScript types defined)
|
||||
|
||||
### Mock Requirements Documented
|
||||
|
||||
- [ ] External service mocking requirements identified
|
||||
- [ ] Mock endpoints documented with URLs and methods
|
||||
- [ ] Success response examples provided
|
||||
- [ ] Failure response examples provided
|
||||
- [ ] Mock requirements documented in ATDD checklist for DEV team
|
||||
|
||||
### data-testid Requirements Listed
|
||||
|
||||
- [ ] All required data-testid attributes identified from E2E tests
|
||||
- [ ] data-testid list organized by page or component
|
||||
- [ ] Each data-testid has clear description of element it targets
|
||||
- [ ] data-testid list included in ATDD checklist for DEV team
|
||||
|
||||
---
|
||||
|
||||
## Step 5: Implementation Checklist Created
|
||||
|
||||
- [ ] Implementation checklist created with clear structure
|
||||
- [ ] Each failing test mapped to concrete implementation tasks
|
||||
- [ ] Tasks include:
|
||||
- [ ] Route/component creation
|
||||
- [ ] Business logic implementation
|
||||
- [ ] API integration
|
||||
- [ ] data-testid attribute additions
|
||||
- [ ] Error handling
|
||||
- [ ] Test execution command
|
||||
- [ ] Completion checkbox
|
||||
- [ ] Red-Green-Refactor workflow documented in checklist
|
||||
- [ ] RED phase marked as complete (TEA responsibility)
|
||||
- [ ] GREEN phase tasks listed for DEV team
|
||||
- [ ] REFACTOR phase guidance provided
|
||||
- [ ] Execution commands provided:
|
||||
- [ ] Run all tests: `npm run test:e2e`
|
||||
- [ ] Run specific test file
|
||||
- [ ] Run in headed mode
|
||||
- [ ] Debug specific test
|
||||
- [ ] Estimated effort included (hours or story points)
|
||||
|
||||
---
|
||||
|
||||
## Step 6: Deliverables Generated
|
||||
|
||||
### ATDD Checklist Document Created
|
||||
|
||||
- [ ] Output file created at `{test_artifacts}/atdd-checklist-{story_id}.md`
|
||||
- [ ] Document follows template structure from `atdd-checklist-template.md`
|
||||
- [ ] Document includes all required sections:
|
||||
- [ ] Story summary
|
||||
- [ ] Acceptance criteria breakdown
|
||||
- [ ] Failing tests created (paths and line counts)
|
||||
- [ ] Data factories created
|
||||
- [ ] Fixtures created
|
||||
- [ ] Mock requirements
|
||||
- [ ] Required data-testid attributes
|
||||
- [ ] Implementation checklist
|
||||
- [ ] Red-green-refactor workflow
|
||||
- [ ] Execution commands
|
||||
- [ ] Next steps for DEV team
|
||||
- [ ] Output shared with DEV workflow (manual handoff; not auto-consumed)
|
||||
|
||||
### All Tests Verified to Fail (RED Phase)
|
||||
|
||||
- [ ] Full test suite run locally before finalizing
|
||||
- [ ] All tests fail as expected (RED phase confirmed)
|
||||
- [ ] No tests passing before implementation (if passing, test is invalid)
|
||||
- [ ] Failure messages documented in ATDD checklist
|
||||
- [ ] Failures are due to missing implementation, not test bugs
|
||||
- [ ] Test run output captured for reference
|
||||
|
||||
### Summary Provided
|
||||
|
||||
- [ ] Summary includes:
|
||||
- [ ] Story ID
|
||||
- [ ] Primary test level
|
||||
- [ ] Test counts (E2E, API, Component)
|
||||
- [ ] Test file paths
|
||||
- [ ] Factory count
|
||||
- [ ] Fixture count
|
||||
- [ ] Mock requirements count
|
||||
- [ ] data-testid count
|
||||
- [ ] Implementation task count
|
||||
- [ ] Estimated effort
|
||||
- [ ] Next steps for DEV team
|
||||
- [ ] Output file path
|
||||
- [ ] Knowledge base references applied
|
||||
|
||||
---
|
||||
|
||||
## Quality Checks
|
||||
|
||||
### Test Design Quality
|
||||
|
||||
- [ ] Tests are readable (clear Given-When-Then structure)
|
||||
- [ ] Tests are maintainable (use factories and fixtures, not hardcoded data)
|
||||
- [ ] Tests are isolated (no shared state between tests)
|
||||
- [ ] Tests are deterministic (no race conditions or flaky patterns)
|
||||
- [ ] Tests are atomic (one assertion per test)
|
||||
- [ ] Tests are fast (no unnecessary waits or delays)
|
||||
|
||||
### Knowledge Base Integration
|
||||
|
||||
- [ ] fixture-architecture.md patterns applied to all fixtures
|
||||
- [ ] data-factories.md patterns applied to all factories
|
||||
- [ ] network-first.md patterns applied to E2E tests with network requests
|
||||
- [ ] component-tdd.md patterns applied to component tests
|
||||
- [ ] test-quality.md principles applied to all test design
|
||||
|
||||
### Code Quality
|
||||
|
||||
- [ ] All TypeScript types are correct and complete
|
||||
- [ ] No linting errors in generated test files
|
||||
- [ ] Consistent naming conventions followed
|
||||
- [ ] Imports are organized and correct
|
||||
- [ ] Code follows project style guide
|
||||
|
||||
---
|
||||
|
||||
## Integration Points
|
||||
|
||||
### With DEV Agent
|
||||
|
||||
- [ ] ATDD checklist provides clear implementation guidance
|
||||
- [ ] Implementation tasks are granular and actionable
|
||||
- [ ] data-testid requirements are complete and clear
|
||||
- [ ] Mock requirements include all necessary details
|
||||
- [ ] Execution commands work correctly
|
||||
|
||||
### With Story Workflow
|
||||
|
||||
- [ ] Story ID correctly referenced in output files
|
||||
- [ ] Acceptance criteria from story accurately reflected in tests
|
||||
- [ ] Technical constraints from story considered in test design
|
||||
|
||||
### With Framework Workflow
|
||||
|
||||
- [ ] Test framework configuration correctly detected and used
|
||||
- [ ] Directory structure matches framework setup
|
||||
- [ ] Fixtures and helpers follow established patterns
|
||||
- [ ] Naming conventions consistent with framework standards
|
||||
|
||||
### With test-design Workflow (If Available)
|
||||
|
||||
- [ ] P0 scenarios from test-design prioritized in ATDD
|
||||
- [ ] Risk assessment from test-design considered in test coverage
|
||||
- [ ] Coverage strategy from test-design aligned with ATDD tests
|
||||
|
||||
---
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
All of the following must be true before marking this workflow as complete:
|
||||
|
||||
- [ ] **Story acceptance criteria analyzed** and mapped to appropriate test levels
|
||||
- [ ] **Failing tests created** at all appropriate levels (E2E, API, Component)
|
||||
- [ ] **Given-When-Then format** used consistently across all tests
|
||||
- [ ] **RED phase verified** by local test run (all tests failing as expected)
|
||||
- [ ] **Network-first pattern** applied to E2E tests with network requests
|
||||
- [ ] **Data factories created** using faker (no hardcoded test data)
|
||||
- [ ] **Fixtures created** with auto-cleanup in teardown
|
||||
- [ ] **Mock requirements documented** for external services
|
||||
- [ ] **data-testid attributes listed** for DEV team
|
||||
- [ ] **Implementation checklist created** mapping tests to code tasks
|
||||
- [ ] **Red-green-refactor workflow documented** in ATDD checklist
|
||||
- [ ] **Execution commands provided** and verified to work
|
||||
- [ ] **ATDD checklist document created** and saved to correct location
|
||||
- [ ] **Output file formatted correctly** using template structure
|
||||
- [ ] **Knowledge base references applied** and documented in summary
|
||||
- [ ] **No test quality issues** (flaky patterns, race conditions, hardcoded data)
|
||||
|
||||
---
|
||||
|
||||
## Common Issues and Resolutions
|
||||
|
||||
### Issue: Tests pass before implementation
|
||||
|
||||
**Problem:** A test passes even though no implementation code exists yet.
|
||||
|
||||
**Resolution:**
|
||||
|
||||
- Review test to ensure it's testing actual behavior, not mocked/stubbed behavior
|
||||
- Check if test is accidentally using existing functionality
|
||||
- Verify test assertions are correct and meaningful
|
||||
- Rewrite test to fail until implementation is complete
|
||||
|
||||
### Issue: Network-first pattern not applied
|
||||
|
||||
**Problem:** Route interception happens after navigation, causing race conditions.
|
||||
|
||||
**Resolution:**
|
||||
|
||||
- Move `await page.route()` calls BEFORE `await page.goto()`
|
||||
- Review `network-first.md` knowledge fragment
|
||||
- Update all E2E tests to follow network-first pattern
|
||||
|
||||
### Issue: Hardcoded test data in tests
|
||||
|
||||
**Problem:** Tests use hardcoded strings/numbers instead of factories.
|
||||
|
||||
**Resolution:**
|
||||
|
||||
- Replace all hardcoded data with factory function calls
|
||||
- Use `faker` for all random data generation
|
||||
- Update data-factories to support all required test scenarios
|
||||
|
||||
### Issue: Fixtures missing auto-cleanup
|
||||
|
||||
**Problem:** Fixtures create data but don't clean it up in teardown.
|
||||
|
||||
**Resolution:**
|
||||
|
||||
- Add cleanup logic after `await use(data)` in fixture
|
||||
- Call deletion/cleanup functions in teardown
|
||||
- Verify cleanup works by checking database/storage after test run
|
||||
|
||||
### Issue: Tests have multiple assertions
|
||||
|
||||
**Problem:** Tests verify multiple behaviors in single test (not atomic).
|
||||
|
||||
**Resolution:**
|
||||
|
||||
- Split into separate tests (one assertion per test)
|
||||
- Each test should verify exactly one behavior
|
||||
- Use descriptive test names to clarify what each test verifies
|
||||
|
||||
### Issue: Tests depend on execution order
|
||||
|
||||
**Problem:** Tests fail when run in isolation or different order.
|
||||
|
||||
**Resolution:**
|
||||
|
||||
- Remove shared state between tests
|
||||
- Each test should create its own test data
|
||||
- Use fixtures for consistent setup across tests
|
||||
- Verify tests can run with `.only` flag
|
||||
|
||||
---
|
||||
|
||||
## Notes for TEA Agent
|
||||
|
||||
- **Preflight halt is critical:** Do not proceed if story has no acceptance criteria or framework is missing
|
||||
- **RED phase verification is mandatory:** Tests must fail before sharing with DEV team
|
||||
- **Network-first pattern:** Route interception BEFORE navigation prevents race conditions
|
||||
- **One assertion per test:** Atomic tests provide clear failure diagnosis
|
||||
- **Auto-cleanup is non-negotiable:** Every fixture must clean up data in teardown
|
||||
- **Use knowledge base:** Load relevant fragments (fixture-architecture, data-factories, network-first, component-tdd, test-quality) for guidance
|
||||
- **Share with DEV agent:** ATDD checklist provides implementation roadmap from red to green
|
||||
45
_bmad/tea/workflows/testarch/atdd/instructions.md
Normal file
45
_bmad/tea/workflows/testarch/atdd/instructions.md
Normal file
@@ -0,0 +1,45 @@
|
||||
<!-- Powered by BMAD-CORE™ -->
|
||||
|
||||
# Acceptance Test-Driven Development (ATDD)
|
||||
|
||||
**Workflow ID**: `_bmad/tea/testarch/atdd`
|
||||
**Version**: 5.0 (Step-File Architecture)
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Generates **failing acceptance tests** before implementation (TDD red phase), plus an implementation checklist. Produces tests at appropriate levels (E2E/API/Component) with supporting fixtures and helpers.
|
||||
|
||||
---
|
||||
|
||||
## WORKFLOW ARCHITECTURE
|
||||
|
||||
This workflow uses **step-file architecture**:
|
||||
|
||||
- **Micro-file Design**: Each step is self-contained
|
||||
- **JIT Loading**: Only the current step file is in memory
|
||||
- **Sequential Enforcement**: Execute steps in order without skipping
|
||||
|
||||
---
|
||||
|
||||
## INITIALIZATION SEQUENCE
|
||||
|
||||
### 1. Configuration Loading
|
||||
|
||||
From `workflow.yaml`, resolve:
|
||||
|
||||
- `config_source`, `test_artifacts`, `user_name`, `communication_language`, `document_output_language`, `date`
|
||||
- `test_dir`
|
||||
|
||||
### 2. First Step
|
||||
|
||||
Load, read completely, and execute:
|
||||
`{project-root}/_bmad/tea/workflows/testarch/atdd/steps-c/step-01-preflight-and-context.md`
|
||||
|
||||
### 3. Resume Support
|
||||
|
||||
If the user selects **Resume** mode, load, read completely, and execute:
|
||||
`{project-root}/_bmad/tea/workflows/testarch/atdd/steps-c/step-01b-resume.md`
|
||||
|
||||
This checks the output document for progress tracking frontmatter and routes to the next incomplete step.
|
||||
@@ -0,0 +1,226 @@
|
||||
---
|
||||
name: 'step-01-preflight-and-context'
|
||||
description: 'Verify prerequisites and load story, framework, and knowledge base'
|
||||
outputFile: '{test_artifacts}/atdd-checklist-{story_id}.md'
|
||||
nextStepFile: './step-02-generation-mode.md'
|
||||
knowledgeIndex: '{project-root}/_bmad/tea/testarch/tea-index.csv'
|
||||
---
|
||||
|
||||
# Step 1: Preflight & Context Loading
|
||||
|
||||
## STEP GOAL
|
||||
|
||||
Verify prerequisites and load all required inputs before generating failing tests.
|
||||
|
||||
## MANDATORY EXECUTION RULES
|
||||
|
||||
- 📖 Read the entire step file before acting
|
||||
- ✅ Speak in `{communication_language}`
|
||||
- 🚫 Halt if requirements are missing
|
||||
|
||||
---
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
- 💾 Record outputs before proceeding
|
||||
- 📖 Load the next step only when instructed
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: config, loaded artifacts, and knowledge fragments
|
||||
- Focus: this step's goal only
|
||||
- Limits: do not execute future steps
|
||||
- Dependencies: prior steps' outputs (if any)
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly. Do not skip, reorder, or improvise.
|
||||
|
||||
## 1. Stack Detection
|
||||
|
||||
**Read `config.test_stack_type`** from `{config_source}`.
|
||||
|
||||
**Auto-Detection Algorithm** (when `test_stack_type` is `"auto"` or not configured):
|
||||
|
||||
- Scan `{project-root}` for project manifests:
|
||||
- **Frontend indicators**: `package.json` with react/vue/angular/next dependencies, `playwright.config.*`, `vite.config.*`, `webpack.config.*`
|
||||
- **Backend indicators**: `pyproject.toml`, `pom.xml`/`build.gradle`, `go.mod`, `*.csproj`/`*.sln`, `Gemfile`, `Cargo.toml`
|
||||
- **Both present** = `fullstack`; only frontend = `frontend`; only backend = `backend`
|
||||
- Explicit `test_stack_type` config value overrides auto-detection
|
||||
- **Backward compatibility**: if `test_stack_type` is not in config, treat as `"auto"` (preserves current frontend behavior for existing installs)
|
||||
|
||||
Store result as `{detected_stack}` = `frontend` | `backend` | `fullstack`
|
||||
|
||||
---
|
||||
|
||||
## 2. Prerequisites (Hard Requirements)
|
||||
|
||||
- Story approved with **clear acceptance criteria**
|
||||
- Test framework configured:
|
||||
- **If {detected_stack} is `frontend` or `fullstack`:** `playwright.config.ts` or `cypress.config.ts`
|
||||
- **If {detected_stack} is `backend`:** relevant test config exists (e.g., `conftest.py`, `src/test/`, `*_test.go`, `.rspec`)
|
||||
- Development environment available
|
||||
|
||||
If any are missing: **HALT** and notify the user.
|
||||
|
||||
---
|
||||
|
||||
## 3. Load Story Context
|
||||
|
||||
- Read story markdown from `{story_file}` (or ask user if not provided)
|
||||
- Extract acceptance criteria and constraints
|
||||
- Identify affected components and integrations
|
||||
|
||||
---
|
||||
|
||||
## 4. Load Framework & Existing Patterns
|
||||
|
||||
- Read framework config
|
||||
- Inspect `{test_dir}` for existing test patterns, fixtures, helpers
|
||||
|
||||
## 4.5 Read TEA Config Flags
|
||||
|
||||
From `{config_source}`:
|
||||
|
||||
- `tea_use_playwright_utils`
|
||||
- `tea_use_pactjs_utils`
|
||||
- `tea_pact_mcp`
|
||||
- `tea_browser_automation`
|
||||
- `test_stack_type`
|
||||
|
||||
---
|
||||
|
||||
### Tiered Knowledge Loading
|
||||
|
||||
Load fragments based on their `tier` classification in `tea-index.csv`:
|
||||
|
||||
1. **Core tier** (always load): Foundational fragments required for this workflow
|
||||
2. **Extended tier** (load on-demand): Load when deeper analysis is needed or when the user's context requires it
|
||||
3. **Specialized tier** (load only when relevant): Load only when the specific use case matches (e.g., contract-testing only for microservices, email-auth only for email flows)
|
||||
|
||||
> **Context Efficiency**: Loading only core fragments reduces context usage by 40-50% compared to loading all fragments.
|
||||
|
||||
### Playwright Utils Loading Profiles
|
||||
|
||||
**If `tea_use_playwright_utils` is enabled**, select the appropriate loading profile:
|
||||
|
||||
- **API-only profile** (when `{detected_stack}` is `backend` or no `page.goto`/`page.locator` found in test files):
|
||||
Load: `overview`, `api-request`, `auth-session`, `recurse` (~1,800 lines)
|
||||
|
||||
- **Full UI+API profile** (when `{detected_stack}` is `frontend`/`fullstack` or browser tests detected):
|
||||
Load: all Playwright Utils core fragments (~4,500 lines)
|
||||
|
||||
**Detection**: Scan `{test_dir}` for files containing `page.goto` or `page.locator`. If none found, use API-only profile.
|
||||
|
||||
### Pact.js Utils Loading
|
||||
|
||||
**If `tea_use_pactjs_utils` is enabled** (and `{detected_stack}` is `backend` or `fullstack`, or microservices indicators detected):
|
||||
|
||||
Load: `pactjs-utils-overview.md`, `pactjs-utils-consumer-helpers.md`, `pactjs-utils-provider-verifier.md`, `pactjs-utils-request-filter.md`
|
||||
|
||||
**If `tea_use_pactjs_utils` is disabled** but contract testing is relevant:
|
||||
|
||||
Load: `contract-testing.md`
|
||||
|
||||
### Pact MCP Loading
|
||||
|
||||
**If `tea_pact_mcp` is `"mcp"`:**
|
||||
|
||||
Load: `pact-mcp.md`
|
||||
|
||||
## 5. Load Knowledge Base Fragments
|
||||
|
||||
Use `{knowledgeIndex}` to load:
|
||||
|
||||
**Core (always):**
|
||||
|
||||
- `data-factories.md`
|
||||
- `component-tdd.md`
|
||||
- `test-quality.md`
|
||||
- `test-healing-patterns.md`
|
||||
|
||||
**If {detected_stack} is `frontend` or `fullstack`:**
|
||||
|
||||
- `selector-resilience.md`
|
||||
- `timing-debugging.md`
|
||||
|
||||
**Playwright Utils (if enabled and {detected_stack} is `frontend` or `fullstack`):**
|
||||
|
||||
- `overview.md`, `api-request.md`, `network-recorder.md`, `auth-session.md`, `intercept-network-call.md`, `recurse.md`, `log.md`, `file-utils.md`, `network-error-monitor.md`, `fixtures-composition.md`
|
||||
|
||||
**Playwright CLI (if tea_browser_automation is "cli" or "auto" and {detected_stack} is `frontend` or `fullstack`):**
|
||||
|
||||
- `playwright-cli.md`
|
||||
|
||||
**MCP Patterns (if tea_browser_automation is "mcp" or "auto" and {detected_stack} is `frontend` or `fullstack`):**
|
||||
|
||||
- (existing MCP-related fragments, if any are added in future)
|
||||
|
||||
**Traditional Patterns (if utils disabled and {detected_stack} is `frontend` or `fullstack`):**
|
||||
|
||||
- `fixture-architecture.md`
|
||||
- `network-first.md`
|
||||
|
||||
**Backend Patterns (if {detected_stack} is `backend` or `fullstack`):**
|
||||
|
||||
- `test-levels-framework.md`
|
||||
- `test-priorities-matrix.md`
|
||||
- `ci-burn-in.md`
|
||||
|
||||
**Pact.js Utils (if enabled):**
|
||||
|
||||
- `pactjs-utils-overview.md`, `pactjs-utils-consumer-helpers.md`, `pactjs-utils-provider-verifier.md`, `pactjs-utils-request-filter.md`
|
||||
|
||||
**Contract Testing (if pactjs-utils disabled but relevant):**
|
||||
|
||||
- `contract-testing.md`
|
||||
|
||||
**Pact MCP (if tea_pact_mcp is "mcp"):**
|
||||
|
||||
- `pact-mcp.md`
|
||||
|
||||
---
|
||||
|
||||
## 6. Confirm Inputs
|
||||
|
||||
Summarize loaded inputs and confirm with the user. Then proceed.
|
||||
|
||||
---
|
||||
|
||||
## 7. Save Progress
|
||||
|
||||
**Save this step's accumulated work to `{outputFile}`.**
|
||||
|
||||
- **If `{outputFile}` does not exist** (first save), create it with YAML frontmatter:
|
||||
|
||||
```yaml
|
||||
---
|
||||
stepsCompleted: ['step-01-preflight-and-context']
|
||||
lastStep: 'step-01-preflight-and-context'
|
||||
lastSaved: '{date}'
|
||||
---
|
||||
```
|
||||
|
||||
Then write this step's output below the frontmatter.
|
||||
|
||||
- **If `{outputFile}` already exists**, update:
|
||||
- Add `'step-01-preflight-and-context'` to `stepsCompleted` array (only if not already present)
|
||||
- Set `lastStep: 'step-01-preflight-and-context'`
|
||||
- Set `lastSaved: '{date}'`
|
||||
- Append this step's output to the appropriate section.
|
||||
|
||||
**Update `inputDocuments`**: Set `inputDocuments` in the output template frontmatter to the list of artifact paths loaded in this step (e.g., knowledge fragments, test design documents, configuration files).
|
||||
|
||||
Load next step: `{nextStepFile}`
|
||||
|
||||
## 🚨 SYSTEM SUCCESS/FAILURE METRICS:
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- Step completed in full with required outputs
|
||||
|
||||
### ❌ SYSTEM FAILURE:
|
||||
|
||||
- Skipped sequence steps or missing outputs
|
||||
**Master Rule:** Skipping steps is FORBIDDEN.
|
||||
96
_bmad/tea/workflows/testarch/atdd/steps-c/step-01b-resume.md
Normal file
96
_bmad/tea/workflows/testarch/atdd/steps-c/step-01b-resume.md
Normal file
@@ -0,0 +1,96 @@
|
||||
---
|
||||
name: 'step-01b-resume'
|
||||
description: 'Resume interrupted workflow from last completed step'
|
||||
outputFile: '{test_artifacts}/atdd-checklist-{story_id}.md'
|
||||
---
|
||||
|
||||
# Step 1b: Resume Workflow
|
||||
|
||||
## STEP GOAL
|
||||
|
||||
Resume an interrupted workflow by loading the existing output document, displaying progress, and routing to the next incomplete step.
|
||||
|
||||
## MANDATORY EXECUTION RULES
|
||||
|
||||
- 📖 Read the entire step file before acting
|
||||
- ✅ Speak in `{communication_language}`
|
||||
|
||||
---
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
- 📖 Load the next step only when instructed
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: Output document with progress frontmatter
|
||||
- Focus: Load progress and route to next step
|
||||
- Limits: Do not re-execute completed steps
|
||||
- Dependencies: Output document must exist from a previous run
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly.
|
||||
|
||||
### 1. Load Output Document
|
||||
|
||||
Read `{outputFile}` and parse YAML frontmatter for:
|
||||
|
||||
- `stepsCompleted` — array of completed step names
|
||||
- `lastStep` — last completed step name
|
||||
- `lastSaved` — timestamp of last save
|
||||
|
||||
**If `{outputFile}` does not exist**, display:
|
||||
|
||||
"⚠️ **No previous progress found.** There is no output document to resume from. Please use **[C] Create** to start a fresh workflow run."
|
||||
|
||||
**THEN:** Halt. Do not proceed.
|
||||
|
||||
---
|
||||
|
||||
### 2. Display Progress Dashboard
|
||||
|
||||
Display progress with ✅/⬜ indicators:
|
||||
|
||||
1. ✅/⬜ Preflight & Context (step-01-preflight-and-context)
|
||||
2. ✅/⬜ Generation Mode (step-02-generation-mode)
|
||||
3. ✅/⬜ Test Strategy (step-03-test-strategy)
|
||||
4. ✅/⬜ Generate Tests + Aggregate (step-04c-aggregate)
|
||||
5. ✅/⬜ Validate & Complete (step-05-validate-and-complete)
|
||||
|
||||
---
|
||||
|
||||
### 3. Route to Next Step
|
||||
|
||||
Based on `lastStep`, load the next incomplete step:
|
||||
|
||||
- `'step-01-preflight-and-context'` → load `./step-02-generation-mode.md`
|
||||
- `'step-02-generation-mode'` → load `./step-03-test-strategy.md`
|
||||
- `'step-03-test-strategy'` → load `./step-04-generate-tests.md`
|
||||
- `'step-04c-aggregate'` → load `./step-05-validate-and-complete.md`
|
||||
- `'step-05-validate-and-complete'` → **Workflow already complete.** Display: "✅ **All steps completed.** Use **[V] Validate** to review outputs or **[E] Edit** to make revisions." Then halt.
|
||||
|
||||
**If `lastStep` does not match any value above**, display: "⚠️ **Unknown progress state** (`lastStep`: {lastStep}). Please use **[C] Create** to start fresh." Then halt.
|
||||
|
||||
**Otherwise**, load the identified step file, read completely, and execute.
|
||||
|
||||
The existing content in `{outputFile}` provides context from previously completed steps.
|
||||
|
||||
---
|
||||
|
||||
## 🚨 SYSTEM SUCCESS/FAILURE METRICS
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- Output document loaded and parsed correctly
|
||||
- Progress dashboard displayed accurately
|
||||
- Routed to correct next step
|
||||
|
||||
### ❌ SYSTEM FAILURE:
|
||||
|
||||
- Not loading output document
|
||||
- Incorrect progress display
|
||||
- Routing to wrong step
|
||||
|
||||
**Master Rule:** Resume MUST route to the exact next incomplete step. Never re-execute completed steps.
|
||||
@@ -0,0 +1,125 @@
|
||||
---
|
||||
name: 'step-02-generation-mode'
|
||||
description: 'Choose AI generation or recording mode'
|
||||
outputFile: '{test_artifacts}/atdd-checklist-{story_id}.md'
|
||||
nextStepFile: './step-03-test-strategy.md'
|
||||
---
|
||||
|
||||
# Step 2: Generation Mode Selection
|
||||
|
||||
## STEP GOAL
|
||||
|
||||
Choose the appropriate generation mode for ATDD tests.
|
||||
|
||||
## MANDATORY EXECUTION RULES
|
||||
|
||||
- 📖 Read the entire step file before acting
|
||||
- ✅ Speak in `{communication_language}`
|
||||
|
||||
---
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
- 💾 Record outputs before proceeding
|
||||
- 📖 Load the next step only when instructed
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: config, loaded artifacts, and knowledge fragments
|
||||
- Focus: this step's goal only
|
||||
- Limits: do not execute future steps
|
||||
- Dependencies: prior steps' outputs (if any)
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly. Do not skip, reorder, or improvise.
|
||||
|
||||
## 1. Default Mode: AI Generation
|
||||
|
||||
Use AI generation when:
|
||||
|
||||
- Acceptance criteria are clear
|
||||
- Scenarios are standard (CRUD, auth, API, navigation)
|
||||
- **If {detected_stack} is `backend`:** Always use AI generation (no browser recording needed)
|
||||
|
||||
Proceed directly to test strategy if this applies.
|
||||
|
||||
---
|
||||
|
||||
## 2. Optional Mode: Recording (Complex UI)
|
||||
|
||||
**Skip this section entirely if {detected_stack} is `backend`.** For backend projects, use AI generation from API documentation, OpenAPI/Swagger specs, or source code analysis instead.
|
||||
|
||||
**If {detected_stack} is `frontend` or `fullstack`:**
|
||||
|
||||
Use recording when UI interactions need live browser verification.
|
||||
|
||||
**Tool selection based on `config.tea_browser_automation`:**
|
||||
|
||||
If `auto`:
|
||||
|
||||
> **Note:** `${timestamp}` is a placeholder the agent should replace with a unique value (e.g., epoch seconds) for session isolation.
|
||||
|
||||
- **Simple recording** (snapshot selectors, capture structure): Use CLI
|
||||
- `playwright-cli -s=tea-atdd-${timestamp} open <url>` → `playwright-cli -s=tea-atdd-${timestamp} snapshot` → extract refs
|
||||
- **Complex recording** (drag/drop, wizards, multi-step state): Use MCP
|
||||
- Full browser automation with rich tool semantics
|
||||
- **Fallback:** If preferred tool unavailable, use the other; if neither, skip recording
|
||||
|
||||
If `cli`:
|
||||
|
||||
- Use Playwright CLI for all recording
|
||||
- `playwright-cli -s=tea-atdd-${timestamp} open <url>`, `snapshot`, `screenshot`, `click <ref>`, etc.
|
||||
|
||||
If `mcp`:
|
||||
|
||||
- Use Playwright MCP tools for all recording (current behavior)
|
||||
- Confirm MCP availability, record selectors and interactions
|
||||
|
||||
If `none`:
|
||||
|
||||
- Skip recording mode entirely, use AI generation from documentation
|
||||
|
||||
---
|
||||
|
||||
## 3. Confirm Mode
|
||||
|
||||
State the chosen mode and why. Then proceed.
|
||||
|
||||
---
|
||||
|
||||
## 4. Save Progress
|
||||
|
||||
**Save this step's accumulated work to `{outputFile}`.**
|
||||
|
||||
- **If `{outputFile}` does not exist** (first save), create it with YAML frontmatter:
|
||||
|
||||
```yaml
|
||||
---
|
||||
stepsCompleted: ['step-02-generation-mode']
|
||||
lastStep: 'step-02-generation-mode'
|
||||
lastSaved: '{date}'
|
||||
---
|
||||
```
|
||||
|
||||
Then write this step's output below the frontmatter.
|
||||
|
||||
- **If `{outputFile}` already exists**, update:
|
||||
- Add `'step-02-generation-mode'` to `stepsCompleted` array (only if not already present)
|
||||
- Set `lastStep: 'step-02-generation-mode'`
|
||||
- Set `lastSaved: '{date}'`
|
||||
- Append this step's output to the appropriate section.
|
||||
|
||||
Load next step: `{nextStepFile}`
|
||||
|
||||
## 🚨 SYSTEM SUCCESS/FAILURE METRICS:
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- Step completed in full with required outputs
|
||||
|
||||
### ❌ SYSTEM FAILURE:
|
||||
|
||||
- Skipped sequence steps or missing outputs
|
||||
**Master Rule:** Skipping steps is FORBIDDEN.
|
||||
@@ -0,0 +1,110 @@
|
||||
---
|
||||
name: 'step-03-test-strategy'
|
||||
description: 'Map acceptance criteria to test levels and priorities'
|
||||
outputFile: '{test_artifacts}/atdd-checklist-{story_id}.md'
|
||||
nextStepFile: './step-04-generate-tests.md'
|
||||
---
|
||||
|
||||
# Step 3: Test Strategy
|
||||
|
||||
## STEP GOAL
|
||||
|
||||
Translate acceptance criteria into a prioritized, level-appropriate test plan.
|
||||
|
||||
## MANDATORY EXECUTION RULES
|
||||
|
||||
- 📖 Read the entire step file before acting
|
||||
- ✅ Speak in `{communication_language}`
|
||||
- 🚫 Avoid duplicate coverage across levels
|
||||
|
||||
---
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
- 💾 Record outputs before proceeding
|
||||
- 📖 Load the next step only when instructed
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: config, loaded artifacts, and knowledge fragments
|
||||
- Focus: this step's goal only
|
||||
- Limits: do not execute future steps
|
||||
- Dependencies: prior steps' outputs (if any)
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly. Do not skip, reorder, or improvise.
|
||||
|
||||
## 1. Map Acceptance Criteria
|
||||
|
||||
- Convert each acceptance criterion into test scenarios
|
||||
- Include negative and edge cases where risk is high
|
||||
|
||||
---
|
||||
|
||||
## 2. Select Test Levels
|
||||
|
||||
Choose the best level per scenario based on `{detected_stack}`:
|
||||
|
||||
**If {detected_stack} is `frontend` or `fullstack`:**
|
||||
|
||||
- **E2E** for critical user journeys
|
||||
- **API** for business logic and service contracts
|
||||
- **Component** for UI behavior
|
||||
|
||||
**If {detected_stack} is `backend` or `fullstack`:**
|
||||
|
||||
- **Unit** for pure functions, business logic, and edge cases
|
||||
- **Integration** for service interactions, database queries, and middleware
|
||||
- **API/Contract** for endpoint validation, request/response schemas, and Pact contracts
|
||||
- **No E2E** for pure backend projects (no browser-based testing needed)
|
||||
|
||||
---
|
||||
|
||||
## 3. Prioritize Tests
|
||||
|
||||
Assign P0–P3 priorities using risk and business impact.
|
||||
|
||||
---
|
||||
|
||||
## 4. Confirm Red Phase Requirements
|
||||
|
||||
Ensure all tests are designed to **fail before implementation** (TDD red phase).
|
||||
|
||||
---
|
||||
|
||||
## 5. Save Progress
|
||||
|
||||
**Save this step's accumulated work to `{outputFile}`.**
|
||||
|
||||
- **If `{outputFile}` does not exist** (first save), create it with YAML frontmatter:
|
||||
|
||||
```yaml
|
||||
---
|
||||
stepsCompleted: ['step-03-test-strategy']
|
||||
lastStep: 'step-03-test-strategy'
|
||||
lastSaved: '{date}'
|
||||
---
|
||||
```
|
||||
|
||||
Then write this step's output below the frontmatter.
|
||||
|
||||
- **If `{outputFile}` already exists**, update:
|
||||
- Add `'step-03-test-strategy'` to `stepsCompleted` array (only if not already present)
|
||||
- Set `lastStep: 'step-03-test-strategy'`
|
||||
- Set `lastSaved: '{date}'`
|
||||
- Append this step's output to the appropriate section.
|
||||
|
||||
Load next step: `{nextStepFile}`
|
||||
|
||||
## 🚨 SYSTEM SUCCESS/FAILURE METRICS:
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- Step completed in full with required outputs
|
||||
|
||||
### ❌ SYSTEM FAILURE:
|
||||
|
||||
- Skipped sequence steps or missing outputs
|
||||
**Master Rule:** Skipping steps is FORBIDDEN.
|
||||
@@ -0,0 +1,334 @@
|
||||
---
|
||||
name: 'step-04-generate-tests'
|
||||
description: 'Orchestrate adaptive FAILING test generation (TDD red phase)'
|
||||
nextStepFile: './step-04c-aggregate.md'
|
||||
---
|
||||
|
||||
# Step 4: Orchestrate Adaptive FAILING Test Generation
|
||||
|
||||
## STEP GOAL
|
||||
|
||||
Select execution mode deterministically, then generate FAILING API and E2E tests (TDD RED PHASE) with consistent output contracts across agent-team, subagent, or sequential execution.
|
||||
|
||||
## MANDATORY EXECUTION RULES
|
||||
|
||||
- 📖 Read the entire step file before acting
|
||||
- ✅ Speak in `{communication_language}`
|
||||
- ✅ Resolve execution mode from config (`tea_execution_mode`, `tea_capability_probe`)
|
||||
- ✅ Apply fallback rules deterministically when requested mode is unsupported
|
||||
- ✅ Generate FAILING tests only (TDD red phase)
|
||||
- ✅ Wait for required worker steps to complete
|
||||
- ❌ Do NOT skip capability checks when probing is enabled
|
||||
- ❌ Do NOT generate passing tests (this is red phase)
|
||||
- ❌ Do NOT proceed until required worker steps finish
|
||||
|
||||
---
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
- 💾 Wait for subagent outputs
|
||||
- 📖 Load the next step only when instructed
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: config, acceptance criteria from Step 1, test strategy from Step 3
|
||||
- Focus: orchestration only (mode selection + worker dispatch)
|
||||
- Limits: do not generate tests directly (delegate to worker steps)
|
||||
- Dependencies: Steps 1-3 outputs
|
||||
|
||||
---
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly. Do not skip, reorder, or improvise.
|
||||
|
||||
### 1. Prepare Execution Context
|
||||
|
||||
**Generate unique timestamp** for temp file naming:
|
||||
|
||||
```javascript
|
||||
const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
|
||||
```
|
||||
|
||||
**Prepare input context for both subagents:**
|
||||
|
||||
```javascript
|
||||
const parseBooleanFlag = (value, defaultValue = true) => {
|
||||
if (typeof value === 'string') {
|
||||
const normalized = value.trim().toLowerCase();
|
||||
if (['false', '0', 'off', 'no'].includes(normalized)) return false;
|
||||
if (['true', '1', 'on', 'yes'].includes(normalized)) return true;
|
||||
}
|
||||
if (value === undefined || value === null) return defaultValue;
|
||||
return Boolean(value);
|
||||
};
|
||||
|
||||
const subagentContext = {
|
||||
story_acceptance_criteria: /* from Step 1 */,
|
||||
test_strategy: /* from Step 3 */,
|
||||
knowledge_fragments_loaded: /* list of fragments */,
|
||||
config: {
|
||||
test_framework: config.test_framework,
|
||||
use_playwright_utils: config.tea_use_playwright_utils,
|
||||
use_pactjs_utils: config.tea_use_pactjs_utils,
|
||||
pact_mcp: config.tea_pact_mcp, // "mcp" | "none"
|
||||
browser_automation: config.tea_browser_automation,
|
||||
execution_mode: config.tea_execution_mode || 'auto', // "auto" | "subagent" | "agent-team" | "sequential"
|
||||
capability_probe: parseBooleanFlag(config.tea_capability_probe, true), // supports booleans and "false"/"true" strings
|
||||
provider_endpoint_map: /* from Step 1/3 context, if use_pactjs_utils enabled */,
|
||||
},
|
||||
timestamp: timestamp
|
||||
};
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 2. Resolve Execution Mode with Capability Probe
|
||||
|
||||
```javascript
|
||||
const normalizeUserExecutionMode = (mode) => {
|
||||
if (typeof mode !== 'string') return null;
|
||||
const normalized = mode.trim().toLowerCase().replace(/[-_]/g, ' ').replace(/\s+/g, ' ');
|
||||
|
||||
if (normalized === 'auto') return 'auto';
|
||||
if (normalized === 'sequential') return 'sequential';
|
||||
if (normalized === 'subagent' || normalized === 'sub agent' || normalized === 'subagents' || normalized === 'sub agents') {
|
||||
return 'subagent';
|
||||
}
|
||||
if (normalized === 'agent team' || normalized === 'agent teams' || normalized === 'agentteam') {
|
||||
return 'agent-team';
|
||||
}
|
||||
|
||||
return null;
|
||||
};
|
||||
|
||||
const normalizeConfigExecutionMode = (mode) => {
|
||||
if (mode === 'subagent') return 'subagent';
|
||||
if (mode === 'auto' || mode === 'sequential' || mode === 'subagent' || mode === 'agent-team') {
|
||||
return mode;
|
||||
}
|
||||
return null;
|
||||
};
|
||||
|
||||
// Explicit user instruction in the active run takes priority over config.
|
||||
const explicitModeFromUser = normalizeUserExecutionMode(runtime.getExplicitExecutionModeHint?.() || null);
|
||||
|
||||
const requestedMode = explicitModeFromUser || normalizeConfigExecutionMode(subagentContext.config.execution_mode) || 'auto';
|
||||
const probeEnabled = subagentContext.config.capability_probe;
|
||||
|
||||
const supports = {
|
||||
subagent: runtime.canLaunchSubagents?.() === true,
|
||||
agentTeam: runtime.canLaunchAgentTeams?.() === true,
|
||||
};
|
||||
|
||||
let resolvedMode = requestedMode;
|
||||
|
||||
if (requestedMode === 'auto') {
|
||||
if (supports.agentTeam) resolvedMode = 'agent-team';
|
||||
else if (supports.subagent) resolvedMode = 'subagent';
|
||||
else resolvedMode = 'sequential';
|
||||
} else if (probeEnabled && requestedMode === 'agent-team' && !supports.agentTeam) {
|
||||
resolvedMode = supports.subagent ? 'subagent' : 'sequential';
|
||||
} else if (probeEnabled && requestedMode === 'subagent' && !supports.subagent) {
|
||||
resolvedMode = 'sequential';
|
||||
}
|
||||
|
||||
subagentContext.execution = {
|
||||
requestedMode,
|
||||
resolvedMode,
|
||||
probeEnabled,
|
||||
supports,
|
||||
};
|
||||
|
||||
if (!probeEnabled && (requestedMode === 'agent-team' || requestedMode === 'subagent')) {
|
||||
const unsupportedRequestedMode =
|
||||
(requestedMode === 'agent-team' && !supports.agentTeam) || (requestedMode === 'subagent' && !supports.subagent);
|
||||
|
||||
if (unsupportedRequestedMode) {
|
||||
subagentContext.execution.error = `Requested execution mode "${requestedMode}" is unavailable because capability probing is disabled.`;
|
||||
throw new Error(subagentContext.execution.error);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Resolution precedence:
|
||||
|
||||
1. Explicit user request in this run (`agent team` => `agent-team`; `subagent` => `subagent`; `sequential`; `auto`)
|
||||
2. `tea_execution_mode` from config
|
||||
3. Runtime capability fallback (when probing enabled)
|
||||
|
||||
If probing is disabled, honor the requested mode strictly. If that mode cannot be executed at runtime, fail with explicit error instead of silent fallback.
|
||||
|
||||
---
|
||||
|
||||
### 3. Dispatch Worker A: Failing API Test Generation
|
||||
|
||||
**Dispatch worker:**
|
||||
|
||||
- **Subagent File:** `./step-04a-subagent-api-failing.md`
|
||||
- **Output File:** `/tmp/tea-atdd-api-tests-${timestamp}.json`
|
||||
- **Context:** Pass `subagentContext`
|
||||
- **Execution:**
|
||||
- `agent-team` or `subagent`: launch non-blocking
|
||||
- `sequential`: run blocking and wait before next dispatch
|
||||
- **TDD Phase:** RED (failing tests)
|
||||
|
||||
**System Action:**
|
||||
|
||||
```
|
||||
🚀 Launching Subagent A: FAILING API Test Generation (RED PHASE)
|
||||
📝 Output: /tmp/tea-atdd-api-tests-${timestamp}.json
|
||||
⚙️ Mode: ${resolvedMode}
|
||||
🔴 TDD Phase: RED (tests will fail until feature implemented)
|
||||
⏳ Status: Running...
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 4. Dispatch Worker B: Failing E2E Test Generation
|
||||
|
||||
**Dispatch worker:**
|
||||
|
||||
- **Subagent File:** `./step-04b-subagent-e2e-failing.md`
|
||||
- **Output File:** `/tmp/tea-atdd-e2e-tests-${timestamp}.json`
|
||||
- **Context:** Pass `subagentContext`
|
||||
- **Execution:**
|
||||
- `agent-team` or `subagent`: launch non-blocking
|
||||
- `sequential`: run blocking and wait before next dispatch
|
||||
- **TDD Phase:** RED (failing tests)
|
||||
|
||||
**System Action:**
|
||||
|
||||
```
|
||||
🚀 Launching Subagent B: FAILING E2E Test Generation (RED PHASE)
|
||||
📝 Output: /tmp/tea-atdd-e2e-tests-${timestamp}.json
|
||||
⚙️ Mode: ${resolvedMode}
|
||||
🔴 TDD Phase: RED (tests will fail until feature implemented)
|
||||
⏳ Status: Running...
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 5. Wait for Required Worker Completion
|
||||
|
||||
**If `resolvedMode` is `agent-team` or `subagent`:**
|
||||
|
||||
```
|
||||
⏳ Waiting for subagents to complete...
|
||||
├── Subagent A (API RED): Running... ⟳
|
||||
└── Subagent B (E2E RED): Running... ⟳
|
||||
|
||||
[... time passes ...]
|
||||
|
||||
├── Subagent A (API RED): Complete ✅
|
||||
└── Subagent B (E2E RED): Complete ✅
|
||||
|
||||
✅ All subagents completed successfully!
|
||||
```
|
||||
|
||||
**If `resolvedMode` is `sequential`:**
|
||||
|
||||
```
|
||||
✅ Sequential mode: each worker already completed during dispatch.
|
||||
```
|
||||
|
||||
**Verify both outputs exist:**
|
||||
|
||||
```javascript
|
||||
const apiOutputExists = fs.existsSync(`/tmp/tea-atdd-api-tests-${timestamp}.json`);
|
||||
const e2eOutputExists = fs.existsSync(`/tmp/tea-atdd-e2e-tests-${timestamp}.json`);
|
||||
|
||||
if (!apiOutputExists || !e2eOutputExists) {
|
||||
throw new Error('One or both subagent outputs missing!');
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 6. TDD Red Phase Report
|
||||
|
||||
**Display TDD status:**
|
||||
|
||||
```
|
||||
🔴 TDD RED PHASE: Failing Tests Generated
|
||||
|
||||
✅ Both subagents completed:
|
||||
- API Tests: Generated with test.skip()
|
||||
- E2E Tests: Generated with test.skip()
|
||||
|
||||
📋 All tests assert EXPECTED behavior
|
||||
📋 All tests will FAIL until feature implemented
|
||||
📋 This is INTENTIONAL (TDD red phase)
|
||||
|
||||
Next: Aggregation will verify TDD compliance
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 7. Execution Report
|
||||
|
||||
**Display performance metrics:**
|
||||
|
||||
```
|
||||
🚀 Performance Report:
|
||||
- Execution Mode: {resolvedMode}
|
||||
- API Test Generation: ~X minutes
|
||||
- E2E Test Generation: ~Y minutes
|
||||
- Total Elapsed: ~mode-dependent
|
||||
- Parallel Gain: ~50% faster when mode is subagent/agent-team
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 8. Proceed to Aggregation
|
||||
|
||||
**Load aggregation step:**
|
||||
Load next step: `{nextStepFile}`
|
||||
|
||||
The aggregation step (4C) will:
|
||||
|
||||
- Read both subagent outputs
|
||||
- Verify TDD red phase compliance (all tests have test.skip())
|
||||
- Write all test files to disk
|
||||
- Generate ATDD checklist
|
||||
- Calculate summary statistics
|
||||
|
||||
---
|
||||
|
||||
## EXIT CONDITION
|
||||
|
||||
Proceed to Step 4C (Aggregation) when:
|
||||
|
||||
- ✅ Subagent A (API failing tests) completed successfully
|
||||
- ✅ Subagent B (E2E failing tests) completed successfully
|
||||
- ✅ Both output files exist and are valid JSON
|
||||
- ✅ TDD red phase status reported
|
||||
|
||||
**Do NOT proceed if:**
|
||||
|
||||
- ❌ One or both subagents failed
|
||||
- ❌ Output files missing or corrupted
|
||||
- ❌ Subagent generated passing tests (wrong - must be failing)
|
||||
|
||||
---
|
||||
|
||||
## 🚨 SYSTEM SUCCESS/FAILURE METRICS:
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- Both subagents launched successfully
|
||||
- Both worker steps completed without errors
|
||||
- Output files generated and valid
|
||||
- Tests generated with test.skip() (TDD red phase)
|
||||
- Fallback behavior respected configuration and capability probe rules
|
||||
|
||||
### ❌ SYSTEM FAILURE:
|
||||
|
||||
- Failed to launch subagents
|
||||
- One or both subagents failed
|
||||
- Output files missing or invalid
|
||||
- Tests generated without test.skip() (wrong phase)
|
||||
- Unsupported requested mode with probing disabled
|
||||
|
||||
**Master Rule:** TDD RED PHASE requires FAILING tests (with test.skip()). Mode selection changes orchestration, never red-phase requirements.
|
||||
@@ -0,0 +1,286 @@
|
||||
---
|
||||
name: 'step-04a-subagent-api-failing'
|
||||
description: 'Subagent: Generate FAILING API tests (TDD red phase)'
|
||||
subagent: true
|
||||
outputFile: '/tmp/tea-atdd-api-tests-{{timestamp}}.json'
|
||||
---
|
||||
|
||||
# Subagent 4A: Generate Failing API Tests (TDD Red Phase)
|
||||
|
||||
## SUBAGENT CONTEXT
|
||||
|
||||
This is an **isolated subagent** running in parallel with E2E failing test generation.
|
||||
|
||||
**What you have from parent workflow:**
|
||||
|
||||
- Story acceptance criteria from Step 1
|
||||
- Test strategy and scenarios from Step 3
|
||||
- Knowledge fragments loaded: api-request, data-factories, api-testing-patterns
|
||||
- Config: test framework, Playwright Utils enabled/disabled, Pact.js Utils enabled/disabled (`use_pactjs_utils`), Pact MCP mode (`pact_mcp`)
|
||||
- Provider Endpoint Map (if `use_pactjs_utils` enabled and provider source accessible)
|
||||
|
||||
**Your task:** Generate API tests that will FAIL because the feature is not implemented yet (TDD RED PHASE).
|
||||
|
||||
---
|
||||
|
||||
## MANDATORY EXECUTION RULES
|
||||
|
||||
- 📖 Read this entire subagent file before acting
|
||||
- ✅ Generate FAILING API tests ONLY
|
||||
- ✅ Tests MUST fail when run (feature not implemented yet)
|
||||
- ✅ Output structured JSON to temp file
|
||||
- ✅ Follow knowledge fragment patterns
|
||||
- ❌ Do NOT generate E2E tests (that's subagent 4B)
|
||||
- ❌ Do NOT generate passing tests (this is TDD red phase)
|
||||
- ❌ Do NOT run tests (that's step 5)
|
||||
|
||||
---
|
||||
|
||||
## SUBAGENT TASK
|
||||
|
||||
### 1. Identify API Endpoints from Acceptance Criteria
|
||||
|
||||
From the story acceptance criteria (Step 1 output), identify:
|
||||
|
||||
- Which API endpoints will be created for this story
|
||||
- Expected request/response contracts
|
||||
- Authentication requirements
|
||||
- Expected status codes and error scenarios
|
||||
|
||||
**Example Acceptance Criteria:**
|
||||
|
||||
```
|
||||
Story: User Registration
|
||||
- As a user, I can POST to /api/users/register with email and password
|
||||
- System returns 201 Created with user object
|
||||
- System returns 400 Bad Request if email already exists
|
||||
- System returns 422 Unprocessable Entity if validation fails
|
||||
```
|
||||
|
||||
### 2. Generate FAILING API Test Files
|
||||
|
||||
For each API endpoint, create test file in `tests/api/[feature].spec.ts`:
|
||||
|
||||
**Test Structure (ATDD - Red Phase):**
|
||||
|
||||
```typescript
|
||||
import { test, expect } from '@playwright/test';
|
||||
// If Playwright Utils enabled:
|
||||
// import { apiRequest } from '@playwright-utils/api';
|
||||
|
||||
test.describe('[Story Name] API Tests (ATDD)', () => {
|
||||
test.skip('[P0] should register new user successfully', async ({ request }) => {
|
||||
// THIS TEST WILL FAIL - Endpoint not implemented yet
|
||||
const response = await request.post('/api/users/register', {
|
||||
data: {
|
||||
email: 'newuser@example.com',
|
||||
password: 'SecurePass123!',
|
||||
},
|
||||
});
|
||||
|
||||
// Expect 201 but will get 404 (endpoint doesn't exist)
|
||||
expect(response.status()).toBe(201);
|
||||
|
||||
const user = await response.json();
|
||||
expect(user).toMatchObject({
|
||||
id: expect.any(Number),
|
||||
email: 'newuser@example.com',
|
||||
});
|
||||
});
|
||||
|
||||
test.skip('[P1] should return 400 if email exists', async ({ request }) => {
|
||||
// THIS TEST WILL FAIL - Endpoint not implemented yet
|
||||
const response = await request.post('/api/users/register', {
|
||||
data: {
|
||||
email: 'existing@example.com',
|
||||
password: 'SecurePass123!',
|
||||
},
|
||||
});
|
||||
|
||||
expect(response.status()).toBe(400);
|
||||
const error = await response.json();
|
||||
expect(error.message).toContain('Email already exists');
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
**CRITICAL ATDD Requirements:**
|
||||
|
||||
- ✅ Use `test.skip()` to mark tests as intentionally failing (red phase)
|
||||
- ✅ Write assertions for EXPECTED behavior (even though not implemented)
|
||||
- ✅ Use realistic test data (not placeholder data)
|
||||
- ✅ Test both happy path and error scenarios from acceptance criteria
|
||||
- ✅ Use `apiRequest()` helper if Playwright Utils enabled
|
||||
- ✅ Use data factories for test data (from data-factories fragment)
|
||||
- ✅ Include priority tags [P0], [P1], [P2], [P3]
|
||||
|
||||
### 1.5 Provider Source Scrutiny for CDC in TDD Red Phase (If `use_pactjs_utils` Enabled)
|
||||
|
||||
When generating Pact consumer contract tests in the ATDD red phase, provider scrutiny applies with TDD-specific rules. Apply the **Seven-Point Scrutiny Checklist** from `contract-testing.md` (Response shape, Status codes, Field names, Enum values, Required fields, Data types, Nested structures) for both existing and new endpoints.
|
||||
|
||||
**If provider endpoint already exists** (extending an existing API):
|
||||
|
||||
- READ the provider route handler, types, and validation schemas
|
||||
- Verify all seven scrutiny points against the provider source: Response shape, Status codes, Field names, Enum values, Required fields, Data types, Nested structures
|
||||
- Add `// Provider endpoint:` comment and scrutiny evidence block documenting findings for each point
|
||||
- Wrap the entire test function in `test.skip()` (so the whole test including `executeTest` is skipped), not just the callback
|
||||
|
||||
**If provider endpoint is new** (TDD — endpoint not implemented yet):
|
||||
|
||||
- Use acceptance criteria as the source of truth for expected behavior
|
||||
- Acceptance criteria should specify all seven scrutiny points where possible (status codes, field names, types, etc.) — note any gaps as assumptions in the evidence block
|
||||
- Add `// Provider endpoint: TODO — new endpoint, not yet implemented`
|
||||
- Document expected behavior from acceptance criteria in scrutiny evidence block
|
||||
- Wrap the entire test function in `test.skip()` and use realistic expectations from the story
|
||||
|
||||
**Graceful degradation when provider source is inaccessible:**
|
||||
|
||||
1. **OpenAPI/Swagger spec available**: Use the spec as the source of truth for response shapes, status codes, and field names
|
||||
2. **Pact Broker available** (when `pact_mcp` is `"mcp"`): Use SmartBear MCP tools to fetch existing provider states and verified interactions as reference
|
||||
3. **Neither available**: For new endpoints, use acceptance criteria; for existing endpoints, use consumer-side types. Mark with `// Provider endpoint: TODO — provider source not accessible, verify manually` and set `provider_scrutiny: "pending"` in output JSON
|
||||
4. **Never silently guess**: Document all assumptions in the scrutiny evidence block
|
||||
|
||||
**Provider endpoint comments are MANDATORY** even in red-phase tests — they document the intent.
|
||||
|
||||
**Example: Red-phase Pact test with provider scrutiny:**
|
||||
|
||||
```typescript
|
||||
// Provider endpoint: TODO — new endpoint, not yet implemented
|
||||
/*
|
||||
* Provider Scrutiny Evidence:
|
||||
* - Handler: NEW — not yet implemented (TDD red phase)
|
||||
* - Expected from acceptance criteria:
|
||||
* - Endpoint: POST /api/v2/users/register
|
||||
* - Status: 201 for success, 400 for duplicate email, 422 for validation error
|
||||
* - Response: { id: number, email: string, createdAt: string }
|
||||
*/
|
||||
test.skip('[P0] should generate consumer contract for user registration', async () => {
|
||||
await provider
|
||||
.given('no users exist')
|
||||
.uponReceiving('a request to register a new user')
|
||||
.withRequest({
|
||||
method: 'POST',
|
||||
path: '/api/v2/users/register',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: { email: 'newuser@example.com', password: 'SecurePass123!' },
|
||||
})
|
||||
.willRespondWith({
|
||||
status: 201,
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: like({
|
||||
id: integer(1),
|
||||
email: string('newuser@example.com'),
|
||||
createdAt: string('2025-01-15T10:00:00Z'),
|
||||
}),
|
||||
})
|
||||
.executeTest(async (mockServer) => {
|
||||
const result = await registerUser({ email: 'newuser@example.com', password: 'SecurePass123!' }, { baseUrl: mockServer.url });
|
||||
expect(result.id).toEqual(expect.any(Number));
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
**Why test.skip():**
|
||||
|
||||
- Tests are written correctly for EXPECTED behavior
|
||||
- But we know they'll fail because feature isn't implemented
|
||||
- `test.skip()` documents this is intentional (TDD red phase)
|
||||
- Once feature is implemented, remove `test.skip()` to verify green phase
|
||||
|
||||
### 3. Track Fixture Needs
|
||||
|
||||
Identify fixtures needed for API tests:
|
||||
|
||||
- Authentication fixtures (if endpoints require auth)
|
||||
- Data factories (user data, etc.)
|
||||
- API client configurations
|
||||
|
||||
**Do NOT create fixtures yet** - just track what's needed for aggregation step.
|
||||
|
||||
---
|
||||
|
||||
## OUTPUT FORMAT
|
||||
|
||||
Write JSON to temp file: `/tmp/tea-atdd-api-tests-{{timestamp}}.json`
|
||||
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"subagent": "atdd-api-tests",
|
||||
"tests": [
|
||||
{
|
||||
"file": "tests/api/user-registration.spec.ts",
|
||||
"content": "[full TypeScript test file content with test.skip()]",
|
||||
"description": "ATDD API tests for user registration (RED PHASE)",
|
||||
"expected_to_fail": true,
|
||||
"acceptance_criteria_covered": [
|
||||
"User can register with email/password",
|
||||
"System returns 201 on success",
|
||||
"System returns 400 if email exists"
|
||||
],
|
||||
"priority_coverage": {
|
||||
"P0": 1,
|
||||
"P1": 2,
|
||||
"P2": 0,
|
||||
"P3": 0
|
||||
}
|
||||
}
|
||||
],
|
||||
"fixture_needs": ["userDataFactory"],
|
||||
"knowledge_fragments_used": ["api-request", "data-factories", "api-testing-patterns"],
|
||||
"test_count": 3,
|
||||
"tdd_phase": "RED",
|
||||
"provider_scrutiny": "completed",
|
||||
"summary": "Generated 3 FAILING API tests for user registration story"
|
||||
}
|
||||
```
|
||||
|
||||
**On Error:**
|
||||
|
||||
```json
|
||||
{
|
||||
"success": false,
|
||||
"subagent": "atdd-api-tests",
|
||||
"error": "Error message describing what went wrong",
|
||||
"partial_output": {
|
||||
/* any tests generated before error */
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## EXIT CONDITION
|
||||
|
||||
Subagent completes when:
|
||||
|
||||
- ✅ All API endpoints from acceptance criteria have test files
|
||||
- ✅ All tests use `test.skip()` (documented failing tests)
|
||||
- ✅ All tests assert EXPECTED behavior (not placeholder assertions)
|
||||
- ✅ JSON output written to temp file
|
||||
- ✅ Fixture needs to be tracked
|
||||
|
||||
**Subagent terminates here.** Parent workflow will read output and proceed to aggregation.
|
||||
|
||||
---
|
||||
|
||||
## 🚨 SUBAGENT SUCCESS METRICS
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- All API tests generated with test.skip()
|
||||
- Tests assert expected behavior (not placeholders)
|
||||
- JSON output valid and complete
|
||||
- No E2E/component/unit tests included (out of scope)
|
||||
- Tests follow knowledge fragment patterns
|
||||
- Every Pact interaction has `// Provider endpoint:` comment (if CDC enabled)
|
||||
- Provider scrutiny completed or TODO markers added for new endpoints (if CDC enabled)
|
||||
|
||||
### ❌ FAILURE:
|
||||
|
||||
- Generated passing tests (wrong - this is RED phase)
|
||||
- Tests without test.skip() (will break CI)
|
||||
- Placeholder assertions (expect(true).toBe(true))
|
||||
- Did not follow knowledge fragment patterns
|
||||
- Invalid or missing JSON output
|
||||
- Pact interactions missing provider endpoint comments (if CDC enabled)
|
||||
@@ -0,0 +1,244 @@
|
||||
---
|
||||
name: 'step-04b-subagent-e2e-failing'
|
||||
description: 'Subagent: Generate FAILING E2E tests (TDD red phase)'
|
||||
subagent: true
|
||||
outputFile: '/tmp/tea-atdd-e2e-tests-{{timestamp}}.json'
|
||||
---
|
||||
|
||||
# Subagent 4B: Generate Failing E2E Tests (TDD Red Phase)
|
||||
|
||||
## SUBAGENT CONTEXT
|
||||
|
||||
This is an **isolated subagent** running in parallel with API failing test generation.
|
||||
|
||||
**What you have from parent workflow:**
|
||||
|
||||
- Story acceptance criteria from Step 1
|
||||
- Test strategy and user journey scenarios from Step 3
|
||||
- Knowledge fragments loaded: fixture-architecture, network-first, selector-resilience
|
||||
- Config: test framework, Playwright Utils enabled/disabled
|
||||
|
||||
**Your task:** Generate E2E tests that will FAIL because the feature UI is not implemented yet (TDD RED PHASE).
|
||||
|
||||
---
|
||||
|
||||
## MANDATORY EXECUTION RULES
|
||||
|
||||
- 📖 Read this entire subagent file before acting
|
||||
- ✅ Generate FAILING E2E tests ONLY
|
||||
- ✅ Tests MUST fail when run (UI not implemented yet)
|
||||
- ✅ Output structured JSON to temp file
|
||||
- ✅ Follow knowledge fragment patterns
|
||||
- ❌ Do NOT generate API tests (that's subagent 4A)
|
||||
- ❌ Do NOT generate passing tests (this is TDD red phase)
|
||||
- ❌ Do NOT run tests (that's step 5)
|
||||
|
||||
---
|
||||
|
||||
## SUBAGENT TASK
|
||||
|
||||
### 1. Identify User Journeys from Acceptance Criteria
|
||||
|
||||
From the story acceptance criteria (Step 1 output), identify:
|
||||
|
||||
- Which UI flows will be created for this story
|
||||
- User interactions required
|
||||
- Expected visual states
|
||||
- Success/error messages expected
|
||||
|
||||
**Example Acceptance Criteria:**
|
||||
|
||||
```
|
||||
Story: User Registration
|
||||
- As a user, I can navigate to /register page
|
||||
- I can fill in email and password fields
|
||||
- I can click "Register" button
|
||||
- System shows success message and redirects to dashboard
|
||||
- System shows error if email already exists
|
||||
```
|
||||
|
||||
### 2. Browser Interaction (Selector Verification)
|
||||
|
||||
**Automation mode:** `config.tea_browser_automation`
|
||||
|
||||
If `auto` (fall back to MCP if CLI unavailable; if neither available, generate from best practices):
|
||||
|
||||
- Open the target page first, then verify selectors with a snapshot:
|
||||
`playwright-cli -s=tea-atdd-{{timestamp}} open <target_url>`
|
||||
`playwright-cli -s=tea-atdd-{{timestamp}} snapshot` → map refs to Playwright locators
|
||||
- ref `{role: "button", name: "Submit"}` → `page.getByRole('button', { name: 'Submit' })`
|
||||
- ref `{role: "textbox", name: "Email"}` → `page.getByRole('textbox', { name: 'Email' })`
|
||||
- `playwright-cli -s=tea-atdd-{{timestamp}} close` when done
|
||||
|
||||
If `cli` (CLI only — do NOT fall back to MCP; generate from best practices if CLI unavailable):
|
||||
|
||||
- Open the target page first, then verify selectors with a snapshot:
|
||||
`playwright-cli -s=tea-atdd-{{timestamp}} open <target_url>`
|
||||
`playwright-cli -s=tea-atdd-{{timestamp}} snapshot` → map refs to Playwright locators
|
||||
- ref `{role: "button", name: "Submit"}` → `page.getByRole('button', { name: 'Submit' })`
|
||||
- ref `{role: "textbox", name: "Email"}` → `page.getByRole('textbox', { name: 'Email' })`
|
||||
- `playwright-cli -s=tea-atdd-{{timestamp}} close` when done
|
||||
|
||||
> **Session Hygiene:** Always close sessions using `playwright-cli -s=tea-atdd-{{timestamp}} close`. Do NOT use `close-all` — it kills every session on the machine and breaks parallel execution.
|
||||
|
||||
If `mcp`:
|
||||
|
||||
- Use MCP tools for selector verification (current behavior)
|
||||
|
||||
If `none`:
|
||||
|
||||
- Generate selectors from best practices without browser verification
|
||||
|
||||
### 3. Generate FAILING E2E Test Files
|
||||
|
||||
For each user journey, create test file in `tests/e2e/[feature].spec.ts`:
|
||||
|
||||
**Test Structure (ATDD - Red Phase):**
|
||||
|
||||
```typescript
|
||||
import { test, expect } from '@playwright/test';
|
||||
|
||||
test.describe('[Story Name] E2E User Journey (ATDD)', () => {
|
||||
test.skip('[P0] should complete user registration successfully', async ({ page }) => {
|
||||
// THIS TEST WILL FAIL - UI not implemented yet
|
||||
await page.goto('/register');
|
||||
|
||||
// Expect registration form but will get 404 or missing elements
|
||||
await page.fill('[name="email"]', 'newuser@example.com');
|
||||
await page.fill('[name="password"]', 'SecurePass123!');
|
||||
await page.click('button:has-text("Register")');
|
||||
|
||||
// Expect success message and redirect
|
||||
await expect(page.getByText('Registration successful!')).toBeVisible();
|
||||
await page.waitForURL('/dashboard');
|
||||
});
|
||||
|
||||
test.skip('[P1] should show error if email exists', async ({ page }) => {
|
||||
// THIS TEST WILL FAIL - UI not implemented yet
|
||||
await page.goto('/register');
|
||||
|
||||
await page.fill('[name="email"]', 'existing@example.com');
|
||||
await page.fill('[name="password"]', 'SecurePass123!');
|
||||
await page.click('button:has-text("Register")');
|
||||
|
||||
// Expect error message
|
||||
await expect(page.getByText('Email already exists')).toBeVisible();
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
**CRITICAL ATDD Requirements:**
|
||||
|
||||
- ✅ Use `test.skip()` to mark tests as intentionally failing (red phase)
|
||||
- ✅ Write assertions for EXPECTED UI behavior (even though not implemented)
|
||||
- ✅ Use resilient selectors: getByRole, getByText, getByLabel (from selector-resilience)
|
||||
- ✅ Follow network-first patterns if API calls involved (from network-first)
|
||||
- ✅ Test complete user journeys from acceptance criteria
|
||||
- ✅ Include priority tags [P0], [P1], [P2], [P3]
|
||||
- ✅ Use proper TypeScript types
|
||||
- ✅ Deterministic waits (no hard sleeps)
|
||||
|
||||
**Why test.skip():**
|
||||
|
||||
- Tests are written correctly for EXPECTED UI behavior
|
||||
- But we know they'll fail because UI isn't implemented
|
||||
- `test.skip()` documents this is intentional (TDD red phase)
|
||||
- Once UI is implemented, remove `test.skip()` to verify green phase
|
||||
|
||||
### 4. Track Fixture Needs
|
||||
|
||||
Identify fixtures needed for E2E tests:
|
||||
|
||||
- Authentication fixtures (if journey requires logged-in state)
|
||||
- Network mocks (if API calls involved)
|
||||
- Test data fixtures
|
||||
|
||||
**Do NOT create fixtures yet** - just track what's needed for aggregation step.
|
||||
|
||||
---
|
||||
|
||||
## OUTPUT FORMAT
|
||||
|
||||
Write JSON to temp file: `/tmp/tea-atdd-e2e-tests-{{timestamp}}.json`
|
||||
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"subagent": "atdd-e2e-tests",
|
||||
"tests": [
|
||||
{
|
||||
"file": "tests/e2e/user-registration.spec.ts",
|
||||
"content": "[full TypeScript test file content with test.skip()]",
|
||||
"description": "ATDD E2E tests for user registration journey (RED PHASE)",
|
||||
"expected_to_fail": true,
|
||||
"acceptance_criteria_covered": [
|
||||
"User can navigate to /register",
|
||||
"User can fill registration form",
|
||||
"System shows success message on registration",
|
||||
"System shows error if email exists"
|
||||
],
|
||||
"priority_coverage": {
|
||||
"P0": 1,
|
||||
"P1": 1,
|
||||
"P2": 0,
|
||||
"P3": 0
|
||||
}
|
||||
}
|
||||
],
|
||||
"fixture_needs": ["registrationPageMock"],
|
||||
"knowledge_fragments_used": ["fixture-architecture", "network-first", "selector-resilience"],
|
||||
"test_count": 2,
|
||||
"tdd_phase": "RED",
|
||||
"summary": "Generated 2 FAILING E2E tests for user registration story"
|
||||
}
|
||||
```
|
||||
|
||||
**On Error:**
|
||||
|
||||
```json
|
||||
{
|
||||
"success": false,
|
||||
"subagent": "atdd-e2e-tests",
|
||||
"error": "Error message describing what went wrong",
|
||||
"partial_output": {
|
||||
/* any tests generated before error */
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## EXIT CONDITION
|
||||
|
||||
Subagent completes when:
|
||||
|
||||
- ✅ All user journeys from acceptance criteria have test files
|
||||
- ✅ All tests use `test.skip()` (documented failing tests)
|
||||
- ✅ All tests assert EXPECTED UI behavior (not placeholder assertions)
|
||||
- ✅ Resilient selectors used (getByRole, getByText)
|
||||
- ✅ JSON output written to temp file
|
||||
- ✅ Fixture needs tracked
|
||||
|
||||
**Subagent terminates here.** Parent workflow will read output and proceed to aggregation.
|
||||
|
||||
---
|
||||
|
||||
## 🚨 SUBAGENT SUCCESS METRICS
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- All E2E tests generated with test.skip()
|
||||
- Tests assert expected UI behavior (not placeholders)
|
||||
- Resilient selectors used (getByRole, getByText)
|
||||
- JSON output valid and complete
|
||||
- No API/component/unit tests included (out of scope)
|
||||
- Tests follow knowledge fragment patterns
|
||||
|
||||
### ❌ FAILURE:
|
||||
|
||||
- Generated passing tests (wrong - this is RED phase)
|
||||
- Tests without test.skip() (will break CI)
|
||||
- Placeholder assertions (expect(true).toBe(true))
|
||||
- Brittle selectors used (CSS classes, XPath)
|
||||
- Did not follow knowledge fragment patterns
|
||||
- Invalid or missing JSON output
|
||||
370
_bmad/tea/workflows/testarch/atdd/steps-c/step-04c-aggregate.md
Normal file
370
_bmad/tea/workflows/testarch/atdd/steps-c/step-04c-aggregate.md
Normal file
@@ -0,0 +1,370 @@
|
||||
---
|
||||
name: 'step-04c-aggregate'
|
||||
description: 'Aggregate subagent outputs and complete ATDD test infrastructure'
|
||||
outputFile: '{test_artifacts}/atdd-checklist-{story_id}.md'
|
||||
nextStepFile: './step-05-validate-and-complete.md'
|
||||
---
|
||||
|
||||
# Step 4C: Aggregate ATDD Test Generation Results
|
||||
|
||||
## STEP GOAL
|
||||
|
||||
Read outputs from parallel subagents (API + E2E failing test generation), aggregate results, verify TDD red phase compliance, and create supporting infrastructure.
|
||||
|
||||
---
|
||||
|
||||
## MANDATORY EXECUTION RULES
|
||||
|
||||
- 📖 Read the entire step file before acting
|
||||
- ✅ Speak in `{communication_language}`
|
||||
- ✅ Read subagent outputs from temp files
|
||||
- ✅ Verify all tests are marked with test.skip() (TDD red phase)
|
||||
- ✅ Generate shared fixtures based on fixture needs
|
||||
- ✅ Write all generated test files to disk
|
||||
- ❌ Do NOT remove test.skip() (that's done after feature implementation)
|
||||
- ❌ Do NOT run tests yet (that's step 5 - verify they fail)
|
||||
|
||||
---
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
- 💾 Record outputs before proceeding
|
||||
- 📖 Load the next step only when instructed
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: config, subagent outputs from temp files
|
||||
- Focus: aggregation and TDD validation
|
||||
- Limits: do not execute future steps
|
||||
- Dependencies: Step 4A and 4B subagent outputs
|
||||
|
||||
---
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly. Do not skip, reorder, or improvise.
|
||||
|
||||
### 1. Read Subagent Outputs
|
||||
|
||||
**Read API test subagent output:**
|
||||
|
||||
```javascript
|
||||
const apiTestsPath = '/tmp/tea-atdd-api-tests-{{timestamp}}.json';
|
||||
const apiTestsOutput = JSON.parse(fs.readFileSync(apiTestsPath, 'utf8'));
|
||||
```
|
||||
|
||||
**Read E2E test subagent output:**
|
||||
|
||||
```javascript
|
||||
const e2eTestsPath = '/tmp/tea-atdd-e2e-tests-{{timestamp}}.json';
|
||||
const e2eTestsOutput = JSON.parse(fs.readFileSync(e2eTestsPath, 'utf8'));
|
||||
```
|
||||
|
||||
**Verify both subagents succeeded:**
|
||||
|
||||
- Check `apiTestsOutput.success === true`
|
||||
- Check `e2eTestsOutput.success === true`
|
||||
- If either failed, report error and stop (don't proceed)
|
||||
|
||||
---
|
||||
|
||||
### 2. Verify TDD Red Phase Compliance
|
||||
|
||||
**CRITICAL TDD Validation:**
|
||||
|
||||
**Check API tests:**
|
||||
|
||||
```javascript
|
||||
apiTestsOutput.tests.forEach((test) => {
|
||||
// Verify test.skip() is present
|
||||
if (!test.content.includes('test.skip(')) {
|
||||
throw new Error(`ATDD ERROR: ${test.file} missing test.skip() - tests MUST be skipped in red phase!`);
|
||||
}
|
||||
|
||||
// Verify not placeholder assertions
|
||||
if (test.content.includes('expect(true).toBe(true)')) {
|
||||
throw new Error(`ATDD ERROR: ${test.file} has placeholder assertions - must assert EXPECTED behavior!`);
|
||||
}
|
||||
|
||||
// Verify expected_to_fail flag
|
||||
if (!test.expected_to_fail) {
|
||||
throw new Error(`ATDD ERROR: ${test.file} not marked as expected_to_fail!`);
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
**Check E2E tests:**
|
||||
|
||||
```javascript
|
||||
e2eTestsOutput.tests.forEach((test) => {
|
||||
// Same validation as API tests
|
||||
if (!test.content.includes('test.skip(')) {
|
||||
throw new Error(`ATDD ERROR: ${test.file} missing test.skip() - tests MUST be skipped in red phase!`);
|
||||
}
|
||||
|
||||
if (test.content.includes('expect(true).toBe(true)')) {
|
||||
throw new Error(`ATDD ERROR: ${test.file} has placeholder assertions!`);
|
||||
}
|
||||
|
||||
if (!test.expected_to_fail) {
|
||||
throw new Error(`ATDD ERROR: ${test.file} not marked as expected_to_fail!`);
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
**If validation passes:**
|
||||
|
||||
```
|
||||
✅ TDD Red Phase Validation: PASS
|
||||
- All tests use test.skip()
|
||||
- All tests assert expected behavior (not placeholders)
|
||||
- All tests marked as expected_to_fail
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 3. Write All Test Files to Disk
|
||||
|
||||
**Write API test files:**
|
||||
|
||||
```javascript
|
||||
apiTestsOutput.tests.forEach((test) => {
|
||||
fs.writeFileSync(test.file, test.content, 'utf8');
|
||||
console.log(`✅ Created (RED): ${test.file}`);
|
||||
});
|
||||
```
|
||||
|
||||
**Write E2E test files:**
|
||||
|
||||
```javascript
|
||||
e2eTestsOutput.tests.forEach((test) => {
|
||||
fs.writeFileSync(test.file, test.content, 'utf8');
|
||||
console.log(`✅ Created (RED): ${test.file}`);
|
||||
});
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 4. Aggregate Fixture Needs
|
||||
|
||||
**Collect all fixture needs from both subagents:**
|
||||
|
||||
```javascript
|
||||
const allFixtureNeeds = [...apiTestsOutput.fixture_needs, ...e2eTestsOutput.fixture_needs];
|
||||
|
||||
// Remove duplicates
|
||||
const uniqueFixtures = [...new Set(allFixtureNeeds)];
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 5. Generate Fixture Infrastructure
|
||||
|
||||
**Create fixtures needed by ATDD tests:**
|
||||
(Similar to automate workflow, but may be simpler for ATDD since feature not implemented)
|
||||
|
||||
**Minimal fixtures for TDD red phase:**
|
||||
|
||||
```typescript
|
||||
// tests/fixtures/test-data.ts
|
||||
export const testUserData = {
|
||||
email: 'test@example.com',
|
||||
password: 'SecurePass123!',
|
||||
};
|
||||
```
|
||||
|
||||
Note: More complete fixtures will be needed when moving to green phase.
|
||||
|
||||
---
|
||||
|
||||
### 6. Generate ATDD Checklist
|
||||
|
||||
**Create ATDD checklist document:**
|
||||
|
||||
```markdown
|
||||
# ATDD Checklist: [Story Name]
|
||||
|
||||
## TDD Red Phase (Current)
|
||||
|
||||
✅ Failing tests generated
|
||||
|
||||
- API Tests: {api_test_count} tests (all skipped)
|
||||
- E2E Tests: {e2e_test_count} tests (all skipped)
|
||||
|
||||
## Acceptance Criteria Coverage
|
||||
|
||||
{list all acceptance criteria with test coverage}
|
||||
|
||||
## Next Steps (TDD Green Phase)
|
||||
|
||||
After implementing the feature:
|
||||
|
||||
1. Remove `test.skip()` from all test files
|
||||
2. Run tests: `npm test`
|
||||
3. Verify tests PASS (green phase)
|
||||
4. If any tests fail:
|
||||
- Either fix implementation (feature bug)
|
||||
- Or fix test (test bug)
|
||||
5. Commit passing tests
|
||||
|
||||
## Implementation Guidance
|
||||
|
||||
Feature endpoints to implement:
|
||||
{list endpoints from API tests}
|
||||
|
||||
UI components to implement:
|
||||
{list UI flows from E2E tests}
|
||||
```
|
||||
|
||||
**Save checklist:**
|
||||
|
||||
```javascript
|
||||
fs.writeFileSync(`{test_artifacts}/atdd-checklist-{story-id}.md`, checklistContent, 'utf8');
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 7. Calculate Summary Statistics
|
||||
|
||||
**Aggregate test counts:**
|
||||
|
||||
```javascript
|
||||
const resolvedMode = subagentContext?.execution?.resolvedMode; // Provided by Step 4's orchestration context
|
||||
const subagentExecutionLabel =
|
||||
resolvedMode === 'sequential'
|
||||
? 'SEQUENTIAL (API → E2E)'
|
||||
: resolvedMode === 'agent-team'
|
||||
? 'AGENT-TEAM (API + E2E)'
|
||||
: resolvedMode === 'subagent'
|
||||
? 'SUBAGENT (API + E2E)'
|
||||
: 'PARALLEL (API + E2E)';
|
||||
const performanceGainLabel =
|
||||
resolvedMode === 'sequential'
|
||||
? 'baseline (no parallel speedup)'
|
||||
: resolvedMode === 'agent-team' || resolvedMode === 'subagent'
|
||||
? '~50% faster than sequential'
|
||||
: 'mode-dependent';
|
||||
|
||||
const summary = {
|
||||
tdd_phase: 'RED',
|
||||
total_tests: apiTestsOutput.test_count + e2eTestsOutput.test_count,
|
||||
api_tests: apiTestsOutput.test_count,
|
||||
e2e_tests: e2eTestsOutput.test_count,
|
||||
all_tests_skipped: true,
|
||||
expected_to_fail: true,
|
||||
fixtures_created: uniqueFixtures.length,
|
||||
acceptance_criteria_covered: [
|
||||
...apiTestsOutput.tests.flatMap((t) => t.acceptance_criteria_covered),
|
||||
...e2eTestsOutput.tests.flatMap((t) => t.acceptance_criteria_covered),
|
||||
],
|
||||
knowledge_fragments_used: [...apiTestsOutput.knowledge_fragments_used, ...e2eTestsOutput.knowledge_fragments_used],
|
||||
subagent_execution: subagentExecutionLabel,
|
||||
performance_gain: performanceGainLabel,
|
||||
};
|
||||
```
|
||||
|
||||
**Store summary for Step 5:**
|
||||
|
||||
```javascript
|
||||
fs.writeFileSync('/tmp/tea-atdd-summary-{{timestamp}}.json', JSON.stringify(summary, null, 2), 'utf8');
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## OUTPUT SUMMARY
|
||||
|
||||
Display to user:
|
||||
|
||||
```
|
||||
✅ ATDD Test Generation Complete (TDD RED PHASE)
|
||||
|
||||
🔴 TDD Red Phase: Failing Tests Generated
|
||||
|
||||
📊 Summary:
|
||||
- Total Tests: {total_tests} (all with test.skip())
|
||||
- API Tests: {api_tests} (RED)
|
||||
- E2E Tests: {e2e_tests} (RED)
|
||||
- Fixtures Created: {fixtures_created}
|
||||
- All tests will FAIL until feature implemented
|
||||
|
||||
✅ Acceptance Criteria Coverage:
|
||||
{list all covered criteria}
|
||||
|
||||
🚀 Performance: {performance_gain}
|
||||
|
||||
📂 Generated Files:
|
||||
- tests/api/[feature].spec.ts (with test.skip())
|
||||
- tests/e2e/[feature].spec.ts (with test.skip())
|
||||
- tests/fixtures/test-data.ts
|
||||
- {test_artifacts}/atdd-checklist-{story-id}.md
|
||||
|
||||
📝 Next Steps:
|
||||
1. Implement the feature
|
||||
2. Remove test.skip() from tests
|
||||
3. Run tests → verify PASS (green phase)
|
||||
4. Commit passing tests
|
||||
|
||||
✅ Ready for validation (Step 5 - verify tests fail as expected)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## EXIT CONDITION
|
||||
|
||||
Proceed to Step 5 when:
|
||||
|
||||
- ✅ All test files written to disk (API + E2E)
|
||||
- ✅ All tests verified to have test.skip()
|
||||
- ✅ All fixtures created
|
||||
- ✅ ATDD checklist generated
|
||||
- ✅ Summary statistics calculated and saved
|
||||
- ✅ Output displayed to user
|
||||
|
||||
---
|
||||
|
||||
### 8. Save Progress
|
||||
|
||||
**Save this step's accumulated work to `{outputFile}`.**
|
||||
|
||||
- **If `{outputFile}` does not exist** (first save), create it with YAML frontmatter:
|
||||
|
||||
```yaml
|
||||
---
|
||||
stepsCompleted: ['step-04c-aggregate']
|
||||
lastStep: 'step-04c-aggregate'
|
||||
lastSaved: '{date}'
|
||||
---
|
||||
```
|
||||
|
||||
Then write this step's output below the frontmatter.
|
||||
|
||||
- **If `{outputFile}` already exists**, update:
|
||||
- Add `'step-04c-aggregate'` to `stepsCompleted` array (only if not already present)
|
||||
- Set `lastStep: 'step-04c-aggregate'`
|
||||
- Set `lastSaved: '{date}'`
|
||||
- Append this step's output to the appropriate section.
|
||||
|
||||
Load next step: `{nextStepFile}`
|
||||
|
||||
---
|
||||
|
||||
## 🚨 SYSTEM SUCCESS/FAILURE METRICS:
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- Both subagents succeeded
|
||||
- All tests have test.skip() (TDD red phase compliant)
|
||||
- All tests assert expected behavior (not placeholders)
|
||||
- All test files written to disk
|
||||
- ATDD checklist generated
|
||||
|
||||
### ❌ SYSTEM FAILURE:
|
||||
|
||||
- One or both subagents failed
|
||||
- Tests missing test.skip() (would break CI)
|
||||
- Tests have placeholder assertions
|
||||
- Test files not written to disk
|
||||
- ATDD checklist missing
|
||||
|
||||
**Master Rule:** TDD RED PHASE requires ALL tests to use test.skip() and assert expected behavior.
|
||||
@@ -0,0 +1,106 @@
|
||||
---
|
||||
name: 'step-05-validate-and-complete'
|
||||
description: 'Validate ATDD outputs and summarize'
|
||||
outputFile: '{test_artifacts}/atdd-checklist-{story_id}.md'
|
||||
---
|
||||
|
||||
# Step 5: Validate & Complete
|
||||
|
||||
## STEP GOAL
|
||||
|
||||
Validate ATDD outputs and provide a completion summary.
|
||||
|
||||
## MANDATORY EXECUTION RULES
|
||||
|
||||
- 📖 Read the entire step file before acting
|
||||
- ✅ Speak in `{communication_language}`
|
||||
- ✅ Validate against the checklist
|
||||
|
||||
---
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
- 💾 Record outputs before proceeding
|
||||
- 📖 Load the next step only when instructed
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: config, loaded artifacts, and knowledge fragments
|
||||
- Focus: this step's goal only
|
||||
- Limits: do not execute future steps
|
||||
- Dependencies: prior steps' outputs (if any)
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly. Do not skip, reorder, or improvise.
|
||||
|
||||
## 1. Validation
|
||||
|
||||
Use `checklist.md` to validate:
|
||||
|
||||
- Prerequisites satisfied
|
||||
- Test files created correctly
|
||||
- Checklist matches acceptance criteria
|
||||
- Tests are designed to fail before implementation
|
||||
- [ ] CLI sessions cleaned up (no orphaned browsers)
|
||||
- [ ] Temp artifacts stored in `{test_artifacts}/` not random locations
|
||||
|
||||
Fix any gaps before completion.
|
||||
|
||||
---
|
||||
|
||||
## 2. Polish Output
|
||||
|
||||
Before finalizing, review the complete output document for quality:
|
||||
|
||||
1. **Remove duplication**: Progressive-append workflow may have created repeated sections — consolidate
|
||||
2. **Verify consistency**: Ensure terminology, risk scores, and references are consistent throughout
|
||||
3. **Check completeness**: All template sections should be populated or explicitly marked N/A
|
||||
4. **Format cleanup**: Ensure markdown formatting is clean (tables aligned, headers consistent, no orphaned references)
|
||||
|
||||
---
|
||||
|
||||
## 3. Completion Summary
|
||||
|
||||
Report:
|
||||
|
||||
- Test files created
|
||||
- Checklist output path
|
||||
- Key risks or assumptions
|
||||
- Next recommended workflow (e.g., implementation or `automate`)
|
||||
|
||||
---
|
||||
|
||||
## 4. Save Progress
|
||||
|
||||
**Save this step's accumulated work to `{outputFile}`.**
|
||||
|
||||
- **If `{outputFile}` does not exist** (first save), create it with YAML frontmatter:
|
||||
|
||||
```yaml
|
||||
---
|
||||
stepsCompleted: ['step-05-validate-and-complete']
|
||||
lastStep: 'step-05-validate-and-complete'
|
||||
lastSaved: '{date}'
|
||||
---
|
||||
```
|
||||
|
||||
Then write this step's output below the frontmatter.
|
||||
|
||||
- **If `{outputFile}` already exists**, update:
|
||||
- Add `'step-05-validate-and-complete'` to `stepsCompleted` array (only if not already present)
|
||||
- Set `lastStep: 'step-05-validate-and-complete'`
|
||||
- Set `lastSaved: '{date}'`
|
||||
- Append this step's output to the appropriate section.
|
||||
|
||||
## 🚨 SYSTEM SUCCESS/FAILURE METRICS:
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- Step completed in full with required outputs
|
||||
|
||||
### ❌ SYSTEM FAILURE:
|
||||
|
||||
- Skipped sequence steps or missing outputs
|
||||
**Master Rule:** Skipping steps is FORBIDDEN.
|
||||
65
_bmad/tea/workflows/testarch/atdd/steps-e/step-01-assess.md
Normal file
65
_bmad/tea/workflows/testarch/atdd/steps-e/step-01-assess.md
Normal file
@@ -0,0 +1,65 @@
|
||||
---
|
||||
name: 'step-01-assess'
|
||||
description: 'Load an existing output for editing'
|
||||
nextStepFile: './step-02-apply-edit.md'
|
||||
---
|
||||
|
||||
# Step 1: Assess Edit Target
|
||||
|
||||
## STEP GOAL:
|
||||
|
||||
Identify which output should be edited and load it.
|
||||
|
||||
## MANDATORY EXECUTION RULES (READ FIRST):
|
||||
|
||||
### Universal Rules:
|
||||
|
||||
- 📖 Read the complete step file before taking any action
|
||||
- ✅ Speak in `{communication_language}`
|
||||
|
||||
### Role Reinforcement:
|
||||
|
||||
- ✅ You are the Master Test Architect
|
||||
|
||||
### Step-Specific Rules:
|
||||
|
||||
- 🎯 Ask the user which output file to edit
|
||||
- 🚫 Do not edit until target is confirmed
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: existing outputs
|
||||
- Focus: select edit target
|
||||
- Limits: no edits yet
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly.
|
||||
|
||||
### 1. Identify Target
|
||||
|
||||
Ask the user to provide the output file path or select from known outputs.
|
||||
|
||||
### 2. Load Target
|
||||
|
||||
Read the provided output file in full.
|
||||
|
||||
### 3. Confirm
|
||||
|
||||
Confirm the target and proceed to edit.
|
||||
|
||||
Load next step: `{nextStepFile}`
|
||||
|
||||
## 🚨 SYSTEM SUCCESS/FAILURE METRICS:
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- Target identified and loaded
|
||||
|
||||
### ❌ SYSTEM FAILURE:
|
||||
|
||||
- Proceeding without a confirmed target
|
||||
@@ -0,0 +1,60 @@
|
||||
---
|
||||
name: 'step-02-apply-edit'
|
||||
description: 'Apply edits to the selected output'
|
||||
---
|
||||
|
||||
# Step 2: Apply Edits
|
||||
|
||||
## STEP GOAL:
|
||||
|
||||
Apply the requested edits to the selected output and confirm changes.
|
||||
|
||||
## MANDATORY EXECUTION RULES (READ FIRST):
|
||||
|
||||
### Universal Rules:
|
||||
|
||||
- 📖 Read the complete step file before taking any action
|
||||
- ✅ Speak in `{communication_language}`
|
||||
|
||||
### Role Reinforcement:
|
||||
|
||||
- ✅ You are the Master Test Architect
|
||||
|
||||
### Step-Specific Rules:
|
||||
|
||||
- 🎯 Only apply edits explicitly requested by the user
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: selected output and user changes
|
||||
- Focus: apply edits only
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly.
|
||||
|
||||
### 1. Confirm Requested Changes
|
||||
|
||||
Restate what will be changed and confirm.
|
||||
|
||||
### 2. Apply Changes
|
||||
|
||||
Update the output file accordingly.
|
||||
|
||||
### 3. Report
|
||||
|
||||
Summarize the edits applied.
|
||||
|
||||
## 🚨 SYSTEM SUCCESS/FAILURE METRICS:
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- Changes applied and confirmed
|
||||
|
||||
### ❌ SYSTEM FAILURE:
|
||||
|
||||
- Unconfirmed edits or missing update
|
||||
@@ -0,0 +1,67 @@
|
||||
---
|
||||
name: 'step-01-validate'
|
||||
description: 'Validate workflow outputs against checklist'
|
||||
outputFile: '{test_artifacts}/atdd-validation-report.md'
|
||||
validationChecklist: '../checklist.md'
|
||||
---
|
||||
|
||||
# Step 1: Validate Outputs
|
||||
|
||||
## STEP GOAL:
|
||||
|
||||
Validate outputs using the workflow checklist and record findings.
|
||||
|
||||
## MANDATORY EXECUTION RULES (READ FIRST):
|
||||
|
||||
### Universal Rules:
|
||||
|
||||
- 📖 Read the complete step file before taking any action
|
||||
- ✅ Speak in `{communication_language}`
|
||||
|
||||
### Role Reinforcement:
|
||||
|
||||
- ✅ You are the Master Test Architect
|
||||
|
||||
### Step-Specific Rules:
|
||||
|
||||
- 🎯 Validate against `{validationChecklist}`
|
||||
- 🚫 Do not skip checks
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
- 💾 Write findings to `{outputFile}`
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: workflow outputs and checklist
|
||||
- Focus: validation only
|
||||
- Limits: do not modify outputs in this step
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly.
|
||||
|
||||
### 1. Load Checklist
|
||||
|
||||
Read `{validationChecklist}` and list all criteria.
|
||||
|
||||
### 2. Validate Outputs
|
||||
|
||||
Evaluate outputs against each checklist item.
|
||||
|
||||
### 3. Write Report
|
||||
|
||||
Write a validation report to `{outputFile}` with PASS/WARN/FAIL per section.
|
||||
|
||||
## 🚨 SYSTEM SUCCESS/FAILURE METRICS:
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- Validation report written
|
||||
- All checklist items evaluated
|
||||
|
||||
### ❌ SYSTEM FAILURE:
|
||||
|
||||
- Skipped checklist items
|
||||
- No report produced
|
||||
@@ -0,0 +1,73 @@
|
||||
---
|
||||
validationDate: 2026-01-27
|
||||
workflowName: testarch-atdd
|
||||
workflowPath: {project-root}/src/workflows/testarch/atdd
|
||||
validationStatus: COMPLETE
|
||||
completionDate: 2026-01-27 10:03:10
|
||||
---
|
||||
|
||||
# Validation Report: testarch-atdd
|
||||
|
||||
**Validation Started:** 2026-01-27 09:50:21
|
||||
**Validator:** BMAD Workflow Validation System (Codex)
|
||||
**Standards Version:** BMAD Workflow Standards
|
||||
|
||||
## File Structure & Size
|
||||
|
||||
- workflow.md present: YES
|
||||
- instructions.md present: YES
|
||||
- workflow.yaml present: YES
|
||||
- step files found: 8
|
||||
|
||||
**Step File Sizes:**
|
||||
|
||||
- steps-c/step-01-preflight-and-context.md: 101 lines [GOOD]
|
||||
- steps-c/step-02-generation-mode.md: 71 lines [GOOD]
|
||||
- steps-c/step-03-test-strategy.md: 70 lines [GOOD]
|
||||
- steps-c/step-04-generate-tests.md: 70 lines [GOOD]
|
||||
- steps-c/step-05-validate-and-complete.md: 61 lines [GOOD]
|
||||
- steps-e/step-01-assess.md: 51 lines [GOOD]
|
||||
- steps-e/step-02-apply-edit.md: 46 lines [GOOD]
|
||||
- steps-v/step-01-validate.md: 53 lines [GOOD]
|
||||
- workflow-plan.md present: YES
|
||||
|
||||
## Frontmatter Validation
|
||||
|
||||
- No frontmatter violations found
|
||||
|
||||
## Critical Path Violations
|
||||
|
||||
- No {project-root} hardcoded paths detected in body
|
||||
- No dead relative links detected
|
||||
|
||||
## Menu Handling Validation
|
||||
|
||||
- No menu structures detected (linear step flow) [N/A]
|
||||
|
||||
## Step Type Validation
|
||||
|
||||
- Last step steps-v/step-01-validate.md has no nextStepFile (final step OK)
|
||||
- Step type validation assumes linear sequence (no branching/menu). Workflow-plan.md present for reference. [INFO]
|
||||
|
||||
## Output Format Validation
|
||||
|
||||
- Templates present: atdd-checklist-template.md
|
||||
- Steps with outputFile in frontmatter:
|
||||
- steps-c/step-04-generate-tests.md
|
||||
- steps-v/step-01-validate.md
|
||||
|
||||
## Validation Design Check
|
||||
|
||||
- checklist.md present: YES
|
||||
- Validation steps folder (steps-v) present: YES
|
||||
|
||||
## Instruction Style Check
|
||||
|
||||
- All steps include STEP GOAL, MANDATORY EXECUTION RULES, EXECUTION PROTOCOLS, CONTEXT BOUNDARIES, and SUCCESS/FAILURE metrics
|
||||
|
||||
## Summary
|
||||
|
||||
- Validation completed: 2026-01-27 10:03:10
|
||||
- Critical issues: 0
|
||||
- Warnings: 0 (informational notes only)
|
||||
- Readiness: READY (manual review optional)
|
||||
@@ -0,0 +1,116 @@
|
||||
---
|
||||
validationDate: 2026-01-27
|
||||
workflowName: testarch-atdd
|
||||
workflowPath: {project-root}/src/workflows/testarch/atdd
|
||||
validationStatus: COMPLETE
|
||||
completionDate: 2026-01-27 10:24:01
|
||||
---
|
||||
|
||||
# Validation Report: testarch-atdd
|
||||
|
||||
**Validation Started:** 2026-01-27 10:24:01
|
||||
**Validator:** BMAD Workflow Validation System (Codex)
|
||||
**Standards Version:** BMAD Workflow Standards
|
||||
|
||||
## File Structure & Size
|
||||
|
||||
- workflow.md present: YES
|
||||
- instructions.md present: YES
|
||||
- workflow.yaml present: YES
|
||||
- step files found: 8
|
||||
|
||||
**Step File Sizes:**
|
||||
|
||||
- steps-c/step-01-preflight-and-context.md: 100 lines [GOOD]
|
||||
- steps-c/step-02-generation-mode.md: 70 lines [GOOD]
|
||||
- steps-c/step-03-test-strategy.md: 69 lines [GOOD]
|
||||
- steps-c/step-04-generate-tests.md: 69 lines [GOOD]
|
||||
- steps-c/step-05-validate-and-complete.md: 60 lines [GOOD]
|
||||
- steps-e/step-01-assess.md: 50 lines [GOOD]
|
||||
- steps-e/step-02-apply-edit.md: 45 lines [GOOD]
|
||||
- steps-v/step-01-validate.md: 52 lines [GOOD]
|
||||
- workflow-plan.md present: YES
|
||||
|
||||
## Frontmatter Validation
|
||||
|
||||
- No frontmatter violations found
|
||||
|
||||
## Critical Path Violations
|
||||
|
||||
### Config Variables (Exceptions)
|
||||
|
||||
Standard BMAD config variables treated as valid exceptions: bmb_creations_output_folder, communication_language, document_output_language, output_folder, planning_artifacts, project-root, project_name, test_artifacts, user_name
|
||||
|
||||
- No {project-root} hardcoded paths detected in body
|
||||
|
||||
- No dead relative links detected
|
||||
|
||||
- No module path assumptions detected
|
||||
|
||||
**Status:** ✅ PASS - No critical violations
|
||||
|
||||
## Menu Handling Validation
|
||||
|
||||
- No menu structures detected (linear step flow) [N/A]
|
||||
|
||||
## Step Type Validation
|
||||
|
||||
- steps-c/step-01-preflight-and-context.md: Init [PASS]
|
||||
- steps-c/step-02-generation-mode.md: Middle [PASS]
|
||||
- steps-c/step-03-test-strategy.md: Middle [PASS]
|
||||
- steps-c/step-04-generate-tests.md: Middle [PASS]
|
||||
- steps-c/step-05-validate-and-complete.md: Final [PASS]
|
||||
- Step type validation assumes linear sequence (no branching/menu). Workflow-plan.md present for reference. [INFO]
|
||||
|
||||
## Output Format Validation
|
||||
|
||||
- Templates present: atdd-checklist-template.md
|
||||
- Steps with outputFile in frontmatter:
|
||||
- steps-c/step-04-generate-tests.md
|
||||
- steps-v/step-01-validate.md
|
||||
- checklist.md present: YES
|
||||
|
||||
## Validation Design Check
|
||||
|
||||
- Validation steps folder (steps-v) present: YES
|
||||
- Validation step(s) present: step-01-validate.md
|
||||
- Validation steps reference checklist data and auto-proceed
|
||||
|
||||
## Instruction Style Check
|
||||
|
||||
- Instruction style: Prescriptive (appropriate for TEA quality/compliance workflows)
|
||||
- Steps emphasize mandatory sequence, explicit success/failure metrics, and risk-based guidance
|
||||
|
||||
## Collaborative Experience Check
|
||||
|
||||
- Overall facilitation quality: GOOD
|
||||
- Steps use progressive prompts and clear role reinforcement; no laundry-list interrogation detected
|
||||
- Flow progression is clear and aligned to workflow goals
|
||||
|
||||
## Subagent Optimization Opportunities
|
||||
|
||||
- No high-priority subagent optimizations identified; workflow already uses step-file architecture
|
||||
- Pattern 1 (grep/regex): N/A for most steps
|
||||
- Pattern 2 (per-file analysis): already aligned to validation structure
|
||||
- Pattern 3 (data ops): minimal data file loads
|
||||
- Pattern 4 (parallel): optional for validation only
|
||||
|
||||
## Cohesive Review
|
||||
|
||||
- Overall assessment: GOOD
|
||||
- Flow is linear, goals are clear, and outputs map to TEA artifacts
|
||||
- Voice and tone consistent with Test Architect persona
|
||||
- Recommendation: READY (minor refinements optional)
|
||||
|
||||
## Plan Quality Validation
|
||||
|
||||
- Plan file present: workflow-plan.md
|
||||
- Planned steps found: 8 (all implemented)
|
||||
- Plan implementation status: Fully Implemented
|
||||
|
||||
## Summary
|
||||
|
||||
- Validation completed: 2026-01-27 10:24:01
|
||||
- Critical issues: 0
|
||||
- Warnings: 0 (informational notes only)
|
||||
- Readiness: READY (manual review optional)
|
||||
21
_bmad/tea/workflows/testarch/atdd/workflow-plan.md
Normal file
21
_bmad/tea/workflows/testarch/atdd/workflow-plan.md
Normal file
@@ -0,0 +1,21 @@
|
||||
# Workflow Plan: testarch-atdd
|
||||
|
||||
## Create Mode (steps-c)
|
||||
- step-01-preflight-and-context.md
|
||||
|
||||
- step-02-generation-mode.md
|
||||
- step-03-test-strategy.md
|
||||
- step-04-generate-tests.md
|
||||
- step-05-validate-and-complete.md
|
||||
|
||||
## Validate Mode (steps-v)
|
||||
- step-01-validate.md
|
||||
|
||||
## Edit Mode (steps-e)
|
||||
- step-01-assess.md
|
||||
- step-02-apply-edit.md
|
||||
|
||||
## Outputs
|
||||
- {test_artifacts}/atdd-checklist-{story_id}.md
|
||||
|
||||
- Failing acceptance tests under {project-root}/tests
|
||||
41
_bmad/tea/workflows/testarch/atdd/workflow.md
Normal file
41
_bmad/tea/workflows/testarch/atdd/workflow.md
Normal file
@@ -0,0 +1,41 @@
|
||||
---
|
||||
name: testarch-atdd
|
||||
description: Generate failing acceptance tests using TDD cycle. Use when user says 'lets write acceptance tests' or 'I want to do ATDD'
|
||||
web_bundle: true
|
||||
---
|
||||
|
||||
# Acceptance Test-Driven Development (ATDD)
|
||||
|
||||
**Goal:** Generate failing acceptance tests before implementation using TDD red-green-refactor cycle
|
||||
|
||||
**Role:** You are the Master Test Architect.
|
||||
|
||||
---
|
||||
|
||||
## WORKFLOW ARCHITECTURE
|
||||
|
||||
This workflow uses **tri-modal step-file architecture**:
|
||||
|
||||
- **Create mode (steps-c/)**: primary execution flow
|
||||
- **Validate mode (steps-v/)**: validation against checklist
|
||||
- **Edit mode (steps-e/)**: revise existing outputs
|
||||
|
||||
---
|
||||
|
||||
## INITIALIZATION SEQUENCE
|
||||
|
||||
### 1. Mode Determination
|
||||
|
||||
"Welcome to the workflow. What would you like to do?"
|
||||
|
||||
- **[C] Create** — Run the workflow
|
||||
- **[R] Resume** — Resume an interrupted workflow
|
||||
- **[V] Validate** — Validate existing outputs
|
||||
- **[E] Edit** — Edit existing outputs
|
||||
|
||||
### 2. Route to First Step
|
||||
|
||||
- **If C:** Load `steps-c/step-01-preflight-and-context.md`
|
||||
- **If R:** Load `steps-c/step-01b-resume.md`
|
||||
- **If V:** Load `steps-v/step-01-validate.md`
|
||||
- **If E:** Load `steps-e/step-01-assess.md`
|
||||
46
_bmad/tea/workflows/testarch/atdd/workflow.yaml
Normal file
46
_bmad/tea/workflows/testarch/atdd/workflow.yaml
Normal file
@@ -0,0 +1,46 @@
|
||||
# Test Architect workflow: atdd
|
||||
name: testarch-atdd
|
||||
# prettier-ignore
|
||||
description: 'Generate failing acceptance tests using TDD cycle. Use when the user says "lets write acceptance tests" or "I want to do ATDD"'
|
||||
|
||||
# Critical variables from config
|
||||
config_source: "{project-root}/_bmad/tea/config.yaml"
|
||||
output_folder: "{config_source}:output_folder"
|
||||
test_artifacts: "{config_source}:test_artifacts"
|
||||
user_name: "{config_source}:user_name"
|
||||
communication_language: "{config_source}:communication_language"
|
||||
document_output_language: "{config_source}:document_output_language"
|
||||
date: system-generated
|
||||
|
||||
# Workflow components
|
||||
installed_path: "{project-root}/_bmad/tea/workflows/testarch/atdd"
|
||||
instructions: "{installed_path}/instructions.md"
|
||||
validation: "{installed_path}/checklist.md"
|
||||
template: "{installed_path}/atdd-checklist-template.md"
|
||||
|
||||
# Variables and inputs
|
||||
variables:
|
||||
test_dir: "{project-root}/tests" # Root test directory
|
||||
|
||||
# Output configuration
|
||||
default_output_file: "{test_artifacts}/atdd-checklist-{story_id}.md"
|
||||
|
||||
# Required tools
|
||||
required_tools:
|
||||
- read_file # Read story markdown, framework config
|
||||
- write_file # Create test files, checklist, factory stubs
|
||||
- create_directory # Create test directories
|
||||
- list_files # Find existing fixtures and helpers
|
||||
- search_repo # Search for similar test patterns
|
||||
|
||||
tags:
|
||||
- qa
|
||||
- atdd
|
||||
- test-architect
|
||||
- tdd
|
||||
- red-green-refactor
|
||||
|
||||
execution_hints:
|
||||
interactive: false # Minimize prompts
|
||||
autonomous: true # Proceed without user input unless blocked
|
||||
iterative: true
|
||||
611
_bmad/tea/workflows/testarch/automate/checklist.md
Normal file
611
_bmad/tea/workflows/testarch/automate/checklist.md
Normal file
@@ -0,0 +1,611 @@
|
||||
# Automate Workflow Validation Checklist
|
||||
|
||||
Use this checklist to validate that the automate workflow has been executed correctly and all deliverables meet quality standards.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Before starting this workflow, verify:
|
||||
|
||||
- [ ] Framework scaffolding configured (playwright.config.ts or cypress.config.ts exists)
|
||||
- [ ] Test directory structure exists (tests/ folder with subdirectories)
|
||||
- [ ] Package.json has test framework dependencies installed
|
||||
|
||||
**Halt only if:** Framework scaffolding is completely missing (run `framework` workflow first)
|
||||
|
||||
**Note:** BMad artifacts (story, tech-spec, PRD) are OPTIONAL - workflow can run without them
|
||||
**Note:** `automate` generates tests; it does not run `*atdd` or `*test-review`. If ATDD outputs exist, use them as input and avoid duplicate coverage.
|
||||
|
||||
---
|
||||
|
||||
## Step 1: Execution Mode Determination and Context Loading
|
||||
|
||||
### Mode Detection
|
||||
|
||||
- [ ] Execution mode correctly determined:
|
||||
- [ ] BMad-Integrated Mode (story_file variable set) OR
|
||||
- [ ] Standalone Mode (target_feature or target_files set) OR
|
||||
- [ ] Auto-discover Mode (no targets specified)
|
||||
|
||||
### BMad Artifacts (If Available - OPTIONAL)
|
||||
|
||||
- [ ] Story markdown loaded (if `{story_file}` provided)
|
||||
- [ ] Acceptance criteria extracted from story (if available)
|
||||
- [ ] Tech-spec.md loaded (if `{use_tech_spec}` true and file exists)
|
||||
- [ ] Test-design.md loaded (if `{use_test_design}` true and file exists)
|
||||
- [ ] PRD.md loaded (if `{use_prd}` true and file exists)
|
||||
- [ ] **Note**: Absence of BMad artifacts does NOT halt workflow
|
||||
|
||||
### Framework Configuration
|
||||
|
||||
- [ ] Test framework config loaded (playwright.config.ts or cypress.config.ts)
|
||||
- [ ] Test directory structure identified from `{test_dir}`
|
||||
- [ ] Existing test patterns reviewed
|
||||
- [ ] Test runner capabilities noted (parallel execution, fixtures, etc.)
|
||||
|
||||
### Coverage Analysis
|
||||
|
||||
- [ ] Existing test files searched in `{test_dir}` (if `{analyze_coverage}` true)
|
||||
- [ ] Tested features vs untested features identified
|
||||
- [ ] Coverage gaps mapped (tests to source files)
|
||||
- [ ] Existing fixture and factory patterns checked
|
||||
|
||||
### Knowledge Base Fragments Loaded
|
||||
|
||||
- [ ] `test-levels-framework.md` - Test level selection
|
||||
- [ ] `test-priorities.md` - Priority classification (P0-P3)
|
||||
- [ ] `fixture-architecture.md` - Fixture patterns with auto-cleanup
|
||||
- [ ] `data-factories.md` - Factory patterns using faker
|
||||
- [ ] `selective-testing.md` - Targeted test execution strategies
|
||||
- [ ] `ci-burn-in.md` - Flaky test detection patterns
|
||||
- [ ] `test-quality.md` - Test design principles
|
||||
|
||||
---
|
||||
|
||||
## Step 2: Automation Targets Identification
|
||||
|
||||
### Target Determination
|
||||
|
||||
**BMad-Integrated Mode (if story available):**
|
||||
|
||||
- [ ] Acceptance criteria mapped to test scenarios
|
||||
- [ ] Features implemented in story identified
|
||||
- [ ] Existing ATDD tests checked (if any)
|
||||
- [ ] Expansion beyond ATDD planned (edge cases, negative paths)
|
||||
|
||||
**Standalone Mode (if no story):**
|
||||
|
||||
- [ ] Specific feature analyzed (if `{target_feature}` specified)
|
||||
- [ ] Specific files analyzed (if `{target_files}` specified)
|
||||
- [ ] Features auto-discovered (if `{auto_discover_features}` true)
|
||||
- [ ] Features prioritized by:
|
||||
- [ ] No test coverage (highest priority)
|
||||
- [ ] Complex business logic
|
||||
- [ ] External integrations (API, database, auth)
|
||||
- [ ] Critical user paths (login, checkout, etc.)
|
||||
|
||||
### Test Level Selection
|
||||
|
||||
- [ ] Test level selection framework applied (from `test-levels-framework.md`)
|
||||
- [ ] E2E tests identified: Critical user journeys, multi-system integration
|
||||
- [ ] API tests identified: Business logic, service contracts, data transformations
|
||||
- [ ] Component tests identified: UI behavior, interactions, state management
|
||||
- [ ] Unit tests identified: Pure logic, edge cases, error handling
|
||||
|
||||
### Duplicate Coverage Avoidance
|
||||
|
||||
- [ ] Same behavior NOT tested at multiple levels unnecessarily
|
||||
- [ ] E2E used for critical happy path only
|
||||
- [ ] API tests used for business logic variations
|
||||
- [ ] Component tests used for UI interaction edge cases
|
||||
- [ ] Unit tests used for pure logic edge cases
|
||||
|
||||
### Priority Assignment
|
||||
|
||||
- [ ] Test priorities assigned using `test-priorities.md` framework
|
||||
- [ ] P0 tests: Critical paths, security-critical, data integrity
|
||||
- [ ] P1 tests: Important features, integration points, error handling
|
||||
- [ ] P2 tests: Edge cases, less-critical variations, performance
|
||||
- [ ] P3 tests: Nice-to-have, rarely-used features, exploratory
|
||||
- [ ] Priority variables respected:
|
||||
- [ ] `{include_p0}` = true (always include)
|
||||
- [ ] `{include_p1}` = true (high priority)
|
||||
- [ ] `{include_p2}` = true (medium priority)
|
||||
- [ ] `{include_p3}` = false (low priority, skip by default)
|
||||
|
||||
### Coverage Plan Created
|
||||
|
||||
- [ ] Test coverage plan documented
|
||||
- [ ] What will be tested at each level listed
|
||||
- [ ] Priorities assigned to each test
|
||||
- [ ] Coverage strategy clear (critical-paths, comprehensive, or selective)
|
||||
|
||||
---
|
||||
|
||||
## Step 3: Test Infrastructure Generated
|
||||
|
||||
### Fixture Architecture
|
||||
|
||||
- [ ] Existing fixtures checked in `tests/support/fixtures/`
|
||||
- [ ] Fixture architecture created/enhanced (if `{generate_fixtures}` true)
|
||||
- [ ] All fixtures use Playwright's `test.extend()` pattern
|
||||
- [ ] All fixtures have auto-cleanup in teardown
|
||||
- [ ] Common fixtures created/enhanced:
|
||||
- [ ] authenticatedUser (with auto-delete)
|
||||
- [ ] apiRequest (authenticated client)
|
||||
- [ ] mockNetwork (external service mocking)
|
||||
- [ ] testDatabase (with auto-cleanup)
|
||||
|
||||
### Data Factories
|
||||
|
||||
- [ ] Existing factories checked in `tests/support/factories/`
|
||||
- [ ] Factory architecture created/enhanced (if `{generate_factories}` true)
|
||||
- [ ] All factories use `@faker-js/faker` for random data (no hardcoded values)
|
||||
- [ ] All factories support overrides for specific scenarios
|
||||
- [ ] Common factories created/enhanced:
|
||||
- [ ] User factory (email, password, name, role)
|
||||
- [ ] Product factory (name, price, SKU)
|
||||
- [ ] Order factory (items, total, status)
|
||||
- [ ] Cleanup helpers provided (e.g., deleteUser(), deleteProduct())
|
||||
|
||||
### Helper Utilities
|
||||
|
||||
- [ ] Existing helpers checked in `tests/support/helpers/` (if `{update_helpers}` true)
|
||||
- [ ] Common utilities created/enhanced:
|
||||
- [ ] waitFor (polling for complex conditions)
|
||||
- [ ] retry (retry helper for flaky operations)
|
||||
- [ ] testData (test data generation)
|
||||
- [ ] assertions (custom assertion helpers)
|
||||
|
||||
---
|
||||
|
||||
## Step 4: Test Files Generated
|
||||
|
||||
### Test File Structure
|
||||
|
||||
- [ ] Test files organized correctly:
|
||||
- [ ] `tests/e2e/` for E2E tests
|
||||
- [ ] `tests/api/` for API tests
|
||||
- [ ] `tests/component/` for component tests
|
||||
- [ ] `tests/unit/` for unit tests
|
||||
- [ ] `tests/support/` for fixtures/factories/helpers
|
||||
|
||||
### E2E Tests (If Applicable)
|
||||
|
||||
- [ ] E2E test files created in `tests/e2e/`
|
||||
- [ ] All tests follow Given-When-Then format
|
||||
- [ ] All tests have priority tags ([P0], [P1], [P2], [P3]) in test name
|
||||
- [ ] All tests use data-testid selectors (not CSS classes)
|
||||
- [ ] One assertion per test (atomic design)
|
||||
- [ ] No hard waits or sleeps (explicit waits only)
|
||||
- [ ] Network-first pattern applied (route interception BEFORE navigation)
|
||||
- [ ] Clear Given-When-Then comments in test code
|
||||
|
||||
### API Tests (If Applicable)
|
||||
|
||||
- [ ] API test files created in `tests/api/`
|
||||
- [ ] All tests follow Given-When-Then format
|
||||
- [ ] All tests have priority tags in test name
|
||||
- [ ] API contracts validated (request/response structure)
|
||||
- [ ] HTTP status codes verified
|
||||
- [ ] Response body validation includes required fields
|
||||
- [ ] Error cases tested (400, 401, 403, 404, 500)
|
||||
- [ ] JWT token format validated (if auth tests)
|
||||
|
||||
### Consumer Contract Tests / CDC (If `use_pactjs_utils` Enabled)
|
||||
|
||||
**Provider Endpoint Comments:**
|
||||
|
||||
- [ ] Every Pact interaction has `// Provider endpoint:` comment
|
||||
- [ ] Comment includes exact file path to provider route handler, OR uses the TODO form when provider is inaccessible
|
||||
- [ ] Comment follows format: `// Provider endpoint: <path> -> <METHOD> <route>` or `// Provider endpoint: TODO — provider source not accessible, verify manually`
|
||||
|
||||
**Provider Source Scrutiny:**
|
||||
|
||||
- [ ] Provider route handlers and/or OpenAPI spec read before generating each interaction
|
||||
- [ ] Status codes verified against provider source (e.g., 201 not assumed 200)
|
||||
- [ ] Field names cross-referenced with provider type/DTO definitions
|
||||
- [ ] Data types verified (string ID vs number ID, date formats)
|
||||
- [ ] Enum/union values extracted from provider validation schemas
|
||||
- [ ] Required request fields and headers checked against provider validation
|
||||
- [ ] Nested response structures match provider's actual response construction
|
||||
- [ ] Scrutiny evidence documented as block comment in each test file
|
||||
|
||||
**CDC Quality Gates:**
|
||||
|
||||
- [ ] Postel's Law enforced: exact values in `withRequest`, matchers in `willRespondWith`
|
||||
- [ ] Response matchers (`like`, `eachLike`, `string`, `integer`) used only in `willRespondWith`
|
||||
- [ ] Provider state names are consistent with provider's state handler naming
|
||||
- [ ] DI pattern used for consumer function imports (actual consumer code, not raw `fetch()`)
|
||||
- [ ] One logical endpoint per Pact interaction (no multi-endpoint interactions)
|
||||
|
||||
### Component Tests (If Applicable)
|
||||
|
||||
- [ ] Component test files created in `tests/component/`
|
||||
- [ ] All tests follow Given-When-Then format
|
||||
- [ ] All tests have priority tags in test name
|
||||
- [ ] Component mounting works correctly
|
||||
- [ ] Interaction testing covers user actions (click, hover, keyboard)
|
||||
- [ ] State management validated
|
||||
- [ ] Props and events tested
|
||||
|
||||
### Unit Tests (If Applicable)
|
||||
|
||||
- [ ] Unit test files created in `tests/unit/`
|
||||
- [ ] All tests follow Given-When-Then format
|
||||
- [ ] All tests have priority tags in test name
|
||||
- [ ] Pure logic tested (no dependencies)
|
||||
- [ ] Edge cases covered
|
||||
- [ ] Error handling tested
|
||||
|
||||
### Quality Standards Enforced
|
||||
|
||||
- [ ] All tests use Given-When-Then format with clear comments
|
||||
- [ ] All tests have descriptive names with priority tags
|
||||
- [ ] No duplicate tests (same behavior tested multiple times)
|
||||
- [ ] No flaky patterns (race conditions, timing issues)
|
||||
- [ ] No test interdependencies (tests can run in any order)
|
||||
- [ ] Tests are deterministic (same input always produces same result)
|
||||
- [ ] All tests use data-testid selectors (E2E tests)
|
||||
- [ ] No hard waits: `await page.waitForTimeout()` (forbidden)
|
||||
- [ ] No conditional flow: `if (await element.isVisible())` (forbidden)
|
||||
- [ ] No try-catch for test logic (only for cleanup)
|
||||
- [ ] No hardcoded test data (use factories with faker)
|
||||
- [ ] No page object classes (tests are direct and simple)
|
||||
- [ ] No shared state between tests
|
||||
|
||||
### Network-First Pattern Applied
|
||||
|
||||
- [ ] Route interception set up BEFORE navigation (E2E tests with network requests)
|
||||
- [ ] `page.route()` called before `page.goto()` to prevent race conditions
|
||||
- [ ] Network-first pattern verified in all E2E tests that make API calls
|
||||
|
||||
---
|
||||
|
||||
## Step 5: Test Validation and Healing (NEW - Phase 2.5)
|
||||
|
||||
### Healing Configuration
|
||||
|
||||
- [ ] Healing configuration checked:
|
||||
- [ ] `{auto_validate}` setting noted (default: true)
|
||||
- [ ] `{auto_heal_failures}` setting noted (default: false)
|
||||
- [ ] `{max_healing_iterations}` setting noted (default: 3)
|
||||
- [ ] `{use_mcp_healing}` setting noted (default: true)
|
||||
|
||||
### Healing Knowledge Fragments Loaded (If Healing Enabled)
|
||||
|
||||
- [ ] `test-healing-patterns.md` loaded (common failure patterns and fixes)
|
||||
- [ ] `selector-resilience.md` loaded (selector refactoring guide)
|
||||
- [ ] `timing-debugging.md` loaded (race condition fixes)
|
||||
|
||||
### Test Execution and Validation
|
||||
|
||||
- [ ] Generated tests executed (if `{auto_validate}` true)
|
||||
- [ ] Test results captured:
|
||||
- [ ] Total tests run
|
||||
- [ ] Passing tests count
|
||||
- [ ] Failing tests count
|
||||
- [ ] Error messages and stack traces captured
|
||||
|
||||
### Healing Loop (If Enabled and Tests Failed)
|
||||
|
||||
- [ ] Healing loop entered (if `{auto_heal_failures}` true AND tests failed)
|
||||
- [ ] For each failing test:
|
||||
- [ ] Failure pattern identified (selector, timing, data, network, hard wait)
|
||||
- [ ] Appropriate healing strategy applied:
|
||||
- [ ] Stale selector → Replaced with data-testid or ARIA role
|
||||
- [ ] Race condition → Added network-first interception or state waits
|
||||
- [ ] Dynamic data → Replaced hardcoded values with regex/dynamic generation
|
||||
- [ ] Network error → Added route mocking
|
||||
- [ ] Hard wait → Replaced with event-based wait
|
||||
- [ ] Healed test re-run to validate fix
|
||||
- [ ] Iteration count tracked (max 3 attempts)
|
||||
|
||||
### Unfixable Tests Handling
|
||||
|
||||
- [ ] Tests that couldn't be healed after 3 iterations marked with `test.fixme()` (if `{mark_unhealable_as_fixme}` true)
|
||||
- [ ] Detailed comment added to test.fixme() tests:
|
||||
- [ ] What failure occurred
|
||||
- [ ] What healing was attempted (3 iterations)
|
||||
- [ ] Why healing failed
|
||||
- [ ] Manual investigation steps needed
|
||||
- [ ] Original test logic preserved in comments
|
||||
|
||||
### Healing Report Generated
|
||||
|
||||
- [ ] Healing report generated (if healing attempted)
|
||||
- [ ] Report includes:
|
||||
- [ ] Auto-heal enabled status
|
||||
- [ ] Healing mode (MCP-assisted or Pattern-based)
|
||||
- [ ] Iterations allowed (max_healing_iterations)
|
||||
- [ ] Validation results (total, passing, failing)
|
||||
- [ ] Successfully healed tests (count, file:line, fix applied)
|
||||
- [ ] Unable to heal tests (count, file:line, reason)
|
||||
- [ ] Healing patterns applied (selector fixes, timing fixes, data fixes)
|
||||
- [ ] Knowledge base references used
|
||||
|
||||
---
|
||||
|
||||
## Step 6: Documentation and Scripts Updated
|
||||
|
||||
### Test README Updated
|
||||
|
||||
- [ ] `tests/README.md` created or updated (if `{update_readme}` true)
|
||||
- [ ] Test suite structure overview included
|
||||
- [ ] Test execution instructions provided (all, specific files, by priority)
|
||||
- [ ] Fixture usage examples provided
|
||||
- [ ] Factory usage examples provided
|
||||
- [ ] Priority tagging convention explained ([P0], [P1], [P2], [P3])
|
||||
- [ ] How to write new tests documented
|
||||
- [ ] Common patterns documented
|
||||
- [ ] Anti-patterns documented (what to avoid)
|
||||
|
||||
### package.json Scripts Updated
|
||||
|
||||
- [ ] package.json scripts added/updated (if `{update_package_scripts}` true)
|
||||
- [ ] `test:e2e` script for all E2E tests
|
||||
- [ ] `test:e2e:p0` script for P0 tests only
|
||||
- [ ] `test:e2e:p1` script for P0 + P1 tests
|
||||
- [ ] `test:api` script for API tests
|
||||
- [ ] `test:component` script for component tests
|
||||
- [ ] `test:unit` script for unit tests (if applicable)
|
||||
|
||||
### Test Suite Executed
|
||||
|
||||
- [ ] Test suite run locally (if `{run_tests_after_generation}` true)
|
||||
- [ ] Test results captured (passing/failing counts)
|
||||
- [ ] No flaky patterns detected (tests are deterministic)
|
||||
- [ ] Setup requirements documented (if any)
|
||||
- [ ] Known issues documented (if any)
|
||||
|
||||
---
|
||||
|
||||
## Step 6: Automation Summary Generated
|
||||
|
||||
### Automation Summary Document
|
||||
|
||||
- [ ] Output file created at `{output_summary}`
|
||||
- [ ] Document includes execution mode (BMad-Integrated, Standalone, Auto-discover)
|
||||
- [ ] Feature analysis included (source files, coverage gaps) - Standalone mode
|
||||
- [ ] Tests created listed (E2E, API, Component, Unit) with counts and paths
|
||||
- [ ] Infrastructure created listed (fixtures, factories, helpers)
|
||||
- [ ] Test execution instructions provided
|
||||
- [ ] Coverage analysis included:
|
||||
- [ ] Total test count
|
||||
- [ ] Priority breakdown (P0, P1, P2, P3 counts)
|
||||
- [ ] Test level breakdown (E2E, API, Component, Unit counts)
|
||||
- [ ] Coverage percentage (if calculated)
|
||||
- [ ] Coverage status (acceptance criteria covered, gaps identified)
|
||||
- [ ] Definition of Done checklist included
|
||||
- [ ] Next steps provided
|
||||
- [ ] Recommendations included (if Standalone mode)
|
||||
|
||||
### Summary Provided to User
|
||||
|
||||
- [ ] Concise summary output provided
|
||||
- [ ] Total tests created across test levels
|
||||
- [ ] Priority breakdown (P0, P1, P2, P3 counts)
|
||||
- [ ] Infrastructure counts (fixtures, factories, helpers)
|
||||
- [ ] Test execution command provided
|
||||
- [ ] Output file path provided
|
||||
- [ ] Next steps listed
|
||||
|
||||
---
|
||||
|
||||
## Quality Checks
|
||||
|
||||
### Test Design Quality
|
||||
|
||||
- [ ] Tests are readable (clear Given-When-Then structure)
|
||||
- [ ] Tests are maintainable (use factories/fixtures, not hardcoded data)
|
||||
- [ ] Tests are isolated (no shared state between tests)
|
||||
- [ ] Tests are deterministic (no race conditions or flaky patterns)
|
||||
- [ ] Tests are atomic (one assertion per test)
|
||||
- [ ] Tests are fast (no unnecessary waits or delays)
|
||||
- [ ] Tests are lean (files under {max_file_lines} lines)
|
||||
|
||||
### Knowledge Base Integration
|
||||
|
||||
- [ ] Test level selection framework applied (from `test-levels-framework.md`)
|
||||
- [ ] Priority classification applied (from `test-priorities.md`)
|
||||
- [ ] Fixture architecture patterns applied (from `fixture-architecture.md`)
|
||||
- [ ] Data factory patterns applied (from `data-factories.md`)
|
||||
- [ ] Selective testing strategies considered (from `selective-testing.md`)
|
||||
- [ ] Flaky test detection patterns considered (from `ci-burn-in.md`)
|
||||
- [ ] Test quality principles applied (from `test-quality.md`)
|
||||
|
||||
### Code Quality
|
||||
|
||||
- [ ] All TypeScript types are correct and complete
|
||||
- [ ] No linting errors in generated test files
|
||||
- [ ] Consistent naming conventions followed
|
||||
- [ ] Imports are organized and correct
|
||||
- [ ] Code follows project style guide
|
||||
- [ ] No console.log or debug statements in test code
|
||||
|
||||
---
|
||||
|
||||
## Integration Points
|
||||
|
||||
### With Framework Workflow
|
||||
|
||||
- [ ] Test framework configuration detected and used
|
||||
- [ ] Directory structure matches framework setup
|
||||
- [ ] Fixtures and helpers follow established patterns
|
||||
- [ ] Naming conventions consistent with framework standards
|
||||
|
||||
### With BMad Workflows (If Available - OPTIONAL)
|
||||
|
||||
**With Story Workflow:**
|
||||
|
||||
- [ ] Story ID correctly referenced in output (if story available)
|
||||
- [ ] Acceptance criteria from story reflected in tests (if story available)
|
||||
- [ ] Technical constraints from story considered (if story available)
|
||||
|
||||
**With test-design Workflow:**
|
||||
|
||||
- [ ] P0 scenarios from test-design prioritized (if test-design available)
|
||||
- [ ] Risk assessment from test-design considered (if test-design available)
|
||||
- [ ] Coverage strategy aligned with test-design (if test-design available)
|
||||
|
||||
**With atdd Workflow:**
|
||||
|
||||
- [ ] ATDD artifacts provided or located (manual handoff; `atdd` not auto-run)
|
||||
- [ ] Existing ATDD tests checked (if story had ATDD workflow run)
|
||||
- [ ] Expansion beyond ATDD planned (edge cases, negative paths)
|
||||
- [ ] No duplicate coverage with ATDD tests
|
||||
|
||||
### With CI Pipeline
|
||||
|
||||
- [ ] Tests can run in CI environment
|
||||
- [ ] Tests are parallelizable (no shared state)
|
||||
- [ ] Tests have appropriate timeouts
|
||||
- [ ] Tests clean up their data (no CI environment pollution)
|
||||
|
||||
---
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
All of the following must be true before marking this workflow as complete:
|
||||
|
||||
- [ ] **Execution mode determined** (BMad-Integrated, Standalone, or Auto-discover)
|
||||
- [ ] **Framework configuration loaded** and validated
|
||||
- [ ] **Coverage analysis completed** (gaps identified if analyze_coverage true)
|
||||
- [ ] **Automation targets identified** (what needs testing)
|
||||
- [ ] **Test levels selected** appropriately (E2E, API, Component, Unit)
|
||||
- [ ] **Duplicate coverage avoided** (same behavior not tested at multiple levels)
|
||||
- [ ] **Test priorities assigned** (P0, P1, P2, P3)
|
||||
- [ ] **Fixture architecture created/enhanced** with auto-cleanup
|
||||
- [ ] **Data factories created/enhanced** using faker (no hardcoded data)
|
||||
- [ ] **Helper utilities created/enhanced** (if needed)
|
||||
- [ ] **Test files generated** at appropriate levels (E2E, API, Component, Unit)
|
||||
- [ ] **Given-When-Then format used** consistently across all tests
|
||||
- [ ] **Priority tags added** to all test names ([P0], [P1], [P2], [P3])
|
||||
- [ ] **data-testid selectors used** in E2E tests (not CSS classes)
|
||||
- [ ] **Network-first pattern applied** (route interception before navigation)
|
||||
- [ ] **Quality standards enforced** (no hard waits, no flaky patterns, self-cleaning, deterministic)
|
||||
- [ ] **Test README updated** with execution instructions and patterns
|
||||
- [ ] **package.json scripts updated** with test execution commands
|
||||
- [ ] **Test suite run locally** (if run_tests_after_generation true)
|
||||
- [ ] **Tests validated** (if auto_validate enabled)
|
||||
- [ ] **Failures healed** (if auto_heal_failures enabled and tests failed)
|
||||
- [ ] **Healing report generated** (if healing attempted)
|
||||
- [ ] **Unfixable tests marked** with test.fixme() and detailed comments (if any)
|
||||
- [ ] **Automation summary created** and saved to correct location
|
||||
- [ ] **Output file formatted correctly**
|
||||
- [ ] **Knowledge base references applied** and documented (including healing fragments if used)
|
||||
- [ ] **No test quality issues** (flaky patterns, race conditions, hardcoded data, page objects)
|
||||
- [ ] **Provider scrutiny completed or gracefully degraded** for all CDC interactions — each interaction either has scrutiny evidence or a TODO marker (if `use_pactjs_utils` enabled)
|
||||
- [ ] **Provider endpoint comments present** on every Pact interaction (if `use_pactjs_utils` enabled)
|
||||
|
||||
---
|
||||
|
||||
## Common Issues and Resolutions
|
||||
|
||||
### Issue: BMad artifacts not found
|
||||
|
||||
**Problem:** Story, tech-spec, or PRD files not found when variables are set.
|
||||
|
||||
**Resolution:**
|
||||
|
||||
- **automate does NOT require BMad artifacts** - they are OPTIONAL enhancements
|
||||
- If files not found, switch to Standalone Mode automatically
|
||||
- Analyze source code directly without BMad context
|
||||
- Continue workflow without halting
|
||||
|
||||
### Issue: Framework configuration not found
|
||||
|
||||
**Problem:** No playwright.config.ts or cypress.config.ts found.
|
||||
|
||||
**Resolution:**
|
||||
|
||||
- **HALT workflow** - framework is required
|
||||
- Message: "Framework scaffolding required. Run `bmad tea *framework` first."
|
||||
- User must run framework workflow before automate
|
||||
|
||||
### Issue: No automation targets identified
|
||||
|
||||
**Problem:** Neither story, target_feature, nor target_files specified, and auto-discover finds nothing.
|
||||
|
||||
**Resolution:**
|
||||
|
||||
- Check if source_dir variable is correct
|
||||
- Verify source code exists in project
|
||||
- Ask user to specify target_feature or target_files explicitly
|
||||
- Provide examples: `target_feature: "src/auth/"` or `target_files: "src/auth/login.ts,src/auth/session.ts"`
|
||||
|
||||
### Issue: Duplicate coverage detected
|
||||
|
||||
**Problem:** Same behavior tested at multiple levels (E2E + API + Component).
|
||||
|
||||
**Resolution:**
|
||||
|
||||
- Review test level selection framework (test-levels-framework.md)
|
||||
- Use E2E for critical happy path ONLY
|
||||
- Use API for business logic variations
|
||||
- Use Component for UI edge cases
|
||||
- Remove redundant tests that duplicate coverage
|
||||
|
||||
### Issue: Tests have hardcoded data
|
||||
|
||||
**Problem:** Tests use hardcoded email addresses, passwords, or other data.
|
||||
|
||||
**Resolution:**
|
||||
|
||||
- Replace all hardcoded data with factory function calls
|
||||
- Use faker for all random data generation
|
||||
- Update data-factories to support all required test scenarios
|
||||
- Example: `createUser({ email: faker.internet.email() })`
|
||||
|
||||
### Issue: Tests are flaky
|
||||
|
||||
**Problem:** Tests fail intermittently, pass on retry.
|
||||
|
||||
**Resolution:**
|
||||
|
||||
- Remove all hard waits (`page.waitForTimeout()`)
|
||||
- Use explicit waits (`page.waitForSelector()`)
|
||||
- Apply network-first pattern (route interception before navigation)
|
||||
- Remove conditional flow (`if (await element.isVisible())`)
|
||||
- Ensure tests are deterministic (no race conditions)
|
||||
- Run burn-in loop (10 iterations) to detect flakiness
|
||||
|
||||
### Issue: Fixtures don't clean up data
|
||||
|
||||
**Problem:** Test data persists after test run, causing test pollution.
|
||||
|
||||
**Resolution:**
|
||||
|
||||
- Ensure all fixtures have cleanup in teardown phase
|
||||
- Cleanup happens AFTER `await use(data)`
|
||||
- Call deletion/cleanup functions (deleteUser, deleteProduct, etc.)
|
||||
- Verify cleanup works by checking database/storage after test run
|
||||
|
||||
### Issue: Tests too slow
|
||||
|
||||
**Problem:** Tests take longer than 90 seconds (max_test_duration).
|
||||
|
||||
**Resolution:**
|
||||
|
||||
- Remove unnecessary waits and delays
|
||||
- Use parallel execution where possible
|
||||
- Mock external services (don't make real API calls)
|
||||
- Use API tests instead of E2E for business logic
|
||||
- Optimize test data creation (use in-memory database, etc.)
|
||||
|
||||
---
|
||||
|
||||
## Notes for TEA Agent
|
||||
|
||||
- **automate is flexible:** Can work with or without BMad artifacts (story, tech-spec, PRD are OPTIONAL)
|
||||
- **Standalone mode is powerful:** Analyze any codebase and generate tests independently
|
||||
- **Auto-discover mode:** Scan codebase for features needing tests when no targets specified
|
||||
- **Framework is the ONLY hard requirement:** HALT if framework config missing, otherwise proceed
|
||||
- **Avoid duplicate coverage:** E2E for critical paths only, API/Component for variations
|
||||
- **Priority tagging enables selective execution:** P0 tests run on every commit, P1 on PR, P2 nightly
|
||||
- **Network-first pattern prevents race conditions:** Route interception BEFORE navigation
|
||||
- **No page objects:** Keep tests simple, direct, and maintainable
|
||||
- **Use knowledge base:** Load relevant fragments (test-levels, test-priorities, fixture-architecture, data-factories, healing patterns) for guidance
|
||||
- **Deterministic tests only:** No hard waits, no conditional flow, no flaky patterns allowed
|
||||
- **Optional healing:** auto_heal_failures disabled by default (opt-in for automatic test healing)
|
||||
- **Graceful degradation:** Healing works without Playwright MCP (pattern-based fallback)
|
||||
- **Unfixable tests handled:** Mark with test.fixme() and detailed comments (not silently broken)
|
||||
50
_bmad/tea/workflows/testarch/automate/instructions.md
Normal file
50
_bmad/tea/workflows/testarch/automate/instructions.md
Normal file
@@ -0,0 +1,50 @@
|
||||
<!-- Powered by BMAD-CORE™ -->
|
||||
|
||||
# Test Automation Expansion
|
||||
|
||||
**Workflow ID**: `_bmad/tea/testarch/automate`
|
||||
**Version**: 5.0 (Step-File Architecture)
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Expands test automation coverage by generating prioritized tests at the appropriate level (E2E, API, Component, Unit) with supporting fixtures and helpers.
|
||||
|
||||
Modes:
|
||||
|
||||
- **BMad-Integrated**: Uses story/PRD/test-design artifacts when available
|
||||
- **Standalone**: Analyzes existing codebase without BMad artifacts
|
||||
|
||||
---
|
||||
|
||||
## WORKFLOW ARCHITECTURE
|
||||
|
||||
This workflow uses **step-file architecture** for disciplined execution:
|
||||
|
||||
- **Micro-file Design**: Each step is self-contained
|
||||
- **JIT Loading**: Only the current step file is in memory
|
||||
- **Sequential Enforcement**: Execute steps in order without skipping
|
||||
|
||||
---
|
||||
|
||||
## INITIALIZATION SEQUENCE
|
||||
|
||||
### 1. Configuration Loading
|
||||
|
||||
From `workflow.yaml`, resolve:
|
||||
|
||||
- `config_source`, `test_artifacts`, `user_name`, `communication_language`, `document_output_language`, `date`
|
||||
- `test_dir`, `source_dir`, `coverage_target`, `standalone_mode`
|
||||
|
||||
### 2. First Step
|
||||
|
||||
Load, read completely, and execute:
|
||||
`{project-root}/_bmad/tea/workflows/testarch/automate/steps-c/step-01-preflight-and-context.md`
|
||||
|
||||
### 3. Resume Support
|
||||
|
||||
If the user selects **Resume** mode, load, read completely, and execute:
|
||||
`{project-root}/_bmad/tea/workflows/testarch/automate/steps-c/step-01b-resume.md`
|
||||
|
||||
This checks the output document for progress tracking frontmatter and routes to the next incomplete step.
|
||||
@@ -0,0 +1,237 @@
|
||||
---
|
||||
name: 'step-01-preflight-and-context'
|
||||
description: 'Determine mode, verify framework, and load context and knowledge'
|
||||
outputFile: '{test_artifacts}/automation-summary.md'
|
||||
nextStepFile: './step-02-identify-targets.md'
|
||||
knowledgeIndex: '{project-root}/_bmad/tea/testarch/tea-index.csv'
|
||||
---
|
||||
|
||||
# Step 1: Preflight & Context Loading
|
||||
|
||||
## STEP GOAL
|
||||
|
||||
Determine execution mode, verify framework readiness, and load the necessary artifacts and knowledge fragments.
|
||||
|
||||
## MANDATORY EXECUTION RULES
|
||||
|
||||
- 📖 Read the entire step file before acting
|
||||
- ✅ Speak in `{communication_language}`
|
||||
- 🚫 Halt if framework scaffolding is missing
|
||||
|
||||
---
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
- 💾 Record outputs before proceeding
|
||||
- 📖 Load the next step only when instructed
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: config, loaded artifacts, and knowledge fragments
|
||||
- Focus: this step's goal only
|
||||
- Limits: do not execute future steps
|
||||
- Dependencies: prior steps' outputs (if any)
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly. Do not skip, reorder, or improvise.
|
||||
|
||||
## 1. Stack Detection & Verify Framework
|
||||
|
||||
**Read `config.test_stack_type`** from `{config_source}`.
|
||||
|
||||
**Auto-Detection Algorithm** (when `test_stack_type` is `"auto"` or not configured):
|
||||
|
||||
- Scan `{project-root}` for project manifests:
|
||||
- **Frontend indicators**: `package.json` with react/vue/angular/next dependencies, `playwright.config.*`, `vite.config.*`, `webpack.config.*`
|
||||
- **Backend indicators**: `pyproject.toml`, `pom.xml`/`build.gradle`, `go.mod`, `*.csproj`/`*.sln`, `Gemfile`, `Cargo.toml`
|
||||
- **Both present** = `fullstack`; only frontend = `frontend`; only backend = `backend`
|
||||
- Explicit `test_stack_type` config value overrides auto-detection
|
||||
- **Backward compatibility**: if `test_stack_type` is not in config, treat as `"auto"` (preserves current frontend behavior for existing installs)
|
||||
|
||||
Store result as `{detected_stack}` = `frontend` | `backend` | `fullstack`
|
||||
|
||||
**Verify framework exists:**
|
||||
|
||||
**If {detected_stack} is `frontend` or `fullstack`:**
|
||||
|
||||
- `playwright.config.ts` or `cypress.config.ts`
|
||||
- `package.json` includes test dependencies
|
||||
|
||||
**If {detected_stack} is `backend` or `fullstack`:**
|
||||
|
||||
- Relevant test config exists (e.g., `conftest.py`, `src/test/`, `*_test.go`, `.rspec`, test project `*.csproj`)
|
||||
|
||||
If missing: **HALT** with message "Run `framework` workflow first."
|
||||
|
||||
---
|
||||
|
||||
## 2. Determine Execution Mode
|
||||
|
||||
- **BMad-Integrated** if story/tech-spec/test-design artifacts are provided or found
|
||||
- **Standalone** if only source code is available
|
||||
- If unclear, ask the user which mode to use
|
||||
|
||||
---
|
||||
|
||||
## 3. Load Context
|
||||
|
||||
### BMad-Integrated (if available)
|
||||
|
||||
- Story with acceptance criteria
|
||||
- PRD and/or tech spec
|
||||
- Test-design document (if exists)
|
||||
|
||||
### Standalone
|
||||
|
||||
- Skip artifacts; proceed to codebase analysis
|
||||
|
||||
### Always Load
|
||||
|
||||
- Test framework config
|
||||
- Existing test structure in `{test_dir}`
|
||||
- Existing tests (for coverage gaps)
|
||||
|
||||
### Read TEA Config Flags
|
||||
|
||||
- From `{config_source}` read `tea_use_playwright_utils`
|
||||
- From `{config_source}` read `tea_use_pactjs_utils`
|
||||
- From `{config_source}` read `tea_pact_mcp`
|
||||
- From `{config_source}` read `tea_browser_automation`
|
||||
- From `{config_source}` read `test_stack_type`
|
||||
|
||||
---
|
||||
|
||||
### Tiered Knowledge Loading
|
||||
|
||||
Load fragments based on their `tier` classification in `tea-index.csv`:
|
||||
|
||||
1. **Core tier** (always load): Foundational fragments required for this workflow
|
||||
2. **Extended tier** (load on-demand): Load when deeper analysis is needed or when the user's context requires it
|
||||
3. **Specialized tier** (load only when relevant): Load only when the specific use case matches (e.g., contract-testing only for microservices, email-auth only for email flows)
|
||||
|
||||
> **Context Efficiency**: Loading only core fragments reduces context usage by 40-50% compared to loading all fragments.
|
||||
|
||||
### Playwright Utils Loading Profiles
|
||||
|
||||
**If `tea_use_playwright_utils` is enabled**, select the appropriate loading profile:
|
||||
|
||||
- **API-only profile** (when `{detected_stack}` is `backend` or no `page.goto`/`page.locator` found in test files):
|
||||
Load: `overview`, `api-request`, `auth-session`, `recurse` (~1,800 lines)
|
||||
|
||||
- **Full UI+API profile** (when `{detected_stack}` is `frontend`/`fullstack` or browser tests detected):
|
||||
Load: all Playwright Utils core fragments (~4,500 lines)
|
||||
|
||||
**Detection**: Scan `{test_dir}` for files containing `page.goto` or `page.locator`. If none found, use API-only profile.
|
||||
|
||||
### Pact.js Utils Loading
|
||||
|
||||
**If `tea_use_pactjs_utils` is enabled** (and `{detected_stack}` is `backend` or `fullstack`, or microservices indicators detected):
|
||||
|
||||
Load: `pactjs-utils-overview.md`, `pactjs-utils-consumer-helpers.md`, `pactjs-utils-provider-verifier.md`, `pactjs-utils-request-filter.md` (~800 lines)
|
||||
|
||||
**If `tea_use_pactjs_utils` is disabled** but contract testing is relevant (microservices architecture detected, existing Pact config found):
|
||||
|
||||
Load: `contract-testing.md` (~960 lines)
|
||||
|
||||
**Detection**: Scan `{project-root}` for Pact indicators: `pact/` directory, `@pact-foundation/pact` in `package.json`, `pactUrls` in test files, `PACT_BROKER` in env files.
|
||||
|
||||
### Pact MCP Loading
|
||||
|
||||
**If `tea_pact_mcp` is `"mcp"`:**
|
||||
|
||||
Load: `pact-mcp.md` (~150 lines) — enables agent to use SmartBear MCP tools for fetching provider states and generating pact tests during automation.
|
||||
|
||||
## 4. Load Knowledge Base Fragments
|
||||
|
||||
Use `{knowledgeIndex}` and load only what is required.
|
||||
|
||||
**Core (always load):**
|
||||
|
||||
- `test-levels-framework.md`
|
||||
- `test-priorities-matrix.md`
|
||||
- `data-factories.md`
|
||||
- `selective-testing.md`
|
||||
- `ci-burn-in.md`
|
||||
- `test-quality.md`
|
||||
|
||||
**Playwright Utils (if enabled):**
|
||||
|
||||
- `overview.md`, `api-request.md`, `network-recorder.md`, `auth-session.md`, `intercept-network-call.md`, `recurse.md`, `log.md`, `file-utils.md`, `burn-in.md`, `network-error-monitor.md`, `fixtures-composition.md`
|
||||
|
||||
**Traditional Patterns (if Playwright Utils disabled):**
|
||||
|
||||
- `fixture-architecture.md`
|
||||
- `network-first.md`
|
||||
|
||||
**Pact.js Utils (if enabled):**
|
||||
|
||||
- `pactjs-utils-overview.md`, `pactjs-utils-consumer-helpers.md`, `pactjs-utils-provider-verifier.md`, `pactjs-utils-request-filter.md`
|
||||
|
||||
**Contract Testing (if pactjs-utils disabled but relevant):**
|
||||
|
||||
- `contract-testing.md`
|
||||
|
||||
**Pact MCP (if tea_pact_mcp is "mcp"):**
|
||||
|
||||
- `pact-mcp.md`
|
||||
|
||||
**Healing (if auto-heal enabled):**
|
||||
|
||||
- `test-healing-patterns.md`
|
||||
- `selector-resilience.md`
|
||||
- `timing-debugging.md`
|
||||
|
||||
**Playwright CLI (if tea_browser_automation is "cli" or "auto"):**
|
||||
|
||||
- `playwright-cli.md`
|
||||
|
||||
**MCP Patterns (if tea_browser_automation is "mcp" or "auto"):**
|
||||
|
||||
- (existing MCP-related fragments, if any are added in future)
|
||||
|
||||
---
|
||||
|
||||
## 5. Confirm Inputs
|
||||
|
||||
Summarize loaded artifacts, framework, and knowledge fragments, then proceed.
|
||||
|
||||
---
|
||||
|
||||
## 6. Save Progress
|
||||
|
||||
**Save this step's accumulated work to `{outputFile}`.**
|
||||
|
||||
- **If `{outputFile}` does not exist** (first save), create it with YAML frontmatter:
|
||||
|
||||
```yaml
|
||||
---
|
||||
stepsCompleted: ['step-01-preflight-and-context']
|
||||
lastStep: 'step-01-preflight-and-context'
|
||||
lastSaved: '{date}'
|
||||
---
|
||||
```
|
||||
|
||||
Then write this step's output below the frontmatter.
|
||||
|
||||
- **If `{outputFile}` already exists**, update:
|
||||
- Add `'step-01-preflight-and-context'` to `stepsCompleted` array (only if not already present)
|
||||
- Set `lastStep: 'step-01-preflight-and-context'`
|
||||
- Set `lastSaved: '{date}'`
|
||||
- Append this step's output to the appropriate section.
|
||||
|
||||
**Update `inputDocuments`**: Set `inputDocuments` in the output template frontmatter to the list of artifact paths loaded in this step (e.g., knowledge fragments, test design documents, configuration files).
|
||||
|
||||
Load next step: `{nextStepFile}`
|
||||
|
||||
## 🚨 SYSTEM SUCCESS/FAILURE METRICS:
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- Step completed in full with required outputs
|
||||
|
||||
### ❌ SYSTEM FAILURE:
|
||||
|
||||
- Skipped sequence steps or missing outputs
|
||||
**Master Rule:** Skipping steps is FORBIDDEN.
|
||||
@@ -0,0 +1,94 @@
|
||||
---
|
||||
name: 'step-01b-resume'
|
||||
description: 'Resume interrupted workflow from last completed step'
|
||||
outputFile: '{test_artifacts}/automation-summary.md'
|
||||
---
|
||||
|
||||
# Step 1b: Resume Workflow
|
||||
|
||||
## STEP GOAL
|
||||
|
||||
Resume an interrupted workflow by loading the existing output document, displaying progress, and routing to the next incomplete step.
|
||||
|
||||
## MANDATORY EXECUTION RULES
|
||||
|
||||
- 📖 Read the entire step file before acting
|
||||
- ✅ Speak in `{communication_language}`
|
||||
|
||||
---
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
- 📖 Load the next step only when instructed
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: Output document with progress frontmatter
|
||||
- Focus: Load progress and route to next step
|
||||
- Limits: Do not re-execute completed steps
|
||||
- Dependencies: Output document must exist from a previous run
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly.
|
||||
|
||||
### 1. Load Output Document
|
||||
|
||||
Read `{outputFile}` and parse YAML frontmatter for:
|
||||
|
||||
- `stepsCompleted` — array of completed step names
|
||||
- `lastStep` — last completed step name
|
||||
- `lastSaved` — timestamp of last save
|
||||
|
||||
**If `{outputFile}` does not exist**, display:
|
||||
|
||||
"⚠️ **No previous progress found.** There is no output document to resume from. Please use **[C] Create** to start a fresh workflow run."
|
||||
|
||||
**THEN:** Halt. Do not proceed.
|
||||
|
||||
---
|
||||
|
||||
### 2. Display Progress Dashboard
|
||||
|
||||
Display progress with ✅/⬜ indicators:
|
||||
|
||||
1. ✅/⬜ Preflight & Context (step-01-preflight-and-context)
|
||||
2. ✅/⬜ Identify Targets (step-02-identify-targets)
|
||||
3. ✅/⬜ Generate Tests + Aggregate (step-03c-aggregate)
|
||||
4. ✅/⬜ Validate & Summarize (step-04-validate-and-summarize)
|
||||
|
||||
---
|
||||
|
||||
### 3. Route to Next Step
|
||||
|
||||
Based on `lastStep`, load the next incomplete step:
|
||||
|
||||
- `'step-01-preflight-and-context'` → load `./step-02-identify-targets.md`
|
||||
- `'step-02-identify-targets'` → load `./step-03-generate-tests.md`
|
||||
- `'step-03c-aggregate'` → load `./step-04-validate-and-summarize.md`
|
||||
- `'step-04-validate-and-summarize'` → **Workflow already complete.** Display: "✅ **All steps completed.** Use **[V] Validate** to review outputs or **[E] Edit** to make revisions." Then halt.
|
||||
|
||||
**If `lastStep` does not match any value above**, display: "⚠️ **Unknown progress state** (`lastStep`: {lastStep}). Please use **[C] Create** to start fresh." Then halt.
|
||||
|
||||
**Otherwise**, load the identified step file, read completely, and execute.
|
||||
|
||||
The existing content in `{outputFile}` provides context from previously completed steps.
|
||||
|
||||
---
|
||||
|
||||
## 🚨 SYSTEM SUCCESS/FAILURE METRICS
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- Output document loaded and parsed correctly
|
||||
- Progress dashboard displayed accurately
|
||||
- Routed to correct next step
|
||||
|
||||
### ❌ SYSTEM FAILURE:
|
||||
|
||||
- Not loading output document
|
||||
- Incorrect progress display
|
||||
- Routing to wrong step
|
||||
|
||||
**Master Rule:** Resume MUST route to the exact next incomplete step. Never re-execute completed steps.
|
||||
@@ -0,0 +1,169 @@
|
||||
---
|
||||
name: 'step-02-identify-targets'
|
||||
description: 'Identify automation targets and create coverage plan'
|
||||
outputFile: '{test_artifacts}/automation-summary.md'
|
||||
nextStepFile: './step-03-generate-tests.md'
|
||||
---
|
||||
|
||||
# Step 2: Identify Automation Targets
|
||||
|
||||
## STEP GOAL
|
||||
|
||||
Determine what needs to be tested and select appropriate test levels and priorities.
|
||||
|
||||
## MANDATORY EXECUTION RULES
|
||||
|
||||
- 📖 Read the entire step file before acting
|
||||
- ✅ Speak in `{communication_language}`
|
||||
- 🚫 Avoid duplicate coverage across test levels
|
||||
|
||||
---
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
- 💾 Record outputs before proceeding
|
||||
- 📖 Load the next step only when instructed
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: config, loaded artifacts, and knowledge fragments
|
||||
- Focus: this step's goal only
|
||||
- Limits: do not execute future steps
|
||||
- Dependencies: prior steps' outputs (if any)
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly. Do not skip, reorder, or improvise.
|
||||
|
||||
## 1. Determine Targets
|
||||
|
||||
**BMad-Integrated:**
|
||||
|
||||
- Map acceptance criteria to test scenarios
|
||||
- Check for existing ATDD outputs to avoid duplication
|
||||
- Expand coverage with edge cases and negative paths
|
||||
|
||||
**Standalone:**
|
||||
|
||||
- If specific target feature/files are provided, focus there
|
||||
- Otherwise auto-discover features in `{source_dir}`
|
||||
- Prioritize critical paths, integrations, and untested logic
|
||||
|
||||
**If {detected_stack} is `frontend` or `fullstack`:**
|
||||
|
||||
**Browser Exploration (if `tea_browser_automation` is `cli` or `auto`):**
|
||||
|
||||
> **Fallback:** If CLI is not installed, fall back to MCP (if available) or skip browser exploration and rely on code/doc analysis.
|
||||
|
||||
Use CLI to explore the application and identify testable pages/flows:
|
||||
|
||||
1. `playwright-cli -s=tea-automate open <target_url>`
|
||||
2. `playwright-cli -s=tea-automate snapshot` → capture page structure and element refs
|
||||
3. Analyze snapshot output to identify testable elements and flows
|
||||
4. `playwright-cli -s=tea-automate close`
|
||||
|
||||
> **Session Hygiene:** Always close sessions using `playwright-cli -s=tea-automate close`. Do NOT use `close-all` — it kills every session on the machine and breaks parallel execution.
|
||||
|
||||
**If {detected_stack} is `backend` or `fullstack`:**
|
||||
|
||||
**Source & API Analysis (no browser exploration):**
|
||||
|
||||
- Scan source code for route handlers, controllers, service classes, and public APIs
|
||||
- Read OpenAPI/Swagger specs (`openapi.yaml`, `swagger.json`) if available
|
||||
- Identify database models, migrations, and data access patterns
|
||||
- Map service-to-service integrations and message queue consumers/producers
|
||||
- Check for existing contract tests (Pact, etc.)
|
||||
|
||||
---
|
||||
|
||||
**If `use_pactjs_utils` is enabled — Provider Endpoint Mapping (all stacks):**
|
||||
|
||||
When consumer-driven contract tests will be generated, build a Provider Endpoint Map during target identification. This applies to all `{detected_stack}` values — frontend, backend, and fullstack consumers all need provider scrutiny.
|
||||
|
||||
1. **Locate provider source and/or OpenAPI spec**: Scan workspace for provider project (from config, monorepo structure, or adjacent repositories). Also check for OpenAPI/Swagger spec files (`openapi.yaml`, `openapi.json`, `swagger.json`) — these document the provider's contract explicitly and can supplement or replace handler code analysis.
|
||||
2. **Map each consumer endpoint** to its provider counterpart:
|
||||
- Provider file path (route handler)
|
||||
- Route pattern (METHOD + path)
|
||||
- Validation schema location (Joi, Zod, class-validator) or OpenAPI request schema
|
||||
- Response type/DTO definition location or OpenAPI response schema
|
||||
- OpenAPI spec path (if available, e.g., `server/openapi.yaml`)
|
||||
3. **Output as "Provider Endpoint Map" table** in the coverage plan:
|
||||
|
||||
```markdown
|
||||
| Consumer Endpoint | Provider File | Route | Validation Schema | Response Type | OpenAPI Spec |
|
||||
| --------------------- | --------------------------------- | ------------------------- | ----------------------------------- | --------------- | ------------------------------------------------- |
|
||||
| GET /api/v2/users/:id | server/src/routes/userHandlers.ts | GET /api/v2/users/:userId | server/src/validation/user.ts | UserResponseDto | server/openapi.yaml#/paths/~1api~1v2~1users~1{id} |
|
||||
| POST /api/v2/users | server/src/routes/userHandlers.ts | POST /api/v2/users | server/src/validation/createUser.ts | UserResponseDto | server/openapi.yaml#/paths/~1api~1v2~1users |
|
||||
```
|
||||
|
||||
4. **If provider source not accessible**: Mark entries with `TODO — provider source not accessible` and note in coverage plan that provider scrutiny will use graceful degradation (see `contract-testing.md` Provider Scrutiny Protocol)
|
||||
|
||||
---
|
||||
|
||||
## 2. Choose Test Levels
|
||||
|
||||
Use `test-levels-framework.md` to select:
|
||||
|
||||
- **E2E** for critical user journeys
|
||||
- **API** for business logic and service contracts
|
||||
- **Component** for UI behavior
|
||||
- **Unit** for pure logic and edge cases
|
||||
|
||||
---
|
||||
|
||||
## 3. Assign Priorities
|
||||
|
||||
Use `test-priorities-matrix.md`:
|
||||
|
||||
- P0: Critical path + high risk
|
||||
- P1: Important flows + medium/high risk
|
||||
- P2: Secondary + edge cases
|
||||
- P3: Optional/rare scenarios
|
||||
|
||||
---
|
||||
|
||||
## 4. Coverage Plan
|
||||
|
||||
Produce a concise coverage plan:
|
||||
|
||||
- Targets by test level
|
||||
- Priority assignments
|
||||
- Justification for coverage scope (critical-paths/comprehensive/selective)
|
||||
|
||||
---
|
||||
|
||||
## 5. Save Progress
|
||||
|
||||
**Save this step's accumulated work to `{outputFile}`.**
|
||||
|
||||
- **If `{outputFile}` does not exist** (first save), create it with YAML frontmatter:
|
||||
|
||||
```yaml
|
||||
---
|
||||
stepsCompleted: ['step-02-identify-targets']
|
||||
lastStep: 'step-02-identify-targets'
|
||||
lastSaved: '{date}'
|
||||
---
|
||||
```
|
||||
|
||||
Then write this step's output below the frontmatter.
|
||||
|
||||
- **If `{outputFile}` already exists**, update:
|
||||
- Add `'step-02-identify-targets'` to `stepsCompleted` array (only if not already present)
|
||||
- Set `lastStep: 'step-02-identify-targets'`
|
||||
- Set `lastSaved: '{date}'`
|
||||
- Append this step's output to the appropriate section.
|
||||
|
||||
Load next step: `{nextStepFile}`
|
||||
|
||||
## 🚨 SYSTEM SUCCESS/FAILURE METRICS:
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- Step completed in full with required outputs
|
||||
|
||||
### ❌ SYSTEM FAILURE:
|
||||
|
||||
- Skipped sequence steps or missing outputs
|
||||
**Master Rule:** Skipping steps is FORBIDDEN.
|
||||
@@ -0,0 +1,394 @@
|
||||
---
|
||||
name: 'step-03-generate-tests'
|
||||
description: 'Orchestrate adaptive test generation (agent-team, subagent, or sequential)'
|
||||
nextStepFile: './step-03c-aggregate.md'
|
||||
---
|
||||
|
||||
# Step 3: Orchestrate Adaptive Test Generation
|
||||
|
||||
## STEP GOAL
|
||||
|
||||
Select execution mode deterministically, then generate tests using agent-team, subagent, or sequential execution while preserving the same output contract. Worker selection depends on `{detected_stack}`.
|
||||
|
||||
## MANDATORY EXECUTION RULES
|
||||
|
||||
- 📖 Read the entire step file before acting
|
||||
- ✅ Speak in `{communication_language}`
|
||||
- ✅ Resolve execution mode from config (`tea_execution_mode`, `tea_capability_probe`)
|
||||
- ✅ Apply fallback rules deterministically when requested mode is unsupported
|
||||
- ✅ Preserve output schema and temp file naming across all modes
|
||||
- ❌ Do NOT skip capability checks when probing is enabled
|
||||
- ❌ Do NOT change output paths or JSON schema by mode
|
||||
|
||||
---
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
- 💾 Wait for subagent outputs
|
||||
- 📖 Load the next step only when instructed
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: config, coverage plan from Step 2, knowledge fragments
|
||||
- Focus: orchestration only (mode selection + worker dispatch)
|
||||
- Limits: do not generate tests directly (delegate to worker steps)
|
||||
- Dependencies: Step 2 outputs (coverage plan, target features)
|
||||
|
||||
---
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly. Do not skip, reorder, or improvise.
|
||||
|
||||
### 1. Prepare Execution Context
|
||||
|
||||
**Generate unique timestamp** for temp file naming:
|
||||
|
||||
```javascript
|
||||
const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
|
||||
```
|
||||
|
||||
**Prepare input context for subagents:**
|
||||
|
||||
```javascript
|
||||
const parseBooleanFlag = (value, defaultValue = true) => {
|
||||
if (typeof value === 'string') {
|
||||
const normalized = value.trim().toLowerCase();
|
||||
if (['false', '0', 'off', 'no'].includes(normalized)) return false;
|
||||
if (['true', '1', 'on', 'yes'].includes(normalized)) return true;
|
||||
}
|
||||
if (value === undefined || value === null) return defaultValue;
|
||||
return Boolean(value);
|
||||
};
|
||||
|
||||
const subagentContext = {
|
||||
features: /* from Step 2 coverage plan */,
|
||||
knowledge_fragments_loaded: /* list of fragments */,
|
||||
config: {
|
||||
test_framework: config.test_framework,
|
||||
use_playwright_utils: config.tea_use_playwright_utils,
|
||||
use_pactjs_utils: config.tea_use_pactjs_utils,
|
||||
pact_mcp: config.tea_pact_mcp, // "mcp" | "none"
|
||||
browser_automation: config.tea_browser_automation, // "auto" | "cli" | "mcp" | "none"
|
||||
detected_stack: '{detected_stack}', // "frontend" | "backend" | "fullstack"
|
||||
execution_mode: config.tea_execution_mode || 'auto', // "auto" | "subagent" | "agent-team" | "sequential"
|
||||
capability_probe: parseBooleanFlag(config.tea_capability_probe, true), // supports booleans and "false"/"true" strings
|
||||
provider_endpoint_map: /* from Step 2 coverage plan, if use_pactjs_utils enabled */,
|
||||
},
|
||||
timestamp: timestamp
|
||||
};
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 2. Resolve Execution Mode with Capability Probe
|
||||
|
||||
```javascript
|
||||
const normalizeUserExecutionMode = (mode) => {
|
||||
if (typeof mode !== 'string') return null;
|
||||
const normalized = mode.trim().toLowerCase().replace(/[-_]/g, ' ').replace(/\s+/g, ' ');
|
||||
|
||||
if (normalized === 'auto') return 'auto';
|
||||
if (normalized === 'sequential') return 'sequential';
|
||||
if (normalized === 'subagent' || normalized === 'sub agent' || normalized === 'subagents' || normalized === 'sub agents') {
|
||||
return 'subagent';
|
||||
}
|
||||
if (normalized === 'agent team' || normalized === 'agent teams' || normalized === 'agentteam') {
|
||||
return 'agent-team';
|
||||
}
|
||||
|
||||
return null;
|
||||
};
|
||||
|
||||
const normalizeConfigExecutionMode = (mode) => {
|
||||
if (mode === 'subagent') return 'subagent';
|
||||
if (mode === 'auto' || mode === 'sequential' || mode === 'subagent' || mode === 'agent-team') {
|
||||
return mode;
|
||||
}
|
||||
return null;
|
||||
};
|
||||
|
||||
// Explicit user instruction in the active run takes priority over config.
|
||||
const explicitModeFromUser = normalizeUserExecutionMode(runtime.getExplicitExecutionModeHint?.() || null);
|
||||
|
||||
const requestedMode = explicitModeFromUser || normalizeConfigExecutionMode(subagentContext.config.execution_mode) || 'auto';
|
||||
const probeEnabled = subagentContext.config.capability_probe;
|
||||
|
||||
const supports = {
|
||||
subagent: false,
|
||||
agentTeam: false,
|
||||
};
|
||||
|
||||
if (probeEnabled) {
|
||||
// Probe using runtime-native capability checks or a no-op launch test.
|
||||
supports.subagent = runtime.canLaunchSubagents?.() === true;
|
||||
supports.agentTeam = runtime.canLaunchAgentTeams?.() === true;
|
||||
}
|
||||
|
||||
let resolvedMode = requestedMode;
|
||||
|
||||
if (requestedMode === 'auto') {
|
||||
if (supports.agentTeam) resolvedMode = 'agent-team';
|
||||
else if (supports.subagent) resolvedMode = 'subagent';
|
||||
else resolvedMode = 'sequential';
|
||||
} else if (probeEnabled && requestedMode === 'agent-team' && !supports.agentTeam) {
|
||||
resolvedMode = supports.subagent ? 'subagent' : 'sequential';
|
||||
} else if (probeEnabled && requestedMode === 'subagent' && !supports.subagent) {
|
||||
resolvedMode = 'sequential';
|
||||
}
|
||||
|
||||
subagentContext.execution = {
|
||||
requestedMode,
|
||||
resolvedMode,
|
||||
probeEnabled,
|
||||
supports,
|
||||
};
|
||||
```
|
||||
|
||||
Resolution precedence:
|
||||
|
||||
1. Explicit user request in this run (`agent team` => `agent-team`; `subagent` => `subagent`; `sequential`; `auto`)
|
||||
2. `tea_execution_mode` from config
|
||||
3. Runtime capability fallback (when probing enabled)
|
||||
|
||||
If probing is disabled, honor the requested mode strictly. If that mode cannot be executed at runtime, fail with explicit error instead of silent fallback.
|
||||
|
||||
Report selected mode before dispatch:
|
||||
|
||||
```
|
||||
⚙️ Execution Mode Resolution:
|
||||
- Requested: {requestedMode}
|
||||
- Probe Enabled: {probeEnabled}
|
||||
- Supports agent-team: {supports.agentTeam}
|
||||
- Supports subagent: {supports.subagent}
|
||||
- Resolved: {resolvedMode}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 3. Subagent Dispatch Matrix
|
||||
|
||||
**Select subagents based on `{detected_stack}`:**
|
||||
|
||||
| `{detected_stack}` | Subagent A (API) | Subagent B (E2E) | Subagent B-backend |
|
||||
| ------------------ | ---------------- | ---------------- | ------------------ |
|
||||
| `frontend` | Launch | Launch | Skip |
|
||||
| `backend` | Launch | Skip | Launch |
|
||||
| `fullstack` | Launch | Launch | Launch |
|
||||
|
||||
### 3A. Runtime-Managed Parallelism
|
||||
|
||||
When `resolvedMode` is `agent-team` or `subagent`, let the runtime decide concurrency and scheduling. TEA does not impose an additional worker ceiling.
|
||||
|
||||
---
|
||||
|
||||
### Contract Test Generation Note
|
||||
|
||||
When `use_pactjs_utils` is enabled, the API test generation subagent (step-03a) also generates:
|
||||
|
||||
- **Consumer contract tests**: Using `createProviderState` for type-safe provider states
|
||||
- **Provider verification tests**: Using `buildVerifierOptions` for one-call verifier setup
|
||||
- **Message contract tests**: Using `buildMessageVerifierOptions` if async/Kafka patterns detected
|
||||
- **Helper files**: Request filter setup with `createRequestFilter`, shared state constants
|
||||
- **Provider scrutiny**: Subagent reads provider route handlers, types, and validation schemas before generating each interaction (see `contract-testing.md` Provider Scrutiny Protocol)
|
||||
|
||||
When `pact_mcp` is `"mcp"`, the subagent can use SmartBear MCP tools to fetch existing provider states and generate tests informed by broker data.
|
||||
|
||||
---
|
||||
|
||||
### 4. Dispatch Worker A: API Test Generation (always)
|
||||
|
||||
**Dispatch worker:**
|
||||
|
||||
- **Subagent File:** `./step-03a-subagent-api.md`
|
||||
- **Output File:** `/tmp/tea-automate-api-tests-${timestamp}.json`
|
||||
- **Context:** Pass `subagentContext`
|
||||
- **Execution:**
|
||||
- `agent-team` or `subagent`: launch non-blocking
|
||||
- `sequential`: run blocking and wait before next dispatch
|
||||
|
||||
**System Action:**
|
||||
|
||||
```
|
||||
🚀 Launching Subagent A: API Test Generation
|
||||
📝 Output: /tmp/tea-automate-api-tests-${timestamp}.json
|
||||
⚙️ Mode: ${resolvedMode}
|
||||
⏳ Status: Running...
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 5. Dispatch Worker B: E2E Test Generation (frontend/fullstack only)
|
||||
|
||||
**If {detected_stack} is `frontend` or `fullstack`:**
|
||||
|
||||
**Dispatch worker:**
|
||||
|
||||
- **Subagent File:** `./step-03b-subagent-e2e.md`
|
||||
- **Output File:** `/tmp/tea-automate-e2e-tests-${timestamp}.json`
|
||||
- **Context:** Pass `subagentContext`
|
||||
- **Execution:**
|
||||
- `agent-team` or `subagent`: launch non-blocking
|
||||
- `sequential`: run blocking and wait before next dispatch
|
||||
|
||||
**System Action:**
|
||||
|
||||
```
|
||||
🚀 Launching Subagent B: E2E Test Generation
|
||||
📝 Output: /tmp/tea-automate-e2e-tests-${timestamp}.json
|
||||
⚙️ Mode: ${resolvedMode}
|
||||
⏳ Status: Running...
|
||||
```
|
||||
|
||||
**If {detected_stack} is `backend`:** Skip this subagent.
|
||||
|
||||
---
|
||||
|
||||
### 6. Dispatch Worker B-backend: Backend Test Generation (backend/fullstack only)
|
||||
|
||||
**If {detected_stack} is `backend` or `fullstack`:**
|
||||
|
||||
**Dispatch worker:**
|
||||
|
||||
- **Subagent File:** `./step-03b-subagent-backend.md`
|
||||
- **Output File:** `/tmp/tea-automate-backend-tests-${timestamp}.json`
|
||||
- **Context:** Pass `subagentContext`
|
||||
- **Execution:**
|
||||
- `agent-team` or `subagent`: launch non-blocking
|
||||
- `sequential`: run blocking and wait before next dispatch
|
||||
|
||||
**System Action:**
|
||||
|
||||
```
|
||||
🚀 Launching Subagent B-backend: Backend Test Generation
|
||||
📝 Output: /tmp/tea-automate-backend-tests-${timestamp}.json
|
||||
⚙️ Mode: ${resolvedMode}
|
||||
⏳ Status: Running...
|
||||
```
|
||||
|
||||
**If {detected_stack} is `frontend`:** Skip this subagent.
|
||||
|
||||
---
|
||||
|
||||
### 7. Wait for Expected Worker Completion
|
||||
|
||||
**If `resolvedMode` is `agent-team` or `subagent`:**
|
||||
|
||||
```
|
||||
⏳ Waiting for subagents to complete...
|
||||
├── Subagent A (API): Running... ⟳
|
||||
├── Subagent B (E2E): Running... ⟳ [if frontend/fullstack]
|
||||
└── Subagent B-backend: Running... ⟳ [if backend/fullstack]
|
||||
|
||||
[... time passes ...]
|
||||
|
||||
├── Subagent A (API): Complete ✅
|
||||
├── Subagent B (E2E): Complete ✅ [if frontend/fullstack]
|
||||
└── Subagent B-backend: Complete ✅ [if backend/fullstack]
|
||||
|
||||
✅ All subagents completed successfully!
|
||||
```
|
||||
|
||||
**If `resolvedMode` is `sequential`:**
|
||||
|
||||
```
|
||||
✅ Sequential mode: each worker already completed during dispatch.
|
||||
```
|
||||
|
||||
**Verify outputs exist (based on `{detected_stack}`):**
|
||||
|
||||
```javascript
|
||||
const apiOutputExists = fs.existsSync(`/tmp/tea-automate-api-tests-${timestamp}.json`);
|
||||
|
||||
// Check based on detected_stack
|
||||
if (detected_stack === 'frontend' || detected_stack === 'fullstack') {
|
||||
const e2eOutputExists = fs.existsSync(`/tmp/tea-automate-e2e-tests-${timestamp}.json`);
|
||||
if (!e2eOutputExists) throw new Error('E2E subagent output missing!');
|
||||
}
|
||||
if (detected_stack === 'backend' || detected_stack === 'fullstack') {
|
||||
const backendOutputExists = fs.existsSync(`/tmp/tea-automate-backend-tests-${timestamp}.json`);
|
||||
if (!backendOutputExists) throw new Error('Backend subagent output missing!');
|
||||
}
|
||||
if (!apiOutputExists) throw new Error('API subagent output missing!');
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Subagent Output Schema Contract
|
||||
|
||||
The aggregate step expects both outputs to include `success`, but the payload shapes are intentionally different:
|
||||
|
||||
- `step-03b-subagent-e2e.md` output includes `success`, `subagent`, `tests`, `fixture_needs`, `knowledge_fragments_used`, `test_count`, and `summary`.
|
||||
- `step-03b-subagent-backend.md` output includes `success`, `subagent`, `subagentType`, `testsGenerated`, `coverageSummary` (with `fixtureNeeds`), `status`, `knowledge_fragments_used`, and `summary`.
|
||||
|
||||
The aggregate step reads whichever output file(s) exist based on `{detected_stack}` and must use the matching schema per subagent type.
|
||||
|
||||
---
|
||||
|
||||
### 8. Execution Report
|
||||
|
||||
**Display performance metrics:**
|
||||
|
||||
```
|
||||
🚀 Performance Report:
|
||||
- Execution Mode: {resolvedMode}
|
||||
- Stack Type: {detected_stack}
|
||||
- API Test Generation: ~X minutes
|
||||
- E2E Test Generation: ~Y minutes [if frontend/fullstack]
|
||||
- Backend Test Generation: ~Z minutes [if backend/fullstack]
|
||||
- Total Elapsed: ~mode-dependent
|
||||
- Parallel Gain: ~40-70% faster when mode is subagent/agent-team
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 9. Proceed to Aggregation
|
||||
|
||||
**Load aggregation step:**
|
||||
Load next step: `{nextStepFile}`
|
||||
|
||||
The aggregation step (3C) will:
|
||||
|
||||
- Read all subagent outputs (based on `{detected_stack}`)
|
||||
- Write all test files to disk
|
||||
- Generate shared fixtures and helpers
|
||||
- Calculate summary statistics
|
||||
|
||||
---
|
||||
|
||||
## EXIT CONDITION
|
||||
|
||||
Proceed to Step 3C (Aggregation) when:
|
||||
|
||||
- ✅ Subagent A (API tests) completed successfully
|
||||
- ✅ Subagent B (E2E tests) completed successfully [if frontend/fullstack]
|
||||
- ✅ Subagent B-backend (Backend tests) completed successfully [if backend/fullstack]
|
||||
- ✅ All expected output files exist and are valid JSON
|
||||
- ✅ Execution metrics displayed
|
||||
|
||||
**Do NOT proceed if:**
|
||||
|
||||
- ❌ Any launched subagent failed
|
||||
- ❌ Output files missing or corrupted
|
||||
- ❌ Timeout occurred (parallel mode only)
|
||||
|
||||
---
|
||||
|
||||
## 🚨 SYSTEM SUCCESS/FAILURE METRICS:
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- All required subagents launched successfully (based on `{detected_stack}`)
|
||||
- All required worker steps completed without errors
|
||||
- Output files generated and valid
|
||||
- Fallback behavior respected configuration and capability probe rules
|
||||
|
||||
### ❌ SYSTEM FAILURE:
|
||||
|
||||
- Failed to launch subagents
|
||||
- One or more subagents failed
|
||||
- Output files missing or invalid
|
||||
- Unsupported requested mode with probing disabled
|
||||
|
||||
**Master Rule:** Deterministic mode selection + stable output contract. Use the best supported mode, then aggregate normally.
|
||||
@@ -0,0 +1,263 @@
|
||||
---
|
||||
name: 'step-03a-subagent-api'
|
||||
description: 'Subagent: Generate API tests only'
|
||||
subagent: true
|
||||
outputFile: '/tmp/tea-automate-api-tests-{{timestamp}}.json'
|
||||
---
|
||||
|
||||
# Subagent 3A: Generate API Tests
|
||||
|
||||
## SUBAGENT CONTEXT
|
||||
|
||||
This is an **isolated subagent** running in parallel with E2E test generation.
|
||||
|
||||
**What you have from parent workflow:**
|
||||
|
||||
- Target features/components identified in Step 2
|
||||
- Knowledge fragments loaded: api-request, data-factories, api-testing-patterns
|
||||
- Config: test framework, Playwright Utils enabled/disabled, Pact.js Utils enabled/disabled, Pact MCP mode
|
||||
- Coverage plan: which API endpoints need testing
|
||||
|
||||
**Your task:** Generate API tests ONLY (not E2E, not fixtures, not other test types).
|
||||
|
||||
**If `use_pactjs_utils` is enabled:** Also generate consumer contract tests and provider verification tests alongside API tests. Use the loaded pactjs-utils fragments (`pactjs-utils-overview`, `pactjs-utils-consumer-helpers`, `pactjs-utils-provider-verifier`, `pactjs-utils-request-filter`) for patterns. If `pact_mcp` is `"mcp"`, use SmartBear MCP tools (Fetch Provider States, Generate Pact Tests) to inform test generation.
|
||||
|
||||
---
|
||||
|
||||
## MANDATORY EXECUTION RULES
|
||||
|
||||
- 📖 Read this entire subagent file before acting
|
||||
- ✅ Generate API tests ONLY
|
||||
- ✅ Output structured JSON to temp file
|
||||
- ✅ Follow knowledge fragment patterns
|
||||
- ❌ Do NOT generate E2E tests (that's subagent 3B)
|
||||
- ❌ Do NOT run tests (that's step 4)
|
||||
- ❌ Do NOT generate fixtures yet (that's step 3C aggregation)
|
||||
|
||||
---
|
||||
|
||||
## SUBAGENT TASK
|
||||
|
||||
### 1. Identify API Endpoints
|
||||
|
||||
From the coverage plan (Step 2 output), identify:
|
||||
|
||||
- Which API endpoints need test coverage
|
||||
- Expected request/response formats
|
||||
- Authentication requirements
|
||||
- Error scenarios to test
|
||||
|
||||
### 2. Generate API Test Files
|
||||
|
||||
For each API endpoint, create test file in `tests/api/[feature].spec.ts`:
|
||||
|
||||
**Test Structure:**
|
||||
|
||||
```typescript
|
||||
import { test, expect } from '@playwright/test';
|
||||
// If Playwright Utils enabled:
|
||||
// import { apiRequest } from '@playwright-utils/api';
|
||||
|
||||
test.describe('[Feature] API Tests', () => {
|
||||
test('[P0] should handle successful [operation]', async ({ request }) => {
|
||||
// Use apiRequest helper if Playwright Utils enabled
|
||||
// Otherwise use standard request fixture
|
||||
const response = await request.post('/api/endpoint', {
|
||||
data: {
|
||||
/* test data */
|
||||
},
|
||||
});
|
||||
|
||||
expect(response.status()).toBe(200);
|
||||
expect(await response.json()).toMatchObject({
|
||||
/* expected */
|
||||
});
|
||||
});
|
||||
|
||||
test('[P1] should handle [error scenario]', async ({ request }) => {
|
||||
// Test error handling
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
**Requirements:**
|
||||
|
||||
- ✅ Use `apiRequest()` helper if Playwright Utils enabled (from api-request fragment)
|
||||
- ✅ Use data factories for test data (from data-factories fragment)
|
||||
- ✅ Follow API testing patterns (from api-testing-patterns fragment)
|
||||
- ✅ Include priority tags [P0], [P1], [P2], [P3]
|
||||
- ✅ Test both happy path and error scenarios
|
||||
- ✅ Use proper TypeScript types
|
||||
- ✅ Deterministic assertions (no timing dependencies)
|
||||
|
||||
**If Pact.js Utils enabled (from `subagentContext.config.use_pactjs_utils`):**
|
||||
|
||||
- ✅ Generate consumer contract tests in `pact/http/consumer/` using `createProviderState({ name, params })` pattern
|
||||
- ✅ Generate provider verification tests in `pact/http/provider/` using `buildVerifierOptions({ provider, port, includeMainAndDeployed, stateHandlers })` pattern
|
||||
- ✅ Generate request filter helpers in `pact/http/helpers/` using `createRequestFilter({ tokenGenerator: () => string })`
|
||||
- ✅ Generate shared state constants in `pact/http/helpers/states.ts`
|
||||
- ✅ If async/message patterns detected, generate message consumer tests in `pact/message/` using `buildMessageVerifierOptions`
|
||||
- ✅ **Provider endpoint comment MANDATORY** on every Pact interaction: `// Provider endpoint: <path> -> <METHOD> <route>`
|
||||
- ⚠️ **Postel's Law for matchers**: Use `like()`, `eachLike()`, `string()`, `integer()` matchers ONLY in `willRespondWith` (responses). Request bodies in `withRequest` MUST use exact values — never wrap request bodies in `like()`. The consumer controls what it sends, so contracts should be strict about request shape.
|
||||
|
||||
### 1.5 Provider Source Scrutiny (CDC Only)
|
||||
|
||||
**CRITICAL**: Before generating ANY Pact consumer interaction, perform provider source scrutiny per the **Seven-Point Scrutiny Checklist** defined in `contract-testing.md`. Do NOT generate response matchers from consumer-side types alone — this is the #1 cause of contract verification failures.
|
||||
|
||||
The seven points to verify for each interaction:
|
||||
|
||||
1. Response shape
|
||||
2. Status codes
|
||||
3. Field names
|
||||
4. Enum values
|
||||
5. Required fields
|
||||
6. Data types
|
||||
7. Nested structures
|
||||
|
||||
**Source priority**: Provider source code is most authoritative. When an OpenAPI/Swagger spec exists (`openapi.yaml`, `openapi.json`, `swagger.json`), use it as a complementary or alternative source — it documents the provider's contract explicitly and can be faster to parse than tracing through handler code. When both exist, cross-reference them; if they disagree, the source code wins. Document the discrepancy in the scrutiny evidence block (e.g., `OpenAPI shows 200 but handler returns 201; using handler behavior`) and flag it in the output JSON `summary` so it is discoverable by downstream consumers or audits.
|
||||
|
||||
**Scrutiny Sequence** (for each endpoint in the coverage plan):
|
||||
|
||||
1. **READ provider route handler and/or OpenAPI spec**: Find the handler file from `subagentContext.config.provider_endpoint_map` or by scanning the provider codebase. Also check for OpenAPI/Swagger spec files. Extract:
|
||||
- Exact status codes returned (`res.status(201)` / OpenAPI `responses` keys)
|
||||
- Response construction (`res.json({ data: ... })` / OpenAPI `schema`)
|
||||
- Error handling paths (what status codes for what conditions)
|
||||
|
||||
2. **READ provider type/model/DTO definitions**: Find the response type referenced by the handler or OpenAPI `$ref` schemas. Extract:
|
||||
- Exact field names (`transaction_id` not `transactionId`)
|
||||
- Field types (`string` ID vs `number` ID / OpenAPI `type` + `format`)
|
||||
- Optional vs required fields (OpenAPI `required` array)
|
||||
- Nested object structures (OpenAPI `$ref`, `allOf`, `oneOf`)
|
||||
|
||||
3. **READ provider validation schemas**: Find Joi/Zod/class-validator schemas or OpenAPI request body `schema.required`. Extract:
|
||||
- Required request fields and headers
|
||||
- Enum/union type allowed values (`"active" | "inactive"` / OpenAPI `enum`)
|
||||
- Request body constraints
|
||||
|
||||
4. **Cross-reference findings** against consumer expectations:
|
||||
- Does the consumer expect the same field names the provider sends?
|
||||
- Does the consumer expect the same status codes the provider returns?
|
||||
- Does the consumer expect the same nesting the provider produces?
|
||||
|
||||
5. **Document scrutiny evidence** as a block comment in the generated test:
|
||||
|
||||
```typescript
|
||||
/*
|
||||
* Provider Scrutiny Evidence:
|
||||
* - Handler: server/src/routes/userHandlers.ts:45
|
||||
* - OpenAPI: server/openapi.yaml paths./api/v2/users/{userId}.get (if available)
|
||||
* - Response type: UserResponseDto (server/src/types/user.ts:12)
|
||||
* - Status: 201 for creation (line 52), 400 for validation error (line 48)
|
||||
* - Fields: { id: number, name: string, email: string, role: "user" | "admin" }
|
||||
* - Required request headers: Authorization (Bearer token)
|
||||
*/
|
||||
```
|
||||
|
||||
6. **Graceful degradation** when provider source is not accessible (follows the canonical four-step protocol from `contract-testing.md`):
|
||||
1. **OpenAPI/Swagger spec available**: Use the spec as the source of truth for response shapes, status codes, and field names
|
||||
2. **Pact Broker available** (when `pact_mcp` is `"mcp"` in `subagentContext.config`): Use SmartBear MCP tools to fetch existing provider states and verified interactions as reference
|
||||
3. **Neither available**: Generate from consumer types but use the TODO form of the mandatory comment: `// Provider endpoint: TODO — provider source not accessible, verify manually`. Set `provider_scrutiny: "pending"` in output JSON
|
||||
4. **Never silently guess**: Document all assumptions in the scrutiny evidence block
|
||||
|
||||
> ⚠️ **Anti-pattern**: Generating response matchers from consumer-side types alone. This produces contracts that reflect what the consumer _wishes_ the provider returns, not what it _actually_ returns. Always read provider source or OpenAPI spec first.
|
||||
|
||||
### 3. Track Fixture Needs
|
||||
|
||||
Identify fixtures needed for API tests:
|
||||
|
||||
- Authentication fixtures (auth tokens, API keys)
|
||||
- Data factories (user data, product data, etc.)
|
||||
- API client configurations
|
||||
|
||||
**Do NOT create fixtures yet** - just track what's needed for aggregation step.
|
||||
|
||||
---
|
||||
|
||||
## OUTPUT FORMAT
|
||||
|
||||
Write JSON to temp file: `/tmp/tea-automate-api-tests-{{timestamp}}.json`
|
||||
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"subagent": "api-tests",
|
||||
"tests": [
|
||||
{
|
||||
"file": "tests/api/auth.spec.ts",
|
||||
"content": "[full TypeScript test file content]",
|
||||
"description": "API tests for authentication endpoints",
|
||||
"priority_coverage": {
|
||||
"P0": 3,
|
||||
"P1": 2,
|
||||
"P2": 1,
|
||||
"P3": 0
|
||||
}
|
||||
},
|
||||
{
|
||||
"file": "tests/api/checkout.spec.ts",
|
||||
"content": "[full TypeScript test file content]",
|
||||
"description": "API tests for checkout endpoints",
|
||||
"priority_coverage": {
|
||||
"P0": 2,
|
||||
"P1": 3,
|
||||
"P2": 1,
|
||||
"P3": 0
|
||||
}
|
||||
}
|
||||
],
|
||||
"fixture_needs": ["authToken", "userDataFactory", "productDataFactory"],
|
||||
"knowledge_fragments_used": ["api-request", "data-factories", "api-testing-patterns"],
|
||||
"provider_scrutiny": "completed",
|
||||
"provider_files_read": ["server/src/routes/authHandlers.ts", "server/src/routes/checkoutHandlers.ts", "server/src/types/auth.ts"],
|
||||
"test_count": 12,
|
||||
"summary": "Generated 12 API test cases covering 3 features"
|
||||
}
|
||||
```
|
||||
|
||||
**On Error:**
|
||||
|
||||
```json
|
||||
{
|
||||
"success": false,
|
||||
"subagent": "api-tests",
|
||||
"error": "Error message describing what went wrong",
|
||||
"partial_output": {
|
||||
/* any tests generated before error */
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## EXIT CONDITION
|
||||
|
||||
Subagent completes when:
|
||||
|
||||
- ✅ All API endpoints have test files generated
|
||||
- ✅ All tests follow knowledge fragment patterns
|
||||
- ✅ JSON output written to temp file
|
||||
- ✅ Fixture needs tracked
|
||||
|
||||
**Subagent terminates here.** Parent workflow will read output and proceed to aggregation.
|
||||
|
||||
---
|
||||
|
||||
## 🚨 SUBAGENT SUCCESS METRICS
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- All API tests generated following patterns
|
||||
- JSON output valid and complete
|
||||
- No E2E/component/unit tests included (out of scope)
|
||||
- Every Pact interaction has `// Provider endpoint:` comment (if CDC enabled)
|
||||
- Provider source scrutiny completed or gracefully degraded with TODO markers (if CDC enabled)
|
||||
- Scrutiny evidence documented as block comments in test files (if CDC enabled)
|
||||
|
||||
### ❌ FAILURE:
|
||||
|
||||
- Generated tests other than API tests
|
||||
- Did not follow knowledge fragment patterns
|
||||
- Invalid or missing JSON output
|
||||
- Ran tests (not subagent responsibility)
|
||||
- Pact interactions missing provider endpoint comments (if CDC enabled)
|
||||
- Response matchers generated from consumer-side types without provider scrutiny (if CDC enabled)
|
||||
@@ -0,0 +1,246 @@
|
||||
---
|
||||
name: 'step-03b-subagent-backend'
|
||||
description: 'Subagent: Generate backend tests only (unit, integration, contract)'
|
||||
subagent: true
|
||||
outputFile: '/tmp/tea-automate-backend-tests-{{timestamp}}.json'
|
||||
---
|
||||
|
||||
# Subagent 3B-backend: Generate Backend Tests
|
||||
|
||||
## SUBAGENT CONTEXT
|
||||
|
||||
This is an **isolated subagent** running in parallel with API test generation (and optionally E2E test generation for fullstack projects).
|
||||
|
||||
**What you have from parent workflow:**
|
||||
|
||||
- Target features/services identified in Step 2
|
||||
- Knowledge fragments loaded: test-levels-framework, test-priorities-matrix, data-factories
|
||||
- Config: test framework, detected stack type
|
||||
- Coverage plan: which services/modules need backend testing
|
||||
|
||||
**Your task:** Generate backend tests ONLY (unit, integration, contract - not API endpoint tests, not E2E).
|
||||
|
||||
---
|
||||
|
||||
## MANDATORY EXECUTION RULES
|
||||
|
||||
- Read this entire subagent file before acting
|
||||
- Generate backend tests ONLY (unit, integration, contract)
|
||||
- Output structured JSON to temp file using the subagent output schema contract
|
||||
- Follow knowledge fragment patterns
|
||||
- Do NOT generate API endpoint tests (that's subagent 3A)
|
||||
- Do NOT generate E2E tests (that's subagent 3B-E2E)
|
||||
- Do NOT run tests (that's step 4)
|
||||
- Do NOT generate fixtures yet (that's step 3C aggregation)
|
||||
|
||||
---
|
||||
|
||||
## SUBAGENT TASK
|
||||
|
||||
### 1. Identify Test Targets
|
||||
|
||||
From the coverage plan (Step 2 output), identify:
|
||||
|
||||
- Which services/modules need unit test coverage
|
||||
- Which integrations need integration test coverage (database, message queues, external services)
|
||||
- Which service contracts need contract test coverage (Pact, schema validation)
|
||||
- Business logic functions requiring edge case coverage
|
||||
|
||||
### 2. Detect Framework & Language
|
||||
|
||||
From `config.test_framework` and project manifests, determine:
|
||||
|
||||
- **Python (pytest)**: Use `pytest` conventions, `conftest.py` fixtures, `@pytest.mark` decorators
|
||||
- **Java/Kotlin (JUnit)**: Use JUnit 5 annotations (`@Test`, `@BeforeEach`, `@Nested`), Mockito for mocking
|
||||
- **Go (go test)**: Use `*_test.go` files, `testing.T`, table-driven tests, `testify` assertions
|
||||
- **C#/.NET (xUnit)**: Use `[Fact]`, `[Theory]`, `[InlineData]`, `Moq` for mocking
|
||||
- **Ruby (RSpec)**: Use `describe`/`context`/`it` blocks, `let`/`before` helpers, `FactoryBot`
|
||||
|
||||
### 3. Generate Unit Tests
|
||||
|
||||
For each module/service, create test files following language-idiomatic patterns:
|
||||
|
||||
**Python (pytest) example:**
|
||||
|
||||
```python
|
||||
import pytest
|
||||
from unittest.mock import MagicMock, patch
|
||||
from myapp.services.user_service import UserService
|
||||
|
||||
class TestUserService:
|
||||
"""[P0] Unit tests for UserService"""
|
||||
|
||||
def test_create_user_with_valid_data(self, user_factory):
|
||||
"""Should create user when data is valid"""
|
||||
user_data = user_factory.build()
|
||||
result = UserService.create(user_data)
|
||||
assert result.email == user_data["email"]
|
||||
|
||||
def test_create_user_rejects_duplicate_email(self, user_factory):
|
||||
"""[P1] Should reject duplicate email"""
|
||||
user_data = user_factory.build(email="existing@test.com")
|
||||
with pytest.raises(DuplicateEmailError):
|
||||
UserService.create(user_data)
|
||||
```
|
||||
|
||||
**Go (go test) example:**
|
||||
|
||||
```go
|
||||
func TestUserService_Create(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
input CreateUserInput
|
||||
wantErr bool
|
||||
}{
|
||||
{"valid user", validInput(), false},
|
||||
{"duplicate email", duplicateInput(), true},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
svc := NewUserService(mockRepo)
|
||||
_, err := svc.Create(tt.input)
|
||||
if (err != nil) != tt.wantErr {
|
||||
t.Errorf("Create() error = %v, wantErr %v", err, tt.wantErr)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Requirements:**
|
||||
|
||||
- Follow the detected framework's idiomatic test patterns
|
||||
- Include priority tags [P0], [P1], [P2], [P3] in test descriptions
|
||||
- Use proper mocking for external dependencies (database, APIs, message queues)
|
||||
- Test both happy path and error cases
|
||||
- Use proper typing/type hints where applicable
|
||||
- No hard-coded test data; use factories or builders
|
||||
|
||||
### 4. Generate Integration Tests
|
||||
|
||||
For service integrations, create integration test files:
|
||||
|
||||
- Database integration tests (with test database or in-memory alternatives)
|
||||
- Message queue consumer/producer tests
|
||||
- Cache integration tests
|
||||
- External service integration tests (with mocked HTTP clients)
|
||||
|
||||
### 5. Generate Contract Tests (if applicable)
|
||||
|
||||
If the project uses microservices or has defined API contracts:
|
||||
|
||||
- Pact consumer/provider tests
|
||||
- Schema validation tests (JSON Schema, Protobuf)
|
||||
- OpenAPI spec compliance tests
|
||||
|
||||
### 6. Track Fixture Needs
|
||||
|
||||
Identify fixtures/helpers needed for backend tests:
|
||||
|
||||
- Database fixtures (seed data, cleanup)
|
||||
- Factory functions (test data builders)
|
||||
- Mock services (HTTP mocks, message queue mocks)
|
||||
- Configuration fixtures (test environment config)
|
||||
|
||||
**Do NOT create fixtures yet** - just track what's needed for aggregation step.
|
||||
|
||||
---
|
||||
|
||||
## OUTPUT FORMAT
|
||||
|
||||
Write JSON to temp file: `/tmp/tea-automate-backend-tests-{{timestamp}}.json`
|
||||
|
||||
```json
|
||||
{
|
||||
"subagentType": "backend",
|
||||
"testsGenerated": [
|
||||
{
|
||||
"file": "tests/unit/test_user_service.py",
|
||||
"content": "[full test file content]",
|
||||
"description": "Unit tests for UserService",
|
||||
"priority_coverage": {
|
||||
"P0": 3,
|
||||
"P1": 2,
|
||||
"P2": 1,
|
||||
"P3": 0
|
||||
}
|
||||
},
|
||||
{
|
||||
"file": "tests/integration/test_user_repository.py",
|
||||
"content": "[full test file content]",
|
||||
"description": "Integration tests for user database operations",
|
||||
"priority_coverage": {
|
||||
"P0": 1,
|
||||
"P1": 2,
|
||||
"P2": 1,
|
||||
"P3": 0
|
||||
}
|
||||
}
|
||||
],
|
||||
"coverageSummary": {
|
||||
"totalTests": 15,
|
||||
"testLevels": ["unit", "integration", "contract"],
|
||||
"fixtureNeeds": ["databaseFixture", "userFactory", "mockHttpClient"]
|
||||
},
|
||||
"status": "complete",
|
||||
"success": true,
|
||||
"subagent": "backend-tests",
|
||||
"knowledge_fragments_used": ["test-levels-framework", "test-priorities-matrix", "data-factories"],
|
||||
"summary": "Generated 15 backend test cases (10 unit, 4 integration, 1 contract)"
|
||||
}
|
||||
```
|
||||
|
||||
**On Error:**
|
||||
|
||||
```json
|
||||
{
|
||||
"subagentType": "backend",
|
||||
"testsGenerated": [],
|
||||
"coverageSummary": {
|
||||
"totalTests": 0,
|
||||
"testLevels": [],
|
||||
"fixtureNeeds": []
|
||||
},
|
||||
"status": "partial",
|
||||
"success": false,
|
||||
"subagent": "backend-tests",
|
||||
"error": "Error message describing what went wrong",
|
||||
"partial_output": {
|
||||
/* any tests generated before error */
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## EXIT CONDITION
|
||||
|
||||
Subagent completes when:
|
||||
|
||||
- All identified modules have backend test files generated
|
||||
- All tests follow language-idiomatic patterns
|
||||
- JSON output written to temp file using the subagent output schema contract
|
||||
- Fixture needs tracked
|
||||
|
||||
**Subagent terminates here.** Parent workflow will read output and proceed to aggregation.
|
||||
|
||||
---
|
||||
|
||||
## SUBAGENT SUCCESS METRICS
|
||||
|
||||
### SUCCESS:
|
||||
|
||||
- All backend tests generated following idiomatic patterns
|
||||
- JSON output valid and complete, matches subagent output schema contract
|
||||
- No E2E or browser tests included (out of scope)
|
||||
- Proper mocking used for external dependencies
|
||||
- Priority tags assigned to all test cases
|
||||
|
||||
### FAILURE:
|
||||
|
||||
- Generated tests other than backend tests (unit/integration/contract)
|
||||
- Did not follow language-idiomatic patterns
|
||||
- Invalid or missing JSON output
|
||||
- Output schema does not match the contract
|
||||
- Ran tests (not subagent responsibility)
|
||||
- Used real external services instead of mocks
|
||||
@@ -0,0 +1,213 @@
|
||||
---
|
||||
name: 'step-03b-subagent-e2e'
|
||||
description: 'Subagent: Generate E2E tests only'
|
||||
subagent: true
|
||||
outputFile: '/tmp/tea-automate-e2e-tests-{{timestamp}}.json'
|
||||
---
|
||||
|
||||
# Subagent 3B: Generate E2E Tests
|
||||
|
||||
## SUBAGENT CONTEXT
|
||||
|
||||
This is an **isolated subagent** running in parallel with API test generation.
|
||||
|
||||
**What you have from parent workflow:**
|
||||
|
||||
- Target features/user journeys identified in Step 2
|
||||
- Knowledge fragments loaded: fixture-architecture, network-first, selector-resilience
|
||||
- Config: test framework, Playwright Utils enabled/disabled
|
||||
- Coverage plan: which user journeys need E2E testing
|
||||
|
||||
**Your task:** Generate E2E tests ONLY (not API, not fixtures, not other test types).
|
||||
|
||||
---
|
||||
|
||||
## MANDATORY EXECUTION RULES
|
||||
|
||||
- 📖 Read this entire subagent file before acting
|
||||
- ✅ Generate E2E tests ONLY
|
||||
- ✅ Output structured JSON to temp file
|
||||
- ✅ Follow knowledge fragment patterns
|
||||
- ❌ Do NOT generate API tests (that's subagent 3A)
|
||||
- ❌ Do NOT run tests (that's step 4)
|
||||
- ❌ Do NOT generate fixtures yet (that's step 3C aggregation)
|
||||
|
||||
---
|
||||
|
||||
## SUBAGENT TASK
|
||||
|
||||
### 1. Identify User Journeys
|
||||
|
||||
From the coverage plan (Step 2 output), identify:
|
||||
|
||||
- Which user journeys need E2E coverage
|
||||
- Critical user paths (authentication, checkout, profile, etc.)
|
||||
- UI interactions required
|
||||
- Expected visual states
|
||||
|
||||
### 2. Browser Interaction (Selector Verification)
|
||||
|
||||
**Automation mode:** `config.tea_browser_automation`
|
||||
|
||||
If `auto` (fall back to MCP if CLI unavailable; if neither available, generate from best practices):
|
||||
|
||||
- Open the target page first, then verify selectors with a snapshot:
|
||||
`playwright-cli -s=tea-automate-{{timestamp}} open <target_url>`
|
||||
`playwright-cli -s=tea-automate-{{timestamp}} snapshot` → map refs to Playwright locators
|
||||
- ref `{role: "button", name: "Submit"}` → `page.getByRole('button', { name: 'Submit' })`
|
||||
- ref `{role: "textbox", name: "Email"}` → `page.getByRole('textbox', { name: 'Email' })`
|
||||
- `playwright-cli -s=tea-automate-{{timestamp}} close` when done
|
||||
|
||||
If `cli` (CLI only — do NOT fall back to MCP; generate from best practices if CLI unavailable):
|
||||
|
||||
- Open the target page first, then verify selectors with a snapshot:
|
||||
`playwright-cli -s=tea-automate-{{timestamp}} open <target_url>`
|
||||
`playwright-cli -s=tea-automate-{{timestamp}} snapshot` → map refs to Playwright locators
|
||||
- ref `{role: "button", name: "Submit"}` → `page.getByRole('button', { name: 'Submit' })`
|
||||
- ref `{role: "textbox", name: "Email"}` → `page.getByRole('textbox', { name: 'Email' })`
|
||||
- `playwright-cli -s=tea-automate-{{timestamp}} close` when done
|
||||
|
||||
> **Session Hygiene:** Always close sessions using `playwright-cli -s=tea-automate-{{timestamp}} close`. Do NOT use `close-all` — it kills every session on the machine and breaks parallel execution.
|
||||
|
||||
If `mcp`:
|
||||
|
||||
- Use MCP tools for selector verification (current behavior)
|
||||
|
||||
If `none`:
|
||||
|
||||
- Generate selectors from best practices without browser verification
|
||||
|
||||
### 3. Generate E2E Test Files
|
||||
|
||||
For each user journey, create test file in `tests/e2e/[feature].spec.ts`:
|
||||
|
||||
**Test Structure:**
|
||||
|
||||
```typescript
|
||||
import { test, expect } from '@playwright/test';
|
||||
|
||||
test.describe('[Feature] E2E User Journey', () => {
|
||||
test('[P0] should complete [user journey]', async ({ page }) => {
|
||||
// Navigate to starting point
|
||||
await page.goto('/feature');
|
||||
|
||||
// Interact with UI
|
||||
await page.getByRole('button', { name: 'Submit' }).click();
|
||||
|
||||
// Assert expected state
|
||||
await expect(page.getByText('Success')).toBeVisible();
|
||||
});
|
||||
|
||||
test('[P1] should handle [edge case]', async ({ page }) => {
|
||||
// Test edge case scenario
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
**Requirements:**
|
||||
|
||||
- ✅ Follow fixture architecture patterns (from fixture-architecture fragment)
|
||||
- ✅ Use network-first patterns: intercept before navigate (from network-first fragment)
|
||||
- ✅ Use resilient selectors: getByRole, getByText, getByLabel (from selector-resilience fragment)
|
||||
- ✅ Include priority tags [P0], [P1], [P2], [P3]
|
||||
- ✅ Test complete user journeys (not isolated clicks)
|
||||
- ✅ Use proper TypeScript types
|
||||
- ✅ Deterministic waits (no hard sleeps, use expect().toBeVisible())
|
||||
|
||||
### 4. Track Fixture Needs
|
||||
|
||||
Identify fixtures needed for E2E tests:
|
||||
|
||||
- Page object models (if complex)
|
||||
- Authentication fixtures (logged-in user state)
|
||||
- Network mocks/intercepts
|
||||
- Test data fixtures
|
||||
|
||||
**Do NOT create fixtures yet** - just track what's needed for aggregation step.
|
||||
|
||||
---
|
||||
|
||||
## OUTPUT FORMAT
|
||||
|
||||
Write JSON to temp file: `/tmp/tea-automate-e2e-tests-{{timestamp}}.json`
|
||||
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"subagent": "e2e-tests",
|
||||
"tests": [
|
||||
{
|
||||
"file": "tests/e2e/authentication.spec.ts",
|
||||
"content": "[full TypeScript test file content]",
|
||||
"description": "E2E tests for user authentication journey",
|
||||
"priority_coverage": {
|
||||
"P0": 2,
|
||||
"P1": 3,
|
||||
"P2": 2,
|
||||
"P3": 0
|
||||
}
|
||||
},
|
||||
{
|
||||
"file": "tests/e2e/checkout.spec.ts",
|
||||
"content": "[full TypeScript test file content]",
|
||||
"description": "E2E tests for checkout journey",
|
||||
"priority_coverage": {
|
||||
"P0": 3,
|
||||
"P1": 2,
|
||||
"P2": 1,
|
||||
"P3": 0
|
||||
}
|
||||
}
|
||||
],
|
||||
"fixture_needs": ["authenticatedUserFixture", "paymentMockFixture", "checkoutDataFixture"],
|
||||
"knowledge_fragments_used": ["fixture-architecture", "network-first", "selector-resilience"],
|
||||
"test_count": 15,
|
||||
"summary": "Generated 15 E2E test cases covering 5 user journeys"
|
||||
}
|
||||
```
|
||||
|
||||
**On Error:**
|
||||
|
||||
```json
|
||||
{
|
||||
"success": false,
|
||||
"subagent": "e2e-tests",
|
||||
"error": "Error message describing what went wrong",
|
||||
"partial_output": {
|
||||
/* any tests generated before error */
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## EXIT CONDITION
|
||||
|
||||
Subagent completes when:
|
||||
|
||||
- ✅ All user journeys have E2E test files generated
|
||||
- ✅ All tests follow knowledge fragment patterns
|
||||
- ✅ JSON output written to temp file
|
||||
- ✅ Fixture needs tracked
|
||||
|
||||
**Subagent terminates here.** Parent workflow will read output and proceed to aggregation.
|
||||
|
||||
---
|
||||
|
||||
## 🚨 SUBAGENT SUCCESS METRICS
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- All E2E tests generated following patterns
|
||||
- JSON output valid and complete
|
||||
- No API/component/unit tests included (out of scope)
|
||||
- Resilient selectors used (getByRole, getByText)
|
||||
- Network-first patterns applied (intercept before navigate)
|
||||
|
||||
### ❌ FAILURE:
|
||||
|
||||
- Generated tests other than E2E tests
|
||||
- Did not follow knowledge fragment patterns
|
||||
- Invalid or missing JSON output
|
||||
- Ran tests (not subagent responsibility)
|
||||
- Used brittle selectors (CSS classes, XPath)
|
||||
@@ -0,0 +1,393 @@
|
||||
---
|
||||
name: 'step-03c-aggregate'
|
||||
description: 'Aggregate subagent outputs and complete test infrastructure'
|
||||
outputFile: '{test_artifacts}/automation-summary.md'
|
||||
nextStepFile: './step-04-validate-and-summarize.md'
|
||||
---
|
||||
|
||||
# Step 3C: Aggregate Test Generation Results
|
||||
|
||||
## STEP GOAL
|
||||
|
||||
Read outputs from parallel subagents (API + E2E and/or Backend test generation based on `{detected_stack}`), aggregate results, and create supporting infrastructure (fixtures, helpers).
|
||||
|
||||
---
|
||||
|
||||
## MANDATORY EXECUTION RULES
|
||||
|
||||
- 📖 Read the entire step file before acting
|
||||
- ✅ Speak in `{communication_language}`
|
||||
- ✅ Read subagent outputs from temp files
|
||||
- ✅ Generate shared fixtures based on fixture needs from both subagents
|
||||
- ✅ Write all generated test files to disk
|
||||
- ❌ Do NOT regenerate tests (use subagent outputs)
|
||||
- ❌ Do NOT run tests yet (that's step 4)
|
||||
|
||||
---
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
- 💾 Record outputs before proceeding
|
||||
- 📖 Load the next step only when instructed
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: config, subagent outputs from temp files
|
||||
- Focus: aggregation and fixture generation only
|
||||
- Limits: do not execute future steps
|
||||
- Dependencies: Step 3A and 3B subagent outputs
|
||||
|
||||
---
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly. Do not skip, reorder, or improvise.
|
||||
|
||||
### 1. Read Subagent Outputs
|
||||
|
||||
**Read API test subagent output (always):**
|
||||
|
||||
```javascript
|
||||
const apiTestsPath = '/tmp/tea-automate-api-tests-{{timestamp}}.json';
|
||||
const apiTestsOutput = JSON.parse(fs.readFileSync(apiTestsPath, 'utf8'));
|
||||
```
|
||||
|
||||
**Read E2E test subagent output (if {detected_stack} is `frontend` or `fullstack`):**
|
||||
|
||||
```javascript
|
||||
let e2eTestsOutput = null;
|
||||
if (detected_stack === 'frontend' || detected_stack === 'fullstack') {
|
||||
const e2eTestsPath = '/tmp/tea-automate-e2e-tests-{{timestamp}}.json';
|
||||
e2eTestsOutput = JSON.parse(fs.readFileSync(e2eTestsPath, 'utf8'));
|
||||
}
|
||||
```
|
||||
|
||||
**Read Backend test subagent output (if {detected_stack} is `backend` or `fullstack`):**
|
||||
|
||||
```javascript
|
||||
let backendTestsOutput = null;
|
||||
if (detected_stack === 'backend' || detected_stack === 'fullstack') {
|
||||
const backendTestsPath = '/tmp/tea-automate-backend-tests-{{timestamp}}.json';
|
||||
backendTestsOutput = JSON.parse(fs.readFileSync(backendTestsPath, 'utf8'));
|
||||
}
|
||||
```
|
||||
|
||||
**Verify all launched subagents succeeded:**
|
||||
|
||||
- Check `apiTestsOutput.success === true`
|
||||
- If E2E was launched: check `e2eTestsOutput.success === true`
|
||||
- If Backend was launched: check `backendTestsOutput.success === true`
|
||||
- If any failed, report error and stop (don't proceed)
|
||||
|
||||
---
|
||||
|
||||
### 2. Write All Test Files to Disk
|
||||
|
||||
**Write API test files:**
|
||||
|
||||
```javascript
|
||||
apiTestsOutput.tests.forEach((test) => {
|
||||
fs.writeFileSync(test.file, test.content, 'utf8');
|
||||
console.log(`✅ Created: ${test.file}`);
|
||||
});
|
||||
```
|
||||
|
||||
**Write E2E test files (if {detected_stack} is `frontend` or `fullstack`):**
|
||||
|
||||
```javascript
|
||||
if (e2eTestsOutput) {
|
||||
e2eTestsOutput.tests.forEach((test) => {
|
||||
fs.writeFileSync(test.file, test.content, 'utf8');
|
||||
console.log(`✅ Created: ${test.file}`);
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
**Write Backend test files (if {detected_stack} is `backend` or `fullstack`):**
|
||||
|
||||
```javascript
|
||||
if (backendTestsOutput) {
|
||||
backendTestsOutput.testsGenerated.forEach((test) => {
|
||||
fs.writeFileSync(test.file, test.content, 'utf8');
|
||||
console.log(`✅ Created: ${test.file}`);
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 3. Aggregate Fixture Needs
|
||||
|
||||
**Collect all fixture needs from all launched subagents:**
|
||||
|
||||
```javascript
|
||||
const allFixtureNeeds = [
|
||||
...apiTestsOutput.fixture_needs,
|
||||
...(e2eTestsOutput ? e2eTestsOutput.fixture_needs : []),
|
||||
...(backendTestsOutput ? backendTestsOutput.coverageSummary?.fixtureNeeds || [] : []),
|
||||
];
|
||||
|
||||
// Remove duplicates
|
||||
const uniqueFixtures = [...new Set(allFixtureNeeds)];
|
||||
```
|
||||
|
||||
**Categorize fixtures:**
|
||||
|
||||
- **Authentication fixtures:** authToken, authenticatedUserFixture, etc.
|
||||
- **Data factories:** userDataFactory, productDataFactory, etc.
|
||||
- **Network mocks:** paymentMockFixture, apiResponseMocks, etc.
|
||||
- **Test helpers:** wait/retry/assertion helpers
|
||||
|
||||
---
|
||||
|
||||
### 4. Generate Fixture Infrastructure
|
||||
|
||||
**Create or update fixture files based on needs:**
|
||||
|
||||
**A) Authentication Fixtures** (`tests/fixtures/auth.ts`):
|
||||
|
||||
```typescript
|
||||
import { test as base } from '@playwright/test';
|
||||
|
||||
export const test = base.extend({
|
||||
authenticatedUser: async ({ page }, use) => {
|
||||
// Login logic
|
||||
await page.goto('/login');
|
||||
await page.fill('[name="email"]', 'test@example.com');
|
||||
await page.fill('[name="password"]', 'password');
|
||||
await page.click('button[type="submit"]');
|
||||
await page.waitForURL('/dashboard');
|
||||
|
||||
await use(page);
|
||||
},
|
||||
|
||||
authToken: async ({ request }, use) => {
|
||||
// Get auth token for API tests
|
||||
const response = await request.post('/api/auth/login', {
|
||||
data: { email: 'test@example.com', password: 'password' },
|
||||
});
|
||||
const { token } = await response.json();
|
||||
|
||||
await use(token);
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
**B) Data Factories** (`tests/fixtures/data-factories.ts`):
|
||||
|
||||
```typescript
|
||||
import { faker } from '@faker-js/faker';
|
||||
|
||||
export const createUserData = (overrides = {}) => ({
|
||||
name: faker.person.fullName(),
|
||||
email: faker.internet.email(),
|
||||
...overrides,
|
||||
});
|
||||
|
||||
export const createProductData = (overrides = {}) => ({
|
||||
name: faker.commerce.productName(),
|
||||
price: faker.number.int({ min: 10, max: 1000 }),
|
||||
...overrides,
|
||||
});
|
||||
```
|
||||
|
||||
**C) Network Mocks** (`tests/fixtures/network-mocks.ts`):
|
||||
|
||||
```typescript
|
||||
import { Page } from '@playwright/test';
|
||||
|
||||
export const mockPaymentSuccess = async (page: Page) => {
|
||||
await page.route('/api/payment/**', (route) => {
|
||||
route.fulfill({
|
||||
status: 200,
|
||||
body: JSON.stringify({ success: true, transactionId: '12345' }),
|
||||
});
|
||||
});
|
||||
};
|
||||
```
|
||||
|
||||
**D) Helper Utilities** (`tests/fixtures/helpers.ts`):
|
||||
|
||||
```typescript
|
||||
import { expect, Page } from '@playwright/test';
|
||||
|
||||
export const waitForApiResponse = async (page: Page, urlPattern: string) => {
|
||||
return page.waitForResponse((response) => response.url().includes(urlPattern) && response.ok());
|
||||
};
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 5. Calculate Summary Statistics
|
||||
|
||||
**Aggregate test counts (based on `{detected_stack}`):**
|
||||
|
||||
```javascript
|
||||
const e2eCount = e2eTestsOutput ? e2eTestsOutput.test_count : 0;
|
||||
const backendCount = backendTestsOutput ? (backendTestsOutput.coverageSummary?.totalTests ?? 0) : 0;
|
||||
|
||||
const resolvedMode = subagentContext?.execution?.resolvedMode;
|
||||
const subagentExecutionLabel =
|
||||
resolvedMode === 'sequential'
|
||||
? 'SEQUENTIAL (API then dependent workers)'
|
||||
: resolvedMode === 'agent-team'
|
||||
? 'AGENT-TEAM (parallel worker squad)'
|
||||
: resolvedMode === 'subagent'
|
||||
? 'SUBAGENT (parallel subagents)'
|
||||
: `PARALLEL (based on ${detected_stack})`;
|
||||
const performanceGainLabel =
|
||||
resolvedMode === 'sequential'
|
||||
? 'baseline (no parallel speedup)'
|
||||
: resolvedMode === 'agent-team' || resolvedMode === 'subagent'
|
||||
? '~40-70% faster than sequential'
|
||||
: 'mode-dependent';
|
||||
|
||||
const summary = {
|
||||
detected_stack: '{detected_stack}',
|
||||
total_tests: apiTestsOutput.test_count + e2eCount + backendCount,
|
||||
api_tests: apiTestsOutput.test_count,
|
||||
e2e_tests: e2eCount,
|
||||
backend_tests: backendCount,
|
||||
fixtures_created: uniqueFixtures.length,
|
||||
api_test_files: apiTestsOutput.tests.length,
|
||||
e2e_test_files: e2eTestsOutput ? e2eTestsOutput.tests.length : 0,
|
||||
backend_test_files: backendTestsOutput ? backendTestsOutput.testsGenerated.length : 0,
|
||||
priority_coverage: {
|
||||
P0:
|
||||
(apiTestsOutput.priority_coverage?.P0 ?? 0) +
|
||||
(e2eTestsOutput?.priority_coverage?.P0 ?? 0) +
|
||||
(backendTestsOutput?.testsGenerated?.reduce((sum, t) => sum + (t.priority_coverage?.P0 ?? 0), 0) ?? 0),
|
||||
P1:
|
||||
(apiTestsOutput.priority_coverage?.P1 ?? 0) +
|
||||
(e2eTestsOutput?.priority_coverage?.P1 ?? 0) +
|
||||
(backendTestsOutput?.testsGenerated?.reduce((sum, t) => sum + (t.priority_coverage?.P1 ?? 0), 0) ?? 0),
|
||||
P2:
|
||||
(apiTestsOutput.priority_coverage?.P2 ?? 0) +
|
||||
(e2eTestsOutput?.priority_coverage?.P2 ?? 0) +
|
||||
(backendTestsOutput?.testsGenerated?.reduce((sum, t) => sum + (t.priority_coverage?.P2 ?? 0), 0) ?? 0),
|
||||
P3:
|
||||
(apiTestsOutput.priority_coverage?.P3 ?? 0) +
|
||||
(e2eTestsOutput?.priority_coverage?.P3 ?? 0) +
|
||||
(backendTestsOutput?.testsGenerated?.reduce((sum, t) => sum + (t.priority_coverage?.P3 ?? 0), 0) ?? 0),
|
||||
},
|
||||
knowledge_fragments_used: [
|
||||
...apiTestsOutput.knowledge_fragments_used,
|
||||
...(e2eTestsOutput ? e2eTestsOutput.knowledge_fragments_used : []),
|
||||
...(backendTestsOutput ? backendTestsOutput.knowledge_fragments_used || [] : []),
|
||||
],
|
||||
subagent_execution: subagentExecutionLabel,
|
||||
performance_gain: performanceGainLabel,
|
||||
};
|
||||
```
|
||||
|
||||
**Store summary for Step 4:**
|
||||
Save summary to temp file for validation step:
|
||||
|
||||
```javascript
|
||||
fs.writeFileSync('/tmp/tea-automate-summary-{{timestamp}}.json', JSON.stringify(summary, null, 2), 'utf8');
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 6. Optional Cleanup
|
||||
|
||||
**Clean up subagent temp files** (optional - can keep for debugging):
|
||||
|
||||
```javascript
|
||||
fs.unlinkSync(apiTestsPath);
|
||||
if (e2eTestsOutput) fs.unlinkSync('/tmp/tea-automate-e2e-tests-{{timestamp}}.json');
|
||||
if (backendTestsOutput) fs.unlinkSync('/tmp/tea-automate-backend-tests-{{timestamp}}.json');
|
||||
console.log('✅ Subagent temp files cleaned up');
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## OUTPUT SUMMARY
|
||||
|
||||
Display to user:
|
||||
|
||||
```
|
||||
✅ Test Generation Complete ({subagent_execution})
|
||||
|
||||
📊 Summary:
|
||||
- Stack Type: {detected_stack}
|
||||
- Total Tests: {total_tests}
|
||||
- API Tests: {api_tests} ({api_test_files} files)
|
||||
- E2E Tests: {e2e_tests} ({e2e_test_files} files) [if frontend/fullstack]
|
||||
- Backend Tests: {backend_tests} ({backend_test_files} files) [if backend/fullstack]
|
||||
- Fixtures Created: {fixtures_created}
|
||||
- Priority Coverage:
|
||||
- P0 (Critical): {P0} tests
|
||||
- P1 (High): {P1} tests
|
||||
- P2 (Medium): {P2} tests
|
||||
- P3 (Low): {P3} tests
|
||||
|
||||
🚀 Performance: {performance_gain}
|
||||
|
||||
📂 Generated Files:
|
||||
- tests/api/[feature].spec.ts [always]
|
||||
- tests/e2e/[feature].spec.ts [if frontend/fullstack]
|
||||
- tests/unit/[feature].test.* [if backend/fullstack]
|
||||
- tests/integration/[feature].test.* [if backend/fullstack]
|
||||
- tests/fixtures/ or tests/support/ [shared infrastructure]
|
||||
|
||||
✅ Ready for validation (Step 4)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## EXIT CONDITION
|
||||
|
||||
Proceed to Step 4 when:
|
||||
|
||||
- ✅ All test files written to disk (API + E2E and/or Backend, based on `{detected_stack}`)
|
||||
- ✅ All fixtures and helpers created
|
||||
- ✅ Summary statistics calculated and saved
|
||||
- ✅ Output displayed to user
|
||||
|
||||
---
|
||||
|
||||
### 7. Save Progress
|
||||
|
||||
**Save this step's accumulated work to `{outputFile}`.**
|
||||
|
||||
- **If `{outputFile}` does not exist** (first save), create it with YAML frontmatter:
|
||||
|
||||
```yaml
|
||||
---
|
||||
stepsCompleted: ['step-03c-aggregate']
|
||||
lastStep: 'step-03c-aggregate'
|
||||
lastSaved: '{date}'
|
||||
---
|
||||
```
|
||||
|
||||
Then write this step's output below the frontmatter.
|
||||
|
||||
- **If `{outputFile}` already exists**, update:
|
||||
- Add `'step-03c-aggregate'` to `stepsCompleted` array (only if not already present)
|
||||
- Set `lastStep: 'step-03c-aggregate'`
|
||||
- Set `lastSaved: '{date}'`
|
||||
- Append this step's output to the appropriate section.
|
||||
|
||||
Load next step: `{nextStepFile}`
|
||||
|
||||
---
|
||||
|
||||
## 🚨 SYSTEM SUCCESS/FAILURE METRICS:
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- All launched subagents succeeded (based on `{detected_stack}`)
|
||||
- All test files written to disk
|
||||
- Fixtures generated based on subagent needs
|
||||
- Summary complete and accurate
|
||||
|
||||
### ❌ SYSTEM FAILURE:
|
||||
|
||||
- One or more subagents failed
|
||||
- Test files not written to disk
|
||||
- Fixtures missing or incomplete
|
||||
- Summary missing or inaccurate
|
||||
|
||||
**Master Rule:** Do NOT proceed to Step 4 if aggregation incomplete.
|
||||
@@ -0,0 +1,106 @@
|
||||
---
|
||||
name: 'step-04-validate-and-summarize'
|
||||
description: 'Validate outputs and produce automation summary'
|
||||
outputFile: '{test_artifacts}/automation-summary.md'
|
||||
---
|
||||
|
||||
# Step 4: Validate & Summarize
|
||||
|
||||
## STEP GOAL
|
||||
|
||||
Validate generated outputs and produce a concise automation summary.
|
||||
|
||||
## MANDATORY EXECUTION RULES
|
||||
|
||||
- 📖 Read the entire step file before acting
|
||||
- ✅ Speak in `{communication_language}`
|
||||
- ✅ Validate against the checklist before completion
|
||||
|
||||
---
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
- 💾 Record outputs before proceeding
|
||||
- 📖 Load the next step only when instructed
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: config, loaded artifacts, and knowledge fragments
|
||||
- Focus: this step's goal only
|
||||
- Limits: do not execute future steps
|
||||
- Dependencies: prior steps' outputs (if any)
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly. Do not skip, reorder, or improvise.
|
||||
|
||||
## 1. Validate
|
||||
|
||||
Use `checklist.md` to validate:
|
||||
|
||||
- Framework readiness
|
||||
- Coverage mapping
|
||||
- Test quality and structure
|
||||
- Fixtures, factories, helpers
|
||||
- [ ] CLI sessions cleaned up (no orphaned browsers)
|
||||
- [ ] Temp artifacts stored in `{test_artifacts}/` not random locations
|
||||
|
||||
Fix gaps before proceeding.
|
||||
|
||||
---
|
||||
|
||||
## 2. Polish Output
|
||||
|
||||
Before finalizing, review the complete output document for quality:
|
||||
|
||||
1. **Remove duplication**: Progressive-append workflow may have created repeated sections — consolidate
|
||||
2. **Verify consistency**: Ensure terminology, risk scores, and references are consistent throughout
|
||||
3. **Check completeness**: All template sections should be populated or explicitly marked N/A
|
||||
4. **Format cleanup**: Ensure markdown formatting is clean (tables aligned, headers consistent, no orphaned references)
|
||||
|
||||
---
|
||||
|
||||
## 3. Summary Output
|
||||
|
||||
Write `{outputFile}` including:
|
||||
|
||||
- Coverage plan by test level and priority
|
||||
- Files created/updated
|
||||
- Key assumptions and risks
|
||||
- Next recommended workflow (e.g., `test-review` or `trace`)
|
||||
|
||||
---
|
||||
|
||||
## 4. Save Progress
|
||||
|
||||
**Save this step's accumulated work to `{outputFile}`.**
|
||||
|
||||
- **If `{outputFile}` does not exist** (first save), create it with YAML frontmatter:
|
||||
|
||||
```yaml
|
||||
---
|
||||
stepsCompleted: ['step-04-validate-and-summarize']
|
||||
lastStep: 'step-04-validate-and-summarize'
|
||||
lastSaved: '{date}'
|
||||
---
|
||||
```
|
||||
|
||||
Then write this step's output below the frontmatter.
|
||||
|
||||
- **If `{outputFile}` already exists**, update:
|
||||
- Add `'step-04-validate-and-summarize'` to `stepsCompleted` array (only if not already present)
|
||||
- Set `lastStep: 'step-04-validate-and-summarize'`
|
||||
- Set `lastSaved: '{date}'`
|
||||
- Append this step's output to the appropriate section.
|
||||
|
||||
## 🚨 SYSTEM SUCCESS/FAILURE METRICS:
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- Step completed in full with required outputs
|
||||
|
||||
### ❌ SYSTEM FAILURE:
|
||||
|
||||
- Skipped sequence steps or missing outputs
|
||||
**Master Rule:** Skipping steps is FORBIDDEN.
|
||||
@@ -0,0 +1,65 @@
|
||||
---
|
||||
name: 'step-01-assess'
|
||||
description: 'Load an existing output for editing'
|
||||
nextStepFile: './step-02-apply-edit.md'
|
||||
---
|
||||
|
||||
# Step 1: Assess Edit Target
|
||||
|
||||
## STEP GOAL:
|
||||
|
||||
Identify which output should be edited and load it.
|
||||
|
||||
## MANDATORY EXECUTION RULES (READ FIRST):
|
||||
|
||||
### Universal Rules:
|
||||
|
||||
- 📖 Read the complete step file before taking any action
|
||||
- ✅ Speak in `{communication_language}`
|
||||
|
||||
### Role Reinforcement:
|
||||
|
||||
- ✅ You are the Master Test Architect
|
||||
|
||||
### Step-Specific Rules:
|
||||
|
||||
- 🎯 Ask the user which output file to edit
|
||||
- 🚫 Do not edit until target is confirmed
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: existing outputs
|
||||
- Focus: select edit target
|
||||
- Limits: no edits yet
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly.
|
||||
|
||||
### 1. Identify Target
|
||||
|
||||
Ask the user to provide the output file path or select from known outputs.
|
||||
|
||||
### 2. Load Target
|
||||
|
||||
Read the provided output file in full.
|
||||
|
||||
### 3. Confirm
|
||||
|
||||
Confirm the target and proceed to edit.
|
||||
|
||||
Load next step: `{nextStepFile}`
|
||||
|
||||
## 🚨 SYSTEM SUCCESS/FAILURE METRICS:
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- Target identified and loaded
|
||||
|
||||
### ❌ SYSTEM FAILURE:
|
||||
|
||||
- Proceeding without a confirmed target
|
||||
@@ -0,0 +1,60 @@
|
||||
---
|
||||
name: 'step-02-apply-edit'
|
||||
description: 'Apply edits to the selected output'
|
||||
---
|
||||
|
||||
# Step 2: Apply Edits
|
||||
|
||||
## STEP GOAL:
|
||||
|
||||
Apply the requested edits to the selected output and confirm changes.
|
||||
|
||||
## MANDATORY EXECUTION RULES (READ FIRST):
|
||||
|
||||
### Universal Rules:
|
||||
|
||||
- 📖 Read the complete step file before taking any action
|
||||
- ✅ Speak in `{communication_language}`
|
||||
|
||||
### Role Reinforcement:
|
||||
|
||||
- ✅ You are the Master Test Architect
|
||||
|
||||
### Step-Specific Rules:
|
||||
|
||||
- 🎯 Only apply edits explicitly requested by the user
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: selected output and user changes
|
||||
- Focus: apply edits only
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly.
|
||||
|
||||
### 1. Confirm Requested Changes
|
||||
|
||||
Restate what will be changed and confirm.
|
||||
|
||||
### 2. Apply Changes
|
||||
|
||||
Update the output file accordingly.
|
||||
|
||||
### 3. Report
|
||||
|
||||
Summarize the edits applied.
|
||||
|
||||
## 🚨 SYSTEM SUCCESS/FAILURE METRICS:
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- Changes applied and confirmed
|
||||
|
||||
### ❌ SYSTEM FAILURE:
|
||||
|
||||
- Unconfirmed edits or missing update
|
||||
@@ -0,0 +1,67 @@
|
||||
---
|
||||
name: 'step-01-validate'
|
||||
description: 'Validate workflow outputs against checklist'
|
||||
outputFile: '{test_artifacts}/automate-validation-report.md'
|
||||
validationChecklist: '../checklist.md'
|
||||
---
|
||||
|
||||
# Step 1: Validate Outputs
|
||||
|
||||
## STEP GOAL:
|
||||
|
||||
Validate outputs using the workflow checklist and record findings.
|
||||
|
||||
## MANDATORY EXECUTION RULES (READ FIRST):
|
||||
|
||||
### Universal Rules:
|
||||
|
||||
- 📖 Read the complete step file before taking any action
|
||||
- ✅ Speak in `{communication_language}`
|
||||
|
||||
### Role Reinforcement:
|
||||
|
||||
- ✅ You are the Master Test Architect
|
||||
|
||||
### Step-Specific Rules:
|
||||
|
||||
- 🎯 Validate against `{validationChecklist}`
|
||||
- 🚫 Do not skip checks
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
- 💾 Write findings to `{outputFile}`
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: workflow outputs and checklist
|
||||
- Focus: validation only
|
||||
- Limits: do not modify outputs in this step
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly.
|
||||
|
||||
### 1. Load Checklist
|
||||
|
||||
Read `{validationChecklist}` and list all criteria.
|
||||
|
||||
### 2. Validate Outputs
|
||||
|
||||
Evaluate outputs against each checklist item.
|
||||
|
||||
### 3. Write Report
|
||||
|
||||
Write a validation report to `{outputFile}` with PASS/WARN/FAIL per section.
|
||||
|
||||
## 🚨 SYSTEM SUCCESS/FAILURE METRICS:
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- Validation report written
|
||||
- All checklist items evaluated
|
||||
|
||||
### ❌ SYSTEM FAILURE:
|
||||
|
||||
- Skipped checklist items
|
||||
- No report produced
|
||||
@@ -0,0 +1,72 @@
|
||||
---
|
||||
validationDate: 2026-01-27
|
||||
workflowName: testarch-automate
|
||||
workflowPath: {project-root}/src/workflows/testarch/automate
|
||||
validationStatus: COMPLETE
|
||||
completionDate: 2026-01-27 10:03:10
|
||||
---
|
||||
|
||||
# Validation Report: testarch-automate
|
||||
|
||||
**Validation Started:** 2026-01-27 09:50:21
|
||||
**Validator:** BMAD Workflow Validation System (Codex)
|
||||
**Standards Version:** BMAD Workflow Standards
|
||||
|
||||
## File Structure & Size
|
||||
|
||||
- workflow.md present: YES
|
||||
- instructions.md present: YES
|
||||
- workflow.yaml present: YES
|
||||
- step files found: 7
|
||||
|
||||
**Step File Sizes:**
|
||||
|
||||
- steps-c/step-01-preflight-and-context.md: 113 lines [GOOD]
|
||||
- steps-c/step-02-identify-targets.md: 85 lines [GOOD]
|
||||
- steps-c/step-03-generate-tests.md: 76 lines [GOOD]
|
||||
- steps-c/step-04-validate-and-summarize.md: 62 lines [GOOD]
|
||||
- steps-e/step-01-assess.md: 51 lines [GOOD]
|
||||
- steps-e/step-02-apply-edit.md: 46 lines [GOOD]
|
||||
- steps-v/step-01-validate.md: 53 lines [GOOD]
|
||||
- workflow-plan.md present: YES
|
||||
|
||||
## Frontmatter Validation
|
||||
|
||||
- No frontmatter violations found
|
||||
|
||||
## Critical Path Violations
|
||||
|
||||
- No {project-root} hardcoded paths detected in body
|
||||
- No dead relative links detected
|
||||
|
||||
## Menu Handling Validation
|
||||
|
||||
- No menu structures detected (linear step flow) [N/A]
|
||||
|
||||
## Step Type Validation
|
||||
|
||||
- Last step steps-v/step-01-validate.md has no nextStepFile (final step OK)
|
||||
- Step type validation assumes linear sequence (no branching/menu). Workflow-plan.md present for reference. [INFO]
|
||||
|
||||
## Output Format Validation
|
||||
|
||||
- No templates found in workflow root
|
||||
- Steps with outputFile in frontmatter:
|
||||
- steps-c/step-04-validate-and-summarize.md
|
||||
- steps-v/step-01-validate.md
|
||||
|
||||
## Validation Design Check
|
||||
|
||||
- checklist.md present: YES
|
||||
- Validation steps folder (steps-v) present: YES
|
||||
|
||||
## Instruction Style Check
|
||||
|
||||
- All steps include STEP GOAL, MANDATORY EXECUTION RULES, EXECUTION PROTOCOLS, CONTEXT BOUNDARIES, and SUCCESS/FAILURE metrics
|
||||
|
||||
## Summary
|
||||
|
||||
- Validation completed: 2026-01-27 10:03:10
|
||||
- Critical issues: 0
|
||||
- Warnings: 0 (informational notes only)
|
||||
- Readiness: READY (manual review optional)
|
||||
@@ -0,0 +1,114 @@
|
||||
---
|
||||
validationDate: 2026-01-27
|
||||
workflowName: testarch-automate
|
||||
workflowPath: {project-root}/src/workflows/testarch/automate
|
||||
validationStatus: COMPLETE
|
||||
completionDate: 2026-01-27 10:24:01
|
||||
---
|
||||
|
||||
# Validation Report: testarch-automate
|
||||
|
||||
**Validation Started:** 2026-01-27 10:24:01
|
||||
**Validator:** BMAD Workflow Validation System (Codex)
|
||||
**Standards Version:** BMAD Workflow Standards
|
||||
|
||||
## File Structure & Size
|
||||
|
||||
- workflow.md present: YES
|
||||
- instructions.md present: YES
|
||||
- workflow.yaml present: YES
|
||||
- step files found: 7
|
||||
|
||||
**Step File Sizes:**
|
||||
|
||||
- steps-c/step-01-preflight-and-context.md: 112 lines [GOOD]
|
||||
- steps-c/step-02-identify-targets.md: 84 lines [GOOD]
|
||||
- steps-c/step-03-generate-tests.md: 75 lines [GOOD]
|
||||
- steps-c/step-04-validate-and-summarize.md: 61 lines [GOOD]
|
||||
- steps-e/step-01-assess.md: 50 lines [GOOD]
|
||||
- steps-e/step-02-apply-edit.md: 45 lines [GOOD]
|
||||
- steps-v/step-01-validate.md: 52 lines [GOOD]
|
||||
- workflow-plan.md present: YES
|
||||
|
||||
## Frontmatter Validation
|
||||
|
||||
- No frontmatter violations found
|
||||
|
||||
## Critical Path Violations
|
||||
|
||||
### Config Variables (Exceptions)
|
||||
|
||||
Standard BMAD config variables treated as valid exceptions: bmb_creations_output_folder, communication_language, document_output_language, output_folder, planning_artifacts, project-root, project_name, test_artifacts, user_name
|
||||
|
||||
- No {project-root} hardcoded paths detected in body
|
||||
|
||||
- No dead relative links detected
|
||||
|
||||
- No module path assumptions detected
|
||||
|
||||
**Status:** ✅ PASS - No critical violations
|
||||
|
||||
## Menu Handling Validation
|
||||
|
||||
- No menu structures detected (linear step flow) [N/A]
|
||||
|
||||
## Step Type Validation
|
||||
|
||||
- steps-c/step-01-preflight-and-context.md: Init [PASS]
|
||||
- steps-c/step-02-identify-targets.md: Middle [PASS]
|
||||
- steps-c/step-03-generate-tests.md: Middle [PASS]
|
||||
- steps-c/step-04-validate-and-summarize.md: Final [PASS]
|
||||
- Step type validation assumes linear sequence (no branching/menu). Workflow-plan.md present for reference. [INFO]
|
||||
|
||||
## Output Format Validation
|
||||
|
||||
- Templates present: NONE
|
||||
- Steps with outputFile in frontmatter:
|
||||
- steps-c/step-04-validate-and-summarize.md
|
||||
- steps-v/step-01-validate.md
|
||||
- checklist.md present: YES
|
||||
|
||||
## Validation Design Check
|
||||
|
||||
- Validation steps folder (steps-v) present: YES
|
||||
- Validation step(s) present: step-01-validate.md
|
||||
- Validation steps reference checklist data and auto-proceed
|
||||
|
||||
## Instruction Style Check
|
||||
|
||||
- Instruction style: Prescriptive (appropriate for TEA quality/compliance workflows)
|
||||
- Steps emphasize mandatory sequence, explicit success/failure metrics, and risk-based guidance
|
||||
|
||||
## Collaborative Experience Check
|
||||
|
||||
- Overall facilitation quality: GOOD
|
||||
- Steps use progressive prompts and clear role reinforcement; no laundry-list interrogation detected
|
||||
- Flow progression is clear and aligned to workflow goals
|
||||
|
||||
## Subagent Optimization Opportunities
|
||||
|
||||
- No high-priority subagent optimizations identified; workflow already uses step-file architecture
|
||||
- Pattern 1 (grep/regex): N/A for most steps
|
||||
- Pattern 2 (per-file analysis): already aligned to validation structure
|
||||
- Pattern 3 (data ops): minimal data file loads
|
||||
- Pattern 4 (parallel): optional for validation only
|
||||
|
||||
## Cohesive Review
|
||||
|
||||
- Overall assessment: GOOD
|
||||
- Flow is linear, goals are clear, and outputs map to TEA artifacts
|
||||
- Voice and tone consistent with Test Architect persona
|
||||
- Recommendation: READY (minor refinements optional)
|
||||
|
||||
## Plan Quality Validation
|
||||
|
||||
- Plan file present: workflow-plan.md
|
||||
- Planned steps found: 7 (all implemented)
|
||||
- Plan implementation status: Fully Implemented
|
||||
|
||||
## Summary
|
||||
|
||||
- Validation completed: 2026-01-27 10:24:01
|
||||
- Critical issues: 0
|
||||
- Warnings: 0 (informational notes only)
|
||||
- Readiness: READY (manual review optional)
|
||||
20
_bmad/tea/workflows/testarch/automate/workflow-plan.md
Normal file
20
_bmad/tea/workflows/testarch/automate/workflow-plan.md
Normal file
@@ -0,0 +1,20 @@
|
||||
# Workflow Plan: testarch-automate
|
||||
|
||||
## Create Mode (steps-c)
|
||||
- step-01-preflight-and-context.md
|
||||
|
||||
- step-02-identify-targets.md
|
||||
- step-03-generate-tests.md
|
||||
- step-04-validate-and-summarize.md
|
||||
|
||||
## Validate Mode (steps-v)
|
||||
- step-01-validate.md
|
||||
|
||||
## Edit Mode (steps-e)
|
||||
- step-01-assess.md
|
||||
- step-02-apply-edit.md
|
||||
|
||||
## Outputs
|
||||
- {test_artifacts}/automation-summary.md
|
||||
|
||||
- Test files under {project-root}/tests
|
||||
41
_bmad/tea/workflows/testarch/automate/workflow.md
Normal file
41
_bmad/tea/workflows/testarch/automate/workflow.md
Normal file
@@ -0,0 +1,41 @@
|
||||
---
|
||||
name: testarch-automate
|
||||
description: Expand test automation coverage for codebase. Use when user says 'lets expand test coverage' or 'I want to automate tests'
|
||||
web_bundle: true
|
||||
---
|
||||
|
||||
# Test Automation Expansion
|
||||
|
||||
**Goal:** Expand test automation coverage after implementation or analyze existing codebase to generate comprehensive test suite
|
||||
|
||||
**Role:** You are the Master Test Architect.
|
||||
|
||||
---
|
||||
|
||||
## WORKFLOW ARCHITECTURE
|
||||
|
||||
This workflow uses **tri-modal step-file architecture**:
|
||||
|
||||
- **Create mode (steps-c/)**: primary execution flow
|
||||
- **Validate mode (steps-v/)**: validation against checklist
|
||||
- **Edit mode (steps-e/)**: revise existing outputs
|
||||
|
||||
---
|
||||
|
||||
## INITIALIZATION SEQUENCE
|
||||
|
||||
### 1. Mode Determination
|
||||
|
||||
"Welcome to the workflow. What would you like to do?"
|
||||
|
||||
- **[C] Create** — Run the workflow
|
||||
- **[R] Resume** — Resume an interrupted workflow
|
||||
- **[V] Validate** — Validate existing outputs
|
||||
- **[E] Edit** — Edit existing outputs
|
||||
|
||||
### 2. Route to First Step
|
||||
|
||||
- **If C:** Load `steps-c/step-01-preflight-and-context.md`
|
||||
- **If R:** Load `steps-c/step-01b-resume.md`
|
||||
- **If V:** Load `steps-v/step-01-validate.md`
|
||||
- **If E:** Load `steps-e/step-01-assess.md`
|
||||
53
_bmad/tea/workflows/testarch/automate/workflow.yaml
Normal file
53
_bmad/tea/workflows/testarch/automate/workflow.yaml
Normal file
@@ -0,0 +1,53 @@
|
||||
# Test Architect workflow: automate
|
||||
name: testarch-automate
|
||||
# prettier-ignore
|
||||
description: 'Expand test automation coverage for codebase. Use when the user says "lets expand test coverage" or "I want to automate tests"'
|
||||
|
||||
# Critical variables from config
|
||||
config_source: "{project-root}/_bmad/tea/config.yaml"
|
||||
output_folder: "{config_source}:output_folder"
|
||||
test_artifacts: "{config_source}:test_artifacts"
|
||||
user_name: "{config_source}:user_name"
|
||||
communication_language: "{config_source}:communication_language"
|
||||
document_output_language: "{config_source}:document_output_language"
|
||||
date: system-generated
|
||||
|
||||
# Workflow components
|
||||
installed_path: "{project-root}/_bmad/tea/workflows/testarch/automate"
|
||||
instructions: "{installed_path}/instructions.md"
|
||||
validation: "{installed_path}/checklist.md"
|
||||
template: false
|
||||
|
||||
# Variables and inputs
|
||||
variables:
|
||||
# Execution mode and targeting
|
||||
standalone_mode: true # Can work without BMad artifacts (true) or integrate with BMad (false)
|
||||
coverage_target: "critical-paths" # critical-paths, comprehensive, selective
|
||||
|
||||
# Directory paths
|
||||
test_dir: "{project-root}/tests" # Root test directory
|
||||
source_dir: "{project-root}" # Source code directory (customize if needed, e.g., {project-root}/src or {project-root}/lib)
|
||||
|
||||
# Output configuration
|
||||
default_output_file: "{test_artifacts}/automation-summary.md"
|
||||
|
||||
# Required tools
|
||||
required_tools:
|
||||
- read_file # Read source code, existing tests, BMad artifacts
|
||||
- write_file # Create test files, fixtures, factories, summaries
|
||||
- create_directory # Create test directories
|
||||
- list_files # Discover features and existing tests
|
||||
- search_repo # Find coverage gaps and patterns
|
||||
- glob # Find test files and source files
|
||||
|
||||
tags:
|
||||
- qa
|
||||
- automation
|
||||
- test-architect
|
||||
- regression
|
||||
- coverage
|
||||
|
||||
execution_hints:
|
||||
interactive: false # Minimize prompts
|
||||
autonomous: true # Proceed without user input unless blocked
|
||||
iterative: true
|
||||
155
_bmad/tea/workflows/testarch/ci/azure-pipelines-template.yaml
Normal file
155
_bmad/tea/workflows/testarch/ci/azure-pipelines-template.yaml
Normal file
@@ -0,0 +1,155 @@
|
||||
# Azure DevOps CI/CD Pipeline for Test Execution
|
||||
# Generated by BMad TEA Agent - Test Architect Module
|
||||
# Optimized for: Parallel Sharding, Burn-In Loop
|
||||
# Stack: {test_stack_type} | Framework: {test_framework}
|
||||
#
|
||||
# Variables to customize per project:
|
||||
# INSTALL_CMD - dependency install command (e.g., npm ci, pnpm install --frozen-lockfile)
|
||||
# TEST_CMD - main test command (e.g., npm run test:e2e, npm test, npx vitest)
|
||||
# LINT_CMD - lint command (e.g., npm run lint)
|
||||
# BROWSER_INSTALL - browser install command (frontend/fullstack only; omit for backend)
|
||||
# DEFAULT_NODE_VERSION - Node.js version (read from .nvmrc or default to 24)
|
||||
|
||||
trigger:
|
||||
branches:
|
||||
include:
|
||||
- main
|
||||
- develop
|
||||
|
||||
pr:
|
||||
branches:
|
||||
include:
|
||||
- main
|
||||
- develop
|
||||
|
||||
variables:
|
||||
DEFAULT_NODE_VERSION: "24"
|
||||
npm_config_cache: $(Pipeline.Workspace)/.npm
|
||||
# Set TEST_STACK_TYPE to 'backend' to skip Playwright browser installs
|
||||
TEST_STACK_TYPE: "" # Values: frontend, backend, fullstack (leave empty for auto)
|
||||
|
||||
stages:
|
||||
# Lint stage - Code quality checks
|
||||
- stage: Lint
|
||||
displayName: "Lint"
|
||||
jobs:
|
||||
- job: LintJob
|
||||
displayName: "Code Quality"
|
||||
pool:
|
||||
vmImage: "ubuntu-latest"
|
||||
timeoutInMinutes: 5
|
||||
steps:
|
||||
- task: NodeTool@0
|
||||
inputs:
|
||||
versionSpec: $(DEFAULT_NODE_VERSION)
|
||||
displayName: "Setup Node.js"
|
||||
|
||||
- task: Cache@2
|
||||
inputs:
|
||||
key: 'npm | "$(Agent.OS)" | package-lock.json'
|
||||
restoreKeys: 'npm | "$(Agent.OS)"'
|
||||
path: $(npm_config_cache)
|
||||
displayName: "Cache npm"
|
||||
|
||||
- script: npm ci
|
||||
displayName: "Install dependencies" # Replace with INSTALL_CMD
|
||||
|
||||
- script: npm run lint
|
||||
displayName: "Run linter" # Replace with LINT_CMD
|
||||
|
||||
# Test stage - Parallel execution with sharding
|
||||
- stage: Test
|
||||
displayName: "Test"
|
||||
dependsOn: Lint
|
||||
jobs:
|
||||
- job: TestShard
|
||||
displayName: "Test Shard"
|
||||
pool:
|
||||
vmImage: "ubuntu-latest"
|
||||
timeoutInMinutes: 30
|
||||
strategy:
|
||||
matrix:
|
||||
Shard1:
|
||||
SHARD_INDEX: 1
|
||||
Shard2:
|
||||
SHARD_INDEX: 2
|
||||
Shard3:
|
||||
SHARD_INDEX: 3
|
||||
Shard4:
|
||||
SHARD_INDEX: 4
|
||||
steps:
|
||||
- task: NodeTool@0
|
||||
inputs:
|
||||
versionSpec: $(DEFAULT_NODE_VERSION)
|
||||
displayName: "Setup Node.js"
|
||||
|
||||
- task: Cache@2
|
||||
inputs:
|
||||
key: 'npm | "$(Agent.OS)" | package-lock.json'
|
||||
restoreKeys: 'npm | "$(Agent.OS)"'
|
||||
path: $(npm_config_cache)
|
||||
displayName: "Cache npm"
|
||||
|
||||
- script: npm ci
|
||||
displayName: "Install dependencies" # Replace with INSTALL_CMD
|
||||
|
||||
# Frontend/Fullstack only — skipped for backend-only stacks
|
||||
- script: npx playwright install --with-deps chromium
|
||||
condition: ne(variables['TEST_STACK_TYPE'], 'backend')
|
||||
displayName: "Install Playwright browsers" # Replace with BROWSER_INSTALL
|
||||
|
||||
- script: npm run test:e2e -- --shard=$(SHARD_INDEX)/4
|
||||
displayName: "Run tests (shard $(SHARD_INDEX)/4)" # Replace with TEST_CMD + shard args
|
||||
|
||||
- task: PublishTestResults@2
|
||||
condition: always()
|
||||
inputs:
|
||||
testResultsFormat: "JUnit"
|
||||
testResultsFiles: "test-results/**/*.xml"
|
||||
mergeTestResults: true
|
||||
displayName: "Publish test results"
|
||||
|
||||
- publish: test-results/
|
||||
artifact: test-results-$(SHARD_INDEX)
|
||||
condition: failed()
|
||||
displayName: "Upload failure artifacts"
|
||||
|
||||
# Burn-in stage - Flaky test detection
|
||||
# Note: Burn-in targets UI flakiness. For backend-only stacks, remove this stage entirely.
|
||||
- stage: BurnIn
|
||||
displayName: "Burn-In (Flaky Detection)"
|
||||
dependsOn: Test
|
||||
condition: and(succeeded(), or(eq(variables['Build.Reason'], 'PullRequest'), eq(variables['Build.CronSchedule.DisplayName'], 'Weekly burn-in')))
|
||||
jobs:
|
||||
- job: BurnInJob
|
||||
displayName: "Burn-In Loop"
|
||||
pool:
|
||||
vmImage: "ubuntu-latest"
|
||||
timeoutInMinutes: 60
|
||||
steps:
|
||||
- task: NodeTool@0
|
||||
inputs:
|
||||
versionSpec: $(DEFAULT_NODE_VERSION)
|
||||
displayName: "Setup Node.js"
|
||||
|
||||
- script: npm ci
|
||||
displayName: "Install dependencies" # Replace with INSTALL_CMD
|
||||
|
||||
# Frontend/Fullstack only — skipped for backend-only stacks
|
||||
- script: npx playwright install --with-deps chromium
|
||||
condition: ne(variables['TEST_STACK_TYPE'], 'backend')
|
||||
displayName: "Install Playwright browsers" # Replace with BROWSER_INSTALL
|
||||
|
||||
- script: |
|
||||
echo "Starting burn-in loop - detecting flaky tests"
|
||||
for i in $(seq 1 10); do
|
||||
echo "Burn-in iteration $i/10"
|
||||
npm run test:e2e || exit 1
|
||||
done
|
||||
echo "Burn-in complete - no flaky tests detected"
|
||||
displayName: "Run burn-in loop (10 iterations)" # Replace npm run test:e2e with TEST_CMD
|
||||
|
||||
- publish: test-results/
|
||||
artifact: burn-in-failures
|
||||
condition: failed()
|
||||
displayName: "Upload burn-in failure artifacts"
|
||||
289
_bmad/tea/workflows/testarch/ci/checklist.md
Normal file
289
_bmad/tea/workflows/testarch/ci/checklist.md
Normal file
@@ -0,0 +1,289 @@
|
||||
# CI/CD Pipeline Setup - Validation Checklist
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- [ ] Git repository initialized (`.git/` exists)
|
||||
- [ ] Git remote configured (`git remote -v` shows origin)
|
||||
- [ ] Test framework configured (appropriate config for detected stack type)
|
||||
- [ ] Local tests pass (test command succeeds)
|
||||
- [ ] Team agrees on CI platform
|
||||
- [ ] Access to CI platform settings (if updating)
|
||||
|
||||
### Multi-Stack Detection
|
||||
|
||||
- [ ] Test stack type detected or configured (`frontend`, `backend`, `fullstack`)
|
||||
- [ ] Test framework detected or configured (Playwright, Cypress, Jest, Vitest, etc.)
|
||||
- [ ] Stack-appropriate test commands identified
|
||||
|
||||
### Multi-Platform Detection
|
||||
|
||||
- [ ] CI platform detected or configured
|
||||
- [ ] Supported platform: GitHub Actions, GitLab CI, Jenkins, Azure DevOps, Harness, or Circle CI
|
||||
- [ ] Platform-specific template selected
|
||||
|
||||
Note: CI setup is typically a one-time task per repo and can be run any time after the test framework is configured.
|
||||
|
||||
## Process Steps
|
||||
|
||||
### Step 1: Preflight Checks
|
||||
|
||||
- [ ] Git repository validated
|
||||
- [ ] Framework configuration detected
|
||||
- [ ] Local test execution successful
|
||||
- [ ] CI platform detected or selected
|
||||
- [ ] Node version identified (.nvmrc or default)
|
||||
- [ ] No blocking issues found
|
||||
|
||||
### Step 2: CI Pipeline Configuration
|
||||
|
||||
- [ ] CI configuration file created at platform-correct path
|
||||
- GitHub Actions: `.github/workflows/test.yml`
|
||||
- GitLab CI: `.gitlab-ci.yml`
|
||||
- Jenkins: `Jenkinsfile`
|
||||
- Azure DevOps: `azure-pipelines.yml`
|
||||
- Harness: `.harness/pipeline.yaml`
|
||||
- Circle CI: `.circleci/config.yml`
|
||||
- [ ] File is syntactically valid (no YAML/Groovy errors)
|
||||
- [ ] Correct framework commands configured for detected stack type
|
||||
- [ ] Node version matches project
|
||||
- [ ] Test directory paths correct
|
||||
- [ ] Stack-conditional steps applied:
|
||||
- [ ] Browser install included for frontend/fullstack stacks
|
||||
- [ ] Browser install omitted for backend-only stacks
|
||||
- [ ] Test commands match detected framework
|
||||
|
||||
### Step 3: Parallel Sharding
|
||||
|
||||
- [ ] Matrix strategy configured (4 shards default)
|
||||
- [ ] Shard syntax correct for framework
|
||||
- [ ] fail-fast set to false
|
||||
- [ ] Shard count appropriate for test suite size
|
||||
|
||||
### Step 4: Burn-In Loop
|
||||
|
||||
- [ ] Burn-in job created (frontend/fullstack stacks) or intentionally skipped (backend-only)
|
||||
- [ ] 10 iterations configured (when enabled)
|
||||
- [ ] Proper exit on failure (`|| exit 1`)
|
||||
- [ ] Runs on appropriate triggers (PR, cron)
|
||||
- [ ] Failure artifacts uploaded
|
||||
- [ ] Backend-only stacks: burn-in skipped by default (documented reason: targets UI flakiness)
|
||||
|
||||
### Step 5: Caching Configuration
|
||||
|
||||
- [ ] Dependency cache configured (npm/yarn)
|
||||
- [ ] Cache key uses lockfile hash
|
||||
- [ ] Browser cache configured (Playwright/Cypress)
|
||||
- [ ] Restore-keys defined for fallback
|
||||
- [ ] Cache paths correct for platform
|
||||
|
||||
### Step 6: Artifact Collection
|
||||
|
||||
- [ ] Artifacts upload on failure only
|
||||
- [ ] Correct artifact paths (test-results/, traces/, etc.)
|
||||
- [ ] Retention days set (30 default)
|
||||
- [ ] Artifact names unique per shard
|
||||
- [ ] No sensitive data in artifacts
|
||||
|
||||
### Step 7: Retry Logic
|
||||
|
||||
- [ ] Retry action/strategy configured
|
||||
- [ ] Max attempts: 2-3
|
||||
- [ ] Timeout appropriate (30 min)
|
||||
- [ ] Retry only on transient errors
|
||||
|
||||
### Step 8: Helper Scripts
|
||||
|
||||
- [ ] `scripts/test-changed.sh` created
|
||||
- [ ] `scripts/ci-local.sh` created
|
||||
- [ ] `scripts/burn-in.sh` created (optional)
|
||||
- [ ] Scripts are executable (`chmod +x`)
|
||||
- [ ] Scripts use correct test commands
|
||||
- [ ] Shebang present (`#!/bin/bash`)
|
||||
|
||||
### Step 9: Documentation
|
||||
|
||||
- [ ] `docs/ci.md` created with pipeline guide
|
||||
- [ ] `docs/ci-secrets-checklist.md` created
|
||||
- [ ] Required secrets documented
|
||||
- [ ] Setup instructions clear
|
||||
- [ ] Troubleshooting section included
|
||||
- [ ] Badge URLs provided (optional)
|
||||
|
||||
## Output Validation
|
||||
|
||||
### Configuration Validation
|
||||
|
||||
- [ ] CI file loads without errors
|
||||
- [ ] All paths resolve correctly
|
||||
- [ ] No hardcoded values (use env vars)
|
||||
- [ ] Triggers configured (push, pull_request, schedule)
|
||||
- [ ] Platform-specific syntax correct
|
||||
|
||||
### Execution Validation
|
||||
|
||||
- [ ] First CI run triggered (push to remote)
|
||||
- [ ] Pipeline starts without errors
|
||||
- [ ] All jobs appear in CI dashboard
|
||||
- [ ] Caching works (check logs for cache hit)
|
||||
- [ ] Tests execute in parallel
|
||||
- [ ] Artifacts collected on failure
|
||||
|
||||
### Performance Validation
|
||||
|
||||
- [ ] Lint stage: <2 minutes
|
||||
- [ ] Test stage (per shard): <10 minutes
|
||||
- [ ] Burn-in stage: <30 minutes
|
||||
- [ ] Total pipeline: <45 minutes
|
||||
- [ ] Cache reduces install time by 2-5 minutes
|
||||
|
||||
## Quality Checks
|
||||
|
||||
### Best Practices Compliance
|
||||
|
||||
- [ ] Burn-in loop follows production patterns
|
||||
- [ ] Parallel sharding configured optimally
|
||||
- [ ] Failure-only artifact collection
|
||||
- [ ] Selective testing enabled (optional)
|
||||
- [ ] Retry logic handles transient failures only
|
||||
- [ ] No secrets in configuration files
|
||||
|
||||
### Knowledge Base Alignment
|
||||
|
||||
- [ ] Burn-in pattern matches `ci-burn-in.md`
|
||||
- [ ] Selective testing matches `selective-testing.md`
|
||||
- [ ] Artifact collection matches `visual-debugging.md`
|
||||
- [ ] Test quality matches `test-quality.md`
|
||||
|
||||
### Security Checks
|
||||
|
||||
- [ ] No credentials in CI configuration
|
||||
- [ ] Secrets use platform secret management
|
||||
- [ ] Environment variables for sensitive data
|
||||
- [ ] Artifact retention appropriate (not too long)
|
||||
- [ ] No debug output exposing secrets
|
||||
- [ ] **MUST**: No `${{ inputs.* }}` or user-controlled GitHub context (`github.event.pull_request.title`, `github.event.issue.body`, `github.event.comment.body`, `github.head_ref`) directly in `run:` blocks — all passed through `env:` intermediaries and referenced as `"$ENV_VAR"`
|
||||
|
||||
## Integration Points
|
||||
|
||||
### Status File Integration
|
||||
|
||||
- [ ] CI setup logged in Quality & Testing Progress section
|
||||
- [ ] Status updated with completion timestamp
|
||||
- [ ] Platform and configuration noted
|
||||
|
||||
### Knowledge Base Integration
|
||||
|
||||
- [ ] Relevant knowledge fragments loaded
|
||||
- [ ] Patterns applied from knowledge base
|
||||
- [ ] Documentation references knowledge base
|
||||
- [ ] Knowledge base references in README
|
||||
|
||||
### Workflow Dependencies
|
||||
|
||||
- [ ] `framework` workflow completed first
|
||||
- [ ] Can proceed to `atdd` workflow after CI setup
|
||||
- [ ] Can proceed to `automate` workflow
|
||||
- [ ] CI integrates with `gate` workflow
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
**All must be true:**
|
||||
|
||||
- [ ] All prerequisites met
|
||||
- [ ] All process steps completed
|
||||
- [ ] All output validations passed
|
||||
- [ ] All quality checks passed
|
||||
- [ ] All integration points verified
|
||||
- [ ] First CI run successful
|
||||
- [ ] Performance targets met
|
||||
- [ ] Documentation complete
|
||||
|
||||
## Post-Workflow Actions
|
||||
|
||||
**User must complete:**
|
||||
|
||||
1. [ ] Commit CI configuration
|
||||
2. [ ] Push to remote repository
|
||||
3. [ ] Configure required secrets in CI platform
|
||||
4. [ ] Open PR to trigger first CI run
|
||||
5. [ ] Monitor and verify pipeline execution
|
||||
6. [ ] Adjust parallelism if needed (based on actual run times)
|
||||
7. [ ] Set up notifications (optional)
|
||||
|
||||
**Recommended next workflows:**
|
||||
|
||||
1. [ ] Run `atdd` workflow for test generation
|
||||
2. [ ] Run `automate` workflow for coverage expansion
|
||||
3. [ ] Run `gate` workflow for quality gates
|
||||
|
||||
## Rollback Procedure
|
||||
|
||||
If workflow fails:
|
||||
|
||||
1. [ ] Delete CI configuration file
|
||||
2. [ ] Remove helper scripts directory
|
||||
3. [ ] Remove documentation (docs/ci.md, etc.)
|
||||
4. [ ] Clear CI platform secrets (if added)
|
||||
5. [ ] Review error logs
|
||||
6. [ ] Fix issues and retry workflow
|
||||
|
||||
## Notes
|
||||
|
||||
### Common Issues
|
||||
|
||||
**Issue**: CI file syntax errors
|
||||
|
||||
- **Solution**: Validate YAML syntax online or with linter
|
||||
|
||||
**Issue**: Tests fail in CI but pass locally
|
||||
|
||||
- **Solution**: Use `scripts/ci-local.sh` to mirror CI environment
|
||||
|
||||
**Issue**: Caching not working
|
||||
|
||||
- **Solution**: Check cache key formula, verify paths
|
||||
|
||||
**Issue**: Burn-in too slow
|
||||
|
||||
- **Solution**: Reduce iterations or run on cron only
|
||||
|
||||
### Platform-Specific
|
||||
|
||||
**GitHub Actions:**
|
||||
|
||||
- Secrets: Repository Settings → Secrets and variables → Actions
|
||||
- Runners: Ubuntu latest recommended
|
||||
- Concurrency limits: 20 jobs for free tier
|
||||
|
||||
**GitLab CI:**
|
||||
|
||||
- Variables: Project Settings → CI/CD → Variables
|
||||
- Runners: Shared or project-specific
|
||||
- Pipeline quota: 400 minutes/month free tier
|
||||
|
||||
**Jenkins:**
|
||||
|
||||
- Credentials: Manage Jenkins → Manage Credentials
|
||||
- Agents: Configure build agents with Node.js
|
||||
- Plugins: Pipeline, JUnit, HTML Publisher recommended
|
||||
|
||||
**Azure DevOps:**
|
||||
|
||||
- Variables: Pipelines → Library → Variable groups
|
||||
- Agent pools: Azure-hosted or self-hosted
|
||||
- Parallel jobs: 1 free (Microsoft-hosted)
|
||||
|
||||
**Harness:**
|
||||
|
||||
- Connectors: Configure container registry and code repo connectors
|
||||
- Delegates: Install Harness delegate in target infrastructure
|
||||
- Steps: Use Run steps with appropriate container images
|
||||
|
||||
---
|
||||
|
||||
**Checklist Complete**: Sign off when all items validated.
|
||||
|
||||
**Completed by:** {name}
|
||||
**Date:** {date}
|
||||
**Platform:** {GitHub Actions, GitLab CI, Other}
|
||||
**Notes:** {notes}
|
||||
328
_bmad/tea/workflows/testarch/ci/github-actions-template.yaml
Normal file
328
_bmad/tea/workflows/testarch/ci/github-actions-template.yaml
Normal file
@@ -0,0 +1,328 @@
|
||||
# GitHub Actions CI/CD Pipeline for Test Execution
|
||||
# Generated by BMad TEA Agent - Test Architect Module
|
||||
# Optimized for: Parallel Sharding, Burn-In Loop
|
||||
# Stack: {test_stack_type} | Framework: {test_framework}
|
||||
#
|
||||
# Variables to customize per project:
|
||||
# INSTALL_CMD - dependency install command (e.g., npm ci, pnpm install --frozen-lockfile, yarn --frozen-lockfile)
|
||||
# TEST_CMD - main test command (e.g., npm run test:e2e, npm test, npx vitest)
|
||||
# LINT_CMD - lint command (e.g., npm run lint)
|
||||
# BROWSER_INSTALL - browser install command (frontend/fullstack only; omit for backend)
|
||||
# BROWSER_CACHE_PATH - browser cache path (frontend/fullstack only; omit for backend)
|
||||
|
||||
name: Test Pipeline
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main, develop]
|
||||
pull_request:
|
||||
branches: [main, develop]
|
||||
schedule:
|
||||
# Weekly burn-in on Sundays at 2 AM UTC
|
||||
- cron: "0 2 * * 0"
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
# Lint stage - Code quality checks
|
||||
lint:
|
||||
name: Lint
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Determine Node version
|
||||
id: node-version
|
||||
run: |
|
||||
if [ -f .nvmrc ]; then
|
||||
echo "value=$(cat .nvmrc)" >> "$GITHUB_OUTPUT"
|
||||
echo "Using Node from .nvmrc"
|
||||
else
|
||||
echo "value=24" >> "$GITHUB_OUTPUT"
|
||||
echo "Using default Node 24 (current LTS)"
|
||||
fi
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: ${{ steps.node-version.outputs.value }}
|
||||
cache: "npm"
|
||||
|
||||
- name: Install dependencies
|
||||
run: npm ci # Replace with INSTALL_CMD
|
||||
|
||||
- name: Run linter
|
||||
run: npm run lint # Replace with LINT_CMD
|
||||
|
||||
# Test stage - Parallel execution with sharding
|
||||
test:
|
||||
name: Test (Shard ${{ matrix.shard }})
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
needs: lint
|
||||
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
shard: [1, 2, 3, 4]
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Determine Node version
|
||||
id: node-version
|
||||
run: |
|
||||
if [ -f .nvmrc ]; then
|
||||
echo "value=$(cat .nvmrc)" >> "$GITHUB_OUTPUT"
|
||||
echo "Using Node from .nvmrc"
|
||||
else
|
||||
echo "value=22" >> "$GITHUB_OUTPUT"
|
||||
echo "Using default Node 22 (current LTS)"
|
||||
fi
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: ${{ steps.node-version.outputs.value }}
|
||||
cache: "npm"
|
||||
|
||||
- name: Cache Playwright browsers
|
||||
uses: actions/cache@v4
|
||||
with:
|
||||
path: ~/.cache/ms-playwright
|
||||
key: ${{ runner.os }}-playwright-${{ hashFiles('**/package-lock.json') }}
|
||||
restore-keys: |
|
||||
${{ runner.os }}-playwright-
|
||||
|
||||
- name: Install dependencies
|
||||
run: npm ci # Replace with INSTALL_CMD
|
||||
|
||||
# Frontend/Fullstack only — remove this step for backend-only stacks
|
||||
- name: Install Playwright browsers
|
||||
run: npx playwright install --with-deps chromium # Replace with BROWSER_INSTALL
|
||||
|
||||
- name: Run tests (shard ${{ matrix.shard }}/4)
|
||||
run: npm run test:e2e -- --shard=${{ matrix.shard }}/4 # Replace with TEST_CMD + shard args
|
||||
|
||||
- name: Upload test results
|
||||
if: failure()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: test-results-${{ matrix.shard }}
|
||||
path: |
|
||||
test-results/
|
||||
playwright-report/
|
||||
retention-days: 30
|
||||
|
||||
# Burn-in stage - Flaky test detection
|
||||
burn-in:
|
||||
name: Burn-In (Flaky Detection)
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 60
|
||||
needs: test
|
||||
# Only run burn-in on PRs to main/develop or on schedule
|
||||
if: github.event_name == 'pull_request' || github.event_name == 'schedule'
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Determine Node version
|
||||
id: node-version
|
||||
run: |
|
||||
if [ -f .nvmrc ]; then
|
||||
echo "value=$(cat .nvmrc)" >> "$GITHUB_OUTPUT"
|
||||
echo "Using Node from .nvmrc"
|
||||
else
|
||||
echo "value=22" >> "$GITHUB_OUTPUT"
|
||||
echo "Using default Node 22 (current LTS)"
|
||||
fi
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: ${{ steps.node-version.outputs.value }}
|
||||
cache: "npm"
|
||||
|
||||
# Frontend/Fullstack only — remove this step for backend-only stacks
|
||||
- name: Cache Playwright browsers
|
||||
uses: actions/cache@v4
|
||||
with:
|
||||
path: ~/.cache/ms-playwright # Replace with BROWSER_CACHE_PATH
|
||||
key: ${{ runner.os }}-playwright-${{ hashFiles('**/package-lock.json') }}
|
||||
|
||||
- name: Install dependencies
|
||||
run: npm ci # Replace with INSTALL_CMD
|
||||
|
||||
# Frontend/Fullstack only — remove this step for backend-only stacks
|
||||
- name: Install Playwright browsers
|
||||
run: npx playwright install --with-deps chromium # Replace with BROWSER_INSTALL
|
||||
|
||||
# Note: Burn-in targets UI flakiness. For backend-only stacks, remove this job entirely.
|
||||
- name: Run burn-in loop (10 iterations)
|
||||
run: |
|
||||
echo "🔥 Starting burn-in loop - detecting flaky tests"
|
||||
for i in {1..10}; do
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
echo "🔥 Burn-in iteration $i/10"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
npm run test:e2e || exit 1 # Replace with TEST_CMD
|
||||
done
|
||||
echo "✅ Burn-in complete - no flaky tests detected"
|
||||
|
||||
- name: Upload burn-in failure artifacts
|
||||
if: failure()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: burn-in-failures
|
||||
path: |
|
||||
test-results/
|
||||
playwright-report/
|
||||
retention-days: 30
|
||||
|
||||
# Report stage - Aggregate and publish results
|
||||
report:
|
||||
name: Test Report
|
||||
runs-on: ubuntu-latest
|
||||
needs: [test, burn-in]
|
||||
if: always()
|
||||
|
||||
steps:
|
||||
- name: Download all artifacts
|
||||
uses: actions/download-artifact@v4
|
||||
with:
|
||||
path: artifacts
|
||||
|
||||
- name: Generate summary
|
||||
run: |
|
||||
echo "## Test Execution Summary" >> $GITHUB_STEP_SUMMARY
|
||||
echo "" >> $GITHUB_STEP_SUMMARY
|
||||
echo "- **Status**: ${{ needs.test.result }}" >> $GITHUB_STEP_SUMMARY
|
||||
echo "- **Burn-in**: ${{ needs.burn-in.result }}" >> $GITHUB_STEP_SUMMARY
|
||||
echo "- **Shards**: 4" >> $GITHUB_STEP_SUMMARY
|
||||
echo "" >> $GITHUB_STEP_SUMMARY
|
||||
|
||||
if [ "${{ needs.burn-in.result }}" == "failure" ]; then
|
||||
echo "⚠️ **Flaky tests detected** - Review burn-in artifacts" >> $GITHUB_STEP_SUMMARY
|
||||
fi
|
||||
|
||||
# ============================================================================
|
||||
# EXTENSION PATTERNS — Script Injection Prevention
|
||||
# ============================================================================
|
||||
# When extending this template into reusable workflows, manual dispatch
|
||||
# workflows, or composite actions, NEVER use ${{ inputs.* }} directly in
|
||||
# run: blocks. Always pass through env: intermediaries.
|
||||
#
|
||||
# KEY PRINCIPLE: Inputs must be DATA, not COMMANDS.
|
||||
# Pass inputs through env: and interpolate as quoted arguments into fixed
|
||||
# commands. NEVER accept command-shaped inputs (e.g., install-command,
|
||||
# test-command) that get executed as shell code — even through env:.
|
||||
#
|
||||
# --- Reusable Workflow (workflow_call) ---
|
||||
#
|
||||
# on:
|
||||
# workflow_call:
|
||||
# inputs:
|
||||
# test-grep:
|
||||
# description: 'Test grep filter (data only — not a command)'
|
||||
# type: string
|
||||
# required: false
|
||||
# default: ''
|
||||
# base-ref:
|
||||
# description: 'Base branch for diff'
|
||||
# type: string
|
||||
# required: false
|
||||
# default: 'main'
|
||||
# burn-in-count:
|
||||
# description: 'Number of burn-in iterations'
|
||||
# type: string
|
||||
# required: false
|
||||
# default: '10'
|
||||
#
|
||||
# jobs:
|
||||
# test:
|
||||
# runs-on: ubuntu-latest
|
||||
# steps:
|
||||
# - uses: actions/checkout@v4
|
||||
# # Fixed command — not derived from inputs
|
||||
# - name: Install dependencies
|
||||
# run: npm ci
|
||||
# # ✅ SAFE — input is DATA passed as an argument to a fixed command
|
||||
# - name: Run tests
|
||||
# env:
|
||||
# TEST_GREP: ${{ inputs.test-grep }}
|
||||
# run: |
|
||||
# # Security: inputs passed through env: to prevent script injection
|
||||
# if [ -n "$TEST_GREP" ]; then
|
||||
# npx playwright test --grep "$TEST_GREP"
|
||||
# else
|
||||
# npx playwright test
|
||||
# fi
|
||||
#
|
||||
# --- Manual Dispatch (workflow_dispatch) ---
|
||||
#
|
||||
# on:
|
||||
# workflow_dispatch:
|
||||
# inputs:
|
||||
# test-grep:
|
||||
# description: 'Test grep filter (data only — not a command)'
|
||||
# type: string
|
||||
# required: false
|
||||
# environment:
|
||||
# description: 'Target environment'
|
||||
# type: choice
|
||||
# options: [staging, production]
|
||||
#
|
||||
# jobs:
|
||||
# run-tests:
|
||||
# runs-on: ubuntu-latest
|
||||
# steps:
|
||||
# - uses: actions/checkout@v4
|
||||
# # ✅ SAFE — input is DATA interpolated into a fixed command
|
||||
# - name: Run selected tests
|
||||
# env:
|
||||
# TEST_GREP: ${{ inputs.test-grep }}
|
||||
# run: |
|
||||
# # Security: inputs passed through env: to prevent script injection
|
||||
# npx playwright test --grep "$TEST_GREP"
|
||||
#
|
||||
# --- Composite Action (action.yml) ---
|
||||
#
|
||||
# inputs:
|
||||
# test-grep:
|
||||
# description: 'Test grep filter (data only — not a command)'
|
||||
# required: false
|
||||
# default: ''
|
||||
# burn-in-count:
|
||||
# description: 'Number of burn-in iterations'
|
||||
# required: false
|
||||
# default: '10'
|
||||
#
|
||||
# runs:
|
||||
# using: composite
|
||||
# steps:
|
||||
# # ✅ SAFE — inputs are DATA arguments to fixed commands
|
||||
# - name: Run burn-in
|
||||
# shell: bash
|
||||
# env:
|
||||
# TEST_GREP: ${{ inputs.test-grep }}
|
||||
# BURN_IN_COUNT: ${{ inputs.burn-in-count }}
|
||||
# run: |
|
||||
# # Security: inputs passed through env: to prevent script injection
|
||||
# for i in $(seq 1 "$BURN_IN_COUNT"); do
|
||||
# echo "Burn-in iteration $i/$BURN_IN_COUNT"
|
||||
# npx playwright test --grep "$TEST_GREP" || exit 1
|
||||
# done
|
||||
#
|
||||
# ❌ NEVER DO THIS:
|
||||
# # Direct ${{ inputs.* }} in run: — GitHub expression injection
|
||||
# - run: npx playwright test --grep "${{ inputs.test-grep }}"
|
||||
#
|
||||
# # Executing input-derived env var as a command — still command injection
|
||||
# - env:
|
||||
# CMD: ${{ inputs.test-command }}
|
||||
# run: $CMD
|
||||
# ============================================================================
|
||||
158
_bmad/tea/workflows/testarch/ci/gitlab-ci-template.yaml
Normal file
158
_bmad/tea/workflows/testarch/ci/gitlab-ci-template.yaml
Normal file
@@ -0,0 +1,158 @@
|
||||
# GitLab CI/CD Pipeline for Test Execution
|
||||
# Generated by BMad TEA Agent - Test Architect Module
|
||||
# Optimized for: Parallel Sharding, Burn-In Loop
|
||||
# Stack: {test_stack_type} | Framework: {test_framework}
|
||||
#
|
||||
# Variables to customize per project:
|
||||
# INSTALL_CMD - dependency install command (e.g., npm ci, pnpm install --frozen-lockfile)
|
||||
# TEST_CMD - main test command (e.g., npm run test:e2e, npm test, npx vitest)
|
||||
# LINT_CMD - lint command (e.g., npm run lint)
|
||||
# BROWSER_INSTALL - browser install command (frontend/fullstack only; omit for backend)
|
||||
# BROWSER_CACHE_PATH - browser cache path (frontend/fullstack only; omit for backend)
|
||||
|
||||
stages:
|
||||
- lint
|
||||
- test
|
||||
- burn-in
|
||||
- report
|
||||
|
||||
variables:
|
||||
# Disable git depth for accurate change detection
|
||||
GIT_DEPTH: 0
|
||||
# Use npm ci for faster, deterministic installs
|
||||
npm_config_cache: "$CI_PROJECT_DIR/.npm"
|
||||
# Playwright browser cache
|
||||
PLAYWRIGHT_BROWSERS_PATH: "$CI_PROJECT_DIR/.cache/ms-playwright"
|
||||
# Default Node version when .nvmrc is missing
|
||||
DEFAULT_NODE_VERSION: "24"
|
||||
|
||||
# Caching configuration
|
||||
cache:
|
||||
key:
|
||||
files:
|
||||
- package-lock.json
|
||||
paths:
|
||||
- .npm/
|
||||
- .cache/ms-playwright/
|
||||
- node_modules/
|
||||
|
||||
# Lint stage - Code quality checks
|
||||
lint:
|
||||
stage: lint
|
||||
image: node:$DEFAULT_NODE_VERSION
|
||||
before_script:
|
||||
- |
|
||||
NODE_VERSION=$(cat .nvmrc 2>/dev/null || echo "$DEFAULT_NODE_VERSION")
|
||||
echo "Using Node $NODE_VERSION"
|
||||
npm install -g n
|
||||
n "$NODE_VERSION"
|
||||
node -v
|
||||
- npm ci # Replace with INSTALL_CMD
|
||||
script:
|
||||
- npm run lint # Replace with LINT_CMD
|
||||
timeout: 5 minutes
|
||||
|
||||
# Test stage - Parallel execution with sharding
|
||||
.test-template: &test-template
|
||||
stage: test
|
||||
image: node:$DEFAULT_NODE_VERSION
|
||||
needs:
|
||||
- lint
|
||||
before_script:
|
||||
- |
|
||||
NODE_VERSION=$(cat .nvmrc 2>/dev/null || echo "$DEFAULT_NODE_VERSION")
|
||||
echo "Using Node $NODE_VERSION"
|
||||
npm install -g n
|
||||
n "$NODE_VERSION"
|
||||
node -v
|
||||
- npm ci # Replace with INSTALL_CMD
|
||||
- npx playwright install --with-deps chromium # Replace with BROWSER_INSTALL; remove for backend-only
|
||||
artifacts:
|
||||
when: on_failure
|
||||
paths:
|
||||
- test-results/
|
||||
- playwright-report/
|
||||
expire_in: 30 days
|
||||
timeout: 30 minutes
|
||||
|
||||
test:shard-1:
|
||||
<<: *test-template
|
||||
script:
|
||||
- npm run test:e2e -- --shard=1/4 # Replace with TEST_CMD + shard args
|
||||
|
||||
test:shard-2:
|
||||
<<: *test-template
|
||||
script:
|
||||
- npm run test:e2e -- --shard=2/4 # Replace with TEST_CMD + shard args
|
||||
|
||||
test:shard-3:
|
||||
<<: *test-template
|
||||
script:
|
||||
- npm run test:e2e -- --shard=3/4 # Replace with TEST_CMD + shard args
|
||||
|
||||
test:shard-4:
|
||||
<<: *test-template
|
||||
script:
|
||||
- npm run test:e2e -- --shard=4/4 # Replace with TEST_CMD + shard args
|
||||
|
||||
# Burn-in stage - Flaky test detection
|
||||
burn-in:
|
||||
stage: burn-in
|
||||
image: node:$DEFAULT_NODE_VERSION
|
||||
needs:
|
||||
- test:shard-1
|
||||
- test:shard-2
|
||||
- test:shard-3
|
||||
- test:shard-4
|
||||
# Only run burn-in on merge requests to main/develop or on schedule
|
||||
rules:
|
||||
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
|
||||
- if: '$CI_PIPELINE_SOURCE == "schedule"'
|
||||
before_script:
|
||||
- |
|
||||
NODE_VERSION=$(cat .nvmrc 2>/dev/null || echo "$DEFAULT_NODE_VERSION")
|
||||
echo "Using Node $NODE_VERSION"
|
||||
npm install -g n
|
||||
n "$NODE_VERSION"
|
||||
node -v
|
||||
- npm ci # Replace with INSTALL_CMD
|
||||
- npx playwright install --with-deps chromium # Replace with BROWSER_INSTALL; remove for backend-only
|
||||
# Note: Burn-in targets UI flakiness. For backend-only stacks, remove this job entirely.
|
||||
script:
|
||||
- |
|
||||
echo "🔥 Starting burn-in loop - detecting flaky tests"
|
||||
for i in {1..10}; do
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
echo "🔥 Burn-in iteration $i/10"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
npm run test:e2e || exit 1 # Replace with TEST_CMD
|
||||
done
|
||||
echo "✅ Burn-in complete - no flaky tests detected"
|
||||
artifacts:
|
||||
when: on_failure
|
||||
paths:
|
||||
- test-results/
|
||||
- playwright-report/
|
||||
expire_in: 30 days
|
||||
timeout: 60 minutes
|
||||
|
||||
# Report stage - Aggregate results
|
||||
report:
|
||||
stage: report
|
||||
image: alpine:latest
|
||||
needs:
|
||||
- test:shard-1
|
||||
- test:shard-2
|
||||
- test:shard-3
|
||||
- test:shard-4
|
||||
- burn-in
|
||||
when: always
|
||||
script:
|
||||
- |
|
||||
echo "## Test Execution Summary"
|
||||
echo ""
|
||||
echo "- Pipeline: $CI_PIPELINE_ID"
|
||||
echo "- Shards: 4"
|
||||
echo "- Branch: $CI_COMMIT_REF_NAME"
|
||||
echo ""
|
||||
echo "View detailed results in job artifacts"
|
||||
159
_bmad/tea/workflows/testarch/ci/harness-pipeline-template.yaml
Normal file
159
_bmad/tea/workflows/testarch/ci/harness-pipeline-template.yaml
Normal file
@@ -0,0 +1,159 @@
|
||||
# Harness CI Pipeline for Test Execution
|
||||
# Generated by BMad TEA Agent - Test Architect Module
|
||||
# Optimized for: Parallel Sharding, Burn-In Loop
|
||||
# Stack: {test_stack_type} | Framework: {test_framework}
|
||||
#
|
||||
# Variables to customize per project:
|
||||
# INSTALL_CMD - dependency install command (e.g., npm ci, pnpm install --frozen-lockfile)
|
||||
# TEST_CMD - main test command (e.g., npm run test:e2e, npm test, npx vitest)
|
||||
# LINT_CMD - lint command (e.g., npm run lint)
|
||||
# BROWSER_INSTALL - browser install command (frontend/fullstack only; omit for backend)
|
||||
|
||||
pipeline:
|
||||
name: Test Pipeline
|
||||
identifier: test_pipeline
|
||||
projectIdentifier: default
|
||||
orgIdentifier: default
|
||||
stages:
|
||||
# Lint stage - Code quality checks
|
||||
- stage:
|
||||
name: Lint
|
||||
identifier: lint
|
||||
type: CI
|
||||
spec:
|
||||
cloneCodebase: true
|
||||
infrastructure:
|
||||
type: KubernetesDirect
|
||||
spec:
|
||||
connectorRef: account.harnessImage
|
||||
namespace: default
|
||||
execution:
|
||||
steps:
|
||||
- step:
|
||||
type: Run
|
||||
name: Install dependencies
|
||||
identifier: install
|
||||
spec:
|
||||
connectorRef: account.harnessImage
|
||||
image: node:24
|
||||
shell: Sh
|
||||
command: npm ci # Replace with INSTALL_CMD
|
||||
|
||||
- step:
|
||||
type: Run
|
||||
name: Run linter
|
||||
identifier: lint
|
||||
spec:
|
||||
connectorRef: account.harnessImage
|
||||
image: node:24
|
||||
shell: Sh
|
||||
command: npm run lint # Replace with LINT_CMD
|
||||
|
||||
# Test stage - Parallel execution with sharding
|
||||
- stage:
|
||||
name: Test
|
||||
identifier: test
|
||||
type: CI
|
||||
spec:
|
||||
cloneCodebase: true
|
||||
infrastructure:
|
||||
type: KubernetesDirect
|
||||
spec:
|
||||
connectorRef: account.harnessImage
|
||||
namespace: default
|
||||
execution:
|
||||
steps:
|
||||
- step:
|
||||
type: Run
|
||||
name: Install dependencies
|
||||
identifier: install
|
||||
spec:
|
||||
connectorRef: account.harnessImage
|
||||
image: node:24
|
||||
shell: Sh
|
||||
command: npm ci # Replace with INSTALL_CMD
|
||||
|
||||
# Frontend/Fullstack only — remove this step for backend-only stacks
|
||||
- step:
|
||||
type: Run
|
||||
name: Install browsers
|
||||
identifier: browsers
|
||||
spec:
|
||||
connectorRef: account.harnessImage
|
||||
image: mcr.microsoft.com/playwright:v1.50.0-noble
|
||||
shell: Sh
|
||||
command: npx playwright install --with-deps chromium # Replace with BROWSER_INSTALL
|
||||
|
||||
- parallel:
|
||||
- step:
|
||||
type: Run
|
||||
name: Test Shard 1
|
||||
identifier: shard_1
|
||||
spec:
|
||||
connectorRef: account.harnessImage
|
||||
image: mcr.microsoft.com/playwright:v1.50.0-noble
|
||||
shell: Sh
|
||||
command: npm run test:e2e -- --shard=1/4 # Replace with TEST_CMD + shard args
|
||||
- step:
|
||||
type: Run
|
||||
name: Test Shard 2
|
||||
identifier: shard_2
|
||||
spec:
|
||||
connectorRef: account.harnessImage
|
||||
image: mcr.microsoft.com/playwright:v1.50.0-noble
|
||||
shell: Sh
|
||||
command: npm run test:e2e -- --shard=2/4 # Replace with TEST_CMD + shard args
|
||||
- step:
|
||||
type: Run
|
||||
name: Test Shard 3
|
||||
identifier: shard_3
|
||||
spec:
|
||||
connectorRef: account.harnessImage
|
||||
image: mcr.microsoft.com/playwright:v1.50.0-noble
|
||||
shell: Sh
|
||||
command: npm run test:e2e -- --shard=3/4 # Replace with TEST_CMD + shard args
|
||||
- step:
|
||||
type: Run
|
||||
name: Test Shard 4
|
||||
identifier: shard_4
|
||||
spec:
|
||||
connectorRef: account.harnessImage
|
||||
image: mcr.microsoft.com/playwright:v1.50.0-noble
|
||||
shell: Sh
|
||||
command: npm run test:e2e -- --shard=4/4 # Replace with TEST_CMD + shard args
|
||||
|
||||
# Burn-in stage - Flaky test detection
|
||||
# Note: Burn-in targets UI flakiness. For backend-only stacks, remove this stage entirely.
|
||||
- stage:
|
||||
name: Burn-In
|
||||
identifier: burn_in
|
||||
type: CI
|
||||
when:
|
||||
condition: <+pipeline.triggerType> == "WEBHOOK" || <+pipeline.triggerType> == "SCHEDULER"
|
||||
spec:
|
||||
cloneCodebase: true
|
||||
infrastructure:
|
||||
type: KubernetesDirect
|
||||
spec:
|
||||
connectorRef: account.harnessImage
|
||||
namespace: default
|
||||
execution:
|
||||
steps:
|
||||
- step:
|
||||
type: Run
|
||||
name: Install and burn-in
|
||||
identifier: burn_in_loop
|
||||
spec:
|
||||
connectorRef: account.harnessImage
|
||||
image: mcr.microsoft.com/playwright:v1.50.0-noble
|
||||
shell: Sh
|
||||
command: |
|
||||
npm ci
|
||||
npx playwright install --with-deps chromium
|
||||
echo "Starting burn-in loop - detecting flaky tests"
|
||||
for i in $(seq 1 10); do
|
||||
echo "Burn-in iteration $i/10"
|
||||
npm run test:e2e || exit 1
|
||||
done
|
||||
echo "Burn-in complete - no flaky tests detected"
|
||||
# Replace npm ci with INSTALL_CMD, npm run test:e2e with TEST_CMD
|
||||
45
_bmad/tea/workflows/testarch/ci/instructions.md
Normal file
45
_bmad/tea/workflows/testarch/ci/instructions.md
Normal file
@@ -0,0 +1,45 @@
|
||||
<!-- Powered by BMAD-CORE™ -->
|
||||
|
||||
# CI/CD Pipeline Setup
|
||||
|
||||
**Workflow ID**: `_bmad/tea/testarch/ci`
|
||||
**Version**: 5.0 (Step-File Architecture)
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Scaffold a production-ready CI/CD quality pipeline with test execution, burn-in loops for flaky detection, parallel sharding, artifact collection, and notifications.
|
||||
|
||||
---
|
||||
|
||||
## WORKFLOW ARCHITECTURE
|
||||
|
||||
This workflow uses **step-file architecture**:
|
||||
|
||||
- **Micro-file Design**: Each step is self-contained
|
||||
- **JIT Loading**: Only the current step file is in memory
|
||||
- **Sequential Enforcement**: Execute steps in order
|
||||
|
||||
---
|
||||
|
||||
## INITIALIZATION SEQUENCE
|
||||
|
||||
### 1. Configuration Loading
|
||||
|
||||
From `workflow.yaml`, resolve:
|
||||
|
||||
- `config_source`, `test_artifacts`, `user_name`, `communication_language`, `document_output_language`, `date`
|
||||
- `ci_platform`, `test_dir`
|
||||
|
||||
### 2. First Step
|
||||
|
||||
Load, read completely, and execute:
|
||||
`{project-root}/_bmad/tea/workflows/testarch/ci/steps-c/step-01-preflight.md`
|
||||
|
||||
### 3. Resume Support
|
||||
|
||||
If the user selects **Resume** mode, load, read completely, and execute:
|
||||
`{project-root}/_bmad/tea/workflows/testarch/ci/steps-c/step-01b-resume.md`
|
||||
|
||||
This checks the output document for progress tracking frontmatter and routes to the next incomplete step.
|
||||
129
_bmad/tea/workflows/testarch/ci/jenkins-pipeline-template.groovy
Normal file
129
_bmad/tea/workflows/testarch/ci/jenkins-pipeline-template.groovy
Normal file
@@ -0,0 +1,129 @@
|
||||
// Jenkinsfile CI/CD Pipeline for Test Execution
|
||||
// Generated by BMad TEA Agent - Test Architect Module
|
||||
// Optimized for: Parallel Sharding, Burn-In Loop
|
||||
// Stack: {test_stack_type} | Framework: {test_framework}
|
||||
//
|
||||
// Variables to customize per project:
|
||||
// INSTALL_CMD - dependency install command (e.g., npm ci, pnpm install --frozen-lockfile)
|
||||
// TEST_CMD - main test command (e.g., npm run test:e2e, npm test, npx vitest)
|
||||
// LINT_CMD - lint command (e.g., npm run lint)
|
||||
// BROWSER_INSTALL - browser install command (frontend/fullstack only; omit for backend)
|
||||
//
|
||||
// Node.js version management — choose one:
|
||||
// Option A (recommended): Configure NodeJS Plugin in Jenkins Global Tool Configuration,
|
||||
// then add to pipeline: tools { nodejs 'NodeJS-24' }
|
||||
// Option B: Use nvm (pre-installed on agent) — this template uses nvm as the default
|
||||
// Option C: Use a Docker agent — agent { docker { image 'node:24' } }
|
||||
|
||||
pipeline {
|
||||
agent any
|
||||
|
||||
environment {
|
||||
CI = 'true'
|
||||
}
|
||||
|
||||
options {
|
||||
timeout(time: 45, unit: 'MINUTES')
|
||||
disableConcurrentBuilds()
|
||||
}
|
||||
|
||||
stages {
|
||||
stage('Checkout') {
|
||||
steps {
|
||||
checkout scm
|
||||
}
|
||||
}
|
||||
|
||||
stage('Install') {
|
||||
steps {
|
||||
// Detect and apply Node.js version from .nvmrc (falls back to v24)
|
||||
// If using NodeJS Plugin instead, remove this block and add: tools { nodejs 'NodeJS-24' }
|
||||
sh '''
|
||||
export NVM_DIR="$HOME/.nvm"
|
||||
[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh"
|
||||
NODE_VERSION=$(cat .nvmrc 2>/dev/null || echo "24")
|
||||
nvm install "$NODE_VERSION" 2>/dev/null || true
|
||||
nvm use "$NODE_VERSION" 2>/dev/null || true
|
||||
node --version
|
||||
npm ci
|
||||
''' // Replace npm ci with INSTALL_CMD
|
||||
// Stash installed dependencies so parallel shards can restore them
|
||||
stash includes: 'node_modules/**', name: 'deps'
|
||||
}
|
||||
}
|
||||
|
||||
stage('Lint') {
|
||||
steps {
|
||||
sh 'npm run lint' // Replace with LINT_CMD
|
||||
}
|
||||
}
|
||||
|
||||
// Test stage - Parallel execution with sharding
|
||||
// Each shard restores dependencies via unstash for workspace safety
|
||||
stage('Test') {
|
||||
parallel {
|
||||
stage('Shard 1') {
|
||||
steps {
|
||||
unstash 'deps'
|
||||
// Frontend/Fullstack only — remove browser install for backend-only stacks
|
||||
sh 'npx playwright install --with-deps chromium' // Replace with BROWSER_INSTALL
|
||||
sh 'npm run test:e2e -- --shard=1/4' // Replace with TEST_CMD + shard args
|
||||
}
|
||||
}
|
||||
stage('Shard 2') {
|
||||
steps {
|
||||
unstash 'deps'
|
||||
sh 'npx playwright install --with-deps chromium' // Replace with BROWSER_INSTALL
|
||||
sh 'npm run test:e2e -- --shard=2/4' // Replace with TEST_CMD + shard args
|
||||
}
|
||||
}
|
||||
stage('Shard 3') {
|
||||
steps {
|
||||
unstash 'deps'
|
||||
sh 'npx playwright install --with-deps chromium' // Replace with BROWSER_INSTALL
|
||||
sh 'npm run test:e2e -- --shard=3/4' // Replace with TEST_CMD + shard args
|
||||
}
|
||||
}
|
||||
stage('Shard 4') {
|
||||
steps {
|
||||
unstash 'deps'
|
||||
sh 'npx playwright install --with-deps chromium' // Replace with BROWSER_INSTALL
|
||||
sh 'npm run test:e2e -- --shard=4/4' // Replace with TEST_CMD + shard args
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Burn-in stage - Flaky test detection
|
||||
// Note: Burn-in targets UI flakiness. For backend-only stacks, remove this stage entirely.
|
||||
stage('Burn-In') {
|
||||
when {
|
||||
anyOf {
|
||||
changeRequest()
|
||||
triggeredBy 'TimerTrigger'
|
||||
}
|
||||
}
|
||||
steps {
|
||||
sh '''
|
||||
echo "Starting burn-in loop - detecting flaky tests"
|
||||
for i in $(seq 1 10); do
|
||||
echo "Burn-in iteration $i/10"
|
||||
npm run test:e2e || exit 1
|
||||
done
|
||||
echo "Burn-in complete - no flaky tests detected"
|
||||
''' // Replace npm run test:e2e with TEST_CMD
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
post {
|
||||
always {
|
||||
// Archive test results and reports
|
||||
archiveArtifacts artifacts: 'test-results/**,playwright-report/**', allowEmptyArchive: true
|
||||
junit testResults: 'test-results/**/*.xml', allowEmptyResults: true
|
||||
}
|
||||
failure {
|
||||
echo 'Pipeline failed - check test results and artifacts'
|
||||
}
|
||||
}
|
||||
}
|
||||
158
_bmad/tea/workflows/testarch/ci/steps-c/step-01-preflight.md
Normal file
158
_bmad/tea/workflows/testarch/ci/steps-c/step-01-preflight.md
Normal file
@@ -0,0 +1,158 @@
|
||||
---
|
||||
name: 'step-01-preflight'
|
||||
description: 'Verify prerequisites and detect CI platform'
|
||||
nextStepFile: './step-02-generate-pipeline.md'
|
||||
outputFile: '{test_artifacts}/ci-pipeline-progress.md'
|
||||
---
|
||||
|
||||
# Step 1: Preflight Checks
|
||||
|
||||
## STEP GOAL
|
||||
|
||||
Verify CI prerequisites and determine target CI platform.
|
||||
|
||||
## MANDATORY EXECUTION RULES
|
||||
|
||||
- 📖 Read the entire step file before acting
|
||||
- ✅ Speak in `{communication_language}`
|
||||
- 🚫 Halt if requirements fail
|
||||
|
||||
---
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
- 💾 Record outputs before proceeding
|
||||
- 📖 Load the next step only when instructed
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: config, loaded artifacts, and knowledge fragments
|
||||
- Focus: this step's goal only
|
||||
- Limits: do not execute future steps
|
||||
- Dependencies: prior steps' outputs (if any)
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly. Do not skip, reorder, or improvise.
|
||||
|
||||
## 1. Verify Git Repository
|
||||
|
||||
- `.git/` exists
|
||||
- Remote configured (if available)
|
||||
|
||||
If missing: **HALT** with "Git repository required for CI/CD setup."
|
||||
|
||||
---
|
||||
|
||||
## 2. Detect Test Stack Type
|
||||
|
||||
Determine the project's test stack type (`test_stack_type`) using the following algorithm:
|
||||
|
||||
1. If `test_stack_type` is explicitly set in config (not `"auto"`), use that value.
|
||||
2. Otherwise, auto-detect by scanning project manifests:
|
||||
- **Frontend indicators**: `playwright.config.*`, `cypress.config.*`, `vite.config.*`, `next.config.*`, `src/components/`, `src/pages/`, `src/app/`
|
||||
- **Backend indicators**: `pyproject.toml`, `pom.xml`/`build.gradle`, `go.mod`, `*.csproj`/`*.sln`, `Gemfile`, `Cargo.toml`, `jest.config.*`, `vitest.config.*`, `src/routes/`, `src/controllers/`, `src/api/`, `Dockerfile`, `serverless.yml`
|
||||
- **Both present** → `fullstack`
|
||||
- **Only frontend** → `frontend`
|
||||
- **Only backend** → `backend`
|
||||
- **Cannot determine** → default to `fullstack` and note assumption
|
||||
|
||||
Record detected `test_stack_type` in step output.
|
||||
|
||||
---
|
||||
|
||||
## 3. Verify Test Framework
|
||||
|
||||
- Check for framework configuration based on detected stack:
|
||||
- **Frontend/Fullstack**: `playwright.config.*` or `cypress.config.*` exists
|
||||
- **Backend (Node.js)**: `jest.config.*` or `vitest.config.*` or test scripts in `package.json`
|
||||
- **Backend (Python)**: `pyproject.toml` with `[tool.pytest]` or `pytest.ini` or `setup.cfg` with pytest config
|
||||
- **Backend (Java/Kotlin)**: `pom.xml` with surefire/failsafe plugins or `build.gradle` with test task
|
||||
- **Backend (Go)**: `*_test.go` files present (Go convention — no config file needed)
|
||||
- **Backend (C#/.NET)**: `*.csproj` with xUnit/NUnit/MSTest references
|
||||
- **Backend (Ruby)**: `Gemfile` with rspec or `.rspec` config file
|
||||
- If `test_framework` is `"auto"`, detect from config files and project manifests found
|
||||
- Verify test dependencies are installed (language-appropriate package manager)
|
||||
|
||||
If missing: **HALT** with "Run `framework` workflow first."
|
||||
|
||||
---
|
||||
|
||||
## 4. Ensure Tests Pass Locally
|
||||
|
||||
- Run the main test command based on detected stack and framework:
|
||||
- **Node.js**: `npm test` or `npm run test:e2e`
|
||||
- **Python**: `pytest` or `python -m pytest`
|
||||
- **Java**: `mvn test` or `gradle test`
|
||||
- **Go**: `go test ./...`
|
||||
- **C#/.NET**: `dotnet test`
|
||||
- **Ruby**: `bundle exec rspec`
|
||||
- If failing: **HALT** and request fixes before CI setup
|
||||
|
||||
---
|
||||
|
||||
## 5. Detect CI Platform
|
||||
|
||||
- If `ci_platform` is explicitly set in config (not `"auto"`), use that value.
|
||||
- Otherwise, scan for existing CI configuration files:
|
||||
- `.github/workflows/*.yml` → `github-actions`
|
||||
- `.gitlab-ci.yml` → `gitlab-ci`
|
||||
- `Jenkinsfile` → `jenkins`
|
||||
- `azure-pipelines.yml` → `azure-devops`
|
||||
- `.harness/*.yaml` → `harness`
|
||||
- `.circleci/config.yml` → `circle-ci`
|
||||
- If found, ask whether to update or replace
|
||||
- If not found, infer from git remote (github.com → `github-actions`, gitlab.com → `gitlab-ci`)
|
||||
- If still unresolved, default to `github-actions`
|
||||
|
||||
Record detected `ci_platform` in step output.
|
||||
|
||||
---
|
||||
|
||||
## 6. Read Environment Context
|
||||
|
||||
- Read environment context based on detected stack:
|
||||
- **Node.js**: Read `.nvmrc` if present (default to Node 24+ LTS if missing); read `package.json` for dependency caching strategy
|
||||
- **Python**: Read `.python-version` or `pyproject.toml` for Python version; note `pip`/`poetry`/`pipenv` for caching
|
||||
- **Java**: Read `pom.xml`/`build.gradle` for Java version; note Maven/Gradle for caching
|
||||
- **Go**: Read `go.mod` for Go version; note Go module cache path
|
||||
- **C#/.NET**: Read `*.csproj`/`global.json` for .NET SDK version; note NuGet cache
|
||||
- **Ruby**: Read `.ruby-version` or `Gemfile` for Ruby version; note Bundler cache
|
||||
|
||||
---
|
||||
|
||||
### 7. Save Progress
|
||||
|
||||
**Save this step's accumulated work to `{outputFile}`.**
|
||||
|
||||
- **If `{outputFile}` does not exist** (first save), create it with YAML frontmatter:
|
||||
|
||||
```yaml
|
||||
---
|
||||
stepsCompleted: ['step-01-preflight']
|
||||
lastStep: 'step-01-preflight'
|
||||
lastSaved: '{date}'
|
||||
---
|
||||
```
|
||||
|
||||
Then write this step's output below the frontmatter.
|
||||
|
||||
- **If `{outputFile}` already exists**, update:
|
||||
- Add `'step-01-preflight'` to `stepsCompleted` array (only if not already present)
|
||||
- Set `lastStep: 'step-01-preflight'`
|
||||
- Set `lastSaved: '{date}'`
|
||||
- Append this step's output to the appropriate section of the document.
|
||||
|
||||
Load next step: `{nextStepFile}`
|
||||
|
||||
## 🚨 SYSTEM SUCCESS/FAILURE METRICS:
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- Step completed in full with required outputs
|
||||
|
||||
### ❌ SYSTEM FAILURE:
|
||||
|
||||
- Skipped sequence steps or missing outputs
|
||||
**Master Rule:** Skipping steps is FORBIDDEN.
|
||||
110
_bmad/tea/workflows/testarch/ci/steps-c/step-01b-resume.md
Normal file
110
_bmad/tea/workflows/testarch/ci/steps-c/step-01b-resume.md
Normal file
@@ -0,0 +1,110 @@
|
||||
---
|
||||
name: 'step-01b-resume'
|
||||
description: 'Resume interrupted workflow from last completed step'
|
||||
outputFile: '{test_artifacts}/ci-pipeline-progress.md'
|
||||
---
|
||||
|
||||
# Step 1b: Resume Workflow
|
||||
|
||||
## STEP GOAL
|
||||
|
||||
Resume an interrupted workflow by loading the existing progress document, displaying progress, verifying previously created artifacts, and routing to the next incomplete step.
|
||||
|
||||
## MANDATORY EXECUTION RULES
|
||||
|
||||
- 📖 Read the entire step file before acting
|
||||
- ✅ Speak in `{communication_language}`
|
||||
|
||||
---
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
- 📖 Load the next step only when instructed
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: Output document with progress frontmatter
|
||||
- Focus: Load progress and route to next step
|
||||
- Limits: Do not re-execute completed steps
|
||||
- Dependencies: Output document must exist from a previous run
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly. Do not skip, reorder, or improvise.
|
||||
|
||||
### 1. Load Output Document
|
||||
|
||||
Read `{outputFile}` and parse YAML frontmatter for:
|
||||
|
||||
- `stepsCompleted` — array of completed step names
|
||||
- `lastStep` — last completed step name
|
||||
- `lastSaved` — timestamp of last save
|
||||
|
||||
**If `{outputFile}` does not exist**, display:
|
||||
|
||||
"⚠️ **No previous progress found.** There is no output document to resume from. Please use **[C] Create** to start a fresh workflow run."
|
||||
|
||||
**THEN:** Halt. Do not proceed.
|
||||
|
||||
---
|
||||
|
||||
### 2. Verify Previously Created Artifacts
|
||||
|
||||
Since this is a file-creation workflow, verify that artifacts from completed steps still exist on disk:
|
||||
|
||||
- If `step-02-generate-pipeline` is in `stepsCompleted`, check that the pipeline config file exists (e.g., `.github/workflows/test.yml` or equivalent)
|
||||
- If any expected artifact is missing, warn the user and suggest re-running from the step that creates it
|
||||
|
||||
---
|
||||
|
||||
### 3. Display Progress Dashboard
|
||||
|
||||
Display:
|
||||
|
||||
"📋 **Workflow Resume — CI/CD Pipeline Setup**
|
||||
|
||||
**Last saved:** {lastSaved}
|
||||
**Steps completed:** {stepsCompleted.length} of 4
|
||||
|
||||
1. Preflight Checks (step-01-preflight) — {✅ if in stepsCompleted, ⬜ otherwise}
|
||||
2. Generate Pipeline (step-02-generate-pipeline) — {✅ if in stepsCompleted, ⬜ otherwise}
|
||||
3. Configure Quality Gates (step-03-configure-quality-gates) — {✅ if in stepsCompleted, ⬜ otherwise}
|
||||
4. Validate & Summary (step-04-validate-and-summary) — {✅ if in stepsCompleted, ⬜ otherwise}"
|
||||
|
||||
---
|
||||
|
||||
### 4. Route to Next Step
|
||||
|
||||
Based on `lastStep`, load the next incomplete step:
|
||||
|
||||
- `'step-01-preflight'` → Load `./step-02-generate-pipeline.md`
|
||||
- `'step-02-generate-pipeline'` → Load `./step-03-configure-quality-gates.md`
|
||||
- `'step-03-configure-quality-gates'` → Load `./step-04-validate-and-summary.md`
|
||||
- `'step-04-validate-and-summary'` → **Workflow already complete.** Display: "✅ **All steps completed.** Use **[V] Validate** to review outputs or **[E] Edit** to make revisions." Then halt.
|
||||
|
||||
**If `lastStep` does not match any value above**, display: "⚠️ **Unknown progress state** (`lastStep`: {lastStep}). Please use **[C] Create** to start fresh." Then halt.
|
||||
|
||||
**Otherwise**, load the identified step file, read completely, and execute.
|
||||
|
||||
The existing content in `{outputFile}` provides context from previously completed steps. Use it as reference for remaining steps.
|
||||
|
||||
---
|
||||
|
||||
## 🚨 SYSTEM SUCCESS/FAILURE METRICS
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- Output document loaded and parsed correctly
|
||||
- Previously created artifacts verified
|
||||
- Progress dashboard displayed accurately
|
||||
- Routed to correct next step
|
||||
|
||||
### ❌ SYSTEM FAILURE:
|
||||
|
||||
- Not loading output document
|
||||
- Incorrect progress display
|
||||
- Routing to wrong step
|
||||
- Re-executing completed steps
|
||||
|
||||
**Master Rule:** Resume MUST route to the exact next incomplete step. Never re-execute completed steps.
|
||||
@@ -0,0 +1,279 @@
|
||||
---
|
||||
name: 'step-02-generate-pipeline'
|
||||
description: 'Generate CI pipeline configuration with adaptive orchestration (agent-team, subagent, or sequential)'
|
||||
nextStepFile: './step-03-configure-quality-gates.md'
|
||||
outputFile: '{test_artifacts}/ci-pipeline-progress.md'
|
||||
---
|
||||
|
||||
# Step 2: Generate CI Pipeline
|
||||
|
||||
## STEP GOAL
|
||||
|
||||
Create platform-specific CI configuration with test execution, sharding, burn-in, and artifacts.
|
||||
|
||||
## MANDATORY EXECUTION RULES
|
||||
|
||||
- 📖 Read the entire step file before acting
|
||||
- ✅ Speak in `{communication_language}`
|
||||
- ✅ Resolve execution mode from explicit user request first, then config
|
||||
- ✅ Apply fallback rules deterministically when requested mode is unsupported
|
||||
|
||||
---
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
- 💾 Record outputs before proceeding
|
||||
- 📖 Load the next step only when instructed
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: config, loaded artifacts, and knowledge fragments
|
||||
- Focus: this step's goal only
|
||||
- Limits: do not execute future steps
|
||||
- Dependencies: prior steps' outputs (if any)
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly. Do not skip, reorder, or improvise.
|
||||
|
||||
## 0. Resolve Execution Mode (User Override First)
|
||||
|
||||
```javascript
|
||||
const orchestrationContext = {
|
||||
config: {
|
||||
execution_mode: config.tea_execution_mode || 'auto', // "auto" | "subagent" | "agent-team" | "sequential"
|
||||
capability_probe: config.tea_capability_probe !== false, // true by default
|
||||
},
|
||||
timestamp: new Date().toISOString().replace(/[:.]/g, '-'),
|
||||
};
|
||||
|
||||
const normalizeUserExecutionMode = (mode) => {
|
||||
if (typeof mode !== 'string') return null;
|
||||
const normalized = mode.trim().toLowerCase().replace(/[-_]/g, ' ').replace(/\s+/g, ' ');
|
||||
|
||||
if (normalized === 'auto') return 'auto';
|
||||
if (normalized === 'sequential') return 'sequential';
|
||||
if (normalized === 'subagent' || normalized === 'sub agent' || normalized === 'subagents' || normalized === 'sub agents') {
|
||||
return 'subagent';
|
||||
}
|
||||
if (normalized === 'agent team' || normalized === 'agent teams' || normalized === 'agentteam') {
|
||||
return 'agent-team';
|
||||
}
|
||||
|
||||
return null;
|
||||
};
|
||||
|
||||
const normalizeConfigExecutionMode = (mode) => {
|
||||
if (mode === 'subagent') return 'subagent';
|
||||
if (mode === 'auto' || mode === 'sequential' || mode === 'subagent' || mode === 'agent-team') {
|
||||
return mode;
|
||||
}
|
||||
return null;
|
||||
};
|
||||
|
||||
// Explicit user instruction in the active run takes priority over config.
|
||||
const explicitModeFromUser = normalizeUserExecutionMode(runtime.getExplicitExecutionModeHint?.() || null);
|
||||
|
||||
const requestedMode = explicitModeFromUser || normalizeConfigExecutionMode(orchestrationContext.config.execution_mode) || 'auto';
|
||||
const probeEnabled = orchestrationContext.config.capability_probe;
|
||||
|
||||
const supports = { subagent: false, agentTeam: false };
|
||||
if (probeEnabled) {
|
||||
supports.subagent = runtime.canLaunchSubagents?.() === true;
|
||||
supports.agentTeam = runtime.canLaunchAgentTeams?.() === true;
|
||||
}
|
||||
|
||||
let resolvedMode = requestedMode;
|
||||
if (requestedMode === 'auto') {
|
||||
if (supports.agentTeam) resolvedMode = 'agent-team';
|
||||
else if (supports.subagent) resolvedMode = 'subagent';
|
||||
else resolvedMode = 'sequential';
|
||||
} else if (probeEnabled && requestedMode === 'agent-team' && !supports.agentTeam) {
|
||||
resolvedMode = supports.subagent ? 'subagent' : 'sequential';
|
||||
} else if (probeEnabled && requestedMode === 'subagent' && !supports.subagent) {
|
||||
resolvedMode = 'sequential';
|
||||
}
|
||||
```
|
||||
|
||||
Resolution precedence:
|
||||
|
||||
1. Explicit user request in this run (`agent team` => `agent-team`; `subagent` => `subagent`; `sequential`; `auto`)
|
||||
2. `tea_execution_mode` from config
|
||||
3. Runtime capability fallback (when probing enabled)
|
||||
|
||||
## 1. Resolve Output Path and Select Template
|
||||
|
||||
Determine the pipeline output file path based on the detected `ci_platform`:
|
||||
|
||||
| CI Platform | Output Path | Template File |
|
||||
| ---------------- | ------------------------------------------- | --------------------------------------------------- |
|
||||
| `github-actions` | `{project-root}/.github/workflows/test.yml` | `{installed_path}/github-actions-template.yaml` |
|
||||
| `gitlab-ci` | `{project-root}/.gitlab-ci.yml` | `{installed_path}/gitlab-ci-template.yaml` |
|
||||
| `jenkins` | `{project-root}/Jenkinsfile` | `{installed_path}/jenkins-pipeline-template.groovy` |
|
||||
| `azure-devops` | `{project-root}/azure-pipelines.yml` | `{installed_path}/azure-pipelines-template.yaml` |
|
||||
| `harness` | `{project-root}/.harness/pipeline.yaml` | `{installed_path}/harness-pipeline-template.yaml` |
|
||||
| `circle-ci` | `{project-root}/.circleci/config.yml` | _(no template; generate from first principles)_ |
|
||||
|
||||
Use templates from `{installed_path}` when available. Adapt the template to the project's `test_stack_type` and `test_framework`.
|
||||
|
||||
---
|
||||
|
||||
## Security: Script Injection Prevention
|
||||
|
||||
> **CRITICAL:** Treat `${{ inputs.* }}` and the entire `${{ github.event.* }}` namespace as unsafe by default. ALWAYS route them through `env:` intermediaries and reference as double-quoted `"$ENV_VAR"` in `run:` blocks. NEVER interpolate them directly.
|
||||
|
||||
When the generated pipeline is extended into reusable workflows (`on: workflow_call`), manual dispatch (`on: workflow_dispatch`), or composite actions, these values become user-controllable and can inject arbitrary shell commands.
|
||||
|
||||
**Two rules for generated `run:` blocks:**
|
||||
|
||||
1. **No direct interpolation** — pass unsafe contexts through `env:`, reference as `"$ENV_VAR"`
|
||||
2. **Inputs must be DATA, not COMMANDS** — never accept command-shaped inputs (e.g., `inputs.install-command`) that get executed as shell code. Even through `env:`, running `$CMD` where CMD comes from an input is still command injection. Use fixed commands and pass inputs only as arguments.
|
||||
|
||||
```yaml
|
||||
# ✅ SAFE — input is DATA interpolated into a fixed command
|
||||
- name: Run tests
|
||||
env:
|
||||
TEST_GREP: ${{ inputs.test-grep }}
|
||||
run: |
|
||||
# Security: inputs passed through env: to prevent script injection
|
||||
npx playwright test --grep "$TEST_GREP"
|
||||
|
||||
# ❌ NEVER — direct GitHub expression injection
|
||||
- name: Run tests
|
||||
run: |
|
||||
npx playwright test --grep "${{ inputs.test-grep }}"
|
||||
|
||||
# ❌ NEVER — executing input-derived env var as a command
|
||||
- name: Install
|
||||
env:
|
||||
CMD: ${{ inputs.install-command }}
|
||||
run: $CMD
|
||||
```
|
||||
|
||||
Include a `# Security: inputs passed through env: to prevent script injection` comment in generated YAML wherever this pattern is applied.
|
||||
|
||||
**Safe contexts** (do NOT need `env:` intermediaries): `${{ steps.*.outputs.* }}`, `${{ matrix.* }}`, `${{ runner.os }}`, `${{ github.sha }}`, `${{ github.ref }}`, `${{ secrets.* }}`, `${{ env.* }}`.
|
||||
|
||||
---
|
||||
|
||||
## 2. Pipeline Stages
|
||||
|
||||
Include stages:
|
||||
|
||||
- lint
|
||||
- test (parallel shards)
|
||||
- contract-test (if `tea_use_pactjs_utils` enabled)
|
||||
- burn-in (flaky detection)
|
||||
- report (aggregate + publish)
|
||||
|
||||
---
|
||||
|
||||
## 3. Test Execution
|
||||
|
||||
- Parallel sharding enabled
|
||||
- CI retries configured
|
||||
- Capture artifacts (HTML report, JUnit XML, traces/videos on failure)
|
||||
- Cache dependencies (language-appropriate: node_modules, .venv, .m2, go module cache, NuGet, bundler)
|
||||
|
||||
Write the selected pipeline configuration to the resolved output path from step 1. Adjust test commands based on `test_stack_type` and `test_framework`:
|
||||
|
||||
- **Frontend/Fullstack**: Include browser install, E2E/component test commands, Playwright/Cypress artifacts
|
||||
- **Backend (Node.js)**: Use `npm test` or framework-specific commands (`vitest`, `jest`), skip browser install
|
||||
- **Backend (Python)**: Use `pytest` with coverage (`pytest --cov`), install via `pip install -r requirements.txt` or `poetry install`
|
||||
- **Backend (Java/Kotlin)**: Use `mvn test` or `gradle test`, cache `.m2/repository` or `.gradle/caches`
|
||||
- **Backend (Go)**: Use `go test ./...` with coverage (`-coverprofile`), cache Go modules
|
||||
- **Backend (C#/.NET)**: Use `dotnet test` with coverage, restore NuGet packages
|
||||
- **Backend (Ruby)**: Use `bundle exec rspec` with coverage, cache `vendor/bundle`
|
||||
|
||||
### Contract Testing Pipeline (if `tea_use_pactjs_utils` enabled)
|
||||
|
||||
When `tea_use_pactjs_utils` is enabled, add a `contract-test` stage after `test`:
|
||||
|
||||
**Required env block** (add to the generated pipeline):
|
||||
|
||||
```yaml
|
||||
env:
|
||||
PACT_BROKER_BASE_URL: ${{ secrets.PACT_BROKER_BASE_URL }}
|
||||
PACT_BROKER_TOKEN: ${{ secrets.PACT_BROKER_TOKEN }}
|
||||
GITHUB_SHA: ${{ github.sha }} # auto-set by GitHub Actions
|
||||
GITHUB_BRANCH: ${{ github.head_ref || github.ref_name }} # NOT auto-set — must be defined explicitly
|
||||
```
|
||||
|
||||
> **Note:** `GITHUB_SHA` is auto-set by GitHub Actions, but `GITHUB_BRANCH` is **not** — it must be derived from `github.head_ref` (for PRs) or `github.ref_name` (for pushes). The pactjs-utils library reads both from `process.env`.
|
||||
|
||||
1. **Consumer test + publish**: Run consumer contract tests, then publish pacts to broker
|
||||
- `npm run test:pact:consumer`
|
||||
- `npm run publish:pact`
|
||||
- Only publish on PR and main branch pushes
|
||||
|
||||
2. **Provider verification**: Run provider verification against published pacts
|
||||
- `npm run test:pact:provider:remote:contract`
|
||||
- `buildVerifierOptions` auto-reads `PACT_BROKER_BASE_URL`, `PACT_BROKER_TOKEN`, `GITHUB_SHA`, `GITHUB_BRANCH`
|
||||
- Verification results published to broker when `CI=true`
|
||||
|
||||
3. **Can-I-Deploy gate**: Block deployment if contracts are incompatible
|
||||
- `npm run can:i:deploy:provider`
|
||||
- Ensure the script adds `--retry-while-unknown 6 --retry-interval 10` for async verification
|
||||
|
||||
4. **Webhook job**: Add `repository_dispatch` trigger for `pact_changed` event
|
||||
- Provider verification runs when consumers publish new pacts
|
||||
- Ensures compatibility is checked on both consumer and provider changes
|
||||
|
||||
5. **Breaking change handling**: When `PACT_BREAKING_CHANGE=true` env var is set:
|
||||
- Provider test passes `includeMainAndDeployed: false` to `buildVerifierOptions` — verifies only matching branch
|
||||
- Coordinate with consumer team before removing the flag
|
||||
|
||||
6. **Record deployment**: After successful deployment, record version in broker
|
||||
- `npm run record:provider:deployment --env=production`
|
||||
|
||||
Required CI secrets: `PACT_BROKER_BASE_URL`, `PACT_BROKER_TOKEN`
|
||||
|
||||
**If `tea_pact_mcp` is `"mcp"`:** Reference the SmartBear MCP `Can I Deploy` and `Matrix` tools for pipeline guidance in `pact-mcp.md`.
|
||||
|
||||
---
|
||||
|
||||
### 4. Save Progress
|
||||
|
||||
**Save this step's accumulated work to `{outputFile}`.**
|
||||
|
||||
- **If `{outputFile}` does not exist** (first save), create it with YAML frontmatter:
|
||||
|
||||
```yaml
|
||||
---
|
||||
stepsCompleted: ['step-02-generate-pipeline']
|
||||
lastStep: 'step-02-generate-pipeline'
|
||||
lastSaved: '{date}'
|
||||
---
|
||||
```
|
||||
|
||||
Then write this step's output below the frontmatter.
|
||||
|
||||
- **If `{outputFile}` already exists**, update:
|
||||
- Add `'step-02-generate-pipeline'` to `stepsCompleted` array (only if not already present)
|
||||
- Set `lastStep: 'step-02-generate-pipeline'`
|
||||
- Set `lastSaved: '{date}'`
|
||||
- Append this step's output to the appropriate section of the document.
|
||||
|
||||
### 5. Orchestration Notes for This Step
|
||||
|
||||
For this step, treat these work units as parallelizable when `resolvedMode` is `agent-team` or `subagent`:
|
||||
|
||||
- Worker A: resolve platform path/template and produce base pipeline skeleton (section 1)
|
||||
- Worker B: construct stage definitions and test execution blocks (sections 2-3)
|
||||
- Worker C: contract-testing block (only when `tea_use_pactjs_utils` is true)
|
||||
|
||||
If `resolvedMode` is `sequential`, execute sections 1→4 in order.
|
||||
|
||||
Load next step: `{nextStepFile}`
|
||||
|
||||
## 🚨 SYSTEM SUCCESS/FAILURE METRICS:
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- Step completed in full with required outputs
|
||||
|
||||
### ❌ SYSTEM FAILURE:
|
||||
|
||||
- Skipped sequence steps or missing outputs
|
||||
**Master Rule:** Skipping steps is FORBIDDEN.
|
||||
@@ -0,0 +1,135 @@
|
||||
---
|
||||
name: 'step-03-configure-quality-gates'
|
||||
description: 'Configure burn-in, quality gates, and notifications'
|
||||
nextStepFile: './step-04-validate-and-summary.md'
|
||||
knowledgeIndex: '{project-root}/_bmad/tea/testarch/tea-index.csv'
|
||||
outputFile: '{test_artifacts}/ci-pipeline-progress.md'
|
||||
---
|
||||
|
||||
# Step 3: Quality Gates & Notifications
|
||||
|
||||
## STEP GOAL
|
||||
|
||||
Configure burn-in loops, quality thresholds, and notification hooks.
|
||||
|
||||
## MANDATORY EXECUTION RULES
|
||||
|
||||
- 📖 Read the entire step file before acting
|
||||
- ✅ Speak in `{communication_language}`
|
||||
|
||||
---
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
- 💾 Record outputs before proceeding
|
||||
- 📖 Load the next step only when instructed
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: config, loaded artifacts, and knowledge fragments
|
||||
- Focus: this step's goal only
|
||||
- Limits: do not execute future steps
|
||||
- Dependencies: prior steps' outputs (if any)
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly. Do not skip, reorder, or improvise.
|
||||
|
||||
## 1. Burn-In Configuration
|
||||
|
||||
Use `{knowledgeIndex}` to load `ci-burn-in.md` guidance:
|
||||
|
||||
- Run N-iteration burn-in for flaky detection
|
||||
- Gate promotion based on burn-in stability
|
||||
|
||||
**Stack-conditional burn-in:**
|
||||
|
||||
- **Frontend or Fullstack** (`test_stack_type` is `frontend` or `fullstack`): Enable burn-in by default. Burn-in targets UI flakiness (race conditions, selector instability, timing issues).
|
||||
- **Backend only** (`test_stack_type` is `backend`): Skip burn-in by default. Backend tests (unit, integration, API) are deterministic and rarely exhibit UI-related flakiness. If the user explicitly requests burn-in for backend, honor that override.
|
||||
|
||||
**Security: Script injection prevention for reusable burn-in workflows:**
|
||||
|
||||
When burn-in is extracted into a reusable workflow (`on: workflow_call`), all `${{ inputs.* }}` values MUST be passed through `env:` intermediaries and referenced as quoted `"$ENV_VAR"`. Never interpolate them directly.
|
||||
|
||||
**Inputs must be DATA, not COMMANDS.** Do not accept command-shaped inputs (e.g., `inputs.install-command`, `inputs.test-command`) that get executed as shell code — even through `env:`, running `$CMD` is still command injection. Use fixed commands (e.g., `npm ci`, `npx playwright test`) and pass inputs only as data arguments.
|
||||
|
||||
```yaml
|
||||
# ✅ SAFE — fixed commands with data-only inputs
|
||||
- name: Install dependencies
|
||||
run: npm ci
|
||||
- name: Run burn-in loop
|
||||
env:
|
||||
TEST_GREP: ${{ inputs.test-grep }}
|
||||
BURN_IN_COUNT: ${{ inputs.burn-in-count }}
|
||||
BASE_REF: ${{ inputs.base-ref }}
|
||||
run: |
|
||||
# Security: inputs passed through env: to prevent script injection
|
||||
for i in $(seq 1 "$BURN_IN_COUNT"); do
|
||||
echo "Burn-in iteration $i/$BURN_IN_COUNT"
|
||||
npx playwright test --grep "$TEST_GREP" || exit 1
|
||||
done
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Quality Gates
|
||||
|
||||
Define:
|
||||
|
||||
- Minimum pass rates (P0 = 100%, P1 ≥ 95%)
|
||||
- Fail CI on critical test failures
|
||||
- Optional: require traceability or nfr-assess output before release
|
||||
|
||||
**Contract testing gate** (if `tea_use_pactjs_utils` is enabled):
|
||||
|
||||
- **can-i-deploy must pass** before any deployment to staging or production
|
||||
- Block the deployment pipeline if contract verification fails
|
||||
- Treat consumer pact publishing failures as CI failures (contracts must stay up-to-date)
|
||||
- Provider verification must pass for all consumer pacts before merge
|
||||
|
||||
---
|
||||
|
||||
## 3. Notifications
|
||||
|
||||
Configure:
|
||||
|
||||
- Failure notifications (Slack/email)
|
||||
- Artifact links
|
||||
|
||||
---
|
||||
|
||||
### 4. Save Progress
|
||||
|
||||
**Save this step's accumulated work to `{outputFile}`.**
|
||||
|
||||
- **If `{outputFile}` does not exist** (first save), create it with YAML frontmatter:
|
||||
|
||||
```yaml
|
||||
---
|
||||
stepsCompleted: ['step-03-configure-quality-gates']
|
||||
lastStep: 'step-03-configure-quality-gates'
|
||||
lastSaved: '{date}'
|
||||
---
|
||||
```
|
||||
|
||||
Then write this step's output below the frontmatter.
|
||||
|
||||
- **If `{outputFile}` already exists**, update:
|
||||
- Add `'step-03-configure-quality-gates'` to `stepsCompleted` array (only if not already present)
|
||||
- Set `lastStep: 'step-03-configure-quality-gates'`
|
||||
- Set `lastSaved: '{date}'`
|
||||
- Append this step's output to the appropriate section of the document.
|
||||
|
||||
Load next step: `{nextStepFile}`
|
||||
|
||||
## 🚨 SYSTEM SUCCESS/FAILURE METRICS:
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- Step completed in full with required outputs
|
||||
|
||||
### ❌ SYSTEM FAILURE:
|
||||
|
||||
- Skipped sequence steps or missing outputs
|
||||
**Master Rule:** Skipping steps is FORBIDDEN.
|
||||
@@ -0,0 +1,92 @@
|
||||
---
|
||||
name: 'step-04-validate-and-summary'
|
||||
description: 'Validate pipeline and summarize'
|
||||
outputFile: '{test_artifacts}/ci-pipeline-progress.md'
|
||||
---
|
||||
|
||||
# Step 4: Validate & Summarize
|
||||
|
||||
## STEP GOAL
|
||||
|
||||
Validate CI configuration and report completion details.
|
||||
|
||||
## MANDATORY EXECUTION RULES
|
||||
|
||||
- 📖 Read the entire step file before acting
|
||||
- ✅ Speak in `{communication_language}`
|
||||
|
||||
---
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
- 💾 Record outputs before proceeding
|
||||
- 📖 Load the next step only when instructed
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: config, loaded artifacts, and knowledge fragments
|
||||
- Focus: this step's goal only
|
||||
- Limits: do not execute future steps
|
||||
- Dependencies: prior steps' outputs (if any)
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly. Do not skip, reorder, or improvise.
|
||||
|
||||
## 1. Validation
|
||||
|
||||
Validate against `checklist.md`:
|
||||
|
||||
- Config file created
|
||||
- Stages and sharding configured
|
||||
- Burn-in and artifacts enabled
|
||||
- Secrets/variables documented
|
||||
|
||||
Fix gaps before completion.
|
||||
|
||||
---
|
||||
|
||||
## 2. Completion Summary
|
||||
|
||||
Report:
|
||||
|
||||
- CI platform and config path
|
||||
- Key stages enabled
|
||||
- Artifacts and notifications
|
||||
- Next steps (set secrets, run pipeline)
|
||||
|
||||
---
|
||||
|
||||
### 3. Save Progress
|
||||
|
||||
**Save this step's accumulated work to `{outputFile}`.**
|
||||
|
||||
- **If `{outputFile}` does not exist** (first save), create it with YAML frontmatter:
|
||||
|
||||
```yaml
|
||||
---
|
||||
stepsCompleted: ['step-04-validate-and-summary']
|
||||
lastStep: 'step-04-validate-and-summary'
|
||||
lastSaved: '{date}'
|
||||
---
|
||||
```
|
||||
|
||||
Then write this step's output below the frontmatter.
|
||||
|
||||
- **If `{outputFile}` already exists**, update:
|
||||
- Add `'step-04-validate-and-summary'` to `stepsCompleted` array (only if not already present)
|
||||
- Set `lastStep: 'step-04-validate-and-summary'`
|
||||
- Set `lastSaved: '{date}'`
|
||||
- Append this step's output to the appropriate section of the document.
|
||||
|
||||
## 🚨 SYSTEM SUCCESS/FAILURE METRICS:
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- Step completed in full with required outputs
|
||||
|
||||
### ❌ SYSTEM FAILURE:
|
||||
|
||||
- Skipped sequence steps or missing outputs
|
||||
**Master Rule:** Skipping steps is FORBIDDEN.
|
||||
65
_bmad/tea/workflows/testarch/ci/steps-e/step-01-assess.md
Normal file
65
_bmad/tea/workflows/testarch/ci/steps-e/step-01-assess.md
Normal file
@@ -0,0 +1,65 @@
|
||||
---
|
||||
name: 'step-01-assess'
|
||||
description: 'Load an existing output for editing'
|
||||
nextStepFile: './step-02-apply-edit.md'
|
||||
---
|
||||
|
||||
# Step 1: Assess Edit Target
|
||||
|
||||
## STEP GOAL:
|
||||
|
||||
Identify which output should be edited and load it.
|
||||
|
||||
## MANDATORY EXECUTION RULES (READ FIRST):
|
||||
|
||||
### Universal Rules:
|
||||
|
||||
- 📖 Read the complete step file before taking any action
|
||||
- ✅ Speak in `{communication_language}`
|
||||
|
||||
### Role Reinforcement:
|
||||
|
||||
- ✅ You are the Master Test Architect
|
||||
|
||||
### Step-Specific Rules:
|
||||
|
||||
- 🎯 Ask the user which output file to edit
|
||||
- 🚫 Do not edit until target is confirmed
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: existing outputs
|
||||
- Focus: select edit target
|
||||
- Limits: no edits yet
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly.
|
||||
|
||||
### 1. Identify Target
|
||||
|
||||
Ask the user to provide the output file path or select from known outputs.
|
||||
|
||||
### 2. Load Target
|
||||
|
||||
Read the provided output file in full.
|
||||
|
||||
### 3. Confirm
|
||||
|
||||
Confirm the target and proceed to edit.
|
||||
|
||||
Load next step: `{nextStepFile}`
|
||||
|
||||
## 🚨 SYSTEM SUCCESS/FAILURE METRICS:
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- Target identified and loaded
|
||||
|
||||
### ❌ SYSTEM FAILURE:
|
||||
|
||||
- Proceeding without a confirmed target
|
||||
@@ -0,0 +1,60 @@
|
||||
---
|
||||
name: 'step-02-apply-edit'
|
||||
description: 'Apply edits to the selected output'
|
||||
---
|
||||
|
||||
# Step 2: Apply Edits
|
||||
|
||||
## STEP GOAL:
|
||||
|
||||
Apply the requested edits to the selected output and confirm changes.
|
||||
|
||||
## MANDATORY EXECUTION RULES (READ FIRST):
|
||||
|
||||
### Universal Rules:
|
||||
|
||||
- 📖 Read the complete step file before taking any action
|
||||
- ✅ Speak in `{communication_language}`
|
||||
|
||||
### Role Reinforcement:
|
||||
|
||||
- ✅ You are the Master Test Architect
|
||||
|
||||
### Step-Specific Rules:
|
||||
|
||||
- 🎯 Only apply edits explicitly requested by the user
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: selected output and user changes
|
||||
- Focus: apply edits only
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly.
|
||||
|
||||
### 1. Confirm Requested Changes
|
||||
|
||||
Restate what will be changed and confirm.
|
||||
|
||||
### 2. Apply Changes
|
||||
|
||||
Update the output file accordingly.
|
||||
|
||||
### 3. Report
|
||||
|
||||
Summarize the edits applied.
|
||||
|
||||
## 🚨 SYSTEM SUCCESS/FAILURE METRICS:
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- Changes applied and confirmed
|
||||
|
||||
### ❌ SYSTEM FAILURE:
|
||||
|
||||
- Unconfirmed edits or missing update
|
||||
81
_bmad/tea/workflows/testarch/ci/steps-v/step-01-validate.md
Normal file
81
_bmad/tea/workflows/testarch/ci/steps-v/step-01-validate.md
Normal file
@@ -0,0 +1,81 @@
|
||||
---
|
||||
name: 'step-01-validate'
|
||||
description: 'Validate workflow outputs against checklist'
|
||||
outputFile: '{test_artifacts}/ci-validation-report.md'
|
||||
validationChecklist: '../checklist.md'
|
||||
---
|
||||
|
||||
# Step 1: Validate Outputs
|
||||
|
||||
## STEP GOAL:
|
||||
|
||||
Validate outputs using the workflow checklist and record findings.
|
||||
|
||||
## MANDATORY EXECUTION RULES (READ FIRST):
|
||||
|
||||
### Universal Rules:
|
||||
|
||||
- 📖 Read the complete step file before taking any action
|
||||
- ✅ Speak in `{communication_language}`
|
||||
|
||||
### Role Reinforcement:
|
||||
|
||||
- ✅ You are the Master Test Architect
|
||||
|
||||
### Step-Specific Rules:
|
||||
|
||||
- 🎯 Validate against `{validationChecklist}`
|
||||
- 🚫 Do not skip checks
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
- 💾 Write findings to `{outputFile}`
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: workflow outputs and checklist
|
||||
- Focus: validation only
|
||||
- Limits: do not modify outputs in this step
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly.
|
||||
|
||||
### 1. Load Checklist
|
||||
|
||||
Read `{validationChecklist}` and list all criteria.
|
||||
|
||||
### 2. Validate Outputs
|
||||
|
||||
Evaluate outputs against each checklist item.
|
||||
|
||||
### 2a. Script Injection Scan
|
||||
|
||||
Scan all generated YAML workflow files for unsafe interpolation patterns inside `run:` blocks.
|
||||
|
||||
**Unsafe patterns to flag (FAIL):**
|
||||
|
||||
- `${{ inputs.* }}` — all workflow inputs are user-controllable
|
||||
- `${{ github.event.* }}` — treat the entire event namespace as unsafe by default (includes PR titles, issue bodies, comment bodies, label names, etc.)
|
||||
- `${{ github.head_ref }}` — PR source branch name (user-controlled)
|
||||
|
||||
**Detection method:** For each `run:` block in generated YAML, check if any of the above expressions appears in the run script body. If found, flag as **FAIL** with the exact line and recommend converting to the safe `env:` intermediary pattern (pass through `env:`, reference as double-quoted `"$ENV_VAR"`).
|
||||
|
||||
**Safe patterns to ignore** (exempt from flagging): `${{ steps.*.outputs.* }}`, `${{ matrix.* }}`, `${{ runner.os }}`, `${{ github.sha }}`, `${{ github.ref }}`, `${{ secrets.* }}`, `${{ env.* }}` — these are safe from GitHub expression injection when used in `run:` blocks.
|
||||
|
||||
### 3. Write Report
|
||||
|
||||
Write a validation report to `{outputFile}` with PASS/WARN/FAIL per section.
|
||||
|
||||
## 🚨 SYSTEM SUCCESS/FAILURE METRICS:
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- Validation report written
|
||||
- All checklist items evaluated
|
||||
|
||||
### ❌ SYSTEM FAILURE:
|
||||
|
||||
- Skipped checklist items
|
||||
- No report produced
|
||||
@@ -0,0 +1,72 @@
|
||||
---
|
||||
validationDate: 2026-01-27
|
||||
workflowName: testarch-ci
|
||||
workflowPath: {project-root}/src/workflows/testarch/ci
|
||||
validationStatus: COMPLETE
|
||||
completionDate: 2026-01-27 10:03:10
|
||||
---
|
||||
|
||||
# Validation Report: testarch-ci
|
||||
|
||||
**Validation Started:** 2026-01-27 09:50:21
|
||||
**Validator:** BMAD Workflow Validation System (Codex)
|
||||
**Standards Version:** BMAD Workflow Standards
|
||||
|
||||
## File Structure & Size
|
||||
|
||||
- workflow.md present: YES
|
||||
- instructions.md present: YES
|
||||
- workflow.yaml present: YES
|
||||
- step files found: 7
|
||||
|
||||
**Step File Sizes:**
|
||||
|
||||
- steps-c/step-01-preflight.md: 87 lines [GOOD]
|
||||
- steps-c/step-02-generate-pipeline.md: 75 lines [GOOD]
|
||||
- steps-c/step-03-configure-quality-gates.md: 67 lines [GOOD]
|
||||
- steps-c/step-04-validate-and-summary.md: 60 lines [GOOD]
|
||||
- steps-e/step-01-assess.md: 51 lines [GOOD]
|
||||
- steps-e/step-02-apply-edit.md: 46 lines [GOOD]
|
||||
- steps-v/step-01-validate.md: 53 lines [GOOD]
|
||||
- workflow-plan.md present: YES
|
||||
|
||||
## Frontmatter Validation
|
||||
|
||||
- No frontmatter violations found
|
||||
|
||||
## Critical Path Violations
|
||||
|
||||
- No {project-root} hardcoded paths detected in body
|
||||
- No dead relative links detected
|
||||
|
||||
## Menu Handling Validation
|
||||
|
||||
- No menu structures detected (linear step flow) [N/A]
|
||||
|
||||
## Step Type Validation
|
||||
|
||||
- Last step steps-v/step-01-validate.md has no nextStepFile (final step OK)
|
||||
- Step type validation assumes linear sequence (no branching/menu). Workflow-plan.md present for reference. [INFO]
|
||||
|
||||
## Output Format Validation
|
||||
|
||||
- No templates found in workflow root
|
||||
- Steps with outputFile in frontmatter:
|
||||
- steps-c/step-02-generate-pipeline.md
|
||||
- steps-v/step-01-validate.md
|
||||
|
||||
## Validation Design Check
|
||||
|
||||
- checklist.md present: YES
|
||||
- Validation steps folder (steps-v) present: YES
|
||||
|
||||
## Instruction Style Check
|
||||
|
||||
- All steps include STEP GOAL, MANDATORY EXECUTION RULES, EXECUTION PROTOCOLS, CONTEXT BOUNDARIES, and SUCCESS/FAILURE metrics
|
||||
|
||||
## Summary
|
||||
|
||||
- Validation completed: 2026-01-27 10:03:10
|
||||
- Critical issues: 0
|
||||
- Warnings: 0 (informational notes only)
|
||||
- Readiness: READY (manual review optional)
|
||||
@@ -0,0 +1,114 @@
|
||||
---
|
||||
validationDate: 2026-01-27
|
||||
workflowName: testarch-ci
|
||||
workflowPath: {project-root}/src/workflows/testarch/ci
|
||||
validationStatus: COMPLETE
|
||||
completionDate: 2026-01-27 10:24:01
|
||||
---
|
||||
|
||||
# Validation Report: testarch-ci
|
||||
|
||||
**Validation Started:** 2026-01-27 10:24:01
|
||||
**Validator:** BMAD Workflow Validation System (Codex)
|
||||
**Standards Version:** BMAD Workflow Standards
|
||||
|
||||
## File Structure & Size
|
||||
|
||||
- workflow.md present: YES
|
||||
- instructions.md present: YES
|
||||
- workflow.yaml present: YES
|
||||
- step files found: 7
|
||||
|
||||
**Step File Sizes:**
|
||||
|
||||
- steps-c/step-01-preflight.md: 86 lines [GOOD]
|
||||
- steps-c/step-02-generate-pipeline.md: 74 lines [GOOD]
|
||||
- steps-c/step-03-configure-quality-gates.md: 66 lines [GOOD]
|
||||
- steps-c/step-04-validate-and-summary.md: 59 lines [GOOD]
|
||||
- steps-e/step-01-assess.md: 50 lines [GOOD]
|
||||
- steps-e/step-02-apply-edit.md: 45 lines [GOOD]
|
||||
- steps-v/step-01-validate.md: 52 lines [GOOD]
|
||||
- workflow-plan.md present: YES
|
||||
|
||||
## Frontmatter Validation
|
||||
|
||||
- No frontmatter violations found
|
||||
|
||||
## Critical Path Violations
|
||||
|
||||
### Config Variables (Exceptions)
|
||||
|
||||
Standard BMAD config variables treated as valid exceptions: bmb_creations_output_folder, communication_language, document_output_language, output_folder, planning_artifacts, project-root, project_name, test_artifacts, user_name
|
||||
|
||||
- No {project-root} hardcoded paths detected in body
|
||||
|
||||
- No dead relative links detected
|
||||
|
||||
- No module path assumptions detected
|
||||
|
||||
**Status:** ✅ PASS - No critical violations
|
||||
|
||||
## Menu Handling Validation
|
||||
|
||||
- No menu structures detected (linear step flow) [N/A]
|
||||
|
||||
## Step Type Validation
|
||||
|
||||
- steps-c/step-01-preflight.md: Init [PASS]
|
||||
- steps-c/step-02-generate-pipeline.md: Middle [PASS]
|
||||
- steps-c/step-03-configure-quality-gates.md: Middle [PASS]
|
||||
- steps-c/step-04-validate-and-summary.md: Final [PASS]
|
||||
- Step type validation assumes linear sequence (no branching/menu). Workflow-plan.md present for reference. [INFO]
|
||||
|
||||
## Output Format Validation
|
||||
|
||||
- Templates present: NONE
|
||||
- Steps with outputFile in frontmatter:
|
||||
- steps-c/step-02-generate-pipeline.md
|
||||
- steps-v/step-01-validate.md
|
||||
- checklist.md present: YES
|
||||
|
||||
## Validation Design Check
|
||||
|
||||
- Validation steps folder (steps-v) present: YES
|
||||
- Validation step(s) present: step-01-validate.md
|
||||
- Validation steps reference checklist data and auto-proceed
|
||||
|
||||
## Instruction Style Check
|
||||
|
||||
- Instruction style: Prescriptive (appropriate for TEA quality/compliance workflows)
|
||||
- Steps emphasize mandatory sequence, explicit success/failure metrics, and risk-based guidance
|
||||
|
||||
## Collaborative Experience Check
|
||||
|
||||
- Overall facilitation quality: GOOD
|
||||
- Steps use progressive prompts and clear role reinforcement; no laundry-list interrogation detected
|
||||
- Flow progression is clear and aligned to workflow goals
|
||||
|
||||
## Subagent Optimization Opportunities
|
||||
|
||||
- No high-priority subagent optimizations identified; workflow already uses step-file architecture
|
||||
- Pattern 1 (grep/regex): N/A for most steps
|
||||
- Pattern 2 (per-file analysis): already aligned to validation structure
|
||||
- Pattern 3 (data ops): minimal data file loads
|
||||
- Pattern 4 (parallel): optional for validation only
|
||||
|
||||
## Cohesive Review
|
||||
|
||||
- Overall assessment: GOOD
|
||||
- Flow is linear, goals are clear, and outputs map to TEA artifacts
|
||||
- Voice and tone consistent with Test Architect persona
|
||||
- Recommendation: READY (minor refinements optional)
|
||||
|
||||
## Plan Quality Validation
|
||||
|
||||
- Plan file present: workflow-plan.md
|
||||
- Planned steps found: 7 (all implemented)
|
||||
- Plan implementation status: Fully Implemented
|
||||
|
||||
## Summary
|
||||
|
||||
- Validation completed: 2026-01-27 10:24:01
|
||||
- Critical issues: 0
|
||||
- Warnings: 0 (informational notes only)
|
||||
- Readiness: READY (manual review optional)
|
||||
20
_bmad/tea/workflows/testarch/ci/workflow-plan.md
Normal file
20
_bmad/tea/workflows/testarch/ci/workflow-plan.md
Normal file
@@ -0,0 +1,20 @@
|
||||
# Workflow Plan: testarch-ci
|
||||
|
||||
## Create Mode (steps-c)
|
||||
- step-01-preflight.md
|
||||
|
||||
- step-02-generate-pipeline.md
|
||||
- step-03-configure-quality-gates.md
|
||||
- step-04-validate-and-summary.md
|
||||
|
||||
## Validate Mode (steps-v)
|
||||
- step-01-validate.md
|
||||
|
||||
## Edit Mode (steps-e)
|
||||
- step-01-assess.md
|
||||
- step-02-apply-edit.md
|
||||
|
||||
## Outputs
|
||||
- CI config (e.g., {project-root}/.github/workflows/test.yml)
|
||||
|
||||
- Pipeline guidance and artifacts configuration
|
||||
41
_bmad/tea/workflows/testarch/ci/workflow.md
Normal file
41
_bmad/tea/workflows/testarch/ci/workflow.md
Normal file
@@ -0,0 +1,41 @@
|
||||
---
|
||||
name: testarch-ci
|
||||
description: Scaffold CI/CD quality pipeline with test execution. Use when user says 'lets setup CI pipeline' or 'I want to create quality gates'
|
||||
web_bundle: true
|
||||
---
|
||||
|
||||
# CI/CD Pipeline Setup
|
||||
|
||||
**Goal:** Scaffold CI/CD quality pipeline with test execution, burn-in loops, and artifact collection
|
||||
|
||||
**Role:** You are the Master Test Architect.
|
||||
|
||||
---
|
||||
|
||||
## WORKFLOW ARCHITECTURE
|
||||
|
||||
This workflow uses **tri-modal step-file architecture**:
|
||||
|
||||
- **Create mode (steps-c/)**: primary execution flow
|
||||
- **Validate mode (steps-v/)**: validation against checklist
|
||||
- **Edit mode (steps-e/)**: revise existing outputs
|
||||
|
||||
---
|
||||
|
||||
## INITIALIZATION SEQUENCE
|
||||
|
||||
### 1. Mode Determination
|
||||
|
||||
"Welcome to the workflow. What would you like to do?"
|
||||
|
||||
- **[C] Create** — Run the workflow
|
||||
- **[R] Resume** — Resume an interrupted workflow
|
||||
- **[V] Validate** — Validate existing outputs
|
||||
- **[E] Edit** — Edit existing outputs
|
||||
|
||||
### 2. Route to First Step
|
||||
|
||||
- **If C:** Load `steps-c/step-01-preflight.md`
|
||||
- **If R:** Load `steps-c/step-01b-resume.md`
|
||||
- **If V:** Load `steps-v/step-01-validate.md`
|
||||
- **If E:** Load `steps-e/step-01-assess.md`
|
||||
48
_bmad/tea/workflows/testarch/ci/workflow.yaml
Normal file
48
_bmad/tea/workflows/testarch/ci/workflow.yaml
Normal file
@@ -0,0 +1,48 @@
|
||||
# Test Architect workflow: ci
|
||||
name: testarch-ci
|
||||
# prettier-ignore
|
||||
description: 'Scaffold CI/CD quality pipeline with test execution. Use when the user says "lets setup CI pipeline" or "I want to create quality gates"'
|
||||
|
||||
# Critical variables from config
|
||||
config_source: "{project-root}/_bmad/tea/config.yaml"
|
||||
output_folder: "{config_source}:output_folder"
|
||||
test_artifacts: "{config_source}:test_artifacts"
|
||||
user_name: "{config_source}:user_name"
|
||||
communication_language: "{config_source}:communication_language"
|
||||
document_output_language: "{config_source}:document_output_language"
|
||||
date: system-generated
|
||||
|
||||
# Workflow components
|
||||
installed_path: "{project-root}/_bmad/tea/workflows/testarch/ci"
|
||||
instructions: "{installed_path}/instructions.md"
|
||||
validation: "{installed_path}/checklist.md"
|
||||
|
||||
# Variables and inputs
|
||||
variables:
|
||||
ci_platform: "auto" # auto, github-actions, gitlab-ci, circle-ci, jenkins, azure-devops, harness - user can override
|
||||
test_dir: "{project-root}/tests" # Root test directory
|
||||
test_stack_type: "auto" # auto, frontend, backend, fullstack - detected or user override
|
||||
test_framework: "auto" # auto, playwright, cypress, jest, vitest - detected or user override
|
||||
|
||||
# Output configuration (resolved dynamically based on ci_platform detection)
|
||||
default_output_file: "{project-root}/.github/workflows/test.yml" # GitHub Actions default; overridden per platform
|
||||
|
||||
# Required tools
|
||||
required_tools:
|
||||
- read_file # Read .nvmrc, package.json, framework config
|
||||
- write_file # Create CI config, scripts, documentation
|
||||
- create_directory # Create .github/workflows/ or .gitlab-ci/ directories
|
||||
- list_files # Detect existing CI configuration
|
||||
- search_repo # Find test files for selective testing
|
||||
|
||||
tags:
|
||||
- qa
|
||||
- ci-cd
|
||||
- test-architect
|
||||
- pipeline
|
||||
- automation
|
||||
|
||||
execution_hints:
|
||||
interactive: false # Minimize prompts, auto-detect when possible
|
||||
autonomous: true # Proceed without user input unless blocked
|
||||
iterative: true
|
||||
345
_bmad/tea/workflows/testarch/framework/checklist.md
Normal file
345
_bmad/tea/workflows/testarch/framework/checklist.md
Normal file
@@ -0,0 +1,345 @@
|
||||
# Test Framework Setup - Validation Checklist
|
||||
|
||||
This checklist ensures the framework workflow completes successfully and all deliverables meet quality standards.
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Before starting the workflow:
|
||||
|
||||
- [ ] Project root contains a valid project manifest (`package.json`, `pyproject.toml`, `pom.xml`, `build.gradle`, `go.mod`, `*.csproj`, `Gemfile`, or `Cargo.toml`)
|
||||
- [ ] No existing test framework detected that conflicts with the target setup
|
||||
- [ ] Project type identifiable (React, Vue, Angular, Next.js, Node, Python, Java, Go, .NET, Ruby, Rust, etc.)
|
||||
- [ ] Bundler identifiable (Vite, Webpack, Rollup, esbuild) or not applicable (backend projects)
|
||||
- [ ] User has write permissions to create directories and files
|
||||
|
||||
---
|
||||
|
||||
## Process Steps
|
||||
|
||||
### Step 1: Preflight Checks
|
||||
|
||||
- [ ] Stack type detected (`frontend`, `backend`, or `fullstack`)
|
||||
- [ ] Project manifest successfully read and parsed (`package.json`, `pyproject.toml`, `pom.xml`, `go.mod`, etc.)
|
||||
- [ ] Project type extracted correctly
|
||||
- [ ] Bundler identified (or marked as N/A for backend projects)
|
||||
- [ ] No framework conflicts detected
|
||||
- [ ] Architecture documents located (if available)
|
||||
|
||||
### Step 2: Framework Selection
|
||||
|
||||
- [ ] Framework auto-detection logic executed
|
||||
- [ ] Framework choice justified (Playwright vs Cypress for frontend; pytest/JUnit/Go test/xUnit/RSpec for backend)
|
||||
- [ ] Framework preference respected (if explicitly set via `config.test_framework`)
|
||||
- [ ] User notified of framework selection and rationale
|
||||
|
||||
### Step 3: Directory Structure
|
||||
|
||||
- [ ] `tests/` root directory created
|
||||
- [ ] `tests/e2e/` directory created (or user's preferred structure)
|
||||
- [ ] `tests/support/` directory created (critical pattern)
|
||||
- [ ] `tests/support/fixtures/` directory created
|
||||
- [ ] `tests/support/fixtures/factories/` directory created
|
||||
- [ ] `tests/support/helpers/` directory created
|
||||
- [ ] `tests/support/page-objects/` directory created (if applicable)
|
||||
- [ ] All directories have correct permissions
|
||||
|
||||
**Note**: Test organization is flexible (e2e/, api/, integration/). The **support/** folder is the key pattern.
|
||||
|
||||
### Step 4: Configuration Files
|
||||
|
||||
- [ ] Framework config file created (`playwright.config.ts` or `cypress.config.ts`)
|
||||
- [ ] Config file uses TypeScript (if `use_typescript: true`)
|
||||
- [ ] Timeouts configured correctly (action: 15s, navigation: 30s, test: 60s)
|
||||
- [ ] Base URL configured with environment variable fallback
|
||||
- [ ] Trace/screenshot/video set to retain-on-failure
|
||||
- [ ] Multiple reporters configured (HTML + JUnit + console)
|
||||
- [ ] Parallel execution enabled
|
||||
- [ ] CI-specific settings configured (retries, workers)
|
||||
- [ ] Config file is syntactically valid (no compilation errors)
|
||||
|
||||
### Step 5: Environment Configuration
|
||||
|
||||
- [ ] `.env.example` created in project root
|
||||
- [ ] `TEST_ENV` variable defined
|
||||
- [ ] `BASE_URL` variable defined with default
|
||||
- [ ] `API_URL` variable defined (if applicable)
|
||||
- [ ] Authentication variables defined (if applicable)
|
||||
- [ ] Feature flag variables defined (if applicable)
|
||||
- [ ] `.nvmrc` created with appropriate Node version
|
||||
|
||||
### Step 6: Fixture Architecture
|
||||
|
||||
- [ ] `tests/support/fixtures/index.ts` created
|
||||
- [ ] Base fixture extended from Playwright/Cypress
|
||||
- [ ] Type definitions for fixtures created
|
||||
- [ ] mergeTests pattern implemented (if multiple fixtures)
|
||||
- [ ] Auto-cleanup logic included in fixtures
|
||||
- [ ] Fixture architecture follows knowledge base patterns
|
||||
|
||||
### Step 7: Data Factories
|
||||
|
||||
- [ ] At least one factory created (e.g., UserFactory)
|
||||
- [ ] Factories use @faker-js/faker for realistic data
|
||||
- [ ] Factories track created entities (for cleanup)
|
||||
- [ ] Factories implement `cleanup()` method
|
||||
- [ ] Factories integrate with fixtures
|
||||
- [ ] Factories follow knowledge base patterns
|
||||
|
||||
### Step 8: Sample Tests
|
||||
|
||||
- [ ] Example test file created (`tests/e2e/example.spec.ts`)
|
||||
- [ ] Test uses fixture architecture
|
||||
- [ ] Test demonstrates data factory usage
|
||||
- [ ] Test uses proper selector strategy (data-testid)
|
||||
- [ ] Test follows Given-When-Then structure
|
||||
- [ ] Test includes proper assertions
|
||||
- [ ] Network interception demonstrated (if applicable)
|
||||
|
||||
### Step 9: Helper Utilities
|
||||
|
||||
- [ ] API helper created (if API testing needed)
|
||||
- [ ] Network helper created (if network mocking needed)
|
||||
- [ ] Auth helper created (if authentication needed)
|
||||
- [ ] Helpers follow functional patterns
|
||||
- [ ] Helpers have proper error handling
|
||||
|
||||
### Step 10: Documentation
|
||||
|
||||
- [ ] `tests/README.md` created
|
||||
- [ ] Setup instructions included
|
||||
- [ ] Running tests section included
|
||||
- [ ] Architecture overview section included
|
||||
- [ ] Best practices section included
|
||||
- [ ] CI integration section included
|
||||
- [ ] Knowledge base references included
|
||||
- [ ] Troubleshooting section included
|
||||
|
||||
### Step 11: Build & Test Script Updates
|
||||
|
||||
- [ ] Minimal test script added to appropriate config (`package.json` for frontend, `Makefile`/`pyproject.toml`/`build.gradle` for backend)
|
||||
- [ ] Test framework dependency added (if not already present)
|
||||
- [ ] Type definitions added (if TypeScript)
|
||||
- [ ] Users can extend with additional scripts as needed
|
||||
|
||||
---
|
||||
|
||||
## Output Validation
|
||||
|
||||
### Configuration Validation
|
||||
|
||||
- [ ] Config file loads without errors
|
||||
- [ ] Config file passes linting (if linter configured)
|
||||
- [ ] Config file uses correct syntax for chosen framework
|
||||
- [ ] All paths in config resolve correctly
|
||||
- [ ] Reporter output directories exist or are created on test run
|
||||
|
||||
### Test Execution Validation
|
||||
|
||||
- [ ] Sample test runs successfully
|
||||
- [ ] Test execution produces expected output (pass/fail)
|
||||
- [ ] Test artifacts generated correctly (traces, screenshots, videos)
|
||||
- [ ] Test report generated successfully
|
||||
- [ ] No console errors or warnings during test run
|
||||
|
||||
### Directory Structure Validation
|
||||
|
||||
- [ ] All required directories exist
|
||||
- [ ] Directory structure matches framework conventions
|
||||
- [ ] No duplicate or conflicting directories
|
||||
- [ ] Directories accessible with correct permissions
|
||||
|
||||
### File Integrity Validation
|
||||
|
||||
- [ ] All generated files are syntactically correct
|
||||
- [ ] No placeholder text left in files (e.g., "TODO", "FIXME")
|
||||
- [ ] All imports resolve correctly
|
||||
- [ ] No hardcoded credentials or secrets in files
|
||||
- [ ] All file paths use correct separators for OS
|
||||
|
||||
---
|
||||
|
||||
## Quality Checks
|
||||
|
||||
### Code Quality
|
||||
|
||||
- [ ] Generated code follows project coding standards
|
||||
- [ ] TypeScript types are complete and accurate (no `any` unless necessary)
|
||||
- [ ] No unused imports or variables
|
||||
- [ ] Consistent code formatting (matches project style)
|
||||
- [ ] No linting errors in generated files
|
||||
|
||||
### Best Practices Compliance
|
||||
|
||||
- [ ] Fixture architecture follows pure function → fixture → mergeTests pattern
|
||||
- [ ] Data factories implement auto-cleanup
|
||||
- [ ] Network interception occurs before navigation
|
||||
- [ ] Selectors use data-testid strategy
|
||||
- [ ] Artifacts only captured on failure
|
||||
- [ ] Tests follow Given-When-Then structure
|
||||
- [ ] No hard-coded waits or sleeps
|
||||
|
||||
### Knowledge Base Alignment
|
||||
|
||||
- [ ] Fixture pattern matches `fixture-architecture.md`
|
||||
- [ ] Data factories match `data-factories.md`
|
||||
- [ ] Network handling matches `network-first.md`
|
||||
- [ ] Config follows `playwright-config.md` or `test-config.md`
|
||||
- [ ] Test quality matches `test-quality.md`
|
||||
|
||||
### Pact Consumer CDC Alignment (when `tea_use_pactjs_utils` enabled)
|
||||
|
||||
- [ ] `vitest.config.pact.ts` is minimal (no pool/coverage/setup copied from unit config)
|
||||
- [ ] Script names match pactjs-utils (`test:pact:consumer`, `publish:pact`, `can:i:deploy:consumer`, `record:consumer:deployment`)
|
||||
- [ ] Scripts source `env-setup.sh` inline in package.json
|
||||
- [ ] Shell scripts use `pact-broker` not `npx pact-broker`
|
||||
- [ ] Shell scripts use `PACTICIPANT` env var pattern (not hardcoded service names)
|
||||
- [ ] `can-i-deploy.sh` has `--retry-while-unknown=10 --retry-interval=30`
|
||||
- [ ] `record-deployment.sh` has branch guard (only records on main/master)
|
||||
- [ ] `env-setup.sh` uses `set -eu`; broker scripts use `set -euo pipefail` — each with explanatory comment
|
||||
- [ ] CI workflow named `contract-test-consumer.yml`
|
||||
- [ ] CI has workflow-level env block (not per-step)
|
||||
- [ ] CI has `detect-breaking-change` step before install
|
||||
- [ ] CI step numbering skips (3) — webhook-triggered provider verification
|
||||
- [ ] CI can-i-deploy has `PACT_BREAKING_CHANGE != 'true'` condition
|
||||
- [ ] CI has NO upload-artifact step (broker is source of truth)
|
||||
- [ ] `.github/actions/detect-breaking-change/action.yml` exists
|
||||
- [ ] Consumer tests use `.pacttest.ts` extension
|
||||
- [ ] Consumer tests use PactV4 `addInteraction()` builder (not PactV3 fluent API)
|
||||
- [ ] Consumer tests call REAL consumer code (actual API client functions), NOT raw `fetch()`
|
||||
- [ ] Consumer code exposes URL injection mechanism (`setApiUrl()`, env var, or constructor param)
|
||||
- [ ] Local consumer-helpers shim present if `@seontechnologies/pactjs-utils` not installed
|
||||
- [ ] `.gitignore` includes `/pacts/` and `pact-logs/`
|
||||
|
||||
### Security Checks
|
||||
|
||||
- [ ] No credentials in configuration files
|
||||
- [ ] .env.example contains placeholders, not real values
|
||||
- [ ] Sensitive test data handled securely
|
||||
- [ ] API keys and tokens use environment variables
|
||||
- [ ] No secrets committed to version control
|
||||
|
||||
---
|
||||
|
||||
## Integration Points
|
||||
|
||||
### Status File Integration
|
||||
|
||||
- [ ] Framework initialization logged in Quality & Testing Progress section
|
||||
- [ ] Status file updated with completion timestamp
|
||||
- [ ] Status file shows framework: Playwright or Cypress
|
||||
|
||||
### Knowledge Base Integration
|
||||
|
||||
- [ ] Relevant knowledge fragments identified from tea-index.csv
|
||||
- [ ] Knowledge fragments successfully loaded
|
||||
- [ ] Patterns from knowledge base applied correctly
|
||||
- [ ] Knowledge base references included in documentation
|
||||
|
||||
### Workflow Dependencies
|
||||
|
||||
- [ ] Can proceed to `ci` workflow after completion
|
||||
- [ ] Can proceed to `test-design` workflow after completion
|
||||
- [ ] Can proceed to `atdd` workflow after completion
|
||||
- [ ] Framework setup compatible with downstream workflows
|
||||
|
||||
---
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
**All of the following must be true:**
|
||||
|
||||
- [ ] All prerequisite checks passed
|
||||
- [ ] All process steps completed without errors
|
||||
- [ ] All output validations passed
|
||||
- [ ] All quality checks passed
|
||||
- [ ] All integration points verified
|
||||
- [ ] Sample test executes successfully
|
||||
- [ ] User can run the appropriate test command without errors (`npm run test:e2e`, `pytest`, `go test ./...`, etc.)
|
||||
- [ ] Documentation is complete and accurate
|
||||
- [ ] No critical issues or blockers identified
|
||||
|
||||
---
|
||||
|
||||
## Post-Workflow Actions
|
||||
|
||||
**User must complete:**
|
||||
|
||||
1. [ ] Copy `.env.example` to `.env`
|
||||
2. [ ] Fill in environment-specific values in `.env`
|
||||
3. [ ] Run `npm install` to install test dependencies
|
||||
4. [ ] Run `npm run test:e2e` to verify setup
|
||||
5. [ ] Review `tests/README.md` for project-specific guidance
|
||||
|
||||
**Recommended next workflows:**
|
||||
|
||||
1. [ ] Run `ci` workflow to set up CI/CD pipeline
|
||||
2. [ ] Run `test-design` workflow to plan test coverage
|
||||
3. [ ] Run `atdd` workflow when ready to develop stories
|
||||
|
||||
---
|
||||
|
||||
## Rollback Procedure
|
||||
|
||||
If workflow fails and needs to be rolled back:
|
||||
|
||||
1. [ ] Delete `tests/` directory
|
||||
2. [ ] Remove test scripts from package.json
|
||||
3. [ ] Delete `.env.example` (if created)
|
||||
4. [ ] Delete `.nvmrc` (if created)
|
||||
5. [ ] Delete framework config file
|
||||
6. [ ] Remove test dependencies from package.json (if added)
|
||||
7. [ ] Run `npm install` to clean up node_modules
|
||||
|
||||
---
|
||||
|
||||
## Notes
|
||||
|
||||
### Common Issues
|
||||
|
||||
**Issue**: Config file has TypeScript errors
|
||||
|
||||
- **Solution**: Ensure `@playwright/test` or `cypress` types are installed
|
||||
|
||||
**Issue**: Sample test fails to run
|
||||
|
||||
- **Solution**: Check BASE_URL in .env, ensure app is running
|
||||
|
||||
**Issue**: Fixture cleanup not working
|
||||
|
||||
- **Solution**: Verify cleanup() is called in fixture teardown
|
||||
|
||||
**Issue**: Network interception not working
|
||||
|
||||
- **Solution**: Ensure route setup occurs before page.goto()
|
||||
|
||||
### Framework-Specific Considerations
|
||||
|
||||
**Playwright:**
|
||||
|
||||
- Requires Node.js 18+
|
||||
- Browser binaries auto-installed on first run
|
||||
- Trace viewer requires running `npx playwright show-trace`
|
||||
|
||||
**Cypress:**
|
||||
|
||||
- Requires Node.js 18+
|
||||
- Cypress app opens on first run
|
||||
- Component testing requires additional setup
|
||||
|
||||
### Version Compatibility
|
||||
|
||||
- [ ] Node.js version matches .nvmrc
|
||||
- [ ] Framework version compatible with Node.js version
|
||||
- [ ] TypeScript version compatible with framework
|
||||
- [ ] All peer dependencies satisfied
|
||||
|
||||
---
|
||||
|
||||
**Checklist Complete**: Sign off when all items checked and validated.
|
||||
|
||||
**Completed by:** {name}
|
||||
**Date:** {date}
|
||||
**Framework:** { Playwright / Cypress or something else}
|
||||
**Notes:** {notes}
|
||||
45
_bmad/tea/workflows/testarch/framework/instructions.md
Normal file
45
_bmad/tea/workflows/testarch/framework/instructions.md
Normal file
@@ -0,0 +1,45 @@
|
||||
<!-- Powered by BMAD-CORE™ -->
|
||||
|
||||
# Test Framework Setup
|
||||
|
||||
**Workflow ID**: `_bmad/tea/testarch/framework`
|
||||
**Version**: 5.0 (Step-File Architecture)
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Initialize a production-ready test framework (Playwright or Cypress) with fixtures, helpers, configuration, and best practices.
|
||||
|
||||
---
|
||||
|
||||
## WORKFLOW ARCHITECTURE
|
||||
|
||||
This workflow uses **step-file architecture**:
|
||||
|
||||
- **Micro-file Design**: Each step is self-contained
|
||||
- **JIT Loading**: Only the current step file is in memory
|
||||
- **Sequential Enforcement**: Execute steps in order without skipping
|
||||
|
||||
---
|
||||
|
||||
## INITIALIZATION SEQUENCE
|
||||
|
||||
### 1. Configuration Loading
|
||||
|
||||
From `workflow.yaml`, resolve:
|
||||
|
||||
- `config_source`, `test_artifacts`, `user_name`, `communication_language`, `document_output_language`, `date`
|
||||
- `test_dir`, `use_typescript`, `framework_preference`, `project_size`
|
||||
|
||||
### 2. First Step
|
||||
|
||||
Load, read completely, and execute:
|
||||
`{project-root}/_bmad/tea/workflows/testarch/framework/steps-c/step-01-preflight.md`
|
||||
|
||||
### 3. Resume Support
|
||||
|
||||
If the user selects **Resume** mode, load, read completely, and execute:
|
||||
`{project-root}/_bmad/tea/workflows/testarch/framework/steps-c/step-01b-resume.md`
|
||||
|
||||
This checks the output document for progress tracking frontmatter and routes to the next incomplete step.
|
||||
@@ -0,0 +1,132 @@
|
||||
---
|
||||
name: 'step-01-preflight'
|
||||
description: 'Verify prerequisites and gather project context'
|
||||
nextStepFile: './step-02-select-framework.md'
|
||||
outputFile: '{test_artifacts}/framework-setup-progress.md'
|
||||
---
|
||||
|
||||
# Step 1: Preflight Checks
|
||||
|
||||
## STEP GOAL
|
||||
|
||||
Verify the project is ready for framework scaffolding and gather key context.
|
||||
|
||||
## MANDATORY EXECUTION RULES
|
||||
|
||||
- 📖 Read the entire step file before acting
|
||||
- ✅ Speak in `{communication_language}`
|
||||
- 🚫 Halt if preflight requirements fail
|
||||
|
||||
---
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
- 💾 Record outputs before proceeding
|
||||
- 📖 Load the next step only when instructed
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: config, loaded artifacts, and knowledge fragments
|
||||
- Focus: this step's goal only
|
||||
- Limits: do not execute future steps
|
||||
- Dependencies: prior steps' outputs (if any)
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly. Do not skip, reorder, or improvise.
|
||||
|
||||
## 1. Stack Detection
|
||||
|
||||
**Read `config.test_stack_type`** from `{config_source}`.
|
||||
|
||||
**Auto-Detection Algorithm** (when `test_stack_type` is `"auto"` or not configured):
|
||||
|
||||
- Scan `{project-root}` for project manifests:
|
||||
- **Frontend indicators**: `package.json` with react/vue/angular/next dependencies, `playwright.config.*`, `vite.config.*`, `webpack.config.*`
|
||||
- **Backend indicators**: `pyproject.toml`, `pom.xml`/`build.gradle`, `go.mod`, `*.csproj`/`*.sln`, `Gemfile`, `Cargo.toml`
|
||||
- **Both present** = `fullstack`; only frontend = `frontend`; only backend = `backend`
|
||||
- Explicit `test_stack_type` config value overrides auto-detection
|
||||
- **Backward compatibility**: if `test_stack_type` is not in config, treat as `"auto"` (preserves current frontend behavior for existing installs)
|
||||
|
||||
Store result as `{detected_stack}` = `frontend` | `backend` | `fullstack`
|
||||
|
||||
---
|
||||
|
||||
## 2. Validate Prerequisites
|
||||
|
||||
**If {detected_stack} is `frontend` or `fullstack`:**
|
||||
|
||||
- `package.json` exists in project root
|
||||
- No existing E2E framework (`playwright.config.*`, `cypress.config.*`, `cypress.json`)
|
||||
|
||||
**If {detected_stack} is `backend` or `fullstack`:**
|
||||
|
||||
- At least one backend project manifest exists (`pyproject.toml`, `pom.xml`, `build.gradle`, `go.mod`, `*.csproj`, `Gemfile`, `Cargo.toml`)
|
||||
- No existing test framework config that conflicts (e.g., `conftest.py` with full pytest suite, `src/test/` with JUnit suite)
|
||||
|
||||
- Architecture/stack context available (project type, bundler, dependencies)
|
||||
|
||||
If any fail, **HALT** and report the missing requirement.
|
||||
|
||||
---
|
||||
|
||||
## 3. Gather Project Context
|
||||
|
||||
**If {detected_stack} is `frontend` or `fullstack`:**
|
||||
|
||||
- Read `package.json` to identify framework, bundler, dependencies
|
||||
|
||||
**If {detected_stack} is `backend` or `fullstack`:**
|
||||
|
||||
- Read the relevant project manifest (`pyproject.toml`, `pom.xml`, `go.mod`, `*.csproj`, `Gemfile`, `Cargo.toml`) to identify language, framework, and dependencies
|
||||
|
||||
- Check for architecture docs (`architecture.md`, `tech-spec*.md`) if available
|
||||
- Note auth requirements and APIs (if documented)
|
||||
|
||||
---
|
||||
|
||||
## 3. Confirm Findings
|
||||
|
||||
Summarize:
|
||||
|
||||
- Project type and bundler
|
||||
- Whether a framework is already installed
|
||||
- Any relevant context docs found
|
||||
|
||||
---
|
||||
|
||||
### 4. Save Progress
|
||||
|
||||
**Save this step's accumulated work to `{outputFile}`.**
|
||||
|
||||
- **If `{outputFile}` does not exist** (first save), create it with YAML frontmatter:
|
||||
|
||||
```yaml
|
||||
---
|
||||
stepsCompleted: ['step-01-preflight']
|
||||
lastStep: 'step-01-preflight'
|
||||
lastSaved: '{date}'
|
||||
---
|
||||
```
|
||||
|
||||
Then write this step's output below the frontmatter.
|
||||
|
||||
- **If `{outputFile}` already exists**, update:
|
||||
- Add `'step-01-preflight'` to `stepsCompleted` array (only if not already present)
|
||||
- Set `lastStep: 'step-01-preflight'`
|
||||
- Set `lastSaved: '{date}'`
|
||||
- Append this step's output to the appropriate section of the document.
|
||||
|
||||
Load next step: `{nextStepFile}`
|
||||
|
||||
## 🚨 SYSTEM SUCCESS/FAILURE METRICS:
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- Step completed in full with required outputs
|
||||
|
||||
### ❌ SYSTEM FAILURE:
|
||||
|
||||
- Skipped sequence steps or missing outputs
|
||||
**Master Rule:** Skipping steps is FORBIDDEN.
|
||||
@@ -0,0 +1,116 @@
|
||||
---
|
||||
name: 'step-01b-resume'
|
||||
description: 'Resume interrupted workflow from last completed step'
|
||||
outputFile: '{test_artifacts}/framework-setup-progress.md'
|
||||
---
|
||||
|
||||
# Step 1b: Resume Workflow
|
||||
|
||||
## STEP GOAL
|
||||
|
||||
Resume an interrupted workflow by loading the existing progress document, verifying previously created artifacts still exist on disk, displaying progress, and routing to the next incomplete step.
|
||||
|
||||
## MANDATORY EXECUTION RULES
|
||||
|
||||
- 📖 Read the entire step file before acting
|
||||
- ✅ Speak in `{communication_language}`
|
||||
|
||||
---
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
- 📖 Load the next step only when instructed
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: Output document with progress frontmatter
|
||||
- Focus: Load progress and route to next step
|
||||
- Limits: Do not re-execute completed steps
|
||||
- Dependencies: Output document must exist from a previous run
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly. Do not skip, reorder, or improvise.
|
||||
|
||||
### 1. Load Output Document
|
||||
|
||||
Read `{outputFile}` and parse YAML frontmatter for:
|
||||
|
||||
- `stepsCompleted` — array of completed step names
|
||||
- `lastStep` — last completed step name
|
||||
- `lastSaved` — timestamp of last save
|
||||
|
||||
**If `{outputFile}` does not exist**, display:
|
||||
|
||||
"⚠️ **No previous progress found.** There is no output document to resume from. Please use **[C] Create** to start a fresh workflow run."
|
||||
|
||||
**THEN:** Halt. Do not proceed.
|
||||
|
||||
---
|
||||
|
||||
### 2. Verify Previously Created Artifacts
|
||||
|
||||
Since this workflow creates code files, verify that artifacts from completed steps still exist on disk:
|
||||
|
||||
- If `step-01-preflight` completed: Confirm `package.json` still exists
|
||||
- If `step-03-scaffold-framework` completed: Confirm directory structure and config files exist
|
||||
- If `step-04-docs-and-scripts` completed: Confirm `{test_dir}/README.md` exists
|
||||
|
||||
If any expected artifacts are missing, warn the user and suggest re-running from the step that created them.
|
||||
|
||||
---
|
||||
|
||||
### 3. Display Progress Dashboard
|
||||
|
||||
Display:
|
||||
|
||||
"📋 **Workflow Resume — Test Framework Setup**
|
||||
|
||||
**Last saved:** {lastSaved}
|
||||
**Steps completed:** {stepsCompleted.length} of 5
|
||||
|
||||
1. ✅/⬜ Preflight Checks (step-01-preflight)
|
||||
2. ✅/⬜ Select Framework (step-02-select-framework)
|
||||
3. ✅/⬜ Scaffold Framework (step-03-scaffold-framework)
|
||||
4. ✅/⬜ Docs & Scripts (step-04-docs-and-scripts)
|
||||
5. ✅/⬜ Validate & Summary (step-05-validate-and-summary)"
|
||||
|
||||
---
|
||||
|
||||
### 4. Route to Next Step
|
||||
|
||||
Based on `lastStep`, load the next incomplete step:
|
||||
|
||||
- `'step-01-preflight'` → `./step-02-select-framework.md`
|
||||
- `'step-02-select-framework'` → `./step-03-scaffold-framework.md`
|
||||
- `'step-03-scaffold-framework'` → `./step-04-docs-and-scripts.md`
|
||||
- `'step-04-docs-and-scripts'` → `./step-05-validate-and-summary.md`
|
||||
- `'step-05-validate-and-summary'` → **Workflow already complete.** Display: "✅ **All steps completed.** Use **[V] Validate** to review outputs or **[E] Edit** to make revisions." Then halt.
|
||||
|
||||
**If `lastStep` does not match any value above**, display: "⚠️ **Unknown progress state** (`lastStep`: {lastStep}). Please use **[C] Create** to start fresh." Then halt.
|
||||
|
||||
**Otherwise**, load the identified step file, read completely, and execute.
|
||||
|
||||
The existing content in `{outputFile}` provides context from previously completed steps.
|
||||
|
||||
---
|
||||
|
||||
## 🚨 SYSTEM SUCCESS/FAILURE METRICS
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- Output document loaded and parsed correctly
|
||||
- Previously created artifacts verified on disk
|
||||
- Progress dashboard displayed accurately
|
||||
- Routed to correct next step
|
||||
|
||||
### ❌ SYSTEM FAILURE:
|
||||
|
||||
- Not loading output document
|
||||
- Not verifying existing artifacts
|
||||
- Incorrect progress display
|
||||
- Routing to wrong step
|
||||
- Re-executing completed steps
|
||||
|
||||
**Master Rule:** Resume MUST route to the exact next incomplete step. Never re-execute completed steps.
|
||||
@@ -0,0 +1,117 @@
|
||||
---
|
||||
name: 'step-02-select-framework'
|
||||
description: 'Select Playwright or Cypress and justify choice'
|
||||
nextStepFile: './step-03-scaffold-framework.md'
|
||||
outputFile: '{test_artifacts}/framework-setup-progress.md'
|
||||
---
|
||||
|
||||
# Step 2: Framework Selection
|
||||
|
||||
## STEP GOAL
|
||||
|
||||
Choose the most appropriate framework and document the rationale.
|
||||
|
||||
## MANDATORY EXECUTION RULES
|
||||
|
||||
- 📖 Read the entire step file before acting
|
||||
- ✅ Speak in `{communication_language}`
|
||||
|
||||
---
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
- 💾 Record outputs before proceeding
|
||||
- 📖 Load the next step only when instructed
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: config, loaded artifacts, and knowledge fragments
|
||||
- Focus: this step's goal only
|
||||
- Limits: do not execute future steps
|
||||
- Dependencies: prior steps' outputs (if any)
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly. Do not skip, reorder, or improvise.
|
||||
|
||||
## 1. Selection Logic
|
||||
|
||||
Use `{detected_stack}` from Step 1 to guide framework selection.
|
||||
|
||||
**If {detected_stack} is `frontend` or `fullstack` (browser-based testing):**
|
||||
|
||||
Default to **Playwright** unless strong reasons suggest Cypress.
|
||||
|
||||
**Playwright recommended when:**
|
||||
|
||||
- Large or complex repo
|
||||
- Multi-browser support needed
|
||||
- Heavy API + UI integration
|
||||
- CI speed/parallelism is important
|
||||
|
||||
**Cypress recommended when:**
|
||||
|
||||
- Small team prioritizes DX
|
||||
- Component testing focus
|
||||
- Simpler setup needed
|
||||
|
||||
**If {detected_stack} is `backend` (no browser-based testing):**
|
||||
|
||||
Select the framework matching the project language:
|
||||
|
||||
- **Python**: pytest (default), unittest
|
||||
- **Java/Kotlin**: JUnit 5 (default), TestNG
|
||||
- **Go**: Go test (built-in)
|
||||
- **C#/.NET**: xUnit (default), NUnit, MSTest
|
||||
- **Ruby**: RSpec (default), Minitest
|
||||
- **Rust**: cargo test (built-in)
|
||||
|
||||
**If {detected_stack} is `fullstack`:**
|
||||
|
||||
Select both a browser-based framework (Playwright/Cypress) AND the appropriate backend framework for the detected language.
|
||||
|
||||
Respect `config.test_framework` if explicitly set (not `"auto"`).
|
||||
|
||||
---
|
||||
|
||||
## 2. Announce Decision
|
||||
|
||||
State the selected framework and reasoning.
|
||||
|
||||
---
|
||||
|
||||
### 3. Save Progress
|
||||
|
||||
**Save this step's accumulated work to `{outputFile}`.**
|
||||
|
||||
- **If `{outputFile}` does not exist** (first save), create it with YAML frontmatter:
|
||||
|
||||
```yaml
|
||||
---
|
||||
stepsCompleted: ['step-02-select-framework']
|
||||
lastStep: 'step-02-select-framework'
|
||||
lastSaved: '{date}'
|
||||
---
|
||||
```
|
||||
|
||||
Then write this step's output below the frontmatter.
|
||||
|
||||
- **If `{outputFile}` already exists**, update:
|
||||
- Add `'step-02-select-framework'` to `stepsCompleted` array (only if not already present)
|
||||
- Set `lastStep: 'step-02-select-framework'`
|
||||
- Set `lastSaved: '{date}'`
|
||||
- Append this step's output to the appropriate section of the document.
|
||||
|
||||
Load next step: `{nextStepFile}`
|
||||
|
||||
## 🚨 SYSTEM SUCCESS/FAILURE METRICS:
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- Step completed in full with required outputs
|
||||
|
||||
### ❌ SYSTEM FAILURE:
|
||||
|
||||
- Skipped sequence steps or missing outputs
|
||||
**Master Rule:** Skipping steps is FORBIDDEN.
|
||||
@@ -0,0 +1,323 @@
|
||||
---
|
||||
name: 'step-03-scaffold-framework'
|
||||
description: 'Create framework scaffold with adaptive orchestration (agent-team, subagent, or sequential)'
|
||||
nextStepFile: './step-04-docs-and-scripts.md'
|
||||
knowledgeIndex: '{project-root}/_bmad/tea/testarch/tea-index.csv'
|
||||
outputFile: '{test_artifacts}/framework-setup-progress.md'
|
||||
---
|
||||
|
||||
# Step 3: Scaffold Framework
|
||||
|
||||
## STEP GOAL
|
||||
|
||||
Generate the test directory structure, configuration files, fixtures, factories, helpers, and sample tests using deterministic mode selection with runtime fallback.
|
||||
|
||||
## MANDATORY EXECUTION RULES
|
||||
|
||||
- 📖 Read the entire step file before acting
|
||||
- ✅ Speak in `{communication_language}`
|
||||
- ✅ Apply knowledge base patterns where required
|
||||
- ✅ Resolve execution mode from explicit user request first, then config
|
||||
- ✅ Apply fallback rules deterministically when requested mode is unsupported
|
||||
|
||||
---
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
- 💾 Record outputs before proceeding
|
||||
- 📖 Load the next step only when instructed
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: config, loaded artifacts, and knowledge fragments
|
||||
- Focus: this step's goal only
|
||||
- Limits: do not execute future steps
|
||||
- Dependencies: prior steps' outputs (if any)
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly. Do not skip, reorder, or improvise.
|
||||
|
||||
## 0. Resolve Execution Mode (User Override First)
|
||||
|
||||
```javascript
|
||||
const parseBooleanFlag = (value, defaultValue = true) => {
|
||||
if (typeof value === 'string') {
|
||||
const normalized = value.trim().toLowerCase();
|
||||
if (['false', '0', 'off', 'no'].includes(normalized)) return false;
|
||||
if (['true', '1', 'on', 'yes'].includes(normalized)) return true;
|
||||
}
|
||||
if (value === undefined || value === null) return defaultValue;
|
||||
return Boolean(value);
|
||||
};
|
||||
|
||||
const orchestrationContext = {
|
||||
config: {
|
||||
execution_mode: config.tea_execution_mode || 'auto', // "auto" | "subagent" | "agent-team" | "sequential"
|
||||
capability_probe: parseBooleanFlag(config.tea_capability_probe, true), // supports booleans and "false"/"true" strings
|
||||
},
|
||||
timestamp: new Date().toISOString().replace(/[:.]/g, '-'),
|
||||
};
|
||||
|
||||
const normalizeUserExecutionMode = (mode) => {
|
||||
if (typeof mode !== 'string') return null;
|
||||
const normalized = mode.trim().toLowerCase().replace(/[-_]/g, ' ').replace(/\s+/g, ' ');
|
||||
|
||||
if (normalized === 'auto') return 'auto';
|
||||
if (normalized === 'sequential') return 'sequential';
|
||||
if (normalized === 'subagent' || normalized === 'sub agent' || normalized === 'subagents' || normalized === 'sub agents') {
|
||||
return 'subagent';
|
||||
}
|
||||
if (normalized === 'agent team' || normalized === 'agent teams' || normalized === 'agentteam') {
|
||||
return 'agent-team';
|
||||
}
|
||||
|
||||
return null;
|
||||
};
|
||||
|
||||
const normalizeConfigExecutionMode = (mode) => {
|
||||
if (mode === 'subagent') return 'subagent';
|
||||
if (mode === 'auto' || mode === 'sequential' || mode === 'subagent' || mode === 'agent-team') {
|
||||
return mode;
|
||||
}
|
||||
return null;
|
||||
};
|
||||
|
||||
// Explicit user instruction in the active run takes priority over config.
|
||||
const explicitModeFromUser = normalizeUserExecutionMode(runtime.getExplicitExecutionModeHint?.() || null);
|
||||
|
||||
const requestedMode = explicitModeFromUser || normalizeConfigExecutionMode(orchestrationContext.config.execution_mode) || 'auto';
|
||||
const probeEnabled = orchestrationContext.config.capability_probe;
|
||||
|
||||
const supports = { subagent: false, agentTeam: false };
|
||||
if (probeEnabled) {
|
||||
supports.subagent = runtime.canLaunchSubagents?.() === true;
|
||||
supports.agentTeam = runtime.canLaunchAgentTeams?.() === true;
|
||||
}
|
||||
|
||||
let resolvedMode = requestedMode;
|
||||
if (requestedMode === 'auto') {
|
||||
if (supports.agentTeam) resolvedMode = 'agent-team';
|
||||
else if (supports.subagent) resolvedMode = 'subagent';
|
||||
else resolvedMode = 'sequential';
|
||||
} else if (probeEnabled && requestedMode === 'agent-team' && !supports.agentTeam) {
|
||||
resolvedMode = supports.subagent ? 'subagent' : 'sequential';
|
||||
} else if (probeEnabled && requestedMode === 'subagent' && !supports.subagent) {
|
||||
resolvedMode = 'sequential';
|
||||
}
|
||||
```
|
||||
|
||||
Resolution precedence:
|
||||
|
||||
1. Explicit user request in this run (`agent team` => `agent-team`; `subagent` => `subagent`; `sequential`; `auto`)
|
||||
2. `tea_execution_mode` from config
|
||||
3. Runtime capability fallback (when probing enabled)
|
||||
|
||||
## 1. Create Directory Structure
|
||||
|
||||
Use `{detected_stack}` from Step 1 to determine directory layout.
|
||||
|
||||
**If {detected_stack} is `frontend` or `fullstack`:**
|
||||
|
||||
- `{test_dir}/e2e/`
|
||||
- `{test_dir}/support/fixtures/`
|
||||
- `{test_dir}/support/helpers/`
|
||||
- `{test_dir}/support/page-objects/` (optional)
|
||||
|
||||
**If {detected_stack} is `backend` or `fullstack`:**
|
||||
|
||||
Create the idiomatic test directory for the detected language:
|
||||
|
||||
- **Python (pytest)**: `tests/` with `conftest.py`, `tests/unit/`, `tests/integration/`, `tests/api/`
|
||||
- **Java/Kotlin (JUnit)**: `src/test/java/` mirroring `src/main/java/` package structure, with `unit/`, `integration/`, `api/` sub-packages
|
||||
- **Go**: `*_test.go` files alongside source files (Go convention), plus `testdata/` for fixtures
|
||||
- **C#/.NET (xUnit)**: `tests/` project with `Unit/`, `Integration/`, `Api/` directories
|
||||
- **Ruby (RSpec)**: `spec/` with `spec/unit/`, `spec/integration/`, `spec/api/`, `spec/support/`
|
||||
- **Rust**: `tests/` for integration tests, inline `#[cfg(test)]` modules for unit tests
|
||||
|
||||
**If `config.tea_use_pactjs_utils` is enabled and runtime is Node.js/TypeScript** (i.e., `{detected_stack}` is `frontend` or `fullstack`, or `{detected_stack}` is `backend` with Node.js/TypeScript runtime):
|
||||
|
||||
Create Node.js/TypeScript contract testing directory structure per `pact-consumer-framework-setup.md`:
|
||||
|
||||
- `tests/contract/consumer/` — consumer contract test files (`.pacttest.ts` extension)
|
||||
- `tests/contract/support/` — pact config, provider state factories, consumer helpers shim
|
||||
- `scripts/` — shell scripts (`env-setup.sh`, `publish-pact.sh`, `can-i-deploy.sh`, `record-deployment.sh`)
|
||||
- `.github/actions/detect-breaking-change/` — PR checkbox-driven breaking change detection
|
||||
- `.github/workflows/contract-test-consumer.yml` — consumer CDC CI workflow
|
||||
|
||||
---
|
||||
|
||||
## 2. Generate Framework Config
|
||||
|
||||
**If {detected_stack} is `frontend` or `fullstack`:**
|
||||
|
||||
Create `playwright.config.ts` or `cypress.config.ts` with:
|
||||
|
||||
- **Timeouts**: action 15s, navigation 30s, test 60s
|
||||
- **Base URL**: env fallback (`BASE_URL`)
|
||||
- **Artifacts**: retain-on-failure (trace/screenshot/video)
|
||||
- **Reporters**: HTML + JUnit + console
|
||||
- **Parallelism**: enabled (CI tuned)
|
||||
|
||||
Use TypeScript if `use_typescript: true`.
|
||||
|
||||
**If {detected_stack} is `backend` or `fullstack`:**
|
||||
|
||||
Create the idiomatic test config for the detected framework:
|
||||
|
||||
- **pytest**: `pyproject.toml` `[tool.pytest.ini_options]` or `pytest.ini` with markers, test paths, coverage settings
|
||||
- **JUnit**: `build.gradle`/`pom.xml` test configuration with JUnit 5 dependencies, Surefire/Failsafe plugins
|
||||
- **Go test**: no config file needed (Go convention); optionally create `Makefile` test targets
|
||||
- **xUnit**: `.csproj` test project with xUnit and coverlet dependencies
|
||||
- **RSpec**: `.rspec` config file with `spec_helper.rb` and `rails_helper.rb` (if Rails)
|
||||
|
||||
---
|
||||
|
||||
## 3. Environment Setup
|
||||
|
||||
Create `.env.example` with `TEST_ENV`, `BASE_URL`, `API_URL`.
|
||||
|
||||
**Stack-conditional environment files:**
|
||||
|
||||
**If {detected_stack} is `frontend` or `fullstack` (Node.js):**
|
||||
|
||||
- `.nvmrc` using current LTS Node (prefer Node 24+)
|
||||
|
||||
**If {detected_stack} is `backend`:**
|
||||
|
||||
Create the idiomatic version file for the detected language:
|
||||
|
||||
- **Python**: `.python-version` with current stable Python (prefer 3.12+)
|
||||
- **Java**: `.java-version` or `JAVA_HOME` documentation in `.env.example`
|
||||
- **Go**: Go version is already in `go.mod` (no additional file needed)
|
||||
- **C#/.NET**: `global.json` with SDK version if not already present
|
||||
- **Ruby**: `.ruby-version` with current stable Ruby
|
||||
|
||||
---
|
||||
|
||||
## 4. Fixtures & Factories
|
||||
|
||||
Read `{config_source}` and use `{knowledgeIndex}` to load fragments based on `config.tea_use_playwright_utils`:
|
||||
|
||||
**If Playwright Utils enabled:**
|
||||
|
||||
- `overview.md`, `fixtures-composition.md`, `auth-session.md`, `api-request.md`, `burn-in.md`, `network-error-monitor.md`, `data-factories.md`
|
||||
- Recommend installing `@seontechnologies/playwright-utils`
|
||||
|
||||
**If disabled:**
|
||||
|
||||
- `fixture-architecture.md`, `data-factories.md`, `network-first.md`, `playwright-config.md`, `test-quality.md`
|
||||
|
||||
**If Pact.js Utils enabled** (`config.tea_use_pactjs_utils`):
|
||||
|
||||
- `pact-consumer-framework-setup.md` (CRITICAL: load this for directory structure, scripts, CI workflow, and PactV4 patterns)
|
||||
- `pactjs-utils-overview.md`, `pactjs-utils-consumer-helpers.md`, `pactjs-utils-provider-verifier.md`, `pactjs-utils-request-filter.md`, `contract-testing.md`
|
||||
- Recommend installing `@seontechnologies/pactjs-utils` and `@pact-foundation/pact`
|
||||
|
||||
**If Pact.js Utils disabled but contract testing relevant:**
|
||||
|
||||
- `contract-testing.md`
|
||||
|
||||
**If Pact MCP enabled** (`config.tea_pact_mcp` is `"mcp"`):
|
||||
|
||||
- `pact-mcp.md`
|
||||
|
||||
Implement:
|
||||
|
||||
- Fixture index with `mergeTests`
|
||||
- Auto-cleanup hooks
|
||||
- Faker-based data factories with overrides
|
||||
|
||||
---
|
||||
|
||||
## 5. Sample Tests & Helpers
|
||||
|
||||
**If {detected_stack} is `frontend` or `fullstack`:**
|
||||
|
||||
Create example tests in `{test_dir}/e2e/` demonstrating:
|
||||
|
||||
- Given/When/Then format
|
||||
- data-testid selector strategy
|
||||
- Factory usage
|
||||
- Network interception pattern (if applicable)
|
||||
|
||||
**If {detected_stack} is `backend` or `fullstack`:**
|
||||
|
||||
Create example tests in the idiomatic location for the detected language:
|
||||
|
||||
- **Python**: `tests/test_example.py` with pytest fixtures, parametrize, and factory usage
|
||||
- **Java**: `src/test/java/.../ExampleTest.java` with JUnit 5 annotations, `@BeforeEach` setup
|
||||
- **Go**: `example_test.go` alongside source with table-driven tests and `testify` assertions
|
||||
- **C#/.NET**: `tests/ExampleTests.cs` with xUnit `[Fact]`/`[Theory]` and fixture injection
|
||||
- **Ruby**: `spec/example_spec.rb` with RSpec `describe`/`context`/`it` and factory_bot
|
||||
|
||||
Create helpers for:
|
||||
|
||||
- API clients (if needed)
|
||||
- Network utilities (frontend/fullstack only)
|
||||
- Auth helpers
|
||||
- Test data factories (language-idiomatic patterns)
|
||||
|
||||
**If `config.tea_use_pactjs_utils` is enabled and runtime is Node.js/TypeScript** (i.e., `{detected_stack}` is `frontend` or `fullstack`, or `{detected_stack}` is `backend` with Node.js/TypeScript runtime):
|
||||
|
||||
Create Node.js/TypeScript contract test samples per `pact-consumer-framework-setup.md`:
|
||||
|
||||
- **Consumer test**: Example using PactV4 `addInteraction()` builder + `createProviderState` + real consumer code with URL injection (`.pacttest.ts` extension)
|
||||
- **Support files**: Pact config factory (`pact-config.ts`), provider state factories (`provider-states.ts`), local consumer-helpers shim (`consumer-helpers.ts`)
|
||||
- **Vitest config**: Minimal `vitest.config.pact.ts` (do NOT copy settings from unit config)
|
||||
- **Shell scripts**: `env-setup.sh`, `publish-pact.sh`, `can-i-deploy.sh`, `record-deployment.sh` in `scripts/`
|
||||
- **CI workflow**: `contract-test-consumer.yml` with detect-breaking-change action
|
||||
- **package.json scripts**: `test:pact:consumer`, `publish:pact`, `can:i:deploy:consumer`, `record:consumer:deployment`
|
||||
- **.gitignore**: Add `/pacts/` and `pact-logs/`
|
||||
|
||||
---
|
||||
|
||||
### 6. Orchestration Notes for This Step
|
||||
|
||||
For this step, treat these work units as parallelizable when `resolvedMode` is `agent-team` or `subagent`:
|
||||
|
||||
- Worker A: directory + framework config + env setup (sections 1-3)
|
||||
- Worker B: fixtures + factories (section 4)
|
||||
- Worker C: sample tests + helpers (section 5)
|
||||
|
||||
In parallel-capable modes, runtime decides worker scheduling and concurrency.
|
||||
|
||||
If `resolvedMode` is `sequential`, execute sections 1→5 in order.
|
||||
|
||||
Regardless of mode, outputs must be identical in structure and quality.
|
||||
|
||||
### 7. Save Progress
|
||||
|
||||
**Save this step's accumulated work to `{outputFile}`.**
|
||||
|
||||
- **If `{outputFile}` does not exist** (first save), create it with YAML frontmatter:
|
||||
|
||||
```yaml
|
||||
---
|
||||
stepsCompleted: ['step-03-scaffold-framework']
|
||||
lastStep: 'step-03-scaffold-framework'
|
||||
lastSaved: '{date}'
|
||||
---
|
||||
```
|
||||
|
||||
Then write this step's output below the frontmatter.
|
||||
|
||||
- **If `{outputFile}` already exists**, update:
|
||||
- Add `'step-03-scaffold-framework'` to `stepsCompleted` array (only if not already present)
|
||||
- Set `lastStep: 'step-03-scaffold-framework'`
|
||||
- Set `lastSaved: '{date}'`
|
||||
- Append this step's output to the appropriate section of the document.
|
||||
|
||||
Load next step: `{nextStepFile}`
|
||||
|
||||
## 🚨 SYSTEM SUCCESS/FAILURE METRICS:
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- Step completed in full with required outputs
|
||||
|
||||
### ❌ SYSTEM FAILURE:
|
||||
|
||||
- Skipped sequence steps or missing outputs
|
||||
**Master Rule:** Skipping steps is FORBIDDEN.
|
||||
@@ -0,0 +1,105 @@
|
||||
---
|
||||
name: 'step-04-docs-and-scripts'
|
||||
description: 'Document setup and add package.json scripts'
|
||||
nextStepFile: './step-05-validate-and-summary.md'
|
||||
outputFile: '{test_dir}/README.md'
|
||||
progressFile: '{test_artifacts}/framework-setup-progress.md'
|
||||
---
|
||||
|
||||
# Step 4: Documentation & Scripts
|
||||
|
||||
## STEP GOAL
|
||||
|
||||
Create test documentation and add build/test scripts appropriate for `{detected_stack}`.
|
||||
|
||||
## MANDATORY EXECUTION RULES
|
||||
|
||||
- 📖 Read the entire step file before acting
|
||||
- ✅ Speak in `{communication_language}`
|
||||
|
||||
---
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
- 💾 Record outputs before proceeding
|
||||
- 📖 Load the next step only when instructed
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: config, loaded artifacts, and knowledge fragments
|
||||
- Focus: this step's goal only
|
||||
- Limits: do not execute future steps
|
||||
- Dependencies: prior steps' outputs (if any)
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly. Do not skip, reorder, or improvise.
|
||||
|
||||
## 1. tests/README.md
|
||||
|
||||
Create `{outputFile}` and include:
|
||||
|
||||
- Setup instructions
|
||||
- Running tests (local/headed/debug)
|
||||
- Architecture overview (fixtures, factories, helpers)
|
||||
- Best practices (selectors, isolation, cleanup)
|
||||
- CI integration notes
|
||||
- Knowledge base references
|
||||
|
||||
---
|
||||
|
||||
## 2. Build & Test Scripts
|
||||
|
||||
**If {detected_stack} is `frontend` or `fullstack`:**
|
||||
|
||||
Add to `package.json` at minimum:
|
||||
|
||||
- `test:e2e`: framework execution command (e.g., `npx playwright test`)
|
||||
|
||||
**If {detected_stack} is `backend` or `fullstack`:**
|
||||
|
||||
Add the idiomatic test commands for the detected framework:
|
||||
|
||||
- **Python (pytest)**: Add to `pyproject.toml` scripts or `Makefile`: `pytest`, `pytest --cov`, `pytest -m integration`
|
||||
- **Java (JUnit)**: Add to `build.gradle`/`pom.xml`: `./gradlew test`, `mvn test`, `mvn verify` (integration)
|
||||
- **Go**: Add to `Makefile`: `go test ./...`, `go test -race ./...`, `go test -cover ./...`
|
||||
- **C#/.NET**: Add to CI scripts or `Makefile`: `dotnet test`, `dotnet test --collect:"XPlat Code Coverage"`
|
||||
- **Ruby (RSpec)**: Add to `Gemfile` binstubs or `Makefile`: `bundle exec rspec`, `bundle exec rspec spec/integration`
|
||||
|
||||
---
|
||||
|
||||
### 3. Save Progress
|
||||
|
||||
**Save this step's accumulated work to `{progressFile}`.**
|
||||
|
||||
- **If `{progressFile}` does not exist** (first save), create it with YAML frontmatter:
|
||||
|
||||
```yaml
|
||||
---
|
||||
stepsCompleted: ['step-04-docs-and-scripts']
|
||||
lastStep: 'step-04-docs-and-scripts'
|
||||
lastSaved: '{date}'
|
||||
---
|
||||
```
|
||||
|
||||
Then write this step's output below the frontmatter.
|
||||
|
||||
- **If `{progressFile}` already exists**, update:
|
||||
- Add `'step-04-docs-and-scripts'` to `stepsCompleted` array (only if not already present)
|
||||
- Set `lastStep: 'step-04-docs-and-scripts'`
|
||||
- Set `lastSaved: '{date}'`
|
||||
- Append this step's output to the appropriate section of the document.
|
||||
|
||||
Load next step: `{nextStepFile}`
|
||||
|
||||
## 🚨 SYSTEM SUCCESS/FAILURE METRICS:
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- Step completed in full with required outputs
|
||||
|
||||
### ❌ SYSTEM FAILURE:
|
||||
|
||||
- Skipped sequence steps or missing outputs
|
||||
**Master Rule:** Skipping steps is FORBIDDEN.
|
||||
@@ -0,0 +1,93 @@
|
||||
---
|
||||
name: 'step-05-validate-and-summary'
|
||||
description: 'Validate against checklist and summarize'
|
||||
outputFile: '{test_artifacts}/framework-setup-progress.md'
|
||||
---
|
||||
|
||||
# Step 5: Validate & Summarize
|
||||
|
||||
## STEP GOAL
|
||||
|
||||
Validate framework setup and provide a completion summary.
|
||||
|
||||
## MANDATORY EXECUTION RULES
|
||||
|
||||
- 📖 Read the entire step file before acting
|
||||
- ✅ Speak in `{communication_language}`
|
||||
|
||||
---
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
- 💾 Record outputs before proceeding
|
||||
- 📖 Load the next step only when instructed
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: config, loaded artifacts, and knowledge fragments
|
||||
- Focus: this step's goal only
|
||||
- Limits: do not execute future steps
|
||||
- Dependencies: prior steps' outputs (if any)
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly. Do not skip, reorder, or improvise.
|
||||
|
||||
## 1. Validation
|
||||
|
||||
Validate against `checklist.md`:
|
||||
|
||||
- Preflight success
|
||||
- Directory structure created
|
||||
- Config correctness
|
||||
- Fixtures/factories created
|
||||
- Docs and scripts present
|
||||
|
||||
Fix any gaps before completion.
|
||||
|
||||
---
|
||||
|
||||
## 2. Completion Summary
|
||||
|
||||
Report:
|
||||
|
||||
- Framework selected
|
||||
- Artifacts created
|
||||
- Next steps (install deps, run tests)
|
||||
- Knowledge fragments applied
|
||||
|
||||
---
|
||||
|
||||
### 3. Save Progress
|
||||
|
||||
**Save this step's accumulated work to `{outputFile}`.**
|
||||
|
||||
- **If `{outputFile}` does not exist** (first save), create it with YAML frontmatter:
|
||||
|
||||
```yaml
|
||||
---
|
||||
stepsCompleted: ['step-05-validate-and-summary']
|
||||
lastStep: 'step-05-validate-and-summary'
|
||||
lastSaved: '{date}'
|
||||
---
|
||||
```
|
||||
|
||||
Then write this step's output below the frontmatter.
|
||||
|
||||
- **If `{outputFile}` already exists**, update:
|
||||
- Add `'step-05-validate-and-summary'` to `stepsCompleted` array (only if not already present)
|
||||
- Set `lastStep: 'step-05-validate-and-summary'`
|
||||
- Set `lastSaved: '{date}'`
|
||||
- Append this step's output to the appropriate section of the document.
|
||||
|
||||
## 🚨 SYSTEM SUCCESS/FAILURE METRICS:
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- Step completed in full with required outputs
|
||||
|
||||
### ❌ SYSTEM FAILURE:
|
||||
|
||||
- Skipped sequence steps or missing outputs
|
||||
**Master Rule:** Skipping steps is FORBIDDEN.
|
||||
@@ -0,0 +1,65 @@
|
||||
---
|
||||
name: 'step-01-assess'
|
||||
description: 'Load an existing output for editing'
|
||||
nextStepFile: './step-02-apply-edit.md'
|
||||
---
|
||||
|
||||
# Step 1: Assess Edit Target
|
||||
|
||||
## STEP GOAL:
|
||||
|
||||
Identify which output should be edited and load it.
|
||||
|
||||
## MANDATORY EXECUTION RULES (READ FIRST):
|
||||
|
||||
### Universal Rules:
|
||||
|
||||
- 📖 Read the complete step file before taking any action
|
||||
- ✅ Speak in `{communication_language}`
|
||||
|
||||
### Role Reinforcement:
|
||||
|
||||
- ✅ You are the Master Test Architect
|
||||
|
||||
### Step-Specific Rules:
|
||||
|
||||
- 🎯 Ask the user which output file to edit
|
||||
- 🚫 Do not edit until target is confirmed
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: existing outputs
|
||||
- Focus: select edit target
|
||||
- Limits: no edits yet
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly.
|
||||
|
||||
### 1. Identify Target
|
||||
|
||||
Ask the user to provide the output file path or select from known outputs.
|
||||
|
||||
### 2. Load Target
|
||||
|
||||
Read the provided output file in full.
|
||||
|
||||
### 3. Confirm
|
||||
|
||||
Confirm the target and proceed to edit.
|
||||
|
||||
Load next step: `{nextStepFile}`
|
||||
|
||||
## 🚨 SYSTEM SUCCESS/FAILURE METRICS:
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- Target identified and loaded
|
||||
|
||||
### ❌ SYSTEM FAILURE:
|
||||
|
||||
- Proceeding without a confirmed target
|
||||
@@ -0,0 +1,60 @@
|
||||
---
|
||||
name: 'step-02-apply-edit'
|
||||
description: 'Apply edits to the selected output'
|
||||
---
|
||||
|
||||
# Step 2: Apply Edits
|
||||
|
||||
## STEP GOAL:
|
||||
|
||||
Apply the requested edits to the selected output and confirm changes.
|
||||
|
||||
## MANDATORY EXECUTION RULES (READ FIRST):
|
||||
|
||||
### Universal Rules:
|
||||
|
||||
- 📖 Read the complete step file before taking any action
|
||||
- ✅ Speak in `{communication_language}`
|
||||
|
||||
### Role Reinforcement:
|
||||
|
||||
- ✅ You are the Master Test Architect
|
||||
|
||||
### Step-Specific Rules:
|
||||
|
||||
- 🎯 Only apply edits explicitly requested by the user
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: selected output and user changes
|
||||
- Focus: apply edits only
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly.
|
||||
|
||||
### 1. Confirm Requested Changes
|
||||
|
||||
Restate what will be changed and confirm.
|
||||
|
||||
### 2. Apply Changes
|
||||
|
||||
Update the output file accordingly.
|
||||
|
||||
### 3. Report
|
||||
|
||||
Summarize the edits applied.
|
||||
|
||||
## 🚨 SYSTEM SUCCESS/FAILURE METRICS:
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- Changes applied and confirmed
|
||||
|
||||
### ❌ SYSTEM FAILURE:
|
||||
|
||||
- Unconfirmed edits or missing update
|
||||
@@ -0,0 +1,67 @@
|
||||
---
|
||||
name: 'step-01-validate'
|
||||
description: 'Validate workflow outputs against checklist'
|
||||
outputFile: '{test_artifacts}/framework-validation-report.md'
|
||||
validationChecklist: '../checklist.md'
|
||||
---
|
||||
|
||||
# Step 1: Validate Outputs
|
||||
|
||||
## STEP GOAL:
|
||||
|
||||
Validate outputs using the workflow checklist and record findings.
|
||||
|
||||
## MANDATORY EXECUTION RULES (READ FIRST):
|
||||
|
||||
### Universal Rules:
|
||||
|
||||
- 📖 Read the complete step file before taking any action
|
||||
- ✅ Speak in `{communication_language}`
|
||||
|
||||
### Role Reinforcement:
|
||||
|
||||
- ✅ You are the Master Test Architect
|
||||
|
||||
### Step-Specific Rules:
|
||||
|
||||
- 🎯 Validate against `{validationChecklist}`
|
||||
- 🚫 Do not skip checks
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
- 💾 Write findings to `{outputFile}`
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: workflow outputs and checklist
|
||||
- Focus: validation only
|
||||
- Limits: do not modify outputs in this step
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly.
|
||||
|
||||
### 1. Load Checklist
|
||||
|
||||
Read `{validationChecklist}` and list all criteria.
|
||||
|
||||
### 2. Validate Outputs
|
||||
|
||||
Evaluate outputs against each checklist item.
|
||||
|
||||
### 3. Write Report
|
||||
|
||||
Write a validation report to `{outputFile}` with PASS/WARN/FAIL per section.
|
||||
|
||||
## 🚨 SYSTEM SUCCESS/FAILURE METRICS:
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- Validation report written
|
||||
- All checklist items evaluated
|
||||
|
||||
### ❌ SYSTEM FAILURE:
|
||||
|
||||
- Skipped checklist items
|
||||
- No report produced
|
||||
@@ -0,0 +1,73 @@
|
||||
---
|
||||
validationDate: 2026-01-27
|
||||
workflowName: testarch-framework
|
||||
workflowPath: {project-root}/src/workflows/testarch/framework
|
||||
validationStatus: COMPLETE
|
||||
completionDate: 2026-01-27 10:03:10
|
||||
---
|
||||
|
||||
# Validation Report: testarch-framework
|
||||
|
||||
**Validation Started:** 2026-01-27 09:50:21
|
||||
**Validator:** BMAD Workflow Validation System (Codex)
|
||||
**Standards Version:** BMAD Workflow Standards
|
||||
|
||||
## File Structure & Size
|
||||
|
||||
- workflow.md present: YES
|
||||
- instructions.md present: YES
|
||||
- workflow.yaml present: YES
|
||||
- step files found: 8
|
||||
|
||||
**Step File Sizes:**
|
||||
|
||||
- steps-c/step-01-preflight.md: 69 lines [GOOD]
|
||||
- steps-c/step-02-select-framework.md: 66 lines [GOOD]
|
||||
- steps-c/step-03-scaffold-framework.md: 107 lines [GOOD]
|
||||
- steps-c/step-04-docs-and-scripts.md: 63 lines [GOOD]
|
||||
- steps-c/step-05-validate-and-summary.md: 61 lines [GOOD]
|
||||
- steps-e/step-01-assess.md: 51 lines [GOOD]
|
||||
- steps-e/step-02-apply-edit.md: 46 lines [GOOD]
|
||||
- steps-v/step-01-validate.md: 53 lines [GOOD]
|
||||
- workflow-plan.md present: YES
|
||||
|
||||
## Frontmatter Validation
|
||||
|
||||
- No frontmatter violations found
|
||||
|
||||
## Critical Path Violations
|
||||
|
||||
- No {project-root} hardcoded paths detected in body
|
||||
- No dead relative links detected
|
||||
|
||||
## Menu Handling Validation
|
||||
|
||||
- No menu structures detected (linear step flow) [N/A]
|
||||
|
||||
## Step Type Validation
|
||||
|
||||
- Last step steps-v/step-01-validate.md has no nextStepFile (final step OK)
|
||||
- Step type validation assumes linear sequence (no branching/menu). Workflow-plan.md present for reference. [INFO]
|
||||
|
||||
## Output Format Validation
|
||||
|
||||
- No templates found in workflow root
|
||||
- Steps with outputFile in frontmatter:
|
||||
- steps-c/step-04-docs-and-scripts.md
|
||||
- steps-v/step-01-validate.md
|
||||
|
||||
## Validation Design Check
|
||||
|
||||
- checklist.md present: YES
|
||||
- Validation steps folder (steps-v) present: YES
|
||||
|
||||
## Instruction Style Check
|
||||
|
||||
- All steps include STEP GOAL, MANDATORY EXECUTION RULES, EXECUTION PROTOCOLS, CONTEXT BOUNDARIES, and SUCCESS/FAILURE metrics
|
||||
|
||||
## Summary
|
||||
|
||||
- Validation completed: 2026-01-27 10:03:10
|
||||
- Critical issues: 0
|
||||
- Warnings: 0 (informational notes only)
|
||||
- Readiness: READY (manual review optional)
|
||||
@@ -0,0 +1,116 @@
|
||||
---
|
||||
validationDate: 2026-01-27
|
||||
workflowName: testarch-framework
|
||||
workflowPath: {project-root}/src/workflows/testarch/framework
|
||||
validationStatus: COMPLETE
|
||||
completionDate: 2026-01-27 10:24:01
|
||||
---
|
||||
|
||||
# Validation Report: testarch-framework
|
||||
|
||||
**Validation Started:** 2026-01-27 10:24:01
|
||||
**Validator:** BMAD Workflow Validation System (Codex)
|
||||
**Standards Version:** BMAD Workflow Standards
|
||||
|
||||
## File Structure & Size
|
||||
|
||||
- workflow.md present: YES
|
||||
- instructions.md present: YES
|
||||
- workflow.yaml present: YES
|
||||
- step files found: 8
|
||||
|
||||
**Step File Sizes:**
|
||||
|
||||
- steps-c/step-01-preflight.md: 68 lines [GOOD]
|
||||
- steps-c/step-02-select-framework.md: 65 lines [GOOD]
|
||||
- steps-c/step-03-scaffold-framework.md: 106 lines [GOOD]
|
||||
- steps-c/step-04-docs-and-scripts.md: 62 lines [GOOD]
|
||||
- steps-c/step-05-validate-and-summary.md: 60 lines [GOOD]
|
||||
- steps-e/step-01-assess.md: 50 lines [GOOD]
|
||||
- steps-e/step-02-apply-edit.md: 45 lines [GOOD]
|
||||
- steps-v/step-01-validate.md: 52 lines [GOOD]
|
||||
- workflow-plan.md present: YES
|
||||
|
||||
## Frontmatter Validation
|
||||
|
||||
- No frontmatter violations found
|
||||
|
||||
## Critical Path Violations
|
||||
|
||||
### Config Variables (Exceptions)
|
||||
|
||||
Standard BMAD config variables treated as valid exceptions: bmb_creations_output_folder, communication_language, document_output_language, output_folder, planning_artifacts, project-root, project_name, test_artifacts, user_name
|
||||
|
||||
- No {project-root} hardcoded paths detected in body
|
||||
|
||||
- No dead relative links detected
|
||||
|
||||
- No module path assumptions detected
|
||||
|
||||
**Status:** ✅ PASS - No critical violations
|
||||
|
||||
## Menu Handling Validation
|
||||
|
||||
- No menu structures detected (linear step flow) [N/A]
|
||||
|
||||
## Step Type Validation
|
||||
|
||||
- steps-c/step-01-preflight.md: Init [PASS]
|
||||
- steps-c/step-02-select-framework.md: Middle [PASS]
|
||||
- steps-c/step-03-scaffold-framework.md: Middle [PASS]
|
||||
- steps-c/step-04-docs-and-scripts.md: Middle [PASS]
|
||||
- steps-c/step-05-validate-and-summary.md: Final [PASS]
|
||||
- Step type validation assumes linear sequence (no branching/menu). Workflow-plan.md present for reference. [INFO]
|
||||
|
||||
## Output Format Validation
|
||||
|
||||
- Templates present: NONE
|
||||
- Steps with outputFile in frontmatter:
|
||||
- steps-c/step-04-docs-and-scripts.md
|
||||
- steps-v/step-01-validate.md
|
||||
- checklist.md present: YES
|
||||
|
||||
## Validation Design Check
|
||||
|
||||
- Validation steps folder (steps-v) present: YES
|
||||
- Validation step(s) present: step-01-validate.md
|
||||
- Validation steps reference checklist data and auto-proceed
|
||||
|
||||
## Instruction Style Check
|
||||
|
||||
- Instruction style: Prescriptive (appropriate for TEA quality/compliance workflows)
|
||||
- Steps emphasize mandatory sequence, explicit success/failure metrics, and risk-based guidance
|
||||
|
||||
## Collaborative Experience Check
|
||||
|
||||
- Overall facilitation quality: GOOD
|
||||
- Steps use progressive prompts and clear role reinforcement; no laundry-list interrogation detected
|
||||
- Flow progression is clear and aligned to workflow goals
|
||||
|
||||
## Subagent Optimization Opportunities
|
||||
|
||||
- No high-priority subagent optimizations identified; workflow already uses step-file architecture
|
||||
- Pattern 1 (grep/regex): N/A for most steps
|
||||
- Pattern 2 (per-file analysis): already aligned to validation structure
|
||||
- Pattern 3 (data ops): minimal data file loads
|
||||
- Pattern 4 (parallel): optional for validation only
|
||||
|
||||
## Cohesive Review
|
||||
|
||||
- Overall assessment: GOOD
|
||||
- Flow is linear, goals are clear, and outputs map to TEA artifacts
|
||||
- Voice and tone consistent with Test Architect persona
|
||||
- Recommendation: READY (minor refinements optional)
|
||||
|
||||
## Plan Quality Validation
|
||||
|
||||
- Plan file present: workflow-plan.md
|
||||
- Planned steps found: 8 (all implemented)
|
||||
- Plan implementation status: Fully Implemented
|
||||
|
||||
## Summary
|
||||
|
||||
- Validation completed: 2026-01-27 10:24:01
|
||||
- Critical issues: 0
|
||||
- Warnings: 0 (informational notes only)
|
||||
- Readiness: READY (manual review optional)
|
||||
22
_bmad/tea/workflows/testarch/framework/workflow-plan.md
Normal file
22
_bmad/tea/workflows/testarch/framework/workflow-plan.md
Normal file
@@ -0,0 +1,22 @@
|
||||
# Workflow Plan: testarch-framework
|
||||
|
||||
## Create Mode (steps-c)
|
||||
- step-01-preflight.md
|
||||
|
||||
- step-02-select-framework.md
|
||||
- step-03-scaffold-framework.md
|
||||
- step-04-docs-and-scripts.md
|
||||
- step-05-validate-and-summary.md
|
||||
|
||||
## Validate Mode (steps-v)
|
||||
- step-01-validate.md
|
||||
|
||||
## Edit Mode (steps-e)
|
||||
- step-01-assess.md
|
||||
- step-02-apply-edit.md
|
||||
|
||||
## Outputs
|
||||
- Framework config (playwright.config.ts or cypress.config.ts)
|
||||
|
||||
- {project-root}/tests/README.md
|
||||
- Test support scaffolding under {project-root}/tests
|
||||
41
_bmad/tea/workflows/testarch/framework/workflow.md
Normal file
41
_bmad/tea/workflows/testarch/framework/workflow.md
Normal file
@@ -0,0 +1,41 @@
|
||||
---
|
||||
name: testarch-framework
|
||||
description: Initialize test framework with Playwright or Cypress. Use when user says 'lets setup test framework' or 'I want to initialize testing framework'
|
||||
web_bundle: true
|
||||
---
|
||||
|
||||
# Test Framework Setup
|
||||
|
||||
**Goal:** Initialize production-ready test framework architecture (Playwright or Cypress) with fixtures, helpers, and configuration
|
||||
|
||||
**Role:** You are the Master Test Architect.
|
||||
|
||||
---
|
||||
|
||||
## WORKFLOW ARCHITECTURE
|
||||
|
||||
This workflow uses **tri-modal step-file architecture**:
|
||||
|
||||
- **Create mode (steps-c/)**: primary execution flow
|
||||
- **Validate mode (steps-v/)**: validation against checklist
|
||||
- **Edit mode (steps-e/)**: revise existing outputs
|
||||
|
||||
---
|
||||
|
||||
## INITIALIZATION SEQUENCE
|
||||
|
||||
### 1. Mode Determination
|
||||
|
||||
"Welcome to the workflow. What would you like to do?"
|
||||
|
||||
- **[C] Create** — Run the workflow
|
||||
- **[R] Resume** — Resume an interrupted workflow
|
||||
- **[V] Validate** — Validate existing outputs
|
||||
- **[E] Edit** — Edit existing outputs
|
||||
|
||||
### 2. Route to First Step
|
||||
|
||||
- **If C:** Load `steps-c/step-01-preflight.md`
|
||||
- **If R:** Load `steps-c/step-01b-resume.md`
|
||||
- **If V:** Load `steps-v/step-01-validate.md`
|
||||
- **If E:** Load `steps-e/step-01-assess.md`
|
||||
48
_bmad/tea/workflows/testarch/framework/workflow.yaml
Normal file
48
_bmad/tea/workflows/testarch/framework/workflow.yaml
Normal file
@@ -0,0 +1,48 @@
|
||||
# Test Architect workflow: framework
|
||||
name: testarch-framework
|
||||
# prettier-ignore
|
||||
description: 'Initialize test framework with Playwright or Cypress. Use when the user says "lets setup test framework" or "I want to initialize testing framework"'
|
||||
|
||||
# Critical variables from config
|
||||
config_source: "{project-root}/_bmad/tea/config.yaml"
|
||||
output_folder: "{config_source}:output_folder"
|
||||
test_artifacts: "{config_source}:test_artifacts"
|
||||
user_name: "{config_source}:user_name"
|
||||
communication_language: "{config_source}:communication_language"
|
||||
document_output_language: "{config_source}:document_output_language"
|
||||
date: system-generated
|
||||
|
||||
# Workflow components
|
||||
installed_path: "{project-root}/_bmad/tea/workflows/testarch/framework"
|
||||
instructions: "{installed_path}/instructions.md"
|
||||
validation: "{installed_path}/checklist.md"
|
||||
|
||||
# Variables and inputs
|
||||
variables:
|
||||
test_dir: "{project-root}/tests" # Root test directory
|
||||
use_typescript: true # Prefer TypeScript configuration
|
||||
framework_preference: "auto" # auto, playwright, cypress - user can override auto-detection
|
||||
project_size: "auto" # auto, small, large - influences framework recommendation
|
||||
|
||||
# Output configuration
|
||||
default_output_file: "{test_dir}/README.md" # Main deliverable is test setup README
|
||||
|
||||
# Required tools
|
||||
required_tools:
|
||||
- read_file # Read package.json, existing configs
|
||||
- write_file # Create config files, helpers, fixtures, tests
|
||||
- create_directory # Create test directory structure
|
||||
- list_files # Check for existing framework
|
||||
- search_repo # Find architecture docs
|
||||
|
||||
tags:
|
||||
- qa
|
||||
- setup
|
||||
- test-architect
|
||||
- framework
|
||||
- initialization
|
||||
|
||||
execution_hints:
|
||||
interactive: false # Minimize prompts; auto-detect when possible
|
||||
autonomous: true # Proceed without user input unless blocked
|
||||
iterative: true
|
||||
407
_bmad/tea/workflows/testarch/nfr-assess/checklist.md
Normal file
407
_bmad/tea/workflows/testarch/nfr-assess/checklist.md
Normal file
@@ -0,0 +1,407 @@
|
||||
# Non-Functional Requirements Assessment - Validation Checklist
|
||||
|
||||
**Workflow:** `testarch-nfr`
|
||||
**Purpose:** Ensure comprehensive and evidence-based NFR assessment with actionable recommendations
|
||||
|
||||
---
|
||||
|
||||
Note: `nfr-assess` evaluates existing evidence; it does not run tests or CI workflows.
|
||||
|
||||
## Prerequisites Validation
|
||||
|
||||
- [ ] Implementation is deployed and accessible for evaluation
|
||||
- [ ] Evidence sources are available (test results, metrics, logs, CI results)
|
||||
- [ ] NFR categories are determined (performance, security, reliability, maintainability, custom)
|
||||
- [ ] Evidence directories exist and are accessible (`test_results_dir`, `metrics_dir`, `logs_dir`)
|
||||
- [ ] Knowledge base is loaded (nfr-criteria, ci-burn-in, test-quality)
|
||||
|
||||
---
|
||||
|
||||
## Context Loading
|
||||
|
||||
- [ ] Tech-spec.md loaded successfully (if available)
|
||||
- [ ] PRD.md loaded (if available)
|
||||
- [ ] Story file loaded (if applicable)
|
||||
- [ ] Relevant knowledge fragments loaded from `tea-index.csv`:
|
||||
- [ ] `nfr-criteria.md`
|
||||
- [ ] `ci-burn-in.md`
|
||||
- [ ] `test-quality.md`
|
||||
- [ ] `playwright-config.md` (if using Playwright)
|
||||
|
||||
---
|
||||
|
||||
## NFR Categories and Thresholds
|
||||
|
||||
### Performance
|
||||
|
||||
- [ ] Response time threshold defined or marked as UNKNOWN
|
||||
- [ ] Throughput threshold defined or marked as UNKNOWN
|
||||
- [ ] Resource usage thresholds defined or marked as UNKNOWN
|
||||
- [ ] Scalability requirements defined or marked as UNKNOWN
|
||||
|
||||
### Security
|
||||
|
||||
- [ ] Authentication requirements defined or marked as UNKNOWN
|
||||
- [ ] Authorization requirements defined or marked as UNKNOWN
|
||||
- [ ] Data protection requirements defined or marked as UNKNOWN
|
||||
- [ ] Vulnerability management thresholds defined or marked as UNKNOWN
|
||||
- [ ] Compliance requirements identified (GDPR, HIPAA, PCI-DSS, etc.)
|
||||
|
||||
### Reliability
|
||||
|
||||
- [ ] Availability (uptime) threshold defined or marked as UNKNOWN
|
||||
- [ ] Error rate threshold defined or marked as UNKNOWN
|
||||
- [ ] MTTR (Mean Time To Recovery) threshold defined or marked as UNKNOWN
|
||||
- [ ] Fault tolerance requirements defined or marked as UNKNOWN
|
||||
- [ ] Disaster recovery requirements defined (RTO, RPO) or marked as UNKNOWN
|
||||
|
||||
### Maintainability
|
||||
|
||||
- [ ] Test coverage threshold defined or marked as UNKNOWN
|
||||
- [ ] Code quality threshold defined or marked as UNKNOWN
|
||||
- [ ] Technical debt threshold defined or marked as UNKNOWN
|
||||
- [ ] Documentation completeness threshold defined or marked as UNKNOWN
|
||||
|
||||
### Custom NFR Categories (if applicable)
|
||||
|
||||
- [ ] Custom NFR category 1: Thresholds defined or marked as UNKNOWN
|
||||
- [ ] Custom NFR category 2: Thresholds defined or marked as UNKNOWN
|
||||
- [ ] Custom NFR category 3: Thresholds defined or marked as UNKNOWN
|
||||
|
||||
---
|
||||
|
||||
## Evidence Gathering
|
||||
|
||||
### Performance Evidence
|
||||
|
||||
- [ ] Load test results collected (JMeter, k6, Gatling, etc.)
|
||||
- [ ] Application metrics collected (response times, throughput, resource usage)
|
||||
- [ ] APM data collected (New Relic, Datadog, Dynatrace, etc.)
|
||||
- [ ] Lighthouse reports collected (if web app)
|
||||
- [ ] Playwright performance traces collected (if applicable)
|
||||
|
||||
### Security Evidence
|
||||
|
||||
- [ ] SAST results collected (SonarQube, Checkmarx, Veracode, etc.)
|
||||
- [ ] DAST results collected (OWASP ZAP, Burp Suite, etc.)
|
||||
- [ ] Dependency scanning results collected (Snyk, Dependabot, npm audit)
|
||||
- [ ] Penetration test reports collected (if available)
|
||||
- [ ] Security audit logs collected
|
||||
- [ ] Compliance audit results collected (if applicable)
|
||||
|
||||
### Reliability Evidence
|
||||
|
||||
- [ ] Uptime monitoring data collected (Pingdom, UptimeRobot, StatusCake)
|
||||
- [ ] Error logs collected
|
||||
- [ ] Error rate metrics collected
|
||||
- [ ] CI burn-in results collected (stability over time)
|
||||
- [ ] Chaos engineering test results collected (if available)
|
||||
- [ ] Failover/recovery test results collected (if available)
|
||||
- [ ] Incident reports and postmortems collected (if applicable)
|
||||
|
||||
### Maintainability Evidence
|
||||
|
||||
- [ ] Code coverage reports collected (Istanbul, NYC, c8, JaCoCo)
|
||||
- [ ] Static analysis results collected (ESLint, SonarQube, CodeClimate)
|
||||
- [ ] Technical debt metrics collected
|
||||
- [ ] Documentation audit results collected
|
||||
- [ ] Test review report collected (from test-review workflow, if available)
|
||||
- [ ] Git metrics collected (code churn, commit frequency, etc.)
|
||||
|
||||
---
|
||||
|
||||
## NFR Assessment with Deterministic Rules
|
||||
|
||||
### Performance Assessment
|
||||
|
||||
- [ ] Response time assessed against threshold
|
||||
- [ ] Throughput assessed against threshold
|
||||
- [ ] Resource usage assessed against threshold
|
||||
- [ ] Scalability assessed against requirements
|
||||
- [ ] Status classified (PASS/CONCERNS/FAIL) with justification
|
||||
- [ ] Evidence source documented (file path, metric name)
|
||||
|
||||
### Security Assessment
|
||||
|
||||
- [ ] Authentication strength assessed against requirements
|
||||
- [ ] Authorization controls assessed against requirements
|
||||
- [ ] Data protection assessed against requirements
|
||||
- [ ] Vulnerability management assessed against thresholds
|
||||
- [ ] Compliance assessed against requirements
|
||||
- [ ] Status classified (PASS/CONCERNS/FAIL) with justification
|
||||
- [ ] Evidence source documented (file path, scan result)
|
||||
|
||||
### Reliability Assessment
|
||||
|
||||
- [ ] Availability (uptime) assessed against threshold
|
||||
- [ ] Error rate assessed against threshold
|
||||
- [ ] MTTR assessed against threshold
|
||||
- [ ] Fault tolerance assessed against requirements
|
||||
- [ ] Disaster recovery assessed against requirements (RTO, RPO)
|
||||
- [ ] CI burn-in assessed (stability over time)
|
||||
- [ ] Status classified (PASS/CONCERNS/FAIL) with justification
|
||||
- [ ] Evidence source documented (file path, monitoring data)
|
||||
|
||||
### Maintainability Assessment
|
||||
|
||||
- [ ] Test coverage assessed against threshold
|
||||
- [ ] Code quality assessed against threshold
|
||||
- [ ] Technical debt assessed against threshold
|
||||
- [ ] Documentation completeness assessed against threshold
|
||||
- [ ] Test quality assessed (from test-review, if available)
|
||||
- [ ] Status classified (PASS/CONCERNS/FAIL) with justification
|
||||
- [ ] Evidence source documented (file path, coverage report)
|
||||
|
||||
### Custom NFR Assessment (if applicable)
|
||||
|
||||
- [ ] Custom NFR 1 assessed against threshold with justification
|
||||
- [ ] Custom NFR 2 assessed against threshold with justification
|
||||
- [ ] Custom NFR 3 assessed against threshold with justification
|
||||
|
||||
---
|
||||
|
||||
## Status Classification Validation
|
||||
|
||||
### PASS Criteria Verified
|
||||
|
||||
- [ ] Evidence exists for PASS status
|
||||
- [ ] Evidence meets or exceeds threshold
|
||||
- [ ] No concerns flagged in evidence
|
||||
- [ ] Quality is acceptable
|
||||
|
||||
### CONCERNS Criteria Verified
|
||||
|
||||
- [ ] Threshold is UNKNOWN (documented) OR
|
||||
- [ ] Evidence is MISSING or INCOMPLETE (documented) OR
|
||||
- [ ] Evidence is close to threshold (within 10%, documented) OR
|
||||
- [ ] Evidence shows intermittent issues (documented)
|
||||
|
||||
### FAIL Criteria Verified
|
||||
|
||||
- [ ] Evidence exists BUT does not meet threshold (documented) OR
|
||||
- [ ] Critical evidence is MISSING (documented) OR
|
||||
- [ ] Evidence shows consistent failures (documented) OR
|
||||
- [ ] Quality is unacceptable (documented)
|
||||
|
||||
### No Threshold Guessing
|
||||
|
||||
- [ ] All thresholds are either defined or marked as UNKNOWN
|
||||
- [ ] No thresholds were guessed or inferred
|
||||
- [ ] All UNKNOWN thresholds result in CONCERNS status
|
||||
|
||||
---
|
||||
|
||||
## Quick Wins and Recommended Actions
|
||||
|
||||
### Quick Wins Identified
|
||||
|
||||
- [ ] Low-effort, high-impact improvements identified for CONCERNS/FAIL
|
||||
- [ ] Configuration changes (no code changes) identified
|
||||
- [ ] Optimization opportunities identified (caching, indexing, compression)
|
||||
- [ ] Monitoring additions identified (detect issues before failures)
|
||||
|
||||
### Recommended Actions
|
||||
|
||||
- [ ] Specific remediation steps provided (not generic advice)
|
||||
- [ ] Priority assigned (CRITICAL, HIGH, MEDIUM, LOW)
|
||||
- [ ] Estimated effort provided (hours, days)
|
||||
- [ ] Owner suggestions provided (dev, ops, security)
|
||||
|
||||
### Monitoring Hooks
|
||||
|
||||
- [ ] Performance monitoring suggested (APM, synthetic monitoring)
|
||||
- [ ] Error tracking suggested (Sentry, Rollbar, error logs)
|
||||
- [ ] Security monitoring suggested (intrusion detection, audit logs)
|
||||
- [ ] Alerting thresholds suggested (notify before breach)
|
||||
|
||||
### Fail-Fast Mechanisms
|
||||
|
||||
- [ ] Circuit breakers suggested for reliability
|
||||
- [ ] Rate limiting suggested for performance
|
||||
- [ ] Validation gates suggested for security
|
||||
- [ ] Smoke tests suggested for maintainability
|
||||
|
||||
---
|
||||
|
||||
## Deliverables Generated
|
||||
|
||||
### NFR Assessment Report
|
||||
|
||||
- [ ] File created at `{test_artifacts}/nfr-assessment.md`
|
||||
- [ ] Template from `nfr-report-template.md` used
|
||||
- [ ] Executive summary included (overall status, critical issues)
|
||||
- [ ] Assessment by category included (performance, security, reliability, maintainability)
|
||||
- [ ] Evidence for each NFR documented
|
||||
- [ ] Status classifications documented (PASS/CONCERNS/FAIL)
|
||||
- [ ] Findings summary included (PASS count, CONCERNS count, FAIL count)
|
||||
- [ ] Quick wins section included
|
||||
- [ ] Recommended actions section included
|
||||
- [ ] Evidence gaps checklist included
|
||||
|
||||
### Gate YAML Snippet (if enabled)
|
||||
|
||||
- [ ] YAML snippet generated
|
||||
- [ ] Date included
|
||||
- [ ] Categories status included (performance, security, reliability, maintainability)
|
||||
- [ ] Overall status included (PASS/CONCERNS/FAIL)
|
||||
- [ ] Issue counts included (critical, high, medium, concerns)
|
||||
- [ ] Blockers flag included (true/false)
|
||||
- [ ] Recommendations included
|
||||
|
||||
### Evidence Checklist (if enabled)
|
||||
|
||||
- [ ] All NFRs with MISSING or INCOMPLETE evidence listed
|
||||
- [ ] Owners assigned for evidence collection
|
||||
- [ ] Suggested evidence sources provided
|
||||
- [ ] Deadlines set for evidence collection
|
||||
|
||||
### Updated Story File (if enabled and requested)
|
||||
|
||||
- [ ] "NFR Assessment" section added to story markdown
|
||||
- [ ] Link to NFR assessment report included
|
||||
- [ ] Overall status and critical issues included
|
||||
- [ ] Gate status included
|
||||
|
||||
---
|
||||
|
||||
## Quality Assurance
|
||||
|
||||
### Accuracy Checks
|
||||
|
||||
- [ ] All NFR categories assessed (none skipped)
|
||||
- [ ] All thresholds documented (defined or UNKNOWN)
|
||||
- [ ] All evidence sources documented (file paths, metric names)
|
||||
- [ ] Status classifications are deterministic and consistent
|
||||
- [ ] No false positives (status correctly assigned)
|
||||
- [ ] No false negatives (all issues identified)
|
||||
|
||||
### Completeness Checks
|
||||
|
||||
- [ ] All NFR categories covered (performance, security, reliability, maintainability, custom)
|
||||
- [ ] All evidence sources checked (test results, metrics, logs, CI results)
|
||||
- [ ] All status types used appropriately (PASS, CONCERNS, FAIL)
|
||||
- [ ] All NFRs with CONCERNS/FAIL have recommendations
|
||||
- [ ] All evidence gaps have owners and deadlines
|
||||
|
||||
### Actionability Checks
|
||||
|
||||
- [ ] Recommendations are specific (not generic)
|
||||
- [ ] Remediation steps are clear and actionable
|
||||
- [ ] Priorities are assigned (CRITICAL, HIGH, MEDIUM, LOW)
|
||||
- [ ] Effort estimates are provided (hours, days)
|
||||
- [ ] Owners are suggested (dev, ops, security)
|
||||
|
||||
---
|
||||
|
||||
## Integration with BMad Artifacts
|
||||
|
||||
### With tech-spec.md
|
||||
|
||||
- [ ] Tech spec loaded for NFR requirements and thresholds
|
||||
- [ ] Performance targets extracted
|
||||
- [ ] Security requirements extracted
|
||||
- [ ] Reliability SLAs extracted
|
||||
- [ ] Architectural decisions considered
|
||||
|
||||
### With test-design.md
|
||||
|
||||
- [ ] Test design loaded for NFR test plan
|
||||
- [ ] Test priorities referenced (P0/P1/P2/P3)
|
||||
- [ ] Assessment aligned with planned NFR validation
|
||||
|
||||
### With PRD.md
|
||||
|
||||
- [ ] PRD loaded for product-level NFR context
|
||||
- [ ] User experience goals considered
|
||||
- [ ] Unstated requirements checked
|
||||
- [ ] Product-level SLAs referenced
|
||||
|
||||
---
|
||||
|
||||
## Quality Gates Validation
|
||||
|
||||
### Release Blocker (FAIL)
|
||||
|
||||
- [ ] Critical NFR status checked (security, reliability)
|
||||
- [ ] Performance failures assessed for user impact
|
||||
- [ ] Release blocker flagged if critical NFR has FAIL status
|
||||
|
||||
### PR Blocker (HIGH CONCERNS)
|
||||
|
||||
- [ ] High-priority NFR status checked
|
||||
- [ ] Multiple CONCERNS assessed
|
||||
- [ ] PR blocker flagged if HIGH priority issues exist
|
||||
|
||||
### Warning (CONCERNS)
|
||||
|
||||
- [ ] Any NFR with CONCERNS status flagged
|
||||
- [ ] Missing or incomplete evidence documented
|
||||
- [ ] Warning issued to address before next release
|
||||
|
||||
### Pass (PASS)
|
||||
|
||||
- [ ] All NFRs have PASS status
|
||||
- [ ] No blockers or concerns exist
|
||||
- [ ] Ready for release confirmed
|
||||
|
||||
---
|
||||
|
||||
## Non-Prescriptive Validation
|
||||
|
||||
- [ ] NFR categories adapted to team needs
|
||||
- [ ] Thresholds appropriate for project context
|
||||
- [ ] Assessment criteria customized as needed
|
||||
- [ ] Teams can extend with custom NFR categories
|
||||
- [ ] Integration with external tools supported (New Relic, Datadog, SonarQube, JIRA)
|
||||
|
||||
---
|
||||
|
||||
## Documentation and Communication
|
||||
|
||||
- [ ] NFR assessment report is readable and well-formatted
|
||||
- [ ] Tables render correctly in markdown
|
||||
- [ ] Code blocks have proper syntax highlighting
|
||||
- [ ] Links are valid and accessible
|
||||
- [ ] Recommendations are clear and prioritized
|
||||
- [ ] Overall status is prominent and unambiguous
|
||||
- [ ] Executive summary provides quick understanding
|
||||
|
||||
---
|
||||
|
||||
## Final Validation
|
||||
|
||||
- [ ] All prerequisites met
|
||||
- [ ] All NFR categories assessed with evidence (or gaps documented)
|
||||
- [ ] No thresholds were guessed (all defined or UNKNOWN)
|
||||
- [ ] Status classifications are deterministic and justified
|
||||
- [ ] Quick wins identified for all CONCERNS/FAIL
|
||||
- [ ] Recommended actions are specific and actionable
|
||||
- [ ] Evidence gaps documented with owners and deadlines
|
||||
- [ ] NFR assessment report generated and saved
|
||||
- [ ] Gate YAML snippet generated (if enabled)
|
||||
- [ ] Evidence checklist generated (if enabled)
|
||||
- [ ] Workflow completed successfully
|
||||
|
||||
---
|
||||
|
||||
## Sign-Off
|
||||
|
||||
**NFR Assessment Status:**
|
||||
|
||||
- [ ] ✅ PASS - All NFRs meet requirements, ready for release
|
||||
- [ ] ⚠️ CONCERNS - Some NFRs have concerns, address before next release
|
||||
- [ ] ❌ FAIL - Critical NFRs not met, BLOCKER for release
|
||||
|
||||
**Next Actions:**
|
||||
|
||||
- If PASS ✅: Proceed to `*gate` workflow or release
|
||||
- If CONCERNS ⚠️: Address HIGH/CRITICAL issues, re-run `*nfr-assess`
|
||||
- If FAIL ❌: Resolve FAIL status NFRs, re-run `*nfr-assess`
|
||||
|
||||
**Critical Issues:** {COUNT}
|
||||
**High Priority Issues:** {COUNT}
|
||||
**Concerns:** {COUNT}
|
||||
|
||||
---
|
||||
|
||||
<!-- Powered by BMAD-CORE™ -->
|
||||
43
_bmad/tea/workflows/testarch/nfr-assess/instructions.md
Normal file
43
_bmad/tea/workflows/testarch/nfr-assess/instructions.md
Normal file
@@ -0,0 +1,43 @@
|
||||
# Non-Functional Requirements Assessment
|
||||
|
||||
**Workflow:** `testarch-nfr`
|
||||
**Version:** 5.0 (Step-File Architecture)
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Assess non-functional requirements (performance, security, reliability, maintainability) with evidence-based validation and deterministic PASS/CONCERNS/FAIL outcomes.
|
||||
|
||||
---
|
||||
|
||||
## WORKFLOW ARCHITECTURE
|
||||
|
||||
This workflow uses **step-file architecture**:
|
||||
|
||||
- **Micro-file Design**: Each step is self-contained
|
||||
- **JIT Loading**: Only the current step file is in memory
|
||||
- **Sequential Enforcement**: Execute steps in order
|
||||
|
||||
---
|
||||
|
||||
## INITIALIZATION SEQUENCE
|
||||
|
||||
### 1. Configuration Loading
|
||||
|
||||
From `workflow.yaml`, resolve:
|
||||
|
||||
- `config_source`, `test_artifacts`, `user_name`, `communication_language`, `document_output_language`, `date`
|
||||
- `custom_nfr_categories`
|
||||
|
||||
### 2. First Step
|
||||
|
||||
Load, read completely, and execute:
|
||||
`{project-root}/_bmad/tea/workflows/testarch/nfr-assess/steps-c/step-01-load-context.md`
|
||||
|
||||
### 3. Resume Support
|
||||
|
||||
If the user selects **Resume** mode, load, read completely, and execute:
|
||||
`{project-root}/_bmad/tea/workflows/testarch/nfr-assess/steps-c/step-01b-resume.md`
|
||||
|
||||
This checks the output document for progress tracking frontmatter and routes to the next incomplete step.
|
||||
470
_bmad/tea/workflows/testarch/nfr-assess/nfr-report-template.md
Normal file
470
_bmad/tea/workflows/testarch/nfr-assess/nfr-report-template.md
Normal file
@@ -0,0 +1,470 @@
|
||||
---
|
||||
stepsCompleted: []
|
||||
lastStep: ''
|
||||
lastSaved: ''
|
||||
workflowType: 'testarch-nfr-assess'
|
||||
inputDocuments: []
|
||||
---
|
||||
|
||||
# NFR Assessment - {FEATURE_NAME}
|
||||
|
||||
**Date:** {DATE}
|
||||
**Story:** {STORY_ID} (if applicable)
|
||||
**Overall Status:** {OVERALL_STATUS} {STATUS_ICON}
|
||||
|
||||
---
|
||||
|
||||
Note: This assessment summarizes existing evidence; it does not run tests or CI workflows.
|
||||
|
||||
## Executive Summary
|
||||
|
||||
**Assessment:** {PASS_COUNT} PASS, {CONCERNS_COUNT} CONCERNS, {FAIL_COUNT} FAIL
|
||||
|
||||
**Blockers:** {BLOCKER_COUNT} {BLOCKER_DESCRIPTION}
|
||||
|
||||
**High Priority Issues:** {HIGH_PRIORITY_COUNT} {HIGH_PRIORITY_DESCRIPTION}
|
||||
|
||||
**Recommendation:** {OVERALL_RECOMMENDATION}
|
||||
|
||||
---
|
||||
|
||||
## Performance Assessment
|
||||
|
||||
### Response Time (p95)
|
||||
|
||||
- **Status:** {STATUS} {STATUS_ICON}
|
||||
- **Threshold:** {THRESHOLD_VALUE}
|
||||
- **Actual:** {ACTUAL_VALUE}
|
||||
- **Evidence:** {EVIDENCE_SOURCE}
|
||||
- **Findings:** {FINDINGS_DESCRIPTION}
|
||||
|
||||
### Throughput
|
||||
|
||||
- **Status:** {STATUS} {STATUS_ICON}
|
||||
- **Threshold:** {THRESHOLD_VALUE}
|
||||
- **Actual:** {ACTUAL_VALUE}
|
||||
- **Evidence:** {EVIDENCE_SOURCE}
|
||||
- **Findings:** {FINDINGS_DESCRIPTION}
|
||||
|
||||
### Resource Usage
|
||||
|
||||
- **CPU Usage**
|
||||
- **Status:** {STATUS} {STATUS_ICON}
|
||||
- **Threshold:** {THRESHOLD_VALUE}
|
||||
- **Actual:** {ACTUAL_VALUE}
|
||||
- **Evidence:** {EVIDENCE_SOURCE}
|
||||
|
||||
- **Memory Usage**
|
||||
- **Status:** {STATUS} {STATUS_ICON}
|
||||
- **Threshold:** {THRESHOLD_VALUE}
|
||||
- **Actual:** {ACTUAL_VALUE}
|
||||
- **Evidence:** {EVIDENCE_SOURCE}
|
||||
|
||||
### Scalability
|
||||
|
||||
- **Status:** {STATUS} {STATUS_ICON}
|
||||
- **Threshold:** {THRESHOLD_DESCRIPTION}
|
||||
- **Actual:** {ACTUAL_DESCRIPTION}
|
||||
- **Evidence:** {EVIDENCE_SOURCE}
|
||||
- **Findings:** {FINDINGS_DESCRIPTION}
|
||||
|
||||
---
|
||||
|
||||
## Security Assessment
|
||||
|
||||
### Authentication Strength
|
||||
|
||||
- **Status:** {STATUS} {STATUS_ICON}
|
||||
- **Threshold:** {THRESHOLD_DESCRIPTION}
|
||||
- **Actual:** {ACTUAL_DESCRIPTION}
|
||||
- **Evidence:** {EVIDENCE_SOURCE}
|
||||
- **Findings:** {FINDINGS_DESCRIPTION}
|
||||
- **Recommendation:** {RECOMMENDATION} (if CONCERNS or FAIL)
|
||||
|
||||
### Authorization Controls
|
||||
|
||||
- **Status:** {STATUS} {STATUS_ICON}
|
||||
- **Threshold:** {THRESHOLD_DESCRIPTION}
|
||||
- **Actual:** {ACTUAL_DESCRIPTION}
|
||||
- **Evidence:** {EVIDENCE_SOURCE}
|
||||
- **Findings:** {FINDINGS_DESCRIPTION}
|
||||
|
||||
### Data Protection
|
||||
|
||||
- **Status:** {STATUS} {STATUS_ICON}
|
||||
- **Threshold:** {THRESHOLD_DESCRIPTION}
|
||||
- **Actual:** {ACTUAL_DESCRIPTION}
|
||||
- **Evidence:** {EVIDENCE_SOURCE}
|
||||
- **Findings:** {FINDINGS_DESCRIPTION}
|
||||
|
||||
### Vulnerability Management
|
||||
|
||||
- **Status:** {STATUS} {STATUS_ICON}
|
||||
- **Threshold:** {THRESHOLD_DESCRIPTION} (e.g., "0 critical, <3 high vulnerabilities")
|
||||
- **Actual:** {ACTUAL_DESCRIPTION} (e.g., "0 critical, 1 high, 5 medium vulnerabilities")
|
||||
- **Evidence:** {EVIDENCE_SOURCE} (e.g., "Snyk scan results - scan-2025-10-14.json")
|
||||
- **Findings:** {FINDINGS_DESCRIPTION}
|
||||
|
||||
### Compliance (if applicable)
|
||||
|
||||
- **Status:** {STATUS} {STATUS_ICON}
|
||||
- **Standards:** {COMPLIANCE_STANDARDS} (e.g., "GDPR, HIPAA, PCI-DSS")
|
||||
- **Actual:** {ACTUAL_COMPLIANCE_STATUS}
|
||||
- **Evidence:** {EVIDENCE_SOURCE}
|
||||
- **Findings:** {FINDINGS_DESCRIPTION}
|
||||
|
||||
---
|
||||
|
||||
## Reliability Assessment
|
||||
|
||||
### Availability (Uptime)
|
||||
|
||||
- **Status:** {STATUS} {STATUS_ICON}
|
||||
- **Threshold:** {THRESHOLD_VALUE} (e.g., "99.9%")
|
||||
- **Actual:** {ACTUAL_VALUE} (e.g., "99.95%")
|
||||
- **Evidence:** {EVIDENCE_SOURCE} (e.g., "Uptime monitoring - uptime-report-2025-10-14.csv")
|
||||
- **Findings:** {FINDINGS_DESCRIPTION}
|
||||
|
||||
### Error Rate
|
||||
|
||||
- **Status:** {STATUS} {STATUS_ICON}
|
||||
- **Threshold:** {THRESHOLD_VALUE} (e.g., "<0.1%")
|
||||
- **Actual:** {ACTUAL_VALUE} (e.g., "0.05%")
|
||||
- **Evidence:** {EVIDENCE_SOURCE} (e.g., "Error logs - logs/errors-2025-10.log")
|
||||
- **Findings:** {FINDINGS_DESCRIPTION}
|
||||
|
||||
### MTTR (Mean Time To Recovery)
|
||||
|
||||
- **Status:** {STATUS} {STATUS_ICON}
|
||||
- **Threshold:** {THRESHOLD_VALUE} (e.g., "<15 minutes")
|
||||
- **Actual:** {ACTUAL_VALUE} (e.g., "12 minutes")
|
||||
- **Evidence:** {EVIDENCE_SOURCE} (e.g., "Incident reports - incidents/")
|
||||
- **Findings:** {FINDINGS_DESCRIPTION}
|
||||
|
||||
### Fault Tolerance
|
||||
|
||||
- **Status:** {STATUS} {STATUS_ICON}
|
||||
- **Threshold:** {THRESHOLD_DESCRIPTION}
|
||||
- **Actual:** {ACTUAL_DESCRIPTION}
|
||||
- **Evidence:** {EVIDENCE_SOURCE}
|
||||
- **Findings:** {FINDINGS_DESCRIPTION}
|
||||
|
||||
### CI Burn-In (Stability)
|
||||
|
||||
- **Status:** {STATUS} {STATUS_ICON}
|
||||
- **Threshold:** {THRESHOLD_VALUE} (e.g., "100 consecutive successful runs")
|
||||
- **Actual:** {ACTUAL_VALUE} (e.g., "150 consecutive successful runs")
|
||||
- **Evidence:** {EVIDENCE_SOURCE} (e.g., "CI burn-in results - ci-burn-in-2025-10-14.log")
|
||||
- **Findings:** {FINDINGS_DESCRIPTION}
|
||||
|
||||
### Disaster Recovery (if applicable)
|
||||
|
||||
- **RTO (Recovery Time Objective)**
|
||||
- **Status:** {STATUS} {STATUS_ICON}
|
||||
- **Threshold:** {THRESHOLD_VALUE}
|
||||
- **Actual:** {ACTUAL_VALUE}
|
||||
- **Evidence:** {EVIDENCE_SOURCE}
|
||||
|
||||
- **RPO (Recovery Point Objective)**
|
||||
- **Status:** {STATUS} {STATUS_ICON}
|
||||
- **Threshold:** {THRESHOLD_VALUE}
|
||||
- **Actual:** {ACTUAL_VALUE}
|
||||
- **Evidence:** {EVIDENCE_SOURCE}
|
||||
|
||||
---
|
||||
|
||||
## Maintainability Assessment
|
||||
|
||||
### Test Coverage
|
||||
|
||||
- **Status:** {STATUS} {STATUS_ICON}
|
||||
- **Threshold:** {THRESHOLD_VALUE} (e.g., ">=80%")
|
||||
- **Actual:** {ACTUAL_VALUE} (e.g., "87%")
|
||||
- **Evidence:** {EVIDENCE_SOURCE} (e.g., "Coverage report - coverage/lcov-report/index.html")
|
||||
- **Findings:** {FINDINGS_DESCRIPTION}
|
||||
|
||||
### Code Quality
|
||||
|
||||
- **Status:** {STATUS} {STATUS_ICON}
|
||||
- **Threshold:** {THRESHOLD_VALUE} (e.g., ">=85/100")
|
||||
- **Actual:** {ACTUAL_VALUE} (e.g., "92/100")
|
||||
- **Evidence:** {EVIDENCE_SOURCE} (e.g., "SonarQube analysis - sonarqube-report-2025-10-14.pdf")
|
||||
- **Findings:** {FINDINGS_DESCRIPTION}
|
||||
|
||||
### Technical Debt
|
||||
|
||||
- **Status:** {STATUS} {STATUS_ICON}
|
||||
- **Threshold:** {THRESHOLD_VALUE} (e.g., "<5% debt ratio")
|
||||
- **Actual:** {ACTUAL_VALUE} (e.g., "3.2% debt ratio")
|
||||
- **Evidence:** {EVIDENCE_SOURCE} (e.g., "CodeClimate analysis - codeclimate-2025-10-14.json")
|
||||
- **Findings:** {FINDINGS_DESCRIPTION}
|
||||
|
||||
### Documentation Completeness
|
||||
|
||||
- **Status:** {STATUS} {STATUS_ICON}
|
||||
- **Threshold:** {THRESHOLD_VALUE} (e.g., ">=90%")
|
||||
- **Actual:** {ACTUAL_VALUE} (e.g., "95%")
|
||||
- **Evidence:** {EVIDENCE_SOURCE} (e.g., "Documentation audit - docs-audit-2025-10-14.md")
|
||||
- **Findings:** {FINDINGS_DESCRIPTION}
|
||||
|
||||
### Test Quality (from test-review, if available)
|
||||
|
||||
- **Status:** {STATUS} {STATUS_ICON}
|
||||
- **Threshold:** {THRESHOLD_DESCRIPTION}
|
||||
- **Actual:** {ACTUAL_DESCRIPTION}
|
||||
- **Evidence:** {EVIDENCE_SOURCE} (e.g., "Test review report - test-review-2025-10-14.md")
|
||||
- **Findings:** {FINDINGS_DESCRIPTION}
|
||||
|
||||
---
|
||||
|
||||
## Custom NFR Assessments (if applicable)
|
||||
|
||||
### {CUSTOM_NFR_NAME_1}
|
||||
|
||||
- **Status:** {STATUS} {STATUS_ICON}
|
||||
- **Threshold:** {THRESHOLD_DESCRIPTION}
|
||||
- **Actual:** {ACTUAL_DESCRIPTION}
|
||||
- **Evidence:** {EVIDENCE_SOURCE}
|
||||
- **Findings:** {FINDINGS_DESCRIPTION}
|
||||
|
||||
### {CUSTOM_NFR_NAME_2}
|
||||
|
||||
- **Status:** {STATUS} {STATUS_ICON}
|
||||
- **Threshold:** {THRESHOLD_DESCRIPTION}
|
||||
- **Actual:** {ACTUAL_DESCRIPTION}
|
||||
- **Evidence:** {EVIDENCE_SOURCE}
|
||||
- **Findings:** {FINDINGS_DESCRIPTION}
|
||||
|
||||
---
|
||||
|
||||
## Quick Wins
|
||||
|
||||
{QUICK_WIN_COUNT} quick wins identified for immediate implementation:
|
||||
|
||||
1. **{QUICK_WIN_TITLE_1}** ({NFR_CATEGORY}) - {PRIORITY} - {ESTIMATED_EFFORT}
|
||||
- {QUICK_WIN_DESCRIPTION}
|
||||
- No code changes needed / Minimal code changes
|
||||
|
||||
2. **{QUICK_WIN_TITLE_2}** ({NFR_CATEGORY}) - {PRIORITY} - {ESTIMATED_EFFORT}
|
||||
- {QUICK_WIN_DESCRIPTION}
|
||||
|
||||
---
|
||||
|
||||
## Recommended Actions
|
||||
|
||||
### Immediate (Before Release) - CRITICAL/HIGH Priority
|
||||
|
||||
1. **{ACTION_TITLE_1}** - {PRIORITY} - {ESTIMATED_EFFORT} - {OWNER}
|
||||
- {ACTION_DESCRIPTION}
|
||||
- {SPECIFIC_STEPS}
|
||||
- {VALIDATION_CRITERIA}
|
||||
|
||||
2. **{ACTION_TITLE_2}** - {PRIORITY} - {ESTIMATED_EFFORT} - {OWNER}
|
||||
- {ACTION_DESCRIPTION}
|
||||
- {SPECIFIC_STEPS}
|
||||
- {VALIDATION_CRITERIA}
|
||||
|
||||
### Short-term (Next Milestone) - MEDIUM Priority
|
||||
|
||||
1. **{ACTION_TITLE_3}** - {PRIORITY} - {ESTIMATED_EFFORT} - {OWNER}
|
||||
- {ACTION_DESCRIPTION}
|
||||
|
||||
2. **{ACTION_TITLE_4}** - {PRIORITY} - {ESTIMATED_EFFORT} - {OWNER}
|
||||
- {ACTION_DESCRIPTION}
|
||||
|
||||
### Long-term (Backlog) - LOW Priority
|
||||
|
||||
1. **{ACTION_TITLE_5}** - {PRIORITY} - {ESTIMATED_EFFORT} - {OWNER}
|
||||
- {ACTION_DESCRIPTION}
|
||||
|
||||
---
|
||||
|
||||
## Monitoring Hooks
|
||||
|
||||
{MONITORING_HOOK_COUNT} monitoring hooks recommended to detect issues before failures:
|
||||
|
||||
### Performance Monitoring
|
||||
|
||||
- [ ] {MONITORING_TOOL_1} - {MONITORING_DESCRIPTION}
|
||||
- **Owner:** {OWNER}
|
||||
- **Deadline:** {DEADLINE}
|
||||
|
||||
- [ ] {MONITORING_TOOL_2} - {MONITORING_DESCRIPTION}
|
||||
- **Owner:** {OWNER}
|
||||
- **Deadline:** {DEADLINE}
|
||||
|
||||
### Security Monitoring
|
||||
|
||||
- [ ] {MONITORING_TOOL_3} - {MONITORING_DESCRIPTION}
|
||||
- **Owner:** {OWNER}
|
||||
- **Deadline:** {DEADLINE}
|
||||
|
||||
### Reliability Monitoring
|
||||
|
||||
- [ ] {MONITORING_TOOL_4} - {MONITORING_DESCRIPTION}
|
||||
- **Owner:** {OWNER}
|
||||
- **Deadline:** {DEADLINE}
|
||||
|
||||
### Alerting Thresholds
|
||||
|
||||
- [ ] {ALERT_DESCRIPTION} - Notify when {THRESHOLD_CONDITION}
|
||||
- **Owner:** {OWNER}
|
||||
- **Deadline:** {DEADLINE}
|
||||
|
||||
---
|
||||
|
||||
## Fail-Fast Mechanisms
|
||||
|
||||
{FAIL_FAST_COUNT} fail-fast mechanisms recommended to prevent failures:
|
||||
|
||||
### Circuit Breakers (Reliability)
|
||||
|
||||
- [ ] {CIRCUIT_BREAKER_DESCRIPTION}
|
||||
- **Owner:** {OWNER}
|
||||
- **Estimated Effort:** {EFFORT}
|
||||
|
||||
### Rate Limiting (Performance)
|
||||
|
||||
- [ ] {RATE_LIMITING_DESCRIPTION}
|
||||
- **Owner:** {OWNER}
|
||||
- **Estimated Effort:** {EFFORT}
|
||||
|
||||
### Validation Gates (Security)
|
||||
|
||||
- [ ] {VALIDATION_GATE_DESCRIPTION}
|
||||
- **Owner:** {OWNER}
|
||||
- **Estimated Effort:** {EFFORT}
|
||||
|
||||
### Smoke Tests (Maintainability)
|
||||
|
||||
- [ ] {SMOKE_TEST_DESCRIPTION}
|
||||
- **Owner:** {OWNER}
|
||||
- **Estimated Effort:** {EFFORT}
|
||||
|
||||
---
|
||||
|
||||
## Evidence Gaps
|
||||
|
||||
{EVIDENCE_GAP_COUNT} evidence gaps identified - action required:
|
||||
|
||||
- [ ] **{NFR_NAME_1}** ({NFR_CATEGORY})
|
||||
- **Owner:** {OWNER}
|
||||
- **Deadline:** {DEADLINE}
|
||||
- **Suggested Evidence:** {SUGGESTED_EVIDENCE_SOURCE}
|
||||
- **Impact:** {IMPACT_DESCRIPTION}
|
||||
|
||||
- [ ] **{NFR_NAME_2}** ({NFR_CATEGORY})
|
||||
- **Owner:** {OWNER}
|
||||
- **Deadline:** {DEADLINE}
|
||||
- **Suggested Evidence:** {SUGGESTED_EVIDENCE_SOURCE}
|
||||
- **Impact:** {IMPACT_DESCRIPTION}
|
||||
|
||||
---
|
||||
|
||||
## Findings Summary
|
||||
|
||||
**Based on ADR Quality Readiness Checklist (8 categories, 29 criteria)**
|
||||
|
||||
| Category | Criteria Met | PASS | CONCERNS | FAIL | Overall Status |
|
||||
| ------------------------------------------------ | ------------------ | ---------------- | -------------------- | ---------------- | ----------------------------------- |
|
||||
| 1. Testability & Automation | {T_MET}/4 | {T_PASS} | {T_CONCERNS} | {T_FAIL} | {T_STATUS} {T_ICON} |
|
||||
| 2. Test Data Strategy | {TD_MET}/3 | {TD_PASS} | {TD_CONCERNS} | {TD_FAIL} | {TD_STATUS} {TD_ICON} |
|
||||
| 3. Scalability & Availability | {SA_MET}/4 | {SA_PASS} | {SA_CONCERNS} | {SA_FAIL} | {SA_STATUS} {SA_ICON} |
|
||||
| 4. Disaster Recovery | {DR_MET}/3 | {DR_PASS} | {DR_CONCERNS} | {DR_FAIL} | {DR_STATUS} {DR_ICON} |
|
||||
| 5. Security | {SEC_MET}/4 | {SEC_PASS} | {SEC_CONCERNS} | {SEC_FAIL} | {SEC_STATUS} {SEC_ICON} |
|
||||
| 6. Monitorability, Debuggability & Manageability | {MON_MET}/4 | {MON_PASS} | {MON_CONCERNS} | {MON_FAIL} | {MON_STATUS} {MON_ICON} |
|
||||
| 7. QoS & QoE | {QOS_MET}/4 | {QOS_PASS} | {QOS_CONCERNS} | {QOS_FAIL} | {QOS_STATUS} {QOS_ICON} |
|
||||
| 8. Deployability | {DEP_MET}/3 | {DEP_PASS} | {DEP_CONCERNS} | {DEP_FAIL} | {DEP_STATUS} {DEP_ICON} |
|
||||
| **Total** | **{TOTAL_MET}/29** | **{TOTAL_PASS}** | **{TOTAL_CONCERNS}** | **{TOTAL_FAIL}** | **{OVERALL_STATUS} {OVERALL_ICON}** |
|
||||
|
||||
**Criteria Met Scoring:**
|
||||
|
||||
- ≥26/29 (90%+) = Strong foundation
|
||||
- 20-25/29 (69-86%) = Room for improvement
|
||||
- <20/29 (<69%) = Significant gaps
|
||||
|
||||
---
|
||||
|
||||
## Gate YAML Snippet
|
||||
|
||||
```yaml
|
||||
nfr_assessment:
|
||||
date: '{DATE}'
|
||||
story_id: '{STORY_ID}'
|
||||
feature_name: '{FEATURE_NAME}'
|
||||
adr_checklist_score: '{TOTAL_MET}/29' # ADR Quality Readiness Checklist
|
||||
categories:
|
||||
testability_automation: '{T_STATUS}'
|
||||
test_data_strategy: '{TD_STATUS}'
|
||||
scalability_availability: '{SA_STATUS}'
|
||||
disaster_recovery: '{DR_STATUS}'
|
||||
security: '{SEC_STATUS}'
|
||||
monitorability: '{MON_STATUS}'
|
||||
qos_qoe: '{QOS_STATUS}'
|
||||
deployability: '{DEP_STATUS}'
|
||||
overall_status: '{OVERALL_STATUS}'
|
||||
critical_issues: { CRITICAL_COUNT }
|
||||
high_priority_issues: { HIGH_COUNT }
|
||||
medium_priority_issues: { MEDIUM_COUNT }
|
||||
concerns: { CONCERNS_COUNT }
|
||||
blockers: { BLOCKER_BOOLEAN } # true/false
|
||||
quick_wins: { QUICK_WIN_COUNT }
|
||||
evidence_gaps: { EVIDENCE_GAP_COUNT }
|
||||
recommendations:
|
||||
- '{RECOMMENDATION_1}'
|
||||
- '{RECOMMENDATION_2}'
|
||||
- '{RECOMMENDATION_3}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Related Artifacts
|
||||
|
||||
- **Story File:** {STORY_FILE_PATH} (if applicable)
|
||||
- **Tech Spec:** {TECH_SPEC_PATH} (if available)
|
||||
- **PRD:** {PRD_PATH} (if available)
|
||||
- **Test Design:** {TEST_DESIGN_PATH} (if available)
|
||||
- **Evidence Sources:**
|
||||
- Test Results: {TEST_RESULTS_DIR}
|
||||
- Metrics: {METRICS_DIR}
|
||||
- Logs: {LOGS_DIR}
|
||||
- CI Results: {CI_RESULTS_PATH}
|
||||
|
||||
---
|
||||
|
||||
## Recommendations Summary
|
||||
|
||||
**Release Blocker:** {RELEASE_BLOCKER_SUMMARY}
|
||||
|
||||
**High Priority:** {HIGH_PRIORITY_SUMMARY}
|
||||
|
||||
**Medium Priority:** {MEDIUM_PRIORITY_SUMMARY}
|
||||
|
||||
**Next Steps:** {NEXT_STEPS_DESCRIPTION}
|
||||
|
||||
---
|
||||
|
||||
## Sign-Off
|
||||
|
||||
**NFR Assessment:**
|
||||
|
||||
- Overall Status: {OVERALL_STATUS} {OVERALL_ICON}
|
||||
- Critical Issues: {CRITICAL_COUNT}
|
||||
- High Priority Issues: {HIGH_COUNT}
|
||||
- Concerns: {CONCERNS_COUNT}
|
||||
- Evidence Gaps: {EVIDENCE_GAP_COUNT}
|
||||
|
||||
**Gate Status:** {GATE_STATUS} {GATE_ICON}
|
||||
|
||||
**Next Actions:**
|
||||
|
||||
- If PASS ✅: Proceed to `*gate` workflow or release
|
||||
- If CONCERNS ⚠️: Address HIGH/CRITICAL issues, re-run `*nfr-assess`
|
||||
- If FAIL ❌: Resolve FAIL status NFRs, re-run `*nfr-assess`
|
||||
|
||||
**Generated:** {DATE}
|
||||
**Workflow:** testarch-nfr v4.0
|
||||
|
||||
---
|
||||
|
||||
<!-- Powered by BMAD-CORE™ -->
|
||||
@@ -0,0 +1,138 @@
|
||||
---
|
||||
name: 'step-01-load-context'
|
||||
description: 'Load NFR requirements, evidence sources, and knowledge base'
|
||||
nextStepFile: './step-02-define-thresholds.md'
|
||||
knowledgeIndex: '{project-root}/_bmad/tea/testarch/tea-index.csv'
|
||||
outputFile: '{test_artifacts}/nfr-assessment.md'
|
||||
---
|
||||
|
||||
# Step 1: Load Context & Knowledge Base
|
||||
|
||||
## STEP GOAL
|
||||
|
||||
Gather NFR requirements, evidence sources, and knowledge fragments needed for assessment.
|
||||
|
||||
## MANDATORY EXECUTION RULES
|
||||
|
||||
- 📖 Read the entire step file before acting
|
||||
- ✅ Speak in `{communication_language}`
|
||||
- 🚫 Halt if implementation or evidence is unavailable
|
||||
|
||||
---
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
- 💾 Record outputs before proceeding
|
||||
- 📖 Load the next step only when instructed
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: config, loaded artifacts, and knowledge fragments
|
||||
- Focus: this step's goal only
|
||||
- Limits: do not execute future steps
|
||||
- Dependencies: prior steps' outputs (if any)
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly. Do not skip, reorder, or improvise.
|
||||
|
||||
## 1. Prerequisites
|
||||
|
||||
- Implementation accessible for evaluation
|
||||
- Evidence sources available (test results, metrics, logs)
|
||||
|
||||
If missing: **HALT** and request the missing inputs.
|
||||
|
||||
---
|
||||
|
||||
## 2. Load Configuration
|
||||
|
||||
From `{config_source}`:
|
||||
|
||||
- Read `tea_browser_automation`
|
||||
|
||||
---
|
||||
|
||||
### Tiered Knowledge Loading
|
||||
|
||||
Load fragments based on their `tier` classification in `tea-index.csv`:
|
||||
|
||||
1. **Core tier** (always load): Foundational fragments required for this workflow
|
||||
2. **Extended tier** (load on-demand): Load when deeper analysis is needed or when the user's context requires it
|
||||
3. **Specialized tier** (load only when relevant): Load only when the specific use case matches (e.g., contract-testing only for microservices, email-auth only for email flows)
|
||||
|
||||
> **Context Efficiency**: Loading only core fragments reduces context usage by 40-50% compared to loading all fragments.
|
||||
|
||||
## 3. Load Knowledge Base Fragments
|
||||
|
||||
From `{knowledgeIndex}` load:
|
||||
|
||||
- `adr-quality-readiness-checklist.md`
|
||||
- `ci-burn-in.md`
|
||||
- `test-quality.md`
|
||||
- `playwright-config.md`
|
||||
- `error-handling.md`
|
||||
|
||||
**Playwright CLI (if `tea_browser_automation` is "cli" or "auto"):**
|
||||
|
||||
- `playwright-cli.md`
|
||||
|
||||
**MCP Patterns (if `tea_browser_automation` is "mcp" or "auto"):**
|
||||
|
||||
- (existing MCP-related fragments, if any are added in future)
|
||||
|
||||
---
|
||||
|
||||
## 4. Load Artifacts
|
||||
|
||||
If available, read:
|
||||
|
||||
- `tech-spec.md` (primary NFRs)
|
||||
- `PRD.md` (product-level NFRs)
|
||||
- `story` or `test-design` docs (feature-level NFRs)
|
||||
|
||||
---
|
||||
|
||||
## 5. Confirm Inputs
|
||||
|
||||
Summarize loaded NFR sources and evidence availability.
|
||||
|
||||
---
|
||||
|
||||
## 6. Save Progress
|
||||
|
||||
**Save this step's accumulated work to `{outputFile}`.**
|
||||
|
||||
- **If `{outputFile}` does not exist** (first save), create it using the workflow template (if available) with YAML frontmatter:
|
||||
|
||||
```yaml
|
||||
---
|
||||
stepsCompleted: ['step-01-load-context']
|
||||
lastStep: 'step-01-load-context'
|
||||
lastSaved: '{date}'
|
||||
---
|
||||
```
|
||||
|
||||
Then write this step's output below the frontmatter.
|
||||
|
||||
- **If `{outputFile}` already exists**, update:
|
||||
- Add `'step-01-load-context'` to `stepsCompleted` array (only if not already present)
|
||||
- Set `lastStep: 'step-01-load-context'`
|
||||
- Set `lastSaved: '{date}'`
|
||||
- Append this step's output to the appropriate section of the document.
|
||||
|
||||
**Update `inputDocuments`**: Set `inputDocuments` in the output template frontmatter to the list of artifact paths loaded in this step (e.g., knowledge fragments, test design documents, configuration files).
|
||||
|
||||
Load next step: `{nextStepFile}`
|
||||
|
||||
## 🚨 SYSTEM SUCCESS/FAILURE METRICS:
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- Step completed in full with required outputs
|
||||
|
||||
### ❌ SYSTEM FAILURE:
|
||||
|
||||
- Skipped sequence steps or missing outputs
|
||||
**Master Rule:** Skipping steps is FORBIDDEN.
|
||||
@@ -0,0 +1,106 @@
|
||||
---
|
||||
name: 'step-01b-resume'
|
||||
description: 'Resume interrupted workflow from last completed step'
|
||||
outputFile: '{test_artifacts}/nfr-assessment.md'
|
||||
---
|
||||
|
||||
# Step 1b: Resume Workflow
|
||||
|
||||
## STEP GOAL
|
||||
|
||||
Resume an interrupted workflow by loading the existing output document, displaying progress, and routing to the next incomplete step.
|
||||
|
||||
## MANDATORY EXECUTION RULES
|
||||
|
||||
- Read the entire step file before acting
|
||||
- Speak in `{communication_language}`
|
||||
|
||||
---
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- Follow the MANDATORY SEQUENCE exactly
|
||||
- Load the next step only when instructed
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: Output document with progress frontmatter
|
||||
- Focus: Load progress and route to next step
|
||||
- Limits: Do not re-execute completed steps
|
||||
- Dependencies: Output document must exist from a previous run
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly.
|
||||
|
||||
### 1. Load Output Document
|
||||
|
||||
Read `{outputFile}` and parse YAML frontmatter for:
|
||||
|
||||
- `stepsCompleted` -- array of completed step names
|
||||
- `lastStep` -- last completed step name
|
||||
- `lastSaved` -- timestamp of last save
|
||||
|
||||
**If `{outputFile}` does not exist**, display:
|
||||
|
||||
"No previous progress found. There is no output document to resume from. Please use **[C] Create** to start a fresh workflow run."
|
||||
|
||||
**THEN:** Halt. Do not proceed.
|
||||
|
||||
---
|
||||
|
||||
### 2. Display Progress Dashboard
|
||||
|
||||
Display progress with checkmark/empty indicators:
|
||||
|
||||
```
|
||||
NFR Assessment - Resume Progress:
|
||||
|
||||
1. Load Context (step-01-load-context) [completed/pending]
|
||||
2. Define Thresholds (step-02-define-thresholds) [completed/pending]
|
||||
3. Gather Evidence (step-03-gather-evidence) [completed/pending]
|
||||
4. Evaluate & Aggregate (step-04e-aggregate-nfr) [completed/pending]
|
||||
5. Generate Report (step-05-generate-report) [completed/pending]
|
||||
|
||||
Last saved: {lastSaved}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 3. Route to Next Step
|
||||
|
||||
Based on `lastStep`, load the next incomplete step:
|
||||
|
||||
| lastStep | Next Step File |
|
||||
| --------------------------- | --------------------------------- |
|
||||
| `step-01-load-context` | `./step-02-define-thresholds.md` |
|
||||
| `step-02-define-thresholds` | `./step-03-gather-evidence.md` |
|
||||
| `step-03-gather-evidence` | `./step-04-evaluate-and-score.md` |
|
||||
| `step-04e-aggregate-nfr` | `./step-05-generate-report.md` |
|
||||
| `step-05-generate-report` | **Workflow already complete.** |
|
||||
|
||||
**If `lastStep` is the final step** (`step-05-generate-report`), display: "All steps completed. Use **[C] Create** to start fresh, **[V] Validate** to review outputs, or **[E] Edit** to make revisions." Then halt.
|
||||
|
||||
**If `lastStep` does not match any value above**, display: "Unknown progress state (`lastStep`: {lastStep}). Please use **[C] Create** to start fresh." Then halt.
|
||||
|
||||
**Otherwise**, load the identified step file, read completely, and execute.
|
||||
|
||||
The existing content in `{outputFile}` provides context from previously completed steps.
|
||||
|
||||
---
|
||||
|
||||
## SYSTEM SUCCESS/FAILURE METRICS
|
||||
|
||||
### SUCCESS:
|
||||
|
||||
- Output document loaded and parsed correctly
|
||||
- Progress dashboard displayed accurately
|
||||
- Routed to correct next step
|
||||
|
||||
### FAILURE:
|
||||
|
||||
- Not loading output document
|
||||
- Incorrect progress display
|
||||
- Routing to wrong step
|
||||
|
||||
**Master Rule:** Resume MUST route to the exact next incomplete step. Never re-execute completed steps.
|
||||
@@ -0,0 +1,107 @@
|
||||
---
|
||||
name: 'step-02-define-thresholds'
|
||||
description: 'Identify NFR categories and thresholds'
|
||||
nextStepFile: './step-03-gather-evidence.md'
|
||||
outputFile: '{test_artifacts}/nfr-assessment.md'
|
||||
---
|
||||
|
||||
# Step 2: Define NFR Categories & Thresholds
|
||||
|
||||
## STEP GOAL
|
||||
|
||||
Establish the NFR categories to assess and the thresholds used for validation.
|
||||
|
||||
## MANDATORY EXECUTION RULES
|
||||
|
||||
- 📖 Read the entire step file before acting
|
||||
- ✅ Speak in `{communication_language}`
|
||||
- 🚫 Never guess thresholds
|
||||
|
||||
---
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
- 💾 Record outputs before proceeding
|
||||
- 📖 Load the next step only when instructed
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: config, loaded artifacts, and knowledge fragments
|
||||
- Focus: this step's goal only
|
||||
- Limits: do not execute future steps
|
||||
- Dependencies: prior steps' outputs (if any)
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly. Do not skip, reorder, or improvise.
|
||||
|
||||
## 1. Select Categories
|
||||
|
||||
Use the ADR Quality Readiness Checklist (8 categories):
|
||||
|
||||
1. Testability & Automation
|
||||
2. Test Data Strategy
|
||||
3. Scalability & Availability
|
||||
4. Disaster Recovery
|
||||
5. Security
|
||||
6. Monitorability/Debuggability/Manageability
|
||||
7. QoS/QoE
|
||||
8. Deployability
|
||||
|
||||
Add any `custom_nfr_categories` if provided.
|
||||
|
||||
---
|
||||
|
||||
## 2. Define Thresholds
|
||||
|
||||
For each category, extract thresholds from:
|
||||
|
||||
- tech-spec (primary)
|
||||
- PRD (secondary)
|
||||
- story or test-design (feature-specific)
|
||||
|
||||
If a threshold is unknown, mark it **UNKNOWN** and plan to report **CONCERNS**.
|
||||
|
||||
---
|
||||
|
||||
## 3. Confirm NFR Matrix
|
||||
|
||||
List each NFR category with its threshold or UNKNOWN status.
|
||||
|
||||
---
|
||||
|
||||
## 4. Save Progress
|
||||
|
||||
**Save this step's accumulated work to `{outputFile}`.**
|
||||
|
||||
- **If `{outputFile}` does not exist** (first save), create it using the workflow template (if available) with YAML frontmatter:
|
||||
|
||||
```yaml
|
||||
---
|
||||
stepsCompleted: ['step-02-define-thresholds']
|
||||
lastStep: 'step-02-define-thresholds'
|
||||
lastSaved: '{date}'
|
||||
---
|
||||
```
|
||||
|
||||
Then write this step's output below the frontmatter.
|
||||
|
||||
- **If `{outputFile}` already exists**, update:
|
||||
- Add `'step-02-define-thresholds'` to `stepsCompleted` array (only if not already present)
|
||||
- Set `lastStep: 'step-02-define-thresholds'`
|
||||
- Set `lastSaved: '{date}'`
|
||||
- Append this step's output to the appropriate section of the document.
|
||||
|
||||
Load next step: `{nextStepFile}`
|
||||
|
||||
## 🚨 SYSTEM SUCCESS/FAILURE METRICS:
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- Step completed in full with required outputs
|
||||
|
||||
### ❌ SYSTEM FAILURE:
|
||||
|
||||
- Skipped sequence steps or missing outputs
|
||||
**Master Rule:** Skipping steps is FORBIDDEN.
|
||||
@@ -0,0 +1,108 @@
|
||||
---
|
||||
name: 'step-03-gather-evidence'
|
||||
description: 'Collect evidence for each NFR category'
|
||||
nextStepFile: './step-04-evaluate-and-score.md'
|
||||
outputFile: '{test_artifacts}/nfr-assessment.md'
|
||||
---
|
||||
|
||||
# Step 3: Gather Evidence
|
||||
|
||||
## STEP GOAL
|
||||
|
||||
Collect measurable evidence to evaluate each NFR category.
|
||||
|
||||
## MANDATORY EXECUTION RULES
|
||||
|
||||
- 📖 Read the entire step file before acting
|
||||
- ✅ Speak in `{communication_language}`
|
||||
|
||||
---
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
- 💾 Record outputs before proceeding
|
||||
- 📖 Load the next step only when instructed
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: config, loaded artifacts, and knowledge fragments
|
||||
- Focus: this step's goal only
|
||||
- Limits: do not execute future steps
|
||||
- Dependencies: prior steps' outputs (if any)
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly. Do not skip, reorder, or improvise.
|
||||
|
||||
## 1. Evidence Sources
|
||||
|
||||
Collect evidence for:
|
||||
|
||||
- **Performance**: load tests, metrics, response time data
|
||||
- **Security**: scans, auth tests, vuln reports
|
||||
- **Reliability**: error rates, burn-in runs, failover tests
|
||||
- **Maintainability**: test quality, code health signals
|
||||
- **Other categories**: logs, monitoring, DR drills, deployability checks
|
||||
|
||||
---
|
||||
|
||||
## 2. Browser-Based Evidence Collection (if `tea_browser_automation` is `cli` or `auto`)
|
||||
|
||||
> **Fallback:** If CLI is not installed, fall back to MCP (if available) or skip browser-based evidence collection.
|
||||
|
||||
For performance and security categories, CLI can gather live evidence:
|
||||
|
||||
**Performance evidence (page load, response times):**
|
||||
|
||||
1. `playwright-cli -s=tea-nfr open <target_url>`
|
||||
2. `playwright-cli -s=tea-nfr network` → capture response times and payload sizes
|
||||
3. `playwright-cli -s=tea-nfr screenshot --filename={test_artifacts}/nfr/perf-<page>.png`
|
||||
4. `playwright-cli -s=tea-nfr close`
|
||||
|
||||
> **Session Hygiene:** Always close sessions using `playwright-cli -s=tea-nfr close`. Do NOT use `close-all` — it kills every session on the machine and breaks parallel execution.
|
||||
|
||||
Store artifacts under `{test_artifacts}/nfr/`
|
||||
|
||||
---
|
||||
|
||||
## 3. Evidence Gaps
|
||||
|
||||
If evidence is missing for a category, mark that category as **CONCERNS**.
|
||||
|
||||
---
|
||||
|
||||
## 4. Save Progress
|
||||
|
||||
**Save this step's accumulated work to `{outputFile}`.**
|
||||
|
||||
- **If `{outputFile}` does not exist** (first save), create it using the workflow template (if available) with YAML frontmatter:
|
||||
|
||||
```yaml
|
||||
---
|
||||
stepsCompleted: ['step-03-gather-evidence']
|
||||
lastStep: 'step-03-gather-evidence'
|
||||
lastSaved: '{date}'
|
||||
---
|
||||
```
|
||||
|
||||
Then write this step's output below the frontmatter.
|
||||
|
||||
- **If `{outputFile}` already exists**, update:
|
||||
- Add `'step-03-gather-evidence'` to `stepsCompleted` array (only if not already present)
|
||||
- Set `lastStep: 'step-03-gather-evidence'`
|
||||
- Set `lastSaved: '{date}'`
|
||||
- Append this step's output to the appropriate section of the document.
|
||||
|
||||
Load next step: `{nextStepFile}`
|
||||
|
||||
## 🚨 SYSTEM SUCCESS/FAILURE METRICS:
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- Step completed in full with required outputs
|
||||
|
||||
### ❌ SYSTEM FAILURE:
|
||||
|
||||
- Skipped sequence steps or missing outputs
|
||||
**Master Rule:** Skipping steps is FORBIDDEN.
|
||||
@@ -0,0 +1,254 @@
|
||||
---
|
||||
name: 'step-04-evaluate-and-score'
|
||||
description: 'Orchestrate adaptive NFR domain assessments (agent-team, subagent, or sequential)'
|
||||
nextStepFile: './step-04e-aggregate-nfr.md'
|
||||
---
|
||||
|
||||
# Step 4: Orchestrate Adaptive NFR Assessment
|
||||
|
||||
## STEP GOAL
|
||||
|
||||
Select execution mode deterministically, then assess NFR domains using agent-team, subagent, or sequential execution while preserving output contracts.
|
||||
|
||||
## MANDATORY EXECUTION RULES
|
||||
|
||||
- 📖 Read the entire step file before acting
|
||||
- ✅ Speak in `{communication_language}`
|
||||
- ✅ Resolve execution mode from config (`tea_execution_mode`, `tea_capability_probe`)
|
||||
- ✅ Apply fallback rules deterministically when requested mode is unsupported
|
||||
- ✅ Wait for required worker steps to complete
|
||||
- ❌ Do NOT skip capability checks when probing is enabled
|
||||
|
||||
---
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
- 💾 Wait for subagent outputs
|
||||
- 📖 Load the next step only when instructed
|
||||
|
||||
---
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
### 1. Prepare Execution Context
|
||||
|
||||
**Generate unique timestamp:**
|
||||
|
||||
```javascript
|
||||
const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
|
||||
```
|
||||
|
||||
**Prepare context:**
|
||||
|
||||
```javascript
|
||||
const parseBooleanFlag = (value, defaultValue = true) => {
|
||||
if (typeof value === 'string') {
|
||||
const normalized = value.trim().toLowerCase();
|
||||
if (['false', '0', 'off', 'no'].includes(normalized)) return false;
|
||||
if (['true', '1', 'on', 'yes'].includes(normalized)) return true;
|
||||
}
|
||||
if (value === undefined || value === null) return defaultValue;
|
||||
return Boolean(value);
|
||||
};
|
||||
|
||||
const subagentContext = {
|
||||
system_context: /* from Step 1 */,
|
||||
nfr_thresholds: /* from Step 2 */,
|
||||
evidence_gathered: /* from Step 3 */,
|
||||
config: {
|
||||
execution_mode: config.tea_execution_mode || 'auto', // "auto" | "subagent" | "agent-team" | "sequential"
|
||||
capability_probe: parseBooleanFlag(config.tea_capability_probe, true), // supports booleans and "false"/"true" strings
|
||||
},
|
||||
timestamp: timestamp
|
||||
};
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 2. Resolve Execution Mode with Capability Probe
|
||||
|
||||
```javascript
|
||||
const normalizeUserExecutionMode = (mode) => {
|
||||
if (typeof mode !== 'string') return null;
|
||||
const normalized = mode.trim().toLowerCase().replace(/[-_]/g, ' ').replace(/\s+/g, ' ');
|
||||
|
||||
if (normalized === 'auto') return 'auto';
|
||||
if (normalized === 'sequential') return 'sequential';
|
||||
if (normalized === 'subagent' || normalized === 'sub agent' || normalized === 'subagents' || normalized === 'sub agents') {
|
||||
return 'subagent';
|
||||
}
|
||||
if (normalized === 'agent team' || normalized === 'agent teams' || normalized === 'agentteam') {
|
||||
return 'agent-team';
|
||||
}
|
||||
|
||||
return null;
|
||||
};
|
||||
|
||||
const normalizeConfigExecutionMode = (mode) => {
|
||||
if (mode === 'subagent') return 'subagent';
|
||||
if (mode === 'auto' || mode === 'sequential' || mode === 'subagent' || mode === 'agent-team') {
|
||||
return mode;
|
||||
}
|
||||
return null;
|
||||
};
|
||||
|
||||
// Explicit user instruction in the active run takes priority over config.
|
||||
const explicitModeFromUser = normalizeUserExecutionMode(runtime.getExplicitExecutionModeHint?.() || null);
|
||||
|
||||
const requestedMode = explicitModeFromUser || normalizeConfigExecutionMode(subagentContext.config.execution_mode) || 'auto';
|
||||
const probeEnabled = subagentContext.config.capability_probe;
|
||||
|
||||
const supports = {
|
||||
subagent: false,
|
||||
agentTeam: false,
|
||||
};
|
||||
|
||||
if (probeEnabled) {
|
||||
supports.subagent = runtime.canLaunchSubagents?.() === true;
|
||||
supports.agentTeam = runtime.canLaunchAgentTeams?.() === true;
|
||||
}
|
||||
|
||||
let resolvedMode = requestedMode;
|
||||
|
||||
if (requestedMode === 'auto') {
|
||||
if (supports.agentTeam) resolvedMode = 'agent-team';
|
||||
else if (supports.subagent) resolvedMode = 'subagent';
|
||||
else resolvedMode = 'sequential';
|
||||
} else if (probeEnabled && requestedMode === 'agent-team' && !supports.agentTeam) {
|
||||
resolvedMode = supports.subagent ? 'subagent' : 'sequential';
|
||||
} else if (probeEnabled && requestedMode === 'subagent' && !supports.subagent) {
|
||||
resolvedMode = 'sequential';
|
||||
}
|
||||
|
||||
subagentContext.execution = {
|
||||
requestedMode,
|
||||
resolvedMode,
|
||||
probeEnabled,
|
||||
supports,
|
||||
};
|
||||
```
|
||||
|
||||
Resolution precedence:
|
||||
|
||||
1. Explicit user request in this run (`agent team` => `agent-team`; `subagent` => `subagent`; `sequential`; `auto`)
|
||||
2. `tea_execution_mode` from config
|
||||
3. Runtime capability fallback (when probing enabled)
|
||||
|
||||
If probing is disabled, honor the requested mode strictly. If that mode cannot be executed at runtime, fail with explicit error instead of silent fallback.
|
||||
|
||||
---
|
||||
|
||||
### 3. Dispatch 4 NFR Workers
|
||||
|
||||
**Subagent A: Security Assessment**
|
||||
|
||||
- File: `./step-04a-subagent-security.md`
|
||||
- Output: `/tmp/tea-nfr-security-${timestamp}.json`
|
||||
- Execution:
|
||||
- `agent-team` or `subagent`: launch non-blocking
|
||||
- `sequential`: run blocking and wait
|
||||
- Status: Running... ⟳
|
||||
|
||||
**Subagent B: Performance Assessment**
|
||||
|
||||
- File: `./step-04b-subagent-performance.md`
|
||||
- Output: `/tmp/tea-nfr-performance-${timestamp}.json`
|
||||
- Status: Running... ⟳
|
||||
|
||||
**Subagent C: Reliability Assessment**
|
||||
|
||||
- File: `./step-04c-subagent-reliability.md`
|
||||
- Output: `/tmp/tea-nfr-reliability-${timestamp}.json`
|
||||
- Status: Running... ⟳
|
||||
|
||||
**Subagent D: Scalability Assessment**
|
||||
|
||||
- File: `./step-04d-subagent-scalability.md`
|
||||
- Output: `/tmp/tea-nfr-scalability-${timestamp}.json`
|
||||
- Status: Running... ⟳
|
||||
|
||||
In `agent-team` and `subagent` modes, runtime decides worker scheduling and concurrency.
|
||||
|
||||
---
|
||||
|
||||
### 4. Wait for Expected Worker Completion
|
||||
|
||||
**If `resolvedMode` is `agent-team` or `subagent`:**
|
||||
|
||||
```
|
||||
⏳ Waiting for 4 NFR subagents to complete...
|
||||
├── Subagent A (Security): Running... ⟳
|
||||
├── Subagent B (Performance): Running... ⟳
|
||||
├── Subagent C (Reliability): Running... ⟳
|
||||
└── Subagent D (Scalability): Running... ⟳
|
||||
|
||||
[... time passes ...]
|
||||
|
||||
✅ All 4 NFR subagents completed!
|
||||
```
|
||||
|
||||
**If `resolvedMode` is `sequential`:**
|
||||
|
||||
```
|
||||
✅ Sequential mode: each worker already completed during dispatch.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 5. Verify All Outputs Exist
|
||||
|
||||
```javascript
|
||||
const outputs = ['security', 'performance', 'reliability', 'scalability'].map((domain) => `/tmp/tea-nfr-${domain}-${timestamp}.json`);
|
||||
|
||||
outputs.forEach((output) => {
|
||||
if (!fs.existsSync(output)) {
|
||||
throw new Error(`Subagent output missing: ${output}`);
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 6. Execution Report
|
||||
|
||||
```
|
||||
🚀 Performance Report:
|
||||
- Execution Mode: {resolvedMode}
|
||||
- Total Elapsed: ~mode-dependent
|
||||
- Parallel Gain: ~67% faster when mode is subagent/agent-team
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 7. Proceed to Aggregation
|
||||
|
||||
Load next step: `{nextStepFile}`
|
||||
|
||||
The aggregation step will:
|
||||
|
||||
- Read all 4 NFR domain outputs
|
||||
- Calculate overall risk level
|
||||
- Aggregate compliance status
|
||||
- Identify cross-domain risks
|
||||
- Generate executive summary
|
||||
|
||||
---
|
||||
|
||||
## EXIT CONDITION
|
||||
|
||||
Proceed when all 4 required worker steps completed and outputs exist.
|
||||
|
||||
---
|
||||
|
||||
## 🚨 SYSTEM SUCCESS METRICS
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- All required worker steps completed
|
||||
- Fallback behavior respected configuration and capability probe rules
|
||||
|
||||
### ❌ FAILURE:
|
||||
|
||||
- One or more subagents failed
|
||||
- Unsupported requested mode with probing disabled
|
||||
@@ -0,0 +1,138 @@
|
||||
---
|
||||
name: 'step-04a-subagent-security'
|
||||
description: 'Subagent: Security NFR assessment'
|
||||
subagent: true
|
||||
outputFile: '/tmp/tea-nfr-security-{{timestamp}}.json'
|
||||
---
|
||||
|
||||
# Subagent 4A: Security NFR Assessment
|
||||
|
||||
## SUBAGENT CONTEXT
|
||||
|
||||
This is an **isolated subagent** running in parallel with other NFR domain assessments.
|
||||
|
||||
**Your task:** Assess SECURITY NFR domain only.
|
||||
|
||||
---
|
||||
|
||||
## MANDATORY EXECUTION RULES
|
||||
|
||||
- ✅ Assess SECURITY only (not performance, reliability, scalability)
|
||||
- ✅ Output structured JSON to temp file
|
||||
- ❌ Do NOT assess other NFR domains
|
||||
|
||||
---
|
||||
|
||||
## SUBAGENT TASK
|
||||
|
||||
### 1. Security Assessment Categories
|
||||
|
||||
**Assess the following security dimensions:**
|
||||
|
||||
**A) Authentication & Authorization:**
|
||||
|
||||
- OAuth2/JWT implementation
|
||||
- Session management
|
||||
- Multi-factor authentication
|
||||
- Role-based access control (RBAC)
|
||||
|
||||
**B) Data Protection:**
|
||||
|
||||
- Encryption at rest
|
||||
- Encryption in transit (HTTPS/TLS)
|
||||
- Sensitive data handling (PII, passwords)
|
||||
- Database encryption
|
||||
|
||||
**C) Input Validation:**
|
||||
|
||||
- SQL injection prevention
|
||||
- XSS prevention
|
||||
- CSRF protection
|
||||
- Input sanitization
|
||||
|
||||
**D) API Security:**
|
||||
|
||||
- Rate limiting
|
||||
- API authentication
|
||||
- CORS configuration
|
||||
- Security headers
|
||||
|
||||
**E) Secrets Management:**
|
||||
|
||||
- Environment variables for secrets
|
||||
- No hardcoded credentials
|
||||
- Secret rotation policies
|
||||
- Key management systems
|
||||
|
||||
### 2. Risk Assessment
|
||||
|
||||
For each category, determine status:
|
||||
|
||||
- **PASS**: Properly implemented
|
||||
- **CONCERN**: Partially implemented or weak
|
||||
- **FAIL**: Not implemented or critical vulnerability
|
||||
- **N/A**: Not applicable to this system
|
||||
|
||||
### 3. Compliance Check
|
||||
|
||||
**Common compliance standards:**
|
||||
|
||||
- SOC2
|
||||
- GDPR
|
||||
- HIPAA
|
||||
- PCI-DSS
|
||||
- ISO 27001
|
||||
|
||||
---
|
||||
|
||||
## OUTPUT FORMAT
|
||||
|
||||
```json
|
||||
{
|
||||
"domain": "security",
|
||||
"risk_level": "MEDIUM",
|
||||
"findings": [
|
||||
{
|
||||
"category": "Authentication",
|
||||
"status": "PASS",
|
||||
"description": "OAuth2 with JWT tokens implemented",
|
||||
"evidence": ["src/auth/oauth.ts", "JWT refresh token rotation"],
|
||||
"recommendations": []
|
||||
},
|
||||
{
|
||||
"category": "Data Encryption",
|
||||
"status": "CONCERN",
|
||||
"description": "Database encryption at rest not enabled",
|
||||
"evidence": ["Database config shows no encryption"],
|
||||
"recommendations": ["Enable database encryption at rest", "Use AWS RDS encryption or equivalent", "Implement key rotation policy"]
|
||||
},
|
||||
{
|
||||
"category": "Input Validation",
|
||||
"status": "FAIL",
|
||||
"description": "SQL injection vulnerability in search endpoint",
|
||||
"evidence": ["src/api/search.ts:42 - direct SQL concatenation"],
|
||||
"recommendations": ["URGENT: Use parameterized queries", "Add input sanitization library", "Implement WAF rules"]
|
||||
}
|
||||
],
|
||||
"compliance": {
|
||||
"SOC2": "PARTIAL",
|
||||
"GDPR": "PASS",
|
||||
"HIPAA": "N/A",
|
||||
"PCI-DSS": "FAIL"
|
||||
},
|
||||
"priority_actions": [
|
||||
"Fix SQL injection vulnerability (URGENT)",
|
||||
"Enable database encryption within 30 days",
|
||||
"Implement rate limiting for all APIs"
|
||||
],
|
||||
"summary": "Security posture is MEDIUM risk with 1 critical vulnerability requiring immediate attention"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## EXIT CONDITION
|
||||
|
||||
Subagent completes when JSON output written to temp file.
|
||||
|
||||
**Subagent terminates here.**
|
||||
@@ -0,0 +1,84 @@
|
||||
---
|
||||
name: 'step-04b-subagent-performance'
|
||||
description: 'Subagent: Performance NFR assessment'
|
||||
subagent: true
|
||||
outputFile: '/tmp/tea-nfr-performance-{{timestamp}}.json'
|
||||
---
|
||||
|
||||
# Subagent 4B: Performance NFR Assessment
|
||||
|
||||
## SUBAGENT CONTEXT
|
||||
|
||||
This is an **isolated subagent** running in parallel with other NFR domain assessments.
|
||||
|
||||
**Your task:** Assess PERFORMANCE NFR domain only.
|
||||
|
||||
---
|
||||
|
||||
## SUBAGENT TASK
|
||||
|
||||
### 1. Performance Assessment Categories
|
||||
|
||||
**A) Response Times:**
|
||||
|
||||
- API response times (<200ms target)
|
||||
- Page load times (<2s target)
|
||||
- Time to interactive (<3s target)
|
||||
|
||||
**B) Throughput:**
|
||||
|
||||
- Requests per second capacity
|
||||
- Concurrent user support
|
||||
- Database query performance
|
||||
|
||||
**C) Resource Usage:**
|
||||
|
||||
- Memory consumption
|
||||
- CPU utilization
|
||||
- Database connection pooling
|
||||
|
||||
**D) Optimization:**
|
||||
|
||||
- Caching strategies
|
||||
- CDN usage
|
||||
- Code splitting/lazy loading
|
||||
- Database indexing
|
||||
|
||||
---
|
||||
|
||||
## OUTPUT FORMAT
|
||||
|
||||
```json
|
||||
{
|
||||
"domain": "performance",
|
||||
"risk_level": "LOW",
|
||||
"findings": [
|
||||
{
|
||||
"category": "Response Times",
|
||||
"status": "PASS",
|
||||
"description": "API endpoints respond in <150ms (P95)",
|
||||
"evidence": ["Load testing results show 140ms P95"],
|
||||
"recommendations": []
|
||||
},
|
||||
{
|
||||
"category": "Caching",
|
||||
"status": "CONCERN",
|
||||
"description": "No CDN for static assets",
|
||||
"evidence": ["Static files served from origin"],
|
||||
"recommendations": ["Implement CDN (CloudFront/Cloudflare)", "Cache static assets for 1 year"]
|
||||
}
|
||||
],
|
||||
"compliance": {
|
||||
"SLA_99.9": "PASS",
|
||||
"SLA_99.99": "CONCERN"
|
||||
},
|
||||
"priority_actions": ["Implement CDN for static assets", "Add database query caching for frequent reads"],
|
||||
"summary": "Performance is acceptable with minor optimization opportunities"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## EXIT CONDITION
|
||||
|
||||
Subagent completes when JSON output written to temp file.
|
||||
@@ -0,0 +1,85 @@
|
||||
---
|
||||
name: 'step-04c-subagent-reliability'
|
||||
description: 'Subagent: Reliability NFR assessment'
|
||||
subagent: true
|
||||
outputFile: '/tmp/tea-nfr-reliability-{{timestamp}}.json'
|
||||
---
|
||||
|
||||
# Subagent 4C: Reliability NFR Assessment
|
||||
|
||||
## SUBAGENT CONTEXT
|
||||
|
||||
This is an **isolated subagent** running in parallel with other NFR domain assessments.
|
||||
|
||||
**Your task:** Assess RELIABILITY NFR domain only.
|
||||
|
||||
---
|
||||
|
||||
## SUBAGENT TASK
|
||||
|
||||
### 1. Reliability Assessment Categories
|
||||
|
||||
**A) Error Handling:**
|
||||
|
||||
- Try-catch blocks for critical operations
|
||||
- Graceful degradation
|
||||
- Circuit breakers
|
||||
- Retry mechanisms
|
||||
|
||||
**B) Monitoring & Observability:**
|
||||
|
||||
- Logging implementation
|
||||
- Error tracking (Sentry/Datadog)
|
||||
- Health check endpoints
|
||||
- Alerting systems
|
||||
|
||||
**C) Fault Tolerance:**
|
||||
|
||||
- Database failover
|
||||
- Service redundancy
|
||||
- Backup strategies
|
||||
- Disaster recovery plan
|
||||
|
||||
**D) Uptime & Availability:**
|
||||
|
||||
- SLA targets
|
||||
- Historical uptime
|
||||
- Incident response
|
||||
|
||||
---
|
||||
|
||||
## OUTPUT FORMAT
|
||||
|
||||
```json
|
||||
{
|
||||
"domain": "reliability",
|
||||
"risk_level": "LOW",
|
||||
"findings": [
|
||||
{
|
||||
"category": "Error Handling",
|
||||
"status": "PASS",
|
||||
"description": "Comprehensive error handling with circuit breakers",
|
||||
"evidence": ["Circuit breaker pattern in src/services/", "Retry logic implemented"],
|
||||
"recommendations": []
|
||||
},
|
||||
{
|
||||
"category": "Monitoring",
|
||||
"status": "CONCERN",
|
||||
"description": "No APM (Application Performance Monitoring) tool",
|
||||
"evidence": ["Logging present but no distributed tracing"],
|
||||
"recommendations": ["Implement APM (Datadog/New Relic)", "Add distributed tracing"]
|
||||
}
|
||||
],
|
||||
"compliance": {
|
||||
"SLA_99.9": "PASS"
|
||||
},
|
||||
"priority_actions": ["Implement APM for better observability"],
|
||||
"summary": "Reliability is good with minor monitoring gaps"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## EXIT CONDITION
|
||||
|
||||
Subagent completes when JSON output written to temp file.
|
||||
@@ -0,0 +1,88 @@
|
||||
---
|
||||
name: 'step-04d-subagent-scalability'
|
||||
description: 'Subagent: Scalability NFR assessment'
|
||||
subagent: true
|
||||
outputFile: '/tmp/tea-nfr-scalability-{{timestamp}}.json'
|
||||
---
|
||||
|
||||
# Subagent 4D: Scalability NFR Assessment
|
||||
|
||||
## SUBAGENT CONTEXT
|
||||
|
||||
This is an **isolated subagent** running in parallel with other NFR domain assessments.
|
||||
|
||||
**Your task:** Assess SCALABILITY NFR domain only.
|
||||
|
||||
---
|
||||
|
||||
## SUBAGENT TASK
|
||||
|
||||
### 1. Scalability Assessment Categories
|
||||
|
||||
**A) Horizontal Scaling:**
|
||||
|
||||
- Stateless architecture
|
||||
- Load balancer configuration
|
||||
- Container orchestration (K8s)
|
||||
- Auto-scaling policies
|
||||
|
||||
**B) Vertical Scaling:**
|
||||
|
||||
- Resource allocation
|
||||
- Database size limits
|
||||
- Memory management
|
||||
- CPU optimization
|
||||
|
||||
**C) Data Scaling:**
|
||||
|
||||
- Database partitioning/sharding
|
||||
- Read replicas
|
||||
- Caching layers
|
||||
- Data archival strategy
|
||||
|
||||
**D) Traffic Handling:**
|
||||
|
||||
- CDN for static assets
|
||||
- Rate limiting
|
||||
- Queue systems for async work
|
||||
- WebSocket scaling
|
||||
|
||||
---
|
||||
|
||||
## OUTPUT FORMAT
|
||||
|
||||
```json
|
||||
{
|
||||
"domain": "scalability",
|
||||
"risk_level": "MEDIUM",
|
||||
"findings": [
|
||||
{
|
||||
"category": "Horizontal Scaling",
|
||||
"status": "PASS",
|
||||
"description": "Stateless architecture with container orchestration",
|
||||
"evidence": ["Docker + Kubernetes setup", "Auto-scaling configured"],
|
||||
"recommendations": []
|
||||
},
|
||||
{
|
||||
"category": "Data Scaling",
|
||||
"status": "CONCERN",
|
||||
"description": "No database sharding strategy for large data growth",
|
||||
"evidence": ["Single database instance", "No partitioning"],
|
||||
"recommendations": ["Plan database sharding strategy", "Implement read replicas", "Consider database clustering"]
|
||||
}
|
||||
],
|
||||
"compliance": {
|
||||
"1M_users": "PASS",
|
||||
"10M_users": "CONCERN",
|
||||
"100M_users": "FAIL"
|
||||
},
|
||||
"priority_actions": ["Design database sharding strategy for future growth", "Implement read replicas for read-heavy workloads"],
|
||||
"summary": "Scalability is good up to 1M users, concerns for 10M+ users"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## EXIT CONDITION
|
||||
|
||||
Subagent completes when JSON output written to temp file.
|
||||
@@ -0,0 +1,264 @@
|
||||
---
|
||||
name: 'step-04e-aggregate-nfr'
|
||||
description: 'Aggregate NFR domain assessments into executive summary'
|
||||
nextStepFile: './step-05-generate-report.md'
|
||||
outputFile: '{test_artifacts}/nfr-assessment.md'
|
||||
---
|
||||
|
||||
# Step 4E: Aggregate NFR Assessment Results
|
||||
|
||||
## STEP GOAL
|
||||
|
||||
Read outputs from 4 parallel NFR subagents, calculate overall risk level, aggregate compliance status, and identify cross-domain risks.
|
||||
|
||||
---
|
||||
|
||||
## MANDATORY EXECUTION RULES
|
||||
|
||||
- 📖 Read the entire step file before acting
|
||||
- ✅ Speak in `{communication_language}`
|
||||
- ✅ Read all 4 subagent outputs
|
||||
- ✅ Calculate overall risk level
|
||||
- ❌ Do NOT re-assess NFRs (use subagent outputs)
|
||||
|
||||
---
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
### 1. Read All Subagent Outputs
|
||||
|
||||
```javascript
|
||||
const domains = ['security', 'performance', 'reliability', 'scalability'];
|
||||
const assessments = {};
|
||||
|
||||
domains.forEach((domain) => {
|
||||
const outputPath = `/tmp/tea-nfr-${domain}-{{timestamp}}.json`;
|
||||
assessments[domain] = JSON.parse(fs.readFileSync(outputPath, 'utf8'));
|
||||
});
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 2. Calculate Overall Risk Level
|
||||
|
||||
**Risk hierarchy:** HIGH > MEDIUM > LOW > NONE
|
||||
|
||||
```javascript
|
||||
const riskLevels = { HIGH: 3, MEDIUM: 2, LOW: 1, NONE: 0 };
|
||||
const domainRisks = domains.map((d) => assessments[d].risk_level);
|
||||
const maxRiskValue = Math.max(...domainRisks.map((r) => riskLevels[r]));
|
||||
const overallRisk = Object.keys(riskLevels).find((k) => riskLevels[k] === maxRiskValue);
|
||||
```
|
||||
|
||||
**Risk assessment:**
|
||||
|
||||
- If ANY domain is HIGH → overall is HIGH
|
||||
- If ANY domain is MEDIUM (and none HIGH) → overall is MEDIUM
|
||||
- If ALL domains are LOW/NONE → overall is LOW
|
||||
|
||||
---
|
||||
|
||||
### 3. Aggregate Compliance Status
|
||||
|
||||
```javascript
|
||||
const allCompliance = {};
|
||||
|
||||
domains.forEach((domain) => {
|
||||
const compliance = assessments[domain].compliance;
|
||||
Object.entries(compliance).forEach(([standard, status]) => {
|
||||
if (!allCompliance[standard]) {
|
||||
allCompliance[standard] = [];
|
||||
}
|
||||
allCompliance[standard].push({ domain, status });
|
||||
});
|
||||
});
|
||||
|
||||
// Determine overall compliance per standard
|
||||
const complianceSummary = {};
|
||||
Object.entries(allCompliance).forEach(([standard, statuses]) => {
|
||||
const hasFail = statuses.some((s) => s.status === 'FAIL');
|
||||
const hasPartial = statuses.some((s) => s.status === 'PARTIAL' || s.status === 'CONCERN');
|
||||
|
||||
complianceSummary[standard] = hasFail ? 'FAIL' : hasPartial ? 'PARTIAL' : 'PASS';
|
||||
});
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 4. Identify Cross-Domain Risks
|
||||
|
||||
**Look for risks that span multiple domains:**
|
||||
|
||||
```javascript
|
||||
const crossDomainRisks = [];
|
||||
|
||||
// Example: Performance + Scalability issue
|
||||
const perfConcerns = assessments.performance.findings.filter((f) => f.status !== 'PASS');
|
||||
const scaleConcerns = assessments.scalability.findings.filter((f) => f.status !== 'PASS');
|
||||
if (perfConcerns.length > 0 && scaleConcerns.length > 0) {
|
||||
crossDomainRisks.push({
|
||||
domains: ['performance', 'scalability'],
|
||||
description: 'Performance issues may worsen under scale',
|
||||
impact: 'HIGH',
|
||||
});
|
||||
}
|
||||
|
||||
// Example: Security + Reliability issue
|
||||
const securityFails = assessments.security.findings.filter((f) => f.status === 'FAIL');
|
||||
const reliabilityConcerns = assessments.reliability.findings.filter((f) => f.status !== 'PASS');
|
||||
if (securityFails.length > 0 && reliabilityConcerns.length > 0) {
|
||||
crossDomainRisks.push({
|
||||
domains: ['security', 'reliability'],
|
||||
description: 'Security vulnerabilities may cause reliability incidents',
|
||||
impact: 'CRITICAL',
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 5. Aggregate Priority Actions
|
||||
|
||||
```javascript
|
||||
const allPriorityActions = domains.flatMap((domain) =>
|
||||
assessments[domain].priority_actions.map((action) => ({
|
||||
domain,
|
||||
action,
|
||||
urgency: assessments[domain].risk_level === 'HIGH' ? 'URGENT' : 'NORMAL',
|
||||
})),
|
||||
);
|
||||
|
||||
// Sort by urgency
|
||||
const prioritizedActions = allPriorityActions.sort((a, b) => (a.urgency === 'URGENT' ? -1 : 1));
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 6. Generate Executive Summary
|
||||
|
||||
```javascript
|
||||
const resolvedMode = subagentContext?.execution?.resolvedMode ?? 'unknown';
|
||||
const subagentExecutionLabel =
|
||||
resolvedMode === 'sequential'
|
||||
? 'SEQUENTIAL (4 NFR domains)'
|
||||
: resolvedMode === 'agent-team'
|
||||
? 'AGENT-TEAM (4 NFR domains)'
|
||||
: resolvedMode === 'subagent'
|
||||
? 'SUBAGENT (4 NFR domains)'
|
||||
: 'MODE-DEPENDENT (4 NFR domains)';
|
||||
|
||||
const performanceGainLabel =
|
||||
resolvedMode === 'sequential'
|
||||
? 'baseline (no parallel speedup)'
|
||||
: resolvedMode === 'agent-team' || resolvedMode === 'subagent'
|
||||
? '~67% faster than sequential'
|
||||
: 'mode-dependent';
|
||||
|
||||
const executiveSummary = {
|
||||
overall_risk: overallRisk,
|
||||
assessment_date: new Date().toISOString(),
|
||||
|
||||
domain_assessments: assessments,
|
||||
|
||||
compliance_summary: complianceSummary,
|
||||
|
||||
cross_domain_risks: crossDomainRisks,
|
||||
|
||||
priority_actions: prioritizedActions,
|
||||
|
||||
risk_breakdown: {
|
||||
security: assessments.security.risk_level,
|
||||
performance: assessments.performance.risk_level,
|
||||
reliability: assessments.reliability.risk_level,
|
||||
scalability: assessments.scalability.risk_level,
|
||||
},
|
||||
|
||||
subagent_execution: subagentExecutionLabel,
|
||||
performance_gain: performanceGainLabel,
|
||||
};
|
||||
|
||||
// Save for Step 5 (report generation)
|
||||
fs.writeFileSync('/tmp/tea-nfr-summary-{{timestamp}}.json', JSON.stringify(executiveSummary, null, 2), 'utf8');
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 7. Display Summary to User
|
||||
|
||||
```
|
||||
✅ NFR Assessment Complete ({subagentExecutionLabel})
|
||||
|
||||
🎯 Overall Risk Level: {overallRisk}
|
||||
|
||||
📊 Domain Risk Breakdown:
|
||||
- Security: {security_risk}
|
||||
- Performance: {performance_risk}
|
||||
- Reliability: {reliability_risk}
|
||||
- Scalability: {scalability_risk}
|
||||
|
||||
✅ Compliance Summary:
|
||||
{list standards with PASS/PARTIAL/FAIL}
|
||||
|
||||
⚠️ Cross-Domain Risks: {cross_domain_risk_count}
|
||||
|
||||
🎯 Priority Actions: {priority_action_count}
|
||||
|
||||
🚀 Performance: {performanceGainLabel}
|
||||
|
||||
✅ Ready for report generation (Step 5)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
### 8. Save Progress
|
||||
|
||||
**Save this step's accumulated work to `{outputFile}`.**
|
||||
|
||||
- **If `{outputFile}` does not exist** (first save), create it using the workflow template (if available) with YAML frontmatter:
|
||||
|
||||
```yaml
|
||||
---
|
||||
stepsCompleted: ['step-04e-aggregate-nfr']
|
||||
lastStep: 'step-04e-aggregate-nfr'
|
||||
lastSaved: '{date}'
|
||||
---
|
||||
```
|
||||
|
||||
Then write this step's output below the frontmatter.
|
||||
|
||||
- **If `{outputFile}` already exists**, update:
|
||||
- Add `'step-04e-aggregate-nfr'` to `stepsCompleted` array (only if not already present)
|
||||
- Set `lastStep: 'step-04e-aggregate-nfr'`
|
||||
- Set `lastSaved: '{date}'`
|
||||
- Append this step's output to the appropriate section of the document.
|
||||
|
||||
---
|
||||
|
||||
## EXIT CONDITION
|
||||
|
||||
Proceed to Step 5 when:
|
||||
|
||||
- ✅ All subagent outputs read
|
||||
- ✅ Overall risk calculated
|
||||
- ✅ Compliance aggregated
|
||||
- ✅ Summary saved
|
||||
- ✅ Progress saved to output document
|
||||
|
||||
Load next step: `{nextStepFile}`
|
||||
|
||||
---
|
||||
|
||||
## 🚨 SYSTEM SUCCESS METRICS
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- All 4 NFR domains aggregated correctly
|
||||
- Overall risk level determined
|
||||
- Executive summary complete
|
||||
|
||||
### ❌ FAILURE:
|
||||
|
||||
- Failed to read subagent outputs
|
||||
- Risk calculation incorrect
|
||||
@@ -0,0 +1,108 @@
|
||||
---
|
||||
name: 'step-05-generate-report'
|
||||
description: 'Create NFR report and validation summary'
|
||||
outputFile: '{test_artifacts}/nfr-assessment.md'
|
||||
---
|
||||
|
||||
# Step 5: Generate Report & Validate
|
||||
|
||||
## STEP GOAL
|
||||
|
||||
Produce the NFR assessment report and validate completeness.
|
||||
|
||||
## MANDATORY EXECUTION RULES
|
||||
|
||||
- 📖 Read the entire step file before acting
|
||||
- ✅ Speak in `{communication_language}`
|
||||
|
||||
---
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
- 💾 Record outputs before proceeding
|
||||
- 📖 Load the next step only when instructed
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: config, loaded artifacts, and knowledge fragments
|
||||
- Focus: this step's goal only
|
||||
- Limits: do not execute future steps
|
||||
- Dependencies: prior steps' outputs (if any)
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly. Do not skip, reorder, or improvise.
|
||||
|
||||
## 1. Report Generation
|
||||
|
||||
Use `nfr-report-template.md` to produce `{outputFile}` containing:
|
||||
|
||||
- Category results (PASS/CONCERNS/FAIL)
|
||||
- Evidence summary
|
||||
- Remediation actions
|
||||
- Gate-ready YAML snippet (if applicable)
|
||||
|
||||
---
|
||||
|
||||
## 2. Polish Output
|
||||
|
||||
Before finalizing, review the complete output document for quality:
|
||||
|
||||
1. **Remove duplication**: Progressive-append workflow may have created repeated sections — consolidate
|
||||
2. **Verify consistency**: Ensure terminology, risk scores, and references are consistent throughout
|
||||
3. **Check completeness**: All template sections should be populated or explicitly marked N/A
|
||||
4. **Format cleanup**: Ensure markdown formatting is clean (tables aligned, headers consistent, no orphaned references)
|
||||
|
||||
---
|
||||
|
||||
## 3. Validation
|
||||
|
||||
Validate against `checklist.md` and fix gaps.
|
||||
|
||||
- [ ] CLI sessions cleaned up (no orphaned browsers)
|
||||
|
||||
---
|
||||
|
||||
## 4. Save Progress
|
||||
|
||||
**Save this step's accumulated work to `{outputFile}`.**
|
||||
|
||||
- **If `{outputFile}` does not exist** (first save), create it using the workflow template (if available) with YAML frontmatter:
|
||||
|
||||
```yaml
|
||||
---
|
||||
stepsCompleted: ['step-05-generate-report']
|
||||
lastStep: 'step-05-generate-report'
|
||||
lastSaved: '{date}'
|
||||
---
|
||||
```
|
||||
|
||||
Then write this step's output below the frontmatter.
|
||||
|
||||
- **If `{outputFile}` already exists**, update:
|
||||
- Add `'step-05-generate-report'` to `stepsCompleted` array (only if not already present)
|
||||
- Set `lastStep: 'step-05-generate-report'`
|
||||
- Set `lastSaved: '{date}'`
|
||||
- Append this step's output to the appropriate section of the document.
|
||||
|
||||
---
|
||||
|
||||
## 5. Completion Summary
|
||||
|
||||
Report:
|
||||
|
||||
- Overall NFR status
|
||||
- Critical blockers or waivers needed
|
||||
- Next recommended workflow (`trace` or release gate)
|
||||
|
||||
## 🚨 SYSTEM SUCCESS/FAILURE METRICS:
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- Step completed in full with required outputs
|
||||
|
||||
### ❌ SYSTEM FAILURE:
|
||||
|
||||
- Skipped sequence steps or missing outputs
|
||||
**Master Rule:** Skipping steps is FORBIDDEN.
|
||||
@@ -0,0 +1,65 @@
|
||||
---
|
||||
name: 'step-01-assess'
|
||||
description: 'Load an existing output for editing'
|
||||
nextStepFile: './step-02-apply-edit.md'
|
||||
---
|
||||
|
||||
# Step 1: Assess Edit Target
|
||||
|
||||
## STEP GOAL:
|
||||
|
||||
Identify which output should be edited and load it.
|
||||
|
||||
## MANDATORY EXECUTION RULES (READ FIRST):
|
||||
|
||||
### Universal Rules:
|
||||
|
||||
- 📖 Read the complete step file before taking any action
|
||||
- ✅ Speak in `{communication_language}`
|
||||
|
||||
### Role Reinforcement:
|
||||
|
||||
- ✅ You are the Master Test Architect
|
||||
|
||||
### Step-Specific Rules:
|
||||
|
||||
- 🎯 Ask the user which output file to edit
|
||||
- 🚫 Do not edit until target is confirmed
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: existing outputs
|
||||
- Focus: select edit target
|
||||
- Limits: no edits yet
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly.
|
||||
|
||||
### 1. Identify Target
|
||||
|
||||
Ask the user to provide the output file path or select from known outputs.
|
||||
|
||||
### 2. Load Target
|
||||
|
||||
Read the provided output file in full.
|
||||
|
||||
### 3. Confirm
|
||||
|
||||
Confirm the target and proceed to edit.
|
||||
|
||||
Load next step: `{nextStepFile}`
|
||||
|
||||
## 🚨 SYSTEM SUCCESS/FAILURE METRICS:
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- Target identified and loaded
|
||||
|
||||
### ❌ SYSTEM FAILURE:
|
||||
|
||||
- Proceeding without a confirmed target
|
||||
@@ -0,0 +1,60 @@
|
||||
---
|
||||
name: 'step-02-apply-edit'
|
||||
description: 'Apply edits to the selected output'
|
||||
---
|
||||
|
||||
# Step 2: Apply Edits
|
||||
|
||||
## STEP GOAL:
|
||||
|
||||
Apply the requested edits to the selected output and confirm changes.
|
||||
|
||||
## MANDATORY EXECUTION RULES (READ FIRST):
|
||||
|
||||
### Universal Rules:
|
||||
|
||||
- 📖 Read the complete step file before taking any action
|
||||
- ✅ Speak in `{communication_language}`
|
||||
|
||||
### Role Reinforcement:
|
||||
|
||||
- ✅ You are the Master Test Architect
|
||||
|
||||
### Step-Specific Rules:
|
||||
|
||||
- 🎯 Only apply edits explicitly requested by the user
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: selected output and user changes
|
||||
- Focus: apply edits only
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly.
|
||||
|
||||
### 1. Confirm Requested Changes
|
||||
|
||||
Restate what will be changed and confirm.
|
||||
|
||||
### 2. Apply Changes
|
||||
|
||||
Update the output file accordingly.
|
||||
|
||||
### 3. Report
|
||||
|
||||
Summarize the edits applied.
|
||||
|
||||
## 🚨 SYSTEM SUCCESS/FAILURE METRICS:
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- Changes applied and confirmed
|
||||
|
||||
### ❌ SYSTEM FAILURE:
|
||||
|
||||
- Unconfirmed edits or missing update
|
||||
@@ -0,0 +1,67 @@
|
||||
---
|
||||
name: 'step-01-validate'
|
||||
description: 'Validate workflow outputs against checklist'
|
||||
outputFile: '{test_artifacts}/nfr-assess-validation-report.md'
|
||||
validationChecklist: '../checklist.md'
|
||||
---
|
||||
|
||||
# Step 1: Validate Outputs
|
||||
|
||||
## STEP GOAL:
|
||||
|
||||
Validate outputs using the workflow checklist and record findings.
|
||||
|
||||
## MANDATORY EXECUTION RULES (READ FIRST):
|
||||
|
||||
### Universal Rules:
|
||||
|
||||
- 📖 Read the complete step file before taking any action
|
||||
- ✅ Speak in `{communication_language}`
|
||||
|
||||
### Role Reinforcement:
|
||||
|
||||
- ✅ You are the Master Test Architect
|
||||
|
||||
### Step-Specific Rules:
|
||||
|
||||
- 🎯 Validate against `{validationChecklist}`
|
||||
- 🚫 Do not skip checks
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
- 💾 Write findings to `{outputFile}`
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: workflow outputs and checklist
|
||||
- Focus: validation only
|
||||
- Limits: do not modify outputs in this step
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly.
|
||||
|
||||
### 1. Load Checklist
|
||||
|
||||
Read `{validationChecklist}` and list all criteria.
|
||||
|
||||
### 2. Validate Outputs
|
||||
|
||||
Evaluate outputs against each checklist item.
|
||||
|
||||
### 3. Write Report
|
||||
|
||||
Write a validation report to `{outputFile}` with PASS/WARN/FAIL per section.
|
||||
|
||||
## 🚨 SYSTEM SUCCESS/FAILURE METRICS:
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- Validation report written
|
||||
- All checklist items evaluated
|
||||
|
||||
### ❌ SYSTEM FAILURE:
|
||||
|
||||
- Skipped checklist items
|
||||
- No report produced
|
||||
@@ -0,0 +1,73 @@
|
||||
---
|
||||
validationDate: 2026-01-27
|
||||
workflowName: testarch-nfr
|
||||
workflowPath: {project-root}/src/workflows/testarch/nfr-assess
|
||||
validationStatus: COMPLETE
|
||||
completionDate: 2026-01-27 10:03:10
|
||||
---
|
||||
|
||||
# Validation Report: testarch-nfr
|
||||
|
||||
**Validation Started:** 2026-01-27 09:50:21
|
||||
**Validator:** BMAD Workflow Validation System (Codex)
|
||||
**Standards Version:** BMAD Workflow Standards
|
||||
|
||||
## File Structure & Size
|
||||
|
||||
- workflow.md present: YES
|
||||
- instructions.md present: YES
|
||||
- workflow.yaml present: YES
|
||||
- step files found: 8
|
||||
|
||||
**Step File Sizes:**
|
||||
|
||||
- steps-c/step-01-load-context.md: 78 lines [GOOD]
|
||||
- steps-c/step-02-define-thresholds.md: 75 lines [GOOD]
|
||||
- steps-c/step-03-gather-evidence.md: 58 lines [GOOD]
|
||||
- steps-c/step-04-evaluate-and-score.md: 61 lines [GOOD]
|
||||
- steps-c/step-05-generate-report.md: 64 lines [GOOD]
|
||||
- steps-e/step-01-assess.md: 51 lines [GOOD]
|
||||
- steps-e/step-02-apply-edit.md: 46 lines [GOOD]
|
||||
- steps-v/step-01-validate.md: 53 lines [GOOD]
|
||||
- workflow-plan.md present: YES
|
||||
|
||||
## Frontmatter Validation
|
||||
|
||||
- No frontmatter violations found
|
||||
|
||||
## Critical Path Violations
|
||||
|
||||
- No {project-root} hardcoded paths detected in body
|
||||
- No dead relative links detected
|
||||
|
||||
## Menu Handling Validation
|
||||
|
||||
- No menu structures detected (linear step flow) [N/A]
|
||||
|
||||
## Step Type Validation
|
||||
|
||||
- Last step steps-v/step-01-validate.md has no nextStepFile (final step OK)
|
||||
- Step type validation assumes linear sequence (no branching/menu). Workflow-plan.md present for reference. [INFO]
|
||||
|
||||
## Output Format Validation
|
||||
|
||||
- Templates present: nfr-report-template.md
|
||||
- Steps with outputFile in frontmatter:
|
||||
- steps-c/step-05-generate-report.md
|
||||
- steps-v/step-01-validate.md
|
||||
|
||||
## Validation Design Check
|
||||
|
||||
- checklist.md present: YES
|
||||
- Validation steps folder (steps-v) present: YES
|
||||
|
||||
## Instruction Style Check
|
||||
|
||||
- All steps include STEP GOAL, MANDATORY EXECUTION RULES, EXECUTION PROTOCOLS, CONTEXT BOUNDARIES, and SUCCESS/FAILURE metrics
|
||||
|
||||
## Summary
|
||||
|
||||
- Validation completed: 2026-01-27 10:03:10
|
||||
- Critical issues: 0
|
||||
- Warnings: 0 (informational notes only)
|
||||
- Readiness: READY (manual review optional)
|
||||
@@ -0,0 +1,116 @@
|
||||
---
|
||||
validationDate: 2026-01-27
|
||||
workflowName: testarch-nfr
|
||||
workflowPath: {project-root}/src/workflows/testarch/nfr-assess
|
||||
validationStatus: COMPLETE
|
||||
completionDate: 2026-01-27 10:24:01
|
||||
---
|
||||
|
||||
# Validation Report: testarch-nfr
|
||||
|
||||
**Validation Started:** 2026-01-27 10:24:01
|
||||
**Validator:** BMAD Workflow Validation System (Codex)
|
||||
**Standards Version:** BMAD Workflow Standards
|
||||
|
||||
## File Structure & Size
|
||||
|
||||
- workflow.md present: YES
|
||||
- instructions.md present: YES
|
||||
- workflow.yaml present: YES
|
||||
- step files found: 8
|
||||
|
||||
**Step File Sizes:**
|
||||
|
||||
- steps-c/step-01-load-context.md: 77 lines [GOOD]
|
||||
- steps-c/step-02-define-thresholds.md: 74 lines [GOOD]
|
||||
- steps-c/step-03-gather-evidence.md: 57 lines [GOOD]
|
||||
- steps-c/step-04-evaluate-and-score.md: 60 lines [GOOD]
|
||||
- steps-c/step-05-generate-report.md: 63 lines [GOOD]
|
||||
- steps-e/step-01-assess.md: 50 lines [GOOD]
|
||||
- steps-e/step-02-apply-edit.md: 45 lines [GOOD]
|
||||
- steps-v/step-01-validate.md: 52 lines [GOOD]
|
||||
- workflow-plan.md present: YES
|
||||
|
||||
## Frontmatter Validation
|
||||
|
||||
- No frontmatter violations found
|
||||
|
||||
## Critical Path Violations
|
||||
|
||||
### Config Variables (Exceptions)
|
||||
|
||||
Standard BMAD config variables treated as valid exceptions: bmb_creations_output_folder, communication_language, document_output_language, output_folder, planning_artifacts, project-root, project_name, test_artifacts, user_name
|
||||
|
||||
- No {project-root} hardcoded paths detected in body
|
||||
|
||||
- No dead relative links detected
|
||||
|
||||
- No module path assumptions detected
|
||||
|
||||
**Status:** ✅ PASS - No critical violations
|
||||
|
||||
## Menu Handling Validation
|
||||
|
||||
- No menu structures detected (linear step flow) [N/A]
|
||||
|
||||
## Step Type Validation
|
||||
|
||||
- steps-c/step-01-load-context.md: Init [PASS]
|
||||
- steps-c/step-02-define-thresholds.md: Middle [PASS]
|
||||
- steps-c/step-03-gather-evidence.md: Middle [PASS]
|
||||
- steps-c/step-04-evaluate-and-score.md: Middle [PASS]
|
||||
- steps-c/step-05-generate-report.md: Final [PASS]
|
||||
- Step type validation assumes linear sequence (no branching/menu). Workflow-plan.md present for reference. [INFO]
|
||||
|
||||
## Output Format Validation
|
||||
|
||||
- Templates present: nfr-report-template.md
|
||||
- Steps with outputFile in frontmatter:
|
||||
- steps-c/step-05-generate-report.md
|
||||
- steps-v/step-01-validate.md
|
||||
- checklist.md present: YES
|
||||
|
||||
## Validation Design Check
|
||||
|
||||
- Validation steps folder (steps-v) present: YES
|
||||
- Validation step(s) present: step-01-validate.md
|
||||
- Validation steps reference checklist data and auto-proceed
|
||||
|
||||
## Instruction Style Check
|
||||
|
||||
- Instruction style: Prescriptive (appropriate for TEA quality/compliance workflows)
|
||||
- Steps emphasize mandatory sequence, explicit success/failure metrics, and risk-based guidance
|
||||
|
||||
## Collaborative Experience Check
|
||||
|
||||
- Overall facilitation quality: GOOD
|
||||
- Steps use progressive prompts and clear role reinforcement; no laundry-list interrogation detected
|
||||
- Flow progression is clear and aligned to workflow goals
|
||||
|
||||
## Subagent Optimization Opportunities
|
||||
|
||||
- No high-priority subagent optimizations identified; workflow already uses step-file architecture
|
||||
- Pattern 1 (grep/regex): N/A for most steps
|
||||
- Pattern 2 (per-file analysis): already aligned to validation structure
|
||||
- Pattern 3 (data ops): minimal data file loads
|
||||
- Pattern 4 (parallel): optional for validation only
|
||||
|
||||
## Cohesive Review
|
||||
|
||||
- Overall assessment: GOOD
|
||||
- Flow is linear, goals are clear, and outputs map to TEA artifacts
|
||||
- Voice and tone consistent with Test Architect persona
|
||||
- Recommendation: READY (minor refinements optional)
|
||||
|
||||
## Plan Quality Validation
|
||||
|
||||
- Plan file present: workflow-plan.md
|
||||
- Planned steps found: 8 (all implemented)
|
||||
- Plan implementation status: Fully Implemented
|
||||
|
||||
## Summary
|
||||
|
||||
- Validation completed: 2026-01-27 10:24:01
|
||||
- Critical issues: 0
|
||||
- Warnings: 0 (informational notes only)
|
||||
- Readiness: READY (manual review optional)
|
||||
19
_bmad/tea/workflows/testarch/nfr-assess/workflow-plan.md
Normal file
19
_bmad/tea/workflows/testarch/nfr-assess/workflow-plan.md
Normal file
@@ -0,0 +1,19 @@
|
||||
# Workflow Plan: testarch-nfr
|
||||
|
||||
## Create Mode (steps-c)
|
||||
- step-01-load-context.md
|
||||
|
||||
- step-02-define-thresholds.md
|
||||
- step-03-gather-evidence.md
|
||||
- step-04-evaluate-and-score.md
|
||||
- step-05-generate-report.md
|
||||
|
||||
## Validate Mode (steps-v)
|
||||
- step-01-validate.md
|
||||
|
||||
## Edit Mode (steps-e)
|
||||
- step-01-assess.md
|
||||
- step-02-apply-edit.md
|
||||
|
||||
## Outputs
|
||||
- {test_artifacts}/nfr-assessment.md
|
||||
41
_bmad/tea/workflows/testarch/nfr-assess/workflow.md
Normal file
41
_bmad/tea/workflows/testarch/nfr-assess/workflow.md
Normal file
@@ -0,0 +1,41 @@
|
||||
---
|
||||
name: testarch-nfr
|
||||
description: Assess NFRs like performance security and reliability. Use when user says 'lets assess NFRs' or 'I want to evaluate non-functional requirements'
|
||||
web_bundle: true
|
||||
---
|
||||
|
||||
# Non-Functional Requirements Assessment
|
||||
|
||||
**Goal:** Assess non-functional requirements (performance, security, reliability, maintainability) before release with evidence-based validation
|
||||
|
||||
**Role:** You are the Master Test Architect.
|
||||
|
||||
---
|
||||
|
||||
## WORKFLOW ARCHITECTURE
|
||||
|
||||
This workflow uses **tri-modal step-file architecture**:
|
||||
|
||||
- **Create mode (steps-c/)**: primary execution flow
|
||||
- **Validate mode (steps-v/)**: validation against checklist
|
||||
- **Edit mode (steps-e/)**: revise existing outputs
|
||||
|
||||
---
|
||||
|
||||
## INITIALIZATION SEQUENCE
|
||||
|
||||
### 1. Mode Determination
|
||||
|
||||
"Welcome to the workflow. What would you like to do?"
|
||||
|
||||
- **[C] Create** — Run the workflow
|
||||
- **[R] Resume** — Resume an interrupted workflow
|
||||
- **[V] Validate** — Validate existing outputs
|
||||
- **[E] Edit** — Edit existing outputs
|
||||
|
||||
### 2. Route to First Step
|
||||
|
||||
- **If C:** Load `steps-c/step-01-load-context.md`
|
||||
- **If R:** Load `steps-c/step-01b-resume.md`
|
||||
- **If V:** Load `steps-v/step-01-validate.md`
|
||||
- **If E:** Load `steps-e/step-01-assess.md`
|
||||
48
_bmad/tea/workflows/testarch/nfr-assess/workflow.yaml
Normal file
48
_bmad/tea/workflows/testarch/nfr-assess/workflow.yaml
Normal file
@@ -0,0 +1,48 @@
|
||||
# Test Architect workflow: nfr-assess
|
||||
name: testarch-nfr
|
||||
# prettier-ignore
|
||||
description: 'Assess NFRs like performance security and reliability. Use when the user says "lets assess NFRs" or "I want to evaluate non-functional requirements"'
|
||||
|
||||
# Critical variables from config
|
||||
config_source: "{project-root}/_bmad/tea/config.yaml"
|
||||
output_folder: "{config_source}:output_folder"
|
||||
test_artifacts: "{config_source}:test_artifacts"
|
||||
user_name: "{config_source}:user_name"
|
||||
communication_language: "{config_source}:communication_language"
|
||||
document_output_language: "{config_source}:document_output_language"
|
||||
date: system-generated
|
||||
|
||||
# Workflow components
|
||||
installed_path: "{project-root}/_bmad/tea/workflows/testarch/nfr-assess"
|
||||
instructions: "{installed_path}/instructions.md"
|
||||
validation: "{installed_path}/checklist.md"
|
||||
template: "{installed_path}/nfr-report-template.md"
|
||||
|
||||
# Variables and inputs
|
||||
variables:
|
||||
# NFR category assessment (defaults to all categories)
|
||||
custom_nfr_categories: "" # Optional additional categories beyond standard (security, performance, reliability, maintainability)
|
||||
|
||||
# Output configuration
|
||||
default_output_file: "{test_artifacts}/nfr-assessment.md"
|
||||
|
||||
# Required tools
|
||||
required_tools:
|
||||
- read_file # Read story, test results, metrics, logs, BMad artifacts
|
||||
- write_file # Create NFR assessment, gate YAML, evidence checklist
|
||||
- list_files # Discover test results, metrics, logs
|
||||
- search_repo # Find NFR-related tests and evidence
|
||||
- glob # Find result files matching patterns
|
||||
|
||||
tags:
|
||||
- qa
|
||||
- nfr
|
||||
- test-architect
|
||||
- performance
|
||||
- security
|
||||
- reliability
|
||||
|
||||
execution_hints:
|
||||
interactive: false # Minimize prompts
|
||||
autonomous: true # Proceed without user input unless blocked
|
||||
iterative: true
|
||||
197
_bmad/tea/workflows/testarch/teach-me-testing/checklist.md
Normal file
197
_bmad/tea/workflows/testarch/teach-me-testing/checklist.md
Normal file
@@ -0,0 +1,197 @@
|
||||
# Teach Me Testing - Quality Checklist
|
||||
|
||||
## Workflow Quality Standards
|
||||
|
||||
Use this checklist to validate the teaching workflow meets quality standards.
|
||||
|
||||
---
|
||||
|
||||
## Foundation Quality
|
||||
|
||||
- [ ] **workflow.md** exists with proper frontmatter
|
||||
- [ ] Tri-modal routing logic present (Create/Edit/Validate)
|
||||
- [ ] Configuration loading references correct module (TEA)
|
||||
- [ ] First step path correct (`./steps-c/step-01-init.md`)
|
||||
- [ ] Folder structure complete (steps-c/, steps-e/, steps-v/, data/, templates/)
|
||||
|
||||
---
|
||||
|
||||
## Template Quality
|
||||
|
||||
- [ ] **progress-template.yaml** has complete schema
|
||||
- [ ] All 7 sessions defined with proper structure
|
||||
- [ ] Session status tracking fields present (not-started/in-progress/completed)
|
||||
- [ ] stepsCompleted array for continuation tracking
|
||||
- [ ] **session-notes-template.md** has all required sections
|
||||
- [ ] **certificate-template.md** includes all 7 sessions
|
||||
|
||||
---
|
||||
|
||||
## Step File Quality (CREATE mode)
|
||||
|
||||
### Initialization Steps
|
||||
|
||||
- [ ] **step-01-init.md** checks for existing progress file
|
||||
- [ ] Continuation detection logic works correctly
|
||||
- [ ] **step-01b-continue.md** loads progress and routes to session menu
|
||||
- [ ] Progress dashboard displays completion status
|
||||
|
||||
### Assessment Step
|
||||
|
||||
- [ ] **step-02-assess.md** gathers role, experience, goals
|
||||
- [ ] Validation for role (QA/Dev/Lead/VP)
|
||||
- [ ] Validation for experience (beginner/intermediate/experienced)
|
||||
- [ ] Assessment data written to progress file
|
||||
|
||||
### Session Menu Hub
|
||||
|
||||
- [ ] **step-03-session-menu.md** displays all 7 sessions
|
||||
- [ ] Completion indicators shown (✓ completed, 🔄 in-progress, ⬜ not-started)
|
||||
- [ ] Branching logic routes to selected session (1-7)
|
||||
- [ ] Exit logic (X) routes to completion if all done, otherwise saves and exits
|
||||
|
||||
### Session Steps (1-7)
|
||||
|
||||
- [ ] Each session loads relevant TEA docs just-in-time
|
||||
- [ ] Teaching content presented (mostly autonomous)
|
||||
- [ ] Quiz validation with ≥70% threshold
|
||||
- [ ] Session notes artifact generated
|
||||
- [ ] Progress file updated (status, score, artifact path)
|
||||
- [ ] Returns to session menu hub after completion
|
||||
|
||||
### Completion Step
|
||||
|
||||
- [ ] **step-05-completion.md** verifies all 7 sessions complete
|
||||
- [ ] Certificate generated with accurate data
|
||||
- [ ] Final progress file update (certificate_generated: true)
|
||||
- [ ] Congratulations message shown
|
||||
|
||||
---
|
||||
|
||||
## Data File Quality
|
||||
|
||||
- [ ] **curriculum.yaml** defines all 7 sessions
|
||||
- [ ] **role-paths.yaml** maps role customizations
|
||||
- [ ] **session-content-map.yaml** references TEA docs/fragments/URLs correctly
|
||||
- [ ] **quiz-questions.yaml** has questions for all sessions
|
||||
- [ ] **tea-resources-index.yaml** has complete documentation index
|
||||
|
||||
---
|
||||
|
||||
## Content Quality
|
||||
|
||||
### TEA Documentation Integration
|
||||
|
||||
- [ ] Local file paths correct (`/docs/*.md`, `/src/testarch/knowledge/*.md`)
|
||||
- [ ] Online URLs correct (<https://bmad-code-org.github.io/...>)
|
||||
- [ ] GitHub fragment links correct
|
||||
- [ ] Triple reference system (local + online + GitHub) implemented
|
||||
|
||||
### Role-Based Content
|
||||
|
||||
- [ ] QA examples present (practical testing focus)
|
||||
- [ ] Dev examples present (integration/TDD focus)
|
||||
- [ ] Lead examples present (architecture/patterns focus)
|
||||
- [ ] VP examples present (strategy/metrics focus)
|
||||
|
||||
### Quiz Quality
|
||||
|
||||
- [ ] Questions test understanding, not memorization
|
||||
- [ ] 3-5 questions per session
|
||||
- [ ] Mix of difficulty levels
|
||||
- [ ] Clear correct answers with explanations
|
||||
|
||||
---
|
||||
|
||||
## Error Handling
|
||||
|
||||
- [ ] Corrupted progress file detection
|
||||
- [ ] Backup and recovery options
|
||||
- [ ] Missing TEA docs fallback (Web-Browsing)
|
||||
- [ ] Quiz failure recovery (review or continue)
|
||||
- [ ] Session interruption handling (auto-save)
|
||||
|
||||
---
|
||||
|
||||
## User Experience
|
||||
|
||||
- [ ] Clear navigation instructions
|
||||
- [ ] Progress visibility (completion percentage, next recommended)
|
||||
- [ ] Auto-save after each session
|
||||
- [ ] Resume capability works seamlessly
|
||||
- [ ] Exit options clear at all decision points
|
||||
|
||||
---
|
||||
|
||||
## State Management
|
||||
|
||||
- [ ] stepsCompleted array updated correctly
|
||||
- [ ] Session tracking accurate (status, dates, scores)
|
||||
- [ ] Completion percentage calculated correctly
|
||||
- [ ] Next recommended session logic works
|
||||
- [ ] lastStep and lastContinued timestamps updated
|
||||
|
||||
---
|
||||
|
||||
## Validation Mode
|
||||
|
||||
- [ ] **step-v-01-validate.md** checks all quality standards
|
||||
- [ ] Generates validation report
|
||||
- [ ] Identifies issues clearly
|
||||
- [ ] Provides remediation suggestions
|
||||
|
||||
---
|
||||
|
||||
## Edit Mode
|
||||
|
||||
- [ ] **step-e-01-assess-workflow.md** identifies what to edit
|
||||
- [ ] **step-e-02-apply-edits.md** applies modifications safely
|
||||
- [ ] Preserves workflow integrity during edits
|
||||
|
||||
---
|
||||
|
||||
## Documentation
|
||||
|
||||
- [ ] **instructions.md** clear and complete
|
||||
- [ ] **checklist.md** (this file) comprehensive
|
||||
- [ ] README (if present) accurate
|
||||
- [ ] Inline comments in complex logic
|
||||
|
||||
---
|
||||
|
||||
## Performance
|
||||
|
||||
- [ ] Just-in-time loading (not loading all docs upfront)
|
||||
- [ ] Session steps complete in reasonable time (<5 min)
|
||||
- [ ] Quiz validation fast (<1 min)
|
||||
- [ ] Progress file writes efficient
|
||||
|
||||
---
|
||||
|
||||
## Security
|
||||
|
||||
- [ ] No hardcoded credentials
|
||||
- [ ] File paths use variables
|
||||
- [ ] Progress files private to user
|
||||
- [ ] No sensitive data in session notes
|
||||
|
||||
---
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
✅ **Workflow is ready for deployment when:**
|
||||
|
||||
- All checkboxes above are checked
|
||||
- All step files exist and follow standards
|
||||
- All templates present and correct
|
||||
- Data files complete and accurate
|
||||
- Error handling robust
|
||||
- User experience smooth
|
||||
- Documentation complete
|
||||
|
||||
---
|
||||
|
||||
**Validation Date:** **\*\***\_\_\_**\*\***
|
||||
**Validated By:** **\*\***\_\_\_**\*\***
|
||||
**Issues Found:** **\*\***\_\_\_**\*\***
|
||||
**Status:** ⬜ Ready for Production | ⬜ Needs Revisions
|
||||
@@ -0,0 +1,129 @@
|
||||
# TEA Academy Curriculum Structure
|
||||
# Defines the 7-session learning path with objectives and content mappings
|
||||
|
||||
sessions:
|
||||
- id: session-01-quickstart
|
||||
name: "Quick Start"
|
||||
duration: "30 min"
|
||||
difficulty: beginner
|
||||
objective: "Get immediate value by seeing TEA in action"
|
||||
description: "TEA Lite intro, run automate workflow, understand engagement models"
|
||||
recommended_for:
|
||||
- beginner
|
||||
- intermediate
|
||||
- experienced
|
||||
prerequisites: []
|
||||
|
||||
- id: session-02-concepts
|
||||
name: "Core Concepts"
|
||||
duration: "45 min"
|
||||
difficulty: beginner
|
||||
objective: "Understand WHY behind TEA principles"
|
||||
description: "Risk-based testing, DoD, testing as engineering philosophy"
|
||||
recommended_for:
|
||||
- beginner
|
||||
- intermediate
|
||||
prerequisites: []
|
||||
|
||||
- id: session-03-architecture
|
||||
name: "Architecture & Patterns"
|
||||
duration: "60 min"
|
||||
difficulty: intermediate
|
||||
objective: "Understand TEA patterns and architecture"
|
||||
description: "Fixtures, network-first patterns, data factories, step-file architecture"
|
||||
recommended_for:
|
||||
- intermediate
|
||||
- experienced
|
||||
prerequisites:
|
||||
- session-02-concepts
|
||||
|
||||
- id: session-04-test-design
|
||||
name: "Test Design"
|
||||
duration: "60 min"
|
||||
difficulty: intermediate
|
||||
objective: "Learn risk assessment and coverage planning"
|
||||
description: "Test Design workflow, risk/testability assessment, coverage planning"
|
||||
recommended_for:
|
||||
- intermediate
|
||||
- experienced
|
||||
prerequisites:
|
||||
- session-02-concepts
|
||||
|
||||
- id: session-05-atdd-automate
|
||||
name: "ATDD & Automate"
|
||||
duration: "60 min"
|
||||
difficulty: intermediate
|
||||
objective: "Generate tests with TDD red-green approach"
|
||||
description: "ATDD workflow (red phase), Automate workflow, component TDD, API testing"
|
||||
recommended_for:
|
||||
- intermediate
|
||||
- experienced
|
||||
prerequisites:
|
||||
- session-02-concepts
|
||||
|
||||
- id: session-06-quality-trace
|
||||
name: "Quality & Trace"
|
||||
duration: "45 min"
|
||||
difficulty: intermediate
|
||||
objective: "Audit quality and ensure traceability"
|
||||
description: "Test Review (5 dimensions), Trace workflow, quality metrics"
|
||||
recommended_for:
|
||||
- intermediate
|
||||
- experienced
|
||||
prerequisites:
|
||||
- session-02-concepts
|
||||
|
||||
- id: session-07-advanced
|
||||
name: "Advanced Patterns"
|
||||
duration: "ongoing"
|
||||
difficulty: advanced
|
||||
objective: "Deep-dive into specific knowledge fragments"
|
||||
description: "Menu-driven exploration of 35 knowledge fragments organized by category"
|
||||
recommended_for:
|
||||
- experienced
|
||||
prerequisites: []
|
||||
|
||||
# Learning Paths by Experience Level
|
||||
learning_paths:
|
||||
beginner:
|
||||
recommended_sequence:
|
||||
- session-01-quickstart
|
||||
- session-02-concepts
|
||||
- session-03-architecture
|
||||
- session-04-test-design
|
||||
- session-05-atdd-automate
|
||||
- session-06-quality-trace
|
||||
- session-07-advanced
|
||||
skip_optional: []
|
||||
|
||||
intermediate:
|
||||
recommended_sequence:
|
||||
- session-01-quickstart
|
||||
- session-02-concepts
|
||||
- session-03-architecture
|
||||
- session-04-test-design
|
||||
- session-05-atdd-automate
|
||||
- session-06-quality-trace
|
||||
- session-07-advanced
|
||||
skip_optional:
|
||||
- session-01-quickstart # Can skip if already familiar
|
||||
certificate_eligible_if_skipped: false
|
||||
|
||||
experienced:
|
||||
recommended_sequence:
|
||||
- session-02-concepts
|
||||
- session-03-architecture
|
||||
- session-04-test-design
|
||||
- session-05-atdd-automate
|
||||
- session-06-quality-trace
|
||||
- session-07-advanced
|
||||
skip_optional:
|
||||
- session-01-quickstart
|
||||
certificate_eligible_if_skipped: false
|
||||
|
||||
# Completion Requirements
|
||||
completion:
|
||||
minimum_sessions: 7 # All sessions required for certificate
|
||||
passing_score: 70 # Minimum quiz score to pass session
|
||||
average_score_threshold: 70 # Minimum average for certificate
|
||||
certificate_note: "Certificate eligibility requires completion.minimum_sessions. If intermediate.skip_optional or experienced.skip_optional sessions are skipped, certificate eligibility is forfeited."
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user