Test Automation

#!/usr/bin/env -S ai --sonnet --skip
Run the test suite for this project. Report which tests passed and which
failed. If any tests fail, explain the root cause.

--skip

# Make it executable
chmod +x run-tests.md

# Run in any project directory
cd ~/projects/my-app
./run-tests.md

[AI Runner] Using: Claude Code + Claude Pro
[AI Runner] Model: Sonnet 5

I'll run the test suite using npm test.

✓ 45 tests passed
✗ 3 tests failed:

1. UserService.createUser - validation error
   File: src/services/user.service.test.ts:42
   Root cause: Missing email validation in createUser function
   The test expects validation but the implementation skips it

2. API /auth/login - 500 error
   File: src/api/auth.test.ts:78  
   Root cause: Database connection not mocked in test setup
   The test tries to connect to a real database

3. Utils.parseDate - timezone handling
   File: src/utils/date.test.ts:91
   Root cause: Date parsing assumes UTC but test runs in local timezone

./run-tests.md > test-report.txt

# Use Haiku for faster/cheaper analysis
ai --haiku run-tests.md

# Use AWS Bedrock
ai --aws --sonnet run-tests.md

# Use local Ollama (free!)
ai --ollama run-tests.md

# .git/hooks/pre-commit
#!/bin/bash
echo "Running test suite..."
if ! ai --haiku --skip << 'EOF' > /tmp/test-results.txt; then
Run the test suite. Only output PASS or FAIL and a count.
EOF
    cat /tmp/test-results.txt
    echo "Tests failed. Commit aborted."
    exit 1
fi
echo "All tests passed."

# .github/workflows/test.yml
name: Test with AI Analysis
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Setup Node
        uses: actions/setup-node@v3
        with:
          node-version: '20'
      
      - name: Install AIRun
        run: |
          curl -fsSL https://claude.ai/install.sh | bash
          git clone https://github.com/andisearch/airun.git
          cd airun && ./setup.sh
      
      - name: Run tests with AI analysis
        run: |
          ai --apikey --sonnet --skip << 'EOF' > test-report.md
          Run the test suite. Report pass/fail counts.
          If any tests fail, analyze root causes and suggest fixes.
          EOF
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
      
      - name: Upload report
        uses: actions/upload-artifact@v3
        with:
          name: test-analysis
          path: test-report.md

# .gitlab-ci.yml
test:
  stage: test
  image: node:20
  script:
    - curl -fsSL https://claude.ai/install.sh | bash
    - git clone https://github.com/andisearch/airun.git
    - cd airun && ./setup.sh && cd ..
    - |
      ai --apikey --sonnet --skip << 'EOF' > test-report.md
      Run the test suite. Report pass/fail counts and analyze failures.
      EOF
  artifacts:
    paths:
      - test-report.md
  variables:
    ANTHROPIC_API_KEY: $ANTHROPIC_API_KEY

#!/bin/bash
# nightly-test-analysis.sh

cd ~/projects/my-app

ai --apikey --opus --skip << 'EOF' > "test-report-$(date +%Y-%m-%d).md"
Run the full test suite including integration tests.

For each failure:
1. Identify the root cause
2. Check git history for recent changes to related files
3. Suggest specific fixes with code snippets
4. Rate the severity (critical/high/medium/low)

Summarize test health trends if you can access previous reports.
EOF

# Email the report
mail -s "Nightly Test Report" dev-team@company.com < "test-report-$(date +%Y-%m-%d).md"

#!/usr/bin/env -S ai --sonnet --skip
Run the test suite. If any tests fail:

1. Analyze the root cause
2. Fix the issue in the source code
3. Run tests again to verify the fix
4. Report what you changed

Do NOT commit changes. Just fix and verify.

#!/usr/bin/env -S ai --sonnet --skip
Run only unit tests (npm run test:unit).
Report pass/fail counts and analyze failures.

#!/usr/bin/env -S ai --sonnet --skip
Run integration tests:
1. Start docker-compose services
2. Wait for services to be healthy
3. Run npm run test:integration
4. Stop services when done

Analyze any failures.

#!/usr/bin/env -S ai --sonnet --skip
Run performance tests:
1. Execute npm run test:perf
2. Compare results to previous baseline in perf-baseline.json
3. Flag any regressions >10%
4. Identify the slowest tests

#!/usr/bin/env -S ai --sonnet --skip
Run tests with coverage:
1. Execute npm run test:coverage
2. Report overall coverage percentage
3. List files with <80% coverage
4. Suggest priority areas for new tests

#!/usr/bin/env -S ai --sonnet --skip --live
Run the test suite. Print a status update after each test file completes.
Finally, report pass/fail counts and analyze failures.

#!/usr/bin/env -S ai --sonnet --allowedTools 'Bash(npm test)' 'Read'
Run the test suite. Report results but do not modify any files.

test:
  image: node:20
  script:
    - ai --skip test-script.md

--allowedTools 'Bash(npm test)' 'Bash(pytest)' 'Read'