ci: Add comprehensive automated testing workflows and documentation

.github/TESTING.md

@@ -0,0 +1,341 @@
+# Testing Guide
+
+This document describes the testing infrastructure for Claude How To.
+
+## Overview
+
+The project uses GitHub Actions to automatically run tests on every push and pull request. Tests cover:
+
+- **Unit Tests**: Python tests using pytest
+- **Code Quality**: Linting and formatting with Ruff
+- **Security**: Vulnerability scanning with Bandit
+- **Type Checking**: Static type analysis with mypy
+- **Build Verification**: EPUB generation test
+
+## Running Tests Locally
+
+### Prerequisites
+
+```bash
+# Install uv (fast Python package manager)
+pip install uv
+
+# Or on macOS with Homebrew
+brew install uv
+```
+
+### Setup Environment
+
+```bash
+# Clone the repository
+git clone https://github.com/luongnv89/claude-howto.git
+cd claude-howto
+
+# Create virtual environment
+uv venv
+
+# Activate it
+source .venv/bin/activate  # macOS/Linux
+# or
+.venv\Scripts\activate     # Windows
+
+# Install development dependencies
+uv pip install -r requirements-dev.txt
+```
+
+### Run Tests
+
+```bash
+# Run all unit tests
+pytest scripts/tests/ -v
+
+# Run tests with coverage
+pytest scripts/tests/ -v --cov=scripts --cov-report=html
+
+# Run specific test file
+pytest scripts/tests/test_build_epub.py -v
+
+# Run specific test function
+pytest scripts/tests/test_build_epub.py::test_function_name -v
+
+# Run tests in watch mode (requires pytest-watch)
+ptw scripts/tests/
+```
+
+### Run Linting
+
+```bash
+# Check code formatting
+ruff format --check scripts/
+
+# Auto-fix formatting issues
+ruff format scripts/
+
+# Run linter
+ruff check scripts/
+
+# Auto-fix linter issues
+ruff check --fix scripts/
+```
+
+### Run Security Scan
+
+```bash
+# Run Bandit security scan
+bandit -c pyproject.toml -r scripts/ --exclude scripts/tests/
+
+# Generate JSON report
+bandit -c pyproject.toml -r scripts/ --exclude scripts/tests/ -f json -o bandit-report.json
+```
+
+### Run Type Checking
+
+```bash
+# Check types with mypy
+mypy scripts/ --ignore-missing-imports --no-implicit-optional
+```
+
+## GitHub Actions Workflow
+
+### Triggered On
+
+- **Push** to `main` or `develop` branches (when scripts change)
+- **Pull Request** to `main` (when scripts change)
+- Manual workflow dispatch
+
+### Jobs
+
+#### 1. Unit Tests (pytest)
+
+- **Runs on**: Ubuntu latest
+- **Python versions**: 3.10, 3.11, 3.12
+- **What it does**:
+  - Installs dependencies from `requirements-dev.txt`
+  - Runs pytest with coverage reporting
+  - Uploads coverage to Codecov
+  - Archives test results and coverage HTML
+
+**Outcome**: If any test fails, the workflow fails (critical)
+
+#### 2. Code Quality (Ruff)
+
+- **Runs on**: Ubuntu latest
+- **Python version**: 3.11
+- **What it does**:
+  - Checks code formatting with `ruff format`
+  - Runs linter with `ruff check`
+  - Reports issues but doesn't fail the workflow
+
+**Outcome**: Non-blocking (warning only)
+
+#### 3. Security Scan (Bandit)
+
+- **Runs on**: Ubuntu latest
+- **Python version**: 3.11
+- **What it does**:
+  - Scans for security vulnerabilities
+  - Generates JSON report
+  - Uploads report as artifact
+
+**Outcome**: Non-blocking (warning only)
+
+#### 4. Type Checking (mypy)
+
+- **Runs on**: Ubuntu latest
+- **Python version**: 3.11
+- **What it does**:
+  - Performs static type analysis
+  - Reports type mismatches
+  - Helps catch bugs early
+
+**Outcome**: Non-blocking (warning only)
+
+#### 5. Build EPUB
+
+- **Runs on**: Ubuntu latest
+- **Depends on**: pytest, lint, security (all must pass)
+- **What it does**:
+  - Builds the EPUB file using `scripts/build_epub.py`
+  - Verifies the EPUB was created successfully
+  - Uploads EPUB as artifact
+
+**Outcome**: If build fails, the workflow fails (critical)
+
+#### 6. Summary
+
+- **Runs on**: Ubuntu latest
+- **Depends on**: All other jobs
+- **What it does**:
+  - Generates workflow summary
+  - Lists all artifacts
+  - Reports overall status
+
+## Writing Tests
+
+### Test Structure
+
+Tests should be placed in `scripts/tests/` with names like `test_*.py`:
+
+```python
+# scripts/tests/test_example.py
+import pytest
+from scripts.example_module import some_function
+
+def test_basic_functionality():
+    """Test that some_function works correctly."""
+    result = some_function("input")
+    assert result == "expected_output"
+
+def test_error_handling():
+    """Test that some_function handles errors gracefully."""
+    with pytest.raises(ValueError):
+        some_function("invalid_input")
+
+@pytest.mark.asyncio
+async def test_async_function():
+    """Test async functions."""
+    result = await async_function()
+    assert result is not None
+```
+
+### Test Best Practices
+
+- **Use descriptive names**: `test_function_returns_correct_value()`
+- **One assertion per test** (when possible): Easier to debug failures
+- **Use fixtures** for reusable setup: See `scripts/tests/conftest.py`
+- **Mock external services**: Use `unittest.mock` or `pytest-mock`
+- **Test edge cases**: Empty inputs, None values, errors
+- **Keep tests fast**: Avoid sleep() and external I/O
+- **Use pytest markers**: `@pytest.mark.slow` for slow tests
+
+### Fixtures
+
+Common fixtures are defined in `scripts/tests/conftest.py`:
+
+```python
+# Use fixtures in your tests
+def test_something(tmp_path):
+    """tmp_path fixture provides temporary directory."""
+    test_file = tmp_path / "test.txt"
+    test_file.write_text("content")
+    assert test_file.read_text() == "content"
+```
+
+## Coverage Reports
+
+### Local Coverage
+
+```bash
+# Generate coverage report
+pytest scripts/tests/ --cov=scripts --cov-report=html
+
+# Open the coverage report in your browser
+open htmlcov/index.html
+```
+
+### Coverage Goals
+
+- **Minimum coverage**: 80%
+- **Branch coverage**: Enabled
+- **Focus areas**: Core functionality and error paths
+
+## Pre-commit Hooks
+
+The project uses pre-commit hooks to run checks automatically before commits:
+
+```bash
+# Install pre-commit hooks
+pre-commit install
+
+# Run hooks manually
+pre-commit run --all-files
+
+# Skip hooks for a commit (not recommended)
+git commit --no-verify
+```
+
+Configured hooks in `.pre-commit-config.yaml`:
+- Ruff formatter
+- Ruff linter
+- Bandit security scanner
+- YAML validation
+- File size checks
+- Merge conflict detection
+
+## Troubleshooting
+
+### Tests Pass Locally but Fail in CI
+
+Common causes:
+1. **Python version difference**: CI uses 3.10, 3.11, 3.12
+2. **Missing dependencies**: Update `requirements-dev.txt`
+3. **Platform differences**: Path separators, environment variables
+4. **Flaky tests**: Tests that depend on timing or order
+
+Solution:
+```bash
+# Test with the same Python versions
+uv python install 3.10 3.11 3.12
+
+# Test with clean environment
+rm -rf .venv
+uv venv
+uv pip install -r requirements-dev.txt
+pytest scripts/tests/
+```
+
+### Bandit Reports False Positives
+
+Some security warnings may be false positives. Configure in `pyproject.toml`:
+
+```toml
+[tool.bandit]
+exclude_dirs = ["scripts/tests"]
+skips = ["B101"]  # Skip assert_used warning
+```
+
+### Type Checking Too Strict
+
+Relax type checking for specific files:
+
+```python
+# Add at the top of file
+# type: ignore
+
+# Or for specific lines
+some_dynamic_code()  # type: ignore
+```
+
+## Continuous Integration Best Practices
+
+1. **Keep tests fast**: Each test should complete in <1 second
+2. **Don't test external APIs**: Mock external services
+3. **Test in isolation**: Each test should be independent
+4. **Use clear assertions**: `assert x == 5` not `assert x`
+5. **Handle async tests**: Use `@pytest.mark.asyncio`
+6. **Generate reports**: Coverage, security, type checking
+
+## Resources
+
+- [pytest Documentation](https://docs.pytest.org/)
+- [Ruff Documentation](https://docs.astral.sh/ruff/)
+- [Bandit Documentation](https://bandit.readthedocs.io/)
+- [mypy Documentation](https://mypy.readthedocs.io/)
+- [GitHub Actions Documentation](https://docs.github.com/en/actions)
+
+## Contributing Tests
+
+When submitting a PR:
+
+1. **Write tests** for new functionality
+2. **Run tests locally**: `pytest scripts/tests/ -v`
+3. **Check coverage**: `pytest scripts/tests/ --cov=scripts`
+4. **Run linting**: `ruff check scripts/`
+5. **Security scan**: `bandit -r scripts/ --exclude scripts/tests/`
+6. **Update documentation** if tests change
+
+Tests are required for all PRs! 🧪
+
+---
+
+For questions or issues with testing, open a GitHub issue or discussion.

.github/workflows/ci.yml

@@ -58,8 +58,42 @@ jobs:
         run: uv run bandit -c pyproject.toml -r scripts/ --exclude scripts/tests/
 
   test:
-    name: Tests
+    name: Unit Tests (Python ${{ matrix.python-version }})
     runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ['3.10', '3.11', '3.12']
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v4
+
+      - name: Set up Python ${{ matrix.python-version }}
+        run: uv python install ${{ matrix.python-version }}
+
+      - name: Create venv and install dependencies
+        run: |
+          uv venv
+          uv pip install -r requirements-dev.txt
+
+      - name: Run pytest with coverage
+        run: uv run pytest scripts/tests/ -v --tb=short --cov=scripts --cov-report=xml --cov-report=term-missing
+
+      - name: Upload coverage to Codecov
+        uses: codecov/codecov-action@v3
+        with:
+          files: ./coverage.xml
+          flags: unittests
+          name: codecov-python-${{ matrix.python-version }}
+          fail_ci_if_error: false
+
+  build:
+    name: Build EPUB
+    runs-on: ubuntu-latest
+    needs: [lint, security, test]
     steps:
       - name: Checkout code
         uses: actions/checkout@v4
@@ -73,22 +107,7 @@ jobs:
       - name: Create venv and install dependencies
         run: |
           uv venv
-          uv pip install -r requirements.txt
+          uv pip install -r requirements-dev.txt
-          uv pip install pytest pytest-asyncio
-
-      - name: Run Tests
-        run: uv run pytest scripts/tests/ -v --tb=short
-
-  build:
-    name: Build EPUB
-    runs-on: ubuntu-latest
-    needs: [lint, security, test]
-    steps:
-      - name: Checkout code
-        uses: actions/checkout@v4
-
-      - name: Install uv
-        uses: astral-sh/setup-uv@v4
 
       - name: Build EPUB
         run: uv run scripts/build_epub.py

.github/workflows/test.yml

@@ -0,0 +1,227 @@
+name: Automated Testing
+
+on:
+  push:
+    branches: [main, develop]
+    paths:
+      - 'scripts/**'
+      - '.github/workflows/test.yml'
+      - 'pyproject.toml'
+      - 'requirements*.txt'
+  pull_request:
+    branches: [main]
+    paths:
+      - 'scripts/**'
+      - '.github/workflows/test.yml'
+      - 'pyproject.toml'
+      - 'requirements*.txt'
+
+# Cancel in-progress runs for the same branch
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  pytest:
+    name: Unit Tests (Python ${{ matrix.python-version }})
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ['3.10', '3.11', '3.12']
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v4
+
+      - name: Set up Python ${{ matrix.python-version }}
+        run: uv python install ${{ matrix.python-version }}
+
+      - name: Create venv and install dependencies
+        run: |
+          uv venv
+          uv pip install -r requirements-dev.txt
+
+      - name: Run pytest
+        run: uv run pytest scripts/tests/ -v --tb=short --cov=scripts --cov-report=xml --cov-report=html
+        continue-on-error: false
+
+      - name: Upload coverage to Codecov
+        uses: codecov/codecov-action@v3
+        with:
+          files: ./coverage.xml
+          flags: unittests
+          name: codecov-python-${{ matrix.python-version }}
+          fail_ci_if_error: false
+          verbose: true
+
+      - name: Archive test results
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: pytest-results-${{ matrix.python-version }}
+          path: |
+            coverage.xml
+            htmlcov/
+          retention-days: 7
+
+  lint:
+    name: Code Quality (Lint & Format)
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v4
+
+      - name: Set up Python
+        run: uv python install 3.11
+
+      - name: Create venv and install Ruff
+        run: |
+          uv venv
+          uv pip install ruff
+
+      - name: Ruff Format Check
+        run: uv run ruff format --check scripts/
+        continue-on-error: true
+
+      - name: Ruff Lint Check
+        run: uv run ruff check scripts/
+        continue-on-error: true
+
+  security:
+    name: Security Scan
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v4
+
+      - name: Set up Python
+        run: uv python install 3.11
+
+      - name: Create venv and install Bandit
+        run: |
+          uv venv
+          uv pip install "bandit[toml]"
+
+      - name: Run Bandit Security Scan
+        run: uv run bandit -c pyproject.toml -r scripts/ --exclude scripts/tests/ -f json -o bandit-report.json
+        continue-on-error: true
+
+      - name: Upload security report
+        uses: actions/upload-artifact@v4
+        if: always()
+        with:
+          name: bandit-security-report
+          path: bandit-report.json
+          retention-days: 7
+
+  type-check:
+    name: Type Checking
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v4
+
+      - name: Set up Python
+        run: uv python install 3.11
+
+      - name: Create venv and install mypy
+        run: |
+          uv venv
+          uv pip install -r requirements-dev.txt mypy
+
+      - name: Run mypy
+        run: uv run mypy scripts/ --ignore-missing-imports --no-implicit-optional
+        continue-on-error: true
+
+  build-epub:
+    name: Build EPUB Artifact
+    runs-on: ubuntu-latest
+    needs: [pytest, lint, security]
+    if: success()
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v4
+
+      - name: Set up Python
+        run: uv python install 3.11
+
+      - name: Create venv and install dependencies
+        run: |
+          uv venv
+          uv pip install -r requirements-dev.txt
+
+      - name: Build EPUB
+        run: uv run scripts/build_epub.py
+
+      - name: Verify EPUB Created
+        run: |
+          if [ -f claude-howto-guide.epub ]; then
+            echo "✅ EPUB built successfully"
+            ls -lh claude-howto-guide.epub
+          else
+            echo "❌ EPUB file not found!"
+            exit 1
+          fi
+
+      - name: Upload EPUB Artifact
+        uses: actions/upload-artifact@v4
+        with:
+          name: claude-howto-guide-epub
+          path: claude-howto-guide.epub
+          retention-days: 7
+          compression-level: 0
+
+  summary:
+    name: Test Summary
+    needs: [pytest, lint, security, type-check, build-epub]
+    runs-on: ubuntu-latest
+    if: always()
+
+    steps:
+      - name: Download all artifacts
+        uses: actions/download-artifact@v4
+
+      - name: Generate Summary
+        run: |
+          echo "# Test Summary" >> $GITHUB_STEP_SUMMARY
+          echo "" >> $GITHUB_STEP_SUMMARY
+          echo "## Results" >> $GITHUB_STEP_SUMMARY
+          echo "- **Unit Tests**: ${{ needs.pytest.result }}" >> $GITHUB_STEP_SUMMARY
+          echo "- **Code Quality**: ${{ needs.lint.result }}" >> $GITHUB_STEP_SUMMARY
+          echo "- **Security Scan**: ${{ needs.security.result }}" >> $GITHUB_STEP_SUMMARY
+          echo "- **Type Checking**: ${{ needs.type-check.result }}" >> $GITHUB_STEP_SUMMARY
+          echo "- **EPUB Build**: ${{ needs.build-epub.result }}" >> $GITHUB_STEP_SUMMARY
+          echo "" >> $GITHUB_STEP_SUMMARY
+          echo "## Artifacts" >> $GITHUB_STEP_SUMMARY
+          echo "- pytest-results-* (coverage and test results)" >> $GITHUB_STEP_SUMMARY
+          echo "- bandit-security-report (security scan JSON)" >> $GITHUB_STEP_SUMMARY
+          echo "- claude-howto-guide-epub (built EPUB file)" >> $GITHUB_STEP_SUMMARY
+
+      - name: Check if all tests passed
+        if: |
+          needs.pytest.result != 'success' ||
+          needs.build-epub.result != 'success'
+        run: |
+          echo "❌ Critical tests failed"
+          exit 1
+
+      - name: All tests passed
+        if: success()
+        run: echo "✅ All tests passed successfully!"