10  Verification & Validation Plan

10.1 V&V Strategy

Per [1, Secs. 2.3.5.9, 2.3.5.11], this plan defines how we confirm the system meets requirements (verification) and stakeholder needs (validation).

Method Scope Environment
Unit Testing Go packages Local (go test)
Integration Testing MCP protocol compliance Local (stdio)
Container Testing Image builds, runtime GitLab CI only
HTTP Transport Testing Remote MCP connections GitLab CI (service containers)
Acceptance Testing End-to-end with Claude/VS Code Local (stdio) + manual

10.2 Verification Methods

Per [1, Sec. 2.3.5.9], verification uses IADT methods:

Method Abbreviation Description When Used
Inspection I Visual examination of artifacts Documentation, code review
Analysis A Mathematical/logical evaluation Performance, security assessment
Demonstration D Functional operation shown MCP protocol interaction
Test T Execution with defined inputs Unit tests, integration tests

10.2.1 Verification Method Assignment

Requirement Method Rationale
FR-MCP-001 T, D Test server initialization, demonstrate with client
FR-MCP-002 T Test tool enumeration and execution
FR-GL-001, FR-GL-002 T Test file read from GitLab repositories
FR-SYS-001 T Test parsing of SysML v2 syntax
NFR-DEP-001 T, A Test binary builds, analyze size
NFR-DEP-002 T Test container builds in CI
NFR-DOC-001 I Inspect Quarto output for completeness

10.3 Acceptance Criteria

Requirement Category Verification Method Acceptance Criteria
MCP Protocol Compliance Integration test Server initializes, lists tools/resources, executes tools
GitLab Integration Integration test Read files from gitlab.com and self-hosted
SysML v2 Validation System test Validates correct/incorrect SysML syntax
Container Deployment CI pipeline Image builds, runs, responds to MCP requests
Documentation Inspection Quarto renders, deploys to GitLab Pages

10.4 Enabling Systems

Per [1, Sec. 2.3.5.9], enabling systems support verification activities.

Enabling System Purpose Responsibility
Go Test Framework Unit and integration testing Built into Go toolchain
GitLab CI/CD Automated pipeline execution GitLab SaaS runners
Buildah/Podman Container image builds CI environment only
Claude Desktop Manual acceptance testing Local development
MCP Inspector Protocol debugging Local development
Quarto Documentation builds Local + CI

10.4.1 Test Environment Configuration

Environment Transport External Services Use Case
Local Dev stdio Mocked/optional Unit tests, rapid iteration
CI Test stdio Mocked Automated test suite
CI Integration HTTP GitLab API (PAT) Integration tests
CI Container HTTP Service containers End-to-end container tests

10.5 Test Cases

10.5.1 MCP Protocol Tests

ID Test Case Expected Result Method
TC-MCP-001 Send initialize request Server responds with capabilities T
TC-MCP-002 Request tools/list Returns list including sysml_parse T
TC-MCP-003 Call sysml_parse with valid SysML Returns parsed elements T
TC-MCP-004 Request resources/list Returns example resources T
TC-MCP-005 Read sysml://examples/hello Returns vehicle model content T

10.5.2 GitLab Integration Tests

ID Test Case Expected Result Method
TC-GL-001 Read file from public repo Returns file content T
TC-GL-002 Read file with PAT auth Returns file content T
TC-GL-003 List .sysml files in directory Returns file list T
TC-GL-004 Read from self-hosted GitLab Returns file content T
TC-GL-005 Handle non-existent file Returns appropriate error T

10.5.3 SysML Parsing Tests

ID Test Case Expected Result Method
TC-SYS-001 Parse package declaration Extracts package name T
TC-SYS-002 Parse part definition Extracts part def name T
TC-SYS-003 Parse requirement definition Extracts requirement name T
TC-SYS-004 Parse nested elements Extracts all element names T
TC-SYS-005 Parse empty input Returns empty element list T

10.6 Known Limitations

  1. Container testing: Cannot be performed locally on macOS; relies on CI
  2. HTTP transport: Requires CI service containers or Linux machine
  3. SysML v2 API: Requires running API server; may use mock for some tests

10.7 CI/CD Verification Pipeline

Per [1, Sec. 2.3.5.9], automated verification integrates into CI/CD.

10.7.1 Pipeline Stages

stages:
  - lint
  - test
  - build
  - integration
  - publish

10.7.2 Test Stage

test:
  stage: test
  image: golang:1.23-alpine
  script:
    - go test -v -race ./...
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == "main"

10.7.3 Integration Test Stage

integration:
  stage: integration
  image: golang:1.23-alpine
  variables:
    GITLAB_TOKEN: $CI_JOB_TOKEN
  script:
    - go test -v -tags=integration ./...
  rules:
    - if: $CI_COMMIT_BRANCH == "main"

10.7.4 Container Test Stage

container-test:
  stage: test
  image: quay.io/buildah/stable
  services:
    - name: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
      alias: mcp-server
  script:
    - echo '{"jsonrpc":"2.0","id":1,"method":"initialize"...}' | nc mcp-server 8080
  rules:
    - if: $CI_COMMIT_BRANCH == "main"

10.8 Validation Approach

Per [1, Sec. 2.3.5.11], validation confirms the system meets stakeholder needs.

10.8.1 Validation Activities

Activity Stakeholder Need Method Acceptance
End-to-end demo AI tool integration Demonstration Claude reads SysML from GitLab
User acceptance Developer experience Interview Positive feedback from pilot users
Paper submission Academic validation Peer review GVSETS acceptance
Capstone review Educational objectives Review Advisor approval

10.8.2 Validation Schedule

Milestone Week Validation Activity
SRR 2 Requirements validated with stakeholders
PDR 4 Architecture validated against requirements
CDR 12 Implementation validated, acceptance tests pass
Final 15 Stakeholder acceptance, capstone submission

10.9 Review Verification

Review Verification Activities
SRR Requirements complete, traceable to stakeholders
PDR Architecture addresses all requirements
CDR All tests pass, acceptance criteria met