Quality Assurance TASK-06
Reviewing completed annotations for accuracy and consistency.
Task Overview
What is Quality Assurance?
Quality Assurance (QA) is the systematic review of completed annotations to verify they meet project standards. QA reviewers sample submitted work, identify errors, provide feedback to annotators, and determine whether submissions can be accepted or need revision.
Why is it Important?
Without QA, errors accumulate and compound throughout the dataset. QA catches mistakes before they become embedded in downstream analysis. It also provides feedback that improves annotator performance over time, reducing error rates across the entire project.
Key Objectives
- Review sampled annotations against ground truth standards
- Calculate accuracy metrics (precision, recall, agreement)
- Identify systematic error patterns
- Provide constructive feedback to annotators
- Approve or reject submissions based on quality thresholds
Current Standard Operating Procedure Authoritative
Sampling Strategy
Not every annotation is reviewed. QA uses statistical sampling:
- New annotators — 100% review for first 5 submissions
- Experienced annotators — 10-20% random sample
- Flagged submissions — Always reviewed
- Complex regions — Higher sampling rate
Review Procedure
- Load the submitted annotation and reference data
- Compare annotation against EM imagery at full resolution
- Check for completeness (all required structures labeled)
- Check for accuracy (labels match actual structures)
- Check for consistency (style matches project standards)
- Document all errors found with screenshots
- Calculate error rate and compare to threshold
Error Classification
- Critical — Wrong structure identity, major omission
- Major — Significant boundary error, missing branch
- Minor — Small boundary imprecision, style inconsistency
Quality Thresholds
- Pass — Less than 5% critical/major error rate
- Conditional — 5-10% error rate, requires feedback
- Reject — Greater than 10% error rate, needs redo
Feedback Process
- Compile error report with specific examples
- Identify patterns (repeated mistakes vs. random errors)
- Write constructive feedback focusing on improvement
- Send to annotator via project tracking system
- Schedule retraining if systematic issues identified
Historical SOPs Archived
Previous QA protocols are preserved here for reference. Always use the Current SOP above.
Version History
-
v1.0 (Current) — 2025
Initial QA workflow with sampling strategy
Visual Examples
Reference images showing acceptable quality vs. common errors requiring rejection.
Quality Reference
- Acceptable — Boundaries within 0.5 µm, all structures labeled
- Marginal — Minor boundary drift, occasional missed detail
- Unacceptable — Major omissions, wrong identifications
Common Failure Modes
- Inconsistent standards Different QA reviewers may apply different criteria. Regular calibration meetings maintain consistency across reviewers.
- Sampling bias Easy regions may be over-sampled while difficult regions slip through. Use stratified sampling based on complexity.
- Feedback without specifics "More careful" isn't actionable. Provide exact examples and explain what the correct annotation should look like.
- Delayed feedback Feedback weeks after submission is less useful. Aim for same-day or next-day turnaround on QA reviews.
- Missing systematic patterns Track errors across annotators to identify training gaps. One person's repeated mistake may indicate unclear instructions.
Tools Used
WebKnossos
Side-by-side annotation comparison
Neuroglancer
3D review of reconstructions
Spreadsheets
Error tracking and metrics
ID Validation
Automated consistency checks
Metrics Calculated
- Error Rate — Errors per unit volume or structure
- Agreement — Inter-annotator consistency
- Completeness — Fraction of structures labeled
- Precision — Correct labels / total labels