Research Methodology — CyberSec Research Lab
Our research relies on quantitative testing protocols designed for reproducibility and objectivity. This page describes our evaluation framework, scoring methodology, peer review process, and conflict-of-interest policies.
Testing Framework
All vendor assessments use our standardized testing framework built on 847 threat samples derived from MITRE ATT&CK techniques observed in real-world incident reports from 2024-2025. Testing is conducted in isolated lab environments that replicate enterprise network topologies.
Threat Sample Set
847 unique samples across 8 attack categories. Curated from public threat intelligence feeds, partner incident reports, and MITRE ATT&CK technique documentation. Refreshed quarterly.
Test Environment
Isolated lab with configurable topologies: 100 to 50,000 simulated endpoints across Windows, Linux, and macOS. Network traffic from anonymized production PCAP captures.
Scoring Dimensions
Vendors are evaluated across 8 weighted dimensions. Each dimension is scored 0-100 using standardized protocols. The composite score is a weighted average.
Detection Efficacy
20%Measures the true positive rate across our standardized 847-sample threat test suite. Each sample is mapped to a MITRE ATT&CK technique and classified by attack category. We report TPR, FPR, and 95% confidence intervals.
False Positive Rate
15%Measures the rate of incorrect alerts generated against benign traffic baselines. Lower FPR indicates better signal-to-noise ratio. We use standardized benign traffic datasets derived from anonymized enterprise network captures.
Response Latency
15%Measures mean time to detect (MTTD), mean time to respond (MTTR), and mean time to contain (MTTC). Measured from threat introduction to platform action in milliseconds. Includes automated response capabilities where applicable.
Architectural Innovation
15%Evaluates novelty and technical sophistication of the platform architecture. Scored by our research team using a structured rubric covering data pipeline design, ML model architecture, inference optimization, and extensibility patterns.
Scalability
10%Tests detection performance degradation across increasing endpoint counts (100, 1K, 10K, 50K simulated endpoints). Measures throughput, latency variance, and resource consumption at each scale tier.
API Coverage
10%Assesses the breadth and quality of programmatic interfaces: REST API completeness, webhook support, SIEM/SOAR integration connectors, documentation quality, and SDK availability across major languages.
Deployment Friction
10%Measures time from initial deployment to first detection across standardized test environments. Includes agent deployment complexity, configuration requirements, and documentation clarity.
Threat Coverage Breadth
5%Evaluates the range of MITRE ATT&CK techniques covered by the platform's detection engine. Broader coverage indicates ability to detect a wider variety of attack patterns.
Peer Review Process
Every publication follows a four-step review process designed to ensure accuracy and objectivity.
Internal Analysis
Our research team conducts all testing and data collection using standardized protocols in isolated lab environments.
Cross-Validation
Results are independently verified by at least two researchers who were not involved in the initial testing. Statistical analysis confirms reproducibility within acceptable variance bounds.
External Review
Draft publications are reviewed by at least one external advisor with relevant domain expertise. Advisors are drawn from our network of former national lab researchers and academic collaborators.
Publication with Disclosure
Published studies include full methodology descriptions, limitation acknowledgments, and any relevant conflict-of-interest disclosures. Raw data is available upon request for academic peer review.
Conflict of Interest Disclosures
CyberSec Research Lab does not accept vendor sponsorship, advertising revenue, or pay-for-placement arrangements. Our research is funded through institutional grants, anonymized aggregate data licensing, and enterprise consulting engagements.
No researcher at CyberSec Research Lab holds equity positions, advisory roles, or consulting relationships with any vendor evaluated in our publications. Any potential conflicts that arise are disclosed in the relevant publication and the affected researcher is recused from scoring decisions.
Statistical Methodology
We report 95% confidence intervals for all detection rate measurements. Statistical significance is assessed using paired t-tests (p < 0.05) when comparing vendor performance. Effect sizes are reported using Cohen's d to provide practical significance context alongside statistical significance.
Minimum Sample Size
Our threat sample set of 847 provides statistical power of 0.95 for detecting differences of 5 percentage points or greater in detection rates between vendors (alpha = 0.05).
Reproducibility
Cross-validation testing shows mean score variance of ±2.1 points across independent test runs, indicating acceptable reproducibility for our evaluation framework.