Module `neuralops.benchmarks`

NeuralOps Benchmarks & Sweepers Suite.

Sweep drivers, scaling analysis, multi-seed evaluation harnesses, and GPU profiling utilities for the full NeuralOps benchmark protocol.

Modules

run_all_benchmarks Runs the complete benchmark matrix for all registered modules. run_gpt2_sweep GPT-2-specific quality/speed sweep. run_long_context_sweep Scaling analysis over sequence lengths 512 – 16384. run_multiseed Multi-seed robustness evaluation. run_multiseed_rigor Rigorous multi-seed evaluation with statistical testing. run_one_module_isolated Isolated single-module benchmark for debugging. run_quality_sweep Quality-vs-rank Pareto frontier sweep. run_rank_sweeper Rank ablation sweep across all registered modules. run_ruler_benchmark RULER long-context benchmark driver. run_seqlen_scaling Sequence-length scaling experiment. run_seqlen_scaling_bench Sequence-length scaling with hardware profiling. run_whitened_svd Whitened SVD compression benchmark. launch_parallel_sweep Parallel multi-GPU sweep launcher. run_all_modules_scheduler Priority-queue-based module scheduler. kd_quick_test Quick knowledge-distillation sanity test. gpu_live_probe Real-time GPU telemetry probe (run on GPU server). harvest_gpu_benchmark_data Post-hoc GPU benchmark data harvesting.

Subpackages

core Shared argument parsing (arguments.py) and stats utilities (stats.py). module_sweeps Not here — per-module run_all_4_*.py scripts live in neuralops.runners.module_sweeps.

Sub-modules

neuralops.benchmarks.core: Shared core utilities for the NeuralOps benchmarks suite …
neuralops.benchmarks.gpu_live_probe: gpu_live_probe.py Executed on the GPU server to return real-time GPU telemetry, active running process, live training loss trajectory, and log streams …
neuralops.benchmarks.harvest_gpu_benchmark_data
neuralops.benchmarks.kd_quick_test: Adaptive Knowledge Distillation (KD) Experiment on MambaBlock …
neuralops.benchmarks.launch_parallel_sweep: Launch and supervise 4 parallel GPT-2 sweep workers …
neuralops.benchmarks.run_all_benchmarks: Master Orchestrator for Comprehensive Model Evaluation …
neuralops.benchmarks.run_all_modules_scheduler: Batch scheduler for KD experiments on all attention and compression modules. Runs modules in parallel with dynamic VRAM guarding, automatic retry on …
neuralops.benchmarks.run_gpt2_sweep: Decoder Benchmark Runner: Comprehensive evaluation of attention/compression modules on GPT-2 with WikiText-2 …
neuralops.benchmarks.run_long_context_sweep: Long-Context Sweep Runner: Evaluates compression modules on Byte-Level IMDb (4096 tokens) …
neuralops.benchmarks.run_multiseed: Multi-Seed Evaluation Runner: Measures variance and statistical confidence (mean +/- std) …
neuralops.benchmarks.run_multiseed_rigor: Multi-Seed Statistical Rigor Runner & Aggregator …
neuralops.benchmarks.run_one_module_isolated: Child runner: executes exactly one module evaluation. Called by run_quality_sweep.py via subprocess.Popen. Writes result JSON to –output_json path …
neuralops.benchmarks.run_quality_sweep: Cached Qwen sequence-classification quality sweep …
neuralops.benchmarks.run_rank_sweeper: Automated rank sweep runner for low-rank compressor modules …
neuralops.benchmarks.run_ruler_benchmark: Needle-in-a-Haystack (NIAH) Context Retrieval Benchmark (RULER-style) for GPT-2 …
neuralops.benchmarks.run_seqlen_scaling: Sequence-Length Scaling Runner: Evaluates VRAM, latency, and throughput scaling across context lengths …
neuralops.benchmarks.run_seqlen_scaling_bench: Sequence Length Scaling Benchmark Harness …
neuralops.benchmarks.run_whitened_svd