A single-device multimodal inference setup using NVFP4 quantization for Q&A, coding, and agent tasks. Tiered loading from local NVMe keeps the model within unified system memory capacity.
Multimodal understandingText, images, video → text
1 MegaCubeTest report available
01 / Model parameters
Model architecture & capabilities
Qwen3.8-Flash-Next is a multimodal mixture-of-experts model for general Q&A and agent tasks. It uses sparse experts, hybrid attention, and embedding components that support tiered loading, drawing on a subset of compute resources at each step. Total parameters, active parameters per step, and actual device memory use are distinct measures.
Local Q&A and content synthesis
Code development and debugging
Agent tool calling
Publisher / base model
Alibaba Qwen · Qwen3.8-Flash-Next
Parameter breakdown
125B language model / 6B active; an additional 51B N-gram embedding and 4B MTP component
Architecture
48-layer MoE; 512 routed experts, 10 selected per step, plus 1 shared expert; Gated DeltaNet + QSA
Model context
Native 262,144 tokens, expandable to 1M; configured limits and tested lengths listed separately
Input / output
Text, image, and video input; text and tool-call output
Key capabilities
Q&A, complex reasoning, coding, tool calling, and multimodal understanding
Quantization used here
NVIDIA ModelOpt: NVFP4 for main routed experts, BF16 for other main layers, FP8 for MTP experts and PLE
Weight size
Approx. 123.57 GiB / 132.7 GB; not all resident in memory simultaneously
License
NVIDIA Open Model License, with additional Qwen Community terms
Single-device tests here primarily cover low-concurrency text and coding tasks. The public model supports vision; 262K is this service's configuration limit, and full-length stress testing has not been completed.
Download NVFP4 weights to fast NVMe; tiered PLE reads require local disk storage.
Pinned runtime versions
Use the vLLM image and rc2 configuration specified in the report, and verify the weights and deployment package.
Start a single-device service
Default to C1 / MTP3, enable Prefix Cache, and retain the default Compile / Graph settings.
Warm up and connect
Run health checks and warm up representative tasks, then connect PINVOU to the model service.
Find and download this model in the PINVOU Model Store, then follow its model card to deploy and enable the service.
The Model Store currently offers an rc3 candidate deployment package that has not completed inference acceptance testing on the device. Results on this page come from the earlier rc2 configuration and should be interpreted for that version.
2026-09-13; 40 questions across eight real-world task types, with three interleaved retest rounds and thinking mode disabled. C1 token generation rates are reported as medians. Prefix caching is tested separately using a shared prefix of approx. 19.9K tokens.
43.70 tok/sC1 · MTP3 · Median token generation rate
0.30 sTTFT P95 for the same real-world workloads
1.486 sMedian TTFT with a stable warm cache for an approx. 19.9K-token prefix
C1: interleaved MTP3 / MTP4 retests
Configuration
Token generation P50
P10
TTFT P50/P95
Quality score
MTP acceptance rate
MTP3 · Recommended production setting
43.70 tok/s
26.90
0.26 / 0.30 s
0.842
66.17%
MTP4 · Optional throughput setting
46.25 tok/s
26.00
0.27 / 0.32 s
0.860
57.83%
C2: low-concurrency aggregate throughput (tok/s)
Configuration
Aggregate throughput
TTFT P95
MTP3
48.20
0.47 s
MTP4
51.60
0.53 s
Shared prefix: uncached input and warm-cache runs
Status
TTFT
Query tokens
Cache-hit tokens
Hit rate
Token generation (tok/s)
Uncached prefix
11.579 s
19,884
0
0%
28.012
First reuse/warm-up
11.568 s
19,884
0
0%
26.475
Warm-cache run 1
1.509 s
19,884
17,600
88.51%
26.792
Warm-cache run 2
1.483 s
19,884
17,600
88.51%
27.675
Warm-cache run 3
1.486 s
19,884
17,600
88.51%
26.537
262K is the service configuration limit; full-length stress testing has not been completed.
The first reuse may still miss the cache. A cache hit reduces time to first token; it does not speed up token generation after the first token.
Approx. 133 GB of weight files does not mean all weights remain resident in memory simultaneously.
C1 / C4 / C8 denotes 1 / 4 / 8 concurrent requests. TTFT (time to first token) measures elapsed time from submitting a request to receiving its first output token. TPOT (time per output token) measures average time per generated token after the first token. Aggregate throughput is the combined output rate across concurrent requests; a single-request token generation rate is measured separately in tok/s.
Public benchmark references
Model capability & platform performance benchmarks
The following results are published by model developers or platform providers and are separate from the MegaCube tests above. Capability scores measure task performance; tok/s measures token throughput. Results obtained with different models, numerical precision, hardware, or test protocols are not directly comparable.
NVIDIA NVFP4 capability benchmarks
NVIDIA · Checked on 2026-09-20
Evaluated model
Qwen3.8-Flash-Next-NVFP4
Test platform
NVIDIA B200 / B300；vLLM
NVIDIA NVFP4 capability benchmarks · Data
Benchmark
Score
What it measures
GPQA Diamond
91.5
Challenging science Q&A
Terminal-Bench 2.1
82.9
Multi-step terminal tasks
MMMU Pro
78.3
Multidisciplinary multimodal understanding
Test conditionstemperature=1, top_p=0.95, maximum output budget 131,072 tokens, reasoning effort xhigh. See the source for benchmark tool frameworks and task environments; this is not a retest of single-device throughput or accuracy on this page.
