MegaCube. Desktop AI SupercomputerDiscuss Your Solution
← All models

General Q&A · Coding

Qwen3.8-Flash-Next

A multimodal model for general Q&A, coding, and tool calling. FP8 weights run across two nodes connected directly by DAC, validating distributed inference and long context.

Multimodal understandingText, images, video → text
2 MegaCubesTest report available

01 / Model parameters

Model architecture & capabilities

Qwen3.8-Flash-Next is a multimodal mixture-of-experts model for general Q&A and agent tasks. It uses sparse experts, hybrid attention, and embedding components that support tiered loading, drawing on a subset of compute resources at each step. Total parameters, active parameters per step, and actual device memory use are distinct measures.

  • Long-document Q&A and information extraction
  • Code writing and project updates
  • Multi-step tool workflows
Publisher / base model
Alibaba Qwen · Qwen3.8-Flash-Next
Parameter breakdown
125B language model / 6B active; an additional 51B N-gram embedding and 4B MTP component
Architecture
48-layer MoE; 512 routed experts, 10 selected per step, plus 1 shared expert; Gated DeltaNet + QSA
Model context
Native 262,144 tokens, expandable to 1M; configured limits and tested lengths listed separately
Input / output
Text, image, and video input; text and tool-call output
Key capabilities
Q&A, complex reasoning, coding, tool calling, and multimodal understanding
Weights used here
Qwen official FP8 version; 131 safetensors shards
License
Qwen Community License 1.0

Tests on this setup primarily cover text and long context. The two-device FP8 and single-device NVFP4 setups differ in precision and loading method, so their relative speed cannot be inferred from device count alone.

Base model & benchmarks ↗Official FP8 weights ↗

Model information checked on 2026-09-20

02 / Deployment

MegaCube deployment configuration

Compute devices
2 × MegaCube, each with 128 GB unified memory
Network
One QSFP DAC direct link; management and API traffic over RJ45
Execution method
vLLM native MP · TP=2 · PP=1
Service settings
Context 262,144; MaxSeq=8; BatchTokens=8192
Acceleration & memory
MTP=1；Eager；GPU memory=0.85；Prefix Cache

  1. Connect the devices

    Connect two Cubes with a compatible DAC and verify RDMA and the office network.

  2. Match the weights and container images

    Keep the same 131 weight shards on both devices, verify the index, and use the same ARM64 image.

  3. Start the worker node before the head node

    Start node-rank 1 first, then node-rank 0, to form a TP2 / PP1 service.

  4. Check the API

    Check /health, /v1/models, and streaming requests, then validate concurrency and long documents.

Inference settings
Inference settings
Core configurationDelivery valuePurpose
Execution backendMP · TP=2 · PP=1Native two-node tensor parallelism
Context / scheduling262,144 · MaxSeq8 · BatchTokens8192Long context & concurrency scheduling
Inference optimizationMTP · spec-tokens=1 · EagerPrioritizes stability on the current hardware and software
Memory / cacheGPU memory 0.85 · Prefix CacheKV cache capacity: 733,710 tokens
View commands and configuration references

Before use, set model paths and service addresses for your devices and check the versions and parameters above.

nvidia-smi
free -h
df -hT
ibdev2netdev
rdma link
docker ps

MODEL=/home/test/models/Qwen--Qwen3.8-Flash-Next-FP8/snapshots/master
find "$MODEL" -maxdepth 1 -name 'model-*.safetensors' | wc -l   # Expected: 131
sha256sum "$MODEL/model.safetensors.index.json"
IMAGE=qwen38-flash-next-vllm:validated
MODEL_DIR=/home/test/models/Qwen--Qwen3.8-Flash-Next-FP8/snapshots/master
MASTER_ADDR=<HEAD-RDMA-IP>
IFACE=<RDMA-IFACE>

docker run -d --name qwen38-native-worker --entrypoint bash \
  --network host --ipc host --gpus all \
  --device /dev/infiniband:/dev/infiniband --ulimit memlock=-1 \
  -v "$MODEL_DIR:/models/target:ro" -v /home/test/qwen38-results:/results \
  -e NCCL_SOCKET_IFNAME="$IFACE" -e GLOO_SOCKET_IFNAME="$IFACE" \
  "$IMAGE" -lc "vllm serve /models/target \
    --served-model-name Qwen3.8-Flash-Next-FP8 \
    --tensor-parallel-size 2 --pipeline-parallel-size 1 \
    --nnodes 2 --master-addr $MASTER_ADDR --master-port 29501 \
    --node-rank 1 --headless --distributed-executor-backend mp \
    --enforce-eager --max-model-len 262144 --max-num-seqs 8 \
    --max-num-batched-tokens 8192 --enable-chunked-prefill \
    --enable-prefix-caching --gpu-memory-utilization 0.85 \
    --spec-method mtp --spec-tokens 1 \
    --enable-auto-tool-choice --tool-call-parser qwen3_xml \
    --no-enable-flashinfer-autotune --host 0.0.0.0 --port 8000"
IMAGE=qwen38-flash-next-vllm:validated
MODEL_DIR=/home/test/models/Qwen--Qwen3.8-Flash-Next-FP8/snapshots/master
MASTER_ADDR=<HEAD-RDMA-IP>
IFACE=<RDMA-IFACE>

docker run -d --name qwen38-native-head --entrypoint bash \
  --network host --ipc host --gpus all \
  --device /dev/infiniband:/dev/infiniband --ulimit memlock=-1 \
  -v "$MODEL_DIR:/models/target:ro" -v /home/test/qwen38-results:/results \
  -e NCCL_SOCKET_IFNAME="$IFACE" -e GLOO_SOCKET_IFNAME="$IFACE" \
  "$IMAGE" -lc "vllm serve /models/target \
    --served-model-name Qwen3.8-Flash-Next-FP8 \
    --tensor-parallel-size 2 --pipeline-parallel-size 1 \
    --nnodes 2 --master-addr $MASTER_ADDR --master-port 29501 \
    --node-rank 0 --distributed-executor-backend mp \
    --enforce-eager --max-model-len 262144 --max-num-seqs 8 \
    --max-num-batched-tokens 8192 --enable-chunked-prefill \
    --enable-prefix-caching --gpu-memory-utilization 0.85 \
    --spec-method mtp --spec-tokens 1 \
    --enable-auto-tool-choice --tool-call-parser qwen3_xml \
    --no-enable-flashinfer-autotune --host 0.0.0.0 --port 8000"
curl -f http://<HEAD-MGMT-IP>:8000/health
curl -s http://<HEAD-MGMT-IP>:8000/v1/models

# Further validation: non-streaming, streaming, fixed 64 tokens, C4 concurrency, multilingual requests

03 / Performance tests

MegaCube performance test results

2026-08-28; two devices directly linked by DAC; 752 performance matrix requests. The table lists fixed input/output lengths and concurrency levels; throughput is aggregated across concurrent requests.

112.98 tok/s512/128 tokens · C8 aggregate output
0.421 s512/128 tokens · C1 TTFT P95
262,144261,888 input + 256 output passed
Fixed-token performance matrix
Input / outputConcurrencyRPSAggregate output throughput (tok/s)TTFT P50 / P95TPOT P50 / P95Success rate
512 / 128C10.253032.380.405 / 0.421 s27.51 / 31.14 ms100%
512 / 128C20.420453.820.476 / 0.731 s32.63 / 38.93 ms100%
512 / 128C40.606577.630.529 / 0.818 s46.68 / 52.97 ms100%
512 / 128C80.8827112.980.642 / 1.868 s63.62 / 75.74 ms100%
2048 / 256C10.127432.620.895 / 1.129 s26.51 / 29.14 ms100%
2048 / 256C20.204652.370.986 / 1.698 s33.51 / 36.71 ms100%
2048 / 256C40.288173.761.659 / 2.816 s46.96 / 54.56 ms100%
2048 / 256C80.4005102.541.806 / 5.338 s67.72 / 79.93 ms100%
8128 / 1024C10.036237.100.892 / 0.903 s26.10 / 26.80 ms100%
8128 / 1024C20.058259.651.006 / 2.978 s30.93 / 34.68 ms100%
8128 / 1024C40.081483.372.910 / 6.109 s44.40 / 49.68 ms100%
8128 / 1024C80.1099112.573.235 / 5.503 s67.94 / 81.24 ms100%
16,384 / 1024C10.032933.722.709 / 3.205 s26.66 / 28.04 ms100%
16,384 / 1024C20.060061.431.137 / 1.803 s30.73 / 33.47 ms100%
16,384 / 1024C40.073274.923.526 / 12.546 s46.00 / 57.95 ms100%
16,384 / 1024C80.090592.637.968 / 23.050 s70.51 / 100.68 ms100%
32,768 / 1024C10.031832.573.550 / 3.557 s27.08 / 28.34 ms100%
32,768 / 1024C20.053754.975.647 / 6.802 s30.88 / 32.63 ms100%
32,768 / 1024C40.060361.778.084 / 19.460 s50.84 / 67.58 ms100%
32,768 / 1024C80.074376.1121.725 / 73.848 s61.27 / 117.96 ms100%
  • Use C1 for full-length 262K requests. Short-input C8 results do not establish concurrency at the full context length.
  • The single-device NVFP4 setup and this two-device FP8 setup use different precision and deployment methods; interpret their results separately.

C1 / C4 / C8 denotes 1 / 4 / 8 concurrent requests. TTFT (time to first token) measures elapsed time from submitting a request to receiving its first output token. TPOT (time per output token) measures average time per generated token after the first token. Aggregate throughput is the combined output rate across concurrent requests; a single-request token generation rate is measured separately in tok/s.

Public benchmark references

Model capability & platform performance benchmarks

The following results are published by model developers or platform providers and are separate from the MegaCube tests above. Capability scores measure task performance; tok/s measures token throughput. Results obtained with different models, numerical precision, hardware, or test protocols are not directly comparable.

Qwen3.8-Flash-Next official capability reference

Qwen · Checked on 2026-09-20

Evaluated model
Qwen3.8-Flash-Next base model; not a retest of the two-device configuration on this page
Test platform
Inference hardware not disclosed by the source
Qwen3.8-Flash-Next official capability reference · Data
BenchmarkScoreWhat it measures
GPQA Diamond91.7Challenging science Q&A
SWE-bench Pro62.5Software engineering problem solving
DeepSWE v1.158.7Software problem solving

Test conditionsCoding benchmarks use temperature=1, top_p=0.95, and 256K context. SWE-bench Pro uses Claude Code and the corrected task set; DeepSWE reports the best result across Claude Code and mini-SWE frameworks.

View original benchmarks ↗

Plan model deployment and application integration

Evaluate device configurations and inference services for your model size, workloads, and concurrency needs.

Discuss this model