ybjhSrSSKJr SSKrSSKrSSKJr SSKr\R"\ 5r \(aSSK r Sr SrSSjr\R "SS 9SS j5rS rSS jrSSS jjrSr"SS5rg)awCUDA weight-only quantization helpers. This module contains small Python utilities for producing the weight layouts consumed by CUDA weight-only kernels. The blockwise quantizers wrap the same C++ pybind entry points used by runtime prepacking, and the mixed-GEMM weight packer is a PyTorch reimplementation of the runtime CUDA packing, so tests and model builders can generate byte-identical quantized weights. The PyTorch packer runs on CUDA when a device is available and falls back to CPU otherwise, which is the only option on platforms where the standalone CUDA packer is not built (Windows). A GPU-gated parity test validates it against that standalone CUDA packer. Two storage families are exposed: * raw MatMulNBits blockwise storage, laid out by output channel as ``[N, K/pack]``; * CUDA mixed-GEMM prepacked storage. MatMulNBits prepacked initializers keep the schema shape ``[N, K/block_size, block_size*bits/8]`` and require the node attribute ``weight_prepacked=1``. QMoE/CUTLASS callers use the kernel-facing shape ``[K, N/pack]``. All public helpers take one logical expert weight matrix with shape ``[N, K]``. ) annotationsN) TYPE_CHECKINGcLSSKnU$![an[S5UeSnAff=f)zQImport torch lazily so importing onnxruntime.quantization does not require torch.rNzMCUDA weight-only quantization requires torch. Please install torch to use it.)torch ImportError)res ڣ/tmp/claude-0/-home-danvics-docker-quiz/c1e0577a-e42c-4a3d-b1ea-3edd61103a4e/scratchpad/stt/lib/python3.13/site-packages/onnxruntime/quantization/cuda_quantizer.py _get_torchr +s5r L rijpqqrs # #cSSKJn UR$![an[S5UeSnAff=f![an[S5UeSnAff=f)aReturn the standalone CUDA mixed-GEMM weight packer (parity oracle). Production packing uses the PyTorch implementation (``_pack_weights_for_cuda_mixed_gemm``). This standalone packer lives in ``onnxruntime.capi.onnxruntime_cuda_quant_preprocess``, a separate extension module that links the CUDA runtime (built only on non-Windows CUDA builds). It is imported lazily here (never at ``import onnxruntime`` time) and is used by the parity test to validate the PyTorch packer byte-for-byte. r)!onnxruntime_cuda_quant_preprocesszThe standalone CUDA weight packer (onnxruntime_cuda_quant_preprocess) is unavailable; it is built only on non-Windows onnxruntime-gpu CUDA builds.NzNonnxruntime_cuda_quant_preprocess is missing pack_weights_for_cuda_mixed_gemm.)onnxruntime.capir r pack_weights_for_cuda_mixed_gemmAttributeError) _cuda_quantrs r %_get_pack_weights_for_cuda_mixed_gemmr5shUs;;;  K   sjkqrrss$ 3 0 +0 A A  Ac:[5 g![a gf=f)a Return True if mixed-GEMM weight prepacking is available. Prepacking is implemented with PyTorch (CUDA when available, CPU otherwise), so it is available whenever torch is importable. Callers use this to skip prepack code paths (and tests) when torch is unavailable. FT)r rr has_cuda_weight_prepackingrLs%   s )maxsizec.[RS5 g)NzCUDA device is not available; packing mixed-GEMM weights on CPU with PyTorch. This is correct but significantly slower for large Mixture-of-Experts models. Pack on a CUDA-enabled machine for best performance.)_loggerwarningrrr _warn_cpu_prepack_oncerZs OO ?rc[5nURR5(aUR"S5$[ 5 UR"S5$)zRPick the torch device for mixed-GEMM weight packing (CUDA if available, else CPU).cudacpu)r r is_availabledevicer)rs r _prepack_devicer!cs@ LE zz  ||F## << rc F [5nSnUS:XaSOSnUR5S:XaURS5n/SQ/SQS.nSnS U-nURSURS URSpn X-S:wdX-S:wa[ S U S U S USUS3 5eUS:a=XdSU3n [ U 5V s/sHoU-U-XU--PM nn USS2USS24nURnUS:XaUR UR5nUS- RSSS 5RS5nUS-S- RSSS 5RS5nUR"UU/SS9RURSSURS 5nUSS2SS2SSS24USS2SS2S SS24S--nUR UR5RU5nO"URSSS 5RU5nXE-nUS :asUS:amSU-nSU-nU U-S:wd U U-S:wa[ SU S35eURU SUU U-US-U-5nURSS SSS5RU5nUS:XaURUR5nUSUS:RUR5-S -- nURSS5SS2/SQ4RU5nURUR5R UR5nGOPUR UR5nUS- RS5nUS-S- RS5nUR"UU/SS9RURSURS S5nURSS5SS2/SQ4RUR5nURUR5nUSUS :RUR5-S-- nUSS2SS2SSS24USS2SS2S SS24S--nURUR5R UR5nURS5R!5$s sn f)!aPyTorch port of the runtime CUDA ``preprocess_weights_for_mixed_gemm``. ``tensor`` is a signed int8 weight in ``(K, N/pack)`` packed row-major layout on any device. Returns the CUTLASS mixed-GEMM layout with the same shape/dtype/device. This mirrors ``preprocess_weights_for_mixed_gemm_cuda`` (permute_B_rows -> subbyte_transpose -> interleave_column_major -> add_bias_and_interleave) so its output is byte-identical to the standalone CUDA packer, for both the SM80 (Ampere) and SM90 (Hopper) layouts. r)rrr% r& r$ ) rrr%r'r#r&r(r)r*r$r+r,r-r.r/r0r1)16_816_4rzweight shape (rows=z, packed_cols=zF) is incompatible with mixed-GEMM packing (rows must be a multiple of z, packed cols a multiple of z).d_NdimZi z num_rows (z0) is incompatible with column-interleave tiling.r(i)rr&rr()rr&r$r.rr(r+r/ir/)r rG unsqueezeshape ValueErrorrangeviewuint8permutecatreshapeint8toint64int16squeeze contiguous)tensorbitssmrbits_abits_bpermutation_map mma_shape_nb_rows_per_mma num_expertsnum_rowsnum_colspmaprrow_idxoriginal_shapeuhighlowmerged interleave rows_per_tile elts_in_int32ts r (_preprocess_weights_for_mixed_gemm_torchrrls LE F!)QF zz|q!!!$G! $OJKv%N&,ll1ov||A UV8K A%)?1)D!(>(D33A2BB^_j^kkm o   Cx6(34^cdl^mn^mYZ'>9D^AS>!  ' ' ))Yos<Rc[5n[U5n[U5nUS;a[SUS35eUS;a[SUS35eSU-n[5nUR"[ R "U55RUR5RXU-5RU5nUS:XaUS-RUR5n US- RUR5n UR"X4URUS 9n XS S 2S S S 24'XS S 2S S S 24'U S- RS S 5R5n U S S 2S S S 24S-U S S 2S S S 24S-S--RUR5RUR5n OpURUR5S- RS S 5R5n U RUR5RUR5n [!U R5X45nURS5R#5R%5$)aAPyTorch implementation of the CUDA ``pack_weights_for_cuda_mixed_gemm``. ``q_weights`` is ORT's unsigned MatMulNBits/QMoE storage ``(N, K/pack)`` (uint8). Returns a flat ``int8`` numpy array with the CUTLASS mixed-GEMM layout, byte-identical to the standalone CUDA packer. Runs on CUDA when available, otherwise on CPU. r$r%zbits must be 4 or 8, got .PrIz/force_arch must be 80 (SM80) or 90 (SM90), got r%r$r1dtyper Nrr&rrCrH)r intrNr! as_tensornpascontiguousarrayrPrQrTrVrXempty transposerZrUrrrnumpy) q_weightsnkr\ force_archrpackr qrlrkunpackedsigned_tpacked_touts r !_pack_weights_for_cuda_mixed_gemmrs LE t9DZJ 64TF!<==!J:,VWXYY 9D  F ,,Y78==ekkJRRST[_V_`ccdjkA qy4xmmEKK(Q{{5;;';;vU[[HADqD ADqDqL++Aq1<<>aAg&-8Aqt!tG3Dt3KPQ2QRVVW\WbWbchhinisistDD%+66q!<GGI;;u{{+00< 283F3F3H$ [C ;;r?   & & ((rcVSSKJnJn X4$![an[S5UeSnAff=f)zCReturn MatMulNBits blockwise quantizers from the ORT pybind module.r)quantize_matmul_4bitsquantize_matmul_8bitszfCUDA blockwise quantization requires quantize_matmul_4bits and quantize_matmul_8bits from onnxruntime.N)onnxruntime.capi._pybind_staterrr)rrrs r _get_quantize_matmul_nbitsrs<  ! 77  t  s ( #(c\rSrSrSr\SS.SSjj5r\SSS.SSjjj5r\SSj5r\SSSSSS .SS jj5r \SSSSSS .SS jjj5r \SSSSSS .SS jjj5r \SS.SSjj5r Sr g) CudaQuantizerizCUDA quantizer utilities for MoE/QMoE and MatMulNBits-style weight-only kernels. The methods are stateless; callers may use the class directly without constructing an object. Tunsigned_full_rangec[5nUR5R5RUR5R 5n[ U5nUS;a[SUS35eURupESU-nXV-S:wa[SUSUS35eUS :XaU(aS upxpOS upxpOU(aS upxpOS upxpUR5RSSS9[U 5- n UR"XR"UR5RS9n UR"UR"X - 5Xx5RUR 5R 5n X-RUR"5n US :Xa>U SS2SSS24S-U SS2SSS24S-S --n U RUR"5n OU n U R 5U R%S5R 54$)aQuantize one QMoE expert with symmetric per-channel storage. ``weights`` has logical shape ``[N, K]``. Returns raw QMoE storage ``[N, K/pack]`` and scales ``[N]``. By default, this emits the ORT CUDA QMoE storage contract: unsigned bytes/nibbles with an implicit zero-point offset, so each stored value is ``q + zero_point`` even though the numeric quantization is symmetric. By default it uses the full ``[-8, 7]`` / ``[-128, 127]`` range. Set ``unsigned_full_range=False`` to use the legacy ``[-7, 7]`` / ``[-127, 127]`` range. rtz=QMoE per-channel quantization only supports 4 or 8 bits, got rur%rK () must be divisible by z# for QMoE per-channel quantization.r$r/r%r%r/r/r%rKrCrCrKrKrCrT)rGkeepdimminNr&r1rH)r detachrrVfloat32rZrzrNrMabsamaxfloatclampfinfoepsroundrXrQrY)weightsr\rrrrrqminqmax scale_divisor zero_pointscales quantizedqweights r #qmoe_symmetric_per_channel_quantize1CudaQuantizer.qmoe_symmetric_per_channel_quantizes" .."&&(++EMM:EEG4y v \]a\bbcde e}}Dy 8q=s1#% !Q1* 5 ; ;A >>>?1> !Q1* 5 ; ;A >>>?IaQRTUTXWXTXjDY\_D_deCee?#((A Q1'.>v.FHXHXYdHee e""133'((A95RXXF1/<hhAIJNq#8:V^`^f^fg 7Q7S44,0AI(;P#v{J1P VVF^F(%*:*:6*BEDTDTU`DaaarF)rreturn_zero_pointsrflatten_qweightrc [RUUUUUUS9upn U(a-URURSS5R 5nU(aXU 4$X4$)akQuantize one expert with ONNX Runtime's MatMulNBits blockwise encoding. ``weights`` has logical shape ``[N, K]``. Returns raw flattened storage ``[N, ceil(K/block_size)*ceil(block_size/pack)]`` and block scales ``[N, ceil(K/block_size)]`` by default. Set ``flatten_qweight=False`` for the MatMulNBits initializer shape ``[N, ceil(K/block_size), ceil(block_size/pack)]``. Set ``return_zero_points=True`` to also return packed block zero-points. Symmetric quantization uses the full ``[-8, 7]`` / ``[-128, 127]`` range by default. Set ``unsigned_full_range=False`` to use the legacy ``[-7, 7]`` / ``[-127, 127]`` range. rrrrrH)rrrTrMrZ) rr\rrrrrrrrrs r matmulnbits_blockwise_quantize,CudaQuantizer.matmulnbits_blockwise_quantizesn0(5'Y'Y   ! 3 (Z( $ oogmmA&6;FFHG K/ /r)rrrrc [5n[U5n[U5n[U5nUS;a[SUS35eUS:XaSOSn X);a[SU SUS US35eURupX-S :wa[S U S US 35e[R UUUUUUS9upn[ nU"U RU S5R5XX5n[R"U5R[R5RU R5nUR"[R"U55nU(aUX4$UU 4$)aQuantize and CUDA-prepack one MatMulNBits weight initializer. ``weights`` has logical shape ``[N, K]``. Returns ``B`` with the standard MatMulNBits initializer shape ``[N, K/block_size, block_size*bits/8]`` and scales with shape ``[N, K/block_size]``. The ``force_arch`` value selects the mixed-GEMM weight layout and must match the ``weight_prepacked`` attribute set on the MatMulNBits node: * ``force_arch=80`` (default): SM80/Ampere layout, consumed by the SM80 kernel (also used on newer GPUs via the compatibility path). Use ``weight_prepacked=1``. * ``force_arch=90``: SM90/Hopper layout, consumed by the native SM90 TMA/WGMMA kernel. Use ``weight_prepacked=2``. Requires ``block_size`` in {64, 128}. rvz3force_arch must be 80 (SM80) or 90 (SM90), but got rurw)rJ@rC)rrCzblock_size must be one of z for force_arch=z , but got rr#) must be divisible by block_size () for CUDA-prepacked weights.rrH)r rzrNrMrrrrTrr|rrPrQrr})rr\rrrrrrrallowed_block_sizesrrrrrrrs r (matmulnbits_prepacked_blockwise_quantize6CudaQuantizer.matmulnbits_prepacked_blockwise_quantizesw4 4y_ _ X %RS]R^^_`a a0:R/?mY  0,-@,AAQR\Q]]ghrgsstu }} >Q s1#%H Tqrs s'4'Y'Y   ! 3 (Z( $,M(1'//!R2H2N2N2PRSX\iF#((2::7==I!!""6"6v">? 6. .v~rc T[U5n[U5nURupSU-n X-S:wa[SU SUS35eX-S:wa[SUSU S35e[R UUUUUUS 9upn [ nU"U R US 5R5XX5n[R"U5R[R5R XU -5n[5nUR"[R"U55nU(aXU 4$X4$) aQuantize one expert and CUTLASS-prepack it for CUDA QMoE fpA_intB GEMM. ``weights`` has logical shape ``[N, K]``. Returns ``qweight`` with shape ``[K, N/pack]`` and block scales with shape ``[N, K/block_size]``. Set ``return_zero_points=True`` to also return packed block zero-points. r%rrrrrrz! for QMoE blockwise quantization.rrH)rzrMrNrrrrTrr|rrPrQr rr})rr\rrrrrrrrrrrrrrrs r !qmoe_prepacked_blockwise_quantize/CudaQuantizer.qmoe_prepacked_blockwise_quantizes3$4y_ }}Dy >Q s1#%H Tqrs s 8q=s1#%? ;. .~rc[5nUR5R5R5nURn[ U5n[ U5nUS:XaU(aSOSupgnO$US:XaU(aSOSupgnO[ SUS35eUS n X-S - U-n X-U - n U S :aG[U5n XS 'UR"XRURS 9n UR"X /S S 9nOUnUR"/USS QU PUP76nUR"UR"U5S S 9S nUR"UU- SS9nUR "UUR#S 5- 5nUR"UXg5nUS:XaUR%UR&5R"/USS QX-P76nU S :a USSU *24nUS-R%UR(5n[U5nUS S -S-US 'UR"UUR(URS 9nUSS SS24S-USSS24'URS S :a4USSUSS SS24RS 24==USS SS24S-S--ss'OSUR%UR&5R"/USS QX-P76nU S :a USSU *24nURU5nUR5UR54$)a)Quantize one expert with a pure-PyTorch symmetric blockwise encoding. This helper is useful for non-CUDA reference paths. Unlike the pybind-backed helpers above, it pads the last dimension when it is not divisible by ``block_size`` and returns storage with the same leading shape as ``weights``. r$)rr/r%)rr/r/r%)rrKrC)rrKrKz;CUDA blockwise quantization only supports 4 or 8 bits, got rurHrrrxrFNg:0yE>r.r&r1)r rrrZrMrzrNlistrryr rSrPrrrrrLrVrUrQ)rr\rrrrirrrlast_dimrpad_size pad_shapepaddingweights_paddedreshaped_weights block_max_absrrquantized_flatquantized_uint4 packed_shapers r symmetric_blockwise_quantize*CudaQuantizer.symmetric_blockwise_quantizeDs$ .."&&(335 4y_ 196I z %D QY *X5 a<^,I$bMkk)==XG"YY'9rBN$N)..\s0C\Z\Q[\ %)),<"=2FqI ]]:EKK 063C3CB3G GH KK 46 19&\\%**5::iN3BGCFO$$R(1,CsADqDy9??CCCDY\^_^bab^bYbIcfiIinoHooDll5::.33b^CR5Hb*JabG!|!#z z/2ll>2G{{}fjjl**rrN)r torch.Tensorr\rzrboolreturn!tuple[torch.Tensor, torch.Tensor]rw) rrr\rzrrrrzrrrr)rrr\rzrrzrrrrrrrz/tuple[torch.Tensor, torch.Tensor, torch.Tensor])rrr\rzrrzrrrrrrrrrrrStuple[torch.Tensor, torch.Tensor] | tuple[torch.Tensor, torch.Tensor, torch.Tensor])rrr\rzrrzrrzrrrrrrrrrr) rrr\rzrrzrrrr)__name__ __module__ __qualname____firstlineno____doc__ staticmethodrrrrrrr__static_attributes__rrr rrs  %) 1E1E1E" 1E + 1E1Ef  F %) FFFF F " F +FFBAbAbAbAb  Ab  Ab"Ab 9AbAbF #( $$($$$$  $ ! $$$"$ ]$$L  ; #($(;;;; ;  ;!;;"; ];;z  + #( $(++++ +  +!++"+ ]++Z %) >+>+>+>+ " >+ + >+>+rr)rr)rNone)r\rzr]rzr) rrzrrzr\rzrrzrz np.ndarray)r __future__r functoolsloggingtypingrrr| getLoggerrrrr rr lru_cacherr!rrrrrrrr r s,#    H %s.  Q n*b!)H 8s+s+r