ybj SSKJr SSKrSSKrSSKrSSKrSSKJr SSKJ r SSK r SSK r SSK J r JrJr SSK JrJrJr SSK Jr SSKJrJrJrJr SS KJr SS KJrJrJr SS K J!r! S r#S r$Sr%Sr&Sr'Sr(Sr)Sr*Sr+Sr,0r-\R"R\\R"R^\R"R`\R"Rb4r2\3"\5Vs0sH$n\4"\5"\U5\65(dM\5"\U5U_M& snr7"SS\5r8"SS\5r9"SS\5r:"SS\5r;\R"Rx\ Rz"S5\R"R|\ Rz"S5\R"R~\ Rz"S 5\R"R\ Rz"S!5\R"R\\ \R"R\\R"R\0rC\R"R|\ R"S\ RS"9\ R"S#\ RS"94\R"Rx\ R"S$\ RS"9\ R"S%\ RS"94\R"R\ R"S\ RS"9\ R"S&\ RS"94\R"R~\ R"S'\ RS"9\ R"S(\ RS"94\R"R\ R"S\S"9\ R"S)\S"94\R"R\ R"S*\S"9\ R"S+\S"940rI\R"Rx\ R"S,\ RS"9\ R"S%\ RS"94\R"R~\ R"S-\ RS"9\ R"S(\ RS"940rJ\R"R|\ R"S\ RS"9\ R"S%\ RS"94\R"Rx\ R"S.\ RS"9\ R"S/\ RS"94\R"R\ R"S\ RS"9\ R"S(\ RS"94\R"R~\ R"S0\ RS"9\ R"S1\ RS"94\R"R\ R"S\S"9\ R"S+\S"94\R"R\ R"S2\S"9\ R"S3\S"940rKS4S5.S6jrLSbS7jrMScS8jrNSdSeS9jjrOS:rPSfS;jrQSgShS<jjrRSgSiS=jjrSSjSkS>jjrTSlS?jrUSlS@jrVSmSAjrWSnSBjrX"SCSD5rY"SESF5rZ"SGSH5r[SIr\SJr]SKr^SLr_SoSMjr`SNraSpSOjrbSqSPjrcSrSQjrdSsSRjreStSSjrfSuSTjrgStSUjrhSuSVjriSvSWjrjSjSwSXjjrkSxSYjrlSySZjrmSzS[jrnS{S\jroS{S]jrpS|S^jrqS|S_jrrS|S`jrsS|Sajrtg!\"a Sr!GNf=fs snf)}) annotationsN)Enum)Path) float8_e4m3fnint4uint4) ModelProto TensorProtoexternal_data_helper)onnx_pb) make_graph make_model make_nodemake_tensor_value_info)ReferenceEvaluator)GraphOptimizationLevelInferenceSessionSessionOptions)to_array_extendedz onnx.quantizez0.1.0ai.onnxz com.microsoftQuantizeLinear_QuantizeLinear_InputDequantizeLinear_DequantizeLinear_Output _quantizedlc2\rSrSrSrSrSr\S5rSr g)QuantizationMode:rcUR$Nnameselfs ڠ/tmp/claude-0/-home-danvics-docker-quiz/c1e0577a-e42c-4a3d-b1ea-3edd61103a4e/scratchpad/stt/lib/python3.13/site-packages/onnxruntime/quantization/quant_utils.py__str__QuantizationMode.__str__> yycF[U$![a [5ef=fr!)rKeyError ValueError)modes r& from_stringQuantizationMode.from_stringAs) #D) ) ,    N) __name__ __module__ __qualname____firstlineno__ IntegerOps QLinearOpsr' staticmethodr/__static_attributes__r2r*r&rr:s%JJr*rc2\rSrSrSrSrSr\S5rSr g)QuantizedValueTypeIrrcUR$r!r"r$s r&r'QuantizedValueType.__str__Mr)r*cF[U$![a [5ef=fr!)r<r,r-)vs r&r/QuantizedValueType.from_stringPs) %a( ( ,  r1r2N) r3r4r5r6Input Initializerr'r9r/r:r2r*r&r<r<Is% EKr*r<cV\rSrSrSrSrSrSrSrSr Sr S r \ S 5r \S 5rS rg ) QuantTypeXrrcUR$r!r"r$s r&r'QuantType.__str__ar)r*cF[U$![a [5ef=fr!)rFr,r-)ts r&r/QuantType.from_stringds( Q<  ,  r1cU[R:Xa[R$U[R:Xa[R $U[R :Xa[R$U[R:Xa[R$U[R:Xa[R$U[R:Xa[R$U[R:Xa[R$[!SU<S35e)NzUnexpected value qtype=.)rFQInt8r INT8QUInt8UINT8QUInt16UINT16QInt16INT16 QFLOAT8E4M3FN FLOAT8E4M3FNQUInt4UINT4QInt4INT4r-r$s r& tensor_typeQuantType.tensor_typeks 9?? "## # 9## #$$ $ 9$$ $%% % 9## #$$ $ 9** *++ + 9## #$$ $ 9?? "## #24(!<==r*r2N)r3r4r5r6rTrVr\rZrXr`r^r'r9r/propertyrbr:r2r*r&rFrFXsR E FM FG E F >>r*rFc2\rSrSrSrSrSr\S5rSr g) QuantFormat~rrcUR$r!r"r$s r&r'QuantFormat.__str__r)r*cF[U$![a [5ef=fr!)rfr,r-)formats r&r/QuantFormat.from_strings) v& & ,  r1r2N) r3r4r5r6 QOperatorQDQr'r9r/r:r2r*r&rfrf~s%I Cr*rfint8uint8int16uint16dtypeiiiiiiii@ii@rIzero_point_indexcd/n[U5GHup4[R"[U5[R5(a&UR [R "U55 OB[U[R5(aUR U5 O[SUSU35eX0:XdMUSnUR[R:Xd UR[R:XdM[SUR35e [U5S:a [U5$US$)Nzarg z is not an array: r{zzero_point cannot be rr) enumeratenumpy issubdtypetypenumberappendarray isinstancendarray TypeErrorrtfloat32float16lentuple)r}argsnew_argsiarAs r& _check_typersH$   DGU\\ 2 2 OOEKKN + 5== ) ) OOA d1#%7s;< <  Aww%--'177emm+C"7y ABB "(ma/5?@Xa[@r*cU[;d SUS35eU[RR[RR[RR [RR 4;Ga/US:wa[SU<S35eUR[R:Xa[RnOHUR[R:Xa[RnO[SURS35e[[![#S/S/[$R&R)SU/S/5S 9[#S /S QS /5/S [+SUS5[+SUS5/[+S US5/55n[-U5n[/UR1SXS.5S5$[Un [3USSS9upUb [5X5OU n Ub [7X5OU n [R8"UR;[R5U- R=5U-5n[R>"XXS9 [/UR;U 55$)NUnexpected data type > requested. Only INT8, UINT8, INT16, and UINT16 are supported.rz2zero_point is expected to be null for float 8 not rSzUnexpected dtype Constant zero_point)valuer)XscalerYqurr)rrF reduce_range symmetric)out) ONNX_TYPE_TO_NP_TYPE onnx_protor r]FLOAT8E4M3FNUZ FLOAT8E5M2FLOAT8E5M2FNUZNotImplementedErrorrtrrFLOATrFLOAT16r-rr ronnxhelper make_tensorrrrrunget_qmin_qmax_for_qTypemaxminasarrayastyperoundclip)qTypearrrrlowhigh onnx_type onnx_modelrefrtqminqmaxcliplowcliphigharr_fp32s r&quantize_nparrayrs! ( ( w&de ( ++--))--  ?%(Z[eZhhi&jk k 99 %#))I YY%-- '#++I0 1=> > "Bdkk>U>UVbdikmpqor>s.0LseT  *3 4@*7ItD(UD9:  !,3774s)CDQGHH%U+,URWX $'O#d.&*&63t?D==#**U]]";e"C!J!J!Lz!YZ 8h=8??5122r*c US:dUS:a[SUSU35e[R"U[R"SURS95n[R "U[R"SURS95nUb*[ X[R"XPRS9-5nU(aE[R "[R"U5[R"U55nU*nU7nX#::d SUSU35e[R"X- [RS9n[R"U[RS9[R"U[RS9- n[R"Xx- 5n U S:dS5eU [R"UR5R:aA[R"SURS9n [R"SURS9n X/$U(aZ[R"[R"X#-[R"S [RS9- 5URS9n O8[R"[R"X U - - 5URS9n U RUR5n X/$) aCalculate the scale s and zero point z for the quantization relation r = s(q-z), where r are the original values and q are the corresponding quantized values. r and z are calculated such that every value within [rmin,rmax] has an approximate representation within [qmin,qmax]. In addition, qmin <= z <= qmax is enforced. If the symmetric flag is set to True, the interval [rmin,rmax] is symmetrized to [-absmax, +absmax], where absmax = max(abs(rmin), abs(rmax)). :parameter rmin: minimum value of r :parameter rmax: maximum value of r :parameter qmin: minimum value representable by the target quantization data type :parameter qmax: maximum value representable by the target quantization data type :parameter symmetric: True if the floating-point range should be made symmetric. Defaults to False. :parameter min_real_range: Minimum floating-point range (i.e., rmax - rmin) to enforce. Defaults to None. :return: zero and scale [z, s] rBqmin and qmax must meet requirement: qmin <= 0 <= qmax while qmin:, qmmax:rszqmin=z > qmax=z scale issue?@)r-rminimumrrtmaximumrrabsfloat64finfotinyrr) rminrmaxrrrmin_real_rangeabsmaxdrdqrrs r&compute_scale_zprs( ax4!8]^b]ccklpkqrss ==u{{1DJJ? @D ==u{{1DJJ? @D!4 nJJ OOPuyy $@ww <55htf55< T[ 6B T /%++d%--2X XB KK E A:$}$: u{{4::&+++ Ctzz2[[$**5      T[EKK5==,QQRZ^ZdZdJU[[u 1D%ETZZXJ TZZ(  r*c[U5n[U5nXV-S-S-n[[R"U55n[[R"U55n[ US5n[ US5nUbUS:a[ X[U5-5nX::aUS:aUOUn[ [ U5[ U55n X:XaXe- O[ SXe- S-5n U S:aU OS[ SU 5- n UbXXe- - :aXFU- - n [R"U[RS9[R"U [RS94$US:aL[R"U[RS9n [R"XU- - [RS9n O^[R"U[RS9n U*Xu- - nXU- - n[R"[ X5[RS9n Ub<[U 5XFU- - :a([R"XFU- - [RS9n X4$)aSnap a uint8 activation zero-point to qmin (when rmin >= 0) or mid (when rmin < 0). Used by the ActivationRestrictedAsymmetric quantization option. Recomputes scale so the dequantized range still covers [rmin, rmax] without clipping. :parameter rmin: calibrated minimum activation value (numpy scalar) :parameter rmax: calibrated maximum activation value (numpy scalar) :parameter qmin: minimum quantized value (int, default 0) :parameter qmax: maximum quantized value (int, default 255) :parameter min_real_range: minimum floating-point range to enforce (same semantics as compute_scale_zp). When not None and > 0, rmax is adjusted to max(rmax, rmin + min_real_range) before scale computation. :return: (zero_point, scale) with zero_point dtype uint8 and scale dtype float32 rrHrrrs) intfloatrsqueezerrrrrpr)rrrrrqmin_valqmax_valmid degenerate_zpabs_maxdenom scale_valrr scale_neg scale_poss r&snap_zero_point_to_uint8r2s4yH4yH   "q (C t$ %D t$ %D tS>D tS>D!nq&84n 556 |%)CKS c$iT+)6)B$APXPchiOiHj '! WAu E  %)H[6\*\&X*=>I{{= iii s{[[=  Dx$78 N[[EKK8 ES^, sN+  C 5U]]K!eEl^RZGZ5[&[ N.AB%--X  r*cSnU[;aU[R:XaSSKJn Un[ R "S[ RS9RU5R[ R5n[ R"UVs/sH?n[ R"U5(aM [ R"U5(aM=UPMA sn[ RS9nO[SUS35eU[U'OU[R:XaSSKJn UnUc[SUS 35e[ R "[U5n[ R"SUS9n[ R"X- UR"S9n X/$s snf) aZCalculate the scale s for a float8 type (E4M3FN). The function assumes the coefficient distribution and the float 8 distribution are similar to two gaussian laws. :return: zero and scale [z, s] More details in notebook `quantization_fp8.ipynb `_. Nr)rrszQuantization to element_type=z not implemented.zUnexpected element_type rS)FLOAT8_DISTRIBUTIONSr r] ml_dtypesrrarangerpviewrrrisnanisinfr-rstdrt) element_typerzp_dtyper all_valuesfvaluesstd_f8zerors r&compute_scale_zp_float8rms5H// ;33 3 /$Hc=BB=QXXY^YfYfgJ[[&TJqekk!nU[[QR^JT\a\i\iF<\NJ[\] ]-3\* 11 1+ 2<.BCC YY+L9 :F ;;q )D KK CII 6E =Us F/F Fc URS:wa&[SURSURS35eURUnXS-S- U-n[[R "[ UR5VVs/sHupxXr:wdM UPM snn55n [R"XS5n U RXY5n [USUS9up[USRn Xc-U- nUS:a7[R"X4U RS 9n[R"X/SS 9nOU nURXcU 5nURSS 9nURSS 9n[R "U[R""U55n[R$"U[R""U55nU(aD[R$"[R&"U5[R&"U55nU*nUn[R("U 5n[R("U 5nUU- R+[R(5nUU- nUU- R+UR5n[R,"UR5R.nUU:n[R0"U[R2"U5U5nU(a=[[R4"UU-S - 55n[R6"Xi4UU S 9nO[R4"UUR+[R(5UR+[R(5- - 5n[R8"UUU5nUR+U 5nSUU'[R"USU5n[R"USU5nUU4$s snnf) asCompute per-block scale and zero-point for a weight tensor. The weight is sliced along *axis* into blocks of *block_size* elements. Per the ONNX opset-21 spec, QuantizeLinear/DequantizeLinear require the scale and zero_point tensors to have the **same rank** as the input tensor. Only rank-2 weight tensors are supported; rank > 2 is explicitly rejected. Returns arrays with the same rank and dimensions as *weight*, except ``shape[axis] == ceil(weight.shape[axis] / block_size)``. This matches the ONNX opset-21 QuantizeLinear/DequantizeLinear blocked-quantization spec. :param weight: Float32/float16 weight array (must be rank-2). :param quant_type: ONNX tensor data type for quantization. :param axis: Axis along which to apply block-wise quantization. :param block_size: Number of elements per block along *axis*. :param symmetric: Whether to use symmetric quantization per block. :return: Tuple of (zero_point, scale), each with shape matching *weight* except ``shape[axis] == n_blocks``, where ``n_blocks == ceil(weight.shape[axis] / block_size)``. :raises NotImplementedError: If weight rank is not 2 (opset-21 constraint). rHzXPer-block (opset-21) quantization is only supported for rank-2 weight tensors. Got rank-z tensor with shape zY. For rank > 2 tensors, reshape to 2-D before quantizing or use per-channel quantization.rrFrrs)axisr)ndimrshaperrprodrmoveaxisreshaperONNX_INT_TYPE_RANGErtzeros concatenaterrr zeros_likerrrrrrwhere ones_likerfullr) weight quant_typer block_sizerkn_blocksrdothermovedrrrpad_lenpad moved_paddedblocksrrrrrrr raw_scaler degeneratescaleszp_val zero_pointsraw_zps r&compute_scale_zp_blockedrs6{{a! }$7 ~Ff f   TA"z1H  )FLL*AO*A$!QYA*AOP QE NN6 +E MM! #E(%S\]JD":.q177H#a'G{kk7*%++>((%A>   ! !( >F ::1: D ::1: D ==u//5 6D ==u//5 6Duyy $@w}}T"H}}T"H +  emm ,B H Bb  .I ;;v|| $ ) )DT!J [[U__Y%? KFU[[(X"5!<=>jj(!2F(K X EMM(BV]]SXS`S`Ea(aabFHh7mmH- "# J ^^FAt ,F..a6K  uPs = O Oc[U[R5(d[S[ U5S35eUbUnO"[ U5(aUR 5OSnUbUnO"[ U5(aUR5OSn[R"XpRS9n[R"XRS9n[R"SURS9n U[R:Xa?U(a [S5e[R"U5n [X5up[XSS9$U[R [R"[R$[R&[R([R*4;aU[-XUS 9up[ U5(a[/XxXX$5upO[R"SU RS9n [XSS9$[1S US35e) a Returns the zero_point and scale for the given data. :param data: The data for which to compute quantization parameters. :param quant_type: The quantization data type. :param symmetric: whether symmetric quantization is used or not. :parameter reduce_range: True if the quantization range should be reduced. Defaults to False. :parameter min_real_range: Minimum floating-point range (i.e., rmax - rmin) to enforce. Defaults to None. :parameter rmin_override: The value of rmin to use if not None. Otherwise, uses min(data). :parameter rmax_override: The value of rmax to use if not None. Otherwise, uses max(data). :return: zero point and scale z%Weight must be given as an array not rSrrsrz1Unsupported option reduce_range=True for float 8.rr|rz Unexpected value for quant_type=)rrrrrrrrrrtr r] RuntimeErrorrrrrUrWr[rYrar_rrr-)datarrrr rmin_override rmax_overriderrrrrrrs r&compute_data_quant_paramsrs* dEMM * *?T |1MNN  YYtxxzC  YYtxxzC ;;t:: .D ;;t:: .D KK4:: .E[--- RS Siio3JD :qAA -ZQZ[  t99 0T c JQdjj9J:qAA 7 |1E FFr*c [UUUUUUU5upxU[R:Xa[XX5n [ U R [ R5R5S-S:H5(af[ R"U5n [SU R5SU R5SU R5SU R5S3 5eXxU 4$U[R[R[R[R [R"[R$4;a[XX5n XxU 4$['SUS35e)a  :param data: data to quantize :param qType: data type to quantize to. :param symmetric: whether symmetric quantization is used or not. :parameter reduce_range: True if the quantization range should be reduced. Defaults to False. :parameter min_real_range: Minimum floating-point range (i.e., rmax - rmin) to enforce. Defaults to None. :parameter rmin_override: The value of rmin to use if not None. Otherwise, uses min(data). :parameter rmax_override: The value of rmax to use if not None. Otherwise, uses max(data). :return: minimum, maximum, zero point, scale, and quantized weights To pack weights, we compute a linear transformation - when data `type == uint8` mode, from `[rmin, rmax]` -> :math:`[0, 2^{b-1}]` and - when data `type == int8`, from `[-m , m]` -> :math:`[-(2^{b-1}-1), 2^{b-1}-1]` where `m = max(abs(rmin), abs(rmax))` and add necessary intermediate nodes to transform quantized weight to full weight using the equation :math:`r = S(q-z)`, where - *r*: real original value - *q*: quantized value - *S*: scale - *z*: zero point rvz+One of the quantized value is NaN data in [z, z], quantized_data in [z].zUnexpected value for qType=rS)rr r]ranyrrrpravelrrrrrUrWr[rYrar_r-) rrrrrrrrrquantized_datanp_datas r& quantize_datar2sO82  J ((()%uI ##EKK06683>3F G GmmD)G=gkkm_Bw{{}o^&&4&8&8&:%;2n>P>P>R=SSUW .00  *%uI.00 25'; <>*A6$$Q'==jI""5??C*$Eej(!,CU|*:eCin 5 ; ; ={3PS8?TV^_bdg_gVh+c 3'$# OOQC;;L;L1M"[1MQRQZ11M"[[ \^_ae  (5F5F5H%\ #))$/ K--.  T&(#}%A&++A4L!HM!+ %5L..0-AS& " ( . .u}}=S/T/\/\]i/j k&))*EtL ):%6;;-PhOi@jMT%%222#//1)3&!!((5$1!(5(=(=(?(D(D(F(N(N(P%  (&&:;E{{k///5==?mF[F[F]3]"/ 0A0A/BBc$,,.s34F5==?3B;O:PP[\b\h\h[iS!56t<=Q@(  ((--t/?/?/E/EF F   tUm 3!7 Ftuv v.}/D/D/FGH  ${{66}jR]R]_jpt6u  ==jI m>JRRSYS^S^_ #00;;M=Y Q Y"#\s T.$T.- T4<T4cU[RR:Xa [S5eSnU(a[R U5nO0U(aU[ ;a [ UnO[R U5nU(d[SUS35eUupEUS:dUS:a'[SUSUSURS US US U3 5eU$) z Return qmin and qmax, the minimum and maximum value representable by the given qType :parameter qType: onnx.onnx_pb.TensorProto.UINT8 or onnx.onnx_pb.TensorProto.UINT8 :return: qmin, qmax z;This function is not implemented for float 8 as not needed.Nrrrrrz, dtype=z, reduce_range=z , symmetric=z, qType=) rr r]rONNX_INT_TYPE_REDUCED_RANGEgetONNX_INT_TYPE_SYMMETRIC_RANGErr-rt)rrrqrangerrs r&rrs  &&333!"_`` F,007 u ==.u5$((/ 07uvwwJD ax4!86$x |?<.Y" 8E7 4  Mr*c"[XUS9up4XC- $)z Helper function to get the quantization range for a type. parameter qType: quantization type. return: quantization range. r)r)rrrrrs r&get_qrange_for_qTyperTs ) RJD ;r*c@US:aX-OUnUS:=(a X!:nX24$)z Helper function that tries to return a normalized axis in the range [0, rank - 1]. :parameter axis: The axis to normalize. :parameter rank: The tensor rank (number of dimensions). :return (is_valid, axis_norm) rr2)rrank axis_normis_valids r&normalize_axisrYs- $ax TIA~2)"2H  r*c[U5nUS:Xa [5$US-S-n[U5nSnSnXAS- :a+XS-S-S-XS--X5'US- nUS- nXAS- :aM+XA:a XS-X5'U$)a. Copies a source array of 8-bit values into a destination bytearray of packed 4-bit values. Assumes that the source values are already in the appropriate int4 range. :parameter src_8bit: The 8-bit element values to pack. :return A bytearray with every two 8-bit src elements packed into a single byte. rrrHrwrJ)r bytearray)src_8bit num_elemsdst_sizedstsrc_idst_is r&r4r4 sH IA~{A !#H H CEE a-  *S0Q68?S;PQ      a-   _s* Jr*c(\rSrSrSr//S4SjrSrg)QuantizedInitializeri)zB Represents a linearly quantized weight input from ONNX operators Nc pXlX lX0lX@lXPlX`lXplXlXlgr!) r# initializerrminsrmaxsrrrrr) r%r#rerfrgrrrrrs r&__init__QuantizedInitializer.__init__.s4 &  &  , r*) rrrer#rrgrfrrr3r4r5r6__doc__rhr:r2r*r&rcrc)s r*rcc*\rSrSrSrSSjrSrg)QuantizedValueiHzA Represents a linearly quantized value (input\output\intializer) Nc pXlX lX0lX@lXPlX`lXplXlXlgr!) original_nameq_name scale_namezp_name value_typer node_type node_qtype scale_type) r%r#new_quantized_namerqzero_point_namequantized_value_typerrtrurvs r&rhQuantizedValue.__init__Ms2"( $& . "$$r*) rrurtrorprqrvrsrr)NNNNrjr2r*r&rmrmHs%r*rmc\rSrSrSrSrSrg)BiasToQuantizeidz# Represents a bias to be quantized c(XlX lX0lgr! bias_name input_name weight_name)r%rrrs r&rhBiasToQuantize.__init__is"$&r*r~Nrjr2r*r&r|r|ds 'r*r|cURS:Xa[SURS35eURS:XaURnGO,URS:XaURnGOURS:Xa UR nOURS:Xa UR nOURS:Xa URnOURS :Xa URnOURS :Xa URnO}URS :Xa URnO`URS :Xa URnOCURS :Xa URnO&[SURSURS35eURU0$)z Convert attribute to kwarg format for use with onnx.helper.make_node. :parameter attribute: attribute in AttributeProto format. :return: attribute in {key: value} format. rz attribute z does not have type specified.rrHrIrJrKrLrx r!z has unsupported type rS) rr-r#rrsrPgfloatsintsstringstensorsgraphs) attributers r&attribute_to_kwargros; ~~:inn%55STUU~~  1   1   1   1   1    1  1 !! 1 !! 2   :inn%55KINNK[[\]^^ NNE ""r*c~UVs/sHo"RU:XdMUPM nn[U5S:aUS$S$s snf)z Helper function to find item by name in a list. parameter item_name: name of the item. parameter item_list: list of items. return: item if found. None otherwise. rN)r#r) item_name item_listitemitemss r& find_by_namers@( Bid99 +ATiE B5zA~58/4/ Cs::cXSn[[U55HnXU:XdM UnM U$)z; Helper function to return index of an item in a node list r{)r'r) elem_name elem_listelem_idxrs r&get_elem_indexrs2H 3y> " <9 $H# Or*cF[RRSX/U5$)z Helper function to create a Mul node. parameter inputs: list of input names. parameter output: output name. parameter name: name of the node. return: Mul node in NodeProto format. Mul)rrr)inputsoutputr#s r& get_mul_noders ;; $ ??r*clURRURU-UR-5$)zh Helper function to generate a identifiable filepath by concatenating the given identifier as a suffix. )parentjoinpathstemsuffix)filename identifiers r&generate_identified_filenamers+ ?? # #HMMJ$>$P QQr*cRSSKnSSKJn SSKnUR"UR S9 [ S5 [ U5 [ S5 [ U5 URXSS9 URS5 URS5 URS 5 UR5 g) Nr) thresholdz Histogram:zHistogram Edges:T)fillz Tensor valueCountszTensor value V.S. Counts) sysmatplotlib.pyplotpyplotrset_printoptionsmaxsizeprintstairsxlabelylabeltitleshow)hist hist_edgesrpltrs r& apply_plotrs{# S[[1 , $K  *JJtdJ+JJ~JJxII()HHJr*c $ SSKnSSKnSSKnSSKJs Js Jn SSKJs Js Jn SSK J n [R"SU35 URXS9n[[R R#US5S5n U R%U5 SSS5 UR'S5n UR)S5n /n [+UR-55Hn X nUR/5n[1UR3S U 5R555[1UR3S U 5R555/n[7[9U55nU R;U 5nU R;U5nUR=U 5 UR?U U5 URAU U5 URCU 5nU REU5 M URGU [IU 55 U HnU RKU5 M U RM5nUROU 5 URQU U5 URSU 5nU RUU5 U RW5n[[R R#US 5S 5n U R%U5 SSS5 [RXR3S S5S;aURR[US5nUR]5n[_U5H\nURaU5n[R"URc55 [R"URe55 M^ [[R R#US5S5n [+UR-55Hn X nUR/5n[1UR3S U 5R555[1UR3S U 5R555/nU S-[7[9U55-nU R%U5 U R%S5 M SSS5 g!,(df  GN=f!,(df  GN=f!,(df  g=f)z6 Helper function to write calibration table to files. rN)CalibrationCacheEncoderzcalibration cache: )clszcalibration.jsonwihighestlowestzcalibration.flatbufferswbQUANTIZATION_DEBUG0)r1zcalibration.cache  )3json flatbuffersr5onnxruntime.quantization.CalTableFlatBuffers.KeyValue quantizationCalTableFlatBuffersKeyValue5onnxruntime.quantization.CalTableFlatBuffers.TrtTableTrtTable"onnxruntime.quantization.calibraterlogginginfodumpsopenospathjoinwriterBuildersortedkeysto_dictrrPrr2r CreateString KeyValueStartKeyValueAddKeyKeyValueAddValue KeyValueEndrTrtTableStartDictVectorrPrependUOffsetTRelative EndVector TrtTableStartTrtTableAddDict TrtTableEndFinishOutputenvironGetRootAsTrtTable DictLengthr'DictKeyValue)calibration_cachedirrrnprrr json_datafilerbuilderkey_value_listkeyrd_valuesrrflat_key flat_value key_value main_dict cal_tablebufdict_lenrs r&write_calibration_tablers LLLLK LL&'8&9:; , JI bggll3 23S 9T 9 : 88A;D!!$'GN',,./"'>># (,,y$/446 7 (,,x.335 6 CK '',))%0 w'2!!':6((1 i(#0& $$Wc..AB# '' 2$!!#I 7# Wi0$$W-I NN9 .. C bggll3 9:D AT 3 B zz~~*C0H<%%77Q? '')xA!q)I LL ) LL* +! bggll3 34c :d+0023C&+F~~'Hhll9d388:;hll8T2779:F#ICK 00E JJu  JJt 4 ; :g : 9L B A ; :s%=QQ/CR Q,/ Q> RcUS:HR[R5nUS:gR[R5nUR5nURU- nU(dgU[ U5-[ U5- nUS:dSUSUSU35eUR[R5nXqU-U*U--- nUS:*R5S:XdeU$)ajGiven a discrete distribution (may have not been normalized to 1), smooth it by replacing zeros with eps multiplied by a scaling factor and taking the corresponding amount off the non-zero values. Ref: http://web.engr.illinois.edu/~hanj/cs412/bk3/KL-divergence.pdf https://github.com//apache/incubator-mxnet/blob/master/python/mxnet/contrib/quantization.py rNrzn_zeros=z , n_nonzeros=z, eps1=)rrrsumsizer)pepsis_zeros is_nonzerosn_zeros n_nonzeroseps1rs r&smooth_distributionr #sQu}}-H6//%--0KllnG'!J  w % "3 3D #:Q'- |74&QQ: 88EMM "D(Nte{2 22D AI??  !! ! Kr*c[R"UR5SS9n[SURR 55$)NF)load_external_datac3N# UHn[R"U5v M g7fr!)r uses_external_data).0 intializers r& *model_has_external_data..>s!mUlz#66zBBUls#%)rloadas_posixrgraphre) model_pathmodels r&model_has_external_datar<s9 IIj))+ FE mUZU`U`UlUlm mmr*c[5nUR5Ul[RUl0nS/US'[ UR5U4SS/0UD6ng)z Generate model that applies graph optimization (constant folding, etc.) parameter model_path: path to the original onnx model parameter opt_model_path: path to the optimized onnx model :return: optimized onnx model ConstantSharingdisabled_optimizers providersCPUExecutionProviderN)rroptimized_model_filepathrORT_ENABLE_BASICgraph_optimization_levelr)ropt_model_path sess_optionkwargs_s r&optimize_modelr$Asb!"K+9+B+B+DK(+A+R+RK( F%6$7F !,,. jH^G_jcijAr*cSS0nUR(a:URH*nURURUR05 M, [R R X5 g)z>Tag the model that it went through quantization pre-processingonnx.quant.pre_processonnxruntime.quantNmetadata_propsupdaterrrrset_model_props)rr)props r&add_pre_process_metadatar-QsS.0CDN ((D  ! !488TZZ"8 9)KK6r*cUR(a7URH'nURS:XdMURS:XdM' g g)zCCheck the model whether it went through quantization pre-processingr&r'TFr)rr)rr,s r&model_has_pre_process_metadatar0Zs< ((Dxx33 FY8Y) r*cSS0nUR(a:URH*nURURUR05 M, [R R X5 g)N onnx.inferr'r()rr)rs r&add_infer_metadatar3csS"$78N %%A  ! !155!''"2 3&KK6r*cUR(a7URH'nURS:XdMURS:XdM' g g)Nr2r'TFr/)rrs r&model_has_infer_metadatar5ks; %%Auu $4G)G& r*cURVs/sH'oR(aURS:XdM%UPM) nn[U5S:wa [S5eUSRnU$s snf)Nrrz$Failed to find proper ai.onnx domainr) opset_importdomainrr-version)ropsetai_onnx_domain opset_versions r&get_opset_versionr=ssf).););m);<a?@@"1%--M  ns $A*A*c[U5nUn[USU5nUb [USU5OSn[RR[RR 4n Xy;=(d X;n U (dU(a[ R[ R1n UR5H`n U HNn U RS5nX;aSn O5U RS5nUcM3URS5nUU ;dMLSn O U (dM` O US:a#US:a[R"SUS 35 SnOUS :a;U[RR :Xa[R"SUS 35 S nOmUS:a$U (a[R"SUS 35 SnOCUS :Xa[R"SUS35 O"US :a[R"SUS35 SnXe:wa*[R"R%X5n['U5nU$![[4a [R"S5 GN1f=f)NrbrTconvertzYSkipping 16-bit opset bump heuristic for TensorQuantOverrides: structure not as expected.rz$The original model opset version is z, which does not support block-wise quantization natively. Please update the model to opset >= 21. Automatically updating the model to opset 21. Please verify the quantized model.z, which does not support quantization to float 8. Please update the model to opset >= 19. Automatically update the model to opset 19. Please verify the quantized model.z, which does not support 16-bit integer quantization natively. Please update the model to opset >= 21. Automatically update the model to opset 21. Please verify the quantized model.r!ze, which does not support node fusions. Please update the model to opset >= 11 for better performance.z, which does not support quantization. Please update the model to opset >= 11. Automatically update the model to opset 11. Please verify the quantized model. )r=getattrrr rYr[rFrZrXrrPAttributeErrorrrdebugwarningr]version_converterconvert_version&save_and_reload_model_with_shape_infer)r weight_typeactivation_typetensor_quant_overridesrr<target_opset_versionweight_quant_typeactivation_quant_type _int16_typesneeds_opset21_for_16bit_int16_quant_typesoverrides_listoverrideqtr? convert_qts r&update_opset_versionrW|sg&e,M( ]KHDSD_@ei$$++T-=-=-C-CDL/?hCXCh #'='.. 0A0AB w"8"?"?"A .H!l3B/26/&ll95G*%,[[%> %);;6:3!!/+*#B&rj1n2=/B1 1 "   1T5E5E5R5R R2=/B1 1 "   72=/B1 1 " " 2=/BM M  2=/B1 1 ",&&66uS7u= Lg * w MMu v ws+AH'H HHH&IIc[US5n[RR[ U5[ U55 [R "UR 55n[U5 UR5 U$)Nz -inferred) rrshape_inferenceinfer_shapes_pathr2rrr3unlink)rinferred_model_pathrs r&load_model_with_shape_inferr]s`6z;O**3z?C@SrsL #   00>>&QQ-PP7      ,$2'!''))%%))  58 4Dq4Dq SZ[fhiSjloHp* Q'*4Dq  t   #>#>L $  V!4  %++g"6  %++g"6!!5;;x#8''  %  5;;q #DekkRU]b]h]hFi"j%++d%**"Eu{{SV^c^h^hGi!j!!EKK$F TYafamamHn#o  5;;vU[[#I5;;W\didodoKp"q  5;;q#> BV[@\"]%++b"=u{{1TX?Y!Z %++d%**"Eu{{SV^c^h^hGi!j  5;;vU[[#I5;;W\didodoKp"q!   5;;q #DekkRU]b]h]hFi"j%++c"DekkRT\a\f\fFg!h!!EKK$F TYafamamHn#o  5;;vU[[#I5;;W\didodoKp"q  5;;q#> AUZ@["\%++b"=u{{1TX?Y!Z )+ A 13h<~8v!H` `` ` `  ` ) `N#'"&"&;G ;G);G;G ;G ! ;G  ;G ;G);G~hl:=7:=D$(c c )c c   c  c " c c c L@ <>%%8''"#J0@R$Rj2n k 77)-*. W WW&W( W  W  Wt7+,2-3/S'0rs"[4\;\4\?\