B 1rDa @sddlZddlmZddlZddlmZddlmZddlmZmZej ej ej ej ej ejfZddd d d gZd dZdddZdd ZddZddd Zdd ZdS)N) OrderedDict) distributed)nn)_get_global_gloo_groupget_world_sizeget_async_norm_states pyobj2tensor tensor2pyobj all_reduceall_reduce_normcCsVt}xJ|D]>\}}t|trx*|D]\}}||d||g<q0WqW|S)N.)r named_modules isinstance ASYNC_NORM state_dictitemsjoin)moduleZasync_norm_statesnamechildkvrJ/home/huangxh/桌面/icvrcautonomous-driving/yolox/utils/allreduce_norm.pyr s  cudacCs$tjt|}t|j|dS)z+serialize picklable python object to tensor)device)torch ByteStorage from_bufferpickledumps ByteTensorto)pyobjrstoragerrrr )scCst|S)z-deserialize tensor to picklable python object)r loadscpunumpytobytes)tensorrrrr /scCstjjtjjd|S)N)summean)distReduceOpSUMlower)op_namerrr_get_reduce_op4s r2r+c st}|dkrS|dkr t}t|dkr2St}t|}tj|ddt|}fdd|D}fdd|D}t fdd|D}tj |t |d |d kr||}d dt t |||D} td d t || DS)a  Apply all reduce function for python dict object. NOTE: make sure that every py_dict has the same keys and values are in the same shape. Args: py_dict (dict): dict to apply all reduce op. op (str): operator, could be "sum" or "mean". rNr)srccsg|]}|jqSr)shape).0r)py_dictrr Rszall_reduce..csg|]}|qSr)numel)r5r)r6rrr7Sscsg|]}|qSr)flatten)r5r)r6rrr7Us)opr,cSsg|]\}}||qSr)reshape)r5xr4rrrr7[scSsi|]\}}||qSrr)r5rrrrr ^szall_reduce..)rrr-listkeysr broadcastr rcatr r2zipsplitr) r6r:group world_sizeZpy_keyZ py_key_tensorZ tensor_shapes tensor_numelsZflatten_tensorZ split_tensorsr)r6rr ;s(  cCs&t|}t|dd}|j|dddS)z: All reduce norm statistics in different devices. r,)r:F)strictN)rr load_state_dict)rstatesrrrr as )r)r+N)r collectionsrrrr-rrr BatchNorm1d BatchNorm2d BatchNorm3dInstanceNorm1dInstanceNorm2dInstanceNorm3dr__all__rr r r2r r rrrrs,     &