ASR工业级代码-WeNet代码逐行分析（1）模型初始化

本文目的是彻底摸清WeNet这个神作的细节，对于一些模棱两可的印象纠偏。我这次学习的版本是自己folk的：https://github.com/Xianchao-Wu/wenet

准备工作

老样子，先交代一下我用的docker，我还是随便找了个nemo的docker。

https://github.com/NVIDIA/NeMo/blob/main/Dockerfile

然后在里面安装了wenet的requirements.txt:

root@1032f8d48655:/workspace/asr/wenet/examples/csj/s0# more ../../../requirements.txt
flake8==3.8.2
Pillow
pyyaml>=5.1
sentencepiece
tensorboard
tensorboardX
typeguard
textgrid
pytest
ipdb

为了逐行调试，需要追加一个ipdb(==0.13.4)包的安装。

ipdb有个坑

ipdb> p args.data_type
'raw'
ipdb> p args.train2022-03-09 01:03:48,859 DEBUG diff parser start
2022-03-09 01:03:48,859 DEBUG line_lengths old: 1; new: 1
2022-03-09 01:03:48,860 DEBUG -> code[replace] old[1:1] new[1:1]
2022-03-09 01:03:48,860 DEBUG parse_part from 1 to 1 (to 0 in part parser)
2022-03-09 01:03:48,861 DEBUG diff parser end
ipdb> p args.train_data

这个可以参考：

https://stackoverflow.com/questions/49442523/prevent-showing-debugging-log-info-inside-ipython-shell

类似于说，你在ipdb提示行里面，一旦按下tab，就出来"debug diff parser start... parser end"之类的玩意儿。目测是和wenet中使用的logging冲突了。

解决方法为：

创建一个新的文件：

vi ~/.ipython/profile_default/startup/00-first.py

然后把下面几行复制进去：

import logging

logging.getLogger('parso.cache').disabled=True
logging.getLogger('parso.cache.pickle').disabled=True
logging.getLogger().setLevel(logging.WARNING);
logging.getLogger('parso').setLevel(logging.WARNING)

这样，整个世界就清净了。按下tab自动补全功能重现，烦人的信息就没有了。

docker

docker里面的python环境，和torch相关的package的版本号分别为：

root@1032f8d48655:/workspace/asr/wenet/examples/csj/s0# pip list | grep torch
nvidia-dlprof-pytorch-nvtx    1.8.0
pytorch-lightning             1.5.8
pytorch-quantization          2.1.2
torch                         1.11.0a0+b6df043
torch-stft                    0.1.4
torch-tensorrt                1.1.0a0
torchaudio                    0.10.1+6f539cf
torchmetrics                  0.7.0
torchtext                     0.11.0a0+d697db5
torchvision                   0.11.0a0

在启动docker的时候，使用了：

sudo nvidia-docker run-dit--shm-size=40gb-v/raid/xianchaow/asr:/workspace/asr nvcr.io/nvidian/tegra-audio/nemo:r1.6.0

有个挂载点，就是在/workspace/asr 这个文件夹下了。wenet的路径在：

root@1032f8d48655:/workspace/asr/wenet# ls
CODE_OF_CONDUCT.md  CPPLINT.cfg  README.md     docs      requirements.txt  test   wenet
CONTRIBUTING.md     LICENSE      README_CN.md  examples  runtime           tools

具体使用哪个asr数据，关系不大，主要是为了分析代码。这里还是延续日文csj下的asr的整体流程了。

启动调试

### Training! ###

if [ ${stage} -le 7 ] && [ ${stop_stage} -ge 7 ]; then
# Training
mkdir -p $dir
INIT_FILE=$dir/ddp_init
rm -f $INIT_FILE # delete old one before starting
init_method=file://$(readlink -f $INIT_FILE)
echo "$0: init method is $init_method"
num_gpus=$(echo $CUDA_VISIBLE_DEVICES | awk -F "," '{print NF}')
# Use "nccl" if it works, otherwise use "gloo"
#dist_backend="gloo"
dist_backend="nccl"
cmvn_opts=
$cmvn && cmvn_opts="--cmvn $wave_data/${train_set}/global_cmvn"
# train.py will write $train_config to $dir/train.yaml with model input
# and output dimension, train.yaml will be used for inference or model
# export later
num_gpus=1
# debug wenet code only usage:
for ((i = 0; i < $num_gpus; ++i)); do
{
gpu_id=$(echo $CUDA_VISIBLE_DEVICES | cut -d',' -f$[$i+1])
python -m ipdb wenet/bin/train.py --gpu $gpu_id \
--config $train_config \
--data_type raw \
--symbol_table $dict \
--train_data $wave_data/$train_set/data.list \
--cv_data $wave_data/$dev_set/data.list \
${checkpoint:+--checkpoint $checkpoint} \
--model_dir $dir \
--ddp.init_method $init_method \
--ddp.world_size $num_gpus \
--ddp.rank $i \
--ddp.dist_backend $dist_backend \
--num_workers 1 \
$cmvn_opts \
--pin_memory
} #&
done
#wait

fi

配置文件

这里是搞了个大的模型，使用的配置文件是：

conf/train_conformer_bidecoder_large.yaml

输入参数

> /workspace/asr/wenet/examples/csj/s0/wenet/bin/train.py(116)main()
115     args = get_args()
--> 116     logging.basicConfig(level=logging.DEBUG,
117                         format='%(asctime)s %(levelname)s %(message)s')

ipdb> args
ipdb> p args
Namespace(bpe_model=None,
checkpoint='/workspace/asr/wenet/examples/csj/s0/exp/sp_spec_aug_conformer_bidecoder_large/119.pt',
cmvn='data/train/global_cmvn',
config='conf/train_conformer_bidecoder_large.yaml',
cv_data='data/dev/data.list',
data_type='raw',
dist_backend='nccl',
fp16_grad_sync=False,
gpu=0,
init_method='file:///workspace/asr/wenet/examples/csj/s0/exp/sp_spec_aug_conformer_bidecoder_large/ddp_init',
model_dir='exp/sp_spec_aug_conformer_bidecoder_large',
non_lang_syms=None,
num_workers=1,
override_config=[],
pin_memory=True,
prefetch=100,
rank=0,
symbol_table='data/lang_char/train_bpe4096_units.txt',
tensorboard_dir='tensorboard',
train_data='data/train/data.list',
use_amp=False,
world_size=1)

读取config

读取config文件之后，得到的是：

ipdb> configs

{'encoder': 'conformer',
'encoder_conf':
{'output_size': 512,
'attention_heads': 8,
'linear_units': 2048,
'num_blocks': 12,
'dropout_rate': 0.1,
'positional_dropout_rate': 0.1,
'attention_dropout_rate': 0.1,
'input_layer': 'conv2d',
'normalize_before': True,
'cnn_module_kernel': 31,
'use_cnn_module': True,
'activation_type': 'swish',
'pos_enc_layer_type': 'rel_pos',
'selfattention_layer_type': 'rel_selfattn',
'cnn_module_norm': 'layer_norm'},

'decoder': 'bitransformer',
'decoder_conf':
{'attention_heads': 8,
'linear_units': 2048,
'num_blocks': 3,
'r_num_blocks': 3,
'dropout_rate': 0.1,
'positional_dropout_rate': 0.1,
'self_attention_dropout_rate': 0.1,
'src_attention_dropout_rate': 0.1},

'model_conf':
{'ctc_weight': 0.3,
'lsm_weight': 0.1,
'length_normalized_loss': False,
'reverse_weight': 0.3},

'dataset_conf':
{'filter_conf':
{'max_length': 2000,
'min_length': 50,
'token_max_length': 400,
'token_min_length': 1,
'min_output_input_ratio': 0.05,
'max_output_input_ratio': 10.0},

'resample_conf': {'resample_rate': 16000},
'speed_perturb': True,
'fbank_conf':
{'num_mel_bins': 80,
'frame_shift': 10,
'frame_length': 25,
'dither': 1.0},
'spec_aug': True,
'spec_aug_conf':
{'num_t_mask': 3,
'num_f_mask': 2,
'max_t': 50,
'max_f': 10},
'shuffle': True,
'shuffle_conf': {'shuffle_size': 1500},
'sort': True,
'sort_conf': {'sort_size': 500},
'batch_conf': {'batch_type': 'static', 'batch_size': 12}
},

'grad_clip': 5,
'accum_grad': 4,
'max_epoch': 200,
'log_interval': 100,
'optim': 'adam',
'optim_conf': {'lr': 0.002},
'scheduler': 'warmuplr',
'scheduler_conf': {'warmup_steps': 50000}
}

读取词典文件：

▁あるいは': 1410, '▁あー': 1411, '▁あーの': 1412, '▁い': 1413

Dataset(): 11关

[wenet/dataset/dataset.py]

本以为这是个构造函数，其实就是个普通的函数。。。没有class，哈哈。

先后构造的是train-dataset和cv-dataset的对象。

> /workspace/asr/wenet/examples/csj/s0/wenet/bin/train.py(150)main()
149     import ipdb; ipdb.set_trace()
--> 150     train_dataset = Dataset(args.data_type, args.train_data, symbol_table,
151                             train_conf, args.bpe_model, non_lang_syms, True)

进入这个Dataset的构造函数的时候，输入参数的取值为：

2022-03-09 01:04:52,809 DEBUG Using selector: EpollSelector
> /workspace/asr/wenet/examples/csj/s0/wenet/bin/train.py(150)main()
149     import ipdb; ipdb.set_trace()
--> 150     train_dataset = Dataset(args.data_type, args.train_data, symbol_table,
151                             train_conf, args.bpe_model, non_lang_syms, True)

2022-03-09 01:04:52,823 DEBUG Using selector: EpollSelector
ipdb> p args.data_type
'raw'
ipdb> p args.train_data
'data/train/data.list'
ipdb> p len(symbol_table)
5502
ipdb> train_conf

{'filter_conf': {'max_length': 2000, 'min_length': 50,
'token_max_length': 400, 'token_min_length': 1,
'min_output_input_ratio': 0.05, 'max_output_input_ratio': 10.0},
'resample_conf': {'resample_rate': 16000},
'speed_perturb': True, 'fbank_conf': {'num_mel_bins': 80,
'frame_shift': 10, 'frame_length': 25, 'dither': 1.0},
'spec_aug': True, 'spec_aug_conf': {'num_t_mask': 3,
'num_f_mask': 2, 'max_t': 50, 'max_f': 10}, 'shuffle': True,
'shuffle_conf': {'shuffle_size': 1500}, 'sort': True,
'sort_conf': {'sort_size': 500},
'batch_conf': {'batch_type': 'static', 'batch_size': 12}}

ipdb> p args.bpe_model
None
ipdb> non_lang_syms
ipdb>

这里区分了'raw', 和'shard'，所谓shard就是tar一下成若干文件块，然后再搞事情了。这个shard，在nemo里面也有很好的支持。

[代码解读]NeMo的ASR模块-第一方面军：

https://zhuanlan.zhihu.com/p/416080015

不过，nemo里面是显示先手工搞定tar文件，然后nemo会自动使用它们。

WeNet的数据集合对象的构造，是类似阶梯式的，即一个processor完成之后，就进入下一个。具体为：

dataset=DataList(lists, shuffle=True, partition=True)
processor.parse_raw
processor.tokenize [切词]
processor.filter
processor.resample
processor.speed_perturb
processor.compute_fbank [计算filter banks]
processor.spec_aug [spectrum augment-谱增强]
processor.shuffle
processor.sort
processor.batch
processor.padding

在创建了dataset对象之后，前后经历了11道关口来搞事情。

DataLoader

这里使用的DataLoader直接来自torch.utils.data

模型初始化

init_asr_model(configs) : wenet/transformer/asr_model.py

我们现在在这里：

模型初始化，目前所在的脑图位置

cmvn

这是直接读取已经做好的.json文件，取出来80维度的mean和variance。

>/workspace/asr/wenet/wenet/utils/cmvn.py(35)_load_json_cmvn()
34means = cmvn_stats['mean_stat']
--->35variance = cmvn_stats['var_stat']
36count = cmvn_stats['frame_num']

ipdb> p means
[1811129088.0,1875043712.0,2068275328.0,2248355328.0,2399715328.0,
2499047936.0,2550326016.0,2541845760.0,2532359168.0,2511235328.0,
2528702976.0,2543875328.0,2582629120.0,2611224832.0,2619879168.0,
2630225408.0,2624540928.0,2601335808.0,2604593408.0,2549471232.0,
2505738752.0,2530102016.0,2481621248.0,2495328000.0,2473367808.0,
2482233856.0,2459522560.0,2467700736.0,2460619776.0,2463341568.0,
2467787264.0,2470003712.0,2473266432.0,2479851520.0,2492325632.0,
2507731712.0,2520133120.0,2531906304.0,2546415616.0,2538470400.0,
2558610688.0,2551794176.0,2569423616.0,2582349824.0,2609854976.0,
2644677376.0,2677323776.0,2698899712.0,2703827456.0,2695868672.0,
2673497856.0,2656272640.0,2648803328.0,2653478144.0,2661783296.0,
2675700992.0,2688753664.0,2699604480.0,2704720128.0,2699744512.0,
2693959936.0,2679148288.0,2663292160.0,2646028544.0,2629203712.0,
2615382016.0,2606205696.0,2604423168.0,2602799616.0,2601224192.0,
2600758784.0,2601683456.0,2604914176.0,2612995584.0,2621194240.0,
2625994752.0,2626787072.0,2590963456.0,2413899776.0,1962116096.0]
ipdb> p len(means)
80
ipdb> n
>/workspace/asr/wenet/wenet/utils/cmvn.py(36)_load_json_cmvn()
35variance = cmvn_stats['var_stat']
--->36count = cmvn_stats['frame_num']
37foriinrange(len(means)):

ipdb> p variance
[19328219136.0,21056569344.0,26132310016.0,30714284032.0,34829242368.0,
37707661312.0,39296671744.0,39233339392.0,38975234048.0,38294093824.0,
38737227776.0,39182794752.0,40338415616.0,41239367680.0,41546645504.0,
41855762432.0,41667940352.0,40956448768.0,41003048960.0,39351541760.0,
38064717824.0,38680506368.0,37238284288.0,37548683264.0,36871151616.0,
37054177280.0,36365819904.0,36563275776.0,36342202368.0,36398952448.0,
36512350208.0,36559269888.0,36627546112.0,36790390784.0,37115179008.0,
37525225472.0,37861744640.0,38189297664.0,38599213056.0,38362722304.0,
38918123520.0,38688272384.0,39159963648.0,39507943424.0,40311492608.0,
41362849792.0,42365263872.0,43028090880.0,43168186368.0,42903535616.0,
42201395200.0,41662509056.0,41427623936.0,41575493632.0,41841123328.0,
42285879296.0,42705707008.0,43033923584.0,43164356608.0,42957029376.0,
42729971712.0,42236821504.0,41731428352.0,41191104512.0,40666808320.0,
40232738816.0,39958773760.0,39911337984.0,39876661248.0,39838826496.0,
39824408576.0,39841591296.0,39916343296.0,40132501504.0,40351125504.0,
40471314432.0,40474038272.0,39402278912.0,34367750144.0,23112441856.0]
ipdb> p len(variance)
80

count=176,228,620

这个_load_json_cmvn方法【wenet/utils/cmv.py】有一些操作：

读取cmvn,并且进行一系列计算

具体的计算，cmvn的脚本为：

tools/compute_cmvn_stats.py --num_workers 16 --train_config $train_config \
--in_scp $wave_data/$train_set/wav.scp \
--out_cmvn $wave_data/$train_set/global_cmvn

这个有待于后续确认细节了。【挖坑】

下面的URL填坑了：

https://zhuanlan.zhihu.com/p/478326259

看i=0的时候的计算：

ipdb>p count
176228620
ipdb>n
>/workspace/asr/wenet/wenet/utils/cmvn.py(38)_load_json_cmvn()
37     for i in range(len(means)):
--->38         means[i] /= count
39         variance[i] = variance[i] / count - means[i] * means[i]

ipdb>means[0]
1811129088.0
ipdb>n
>/workspace/asr/wenet/wenet/utils/cmvn.py(39)_load_json_cmvn()
38         means[i] /= count
--->39         variance[i] = variance[i] / count - means[i] * means[i]
40         if variance[i] < 1.0e-20:

ipdb>means[0]
10.277156389240295
ipdb>variance[0]
19328219136.0
ipdb>n
>/workspace/asr/wenet/wenet/utils/cmvn.py(40)_load_json_cmvn()
39         variance[i] = variance[i] / count - means[i] * means[i]
--->40ifvariance[i] < 1.0e-20:
41             variance[i] = 1.0e-20

ipdb>variance[0]
4.057015583063929
ipdb>n
>/workspace/asr/wenet/wenet/utils/cmvn.py(42)_load_json_cmvn()
41             variance[i] = 1.0e-20
--->42         variance[i] = 1.0 / math.sqrt(variance[i])
43     cmvn = np.array([means, variance])

ipdb>n
>/workspace/asr/wenet/wenet/utils/cmvn.py(37)_load_json_cmvn()
36     count = cmvn_stats['frame_num']
--->37foriinrange(len(means)):
38         means[i] /= count

ipdb>variance[0]
0.4964741741680437

最终，这块代码的输出为：

ipdb> p cmvn
array([[10.27715639, 10.63983655, 11.73631915, 12.7581736 , 13.61705793,
...,
0.36248799,  0.36534477,  0.36689017,  0.36772673,  0.37303921]])

里面囊括了mean和variance (istd= inverse standard derivation)。

GlobalCMVN

[wenet/utils/cmvn.py]

目前我们的位置脑图，通过GlobalCMVN是把mean和istd这俩给注册上去，以后使用

然后可以看到，后续的encoder type是conformer，而decoder type是bi-transformer。

ConformerEncoder

最重要的要数这个基于conformer block的编码器了。

wenet/transformer/encoder.py

三大块：self-attention, FFN, 以及卷积模块

BaseEncoder

先是parent = BaseEncoder里面的关于embedding的：

BaseEncoder的构造函数

上面的self.embed展开之后：

这样，我们有：

(global_cmvn): GlobalCMVN()
(embed): Conv2dSubsampling4(
(conv): Sequential(
(0): Conv2d(1,512, kernel_size=(3,3), stride=(2,2))
(1): ReLU()
(2): Conv2d(512,512, kernel_size=(3,3), stride=(2,2))
(3): ReLU()
)
(out): Sequential(
(0): Linear(in_features=9728, out_features=512, bias=True)
)
(pos_enc): RelPositionalEncoding(
(dropout): Dropout(p=0.1, inplace=False)
)
)
(after_norm): LayerNorm((512,), eps=1e-12, elementwise_affine=True)

我们从BaseEncoder类回到ConformerEncoder。

encoder的脑图，给出了使用SiLU()激活函数，定了三个重要模块，注意力，FFN以及卷积，以及它们的分别的初始化参数。

重要的在ConformerEncoderLayer里面了。

ConformerEncoderLayer

为了初始化出来一个ConformerEncoderLayer对象，需要先对其输入的参数进行初始化。

上面的脑图，就给出了输入的参数的初始化的过程。具体包括：

encoder_selfattn_layer，自注意力模块
positionwise_layer，FFN模块
convolution_layer，卷积模块

encoder_selfattn_layer

[wenet/transformer/attention.py]

把上面的parent class, MultiHeadAttention里面的初始化函数展开：

里面就是定义了经典的四个线性层了：

初始化得到的是：

ipdb> self_attn
RelPositionMultiHeadedAttention(
(linear_q): Linear(in_features=512, out_features=512, bias=True)
(linear_k): Linear(in_features=512, out_features=512, bias=True)
(linear_v): Linear(in_features=512, out_features=512, bias=True)
(linear_out): Linear(in_features=512, out_features=512, bias=True)
(dropout): Dropout(p=0.1, inplace=False)
(linear_pos): Linear(in_features=512, out_features=512, bias=False)
)

**FFN*2**

[wenet/transformer/positionwise_feed_forward.py]

因为是“马卡龙”结构，定义了两个“独立”的FFN层：

执行完毕这两个FFN之后，得到：

ipdb> feed_forward
PositionwiseFeedForward(
(w_1): Linear(in_features=512, out_features=2048, bias=True)
(activation): SiLU()
(dropout): Dropout(p=0.1, inplace=False)
(w_2): Linear(in_features=2048, out_features=512, bias=True)
)
ipdb> feed_forward_macaron
PositionwiseFeedForward(
(w_1): Linear(in_features=512, out_features=2048, bias=True)
(activation): SiLU()
(dropout): Dropout(p=0.1, inplace=False)
(w_2): Linear(in_features=2048, out_features=512, bias=True)
)

卷积模块

[wenet/transformer/convolution.py]

一个卷积模块里面有三个卷积：

self.pointwise_conv1, = Conv1d(512, 1024, kernel_size=(1,), stride=(1,))
self.depthwise_conv = Conv1d(512, 512, kernel_size=(31,), padding=(15,), groups=512)
self.pointwise_conv2, = Conv1d(512, 512, kernel_size=(1,), stride=(1,))

即：

ipdb> conv_module
ConvolutionModule(
(pointwise_conv1): Conv1d(512, 1024, kernel_size=(1,), stride=(1,))
(depthwise_conv): Conv1d(512, 512, kernel_size=(31,), stride=(1,), padding=(15,), groups=512)
(norm): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
(pointwise_conv2): Conv1d(512, 512, kernel_size=(1,), stride=(1,))
(activation): SiLU()
)

接着是正菜的

ConformerEncoderLayer细节

[wenet/transformer/encoder_layer.py]

主要是关于layernorm等一些参数的赋值的：

执行完毕一个ConformerEncoderLayer的初始化之后，得到的是：

(0): ConformerEncoderLayer(
(self_attn): RelPositionMultiHeadedAttention(
(linear_q): Linear(in_features=512, out_features=512, bias=True)
(linear_k): Linear(in_features=512, out_features=512, bias=True)
(linear_v): Linear(in_features=512, out_features=512, bias=True)
(linear_out): Linear(in_features=512, out_features=512, bias=True)
(dropout): Dropout(p=0.1, inplace=False)
(linear_pos): Linear(in_features=512, out_features=512, bias=False)
)
(feed_forward): PositionwiseFeedForward(
(w_1): Linear(in_features=512, out_features=2048, bias=True)
(activation): SiLU()
(dropout): Dropout(p=0.1, inplace=False)
(w_2): Linear(in_features=2048, out_features=512, bias=True)
)
(feed_forward_macaron): PositionwiseFeedForward(
(w_1): Linear(in_features=512, out_features=2048, bias=True)
(activation): SiLU()
(dropout): Dropout(p=0.1, inplace=False)
(w_2): Linear(in_features=2048, out_features=512, bias=True)
)
(conv_module): ConvolutionModule(
(pointwise_conv1): Conv1d(512, 1024, kernel_size=(1,), stride=(1,))
(depthwise_conv): Conv1d(512, 512, kernel_size=(31,), stride=(1,),
padding=(15,), groups=512)
(norm): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
(pointwise_conv2): Conv1d(512, 512, kernel_size=(1,), stride=(1,))
(activation): SiLU()
)
(norm_ff): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
(norm_mha): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
(norm_ff_macaron): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
(norm_conv): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
(norm_final): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
(dropout): Dropout(p=0.1, inplace=False)
(concat_linear): Linear(in_features=1024, out_features=512, bias=True)
)

嘿嘿，一个ConformerEncoderLayer里面（one conformer block）居然有这么多LayerNorm!

卷积模块有一个layernorm
norm_ff
norm_mha
norm_ff_macaron
norm_conv
norm_final

代码那边是一共12层和上面一毛一样的conformer blocks。

BiTransformerDecoder

[wenet/transformer/decoder.py]

decoder的脑图，包括了left_decoder和right_decoder

主要是对这里的left_decoder和right_decoder进行初始化。

脑图left/right-decoder

上面的脑图，展示了left_decoder的初始化函数。重点包括了

embedding
位置编码
decoders-三层

TransformerDecoder

[wenet/transformer/decoder.py]

TransformerDecoder的__init__函数的输入参数

文本序列位置编码

目标文本序列中使用的位置编码是绝对编码（和香草transformer中的相同）

之后，token embed和positionencoding的结合，就构成了

self.embed

self.decoders

继续到TransformerDecoder里面最重要的self.decoders:

一个模块里面有三个部分，self-attention of text seq, cross-attention with source wav seq；以及最后的FFN

这里面就包括了self-attention，即文本序列自己的自注意力；

cross-attention，即当前的文本序列和source wav seq的交叉注意力；

以及最后的FFN层。

和encoder中使用的self-attention类似，这里的self-attention和cross-attention的构造也是都包括四个线性层

看看FFN层：

从而，DecoderLayer的__init__函数就是：

这样的话DecoderLayer就算齐备了

细节示意

ipdb>self.decoders ->self.left_decoder same architecture withself.right_decoder
ModuleList(
(0,1,2): DecoderLayer(
(self_attn): MultiHeadedAttention(
(linear_q): Linear(in_features=512, out_features=512, bias=True)
(linear_k): Linear(in_features=512, out_features=512, bias=True)
(linear_v): Linear(in_features=512, out_features=512, bias=True)
(linear_out): Linear(in_features=512, out_features=512, bias=True)
(dropout): Dropout(p=0.1, inplace=False)
)
(src_attn): MultiHeadedAttention(
(linear_q): Linear(in_features=512, out_features=512, bias=True)
(linear_k): Linear(in_features=512, out_features=512, bias=True)
(linear_v): Linear(in_features=512, out_features=512, bias=True)
(linear_out): Linear(in_features=512, out_features=512, bias=True)
(dropout): Dropout(p=0.1, inplace=False)
)
(feed_forward): PositionwiseFeedForward(
(w_1): Linear(in_features=512, out_features=2048, bias=True)
(activation): ReLU()
(dropout): Dropout(p=0.1, inplace=False)
(w_2): Linear(in_features=2048, out_features=512, bias=True)
)
(norm1): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
(norm2): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
(norm3): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
(dropout): Dropout(p=0.1, inplace=False)
(concat_linear1): Linear(in_features=1024, out_features=512, bias=True)
(concat_linear2): Linear(in_features=1024, out_features=512, bias=True)
)
)

一个DecoderLayer里面，三个LayerNorm.

CTC

ipdb> ctc
CTC(
(ctc_lo): Linear(in_features=512, out_features=5502, bias=True)
(ctc_loss): CTCLoss()
)

最后看一下

ASRModel

把最后一个label smooth loss展开就是：

标签平滑loss

最后的最后

ipdb> model
ASRModel(
(encoder): ConformerEncoder(
(global_cmvn): GlobalCMVN()
(embed): Conv2dSubsampling4(
(conv): Sequential(
(0): Conv2d(1, 512, kernel_size=(3, 3), stride=(2, 2))
(1): ReLU()
(2): Conv2d(512, 512, kernel_size=(3, 3), stride=(2, 2))
(3): ReLU()
)
(out): Sequential(
(0): Linear(in_features=9728, out_features=512, bias=True)
)
(pos_enc): RelPositionalEncoding(
(dropout): Dropout(p=0.1, inplace=False)
)
)
(after_norm): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
(encoders): ModuleList(
(0...11): ConformerEncoderLayer(
(self_attn): RelPositionMultiHeadedAttention(
(linear_q): Linear(in_features=512, out_features=512, bias=True)
(linear_k): Linear(in_features=512, out_features=512, bias=True)
(linear_v): Linear(in_features=512, out_features=512, bias=True)
(linear_out): Linear(in_features=512, out_features=512, bias=True)
(dropout): Dropout(p=0.1, inplace=False)
(linear_pos): Linear(in_features=512, out_features=512, bias=False)
)
(feed_forward): PositionwiseFeedForward(
(w_1): Linear(in_features=512, out_features=2048, bias=True)
(activation): SiLU()
(dropout): Dropout(p=0.1, inplace=False)
(w_2): Linear(in_features=2048, out_features=512, bias=True)
)
(feed_forward_macaron): PositionwiseFeedForward(
(w_1): Linear(in_features=512, out_features=2048, bias=True)
(activation): SiLU()
(dropout): Dropout(p=0.1, inplace=False)
(w_2): Linear(in_features=2048, out_features=512, bias=True)
)
(conv_module): ConvolutionModule(
(pointwise_conv1): Conv1d(512, 1024, kernel_size=(1,), stride=(1,))
(depthwise_conv): Conv1d(512, 512, kernel_size=(31,), stride=(1,), padding=(15,), groups=512)
(norm): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
(pointwise_conv2): Conv1d(512, 512, kernel_size=(1,), stride=(1,))
(activation): SiLU()
)
(norm_ff): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
(norm_mha): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
(norm_ff_macaron): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
(norm_conv): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
(norm_final): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
(dropout): Dropout(p=0.1, inplace=False)
(concat_linear): Linear(in_features=1024, out_features=512, bias=True)
)

)
)
(decoder): BiTransformerDecoder(
(left_decoder): TransformerDecoder(
(embed): Sequential(
(0): Embedding(5502, 512)
(1): PositionalEncoding(
(dropout): Dropout(p=0.1, inplace=False)
)
)
(after_norm): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
(output_layer): Linear(in_features=512, out_features=5502, bias=True)
(decoders): ModuleList(
(0,1,2): DecoderLayer(
(self_attn): MultiHeadedAttention(
(linear_q): Linear(in_features=512, out_features=512, bias=True)
(linear_k): Linear(in_features=512, out_features=512, bias=True)
(linear_v): Linear(in_features=512, out_features=512, bias=True)
(linear_out): Linear(in_features=512, out_features=512, bias=True)
(dropout): Dropout(p=0.1, inplace=False)
)
(src_attn): MultiHeadedAttention(
(linear_q): Linear(in_features=512, out_features=512, bias=True)
(linear_k): Linear(in_features=512, out_features=512, bias=True)
(linear_v): Linear(in_features=512, out_features=512, bias=True)
(linear_out): Linear(in_features=512, out_features=512, bias=True)
(dropout): Dropout(p=0.1, inplace=False)
)
(feed_forward): PositionwiseFeedForward(
(w_1): Linear(in_features=512, out_features=2048, bias=True)
(activation): ReLU()
(dropout): Dropout(p=0.1, inplace=False)
(w_2): Linear(in_features=2048, out_features=512, bias=True)
)
(norm1): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
(norm2): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
(norm3): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
(dropout): Dropout(p=0.1, inplace=False)
(concat_linear1): Linear(in_features=1024, out_features=512, bias=True)
(concat_linear2): Linear(in_features=1024, out_features=512, bias=True)
)

)
)
(right_decoder): TransformerDecoder(
(embed): Sequential(
(0): Embedding(5502, 512)
(1): PositionalEncoding(
(dropout): Dropout(p=0.1, inplace=False)
)
)
(after_norm): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
(output_layer): Linear(in_features=512, out_features=5502, bias=True)
(decoders): ModuleList(
(0, 1, 2): DecoderLayer(
(self_attn): MultiHeadedAttention(
(linear_q): Linear(in_features=512, out_features=512, bias=True)
(linear_k): Linear(in_features=512, out_features=512, bias=True)
(linear_v): Linear(in_features=512, out_features=512, bias=True)
(linear_out): Linear(in_features=512, out_features=512, bias=True)
(dropout): Dropout(p=0.1, inplace=False)
)
(src_attn): MultiHeadedAttention(
(linear_q): Linear(in_features=512, out_features=512, bias=True)
(linear_k): Linear(in_features=512, out_features=512, bias=True)
(linear_v): Linear(in_features=512, out_features=512, bias=True)
(linear_out): Linear(in_features=512, out_features=512, bias=True)
(dropout): Dropout(p=0.1, inplace=False)
)
(feed_forward): PositionwiseFeedForward(
(w_1): Linear(in_features=512, out_features=2048, bias=True)
(activation): ReLU()
(dropout): Dropout(p=0.1, inplace=False)
(w_2): Linear(in_features=2048, out_features=512, bias=True)
)
(norm1): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
(norm2): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
(norm3): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
(dropout): Dropout(p=0.1, inplace=False)
(concat_linear1): Linear(in_features=1024, out_features=512, bias=True)
(concat_linear2): Linear(in_features=1024, out_features=512, bias=True)
)

)
)
)
(ctc): CTC(
(ctc_lo): Linear(in_features=512, out_features=5502, bias=True)
(ctc_loss): CTCLoss()
)
(criterion_att): LabelSmoothingLoss(
(criterion): KLDivLoss()
)
)

至此，model的初始化工作就完成了。

WeNet的代码，整洁易读，（读起来比看NeMo的迷宫式的代码，好很多。。。）。