synthetic speech pacing is very fast with frontend script

[zyb8543d]

@Jackiexiao synthetic speech pacing is very fast, do you think what is the reason?
how to solve it?

37050000 37800000 x^iou4-m+ei4=sh@/A:4-4^1@/B:25+4@2^1^26+5#26-5-/C:a_a^n#2+2+2&/D:xx=30!xx@1-1&/E:xx|30-xx@xx#1&xx!1-1#/F:xx^30=17_1-1!
37800000 38600000 iou4^m-ei4+sh=ih1@/A:4-4^1@/B:25+4@2^1^26+5#26-5-/C:a_a^n#2+2+2&/D:xx=30!xx@1-1&/E:xx|30-xx@xx#1&xx!1-1#/F:xx^30=17_1-1!
38600000 39750000 m^ei4-sh+ih1=y@/A:4-1^4@/B:26+3@1^2^27+4#27-4-/C:a_n^z#2+2+2&/D:xx=30!xx@1-1&/E:xx|30-xx@xx#1&xx!1-1#/F:xx^30=17_1-1!
39750000 40400000 ei4^sh-ih1+y=i4@/A:4-1^4@/B:26+3@1^2^27+4#27-4-/C:a_n^z#2+2+2&/D:xx=30!xx@1-1&/E:xx|30-xx@xx#1&xx!1-1#/F:xx^30=17_1-1!
40400000 41200000 sh^ih1-y+i4=ang4@/A:1-4^4@/B:27+2@2^1^28+3#28-3-/C:a_n^z#2+2+2&/D:xx=30!xx@1-1&/E:xx|30-xx@xx#1&xx!1-1#/F:xx^30=17_1-1!

[makerjackie]

I'm not sure about that, do you use DNN architecture in your duration model? Before training the duration model, have you checked your training data (forced-alignment result)? Do you use montreal-forced-alignment to do forced align?

[zyb8543d]

1. I use montreal-forced-alignment to do forced align and feed the train data to merlin.
2. i use default duration config parameter set:
   hidden_layer_size : [1024, 1024, 1024, 1024, 1024, 1024]
   hidden_layer_type : ['TANH', 'TANH', 'TANH', 'TANH', 'TANH', 'TANH']
3. forced-alignment result seems right.
4. othewise, the speed of speech which are synthesis by HTS label which are genarate by montreal-forced-alignment is normal, but the speed of speech which are synthesis by HTS label which are genarate by mandarin_frontend.py is very fast.

when use HTS label which are genarate by mandarin_frontend.py to synthesis speech , get some warning:
2018-06-06 13:57:16,701 INFO labels : loaded /data02/zhangyb/merlin/egs/mandarin_voice/s1/experiments/mandarin_voice/test_synthesis/gen-lab/B11_8.lab, 78 labels
WARNING: no silence found!
WARNING: no silence found!
WARNING: no silence found!
WARNING: no silence found!
WARNING: no silence found!
WARNING: no silence found!
WARNING: no silence found!
WARNING: no silence found!
WARNING: no silence found!
WARNING: no silence found!
WARNING: no silence found!
WARNING: no silence found!
WARNING: no silence found!
WARNING: no silence found!
2018-06-06 13:57:16,845 INFO acoustic_norm: Loaded min max values from the trained data for feature dimension of 471
2018-06-06 13:57:17,193 INFO main : label dimension is 471
2018-06-06 13:57:17,194 INFO main : generating from DNN

[makerjackie]

In synthesis section, mandarin_frontend.py generate HTS Label without predicting phone duration. If your synthetic speech pacing is very fast, then you should check your duration model, both your traing data and training result.

A duration model result (slt_arctic_demo)

2018-01-30 20:30:02,166     INFO           main: Develop: DNN -- RMSE: 6.790 frames/phoneme; CORR: 0.629; 
2018-01-30 20:30:02,166     INFO           main: Test: DNN -- RMSE: 7.842 frames/phoneme; CORR: 0.562; 

[zyb8543d]

@Jackiexiao i check the train data and notice some warning with using montreal-forced-alignment to genarate lab file:

2018-06-06 15:57:01,853 WARNING : --Miss: database/textgrid/mandarin_voice/A11_242.TextGrid
2018-06-06 15:57:01,853 WARNING : --Miss: database/textgrid/mandarin_voice/A11_243.TextGrid
2018-06-06 15:57:01,853 WARNING : --Miss: database/textgrid/mandarin_voice/A11_244.TextGrid
2018-06-06 15:57:01,853 WARNING : --Miss: database/textgrid/mandarin_voice/A11_245.TextGrid
2018-06-06 15:57:01,853 WARNING : --Miss: database/textgrid/mandarin_voice/A11_246.TextGrid
2018-06-06 15:57:01,854 WARNING : --Miss: database/textgrid/mandarin_voice/A11_247.TextGrid
2018-06-06 15:57:01,854 WARNING : --Miss: database/textgrid/mandarin_voice/A11_248.TextGrid
2018-06-06 15:57:01,854 WARNING : --Miss: database/textgrid/mandarin_voice/A11_249.TextGrid

do you think it may have some effect on that case?

[makerjackie]

it means that montreal-forced-align didn't work, so no TextGrid file generated

[zyb8543d]

@Jackiexiao 我大概回想了一下，我是直接下载的thchs30_250_demo.tar.gz,，thchs30_250_demo.tar.gz包里的数据已经有了labels文件，应该是已经做过alignment的吧，所以应该和我自己有没有做alignment没啥关系，因为我没用新的数据。你觉得有毛病么

[makerjackie]

发一个合成音频样例来听听？

[zyb8543d]

@Jackiexiao A11是在训练过程中，自动生成的WAV。
B11是训练完成之后，通过前端获得没有时间戳的lab文件之后生成的。
wav.zip

[makerjackie]

确实很奇怪，按理来说不会出现这种情况，应该是duration model 的问题。你看一下时长模型训练的Log，对比一下之前我发的训练结果记录。

2018-01-30 20:30:02,166     INFO           main: Develop: DNN -- RMSE: 6.790 frames/phoneme; CORR: 0.629; 
2018-01-30 20:30:02,166     INFO           main: Test: DNN -- RMSE: 7.842 frames/phoneme; CORR: 0.562; 

[zyb8543d]

feed_forward_6_tanh_01_57PM_June_06_2018.log
@Jackiexiao 这是其中一个log,没看到你说的那个信息、

[makerjackie]

feed_forward_4_tanh_08_29PM_January_30_2018.log 训练duration model的log

[zyb8543d]

从头开始以后，duration model:
2018-06-08 12:07:49,213 INFO main: calculating MCD
2018-06-08 12:07:49,467 INFO main: Develop: DNN -- RMSE: 9.061 frames/phoneme; CORR: 0.659;
2018-06-08 12:07:49,468 INFO main: Test: DNN -- RMSE: 8.584 frames/phoneme; CORR: 0.668;

[makerjackie]

看起来duration model训练的没有问题......我也不清楚原因

[zyb8543d]

我又重新跑了一遍，发现之前生成不了mgc文件是因为他的路径：world/extract_features_for_merlin.py, 我发现他要找的是WORLD/extract_features_for_merlin.py,于是乎将world夹子改成了WORLD，生成mgc成功。
除了将world文件夹，名字改为大写的，什么都没改，第七步生成wav的时又神奇的报错了：
2018-06-08 19:04:07,893 INFO param_generation: processing 2 of 3: /data02/zhangyb/merlin/egs/mandarin_voice/s1/experiments/mandarin_voice/test_synthesis/wav/A11_1.cmp
2018-06-08 19:04:07,949 INFO param_generation: processing 3 of 3: /data02/zhangyb/merlin/egs/mandarin_voice/s1/experiments/mandarin_voice/test_synthesis/wav/A11_2.cmp
2018-06-08 19:04:07,996 INFO main : reconstructing waveform(s)
2018-06-08 19:04:07,997 CRITICAL wav_generation: The vocoder world is not supported yet!
Traceback (most recent call last):
File "/data02/zhangyb/merlin/src/run_merlin.py", line 1320, in
main_function(cfg)
File "/data02/zhangyb/merlin/src/run_merlin.py", line 989, in main_function
generate_wav(gen_dir, gen_file_id_list, cfg) # generated speech
File "/data02/zhangyb/merlin/src/utils/generate.py", line 335, in generate_wav
raise
RuntimeError: No active exception to reraise