【游戏视频汉化 #1】 AI语音识别软件 Whisper编译

shane007

多年之前，就有为不带字幕的游戏视频配上字幕的想法。
但是当时条件不成熟，但是目前来看，条件似乎成熟了

Whisper是openAI的开源语音识别软件。
6 F7 S7 J) o! g: E6 L6 W 它有一个.net的版本，在这个版本的基础上进行少量修改，就能将游戏视频对应的字幕识别成srt格式。
: n0 y) j( o) R9 V4 k( ?- ~ 之后，对这个srt文件再进行在线批量翻译之后，进行少量调整之后，汉化工作就完成了。

地址如下
https://github.com/sandrohanea/whisper.net

& z# z8 O( b" z
编译最好使用vs2022编译，否则在.net sdk版本上会出很多问题。

编译好之后，有几个注意点

<0>使用的模型文件修改为大模型，ggml-large.bin，用这个模型效果比较好。
    当然，所有时间也会比较多，估计转换一批文件需要几个甚至几十个小时。

<1>Language要设定为"english"。

1. /*    var builder = factory.CreateBuilder()
   9 l' w$ m& ?7 |; g3 J
2.         .WithLanguage(opt.Language);*/
   7 }0 ^" E1 i+ d8 \
3.     var builder = factory.CreateBuilder()
   ; p8 C' n# \( \8 N4 y& W. f
4.     .WithLanguage("english");

复制代码

<2>缺省好像只支持Wav格式，而且是要16K采样率的，需要实现转换成这种格式，否则会出错。
3 F9 `$ r. ]$ @, R$ E* ?
<3>缺省只提供了一个例子wav文件的转换，需要改为批量形式。
   （遍历某个目录中的所有文件）

: p5 m: s- r: N( b: W# X2 _ <4>输出的文件，需要稍加整理，以符合srt格式

1 L8 U3 ?, g) K: j$ Q   以下是一个Wav文件的控制台输出（幽魂开场动画）
. c9 \) E% H) Q3 f" B% b

1. 
   ( j; R3 V( u' R$ }4 {: V' R+ H
2. whisper_init_from_file_no_state: loading model from 'ggml-large.bin'
   ! P: M2 X, o, e( \
3. whisper_model_load: loading model
   6 D/ w$ W6 F1 c) W
4. whisper_model_load: n_vocab       = 51865
   * Q: F) C' N% w; L# C% m
5. whisper_model_load: n_audio_ctx   = 1500
   
6. whisper_model_load: n_audio_state = 1280
   
7. whisper_model_load: n_audio_head  = 20
   % D, C& ?# q4 o: {8 i
8. whisper_model_load: n_audio_layer = 32
   
9. whisper_model_load: n_text_ctx    = 448
   
10. whisper_model_load: n_text_state  = 1280
    & U  Z, @- z- E& ~2 w
11. whisper_model_load: n_text_head   = 20
    
12. whisper_model_load: n_text_layer  = 32
    ' L  M3 ^- a2 W- k9 z9 F
13. whisper_model_load: n_mels        = 80
    - k. ^4 n5 t: G/ R' w
14. whisper_model_load: ftype         = 1
    
15. whisper_model_load: qntvr         = 0
    * p: w5 Z' ?- g2 y# H; }  h! c8 h
16. whisper_model_load: type          = 5
    
17. whisper_model_load: mem required  = 3557.00 MB (+   71.00 MB per decoder)
    
18. whisper_model_load: adding 1608 extra tokens
    7 C% N% [  c9 ?3 n! E
19. whisper_model_load: model ctx     = 2951.27 MB
    6 x; k9 o6 b, F* Y" d1 Q/ H$ I4 A
20. whisper_model_load: model size    = 2950.66 MB
    3 K& z% [, |. K/ n( _/ \  e, O
21. whisper_init_state: kv self size  =   70.00 MB
      o  Z& y$ t& o5 d  J% u) R
22. whisper_init_state: kv cross size =  234.38 MB
    
23. New Segment: 00:00:00 ==> 00:00:02.7600000 :  (birds chirping)
    
24. New Segment: 00:00:03.6600000 ==> 00:00:05.9000000 :  (exhaling)
    4 o1 v. r* @. ^5 b
25. New Segment: 00:00:05.9000000 ==> 00:00:08.6600000 :  (birds chirping)
    * W5 e, N% f4 d
26. New Segment: 00:00:08.6600000 ==> 00:00:35.1200000 :  (gun firing)
    + p/ u0 O* l* E0 ]- ~! U
27. New Segment: 00:00:36.1200000 ==> 00:00:38.5400000 :  (gun firing)
    0 {9 e4 _3 O1 C
28. New Segment: 00:00:39.0600000 ==> 00:00:41.4800000 :  (gun firing)
    
29. New Segment: 00:00:41.4800000 ==> 00:00:49.4000000 :  (tires screeching)
    
30. New Segment: 00:00:49.4000000 ==> 00:00:58.5800000 :  (glass shattering)
    
31. New Segment: 00:00:58.5800000 ==> 00:01:07.7400000 :  (singing in foreign language)
    
32. New Segment: 00:01:07.7400000 ==> 00:01:11.5800000 :  (singing in foreign language)
    ' r! R% c. w2 h5 O! K
33. New Segment: 00:01:11.5800000 ==> 00:01:17 :  (tires screeching)
    $ }3 |. G# b- ]3 V& t$ |% X
34. New Segment: 00:01:17 ==> 00:01:24.8400000 :  (singing in foreign language)
    " K/ L" T  b, U; ~4 U& w; j
35. New Segment: 00:01:24.8400000 ==> 00:01:28.6400000 :  (panting)
    % j1 ?  u8 O* V: j- L
36. New Segment: 00:01:36.7800000 ==> 00:01:39.2000000 :  (gun firing)
    6 S- l8 j- c3 j. a6 Y
37. New Segment: 00:01:39.2000000 ==> 00:01:43.4600000 :  - Adrian.
    
38. New Segment: 00:01:43.4600000 ==> 00:01:45.6200000 :  - Oh God.
    
39. New Segment: 00:01:45.6200000 ==> 00:01:48.2000000 :  - What's the matter sweetheart?
    1 ]: G( @) `  l- |3 i/ n$ \
40. New Segment: 00:01:48.2000000 ==> 00:01:50.4200000 :  Oh.
    5 A/ t% h& L2 W6 [
41. New Segment: 00:01:50.4200000 ==> 00:01:53.4600000 :  - Oh it's horrible.
    
42. New Segment: 00:01:53.4600000 ==> 00:01:55.3000000 :  - Shh.
    
43. New Segment: 00:01:55.3000000 ==> 00:02:02.3400000 :  It was just a bad dream.
    : F; T! N! \/ G5 Y2 V( O+ E
44. New Segment: 00:02:05.4200000 ==> 00:02:09.8800000 :  - You don't ever have to be afraid of anything.
    
45. New Segment: 00:02:09.8800000 ==> 00:02:12.8000000 :  I'll always be here to protect you.
    
46. New Segment: 00:02:12.9200000 ==> 00:02:15.5000000 :  (gentle music)
    # @, w% f( b! m" u' W
47. New Segment: 00:02:16.4800000 ==> 00:02:19.0600000 :  (gentle music)
    
48. New Segment: 00:02:19.0600000 ==> 00:02:21.6400000 :  (gentle music)
    
49. New Segment: 00:02:21.6400000 ==> 00:02:24.2200000 :  (gentle music)
    ) O% P3 g% Z% Y% l& w* H* y  C3 q
50. New Segment: 00:02:24.5400000 ==> 00:02:27.1200000 :  (gentle music)
    
51. New Segment: 00:02:27.1200000 ==> 00:02:29.7000000 :  (gentle music)
    2 ?' C* C. M3 }2 l9 ]
52. New Segment: 00:02:29.7000000 ==> 00:02:33.1800000 :  [Music]
    

复制代码

7 K  N8 o. u6 u+ ]* R- {3 `/ X6 w