【游戏视频汉化 #1】 AI语音识别软件 Whisper编译

shane007

多年之前，就有为不带字幕的游戏视频配上字幕的想法。
但是当时条件不成熟，但是目前来看，条件似乎成熟了

Whisper是openAI的开源语音识别软件。
它有一个.net的版本，在这个版本的基础上进行少量修改，就能将游戏视频对应的字幕识别成srt格式。
+ \1 X1 A6 N6 k0 \1 h( e/ p  p 之后，对这个srt文件再进行在线批量翻译之后，进行少量调整之后，汉化工作就完成了。

4 `& V0 s4 W  o  w! ~  n+ ` 地址如下
5 ^! \/ B6 h! n% ]# R https://github.com/sandrohanea/whisper.net
9 d  l2 [4 J; I8 D+ ?
' E! o6 ~- C8 H7 l4 f: g
- N# O- u, H+ |- c, S+ Z+ @ 编译最好使用vs2022编译，否则在.net sdk版本上会出很多问题。
. S& ]+ A3 e: m" |; a/ c
编译好之后，有几个注意点

$ g% C1 z) `# n <0>使用的模型文件修改为大模型，ggml-large.bin，用这个模型效果比较好。
$ ?, C+ \5 w$ f4 @- j9 {$ {! s2 ^1 a    当然，所有时间也会比较多，估计转换一批文件需要几个甚至几十个小时。
' N% T+ |# p8 _! ~* X
<1>Language要设定为"english"。
9 E6 G( X2 }4 s
" R/ |! h+ t3 B3 U5 G, X) V3 U

1. /*    var builder = factory.CreateBuilder()
   
2.         .WithLanguage(opt.Language);*/
   
3.     var builder = factory.CreateBuilder()
   - G" R! T/ }$ u0 F) h' R
4.     .WithLanguage("english");

复制代码

<2>缺省好像只支持Wav格式，而且是要16K采样率的，需要实现转换成这种格式，否则会出错。

- }3 F7 y/ Q  S4 g <3>缺省只提供了一个例子wav文件的转换，需要改为批量形式。
   （遍历某个目录中的所有文件）

<4>输出的文件，需要稍加整理，以符合srt格式

: Z$ F# c" B4 _! Z   以下是一个Wav文件的控制台输出（幽魂开场动画）
0 q/ l, D; }! Z) d2 K1 G- [
5 G+ z6 d- d+ p4 ~. j

1. 
   * y8 ?% z% T2 |, O) Z, w1 e3 B
2. whisper_init_from_file_no_state: loading model from 'ggml-large.bin'
   6 A( O: x4 Z( l2 d
3. whisper_model_load: loading model
   / Z9 v' z- |: Z% ]/ M( }+ X+ K
4. whisper_model_load: n_vocab       = 51865
   : m" F5 _& K9 @4 F
5. whisper_model_load: n_audio_ctx   = 1500
   ' g7 z, O) D8 \
6. whisper_model_load: n_audio_state = 1280
   ( A4 q2 [8 H7 O' \' L: M# L
7. whisper_model_load: n_audio_head  = 20
   
8. whisper_model_load: n_audio_layer = 32
   ! `% X) j2 V/ `# k
9. whisper_model_load: n_text_ctx    = 448
   / q7 m3 n# d8 X
10. whisper_model_load: n_text_state  = 1280
    $ f( D5 g' ?; J! F& F" ~
11. whisper_model_load: n_text_head   = 20
    # X2 j9 I6 y3 Y
12. whisper_model_load: n_text_layer  = 32
    4 H6 r) x( R  v" G! @
13. whisper_model_load: n_mels        = 80
    4 @0 q9 O* T/ O% R  I/ w
14. whisper_model_load: ftype         = 1
    
15. whisper_model_load: qntvr         = 0
    
16. whisper_model_load: type          = 5
    
17. whisper_model_load: mem required  = 3557.00 MB (+   71.00 MB per decoder)
    
18. whisper_model_load: adding 1608 extra tokens
    0 j3 j: \& m9 @$ w% S( ?4 U6 y
19. whisper_model_load: model ctx     = 2951.27 MB
    
20. whisper_model_load: model size    = 2950.66 MB
    0 X0 n* W9 f: V3 b
21. whisper_init_state: kv self size  =   70.00 MB
    7 d* j+ u2 w: c/ N
22. whisper_init_state: kv cross size =  234.38 MB
    
23. New Segment: 00:00:00 ==> 00:00:02.7600000 :  (birds chirping)
    & p/ W; f9 T  W2 P
24. New Segment: 00:00:03.6600000 ==> 00:00:05.9000000 :  (exhaling)
    
25. New Segment: 00:00:05.9000000 ==> 00:00:08.6600000 :  (birds chirping)
    ( k) h6 C6 b4 g3 w1 }! O$ J
26. New Segment: 00:00:08.6600000 ==> 00:00:35.1200000 :  (gun firing)
    1 E* y' Q; N8 _( U1 s
27. New Segment: 00:00:36.1200000 ==> 00:00:38.5400000 :  (gun firing)
    4 ~% ~* C. A, x3 J# `7 ?4 u  L
28. New Segment: 00:00:39.0600000 ==> 00:00:41.4800000 :  (gun firing)
    
29. New Segment: 00:00:41.4800000 ==> 00:00:49.4000000 :  (tires screeching)
    * [$ O  h! a% P
30. New Segment: 00:00:49.4000000 ==> 00:00:58.5800000 :  (glass shattering)
    $ a8 _# y. O8 J9 D  i5 ^
31. New Segment: 00:00:58.5800000 ==> 00:01:07.7400000 :  (singing in foreign language)
    & d1 V1 W( a: b8 S. U; t( K$ v5 S) S
32. New Segment: 00:01:07.7400000 ==> 00:01:11.5800000 :  (singing in foreign language)
    ) |8 i2 r7 @$ h; H& F8 _
33. New Segment: 00:01:11.5800000 ==> 00:01:17 :  (tires screeching)
    
34. New Segment: 00:01:17 ==> 00:01:24.8400000 :  (singing in foreign language)
    . [- V8 m8 f8 A0 f
35. New Segment: 00:01:24.8400000 ==> 00:01:28.6400000 :  (panting)
    " {. ^0 Q, m$ U* A" h9 _" ^
36. New Segment: 00:01:36.7800000 ==> 00:01:39.2000000 :  (gun firing)
    
37. New Segment: 00:01:39.2000000 ==> 00:01:43.4600000 :  - Adrian.
    : k* f9 ?+ {* Q4 ?4 A
38. New Segment: 00:01:43.4600000 ==> 00:01:45.6200000 :  - Oh God.
    - W' [" }+ u8 l
39. New Segment: 00:01:45.6200000 ==> 00:01:48.2000000 :  - What's the matter sweetheart?
    
40. New Segment: 00:01:48.2000000 ==> 00:01:50.4200000 :  Oh.
    
41. New Segment: 00:01:50.4200000 ==> 00:01:53.4600000 :  - Oh it's horrible.
    & E8 k; f+ y) D/ O: `
42. New Segment: 00:01:53.4600000 ==> 00:01:55.3000000 :  - Shh.
    ' v* r# g* h. s& B( ?; c# h9 c
43. New Segment: 00:01:55.3000000 ==> 00:02:02.3400000 :  It was just a bad dream.
    6 V9 V  J/ U7 U* m3 d2 @( t
44. New Segment: 00:02:05.4200000 ==> 00:02:09.8800000 :  - You don't ever have to be afraid of anything.
    
45. New Segment: 00:02:09.8800000 ==> 00:02:12.8000000 :  I'll always be here to protect you.
    ( p- b7 @" X- B3 g" _
46. New Segment: 00:02:12.9200000 ==> 00:02:15.5000000 :  (gentle music)
    
47. New Segment: 00:02:16.4800000 ==> 00:02:19.0600000 :  (gentle music)
    # f$ F- D0 z  U3 ]7 ^7 ], r
48. New Segment: 00:02:19.0600000 ==> 00:02:21.6400000 :  (gentle music)
    
49. New Segment: 00:02:21.6400000 ==> 00:02:24.2200000 :  (gentle music)
    / p  S, Q2 {3 U  N- b
50. New Segment: 00:02:24.5400000 ==> 00:02:27.1200000 :  (gentle music)
    
51. New Segment: 00:02:27.1200000 ==> 00:02:29.7000000 :  (gentle music)
    
52. New Segment: 00:02:29.7000000 ==> 00:02:33.1800000 :  [Music]
    : J/ B* ]+ b7 ^' W' J7 I

复制代码

* F+ D; h- e! D8 d, v