Rubyで音を視る

松江Ruby会議12「Rubyで音を視る」の発表スライド
https://matsue.rubyist.net/matrk12/ #matrk12

Transcript

1. 2026/06/06 Ruby 12 Ruby seeing sound with ruby Yudai Takada, ydah
7. Otsu City Traditional Performing Arts Hall Saturday, July 18, 2026
14. 2026/06/06 Ruby 12 Ruby seeing sound with ruby Yudai Takada, ydah
17. vizcore https://github.com/ydah/vizcore
18. Vizcore visualize( ) × core( ) Audio-reactive visuals Ruby DSL Ruby scene renderer
19. 2D/3D OpenGL
26. PCM samples Mic / File samples Array<Float> [ 0.0012, 0.0031, -0.0028, . . . ]
28. Sample Window tick window 1024 60fps tick sample 圧
29. Samples RMS amplitude Sample RMS 夏 圧 0 [0.5, -0.5, 0.5, -0.5] (0.5 + -0.5 + 0.5 + -0.5) / 4 = 0 RMS = 1 N N ∑ i=1 x2 i
30. Samples FFT magnitudes / peak_frequency FFT 圧 WN = e−2πi/N Ek = N/2−1 ∑ m=0 x2m Wkm N/2 Ok = N/2−1 ∑ m=0 x2m+1 Wkm N/2 Xk = Ek + Wk N Ok Xk+N/2 = Ek − Wk N Ok (k = 0,1,…, N 2 − 1)
31. magnitudes BandSplitter sub / low / mid / high FFT Vizcore sub, low, mid, high sub: 20Hz . . 60Hz low: 60Hz . . 250Hz mid: 250Hz . . 4000Hz high: 4000Hz . . 20000Hz
32. Samples frame energy BeatDetector beat FFT beat beat frame 圧 E = 1 N N ∑ i=1 x2 i
33. beat beat_pulse beat boolean frame true 1 frame beat 圧 1.0 beat frame 0.86 長
34. beat history BPMEstimator bpm / beat_phase / bar_phase BPM beat frame lag BPM beat beat beat = beat_pulse = bpm = beat_phase = bar_phase = bpm = 60.0 * frame_rate / best_lag
35. bands + onset drums con fi dence 圧 band onset kick = sublow_strength * onset # sub/low ͷڧ͞ × onset snare = mid_strength * onset # mid ͷڧ͞ × onset hihat = high_strength * onset # high ͷڧ͞ × onset
36. magnitudes / spectrum spectral features FFT Spectral features FFT 圧 amplitude bands spectral features centroid: / rollo ff : / fl atness: / fl ux: /
37. raw features smoother audio features frame を EMA EMAt = αxt + (1 − α)EMAt−1 amplitude: @sm.smooth(:amplitude, normalized[:amplitude]) bands: @sm.smooth_hash(normalized[:bands], namespace: :bands) fft: @sm.smooth_array(normalized[:fft], namespace: :fft)
38. amplitude bass / low high / treble beat beat_pulse bpm / phase spectral_* / / / / / を 長
40. audio features MappingResolver layer params map beat_pulse, to: :radius, gain: 160.0, min: 56, max: 164, attack: 1.0, release: 0.2
41. beat beat_pulse map beat_pulse, to: :radius, gain: 100, min: 50, max: 150
42. beat map beat_pulse, to: :radius, gain: 100, min: 50, max: 150 100
43. beat map beat_pulse, to: :radius, gain: 100, min: 50, max: 150 50 150
44. beat map beat_pulse, to: :radius, gain: 100, min: 50, max: 150 radius
45. layer params audio_frame audio scene layers audio frame { “audio”: { "amplitude": 0.42, "beat": true, "beat_pulse": 1.0, "bands": { "low": 0.8 } }, "scene": { "name": "main", "layers": [ { "name": "rings", "params": { "radius": 140 } …
46. audio_frame WebSocket frontend WebGL renderer Frontend audio_frame Ruby Runtime Browser Renderer (WebGL Runtime) Web Socket
47. 6/4
48. BGM
51. 50 6 4
52. 2026/06/06 Ruby 12 Ruby seeing sound with ruby Yudai Takada, ydah
53. 圧 母 /
54. 母
55. 母 母 F1 / F2 spectral centroid low / mid / high ratio
56. F1/F2 母 F 圧 圧 : : : / : :
57. 母 0.0 1.0 1 F1 F2 母
59. / / 圧 圧 burst fl atness ZCR nasal high ratio
60. burst amplitude_delta = [ current[:amplitude] - previous[:amplitude], # લϑϨʔϜ͔ΒͷԻྔ૿Ճ 0.0 ].max burst_strength = [ current[:onset], # Իͷ্ཱ͕ͪΓ current[:spectral_flux], # εϖΫτϧมԽྔ current.dig(:onsets, :mid), # தҬͷ্ཱ͕ͪΓ current.dig(:onsets, :high), # ߴҬͷ্ཱ͕ͪΓ amplitude_delta, # Իྔͷ૿Ճྔ recent_onset_max # ௚ۙ૭ͷ࠷େ্ཱ͕ͪΓ ].max
61. spectral fl atness s_score = weighted_average([ high_ratio, # ߴҬͷڧ͞ flatness, # ϊΠζͬΆ͞ zcr, # ೾ܗͷࡉ͔͍༳Ε centroid # Իͷ໌Δ͞ ]) n_score = weighted_average([ nasal_score, # ඓԻͬΆ͞ 1.0 - high_ratio, # ߴҬ͕গͳ͍ low_mid_ratio, # ௿ʙதҬͷڧ͞ 1.0 - zcr # ༳Ε͕গͳ͍ ])
62. attack feature attack = { strength: attack[:attack_strength], # ग़ͩ͠ͷڧ͞ centroid: attack[:centroid_norm], # ग़ͩ͠ͷ໌Δ͞ high: attack[:high_ratio], # ߴҬͷڧ͞ low: attack[:low_ratio], # ௿Ҭͷڧ͞ flatness: attack[:spectral_flatness],# ϊΠζͬΆ͞ zcr: attack[:zero_crossing_rate], # ࡉ͔͍༳Ε f1 : attack[:f1_frequency], # F1 ͬΆ͍ࢁ f2 : attack[:f2_frequency], # F2 ͬΆ͍ࢁ nasal: attack_nasal_score # ඓԻͬΆ͞ }
63. 母 × kana_score = [ vowel_score * 0.55, # ฼ԻΒ͠͞ consonant_score * 0.35, # ࢠԻΒ͠͞ burst_bonus * 0.10 # ग़ͩ͠ͷิਖ਼ ].sum candidate = { text: kana_table[consonant][vowel], # ͔ͳจࣈ vowel: vowel, # ਪఆ฼Ի consonant: consonant, # ਪఆࢠԻ conf i dence: kana_score # ީิͷࣗ৴౓ }
65. vizcore Ruby ォ Ruby
66. $whoami ォ ydah ( ) SmartHR Ruby 西 Ruby Ruby ( 西 ) 西 圧 OSS 2