SwiftでVRMファイルを表示してみた話

https://github.com/tattn/VTuberKit
https://github.com/tattn/VRMKit

UnityなしでVRMを読み込んでVTuber機能をアプリに組み込めるライブラリを作りました
https://qiita.com/tattn/items/c2d4505064f78da93ef3

#potatotips 59
https://potatotips.connpass.com/event/119277/

Transcript

1. SwiftͰVRMϑΝΠϧΛදࣔͯ͠Έͨ࿩
   ాதୡ໵ (@tattn)
   #potatotips 59
   

   View Slide

2. ాத ୡ໵ / ͨͳͨͭ (@tattn)
   • Yahoo!৐׵Ҋ಺
   • iOSΞϓϦΤϯδχΞ
   • Unity΋΍ͬͯ·͢
   @tattn
   @tanakasan2525
   @tattn
   

   View Slide

3. VRMͱ͸
   

   View Slide

4. VRMͱ͸
   IUUQTEXBOHPHJUIVCJPWSN
   VR޲͚ͷਓܕ3DΞόλʔϑΥʔϚοτ

   ϙʔλϏϦςΟ͕ߴͯ͘ѻ͍΍͍͢

   ެࣜͰఏڙ͞Ε͍ͯΔͷ͸Unity࣮૷ͷΈ
   ɾχίχཱମ

   ɾόʔνϟϧΩϟετ

   ɾcluster

   ɾVDraw

   ɾVRoid ͳͲͰར༻͞Ε͍ͯΔ
   

   View Slide

5. SwiftͰVRMΛදࣔͰ͖Δ

   ϥΠϒϥϦΛ࡞Γ·ͨ͠
   

   View Slide

6. DEMO
   

   View Slide

7. PDF / DEMOࣦഊ༻εΫγϣ
   IUUQTUXJUUFSDPNUBOBLBTBOTUBUVT
   ಈըˠ
   © Kizuna AI
   © SSS LLC.
   © DWANGO Co., Ltd.
   

   View Slide

8. VTuberKit / VRMKit / VRMSceneKit
   VTuberKit
   IUUQTHJUIVCDPNUBUUO75VCFS,JU
   IUUQTHJUIVCDPNUBUUO73.,JU
   VTuberͬΆ͍͜ͱ͕Ͱ͖Δ (؆୯ʹΩϟϥΛදࣔͰ͖Δ)
   VRMKit
   VRMSceneKit
   VRMͷಡΈࠐΈ͕Ͱ͖Δ
   ಡΈࠐΜͩVRMΛSceneKitͰදࣔͰ͖Δ
   ґଘάϥϑ
   

   View Slide

9. ࣮૷
   (ϥΠϒϥϦͷ࢖͍ํ͸GitHubΛݟͯͶ)
   

   View Slide

10. VRMͷಡΈࠐΈ
    

    View Slide

11. VRM (GLTF/GLB) ͷσʔλߏ଄
    IUUQTHJUIVCDPN,ISPOPT(SPVQHM5'CMPCNBTUFSTQFDJpDBUJPO3&"%.&NE
    magic = ϑΝΠϧܗࣜͷνΣοΫ༻

    version = GLBͷόʔδϣϯ (VRMͷ৔߹ɺ2)

    length = σʔλͷαΠζ

    Chunk = Ϟσϧσʔλ (0͸JSONɺ1͸όΠφϦ)
    

    View Slide

12. SwiftͰόΠφϦσʔλΛಡΈࠐΉ
    let data = try Data(contentsOf: url)
    let version: UInt32 = data.subdata(in: 0..<4)
    .withUnsafeBytes { $0.pointee }
    subdataͰόΠφϦσʔλΛ੾Γग़ͯ͠

    withUnsafeBytesͰద੾ͳσʔλܕͱͯ͠औΓग़͢
    

    View Slide

13. Chunk 0͸JSON
    

    View Slide

14. Chunk 0ͷJSONΛಡΈࠐΉ
    let endOffset = offset + Int(length)
    let jsonData = data.subdata(in: offset..let decoder = JSONDecoder()
    self.jsonData = try decoder.decode(GLTF.self, from: jsonData)
    JSON෦෼ͷσʔλΛ੾Γग़ͯ͠ɺ

    JSONDecoderʹ౤͛Δ
    

    View Slide

15. GLTFܕ
    public struct GLTF: Codable {
    let extensionsUsed: [String]?
    let extensionsRequired: [String]?
    let accessors: [Accessor]?
    let animations: [Animation]?
    let asset: Asset
    let buffers: [Buffer]?
    let bufferViews: [BufferView]?
    let cameras: [Camera]?
    let images: [Image]?
    let materials: [Material]?
    let meshes: [Mesh]?
    let nodes: [Node]?
    let samplers: [Sampler]?
    let _scene: Int?
    var scene: Int { return _scene ?? 0 }
    let scenes: [Scene]?
    let skins: [Skin]?
    let textures: [Texture]?
    let extensions: Extension?
    let extras: Extras?
    private enum CodingKeys: String, CodingKey {
    case extensionsUsed
    case extensionsRequired
    case accessors
    case animations
    case asset
    case buffers
    case bufferViews
    case cameras
    case images
    case materials
    case meshes
    case nodes
    case samplers
    case _scene = "scene"
    case scenes
    case skins
    case textures
    case extensions
    case extras
    }
    }
    εϥΠυͰ͸঺հ͕೉͍͠ͷͰ

    GitHubͰݟͯͶ
    IUUQTHJUIVCDPNUBUUO73.,JUCMPCNBTUFS4PVSDFT73.,JU73.(-5'TXJGU
    

    View Slide

16. ࢓༷ॻΛݟͳ͕ΒCodableͳϞσϧΛ࡞Δ
    ࢓༷ॻ͕ΊͪΌΊͪΌ

    Θ͔Γ΍͍͢ʂ
    IUUQTHJUIVCDPN,ISPOPT(SPVQHM5'CMPCNBTUFSTQFDJpDBUJPO
    3&"%.&NESFGFSFODFBDDFTTPS
    

    View Slide

17. VRMͷදࣔ
    

    View Slide

18. ϝογϡͷ࡞੒ (ൈਮ)
    let meshNode = SCNNode()
    for primitive in mesh.primitives { // primitive = GLTF.Mesh.Primitive
    let attributes: [SCNGeometrySource] = <௖఺ɾ๏ઢσʔλͷಡΈࠐΈ>
    let elements: [SCNGeometryElement] = <௖఺ΠϯσοΫεͷಡΈࠐΈ>
    let geometry = SCNGeometry(sources: attributes, elements: elements)
    geometry.materials = <ϚςϦΞϧͷಡΈࠐΈ>
    let primitiveNode = SCNNode()
    primitiveNode.geometry = geometry
    meshNode.addChildNode(primitiveNode)
    }
    

    View Slide

19. ௖఺σʔλͷ࡞੒ (ൈਮ)
    let buffer = vrm.gltf.binaryBuffer
    let bufferView = buffer.subdata(in: byteOffset..let source = SCNGeometrySource(data: bufferView,
    semantic: .vertex,
    vectorCount: accessor.count,
    usesFloatComponents: true,
    componentsPerVector: 3, // x, y, z
    bytesPerComponent: 4, // float
    dataOffset: accessor.byteOffset,
    dataStride: 3 * 4)
    

    View Slide

20. ීஈ͋·Γ࢖Θͳ͍໘ന͍Ϋϥε
    SCNGeometrySource = ௖఺΍๏ઢɺϘʔϯͳͲͷσʔλ
    SCNGeometryElement = ௖఺ΠϯσοΫε
    SCNMaterial / SCNMaterialProperty = ςΫενϟ΍ը૾ͳͲͷσʔλ
    SCNMorpher = ϞʔϑΟϯά (ද৘ͷมߋ)
    SCNSkinner = εΩχϯά (ϘʔϯΞχϝʔγϣϯ)
    CAKeyframeAnimation = ΩʔϑϨʔϜΞχϝʔγϣϯ
    

    View Slide

21. ͕࣌ؒͳ͍ͷͰ

    ࠷ޙʹ
    

    View Slide

22. ࠷ޙʹ
    QiitaʹϥΠϒϥϦͷ࢖͍ํΛॻ͖·ͨ͠
    IUUQTRJJUBDPNUBUUOJUFNTDEGEBFG
    ͨ͘͞Μͷ͍͍ͶΛ௖͍ͨͷͰ

    ػೳ֦ுΛؤுΖ͏ͱࢥ͍·͢
    IUUQTUXJUUFSDPNUBOBLBTBOTUBUVT
    

    View Slide

23. ࠓ͸UnityͷAnimationΛimportͰ͖Δ࢓૊ΈΛ࡞ͬͯ·͢
    ↑͋ͱগͬ͠Ά͍
    ࣦഊͯ͠δϣδϣཱͪʁʹ
    Mecanimͷ

    σϑΥϧτϙʔζ͸

    Ҡ২੒ޭ
    ˜6OJUZ5FDIOPMPHJFT+BQBO6$-
    Unityͷ๛෋ͳΞηοτ͕࢖͑ΔΑ͏ʹͳͬͨΒ࠷ߴͰ͢
    

    View Slide