https://speakerdeck.com/rikkabotan7 2025-09-07 15:45:01 -0400 ローカルAIに向き合う展示会 vol.2における発表スライドです。 双方向エンコーダーモデルの研究進捗に関する発表です。 ローカルAIに向き合う展示会 vol.2における発表スライドです。 双方向エンコーダーモデルの研究進捗に関する発表です。 Tue, 30 Jun 2026 00:00:00 -0400 https://speakerdeck.com/rikkabotan7/rokaruainixiang-kihe-uzhan-shi-hui-vol-dot-2-ye-ti-shi-jian-ding-shu-xing-moziyuruwoyong-ita-orizinarunoshuang-fang-xiang-enkodamoderunexterabert-tui-lun-su-du-xiang-shang-jian-tao-bing-binidaunsutorimuping-jia https://speakerdeck.com/rikkabotan7/rokaruainixiang-kihe-uzhan-shi-hui-vol-dot-2-ye-ti-shi-jian-ding-shu-xing-moziyuruwoyong-ita-orizinarunoshuang-fang-xiang-enkodamoderunexterabert-tui-lun-su-du-xiang-shang-jian-tao-bing-binidaunsutorimuping-jia 双方向エンコーダーモデルをLFM2などで用いられている液体時間定数型のモジュールと、Gated Attention, Scalable Softmaxなどのモジュールを組み合わせて作成し、事前学習を行なったという内容です。ELECTRAのようなGeneratorとDiscriminatorを用いた学習を取り入れることで迅速な収束を示し、かつ推論速度の向上も確認されました。 双方向エンコーダーモデルをLFM2などで用いられている液体時間定数型のモジュールと、Gated Attention, Scalable Softmaxなどのモジュールを組み合わせて作成し、事前学習を行なったという内容です。ELECTRAのようなGeneratorとDiscriminatorを用いた学習を取り入れることで迅速な収束を示し、かつ推論速度の向上も確認されました。 Wed, 06 May 2026 00:00:00 -0400 https://speakerdeck.com/rikkabotan7/sheng-cheng-ainandemozhan-shi-hui-vol-dot-5-ltdeng-tan-nexterabertfa-biao-zi-liao https://speakerdeck.com/rikkabotan7/sheng-cheng-ainandemozhan-shi-hui-vol-dot-5-ltdeng-tan-nexterabertfa-biao-zi-liao Static embedding models enable fast inference due to their simple architecture, but, it is well known that improving their structural expressiveness is challenging. At the same time, as corpora continue to grow in scale, the demand for both higher efficiency and higher accuracy in embedding models has increased significantly. In this work, we propose a simple yet effective method called SSE (Stable Static Embedding), which incorporates Separable DyT (Dynamic Tanh normalization). We demonstrate that SSE achieves higher retrieval performance than prior approaches while using only half the number of parameters. Despite having only 16 million parameters, SSE attains a mean NanoBEIR (English) nDCG@10 score of 0.512. By leveraging Separable DyT, SSE effectively regulates gradient flow and suppresses inter-dimensional imbalance and overfitting, thereby improving generalization performance. Our method provides a new perspective on static embedding models and offers a pathway toward faster and more accurate retrieval systems. Static embedding models enable fast inference due to their simple architecture, but, it is well known that improving their structural expressiveness is challenging. At the same time, as corpora continue to grow in scale, the demand for both higher efficiency and higher accuracy in embedding models has increased significantly. In this work, we propose a simple yet effective method called SSE (Stable Static Embedding), which incorporates Separable DyT (Dynamic Tanh normalization). We demonstrate that SSE achieves higher retrieval performance than prior approaches while using only half the number of parameters. Despite having only 16 million parameters, SSE attains a mean NanoBEIR (English) nDCG@10 score of 0.512. By leveraging Separable DyT, SSE effectively regulates gradient flow and suppresses inter-dimensional imbalance and overfitting, thereby improving generalization performance. Our method provides a new perspective on static embedding models and offers a pathway toward faster and more accurate retrieval systems. Fri, 03 Apr 2026 00:00:00 -0400 https://speakerdeck.com/rikkabotan7/sse-stable-static-embedding https://speakerdeck.com/rikkabotan7/sse-stable-static-embedding SSE(Stable Static Embedding): Unlocking the Potential of Static Embeddings, A Dynamic Tanh Normalization Approach without Speed Penalty SSE(Stable Static Embedding): Unlocking the Potential of Static Embeddings, A Dynamic Tanh Normalization Approach without Speed Penalty Wed, 18 Mar 2026 00:00:00 -0400 https://speakerdeck.com/rikkabotan7/rokaruai-ltda-hui-sse-stable-static-embedding-su-du-di-xia-woban-wazu-jing-de-mai-meip-mimoderunoqian-zai-neng-li-woyin-kichu-su-dynamic-tanhshou-fa-noti-an https://speakerdeck.com/rikkabotan7/rokaruai-ltda-hui-sse-stable-static-embedding-su-du-di-xia-woban-wazu-jing-de-mai-meip-mimoderunoqian-zai-neng-li-woyin-kichu-su-dynamic-tanhshou-fa-noti-an A novel architecture and framework to bridge the gap between Attention mechanism and SSMs: State Space Models A novel architecture and framework to bridge the gap between Attention mechanism and SSMs: State Space Models Sun, 07 Sep 2025 00:00:00 -0400 https://speakerdeck.com/rikkabotan7/sea-model-series-op-dot-1-saint-lupinus-pre-release https://speakerdeck.com/rikkabotan7/sea-model-series-op-dot-1-saint-lupinus-pre-release