Notes from the March 2025 TPC Hackathon

Event: SambaNova Cloud Tokyo Mingle 2 (2025-04-16)
Speaker: Aleksandr Drozd (Research scientist, RIKEN Center for Computational Science)

Transcript

1. Notes from the March 2025 TPC Hackathon Aleksandr Drozd 💼 RIKEN Center For Computational Science 💼 Institute of Science Tokyo 💼 Amigawa GK ⌂ blackbird.pw X-TWITTE bkbrd GITHUB undertherain GITLAB undertherain April 17, 2025
2. RIKEN CCS one of the largest research institutions in Japan pronounced as REE - KEN short for 理化学研究所 centers and facilities across the country CCS, AIP, brain science, quantum etc designated to host country’s largest supercomputer 2 / 15
3. Kobe city, Hyogo prefecture 3 / 15
4. Kobe Port Island 4 / 15
5. RIKEN Center for Computational Science science of computing × computing for science 5 / 15
6. Supercomputer Fugaku [富岳] Japanese ”flagship” supercomputer 158,976 nodes 442 PFLOPS (per TOP500 Rmax) 30 megawatt of electric power World’s faster machine 2020-2022 6 / 15
7. TPC Trillion Parameter Consortium: New international consortium formed to create trustworthy and reliable generative AI models for science Includes dozens of members from around the world, including RIKEN... and SambaNova! https://tpc.dev/ 7 / 15
8. Spring Hackathon (March 2025) Architectures, Evaluation, and Life Sciences AI Challenge Problems breakout intro three groups: Scientific Skills, Safety, and Trust Evaluation (EVAL) Model Architecture and Performance Evaluation (MAPE) Life Sciences Challenge Problems (BIO) 8 / 15
9. Institution Count RIKEN 23 Argonne 11 BSC 7 Institute of Science Tokyo 5 LBNL 2 SambaNova Systems 2 Epistra Inc. 2 The University of Tokyo 2 Rikken BDR 1 University of New South Wales 1 NCI Australia 1 PNNL 1 Metagov 1 Coalition for Epidemic Preparedness Innovations 1 Yonsei University 1 AIST 1 HPE 1 NVIDIA 1 Fixstars Corporation 1 FINAL HEAD COUNT 65 9 / 15
10. Why LLM inference DeepSeek V3 just came out and we wanted to use it at the hackathon we have compute infrastructure, but it was hard to expediently provide access to outsiders friends from SambaNova offered help our interaction goes back to pre-LLM era 10 / 15
11. How we used LLMs: chat interface to let people try prompts/ experience new model in general extracting bio-medical data using DSPy collecting reasoning chains for later reuse experiments with using LLMs to judge how other LLMs solve math problems 11 / 15
12. Experience with SN very easy to get started! registering on the site, getting API key OpenAI -compatible out of the box support through liteLLM and everything built on top of it very fast useful details in API support for very large models like DeepSeek 671B, very promptly 12 / 15
13. Usage report from API { "completion_tokens": 420, "prompt_tokens": 10, "total_tokens": 430, "completion_tokens_after_first_per_sec": 216.63, "completion_tokens_after_first_per_sec_first_ten": 217.67, "completion_tokens_per_sec": 203.33, "end_time": 1744677052.79, "is_last_response": true, "start_time": 1744677050.73, "stop_reason": "stop", "time_to_first_token": 0.13, "total_latency": 2.06, "total_tokens_per_sec": 208.187 } 13 / 15
14. Features I’d love to see added multiple API keys higher / transparent / configurable rate limits lower prices prompt caching custom models fine-tuning as a service 14 / 15
15. Thank you for your attention! Questions are welcome! 15 / 15