The Challenge of Automatically Configuring Numerai Compute Environment

I have tried to use Terraform to build the Numerai Compute environment so that it can be built automatically.

Transcript

1. Regonn @Numerai Community Tokyo Meetup 2023/04/08
   The Challenge of Automatically
   Configuring Numerai Compute
   Environment
   

   View Slide

2. Regonn (@regonn_haizine)
   • Shimane, Japan
   • Freelancer(Machine Learning Engineer, Metaverse, NeuroTech)
   • Orca(Solana Chain DEX) Japanese community manager
   • Farmer
   My past glory
   

   View Slide

3. Numerai Compute
   • ࣗಈͰNumeraiʹαϒϛοτ͢Δ࢓૊Έ
   • Webhook URL Λొ࿥͢Δ
   • Daily submission͕͍ͨ͠
   • numerai-cli ͩͱ AWS Fargate Λར༻
   • https://pypi.org/project/numerai-cli/
   • ࠓճ͸ࣗ෼Ͱͦͷ؀ڥΛߏஙͯ͠ΈΔ
   

   View Slide

4. ར༻ٕज़
   • Google Cloud Functions
   ɾαʔόʔϨεͰಈ͔͢͜ͱ͕Ͱ͖ͯɺಈ͍͍ͯΔ͚࣌ؒͩ՝ۚ
   ɾྨࣅαʔϏεͷAWS Lambdaͷ15෼λΠϜΞ΢τΑΓ௕͘Ͱ͖Δ(࠷େ60෼)
   • Terraform
   ɾInfrastructure as Code Ͱɺ͍͍ͪͪ؅ཧը໘Λ৮ͬͯΠϯϑϥΛߏஙͤͣ
   ʹɺ࢖͍·ΘͤΔΑ͏ʹͳΔ
   

   View Slide

5. Google Cloud Functions with Terraform
   • Fargateͱಉ͡Α͏ʹGPU͸࢖͑ͳ͍
   • αʔόʔϨεͰॲཧΛ࣮ߦՄೳ
   • ϝϞϦ͸࠷େ16GB(Preview 32GB) ※ϑΝΠϧ༰ྔ΋ΠϯϝϞϦͳͨΊ஫ҙ
   • GCFଆ͸ PythonϑΝΠϧ(main.py)ͱґଘϥΠϒϥϦఆٛ(requirements.txt)
   • Terraform ͰΠϯϑϥߏங༻ͷϑΝΠϧΛఆٛ
   • https://github.com/regonn/numerai-gcf-terraform
   

   View Slide

6. https://github.com/regonn/numerai-gcf-terraform
   ίʔυͬ͘͟Γղઆ
   

   View Slide

7. Google Cloud Run with Terraform
   • ϝϞϦ͕࠷େ 32 GB(Cloud functions ͸࠷େ16GB ※Preview: 32GB)
   • ࣮ߦ࣌ؒ͸ಉ͘͡࠷େ60෼
   • ͨͩ͠ɺFastAPI౳ͷΤϯυϙΠϯτΛར༻ͨ͠DockerΛ࡞੒͢Δඞཁ͕͋Δ
   • Cloud functions Ͱ͸ରԠݴޠͷ੍ݶ͕͋Δ͕ɺCloud Run Ͱ͸ࣗ෼ͷ޷͖ͳ
   ݴޠΛબ୒Ͱ͖Δ
   

   View Slide

8. ίετΛԼ͛ΔͨΊʹ
   • ϝϞϦͷઅ໿(int8 dataset ౳)
   • Polars
   ɹɾRust੡ߴ଎σʔλϑϨʔϜϥΠϒϥϦ
   ɹɾhttps://www.pola.rs/
   ɹɾϝϞϦ΍ॲཧ଎౓Λ্͛ΒΕΔͷͰɺίετ࡟ݮʹͭͳ͕Δ
   

   View Slide

9. ऴΘΓ
   

   View Slide