The Challenge of Automatically Configuring Numerai Compute Environment

I have tried to use Terraform to build the Numerai Compute environment so that it can be built automatically.

Transcript

1. Regonn @Numerai Community Tokyo Meetup 2023/04/08 The Challenge of Automatically Configuring Numerai Compute Environment
2. Regonn (@regonn_haizine) • Shimane, Japan • Freelancer(Machine Learning Engineer, Metaverse, NeuroTech) • Orca(Solana Chain DEX) Japanese community manager • Farmer My past glory
3. Numerai Compute • ࣗಈͰNumeraiʹαϒϛοτ͢Δ࢓૊Έ • Webhook URL Λొ࿥͢Δ • Daily submission͕͍ͨ͠ • numerai-cli ͩͱ AWS Fargate Λར༻ • https://pypi.org/project/numerai-cli/ • ࠓճ͸ࣗ෼Ͱͦͷ؀ڥΛߏஙͯ͠ΈΔ
4. ར༻ٕज़ • Google Cloud Functions ɾαʔόʔϨεͰಈ͔͢͜ͱ͕Ͱ͖ͯɺಈ͍͍ͯΔ͚࣌ؒͩ՝ۚ ɾྨࣅαʔϏεͷAWS Lambdaͷ15෼λΠϜΞ΢τΑΓ௕͘Ͱ͖Δ(࠷େ60෼) • Terraform ɾInfrastructure as Code Ͱɺ͍͍ͪͪ؅ཧը໘Λ৮ͬͯΠϯϑϥΛߏஙͤͣ ʹɺ࢖͍·ΘͤΔΑ͏ʹͳΔ
5. Google Cloud Functions with Terraform • Fargateͱಉ͡Α͏ʹGPU͸࢖͑ͳ͍ • αʔόʔϨεͰॲཧΛ࣮ߦՄೳ • ϝϞϦ͸࠷େ16GB(Preview 32GB) ※ϑΝΠϧ༰ྔ΋ΠϯϝϞϦͳͨΊ஫ҙ • GCFଆ͸ PythonϑΝΠϧ(main.py)ͱґଘϥΠϒϥϦఆٛ(requirements.txt) • Terraform ͰΠϯϑϥߏங༻ͷϑΝΠϧΛఆٛ • https://github.com/regonn/numerai-gcf-terraform
6. https://github.com/regonn/numerai-gcf-terraform ίʔυͬ͘͟Γղઆ
7. Google Cloud Run with Terraform • ϝϞϦ͕࠷େ 32 GB(Cloud functions ͸࠷େ16GB ※Preview: 32GB) • ࣮ߦ࣌ؒ͸ಉ͘͡࠷େ60෼ • ͨͩ͠ɺFastAPI౳ͷΤϯυϙΠϯτΛར༻ͨ͠DockerΛ࡞੒͢Δඞཁ͕͋Δ • Cloud functions Ͱ͸ରԠݴޠͷ੍ݶ͕͋Δ͕ɺCloud Run Ͱ͸ࣗ෼ͷ޷͖ͳ ݴޠΛબ୒Ͱ͖Δ
8. ίετΛԼ͛ΔͨΊʹ • ϝϞϦͷઅ໿(int8 dataset ౳) • Polars ɹɾRust੡ߴ଎σʔλϑϨʔϜϥΠϒϥϦ ɹɾhttps://www.pola.rs/ ɹɾϝϞϦ΍ॲཧ଎౓Λ্͛ΒΕΔͷͰɺίετ࡟ݮʹͭͳ͕Δ
9. ऴΘΓ