I have tried to use Terraform to build the Numerai Compute environment so that it can be built automatically.
Regonn @Numerai Community Tokyo Meetup 2023/04/08The Challenge of AutomaticallyConfiguring Numerai ComputeEnvironment
View Slide
Regonn (@regonn_haizine)• Shimane, Japan• Freelancer(Machine Learning Engineer, Metaverse, NeuroTech)• Orca(Solana Chain DEX) Japanese community manager• FarmerMy past glory
Numerai Compute• ࣗಈͰNumeraiʹαϒϛοτ͢ΔΈ• Webhook URL Λొ͢Δ• Daily submission͕͍ͨ͠• numerai-cli ͩͱ AWS Fargate Λར༻• https://pypi.org/project/numerai-cli/• ࠓճࣗͰͦͷڥΛߏஙͯ͠ΈΔ
ར༻ٕज़• Google Cloud FunctionsɾαʔόʔϨεͰಈ͔͢͜ͱ͕Ͱ͖ͯɺಈ͍͍ͯΔ͚࣌ؒͩ՝ۚɾྨࣅαʔϏεͷAWS Lambdaͷ15λΠϜΞτΑΓ͘Ͱ͖Δ(࠷େ60)• TerraformɾInfrastructure as Code Ͱɺ͍͍ͪͪཧը໘Λ৮ͬͯΠϯϑϥΛߏஙͤͣʹɺ͍·ΘͤΔΑ͏ʹͳΔ
Google Cloud Functions with Terraform• Fargateͱಉ͡Α͏ʹGPU͑ͳ͍• αʔόʔϨεͰॲཧΛ࣮ߦՄೳ• ϝϞϦ࠷େ16GB(Preview 32GB) ※ϑΝΠϧ༰ྔΠϯϝϞϦͳͨΊҙ• GCFଆ PythonϑΝΠϧ(main.py)ͱґଘϥΠϒϥϦఆٛ(requirements.txt)• Terraform ͰΠϯϑϥߏங༻ͷϑΝΠϧΛఆٛ• https://github.com/regonn/numerai-gcf-terraform
https://github.com/regonn/numerai-gcf-terraformίʔυͬ͘͟Γղઆ
Google Cloud Run with Terraform• ϝϞϦ͕࠷େ 32 GB(Cloud functions ࠷େ16GB ※Preview: 32GB)• ࣮ߦ࣌ؒಉ͘͡࠷େ60• ͨͩ͠ɺFastAPIͷΤϯυϙΠϯτΛར༻ͨ͠DockerΛ࡞͢Δඞཁ͕͋Δ• Cloud functions ͰରԠݴޠͷ੍ݶ͕͋Δ͕ɺCloud Run Ͱࣗͷ͖ͳݴޠΛબͰ͖Δ
ίετΛԼ͛ΔͨΊʹ• ϝϞϦͷઅ(int8 dataset )• PolarsɹɾRustߴσʔλϑϨʔϜϥΠϒϥϦɹɾhttps://www.pola.rs/ɹɾϝϞϦॲཧΛ্͛ΒΕΔͷͰɺίετݮʹͭͳ͕Δ
ऴΘΓ