The Challenge of Automatically Configuring Numerai Compute Environment

Slide 1

Slide 1 text

Regonn @Numerai Community Tokyo Meetup 2023/04/08 The Challenge of Automatically Configuring Numerai Compute Environment

Slide 2

Slide 2 text

Regonn (@regonn_haizine) • Shimane, Japan • Freelancer(Machine Learning Engineer, Metaverse, NeuroTech) • Orca(Solana Chain DEX) Japanese community manager • Farmer My past glory

Slide 3

Slide 3 text

Numerai Compute • ࣗಈͰNumeraiʹαϒϛοτ͢Δ࢓૊Έ • Webhook URL Λొ࿥͢Δ • Daily submission͕͍ͨ͠ • numerai-cli ͩͱ AWS Fargate Λར༻ • https://pypi.org/project/numerai-cli/ • ࠓճ͸ࣗ෼Ͱͦͷ؀ڥΛߏஙͯ͠ΈΔ

Slide 4

Slide 4 text

ར༻ٕज़ • Google Cloud Functions ɾαʔόʔϨεͰಈ͔͢͜ͱ͕Ͱ͖ͯɺಈ͍͍ͯΔ͚࣌ؒͩ՝ۚ ɾྨࣅαʔϏεͷAWS Lambdaͷ15෼λΠϜΞ΢τΑΓ௕͘Ͱ͖Δ(࠷େ60෼) • Terraform ɾInfrastructure as Code Ͱɺ͍͍ͪͪ؅ཧը໘Λ৮ͬͯΠϯϑϥΛߏஙͤͣ ʹɺ࢖͍·ΘͤΔΑ͏ʹͳΔ

Slide 5

Slide 5 text

Google Cloud Functions with Terraform • Fargateͱಉ͡Α͏ʹGPU͸࢖͑ͳ͍ • αʔόʔϨεͰॲཧΛ࣮ߦՄೳ • ϝϞϦ͸࠷େ16GB(Preview 32GB) ※ϑΝΠϧ༰ྔ΋ΠϯϝϞϦͳͨΊ஫ҙ • GCFଆ͸ PythonϑΝΠϧ(main.py)ͱґଘϥΠϒϥϦఆٛ(requirements.txt) • Terraform ͰΠϯϑϥߏங༻ͷϑΝΠϧΛఆٛ • https://github.com/regonn/numerai-gcf-terraform

Slide 6

Slide 6 text

https://github.com/regonn/numerai-gcf-terraform ίʔυͬ͘͟Γղઆ

Slide 7

Slide 7 text

Google Cloud Run with Terraform • ϝϞϦ͕࠷େ 32 GB(Cloud functions ͸࠷େ16GB ※Preview: 32GB) • ࣮ߦ࣌ؒ͸ಉ͘͡࠷େ60෼ • ͨͩ͠ɺFastAPI౳ͷΤϯυϙΠϯτΛར༻ͨ͠DockerΛ࡞੒͢Δඞཁ͕͋Δ • Cloud functions Ͱ͸ରԠݴޠͷ੍ݶ͕͋Δ͕ɺCloud Run Ͱ͸ࣗ෼ͷ޷͖ͳ ݴޠΛબ୒Ͱ͖Δ

Slide 8

Slide 8 text

ίετΛԼ͛ΔͨΊʹ • ϝϞϦͷઅ໿(int8 dataset ౳) • Polars ɹɾRust੡ߴ଎σʔλϑϨʔϜϥΠϒϥϦ ɹɾhttps://www.pola.rs/ ɹɾϝϞϦ΍ॲཧ଎౓Λ্͛ΒΕΔͷͰɺίετ࡟ݮʹͭͳ͕Δ

Slide 9

Slide 9 text

ऴΘΓ