Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The Challenge of Automatically Configuring Numerai Compute Environment

regonn
April 08, 2023

The Challenge of Automatically Configuring Numerai Compute Environment

I have tried to use Terraform to build the Numerai Compute environment so that it can be built automatically.

regonn

April 08, 2023
Tweet

More Decks by regonn

Other Decks in Technology

Transcript

  1. Regonn @Numerai Community Tokyo Meetup 2023/04/08
    The Challenge of Automatically
    Configuring Numerai Compute
    Environment

    View Slide

  2. Regonn (@regonn_haizine)
    • Shimane, Japan
    • Freelancer(Machine Learning Engineer, Metaverse, NeuroTech)
    • Orca(Solana Chain DEX) Japanese community manager
    • Farmer
    My past glory

    View Slide

  3. Numerai Compute
    • ࣗಈͰNumeraiʹαϒϛοτ͢Δ࢓૊Έ
    • Webhook URL Λొ࿥͢Δ
    • Daily submission͕͍ͨ͠
    • numerai-cli ͩͱ AWS Fargate Λར༻
    • https://pypi.org/project/numerai-cli/
    • ࠓճ͸ࣗ෼Ͱͦͷ؀ڥΛߏஙͯ͠ΈΔ

    View Slide

  4. ར༻ٕज़
    • Google Cloud Functions
    ɾαʔόʔϨεͰಈ͔͢͜ͱ͕Ͱ͖ͯɺಈ͍͍ͯΔ͚࣌ؒͩ՝ۚ
    ɾྨࣅαʔϏεͷAWS Lambdaͷ15෼λΠϜΞ΢τΑΓ௕͘Ͱ͖Δ(࠷େ60෼)
    • Terraform
    ɾInfrastructure as Code Ͱɺ͍͍ͪͪ؅ཧը໘Λ৮ͬͯΠϯϑϥΛߏஙͤͣ
    ʹɺ࢖͍·ΘͤΔΑ͏ʹͳΔ

    View Slide

  5. Google Cloud Functions with Terraform
    • Fargateͱಉ͡Α͏ʹGPU͸࢖͑ͳ͍
    • αʔόʔϨεͰॲཧΛ࣮ߦՄೳ
    • ϝϞϦ͸࠷େ16GB(Preview 32GB) ※ϑΝΠϧ༰ྔ΋ΠϯϝϞϦͳͨΊ஫ҙ
    • GCFଆ͸ PythonϑΝΠϧ(main.py)ͱґଘϥΠϒϥϦఆٛ(requirements.txt)
    • Terraform ͰΠϯϑϥߏங༻ͷϑΝΠϧΛఆٛ
    • https://github.com/regonn/numerai-gcf-terraform

    View Slide

  6. https://github.com/regonn/numerai-gcf-terraform
    ίʔυͬ͘͟Γղઆ

    View Slide

  7. Google Cloud Run with Terraform
    • ϝϞϦ͕࠷େ 32 GB(Cloud functions ͸࠷େ16GB ※Preview: 32GB)
    • ࣮ߦ࣌ؒ͸ಉ͘͡࠷େ60෼
    • ͨͩ͠ɺFastAPI౳ͷΤϯυϙΠϯτΛར༻ͨ͠DockerΛ࡞੒͢Δඞཁ͕͋Δ
    • Cloud functions Ͱ͸ରԠݴޠͷ੍ݶ͕͋Δ͕ɺCloud Run Ͱ͸ࣗ෼ͷ޷͖ͳ
    ݴޠΛબ୒Ͱ͖Δ

    View Slide

  8. ίετΛԼ͛ΔͨΊʹ
    • ϝϞϦͷઅ໿(int8 dataset ౳)
    • Polars
    ɹɾRust੡ߴ଎σʔλϑϨʔϜϥΠϒϥϦ
    ɹɾhttps://www.pola.rs/
    ɹɾϝϞϦ΍ॲཧ଎౓Λ্͛ΒΕΔͷͰɺίετ࡟ݮʹͭͳ͕Δ

    View Slide

  9. ऴΘΓ

    View Slide