Self-improving RLVR for secure Terraform

Transcript

1. rl-gym Audited RLVR for secure Terraform An 8B open model trained against a security verifier. Self-repairing, self-improving. Runs on Nebius — Token Factory, a container, or your VPC. security 0.69 → 0.80 on real requests a self-improvement flywheel 25–30× cheaper than APIs Nebius Serverless AI Builders Challenge · 2026 · Nikita Shigarov
2. THE PROBLEM Terraform is a security liability — and the bar keeps moving 46% human-written Terraform avg security score of real reference configs (rules passed) 1 in 5 has a CRITICAL flaw hardcoded secrets · open SSH · public DBs · wildcard IAM 57–71% of LLM "fixes" are fake look secure, keep the vulnerability And the standard drifts: new CVEs, new services, new rules — every quarter.
3. THE IDEA The security scanner IS the teacher 1 Verifier Rules + hard gates. Code, not opinions. → 2 GRPO (RL) Model explores. Scanner scores. → 3 Audit Catches reward- hacking pre- release. → 4 Serve pass@1 + self-repair. → 5 Continual New rules absorbed for ~$1.30. Prompting rents compliance per request. RL compiles it into the weights — once.
4. THE MECHANISM How the reward + gates are calculated reward = pass-rate of applicable rules × [ all hard gates pass ? 1 : 0 ] Soft components — [0,1] • Each of ~18 rules returns pass / fail / na. • reward = passes ÷ applicable rules. • A smooth gradient GRPO can climb. Hard gates — zero it • builds_required: declared what was asked (non- empty). • no_critical: zero critical findings (secrets, public DB…). • Either fails → reward = 0. No partial credit. Example: a config passes 6/7 rules (0.86) but ships a hardcoded secret → no_critical FAILS → reward 0.00. That's why gates exist.
5. THE VERIFIER · THE MOAT Built as code — and it grows itself Each rule is a deterministic predicate fn(hcl) → pass / fail / na · severity-tiered · hard gates zero anything gamed · CPU, no LLM in the reward. What it checks • Encryption at rest — S3 · RDS · EBS · KMS rotation • Public exposure — buckets · DBs · open SG ports • IAM least-privilege — no Action/Resource "*" • Secrets — no hardcoded credentials • Logging & resilience — flow logs · backups · deletion • ~18 built-in → escalates to Checkov's 1,000+ How it grows • A big model AUTHORS rules from a spec — AST sandbox → passes every test → human review • Traffic × Checkov DISCOVERS blind spots → new specs (found the KMS-key-policy gap this way) • Both gated: the model proposes, tests + humans dispose • The model never becomes the judge — the reward stays deterministic LLMs write the verifier's rules; the verifier is never an LLM — that's what keeps the reward deterministic and un-gameable.
6. THE DATA Datasets — how each is built and used dataset labeled? used where IaC-Eval — 455 human request→TF reference ~0.46 secure — NOT a target dev-120 benchmark + RL requests Parametric generator — 395 stacks verified-1.0 oracle SFT target + GRPO GitHub-mined — 274 / 92 repos mostly none (req + required only) GRPO Checkov corpus + mined holdout — 83 none HOLDOUT — never trained RAFT / repair / served traffic self-generated, scanner-scored SFT / repair-GRPO / flywheel The point: GRPO needs only (request, required types) — no gold output; the scanner supplies the reward. The only imitation targets (SFT) are machine-verified-secure, never the ~0.46 human references. Splits: dev = fixed 120 disjoint from training (the +0.04 leak we caught & fixed); holdout = a source training never samples (hash- disjoint).
7. RESULTS · REAL REQUESTS, FROZEN & DISJOINT Every stage earns its keep policy reward gates meaning human reference 0.459 85.0% what practitioners wrote base granite-4.1-8b 0.629 76.7% best open 8B, untouched + SFT → + GRPO 0.694 81.7% reinforcement on the scanner FULL SYSTEM (+ repair) 0.802 90.0% generates → scans itself → repairs 1.5× the human baseline 0.80 vs 0.46, with better gates 0.876 / 93% on a never-trained source holdout: gains grow, not vanish audit clean no reward hacking uniq 93% · balanced component lift
8. THE REWARD-HACKING AUDIT How we prove it isn't gaming the reward Output-diversity alarm A gamed reward collapses outputs to one pattern — we track output uniqueness every training step. uniq 93.3% · flat Per-rule breakdown Real learning lifts many rules; gaming spams one — the security score is broken out per rule. 0.842→0.881→0.913 Hard gates + exploit tests Empty or hidden-secret configs that game the soft score are zeroed; a unit test proves the empty-config exploit scores 0. exploit → 0.00 Frozen-anchor promotion gate Every candidate is gated vs the incumbent on frozen, disjoint anchors — it caught our own +0.04 data leak and blocked a fashionable upgrade that regressed OOD. 0.737 → 0.694
9. THE MOAT It improves itself — and a program decides what ships generate → scan → repair → ship the better pass flywheel: served traffic → next model incumbent after gate real dev (n=60) 0.697 0.738 PASS holdout (n=60) 0.859 0.906 PASS a program promotes it — no human PROMOTE ▲ ~$4 / cycle traffic → better model gate-checked, repeatable $1.30 absorb a new policy continual update, no forgetting + new rules the verifier grows too a big model drafts them, gated by tests
10. THE SELF-IMPROVING VERIFIER The big model proposes new rules — from logs × checkers 1 Mine served logs the model's own traffic → 2 Check vs Checkov disagreements surface blind spots → 3 Big model drafts DeepSeek-V4-Pro writes the rule from a spec → 4 Gate → merge AST sandbox · tests · human review Case — KMS. Logs × Checkov surfaced an aws_kms_key with no policy; DeepSeek-V4-Pro drafted the kms_key_policy rule. The attempt even exposed a false-positive in our own critical IAM rule — the loop auditing its own reward.
11. HOW IT RUNS · NEBIUS TOKEN FACTORY + IN-VPC Two open models, two roles Serve the specialist — tuned 8B • In your VPC, a Nebius container, or Token Factory • ~$0.006 / request on an L40S — stays in-boundary • Needs a Llama/Qwen base (not granite) Grow the verifier — Token Factory • DeepSeek-V4-Pro drafts rules from log × checker gaps • AST sandbox → passes every test → human merge • per-token, pennies — the self-improving verifier tuned 8B SERVES the traffic verifier JUDGES — on CPU DeepSeek-V4-Pro WRITES the rules All open-weight. Rule-authoring is serverless on Token Factory; the tuned model self-hosts in-VPC or on a managed Nebius container. The verifier is CPU Python — no GPU for the judge.
12. VS. FRONTIER APIS Knowing the verifier beats being bigger condition reward gates $ / request Claude Fable 5, blind 0.598 70% ~$0.17 our 8B system (in your VPC) 0.80+ 90% ~$0.006 Blind, the frontier model ties our untouched base and zeros 3/10 on critical gates. The verifier is what closes that gap — knowledge of it, not model scale, is the moat. 25–30× cheaper / request $0.006 vs ~$0.16 API ~$30 total compute — this whole project training is $1–12 a cycle 1,000+ rules in real policy stacks too big for any prompt — not for weights
13. WHO BUYS · WHERE IT GOES Anyone who already owns a policy-as-code stack Platform / DevSecOps A PR bot that writes & repairs compliant Terraform in their VPC. Regulated industries Banks, health, gov: topology can't leave the boundary. APIs fail procurement. Security vendors (OEM) Turn their policy engine into their remediation model — white-label. The wedge: their verifier already exists and is already trusted — we make their model meet it. Next: full Checkov (1,000+ policies) as the reward · Kubernetes & CI hardening · semantic rubrics (an LLM judge for what deterministic rules can't express). Rule-authoring by a big model already ships today.
14. TEAM Who's building it NS Nikita Shigarov Founder · Research Engineer ex-Meta · Huawei — built rl-gym end to end NK Nikita Karaev Advisor AI @ Amazon Frontier AI & Robotics, Scout @ a16z speedrun | Ex Meta AI, University of Oxford
15. If the task is verifiable, the verifier is the moat. Any scanner → a self-repairing, continually-updated model that improves from its own traffic — where your data lives. 0.80+ / 90% on real requests flywheel traffic → better model ~$30 total compute github.com/SkyForce/rl-gym · live demo now