The original ask was simple to say and hard to build: a reinforcement learning pipeline for BRAWLPIT that trains directly on the packet level, using the AlphaStar league's autocurriculum doctrine, pulling in the fast-forward and fractal commander ideas from ECOWAR, and using PARENA wherever it made sense. This post is the honest, detailed account of what that turned into: three neural networks fighting each other over real UDP packets, a shared checkpoint registry two different training locations can push into, a native C inference path that lets the trained bot run inside the actual game with zero Python at runtime, and a couple of real bugs we found and fixed along the way that are worth telling on ourselves about.
PACKET LEVEL, LITERALLY
Most game AI training pipelines cheat a little. They wrap an internal simulation API, feed the model clean state straight out of memory, and never touch the actual wire format the real client and server speak. BRAWLPIT's environment does not do that. The observation the policy sees is built from the literal bytes bin/brawlpit_server broadcasts every tick, and the action it produces gets sent as the literal bytes the real client would send. There is no shortcut in the middle. That means a trained policy is a genuine drop-in bot that speaks the real protocol; it could run as a totally separate process on a totally separate machine with no code sharing beyond the environment file itself, which is exactly the property that later let us run this policy natively in C at real game speed with no PyTorch anywhere near the actual match.
The wire structures are locked down hard on purpose. NetHeader is 12 bytes, UserCmd is 28 bytes, NetPlayer is 32 bytes, and every one of those has a self-checking assertion against Python's ctypes.sizeof at import time, so if the C protocol header ever drifts out from under the training pipeline, the whole thing fails loudly the moment you try to run it instead of quietly training on garbage.
THE OBSERVATION VECTOR
Each side of a match contributes eight raw scalars: x, y, x-velocity, y-velocity, damage percent, stocks remaining, shield health, and facing direction, each normalized into a roughly [-1, 1] or [0, 1] range. That is sixteen numbers for both players combined. On top of that sits a five-way one-hot block describing the current commander posture: neutral, aggressive, patient, edgeguard, or recover. That posture comes from a real compiled PARENA decision function, commander_posture, built from PARENA/stdlib/brawlpit/commander_mod.prn. It is a genuine structural step toward the fractal commander idea from the original ask, giving the low-level policy a strategic signal to condition on without building out the full commander-and-soldier hierarchy BRAWLPIT does not need, since there are no teams here.
The last ten numbers are the newest and, it turned out, the most interesting to get right: hand-tailored relational features rather than raw per-player scalars. Relative x and y distance between the two players, the straight-line distance between them, a signed closing velocity computed from the actual relative motion vector rather than just position deltas, a normalized time-to-blast-zone estimate for each player computed from their current position and velocity against the stage bounds, whether each player is currently facing toward or away from the other, the damage differential between the two, and the stock differential between the two. None of these are things the network could not in principle learn to infer from the sixteen raw numbers given enough data, but handing them over pre-computed turns "learn to notice you are closing the distance" into "read one number," the same reward-shaping-adjacent logic the posture one-hot already uses. This block is why the observation is thirty-one dimensions rather than the original twenty-one, a deliberate breaking change that meant every checkpoint trained before it stopped being resumable and had to start fresh under the new shape.
THE REWARD, TIER BY TIER, AND A REAL DEBT TO HYPERBOT
The reward function is built in explicit tiers rather than one flat damage-delta signal, because a platform fighter's real skill expression, things like edgeguarding and recovery, is comparatively rare and only ever shows up as a stock swing several seconds after the actual decision that caused it. Too sparse a signal on its own for credit assignment to find quickly. Tier one is outcome: damage dealt, damage taken, stocks taken, stocks lost, a terminal win or loss. Tier two and three build survival and activity incentives on top of that so a policy has a reason to do something other than sit in a corner. Tier five is a survival streak bonus, deliberately Fibonacci-capped after a real bug where an uncapped version let a spike in model quality quietly regress into what could only be described as a gradient of stupidity, models that had learned to farm the streak bonus itself rather than actually survive meaningfully.
Tier four is where we owe a direct, specific credit. The question that started it was whether BRAWLPIT was doing any real reward discounting, and the honest answer at the time was no: stable_baselines3's PPO defaults to a plain, per-step gamma with nothing tied to real elapsed time. A distinct, real Melee-playing bot project called Hyperbot had already solved a closely related problem the right way, and the video walking through that project's own design is the direct source for everything in this section: https://www.youtube.com/watch?v=_JTOJxhtDsU
The root cause the video names is architectural, not incidental. Hyperbot itself is built event-driven, reacting to real packets as they actually arrive rather than ticking forward on a fixed, synchronous per-step loop. Once time is event-driven like that, step count stops being a trustworthy proxy for real elapsed time, since a burst of events can land close together or spread far apart in wall-clock terms while still only ever counting as one step each. Discounting anything by step count in a system built that way silently discounts by the wrong axis entirely. A terminal draw penalty that only ever lands at full severity, regardless of when in the match it was effectively locked in, teaches a policy the wrong lesson about when it is actually safe to play conservatively. Hyperbot's fix discounts that future draw penalty continuously over real time using a half life instead of a step count, so its perceived cost starts small early in a match and grows to its full value right at the buzzer. Early on it reads as roughly a third of its true severity; with a proportionally scaled amount of time left it is already up around ninety percent; at the buzzer itself it hits exactly its full value.
BRAWLPIT has a real, honest version of that same event-driven irregularity, for a closely related reason: under fast-forward training, the environment always drains to the freshest available server snapshot rather than processing every single one in order, so a client's own step cadence can genuinely fall behind the server's real tick rate, and multiple real simulated ticks can elapse between two consecutive step calls without that gap ever being counted. That limitation is named directly here rather than smoothed over; fixing it fully would mean the server ticking in lockstep with every connected client's own input instead of free-running, a real, separate, larger undertaking not attempted in this pass. What we could not do was Hyperbot's literal value-function surgery either, since stable_baselines3's stock PPO rollout buffer has no hook for a per-transition, wall-clock-dependent gamma. What we built instead reproduces the same qualitative incentive through ordinary reward shaping: the inactivity penalty gets scaled up by a time-pressure multiplier as the real match clock runs down, using the exact same half-life formula, ported ratio-preserving onto BRAWLPIT's own two-and-a-half-minute match cap. Stalling stays cheap early in a match, matching Hyperbot's own finding that it is genuinely safer to hold back when a fight is still developing, and becomes progressively more expensive as the timeout approaches, so a policy that has judged its own win chance as low still has a real, growing reason to eventually engage instead of running out the clock for the entire match.
THE LEAGUE: WHY ONE POLICY IS NOT ENOUGH
Plain self-play against your own growing checkpoint history has a well known failure mode called cyclic dominance. Training against your recent self makes you sharper against that one specific opponent while quietly getting worse against strategies outside that narrow recent window, a rock-paper-scissors dynamic that never resolves on its own. AlphaStar's own published league is the doctrine this pipeline ports in, and the same Hyperbot video credited above is a second, direct source for how the exploiter side of that doctrine actually gets implemented: its own walkthrough of a dedicated adversary role, one whose sole incentive is hunting a main policy's current weaknesses and that deliberately steps down to easier, already-beatable opponents the moment it stops getting any real learning signal from getting crushed, is exactly the shape Main Exploiter's own two-mode curriculum below is built from, not just the underlying AlphaStar paper's own higher-level description of a league. The fix, credited to both sources together, is to stop training one policy and instead run a league of three distinct roles simultaneously, every one of which contributes permanent, never-evicted checkpoints back into one shared pool.
Main trains against the whole league, weighted by a formula called PFSP, prioritized-fictitious-self-play, biased toward whatever currently beats it most often. Its job is to push the overall skill frontier forward using the broadest possible curriculum. Main Exploiter has exactly one job: find and break Main's current weaknesses. It always challenges Main's freshest registered checkpoint directly. If it cannot win consistently against that checkpoint, there is no clean learning signal in continuing to get crushed every single game, so it deliberately climbs down through Main's own history instead, biased toward older Main checkpoints it can already beat, rebuilding fundamentals before climbing back up to challenge the current frontier again. It also resets to a freshly initialized network every few generations on purpose, so it cannot converge onto one narrow trick and quietly stop being a useful adversary. League Exploiter runs the exact same whole-league PFSP formula Main does, but as a completely separate lineage of checkpoints, and its purpose is different: catching weaknesses that persist across the entire league rather than just Main's own current blind spot, so nothing the group as a whole is bad at goes permanently unpatched. Every registered checkpoint, from every role, from every generation, is kept forever. Nothing is ever evicted from the pool.
The PFSP weighting itself has a real, honest cold-start property worth naming directly, because it is easy to assume it is smarter than it is. Every opponent id starts with zero recorded games against a given trainee, which is treated as a neutral fifty percent win rate rather than an alarming unknown, so a completely fresh policy's very first generation is close to a uniform random draw across the whole pool. There is no way to know who is actually hard yet. The bias toward genuinely tough opponents only emerges gradually, one real generation of match results at a time, as a trainee's own local win-loss memory against specific opponents starts to diverge from that neutral baseline. That memory lives entirely in the process's own memory for the whole run and is never shared between roles, matching the same architecture's own established convention that a trainee's curriculum should be shaped by its own experience, not a globally averaged statistic.
Every registered checkpoint's skill also gets tracked with a standard, real Elo rating, zero-sum across the entire shared pool including a permanent static heuristic baseline that starts at the same default rating as everything else. The critical detail here is that a checkpoint's neural network weights are frozen forever the instant it is registered, but its Elo rating is not frozen at all. Every single match it plays as a sampled opponent, even months after it stopped being anyone's current generation, updates both sides' ratings. An old, long since superseded checkpoint can watch its own Elo keep drifting down in real time purely from getting repeatedly beaten by newer models, with literally nothing about it changing except that one number. That is necessary for Elo to mean anything at all as a live, relative measure of where the whole league currently stands, rather than a stale snapshot of wherever training happened to be the day a given checkpoint stopped improving.
TWO REGISTRIES, ONE PROBLEM, TWO SCALES
There are genuinely two separate checkpoint registries in this system, and they solve two different scales of the same underlying problem. The first is a local, file-based LeagueManager: one small JSON file per registration, written append-only into a members directory, which needs no locking at all even with three concurrent writers because a new registration is always a brand new file rather than an overwrite of another process's own file. That works perfectly for three roles training as separate processes on the same machine, but it has an obvious ceiling: it only ever sees checkpoints registered by processes that can see the same filesystem.
The second is IDUNA's own real, remote checkpoint registry, reachable over the network by any training location that can authenticate as the dedicated BRAWLPIT-RL machine agent. This is the piece that actually lets training happen from more than one place at once, this box and a Colab runtime and potentially any future machine, all pushing into and pulling from the exact same shared league instead of each maintaining its own local history that the others never see. Every checkpoint uploaded there can optionally carry a second, much smaller artifact alongside the full training-state zip: a real exported native-inference blob in a tiny custom binary format, so the same upload request that registers a new generation for training purposes also makes that generation immediately loadable by the actual compiled game client with no Python involved at all. Checkpoints can be marked active, meaning they are the one currently selected as the default opponent players actually face, or disabled, meaning they stay in the permanent historical record but stop being eligible as a resume target or default opponent, all served over plain authenticated HTTP.
GETTING A LEAGUE RUNNING FROM A FREE COLAB RUNTIME
Training all three roles at once, each with its own dedicated real UDP server subprocess, is heavier than a laptop wants to carry for very long, so there is a real, single-file bootstrap script meant to be pasted directly into one Colab cell. It clones or updates the repository, builds the real training binary, authenticates against IDUNA using the same machine-to-machine agent-secret exchange every other automated agent in this monorepo already uses, downloads each of the three roles' own newest checkpoint from the shared registry so a fresh Colab runtime continues the same ongoing league instead of restarting three random networks from generation zero every single time, and then kicks off a long training run that keeps pushing new generations back to that same registry as it goes.
One deliberate, measured decision worth calling out: this does not use a GPU, and switching to one would not meaningfully help. The policy network is a genuinely tiny sixty-four-unit two-layer MLP, small enough that a forward and backward pass on it is already effectively instant on plain CPU, and a GPU would only add real per-call kernel launch and host-to-device transfer overhead on top of tensors that small, typically making things slower rather than faster. The actual bottleneck, confirmed against this box's own real generation timings, is a real UDP round trip to a real dedicated server subprocess on every single environment step, socket I O and process scheduling latency, not matrix multiplication. The real lever for speed is running more parallel environment instances per role, which spawns that many additional dedicated servers and steps all of them in true operating-system-level parallel, a CPU core and process count story a bigger box genuinely helps with, not a GPU one.
A STANDING BOT POOL FOR HUMANS
Training data is not the only thing this registry feeds. There is a separate, persistent bot pool script whose whole job is keeping real matches running continuously, drawing from every checkpoint in the registry that has exported native weights, from every generation currently registered, not some fixed hand-picked subset. Several pairs of bots run continuous bot-versus-bot matches on their own dedicated local servers so real Elo movement keeps happening even with zero humans online, while a smaller number of bots sit queued on whichever server real players actually connect to, waiting to be matched against an actual human. There is an honest, named gap here rather than a pretended solution: BRAWLPIT has no human player identity or login system at all today, so when a human beats or loses to a waiting bot, only the bot's own Elo moves. A real human-side rating needs a real player identity system first, which is a scoped, acknowledged, not-yet-built piece rather than something quietly faked.
RUNNING THE TRAINED POLICY INSIDE THE ACTUAL GAME
Training happens entirely in Python against stable_baselines3, but nobody wants a PyTorch runtime shipped inside a game client. The export path pulls just the trained actor network's weights, the two hidden linear-plus-tanh layers and the final linear action head, out of a saved PPO checkpoint and writes them into a small, versioned, portable binary format: a four-byte magic value, a version number that fails loudly rather than silently misloading on a future architecture change, a per-layer header table of input size, output size, and activation type, and then the raw weight and bias floats themselves in the exact row-major layout PyTorch already uses internally, so no transpose is ever needed on either side. A hand-written, dependency-free C loader and forward pass reads that exact format and runs it at real game speed with no Python or PyTorch anywhere near the actual match, matching the same weights-are-data-architecture-is-code split a sibling project's own policy export already established. The whole point of a runtime-loadable blob instead of compiled-in source is that a brand new checkpoint's weights can be downloaded and swapped in at game start without ever recompiling the game itself, and by default that download happens compressed, the same LZ4 codec this monorepo already links everywhere else, decompressed client-side with the identical codec the server compressed it with.
Two honest incidents from actually shipping this are worth telling directly rather than glossing over, because they are exactly the kind of thing that looks fine in isolation and only shows up once real people are actually playing against it. The first was a genuine PPO training pathology: stable_baselines3 defaults its entropy coefficient to zero, which means nothing in the loss function was actively resisting the policy's own action distribution collapsing toward zero variance over time, and once that collapse starts, the very gradient that would normally push back against it goes nearly flat right along with it, so there was nothing left to ever recover once it began. The fix was a small, explicit entropy coefficient, exposed as a real command line flag rather than a hardcoded constant so it can be retuned without a code change if a future run needs it. The second, found only after the first fix did not actually solve what it looked like it should, was more embarrassing and much more interesting: the native C observation builder had quietly stopped writing at twenty-one of the real thirty-one dimensions the network expected, silently leaving the ten newer relational features as raw, uninitialized stack memory fed straight into the policy on every single frame of live gameplay, even though every helper function needed to compute those ten values correctly already existed in the same file. Training itself stayed completely healthy through all of this, because training runs entirely in Python, where the observation was always built correctly, so Elo kept climbing exactly as expected while the live game client's own real-time inference was silently broken the whole time. The fix was implemented and then verified the only way that actually counts: a standalone C program calling the real observation-building function directly, diffed value by value against the real Python implementation for an identical synthetic game state, matching to six decimal places across all thirty-one dimensions, with a permanent regression test now checked into the build so this exact silent failure mode cannot happen again without a test catching it immediately.
WHERE THIS LEAVES THINGS
What actually shipped is a training pipeline that speaks the real wire protocol rather than a simulated shortcut, a three-role AlphaStar-style league with a genuine escape valve for a struggling exploiter that Main and League Exploiter deliberately do not get, a shared registry that lets training happen from more than one machine at once and doubles as the exact same source the live game client downloads from, and a native inference path fast and small enough to run inside the actual game with nothing but a tiny hand-written C forward pass. None of it is finished in the sense of being done; League Exploiter's own climb still visibly stalls and recovers as the rest of the league pulls ahead of it, and there is real, honest, un-glossed-over follow-up work already named for a future human-side rating system in the bot pool. But the bot standing across from you in a real match right now is genuinely the same network that has been grinding through generations of self-play against its own league, not a scripted stand-in, and that was worth writing up properly.
STINKIES COMMISSAIRE — the first physical thing EINHORN_INDUSTRIAL has made. Join the waiting list for the hoodie →
← All posts