Halite Home

Building a good ML Bot


#1

Introduction

Hey, so after seeing other people so openly willing to share knowledge, I figured I should go ahead and share some of my ML strategies. My Bot is sitting around 9-11 right now, and I have another one that is also doing ok. I started from the Halite-ML-starter-bot . I'm just going to go over some important topics that should be useful to other people when thinking about how to make their ML bot better! Be warned, this article assumes a fair bit of knowledge about deep learning, so anything that's unclear, please post a question and I will definitely make sure to respond.

1. Optimizer

This is my far the most underrated topic. People just put their favourite in-built default optimizer on the job and then assume that they are all equivalent. They aren't. Try different ones. Here are some general guidelines for Halite:
- Momentum isn't always the best because it tends to overshoot the minima. After moving from Nadam to RMSProp I went up from 77% accuracy to around 81%.
- Look at what's happening when it's training. Is it going up then down? You probably have a large step size/learning rate. Is it barely moving? It's probably too small. Does it start off well and then taper off? Maybe the decay is to steep etc.
Here is a good resource that goes over all the optimizers.

2. Less is More:

Everyone thinks that get as much training data as possible, but it's important to maintain consistently. If I feed it data from acoutte and erdman, they have very different strategies. So it would get confused. There's a funny example. If you're trying to teach a model to avoid a tree. Then giving it examples of going left and going right, then going into the tree minimizes loss! So make sure you give consistent data. When I trained it on 3k samples it broke top 40. Then I trained it on 175 samples of erdman v11 only, and it broke top 16.

3. Structure:

So there are a couple challenges we face when designing a good deep learning architecture. Changing Map sizes, varying number of players, symmetry etc. The current way people have tried to do this is by saying that for each square you own, the move is calculated independantly. So you take a nxn square around a square you own and ask an NN to predict the move it should make. In the starter bot it's 3 fully connected layers that are each of size 512 then one final that outputs a softmax over 5 values. This is ok, but upon closer inspection, this is actually equivalent to running a convolution network of the shape nxnx4x512->1x1x512x512->1x1x512x512->1x1x512x5 then evaluating the board for positions you own. And this is weird, since nobody actually runs a bunch of 1x1 convolutions for a task like this. So instead I run a 3x3x4x200->3x3x200x150->3x3x150x100->3x3x100x100->11x11x100x5. This improved accuracy from about 72->77. This bot finished 10th, altho I submitted it again and it finished 13th now.

4. Symmetry:

Another important thing to realize is 170 games can turn into 1360 games by adding 4 rotations, then a flip and then 4 rotations for every training example. This remains consistent and helps avoid games where your bot has never seen this before and so it stay's still for the whole game.

5. Experimental Process:

Make sure to understand all of the ML starter bot's code. There's a lot going on. And make sure to evaluate every change by keeping everything constant and adding the change. Because sometimes you will be surprised at what happens when you make many changes together and no idea why.

Good Luck!
Happy Machine Learning!


#2

Do you believe Deep Learning is the correct direction for learning or have you also tried 'older' methods such as penalized regression?


#3

I can't entirely speak to @KalraA's process (since I'm not him) but I've chatted a lot with him on the Discord channel (incidentally, if you're interested, there's a machine learning thread on the channel) and my understanding is that he's been mostly working in the realm of supervised deep learning with CNNs and occasionally a touch of RL?


#4

That makes sense, thanks!


#5

Accuracy can be a misleading metric. A bot that just picks STILL every turn achieves a similar level of accuracy, depending on what samples you are training on.


#6

I get around 81% accuracy for actual moves and 90% accuracy for STILLs.


#7

So I have it print the number of each class before training, and I always make sure it's close to even! But good observation.