Halite Home

ML/AI Strategies to explore?


#1

I figured I'd use this challenge as a learning opportunity and explore the world of ML a bit. My current bot uses a heuristic approach, effective just minor improvements on the "Improving the Random Bot" with a few more if statements to handle specific scenarios. I suspect that approach has limits especially as I get into implementing "find the closest optimal location maximizing production, while minimizing the strength required to get to it".

Looking for a few pointers on where to start, which Python libraries to consider? Which techniques to explore/learn? I'm familiar with NumPy/Pandas, but haven't gone too far beyond that. Not looking for game-specific strategies, rather for applicable libs/areas to explore.


#2

I too am trying this approach for the first time. Python is very common in the machine-learning field.

  • scikit-learn is the most popular library for general machine learning in Python
  • Keras is basically a high-level wrapper around TensorFlow or Theano and is used for deep-learning problems
  • DEAP is a library with many tools regarding genetic and evoluationary algorithms

Currently I still struggle with evaluating the performance of model vs model, as most machine-learning problems deal only with independent models.

I've got my feet wet in ML with following tutorials from http://machinelearningmastery.com/ and there are many guides to solve challenges on Kaggle, but those are all quite different problems compared to this game.


#3

I think Reinforcement learning is an option here.

That's how i am gonna try anyway. But i'm starting just now so i can't give many pointers sorry.

Edit: Also here is a list of awesome links to know more about RL: RL links


#4

Check out Q-Learning https://en.wikipedia.org/wiki/Q-learning


#6

My bot (currently rank 10) is purely ML using weights trained with keras/theano. Most of my 50-line submission is for processing the input and then there is one single line that runs the network. All of the game logic and strategy is learned and encoded in the weights; there is no ad-hoc handcrafted logic.

If there is interest, I could probably write a tutorial on training an ML halite bot. An ML bot got 3rd in our internal competition; I would love to see an ML bot win the public competition :smiley:


#7

@brianvanleeuwen, yes, please do so !

(you don't have to give away your "secret sauce" but already an explanation, how to formulate the ml problem properly would be extremely helpful)


#8

I would second this, I'd find it very useful as I'm in the same position as the OP


#9

@brianvanleeuwen I've introduced Halite to my High School Computer Science students and they are loving it. We are currently trying to utilize ML in our bots. I am really enjoying using this competition as a teaching tool. So any help that you can provide would be greatly appreciated!


#10

That would be great. We would totally put your tutorial up on the site.


#11

Two questions:

  • What is your network output? There are 5 actions for each square so does your network output 30 * 30 * 5 neurons?
  • Since board sizes can differ, do your neural networks change?

#12

+10,000

I'm taking an ML course right now and I'd love to see even a rough outline of how to do this.


#13

Any update? Will you do a little writeup?


#14

Hi @hmate9

I and especially @truell20 have been chatting with @brianvanleeuwen regarding the potential ML tutorial. He is planning to write one, and while he didn't specify a date I would guess next weekend.

For those interested in this, I would also recommend joining the Discord channel, as there was an earlier conversation on that regarding ML strategies.


#15

Awesome! Could you please send the Discord invitation again? It says it has expired.


#16

This one should work
https://discord.gg/rbVDB4n


#17

Sorry, we moved to a new apartment last week and I don't have my PC set up yet. I'll write up a tutorial next weekend.

My current bot generates the move for each tile independently.

You could upload multiple networks and/or sets of weights and select which to use during initialization, but I don't do this currently. For example, you could have the bot look at the game map during initialization and decide to load from BigMapHighProdLowStrWeights.h5 or something. I suspect that the advantage of this would be minimal, probably not worth the increase in complexity.


#18

What do you do instead? Is your network's window just 20x20? Is it larger and do you just pad the outer sites with 0 strength and 0 production for smaller maps?


Welcome to Halite Forums
#19

I use a 35x35 area centered around the tile. There is a numpy function that pads by wrapping periodically.


#20

I just ran a first baseline using a purely machine learning based approach. It took a while to get everything set up, prepare data, etc., but the movement patterns are quite interesting and not unreasonable for a first run.


#21

Here's my first bot playing against itself. No hand-coded rules; purely machine trained using downloaded replays. Lots of room for improvement.277234-3156807350.hlt (2.4 MB)

Let it train a little longer and it got a little more interesting. Starting to see some tunneling patterns, which is promising :smiley: 277930-3852875703.hlt (2.3 MB)

Things are getting stranger... 286277-3609305836.hlt (2.3 MB)