Thibaut Platet

augmented reality · project lead, team of 3 · ENSIM

TerraPolis

–

TerraPolis visual: a tablet in augmented reality held above a grid map, and a diagram linking projector, computer and tablet.

At a glance

Dates
–
Context
augmented reality · project lead, team of 3 · ENSIM
Technologies
  • AR
  • Deep Reinforcement Learning
  • Unity
  • Python

Overview

Environmental and anthropological simulation in AR, driven by Deep Reinforcement Learning (CNN).

Problem

Show the consequences of a planning choice (pollution, floods) before anything is built: a sustainable city simulation designed for urban planning studies.

Role

Project lead, team of 3.

Decisions

An AI trained with Deep Reinforcement Learning (CNN), learning in stages; a Python simulation engine linked over UDP to a Unity app in augmented reality.

Result

Working prototype, tried by visitors at the ENSIM open day. Over 1,000 games, the AI scores on average 4.75 times higher than a random strategy.

My contribution

  • Most of the artificial intelligence: problem modelling, neural network, training strategy, tuning and evaluation.
  • The larger share of the state of the art: mixed reality and AI for disaster prevention.
  • A part in designing the rules, in the Python simulation engine and in project management.

Specification

RequirementAnswer
A city to buildA fixed map of 15 × 10 tiles (mountain, forest, plain, river), two resources (wood, stone) and six buildings: sawmill, quarry, housing, coal power plant, wind turbine, nuclear power plant.
A score, with no win or lossFinal score = virtue of what is built − pollution (at construction, then over time) − flood damage. A game lasts 15 minutes.
Two hazardsRandom floods (up to two per game) destroy the buildings next to rivers; industrial pollution contaminates the tiles next to sawmills, quarries and coal plants.
Advice on demandAn “AI advice” button suggests the best action given the resources and the free tiles: build, demolish or wait.
Augmented realityThe map appears on a table after scanning a QR code; buildings are placed on it in 3D, with a preview before confirming.
Two devices in syncThe computer computes (rules and AI in Python), the tablet displays (Unity app); they talk over UDP on the local network, in JSON.

How it works

  1. Tablet or phone Unity app (AR Foundation): augmented reality view, 2D view, building menu.
    2D view of the mobile app: the grid of tiles and the menu to choose among six buildings.
    On the tablet: the 2D view and the menu of the six buildings to place (interface in French).
  2. Local network UDP, JSON messages: low latency, map and actions kept in sync.
    A tablet held above the map projected on a table: its screen shows the same map in augmented reality, with 3D buildings.
    The tablet and the projected map show the same state, kept in sync over the network.
  3. Computer Simulation engine and AI in Python: rules, scores, advice.
    The AI’s dashboard on the computer: the city map at turn 54 of 60, the score, the moves ranked by value and the log of actions.
    On the computer: the map, the moves the AI evaluates at each turn and the log of its actions (interface in French). View full size

The artificial intelligence

The game grid is handled like an image: a convolutional neural network (CNN) learns, through reinforcement, which action pays off most over a whole game.

  • Inputs: the state of the land (one matrix per type of tile and building) and the available resources. Output: the estimated value of each action.
  • Three convolution layers (32, 64 then 128 filters, 3 × 3) that learn neighbourhoods: riverside, nearby forest, map corner…
  • One game = 60 decisions, one every 15 seconds. Exploration, high at first, decreases with every episode in favour of the best moves.

The turning point: learning in stages. Given all the rules at once, the AI mostly learned to build nothing so as not to lose points. Training was therefore split into three phases (curriculum learning):

  1. Unlimited resources, pollution ignored, a reward for building: learning the mechanics.
  2. Pollution back, resources still unlimited: learning that a bad location is costly in the long run.
  3. Resources reset to zero and decreasing exploration: exploiting the best behaviours.
Diagram of the training loop, labelled in French: state initialisation, choice of the best action, sum of the scores after 60 actions, loss, model update.
One episode: state, choice of the best action, score, then model update (diagram in French). View full size
On video (2 min): the AI plays a full game. At each turn it simulates the possible moves, has the network score each future state and plays the best one. This is its best game out of 1,520 simulated: 205,063 points (interface in French).

Results

average score of the AI against a random strategy, over 1,000 games
× 4.75
average score of the AI (12,940 at random)
61,428
the AI’s record: its best game out of 1,520 simulated (best random score: 54,685)
205,063
episodes in the final training run
1,500

An average game by the AI (61,428) beats the best game obtained at random (54,685). The working prototype was tried by visitors at the ENSIM open day.

Scores: random strategy and trained AI
  • Random strategy
  • Trained AI
ScoresRandom strategyTrained AI
Worst score−20,174−3,639
Average score12,94061,428
Best score54,685205,063

Worst and average scores: 1,000 test games. Best score of the AI: its best game out of 1,520 simulated.

TensorBoard curves of the average score over 1,500 episodes: three training configurations rise, the best one to about 100,000, while the random agent stays flat around 12,000.
Average score (over 100 episodes) during the final training run of 1,500 episodes, tracked with TensorBoard. In green, the configuration that was kept (with a dropout of 0.3) ends around 100,600; at the bottom, the random agent stays around 12,400. View full size