Simulation · v0.3 · what makes it learn

Can a dish of neurons learn Pong?

Two copies of the same culture play Pong through their electrodes and face the same serves. They differ in one condition only. DishBrain's feedback alone did not teach our cultures anything. What did: the input about the ball goes quiet while the paddle moves toward it, so acting on the input is what silences it. The control gets the same rule for a mirror image of the ball. If the culture learns, its rallies grow longer than the control's.

DishBrain feedback

Predictable pulse after a hit, noise after a miss.

Playing
Rallies
0
Hits per rally
–
Last 5 min
–
Aces
–
Mean synapse
–
Ball → paddle
–

Control

Same pulses, at random.

Playing
Rallies
0
Hits per rally
–
Last 5 min
–
Aces
–
Mean synapse
–
Ball → paddle
–

Hits per rally

mean over the last 2 minutes of each culture's play

DishBrain feedback Control, random feedback Simulated time 0:00

Seed – · both cultures grow from it, so they start identical.

sensory motor up motor down

How the loop works

  • Sensing. The two middle columns of electrodes are sensory. The row that matches the ball's height is pulsed, from 4 Hz when the ball is far away to 40 Hz when it reaches the paddle.
  • Acting. The outer columns are motor regions. More spikes in the top half move the paddle up, more in the bottom half move it down.
  • Feedback. After a hit, all sensory electrodes pulse at 100 Hz for 100 ms. After a miss, random electrodes fire at random times for 4 s while the ball waits to be served.
  • Going quiet. In the first experiment, the ball input pauses while the paddle heads for the ball. The control pauses for a mirror image of the ball instead, so its quiet carries no information about the game.
  • Plasticity. Synapses change with spike timing (STDP): input that arrives just before a neuron fires gets stronger, input that arrives just after gets weaker.

What we found so far

We ran 16 culture pairs of 2 048 neurons for 20 simulated minutes per condition and compared the first five minutes with the last five. DishBrain's feedback alone did not help: rallies grew by +0.15 ± 0.05 hits, but by +0.07 ± 0.04 with random feedback and +0.04 without any, and with 4 096 neurons the feedback gained nothing at all. Giving the culture a reward signal through eligibility traces did not help either.

Two things did. First, the wiring: with mostly short connections, the ball's position never reached the motor regions, so there was nothing to learn. With connections reaching about half a millimetre, it does (the Ball → paddle number on each culture). Second, letting the input go quiet when the paddle moves toward the ball. Then rallies grew by +0.25 ± 0.10 hits, to 0.97 per rally, while the mirrored control ended at 0.60. With plasticity switched off, neither changed. With 4 096 neurons the gain was +0.75 ± 0.18, in 15 of 16 cultures, to 1.52 hits per rally against 0.61.

Background

DishBrain wired real cortical neurons to Pong and reported that rallies improved within minutes of feedback. The interpretation is debated. One proposal is that cultures learn to act so as to avoid stimulation they cannot predict. Hotaru is a simplified model of that protocol, not of real neurons, and makes no claim about either.

Kagan et al. (2022), In vitro neurons learn and exhibit sentience when embodied in a simulated game-world, Neuron. Izhikevich (2003). Tsodyks & Markram (1997). Sinapayen, Masumori & Ikegami (2017), Learning by stimulation avoidance. Reproduce the numbers with make experiment ARGS="seeds=16 avoidance=1".