On Artificial Karamuto Oscillatory Neurons

Recently, I had the pleasure of attending a talk by Takeru Miyato, on his ICLR 2025 oral paper titled `Artificial Karamuto Oscillatory Neurons’. The idea seems pretty interesting, and authors have done a great job explaining their paper in great detail here. Obviously, i cannot do justice to their amazing work, but i will merely try to distill some of their key points. First, we cover what is known in neuroscience has the binding problem.

Binding Problem

Around 60 years ago, people got interested in how we can look at an image, and automatically cluster different objects together. For eg, if the image contains a cup, glass, then we can automatically group them into corresponding regions.

Suppose you are given a blue cup, and a red cup. Somehow, the brain is able to combine the representation of “blue” with “cup”, and “red” with “cup”, to yield two different representations for blue cup/red cup respectively. There are two plausible arguments here. (i) brain has learnt separate representation for blue cup/red cup. Given such an input, it just “selects” the correct representation out of a dictionary of objects. (ii) brain `dynamically’ combined blue and cup together to form blue-cup on the fly.

A criticism of (i) is that there are infinite objects, and their combinations available, which means we don’t learn a massive superset. Rather it seems that we rely on (ii): learn only a few abstract concepts like colors, objects, and combine them whenever needed. This second problem is known as the binding problem.

The key issue is : 1) how does this combination actually happen? 2) given an image containing both blue-cup, red-cup, how do we `simultaneously’ maintain two-different representations at the same time in our heads?

The hope is that by figuring out how these different attributes combine during inference, we can get better generalization on objects we have never seen during training. Classical AI will just train model on separate samples of blue cup, red-cup, and hope it figures it out. But, that didn’t work quite well.