We present the first practical method towards making Geoffrey Hinton's GLOM work, and showcase the feasibility of encoding part-whole relationships in a neural net via global supervision.

In this work, we propose Asynchronous Perception Machine (APM), a computationally-efficient architecture for test-time-training (TTT). APM can process patches of an image one at a time in any order asymmetrically, and still encode semantic-awareness in the net. We demonstrate APM’s ability to recognize out-of-distribution images without dataset-specific pre-training, augmentation or any-pretext task. APM offers competitive performance over existing TTT approaches. To perform TTT, APM just distills test sample’s representation once. APM possesses a unique property: it can learn using just this single representation and starts predicting semantically-aware features.

APM demostrates potential applications beyond test-time-training: APM can scale up to a dataset of 2D images and yield semantic-clusterings in a single forward pass. APM also provides first empirical evidence towards validating GLOM’s insight, i.e. if input percept is a field. Therefore, APM helps us converge towards an implementation which can do both interpolation and perception on a sharedconnectionist hardware. Our code is publicly available at this link.

APM builds upon a new representation called Hinton's Islands of Agreement. The key idea is that at each location of the input image there is a high dimensional column vector. Clustering these column vectors yields islands of agreement, aka, regions of different colours which represent different parts of an object. Note that the above visualizations were obtained WITHOUT using ANY semantic-labels. Grouping happens as a part of bottom-up recognition only in a transformer like Mvitv2. Bounding-Box supervision is NOT needed.

APM proposes two technical ideas. 1) The first idea is the proposed column representation T 2) The second idea is the folding-unfolding mechanism. However, there are several deeper non-technical/non- scientific inspirations which motivated the design of APM. We discuss some of those, to help facilitate a deeper-connection and ground our intuitions.

A biological analogy : Consider how an organism starts its existence from a cell. The cell is copied across different body locations. Each location possesses identical DNA. However, depending on the location, the DNA decides whether to form an eye or nose. We term this process as unfolding, i.e. a cell ‘expands’ to yield an organism. Next, there is evidence of jellyfish like Turritopsis dohrnii reverting from their fully grown form to younger polyp states. We term this process as folding, i.e. cells of an organism collapse back to the single cell it began from.

A computational analogy: We now start treating an image I as a digital organism. It starts from some compressed representation T . T unfolds to yield the image I. I then folds back to yield the compressed representation T . Learning proceeds by oscillating between these unfolded and folded phases. At every step, the net is trying to reconstruct image I from T . T is then expected to be a dense vector-space

A cosmological analogy : In physics, one of the famous theories of the origin of universe has been starting from a single point, and undergoing a continuous expansion[ 50 ]. There are alternate theories for eg, Conformal Cyclic Cosmology[75 ] which hypothesize the universe undergoing periodic cycles of expansion and contraction[ 79]. Drawing inspiration from these fundamental insights, the trigger column T undergoes these cycles of folding and unfolding during the learning iterations.

A cellular-automaton analogy: On surface it seems a pretty trivial matter to discuss: a point can expand and yield beautiful patterns which can either be an entire universe in accordance with the theory of big-bang, or can be reproduction of an organism from a singular zygote. But, it is funny: if you start from a point, and unfold it, then all you can get is a sphere. This appears to be true for the behaviour of light, in accordance to Huygens principle11. However, we observe non-spherical objects around us all the time. Turing posited that the symmetry breaking in the sphere must happen somewhere while the organism unfolds: such patterns could then be explained a variety of diffusion based equations.This idea has been explored in cellular automaton: different replication rules of starting point can yield different final patterns. Scientists then continue to derive different rules which yield different patterns, which is akin to how we were resorting to hand-engineering features in deep-learning for a long time. APM attempts to answer the question: Is it possible to build a learning machine which can start from a point, unfold, and then express correct features at the correct place?. We want to then push the job of rule-learning to what backpropogation does best. We have lost the "why" for the knowledge was encoded in the weights of the net, but we seem to have gained the ability of correct features presenting themselves at correct locations. This location-aware-disentanglement procedure thereby represents a step towards solving Arnold’s superposition theoram and Hilbert’s thirteenth problem. However, backpropogation can only approximate solutions and not yet reach exact ones, and mathematical formulations are lost into the weights of the neural-net.

We then begin to imagine learning machines which can solve a complex problem like cryptography/breaking-a-cipher in two phases 1) relax the system towards an approximate solution 2) have the system spit-out which parts of the solution are uncertain, and brute-force towards the remaining solution. Or, we could make the loss of the learning-machine reach perfectly zero, thereby representing a perfect solution13. Hard problems like recognizing faces are approximately solved as a consequence of a single forward pass through a learning machine. If the loss could be made to reach zero, then we could consider the problem to be perfectly solved. Solvability can then happen in a feed-forward phase, which for practical purposes appears to be polynomial. Next, we redirect the reader’s attention to neumann’s theory of self-reproducing automaton. His idea of a self-replicating colony was that there is an infinite source of resources a.ka. reservoir which is shared by the automaton which operate at different locations of a colony. The colony uses up the shared resources, does self-replication and in this way converts raw materials/matter into useful intelligent-behaviour. The infinite reservoir of machines he talks about then reduces to the trigger column T in APM: since features are sampled from a same space, they automatically become aware of themselves, thereby making explicit attention unnecessary. This also then is same as how latents have been classically sampled in the generative models. One might argue that multiple automaton although starting their lives at the same point will need to communicate among themselves, as they differ among their configurations at later point in their lifespans. Fluctuations in T are then akin to mutations. We compensate for this fact by weight-sharing the MLP across different locations in APM.