Learning Associative Reasoning Towards Systematicity Using Modular Networks

Abstract

Learning associative reasoning is necessary to implement human-level artificial intelligence even when a model faces unfamiliar associations of learned components. However, conventional memory augmented neural networks (MANNs) have shown degraded performance on systematically different data since they lack consideration of systematic generalization. In this work, we propose a novel architecture for MANNs which explicitly aims to learn recomposable representations with a modular structure of RNNs. Our method binds learned representations with a Tensor Product Representation (TPR) to manifest their associations and stores the associations into TPR-based external memory. In addition, to demonstrate the effectiveness of our approach, we introduce a new benchmark for evaluating systematic generalization performance on associative reasoning, which contains systematically different combinations of words between training and test data. From the experimental results, our method shows superior test accuracy on systematically different data compared to other models. Furthermore, we validate the models using TPR by analyzing whether the learned representations have symbolic properties.

Overview

We propose a novel architecture combining a modular encoder with TPR-based external memory to achieve systematic generalization in associative reasoning, where conventional MANNs fail on systematically different test data.

1. Modular encoding — Recurrent Independent Mechanisms (RIMs) encode input through \(N\) independent modules via competitive learning, producing recomposable representations.
2. TPR binding — Tensor Product Representation mathematically binds role-filler associations: \(T = \sum_{k=1}^N \mathbf{r}_k \otimes \mathbf{f}_k\)
3. Memory-based recall — Associations are stored in TPR-based external memory, enabling systematic reasoning over unseen combinations.

Method

Conventional memory augmented neural networks (MANNs) suffer from severe performance degradation on systematically different test data. We combine a modular RNN encoder + TPR-based external memory to achieve systematic generalization.

Key components:

• Recurrent Independent Mechanisms (RIMs): \(N\) RNN modules learn independent encoding mechanisms via competitive learning
• Tensor Product Representation (TPR): Associations are mathematically bound via tensor products of roles and fillers — \(T = \sum_{k=1}^N \mathbf{r}_k \otimes \mathbf{f}_k\)
• TPR-based External Memory: Role/filler representations are extracted at each time step and superposed into memory
• Systematic Associative Recall (SAR): A new benchmark proposed for evaluating systematic generalization in associative reasoning

Results

Quantitative

Training/test accuracy comparison of DNC, FWM, and our proposed method on the SAR task. While DNC and FWM show significant performance degradation on systematically different test data, our model successfully achieves systematic generalization.

On the large-scale question answering task (catbAbI), our method achieves performance on par with FWM, confirming the general effectiveness of the modular encoder.

Analysis

We verify whether the learned representations possess correct symbolic properties. Analyzing the similarity between role vectors and unbinding vectors shows that FWM fails to achieve orthogonality, while our method exhibits near-perfect orthogonality.

(a) FWM

(b) Ours

Similarity matrices between role vectors and unbinding vectors. FWM fails to achieve orthogonality, while our method shows near-perfect orthogonality, confirming that it has learned proper symbolic representations.

Analyzing the consistency of read vectors for the same target object shows that our method produces nearly identical outputs regardless of the combination.

(a) FWM

(b) Ours

Similarity between read vectors for the same target object. In our method, the read outputs are nearly identical regardless of the combination, demonstrating systematic associative reasoning.

BibTeX

@inproceedings{bae2022learning,
  author    = {Bae, Jun-Hyun and Park, Taewon and Lee, Minho},
  title     = {Learning Associative Reasoning Towards Systematicity Using Modular Networks},
  booktitle = {International Conference on Neural Information Processing (ICONIP)},
  year      = {2022},
  publisher = {Springer},
  doi       = {10.1007/978-3-031-30108-7_10}
}