ICONIP 2022
Jun-Hyun Bae*, Taewon Park*, Minho Lee
Kyungpook National University
* Equal Contribution
๐Ÿ“„ Paper

Abstract

Learning associative reasoning is necessary to implement human-level artificial intelligence even when a model faces unfamiliar associations of learned components. However, conventional memory augmented neural networks (MANNs) have shown degraded performance on systematically different data since they lack consideration of systematic generalization. In this work, we propose a novel architecture for MANNs which explicitly aims to learn recomposable representations with a modular structure of RNNs. Our method binds learned representations with a Tensor Product Representation (TPR) to manifest their associations and stores the associations into TPR-based external memory. In addition, to demonstrate the effectiveness of our approach, we introduce a new benchmark for evaluating systematic generalization performance on associative reasoning, which contains systematically different combinations of words between training and test data. From the experimental results, our method shows superior test accuracy on systematically different data compared to other models. Furthermore, we validate the models using TPR by analyzing whether the learned representations have symbolic properties.


Overview

์ด ๋…ผ๋ฌธ์—์„œ ๋งŒ๋“  Systematic Associative Recall(SAR)์€ ํ•™์Šต๊ณผ ํ…Œ์ŠคํŠธ์˜ ์—ฐ๊ด€ ๊ด€๊ณ„๋ฅผ ๋‹ค๋ฅด๊ฒŒ ๊ตฌ์„ฑํ•œ๋‹ค. ํ•™์Šต ์ค‘ ์ˆซ์ž์™€๋งŒ ์—ฐ๊ฒฐ๋œ ์‚ฌ๋žŒ ์ด๋ฆ„์„ ํ…Œ์ŠคํŠธ์—์„œ๋Š” ๊ณผ์ผ๊ณผ ์—ฐ๊ฒฐํ•œ๋‹ค. ๋“ฑ์žฅํ•˜๋Š” ๋‹จ์–ด๋Š” ์ด๋ฏธ ํ•™์Šตํ–ˆ์ง€๋งŒ ์กฐํ•ฉ์€ ๋‹ฌ๋ผ์ง€๋Š” ์„ค์ •์ด๋‹ค.

DNC์™€ FWM์€ ํ•™์Šต๊ณผ ๊ฐ™์€ ์กฐํ•ฉ์—์„œ๋Š” ์ •ํ™•๋„๊ฐ€ ๋†’์•˜์ง€๋งŒ, ์กฐํ•ฉ์„ ๋ฐ”๊พธ๋ฉด ์„ฑ๋Šฅ์ด ํฌ๊ฒŒ ๋–จ์–ด์กŒ๋‹ค. ์ œ์•ˆ ๋ชจ๋ธ์€ modular RNN encoder์™€ TPR ๊ธฐ๋ฐ˜ ๋ฉ”๋ชจ๋ฆฌ๋ฅผ ๊ฒฐํ•ฉํ•ด ์ด ๊ฒฉ์ฐจ๋ฅผ ์ค„์˜€๋‹ค. FWM๋„ TPR ๋ฉ”๋ชจ๋ฆฌ๋ฅผ ์‚ฌ์šฉํ•˜๋ฏ€๋กœ, ๋‘ ๋ชจ๋ธ์˜ ๋น„๊ต์—์„œ๋Š” encoder๊ฐ€ ๋งŒ๋“œ๋Š” roleยทfiller ํ‘œํ˜„์„ ํ•จ๊ป˜ ๋ถ„์„ํ–ˆ๋‹ค. ์ œ์•ˆ ๋ชจ๋ธ์€ ๊ฐ™์€ ๊ฐ์ฒด๋ฅผ ๋‹ค๋ฅธ ์กฐํ•ฉ์—์„œ ๊บผ๋‚ผ ๋•Œ ๋” ์ผ๊ด€๋œ ํ‘œํ˜„์„ ๋ณด์˜€๋‹ค.

Modular TPR Architecture

์ „์ฒด ์•„ํ‚คํ…์ฒ˜. ๊ฐ ์‹œ์  $t$๋งˆ๋‹ค modular encoder์˜ hidden state $h_t$์—์„œ role $r_t$์™€ filler $f_t$๋ฅผ ๊ฐ๊ฐ ์ถ”์ถœํ•˜๊ณ , ๋‘ ๋ฒกํ„ฐ์˜ tensor product($\otimes$)๋ฅผ ์™ธ๋ถ€ ๋ฉ”๋ชจ๋ฆฌ $\mathbf{M}_t$์— ๋ˆ„์ ํ•œ๋‹ค. ์งˆ์˜ ์‹œ์—๋Š” ๊ฐ™์€ encoder๊ฐ€ unbinding ๋ฒกํ„ฐ $u_t$๋ฅผ ๋งŒ๋“ค์–ด ๋ฉ”๋ชจ๋ฆฌ์—์„œ ํ•ด๋‹น filler๋ฅผ ๊บผ๋‚ด๊ณ (unbind), ์ด๋ฅผ ์„ ํ˜• ๋ณ€ํ™˜ํ•ด ์ถœ๋ ฅ $o_t$๋ฅผ ์–ป๋Š”๋‹ค.


Method

๋ฉ”๋ชจ๋ฆฌ์— ๊ด€๊ณ„๋ฅผ ์ €์žฅํ•  ์ˆ˜ ์žˆ๋‹ค๊ณ  ํ•ด์„œ ์ƒˆ๋กœ์šด ์กฐํ•ฉ๋„ ์ž˜ ๋‹ค๋ฃจ๋Š” ๊ฒƒ์€ ์•„๋‹ˆ๋‹ค. ๋…ผ๋ฌธ์€ encoder๊ฐ€ ํ•™์Šต์—์„œ ๋ณธ ์กฐํ•ฉ์— ์–ฝํžŒ ํ‘œํ˜„์„ ๋งŒ๋“ค ์ˆ˜ ์žˆ๋‹ค๋Š” ์ ์— ์ฃผ๋ชฉํ•œ๋‹ค. ๊ทธ๋ž˜์„œ modular RNN encoder์™€ TPR ๊ธฐ๋ฐ˜ ์™ธ๋ถ€ ๋ฉ”๋ชจ๋ฆฌ๋ฅผ ๊ฒฐํ•ฉํ•ด, ๊ตฌ์„ฑ ์š”์†Œ๋ฅผ ๋‹ค์‹œ ์กฐํ•ฉํ•˜๊ธฐ ์ข‹์€ ํ‘œํ˜„์„ ๋ฐฐ์šฐ๋„๋ก ์œ ๋„ํ•œ๋‹ค.

ํ•ต์‹ฌ ๊ตฌ์„ฑ:

  • Recurrent Independent Mechanisms (RIMs): ์—ฌ๋Ÿฌ RNN ๋ชจ๋“ˆ์ด competitive learning์œผ๋กœ ๊ฐ์ž ๋…๋ฆฝ์ ์ธ ์ธ์ฝ”๋”ฉ ๋ฉ”์ปค๋‹ˆ์ฆ˜์„ ํ•™์Šต
  • Tensor Product Representation (TPR): role๊ณผ filler๋ฅผ tensor product๋กœ ๋ฌถ์–ด ์—ฐ๊ด€ ๊ด€๊ณ„๋ฅผ ํ‘œํ˜„ โ€” \(T = \sum_{k=1}^N \mathbf{r}_k \otimes \mathbf{f}_k\)
  • TPR-based External Memory: ๊ฐ ์‹œ์  \(t\)์—์„œ hidden state๋กœ๋ถ€ํ„ฐ role/filler ํ‘œํ˜„์„ ์ถ”์ถœํ•ด ๋ฉ”๋ชจ๋ฆฌ์— ์ค‘์ฒฉ(superpose)ํ•œ๋‹ค. ์“ฐ๊ธฐ ๊ฐ•๋„ \(\beta = \sigma(W_\beta h_t)\)๋ฅผ ์‚ฌ์šฉํ•ด \(\mathbf{M}_t = \mathbf{M}_{t-1} + \mathbf{r}_t \otimes (\beta \mathbf{f}_t - (1-\beta) \mathbf{f}_{t-1})\)๋กœ ๊ฐฑ์‹ ํ•œ๋‹ค.
  • Systematic Associative Recall (SAR): ์ฒด๊ณ„์  ์ผ๋ฐ˜ํ™” ํ‰๊ฐ€๋ฅผ ์œ„ํ•œ ์ƒˆ ๋ฒค์น˜๋งˆํฌ ์ œ์•ˆ

Results

Systematic Associative Recall (SAR) Task

SAR์€ ๋ณธ ๋…ผ๋ฌธ์—์„œ ์ œ์•ˆํ•˜๋Š” ๋ฒค์น˜๋งˆํฌ๋กœ, associative reasoning์—์„œ์˜ ์ฒด๊ณ„์  ์ผ๋ฐ˜ํ™”๋ฅผ ์ธก์ •ํ•˜๊ธฐ ์œ„ํ•ด ์„ค๊ณ„๋˜์—ˆ๋‹ค. ์„ธ ๊ฐ€์ง€ ๊ฐ์ฒด ์ง‘ํ•ฉ(์‚ฌ๋žŒ ์ด๋ฆ„ \(S_h\), ๊ณผ์ผ ์ด๋ฆ„ \(S_f\), ์ˆซ์ž ์ด๋ฆ„ \(S_n\))์„ ์‚ฌ์šฉํ•˜๋ฉฐ, ํ•™์Šต ๋ฐ์ดํ„ฐ์™€ ํ…Œ์ŠคํŠธ ๋ฐ์ดํ„ฐ ๊ฐ„์— ๊ฐ์ฒด ์กฐํ•ฉ์„ ์ฒด๊ณ„์ ์œผ๋กœ ๋‹ค๋ฅด๊ฒŒ ๊ตฌ์„ฑํ•œ๋‹ค.

๊ตฌ์ฒด์ ์œผ๋กœ, \(S_h\)์˜ ์ผ๋ถ€(\(S_h^1\))๋Š” ํ•™์Šต ์‹œ ์ˆซ์ž์™€๋งŒ ์—ฐ๊ด€๋˜๊ณ , ๋‹ค๋ฅธ ์ผ๋ถ€(\(S_h^2\))๋Š” ๊ณผ์ผ๊ณผ๋งŒ ์—ฐ๊ด€๋œ๋‹ค. ํ…Œ์ŠคํŠธ๋Š” ๋‘ ๊ฐ€์ง€๋‹ค. ํ•™์Šต๊ณผ ๊ฐ™์€ ๊ด€๊ณ„๋ฅผ ๋ฌป๋Š” test (same)๊ณผ, ๊ด€๊ณ„๋ฅผ ๋’ค๋ฐ”๊ฟ” \(S_h^1\)์€ ๊ณผ์ผ๊ณผ, \(S_h^2\)๋Š” ์ˆซ์ž์™€ ์—ฐ๊ด€์‹œํ‚จ test (different)์ด๋‹ค. ๋‚œ์ด๋„ ํŒŒ๋ผ๋ฏธํ„ฐ \(p = |S_h^3| / |S_h|\)๋Š” ํ•™์Šต ์‹œ ๊ณผ์ผยท์ˆซ์ž ๋ชจ๋‘์™€ ์—ฐ๊ด€๋˜๋Š” ์‚ฌ๋žŒ ์ด๋ฆ„(\(S_h^3\))์˜ ๋น„์œจ๋กœ, ๊ฐ’์ด ์ž‘์„์ˆ˜๋ก ํ•™์Šต/ํ…Œ์ŠคํŠธ ๊ฐ„ ์ฒด๊ณ„์  ์ฐจ์ด๊ฐ€ ํฌ๋‹ค.

DNC accuracy on SAR

(a) DNC

FWM accuracy on SAR

(b) FWM

Ours accuracy on SAR

(c) Ours

SAR ํƒœ์Šคํฌ์—์„œ DNC, FWM, ์ œ์•ˆ ๋ฐฉ๋ฒ•์˜ ํ•™์Šต/ํ…Œ์ŠคํŠธ ์ •ํ™•๋„ ๋น„๊ต(15k iteration).

DNC์™€ FWM์€ ๋ชจ๋“  \(p\)์—์„œ test (same)๊ณผ test (different) ์‚ฌ์ด์— ํฐ ๊ฒฉ์ฐจ๋ฅผ ๋ณด์ธ๋‹ค. ํ•™์Šต ๋•Œ ๋ณธ ์กฐํ•ฉ์—๋Š” ๋งž์ถ”์ง€๋งŒ, ๊ด€๊ณ„๊ฐ€ ๋’ค๋ฐ”๋€ ์กฐํ•ฉ์œผ๋กœ๋Š” ์ผ๋ฐ˜ํ™”ํ•˜์ง€ ๋ชปํ•œ๋‹ค๋Š” ๋œป์ด๋‹ค. ์ œ์•ˆ ๋ฐฉ๋ฒ•์€ \(p=0.3\)๊ณผ \(p=0.5\)์—์„œ test (different) ์ •ํ™•๋„๊ฐ€ 90%๋ฅผ ๋„˜์–ด ๊ฒฉ์ฐจ๋ฅผ ํฌ๊ฒŒ ์ขํžŒ๋‹ค. ๋‹ค๋งŒ ๊ฐ€์žฅ ์–ด๋ ค์šด \(p=0.1\)์—์„œ๋Š” ์ œ์•ˆ ๋ฐฉ๋ฒ•๋„ test (different) ์ •ํ™•๋„๊ฐ€ 40%์— ๋ชป ๋ฏธ์ณ, ์—ฌ์ „ํžˆ ํฐ ๊ฒฉ์ฐจ๊ฐ€ ๋‚จ๋Š”๋‹ค. FWM์ด TPR ๊ธฐ๋ฐ˜ ๋ฉ”๋ชจ๋ฆฌ๋ฅผ ์‚ฌ์šฉํ•จ์—๋„ ๋ถˆ๊ตฌํ•˜๊ณ  ์ฒด๊ณ„์  ์ผ๋ฐ˜ํ™”์— ์‹คํŒจํ•œ๋‹ค๋Š” ๊ฒƒ์€, TPR ๋ฉ”๋ชจ๋ฆฌ๋งŒ์œผ๋กœ๋Š” ์ถฉ๋ถ„ํ•˜์ง€ ์•Š์œผ๋ฉฐ encoder๊ฐ€ ์˜ฌ๋ฐ”๋ฅธ symbolic representation์„ ํ•™์Šตํ•˜๋Š” ๊ฒƒ์ด ํ•ต์‹ฌ์ž„์„ ์‹œ์‚ฌํ•œ๋‹ค.

Concatenated-bAbI (catbAbI)

SAR์ด ์ฒด๊ณ„์  ์ผ๋ฐ˜ํ™”์— ์ดˆ์ ์„ ๋งž์ถ˜ ๋ฐ˜๋ฉด, catbAbI๋Š” ์ผ๋ฐ˜์ ์ธ ์žฅ๊ธฐ ์—ฐ๊ด€ ์ถ”๋ก  ์„ฑ๋Šฅ์„ ํ‰๊ฐ€ํ•œ๋‹ค. bAbI์˜ story๋ฅผ ๋์—†์ด ์ด์–ด ๋ถ™์ธ sequence์—์„œ ์งˆ์˜์‘๋‹ต์„ ์ˆ˜ํ–‰ํ•˜๋Š” ํƒœ์Šคํฌ์ด๋‹ค.

catbAbI test accuracy (3ํšŒ ํ‰๊ท ). our trial์€ ๋™์ผ ์„ธํŒ…์—์„œ ์ง์ ‘ ์žฌํ˜„ํ•œ ๊ฒฐ๊ณผ๋‹ค.

ModelTest Accuracy
LSTM80.88%
Transformer-XL87.66%
Meta-learned Neural Memory88.97%
Fast Weight Memory (FWM)96.75%
FWM (our trial)94.94%
Ours96.63%

๋™์ผํ•œ ์‹คํ—˜ ์„ธํŒ…(our trial)์—์„œ ๋น„๊ตํ•˜๋ฉด, ์ œ์•ˆ ๋ฐฉ๋ฒ•(96.63%)์ด FWM(94.94%)๋ณด๋‹ค 1.7%p ๋†’๊ณ , FWM ๋…ผ๋ฌธ์˜ ๋ณด๊ณ ์น˜(96.75%)์™€๋„ ๊ฑฐ์˜ ๊ฐ™๋‹ค. modular encoder๋ฅผ ๋„์ž…ํ•ด ์ฒด๊ณ„์  ์ผ๋ฐ˜ํ™” ๋Šฅ๋ ฅ์„ ๋”ํ•˜๋ฉด์„œ๋„, ์ผ๋ฐ˜์ ์ธ ์—ฐ๊ด€ ์ถ”๋ก  ์„ฑ๋Šฅ์€ ์ตœ๊ณ  ์ˆ˜์ค€์— ๊ฐ€๊น๊ฒŒ ์œ ์ง€ํ•จ์„ ๋ณด์—ฌ์ค€๋‹ค.

Symbolic Representation ๋ถ„์„

ํ•™์Šต๋œ ํ‘œํ˜„์ด ์˜ฌ๋ฐ”๋ฅธ symbolic property๋ฅผ ๊ฐ–๋Š”์ง€ ๋‘ ๊ฐ€์ง€ ๋ถ„์„์œผ๋กœ ๊ฒ€์ฆํ•œ๋‹ค.

Role-Unbinding Orthogonality: TPR์—์„œ filler๋ฅผ ์˜ฌ๋ฐ”๋ฅด๊ฒŒ ๊บผ๋‚ด๋ ค๋ฉด(unbinding), ๊ฐ™์€ ๊ฐ์ฒด์˜ role ๋ฒกํ„ฐ์™€ unbinding ๋ฒกํ„ฐ๋Š” ์œ ์‚ฌ๋„๊ฐ€ 1์— ๊ฐ€๊น๊ณ , ์„œ๋กœ ๋‹ค๋ฅธ ๊ฐ์ฒด๋ผ๋ฆฌ๋Š” 0์— ๊ฐ€๊นŒ์›Œ์•ผ(์ง๊ต) ํ•œ๋‹ค. FWM์€ ์œ ์‚ฌ๋„ ํ–‰๋ ฌ์˜ off-diagonal ๊ฐ’์ด ์ƒ๋‹นํžˆ ๋‚จ์•„ ์žˆ์ง€๋งŒ, ์ œ์•ˆ ๋ฐฉ๋ฒ•์€ ํ›จ์”ฌ ์ง๊ต์— ๊ฐ€๊น๋‹ค. ์ด๋Š” modular encoder๊ฐ€ ๊ฐ ๊ฐ์ฒด์— ๋Œ€ํ•ด ๋ถ„๋ฆฌ ๊ฐ€๋Šฅํ•œ(separable) symbolic representation์„ ํ•™์Šตํ–ˆ์Œ์„ ์˜๋ฏธํ•œ๋‹ค.

FWM role-unbinding

(a) FWM

Ours role-unbinding

(b) Ours

์„œ๋กœ ๋‹ค๋ฅธ ์‚ฌ๋žŒ ๊ฐ์ฒด์— ๋Œ€ํ•œ role ๋ฒกํ„ฐ์™€ unbinding ๋ฒกํ„ฐ ๊ฐ„ ์œ ์‚ฌ๋„ ํ–‰๋ ฌ. FWM์€ off-diagonal ์œ ์‚ฌ๋„๊ฐ€ ๋‚จ์•„ ์žˆ์ง€๋งŒ, ์ œ์•ˆ ๋ฐฉ๋ฒ•์€ ๊ฑฐ์˜ ์ง๊ตํ•œ๋‹ค.

Filler Consistency: ๋™์ผํ•œ ๋Œ€์ƒ ๊ฐ์ฒด์— ๋Œ€ํ•ด, ์–ด๋–ค ์กฐํ•ฉ์—์„œ ์งˆ์˜ํ•˜๋“  ๋™์ผํ•œ read ๋ฒกํ„ฐ๊ฐ€ ๋ฐ˜ํ™˜๋˜์–ด์•ผ ์ฒด๊ณ„์  ์ถ”๋ก ์ด๋ผ ํ•  ์ˆ˜ ์žˆ๋‹ค. FWM์€ ์กฐํ•ฉ์— ๋”ฐ๋ผ read ๋ฒกํ„ฐ๊ฐ€ ๋‹ฌ๋ผ์ง€์ง€๋งŒ, ์ œ์•ˆ ๋ฐฉ๋ฒ•์€ ์กฐํ•ฉ์— ๊ด€๊ณ„์—†์ด ๊ฑฐ์˜ ๋™์ผํ•œ read ๋ฒกํ„ฐ๋ฅผ ๋ฐ˜ํ™˜ํ•œ๋‹ค. ์ด๋Š” ๋ชจ๋ธ์ด ํŠน์ • ์กฐํ•ฉ์„ ์•”๊ธฐํ•˜๋Š” ๊ฒƒ์ด ์•„๋‹ˆ๋ผ, ๊ฐœ๋ณ„ ๊ตฌ์„ฑ ์š”์†Œ๋ฅผ ๋…๋ฆฝ์ ์œผ๋กœ ์ธ์ฝ”๋”ฉํ•˜๊ณ  ์žฌ์กฐํ•ฉํ•˜์—ฌ ์ถ”๋ก ํ•˜๊ณ  ์žˆ์Œ์„ ๋ณด์—ฌ์ฃผ๋Š” ์ฆ๊ฑฐ์ด๋‹ค.

FWM read vectors

(a) FWM

Ours read vectors

(b) Ours

๊ฐ™์€ ๊ณผ์ผ ๊ฐ์ฒด๋ฅผ ๊บผ๋‚ธ read ๋ฒกํ„ฐ ๊ฐ„ ์œ ์‚ฌ๋„. ์ œ์•ˆ ๋ฐฉ๋ฒ•์€ ์กฐํ•ฉ์— ๊ด€๊ณ„์—†์ด ์ผ๊ด€๋œ ์ถœ๋ ฅ์„ ๋ณด์ธ๋‹ค.