Attention is not quite all you need

While working on MacGraph, a neural network that answers questions using knowledge graphs, we came across a problem: How to tell if something’s not present in a list.

We (Octavian.AI) have developed a novel and robust neural network solution to detecting if an item is present in a list. We've created this as an extension of the popular attention mechanism.