Chapter 1: Understanding LLMs and Their Limitations

Large language models (LLMs) have achieved remarkable success due to their extensive capabilities; however, it remains uncertain if they represent the pathway to artificial general intelligence (AGI). The constraints of current LLMs are intrinsically linked to the transformer architecture, which struggles with generalization.

The core debate revolves around the utilization of knowledge acquired by models. While LLMs can amass vast amounts of information, the efficiency of this knowledge's application is questionable. For instance, LLMs often fail to effectively compose internalized facts, highlighting a significant distinction between mere memorization and genuine reasoning. This challenge is particularly evident when reasoning involves multiple steps, where even advanced models like GPT-4 face obstacles.

Consequently, this inefficacy in knowledge application leads to redundancy in stored facts and complications in updating knowledge. Although the lack of compositionality may appear trivial, it critically hampers a model's generalization capabilities. Humans, conversely, leverage compositionality to generalize effectively, even in data-scarce scenarios.

A pertinent inquiry is whether scaling up models can address these limitations. Recent discussions have focused on whether increasing the number of parameters or the volume of data could not only enhance performance but also foster emergent properties.

Chapter 2: Investigating Implicit Reasoning through Scaling

Is it possible that implicit reasoning will emerge from further scaling? Can we transcend the constraints of the transformer by increasing its scale? Answering these questions is undoubtedly complex, particularly when evaluating already trained models. A recent study aimed to tackle these queries by not only generating synthetic datasets but also by training the transformer from the ground up.

The authors conceptualize reasoning as a process of induction and the application of inference rules. To systematically examine this, they define a set of "atomic facts" and "inferred facts," which can be derived from atomic facts using a collection of latent rules. Their goal is to ascertain whether a transformer can make inferences in both in-distribution (ID) and out-of-distribution (OOD) contexts, thereby understanding if the model learns latent rules and can generalize them.

The study begins with compositionality, which involves linking different facts. For example, if "Barack's wife is Michelle" and "Michelle was born in 1964," a competent model should be able to infer that "Barack's wife was born in 1964." Despite seeming straightforward, literature indicates that transformers struggle with such tasks.

In lieu of traditional knowledge sources like Wikipedia, the authors create a random knowledge graph (KG) made up of entities and relations. A KG comprises triplets (subject, relation, object), facilitating multi-hop reasoning within the graph. Essentially, if 'a' is linked to 'b' via relation 'r1,' and 'b' connects to 'c' through 'r2,' the model should deduce a connection among 'a', 'r1', 'r2', and 'c'. They utilize GPT-2 trained on this dataset from scratch, controlling data points to examine behavior at scale.

Chapter 3: The Phenomenon of Grokking

Recently, the concept of "grokking" has garnered attention. This phenomenon describes a model that initially appears to overfit but eventually achieves generalization with continued training. Grokking indicates a nuanced relationship between memorization and generalization, which is why the authors explore the scaling of training to discern whether the model encounters memorization or succeeds in generalization.

Their findings reveal that while the model can generalize to ID test examples, high performance is only realized through extended training well beyond the point of overfitting. This observation underscores the significance of grokking, where prolonged training allows for substantial improvements in generalization.

Despite achieving high accuracy on training sets, this does not directly translate to generalization. However, as training persists, generalization improves dramatically, particularly for in-distribution scenarios. In contrast, even prolonged training fails to yield improvements in OOD generalization.

The authors highlight that grokking's emergence hinges on a critical data size (CDS). Beyond this threshold, the model shifts from memorization to generalization. Interestingly, their results suggest that the distribution size plays a more pivotal role than merely the dataset's size. In essence, grokking facilitates generalization, contingent upon a sufficiently varied dataset.

Chapter 4: Investigating Internal Mechanisms

What mechanisms enable grokking? Why does the model continue to struggle with OOD examples? The authors delve into these questions through two analytical techniques: Logit lens and causal tracing. By interpreting hidden states and analyzing information propagation through intermediate states, they can scrutinize the model at various training checkpoints.

The investigation aims to identify and analyze "generalizing circuits," which are networks of neurons that collectively contribute to generalization capabilities. They discover that such circuits exist across different layers, facilitating compositionality by connecting various entities within relationships. Notably, these circuits strengthen throughout training, particularly during grokking, at which point the model transitions from mere memorization to effective association.

The study reveals that both memorization and generalization circuits can fit a dataset, although memorization circuits tend to be learned first. The generalization circuit is more efficient, requiring fewer parameters. Typically, memorization circuits are established rapidly, but generalization circuits emerge over time, especially when external factors like weight decay are involved.

Despite the model acquiring compositionality through grokking, it lacks the incentive to retain atomic facts in upper layers if they do not appear during training's second hop. This limitation results in failure during the second hop, as the model cannot effectively query.

To facilitate generalization, modifications to the transformer, such as memory augmentation and explicit recurrence, are necessary. The authors probe deeper into this concept by examining a task termed "comparison."

Chapter 5: The Comparison Task and Systematicity

For instance, considering the attribute of age, the initial rule posits that if the age of entity e1 is less than that of entity e2, we can infer that "e1 is younger than e2." This task presents challenges even for advanced models like GPT-4. However, if knowledge resides in the same vicinity, the model can solve it effectively.

Their analysis reveals that the comparison task generates a "parallel circuit" learned during grokking, enabling the storage and retrieval of atomic facts in proximity, thus facilitating systematicity.

The transformer can resolve this task primarily due to grokking, which also allows for systematicity in generalization (particularly for OOD scenarios). Unlike the composition task, facts relevant to OOD are now stored and accessed similarly to ID facts. This observation is intriguing because it demonstrates that the model can tackle a seemingly sequential task using a parallel circuit.

The authors conclude that the ability to generalize is linked to parametric memory, prompting further questions regarding its practical significance. Can we not simply enhance LLMs with non-parametric memory, such as long-context modes or explicit retrieval, to address tasks effectively?

They argue that parametric memory is crucial for deep compression and integration of information necessary for complex reasoning. Despite the challenges posed by reasoning in expansive search spaces, a fully grokked transformer can achieve near-perfect accuracy.

To substantiate this claim, they extend the comparison task with additional complexity and rules to test OOD capabilities. They compare a fully grokked model with other models like GPT-4 and Gemini, which employ chain-of-thought (CoT) and retrieval strategies. The results indicate that these models struggle, often defining problems as unsolvable, and exhibit hallucinations and flawed rationale. In contrast, the fully grokked transformer excels, achieving near-perfect results.

In conclusion, the study asserts that transformers can learn to implicitly reason over parametric knowledge, but this skill is robustly developed only through extensive training well beyond the overfitting point—grokking.

Chapter 6: Implications for Future Research

The findings illustrate that LLMs are fundamentally underfitted, and training them is an expensive endeavor. While some studies suggest that excessive training epochs are detrimental, this research indicates that surpassing the overfitting threshold is essential for eliciting generalization. The ability to efficiently utilize knowledge through grokking is particularly noteworthy.

These insights inform data and training setups to better promote implicit reasoning and suggest potential modifications to transformer architecture to unlock enhanced generalization capabilities. The authors underscore the significance of parametric memory in addressing challenging reasoning tasks with large search spaces.

The ongoing debate regarding transformer modifications or replacements continues, with various studies concentrating on increasing efficiency and mitigating the computational costs associated with self-attention. However, the authors emphasize that aside from issues of knowledge utilization, reasoning transformers may not benefit from additional context provided by prompts. This suggests that in certain domains, a model fine-tuned for a specific area may outperform a generalist LLM supplemented with external memory.

Ultimately, the authors frame the implicit reasoning challenge as one of induction and the application of inference rules derived from a blend of atomic and inferred facts. This approach, while insightful, does not encompass the entire spectrum of reasoning, which encompasses various types and meanings. Further investigation is warranted, as the authors acknowledge the challenges in ensuring their results are applicable across different setups.

What are your thoughts? Do you believe the transformer can achieve greater generalization in the future? If you found this exploration engaging, feel free to connect with me on LinkedIn, and check out my other articles. You can also subscribe for updates when I publish new insights.

Here is the link to my GitHub repository, where I collect resources and code related to machine learning and artificial intelligence.

Recent Articles of Interest:

• Yang, 2024: Do Large Language Models Latently Perform Multi-Hop Reasoning?
• Press, 2023: Measuring and Narrowing the Compositionality Gap in Language Models
• Wang, 2024: Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalization
• Lake, 2017: Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks
• Zhong, 2023: MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions

Thank you for being a part of the In Plain English community! Be sure to follow us for more content on LinkedIn, YouTube, and other platforms.