As LLMs get better, do we still need RAG?

%20(500%20%C3%97%20500%20px)%20(1).avif)

Julian Seidenberg
Published 28 May 2025

Learn how Mobile AI solutions can elevate your business

🤯 Meta's Llama 4 Scout Large Language Model (LLM) claims a 10 Million token context window! Is this real? How did they pull that off?

Just How Big is 10 Million Tokens?

Let’s put that in perspective. The average English word is ~1.3 tokens. This means a 10M token context could theoretically hold:

  • Robert Jordan’s "Wheel of Time" book series (4.1M tokens)
  • George R.R. Martin's "A Song of Ice and Fire" series (2.2M tokens)
  • J.K. Rowling's "Harry Potter" series (1.3M tokens)
  • William Shakespeare’s complete works (1.2M tokens)
  • J.R.R. Tolkien's "Lord of the Rings" series (0.8M tokens)

Total: 9.6M tokens.

Why stop at 10M? Many big-tech CEOs are talking about building models with infinite context windows. Mark Zuckerberg boasts about a nearly infinite context window and Sergey Brin responds to rumours of infinite context windows with Google Gemini. With such a big context windows, who needs RAG and a VectorDB? You can just feed in all your context and ask questions, right?

Well, not so fast!

What’s the catch?

There are trade-offs with increasing the context window size:

  1. Computational Cost & Memory: While a model like Llama 4 Scout can run on a single NVIDIA H100 GPU for certain tasks, efficiently using its super-long context requires serious horsepower. Meta benchmarked the 10M-token window model on a cluster using 512 H100 GPUs. Renting 64 × p5.48xlarge instances (each with 8 H100s) on AWS to match that setup costs roughly USD $150,000 per day!
  2. The "Needle in a Haystack" Problem: Long context window models like Llama 4 can successfully find a specific piece of information (the "needle") within the vast "haystack" (often text). For instance, if we insert "Hamlet said: I'm Batman" somewhere into the 9.6M token text, then ask the LLM “What’s the Dark Knight’s secret identity?”, it likely would do an impressive job of finding the “correct” answer. However, asking questions that require reading and understanding the full context could yield poor responses.
  3. Model Size vs. True Understanding: Llama 4 Scout is a relatively small model with 17 billion parameters. This small size means it will likely struggle with remembering long prompts and maintaining a nuanced understanding of the given context. Larger models generally have a better capacity for detailed understanding, which makes implementation more challenging due to resource requirements.

In summary, to cost-effectively ask non-trivial questions of a large text, RAG using a Vector database is almost certainly the better solution. Even better is using some LLM memory or knowledge graph technology to intelligently retrieve the most relevant text for the LLM to reason about.

How Does the Underlying Technology Work?

Positional Encoding: Classic Sinusoidal vs. RoPE + PI

To understand a 10M token model, we first need to understand how a Sinusoidal Positional Encoding (SPE) works. Positional encoding is essential for the LLM to discern the meaning of tokens based on their placement.

Classic Sinusoidal Positional Encoding (SPE)

  • How it works: SPE adds a deterministic vector, generated using sine and cosine functions of a token's absolute position, to each token embedding. This allows the transformer layers to learn correlations between word order and meaning.
  • Limitations: The periodic nature can hinder the model's ability to generalize with unseen long sequences.

Rotary Position Embedding (RoPE) + Positional Interpolation (PI)

  • How RoPE works: Instead of adding, RoPE rotates each token’s embedding dimensions by an angle θ based on the token's position in the input. This preserves relative rotation positions regardless of text length.
  • Advantages: Stability of relative positions leads to better extrapolation beyond trained windows and effective Key-Value caching in LLMs.

Key Implications

Long context windows like Meta’s 4 Scout’s 10M tokens excel at specific queries but struggle with deep reasoning. They are also resource-intensive, making solutions like Knowledge Graphs and Vector DBs essential for practical LLM applications.

Please get in touch with us at Datch to learn more about our technology for handling large amounts of context.