Expanding the Net: How Google’s Transformer Continuation Patents Cover Decoder-Only GPT Architectures

Google’s Transformer patent (“Attention Is All You Need”) is US10452978B2. While the patent itself covers an encoder-decoder architecture, it has spawned multiple continuation patents.

Relationship Between US10452978B2 and US10719764B2

Overview of the Two Patents

Based on the information collected, the relationship between the two patents can be summarized as follows.

US10452978B2 (Parent Patent) — How the Relationship Is Described

In the prosecution history of US10452978B2, the following entry appears: “2019–09–03 Priority to US16/559,392 priority patent/US10719764B2/en.”

In other words, the parent patent US10452978B2 records US10719764B2 (Application No. US16/559,392) as a subsequent continuation application, reflecting the priority relationship between the two.

US10719764B2 (Continuation Patent) — How the Relationship Is Described

US10719764B2 is a Continuation Application of US10452978B2. As a matter of U.S. patent law, a continuation application must satisfy the following requirements:

• Uses the same specification as the parent application
• Inherits the parent’s priority date (May 23, 2017) unchanged
• Names at least one of the same inventors as the parent (Vaswani, Shazeer, Kaiser, and the other five co-inventors)
• Assignee: Google LLC

Importantly, a continuation application may pursue different claims from those of the parent — and this is precisely the primary purpose of filing a continuation.

Diagram: Relationship Between the Two Patents

PCT/US2018/034224 (International Application, filed May 23, 2018)
        ↓
US10452978B2 (Parent Patent)
  · Application No.: US16/021,971
  · Filing Date: June 28, 2018
  · Grant Date: October 22, 2019
  · Claims: Encoder + Decoder architecture
        ↓ Continuation Application
US10719764B2 (Child Patent)
  · Application No.: US16/559,392
  · Filing Date: September 3, 2019
  · Grant Date: July 21, 2020
  · Claims: To be confirmed (potentially covering
            different aspects of the invention)

Key Points for Practical Purposes

Because a continuation application shares the same specification as the parent but allows the applicant to pursue different claims, Google may have used US10719764B2 to secure protection for aspects not fully covered by the claims of US10452978B2.

As noted in the previous investigation, information retrieved from Justia included references to a “self-attention decoder neural network,” suggesting that claims covering a decoder-only architecture may exist in US10719764B2. This point warrants particular attention and makes a full review of the claims of US10719764B2 strongly advisable.