Understanding and Overcoming the 22-Field Limitation in Scala 2.x Case Classes

In Scala 2.x, case classes are limited to 22 fields. If you try to define a case class with more than 22 fields, you will encounter the error: No unapply or unapplySeq function found. This limitation arises due to the way tuples and functions are implemented in the Scala standard library. Here’s a detailed explanation:

1. Tuple and Function Arity Limitation

Scala 2.x supports tuples and functions up to an arity of 22. This means the standard library defines Tuple1 through Tuple22 and Function1 through Function22. This limitation directly impacts the case class implementation:

• Tuples: A case class with n fields can be seen as a tuple of n elements. Since the Scala 2.x standard library only defines tuples up to Tuple22, case classes are consequently limited to 22 fields.
• Functions: The unapply method used in pattern matching with case classes also adheres to this limit, as it essentially involves functions of certain arity.

2. Pattern Matching and unapply Method

When you define a case class, Scala automatically generates several methods, including unapply or unapplySeq. These methods are used for pattern matching and deconstruction of the case class. For a case class with more than 22 fields, the unapply method would exceed the maximum arity supported by Scala's function definitions.

3. Implementation Complexity and Performance

Supporting tuples and functions with more than 22 elements would complicate the Scala compiler and runtime. The designers of Scala chose the limit of 22 as a practical compromise to avoid excessive complexity and performance overhead in the compiler and standard library.

4. Historical and Practical Reasons

The choice of 22 as the upper limit was somewhat arbitrary but driven by practical considerations. It was deemed rare for most case classes to require more than 22 fields. This limit balances usability with the overhead associated with supporting very large arities.

Overcoming the Limitation

If you need to define a case class with more than 22 fields, you can work around this limitation by:

1. Nested Case Classes: Break down your large case class into multiple nested case classes, each with fewer fields.
2. Custom Data Structures: Define your own data structures to handle more fields if necessary.
3. Using Collections: Store additional fields in a collection like a Map or List if they are not central to the logic of the case class.

If adding more fields to an existing case class, Options 1 and 3 can be challenging in some situations. This article will discuss how to choose Option 2 and implement a custom data structure to read more than 22 fields.

Scala 3.x Changes

Scala 3.x (Dotty) removes this limitation by introducing support for higher arity functions and tuples. The new implementation allows for defining case classes with more than 22 fields without encountering the same issues present in Scala 2.x. However, updating to Scala 3.x might not be possible in all cases.

How to Create Custom Read and Write Methods

Here is an example implementation of a custom data structure to overcome the 22-field limitation:

Dummy.scala

package com.arun.mittal

import org.apache.spark.sql.{Encoder, Encoders, Row}
import play.api.libs.json.{JsError, JsNumber, JsObject, JsResult, JsSuccess, JsValue, Json, Reads, Writes, __}
import play.api.libs.functional.syntax.toFunctionalBuilderOps

case class Dummy(
                  a: Int = 0,
                  b: Int = 0,
                  c: Int = 0,
                  d: Int = 0,
                  e: Int = 0,
                  f: Int = 0,
                  g: Int = 0,
                  h: Int = 0,
                  i: Int = 0,
                  j: Int = 0,
                  k: Int = 0,
                  l: Int = 0,
                  m: Int = 0,
                  n: Int = 0,
                  o: Int = 0,
                  p: Int = 0,
                  q: Int = 0,
                  r: Int = 0,
                  s: Int = 0,
                  t: Int = 0,
                  u: Int = 0,
                  v: Int = 0,
                  w: Int = 0,
                  x: Int = 0,
                  y: Int = 0,
                  z: scala.collection.immutable.Map[String, Long]
                )

object Dummy{
  implicit val encoder: Encoder[Dummy] = Encoders.product[Dummy]

  implicit val reads: Reads[Dummy] = (
    (
      (__ \ "a").read[Int] and
        (__ \ "b").read[Int] and
        (__ \ "c").read[Int] and
        (__ \ "d").read[Int] and
        (__ \ "e").read[Int] and
        (__ \ "f").read[Int] and
        (__ \ "g").read[Int] and
        (__ \ "h").read[Int] and
        (__ \ "i").read[Int] and
        (__ \ "j").read[Int] and
        (__ \ "k").read[Int] and
        (__ \ "l").read[Int] and
        (__ \ "m").read[Int]
      ).tupled and
      (
        (__ \ "n").read[Int] and
          (__ \ "o").read[Int] and
          (__ \ "p").read[Int] and
          (__ \ "q").read[Int] and
          (__ \ "r").read[Int] and
          (__ \ "s").read[Int] and
          (__ \ "t").read[Int] and
          (__ \ "u").read[Int] and
          (__ \ "v").read[Int] and
          (__ \ "w").read[Int] and
          (__ \ "x").read[Int] and
          (__ \ "y").read[Int] and
          (__ \ "z").read[Map[String, Long]]
        ).tupled
    )( (firstGroup, secondGroup) => Dummy(
    firstGroup._1, firstGroup._2, firstGroup._3,
    firstGroup._4, firstGroup._5, firstGroup._6,
    firstGroup._7, firstGroup._8, firstGroup._9,
    firstGroup._10, firstGroup._11, firstGroup._12,
    firstGroup._13, secondGroup._1, secondGroup._2,
    secondGroup._3, secondGroup._4, secondGroup._5,
    secondGroup._6, secondGroup._7, secondGroup._8,
    secondGroup._9, secondGroup._10, secondGroup._11,
    secondGroup._12, secondGroup._13
  ))

  implicit val writes: Writes[Dummy] = new Writes[Dummy] {
    def writes(dfResponse: Dummy): JsValue = Json.obj(
      "a" -> dfResponse.a,
      "b" -> dfResponse.b,
      "c" -> dfResponse.c,
      "d" -> dfResponse.d,
      "e" -> dfResponse.e,
      "f" -> dfResponse.f,
      "g" -> dfResponse.g,
      "h" -> dfResponse.h,
      "i" -> dfResponse.i,
      "j" -> dfResponse.j,
      "k" -> dfResponse.k,
      "l" -> dfResponse.l,
      "m" -> dfResponse.m,
      "n" -> dfResponse.n,
      "o" -> dfResponse.o,
      "p" -> dfResponse.p,
      "q" -> dfResponse.q,
      "r" -> dfResponse.r,
      "s" -> dfResponse.s,
      "t" -> dfResponse.t,
      "u" -> dfResponse.u,
      "v" -> dfResponse.v,
      "w" -> dfResponse.w,
      "x" -> dfResponse.x,
      "y" -> dfResponse.y,
      "z" -> dfResponse.z
    )
  }


  def apply(row: Row): Dummy = {
    new Dummy(
      a = row.getAs[Int]("a"),
      b = row.getAs[Int]("b"),
      c = row.getAs[Int]("c"),
      d = row.getAs[Int]("d"),
      e = row.getAs[Int]("e"),
      f = row.getAs[Int]("f"),
      g = row.getAs[Int]("g"),
      h = row.getAs[Int]("h"),
      i = row.getAs[Int]("i"),
      j = row.getAs[Int]("j"),
      k = row.getAs[Int]("k"),
      l = row.getAs[Int]("l"),
      m = row.getAs[Int]("m"),
      n = row.getAs[Int]("n"),
      o = row.getAs[Int]("o"),
      p = row.getAs[Int]("p"),
      q = row.getAs[Int]("q"),
      r = row.getAs[Int]("r"),
      s = row.getAs[Int]("s"),
      t = row.getAs[Int]("t"),
      u = row.getAs[Int]("u"),
      v = row.getAs[Int]("v"),
      w = row.getAs[Int]("w"),
      x = row.getAs[Int]("x"),
      y = row.getAs[Int]("y"),
      z = row.getAs[Map[String, Long]]("z")
    )
  }
}

Usage Example

To test the above implementation:

##test method for testing above implementation

test("more than 22 fields"){
    import sqlContext.implicits._
    import org.apache.spark.sql.functions._

    val mm = Map[String, Long]("101" -> 1L, "102" -> 2L)

    val df = Seq(
      (mm, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21)
    ).toDF("z", "a", "b", "c", "d", "e", "f", "g", "h", "i", "j", "k", "l", "m", "n", "o", "p", "q", "r", "s", "t", "u")
      .withColumn("v", lit(22))
      .withColumn("w", lit(23))
      .withColumn("x", lit(24))
      .withColumn("y", lit(25))

    val forwarder = Forwarder()

    df.pipe(rows => forwarder.fireRequest(rows)).show(false)

  }

##Forwarder.scala

package com.arun.mittal.helper

import com.arun.mittal.Dummy
import org.apache.spark.sql.DataFrame

case class Forwarder(){
  def fireRequest(df: DataFrame): DataFrame = {
    df.map(row => {
      val tt = Dummy(row)
      tt
    }).toDF()
  }
}

NOTE: If there are more than 44 fields, we have to create 3 tuples.