SQL Execution Via-SparkThriftServer throws “Error: java.lang.StackOverflowError”

Overview 

This article describes "Error: java.lang.StackOverflowError (state=,code=0)” seen while executing very long SQL in Via-SPARK connection. Issue and resolution were tested on Ambari managed 3 node Hortonworks 2.6.5.0 cluster.

The test query contains >=133 number of predicates in where clause. For such query with default configuration java.lang.StackOverflowError
is seen.

From Spark log file  we see following error messages.

When we use Simba Driver for Spark , the console error seen might be different.

Root Cause

It can be noticed from log messages that default stack size of JVM is allowing to push 132 calls to nodeToExpr I.e. 132 predicates. Hence, when 133^rd predicate is added to where clause of query , java.lang.StackOverflowError was seen.

Resolution

We need to increase StackSize by using Xss option of java. To support query containing many more predicates in where clause we can set stack size to large value say 1500m.

We may add following line at end of spark-env.sh file (or increase value if such statement already exist).

After saving spark-env.sh , restart Spark Thrift Server, Spark History Server and refresh Spark client configs. Now same test query will runs successfully.