Recently, when I was working on Join Stages – I encountered a weird problem when I am using Join stages to join a Table and flat file
Job with a join stage used to terminate with the following error Unexpected terminated by UNIX Signal 11 (SIGSEGV)
This is due to record size which exceed the block size setting at Project level
In order to resolve this :
1. Calculate the size of your largest record in bytes.
2. Double the size and set the environment variable APT_TSORT_STRESS_BLOCKSIZE to that value in bytes.
If this environment variable does not exist, create it first.
The variable APT_TSORT_STRESS_BLOCKSIZE sets the size of the shared memory block that is used to pass data between the writer, the sorter, and merger processes within the sort operation.
The default is set so that each sort in a job attempts to map 20 MB each for these in-memory functions.
In the case when a single record is larger than 20 mb, this step is necessary to increase the size of the block.
The APT_TSORT_STRESS_BLOCKSIZE environment variable can be set at the job level, which is recommended.
The APT_TSORT_STRESS_BLOCKSIZE environment variable can also be set at the project level by defining a project level default value but consumes more scratch and resource disk
Hope this helps!!