Frame Size Configuration and RowTooLarge Error Resolution in Druid

Background

In Apache Druid, data processing usually involves dividing large-scale data into multiple “frames” for processing. In this process, the frame size is a key parameter that determines how much data can be processed in each frame. A frame size that is too large or too small can affect performance or cause processing to fail. This error log indicates that while processing a particular piece of data, a row of data was encountered whose size exceeded the specified frame size, causing the task to fail.

Error Message:

RowTooLarge: Encountered row that cannot fit in a single frame (max frame size = 1,000,000)

Indicates that while processing data, Druid encountered a row of data whose size exceeded 1,000,000 bytes (i.e., STANDARD_FRAME_SIZE ) and was unable to fit the current frame size limit.

Debug

1. Frame Size Configuration: As you can see from the code snippet, Druid’s default frame size is 1,000,000 bytes. The value of STANDARD_FRAME_SIZE is statically defined:

   private static final int STANDARD_FRAME_SIZE = 1_000_000;

   This value is used as the standard frame size for memory allocation during Druid task execution:

     final InputChannels inputChannels =
         new InputChannelsImpl(
             workOrder.getQueryDefinition(),
             InputSlices.allReadablePartitions(workOrder.getInputs()),
             inputChannelFactory,
             () -> ArenaMemoryAllocator.createOnHeap(frameContext.memoryParameters().getStandardFrameSize()),
             exec,
             cancellationId
         );
       
   public int getStandardFrameSize()
   {
     return STANDARD_FRAME_SIZE;
   }

2. Task Failure Reason: The task failed because a row of data exceeded the 1,000,000 byte size limit and could not fit into a single frame. This error message indicates that during execution, a row of data was too large for the set maximum frame size, causing the task to fail.

3. Modifying the frame size: If larger rows of data need to be processed, consider increasing the size of STANDARD_FRAME_SIZE to allow larger rows of data to be processed per frame. For example, this could be modified to 2,000,000 or higher:

   private static final int STANDARD_FRAME_SIZE = 2_000_000;

4. Memory Optimization: Increasing the frame size may increase memory consumption, so there is a trade-off between memory usage and data processing performance. If there is not enough memory, increasing the frame size may cause problems such as memory overflow. When adjusting this value, you need to consider the overall system memory and load.

Conclusion

The task failed because a row of data exceeded the maximum frame size ( 1,000,000 bytes) configured by Druid. To resolve this issue, you can consider the following options:

1. Increase the frame size: Increase STANDARD_FRAME_SIZE to a larger value, such as 2,000,000 or higher, to be able to handle larger rows of data.
2. Optimize data row size: If possible, adjust how data is generated to reduce the size of a single row of data.
3. Memory management: Increasing the frame size may increase memory consumption and you need to make sure that the system has enough memory available.

When adjusting the frame size, it is recommended to test it in conjunction with actual memory usage to avoid other problems caused by insufficient memory.