IBM Support

Record size limit exceeded on generate statistics

Troubleshooting


Problem

the table size is less than 64KB and the record is loadable but generate statistics failed because of Record size limit exceeded

Symptom

[nz@netezza nzscratch]$ nzsql -c 'generate statistics on sample_table'
ERROR: GENERATE STATISTICS: SAMPLE_TABLE(C12,C13,C14,C15,C16,C17,C18,C19,C20,C21) - 68032 : Record size limit exceeded

Cause

when we do a GENERATE STATISTICS, we must collect multiple pieces of information about each of the columns.
MIN value (so, 1X the max column width)
MAX value (and another 1X the max column width)
# of null values, # of unique values, average # of bytes

So the "projected row size" is going to be somewhat greater (at least 2X+). Which is why we don't try to process all of the columns all at once ... as the projected row size would frequently be exceeding the 65K limit and throwing an error.

Environment

apply to all NPS version

Diagnosing The Problem

[nz@netezza nzscratch]$ cat qq.sql
CREATE TABLE sample_table
(
C1 numeric(19,0) not null,
C2 national character varying(63) not null,
C3 national character varying(153) not null,
C4 national character varying(7),
C5 timestamp not null,
C6 timestamp,
C7 timestamp,
C8 national character varying(37),
C9 numeric(19,0),
C11 national character varying(25),
C12 national character varying(20),
C13 national character varying(11),
C14 national character varying(13),
C15 national character varying(9),
C16 national character varying(91),
C17 national character varying(19),
C18 national character varying(4000),
C19 national character varying(12),
C20 national character varying(1799),
C21 national character varying(2472),
C22 national character varying(5),
C23 national character varying(100),
C24 national character varying(13),
C25 national character varying(5),
C26 timestamp,
C27 national character varying(39),
C28 timestamp,
C29 timestamp not null,
C30 timestamp not null,
CUSTOM1 national character varying(60),
CUSTOM2 national character varying(25),
CUSTOM3 national character varying(307),
CUSTOM4 national character varying(218),
CUSTOM5 national character varying(46),
CUSTOM6 national character varying(9),
CUSTOM7 national character varying(17),
CUSTOM8 national character varying(60),
CUSTOM9 national character varying(60),
CUSTOM10 national character varying(21),
CUSTOM11 national character varying(20),
CUSTOM12 national character varying(44),
CUSTOM13 national character varying(36),
CUSTOM14 national character varying(9),
CUSTOM15 national character varying(65),
CUSTOM16 national character varying(15),
CUSTOM17 national character varying(15),
CUSTOM18 national character varying(18),
CUSTOM19 national character varying(70),
CUSTOM20 national character varying(60),
CUSTOM21 national character varying(18),
CUSTOM22 national character varying(100),
CUSTOM23 national character varying(148),
CUSTOM24 national character varying(31),
CUSTOM25 national character varying(14),
CUSTOM26 national character varying(18),
CUSTOM27 national character varying(4),
CUSTOM28 national character varying(5),
CUSTOM29 national character varying(19),
CUSTOM30 national character varying(75),
CUSTOM31 national character varying(3),
CUSTOM32 national character varying(3),
CUSTOM33 national character varying(61),
CUSTOM34 national character varying(12),
CUSTOM35 national character varying(31),
CUSTOM36 national character varying(4),
CUSTOM37 national character varying(10),
CUSTOM38 national character varying(10),
CUSTOM39 national character varying(10),
CUSTOM40 national character varying(10),
CUSTOM41 timestamp,
CUSTOM42 timestamp,
CUSTOM43 timestamp,
CUSTOM44 timestamp,
CUSTOM45 timestamp
)
DISTRIBUTE ON (CUSTOM3)
;
[nz@netezza nzscratch]$ nzsql -f qq.sql
CREATE TABLE
[nz@netezza nzscratch]$ nzsql -c 'generate statistics on sample_table'
ERROR: GENERATE STATISTICS: SAMPLE_TABLE(C12,C13,C14,C15,C16,C17,C18,C19,C20,C21) - 68032 : Record size limit exceeded


So in this particular case we are generating FULL statistics as the table is quite small. So we are going to process 10 columns at a time. The GENSTATS is actually dying on the 2nd set of 10 columns, which are defined thusly. when we generate statistics on below column
C12,C13,C14,C15,C16,C17,C18,C19,C20,C21
And the size of these 10 columns, combined, is somewhere around 33K (as all NATIONAL varchar columns need to be able to accommodate as many as 4 bytes per character).

So that is 33K. But the projected row, produced by the GENSTATS, will need to be 2X that size ... to handle the MIN column value and the MAX column value.

And 2X times 33K puts us at 66K ... which puts us > 65K ... which is the maximum records width. Which is why you are getting this error.

You have just hit an edge case where ... even though NPS is processing a small number of columns at once (to try to avoid having any problems) ... it is still having problems because of the particular mix and size of data type

Resolving The Problem

Work around for tihis GENSTATS operation

set STATS_COL_GRP_LIMIT = 9;

[nz@netezza nzscratch]$ nzsql -c 'set STATS_COL_GRP_LIMIT = 9;generate statistics on sample_table'
GENERATE STATISTICS

[{"Product":{"code":"SSULQD","label":"IBM PureData System"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":"Database","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"1.0.0","Edition":"","Line of Business":{"code":"LOB10","label":"Data and AI"}}]

Document Information

Modified date:
17 October 2019

UID

swg22005187