Sample Header Ad - 728x90

Trim_reason = DICTIONARY_SIZE on Clustered ColumnStore index

5 votes
3 answers
1042 views
On a partitioned table with 17 billion rows, I'm getting a very high number of row_groups. I've heard that I should try to get 1.048.576 rows in each rowgroup. How do I achieve that? The trim_reason is DICTIONARY_SIZE, as reported by sys.dm_db_column_store_row_group_physical_stats. I get on average 100.000 rows per rowgroup, when my batch job is inserting 1,048,576 rows per batch. I'm fairly new to columnstore indexes, but the only reference I can find to DICTIONARY_SIZE is for string columns. My table only has bit, bigint, and numeric columns. My table looks like this (column names obscured), and I know that a lot of the columns contain mostly NULLs. If there are values, they are likely to vary a lot (large standard deviation). It is vibration data. CREATE TABLE fct.MeasureV2 ( MeasureV2ID bigint NOT NULL IDENTITY(1, 1), DT1ModelID int NOT NULL CONSTRAINT DF_MeasureV2_DT1ModelID DEFAULT ((0)), DT1TID int NOT NULL CONSTRAINT DF_MeasureV2_DT1TID DEFAULT ((0)), TimeStampUTC datetime NOT NULL, AChkSum bigint NULL, OChkSum bigint NULL, SChkSum bigint NULL, ATp numeric (18, 10) NULL, ATpAvg numeric (18, 10) NULL, AWDAbsAvg numeric (18, 10) NULL, G1TMF numeric (18, 10) NULL, G2Ps_1TMF numeric (18, 10) NULL, G2Ps_2TMF numeric (18, 10) NULL, /* about 600 more numeric (18, 10) NULL columns */ WdSdv numeric (18, 10) NULL, UpdatedDate datetime NOT NULL, UpdatedID bigint NULL, IsCorrected bit NOT NULL CONSTRAINT DF_MeasureV2_IsCorrected DEFAULT ((0)), TRATransformJobID int NOT NULL ) ON PS_FctMeasure (TimeStampUTC) GO CREATE CLUSTERED COLUMNSTORE INDEX CCI_MeasureV2 ON fct.MeasureV2 ON PS_FctMeasure (TimeStampUTC) GO There are a lot of columns, but the alternative attribute-value table would contain >200 billion rows, and be almost impossible to query. This format is slow, but not too slow. We've also tried SPARSE columns, but that took a lot of disk space.
Asked by Henrik Staun Poulsen (2291 rep)
Aug 29, 2022, 12:08 PM
Last activity: Nov 5, 2024, 10:11 AM