Teradata Multivalue Compression and Presence Bytes for NULL Values

Multivalue compression and null indication serve distinct purposes yet share a common objective of storing information about the values of each row's columns. To conserve space, Teradata utilizes presence bytes to retain information on multivalue compression and nullability.

Readers familiar with earlier releases will recognize that the name "presence byte" originally referred only to NULL values. This naming convention was introduced when only nullability information was stored, before multivalue compression existed.

Each row is always equipped with at least one presence byte. More presence bytes are added if needed. Only 7 bits of the first presence byte store information about nullability or multivalue compression, as the system occupies the first bit (and it's always set to 1).

Want more practical data engineering analysis like this?

Join DWHPro Letters and get field-tested notes on Teradata, Snowflake, AI, migrations, performance, and enterprise data work. Early subscribers keep launch access before the paid plan launches.

Get the next issue

How is Teradata storing Information about NULL Values?

One bit per row finds NULL values for each nullable column. The bits used to show nullability are always stored in the bits used by the multivalue compression feature (which means the bits are used from right to left). Of course, columns defined as NOT NULL don't need the nullability indicator bit.

The logic of indicating NULL values is simple:

• If the presence bit is set to 1, a NULL value is present in the column.
• If the presence bit is set to 0, any other value is present in the column.

See below an example table containing eight rows. As TheCol is nullable, an indicator bit is used to show rows containing NULL values in TheCol (as mentioned earlier, the first bit is always set to 1 as the system utilizes it):

We already said that only 7 bits of the first presence byte could be used. We can cover seven nullable columns with the always-existing presence byte. Each additional presence byte can store the null value information of 8 more nullable columns.

Get the next issue by email.

An extra presence byte is added if we increase the number of nullable columns from 7 to 8. As a result, each row consumes one more byte of permanent disk space.
Keep the following limits in mind when designing your tables:

Up to 7   nullable columns -> The already existing presence byte is used.
Up to 15 nullable columns -> 1 presence byte is added (2 in total)
Up to 23 nullable columns -> 2 presence bytes are added (3 in total)
...

To minimize table size, it is advisable to define columns as NOT NULL whenever feasible. The multivalue compression feature can also utilize unused bits in a presence byte.

Teradata Multivalue Compression and the Presence Bytes

Presence bytes are also used in multivalue compression. For each column, up to 255 different values can be compressed.

The original column values are kept in the table header, and each row will only hold the binary number representing its corresponding column value. If we store 255 different values, 8 bits are needed (a total presence byte).

The algorithm will only occupy as many bits as needed to encode the original column value:

Let's do another example. The below table has eight rows. If we compress 3 column values 'A','B','C', we need two presence bits: 'A' could be represented by the binary number 01,  'B' by binary 10, 'C' by binary 11 (the presence bits are occupied from right to left):

We extend our example by making TheCol additionally nullable. In this case, one presence bit is needed to show rows containing NULL values in column TheCol. The compression is unchanged, just shifted by one bit to the left, as nullability bits are always stored first:

When we compress at least one value of TheCol, NULL values are automatically compressed without any more cost. The column's nullability bit can be used for NULL values. No encoding and storing in the table header is required.

To decrease space usage, employing identical presence bytes for multivalue compression and nullability is possible. However, it's crucial to consider the combined expenses of compressed columns and nullable indicators when constructing tables and how adding another compressed value or nullable column may impact table size.

Planning or surviving an enterprise data platform migration?

I write regularly about the performance, cost, architecture, and project mistakes that show up in real Teradata, Snowflake, Databricks, and enterprise data work.

Subscribe before the paid plan launches and keep launch access.

Written by Roland Wenzlofsky, founder of DWHPro and author of Teradata Query Performance Tuning. DWHPro has helped data warehouse practitioners for 15+ years.