Sorting and Grouping07. Grouping

Authors Authors: Egor Rogov, Pavel Luzanov, Ilya Bashtanov

Photo by: Oleg Bartunov (Phu monastery, Bhrikuti summit, Nepal)

Use of course materials

Non-commercial use of course materials (presentations, demonstrations) is

allowed without restrictions. Commercial use is possible only with the written

permission of Postgres Professional. It is prohibited to make changes to the

course materials.

Feedback

Please send your feedback, comments and suggestions to:

edu@postgrespro.ru

Disclaimer

In no event shall Postgres Professional company be liable for any damages

or loss, including loss of profits, that arise from direct or indirect, special or

incidental use of course materials. Postgres Professional company

specifically disclaims any warranties on course materials. Course materials

are provided “as is,” and Postgres Professional company has no obligations

to provide maintenance, support, updates, enhancements, or modifications.

Topics

Grouping Application

Hash-based grouping

Sorting-based grouping

Hybrid grouping

Grouping in Parallel Execution Plans

Window functions

Grouping Application

Explicit Indication

GROUP BY

Removing Duplicates

DISTINCT

UNION, INTERSECT, EXCEPT

In a SQL query, you can group a set of rows using the GROUP BY clause,

which is typically used with aggregate functions that process all rows in the

group. The PARTITION BY clause provides a similar capability for window

functions.

Grouping can occur implicitly when duplicates are removed in a query using

the DISTINCT keyword or in set operations (UNION, INTERSECT, EXCEPT

without the ALL keyword), which eliminate duplicates.

The server chooses one of the available grouping methods, considering

factors such as resource availability, the ability to retrieve sorted data for the

grouping key, and other factors.

Hash-based grouping

Yellow Submarine

Abbey Road

The Beatles

3 Let It Be

1969

1968

1970

id title year

Basket 00

010001 00

Bucket 11

SELECT year, count(*)

cntFROM albumsGROUP BY year;

Bucket 10

Bucket 01

1970

counter

The concept of hashing was discussed in the "Types of Indexes"

section.When grouping values, a similar approach can be used.

The example on the slide demonstrates how the COUNT aggregate

function's value is calculated during grouping by the year column. The server

receives and processes the table rows in turn.

The number of buckets is precomputed as a power of two, and the number

of bits in the hash code that represent the bucket number is determined. In

the example on the slide, the last two bits are used.

The hash function is computed for the row's year value. The bucket number

is determined by the hash code, and a record with the hash code is added to

the hash table if it is not already present.

Since the count aggregate function's value is calculated in our example,

each hash table entry also stores a counter that increments as each row is

processed.

In our example, the first row creates a new entry in bucket 00.

SELECT year, count(*)

cntFROM albumsGROUP BY year;

Basket 00

Bucket 11

Bucket 10

Bucket 01

Hash-based grouping

Yellow Submarine

Abbey Road

The Beatles

3 Let It Be

1969

1968

1970

id title year

100010 01 1

1970

1969

010001 00

The second string is added to bucket 01, generating a new record.

Basket 00

Bucket 11

Bucket 10

Bucket 01

Hash-based grouping

Yellow Submarine

Abbey Road

The Beatles

3 Let It Be

1969

1968

1970

id title year

010001 00 1

100010 01 1

110001 11 1

1970

1969

1968

SELECT year, count(*)

cntFROM albumsGROUP BY year;

The third line creates a new record in bucket 11.

Basket 00

Bucket 11

Bucket 10

Bucket 01

Hash-based grouping

Yellow Submarine

Abbey Road

The Beatles

3 Let It Be

1969

1968

1970

id title year

110001 11 1

1970

1969

1968

010001 00

100010 01

SELECT year, count(*)

cntFROM albumsGROUP BY year;

The hash code for the last row is assigned to bucket 01. A record with the

same key value already exists in the bucket, so the server increases its

counter by one. The table processing is complete at this point: the hash

table now contains all unique values. The algorithm returns the values and

their counts.

The same algorithm, with minor modifications, is used both for retrieving

unique records without an aggregate function and for computing other

aggregate functions.

Basket 00

Bucket 11

Bucket 10

Bucket 01

Hash-Based Grouping and

Packets

Yellow Submarine

Abbey Road

The Beatles

3 Let It Be

1969

1968

1970

id title year

01011 0 11 1

work_mem × hash_mem_multiplier =

= 4MB × 2

1965

1969

1966

01010 0 00

10001 0 01

Help!

Revolver

A Hard Day’s Night

2 Please Please Me

1965

1966

1964

1963

00110 1 10

01000 1 00

1964

1970

11000 1 11 1968

Package 1

11010 0 10 11963

If the allocated memory is insufficient, the original dataset is split into

packets to ensure each packet's hash table fits in random access memory.

The expected number of packets is a power of 2; the hash code reserves

the necessary number of bits to determine which packet a record belongs to

(since there are two packets on the slide, one bit suffices). The hash table

for packet 0 resides in random access memory, while the rows of other

packets are stored in temporary files — each packet in its own file.

Using temporary files significantly reduces performance, and this effect is

more pronounced in hashing than in other algorithms. Therefore, during

hash-based grouping, [work_mem] × hash_mem_multiplie of random access

memory is allocated — by default, twice the usual amount.

Hash-Based Grouping and

Packets

work_mem × hash_mem_multiplier

00110 1 10

01000 1 00

1964

1970

11000 1 11 1968

Package 1

Basket 00

Bucket 11

Bucket 10

Bucket 01

11000 1 11 1

1970

1968

01000 1 00

00110 1 10 11964

Then, each package stored in temporary files is loaded into memory to

create a new hash table.

Once all original rows and additional data chunks have been processed, the

server may return partial results, with each group's rows always contained

within a single package.

Sorting-based grouping

1968

1969

1970

SELECT DISTINCT year

FROM albums;

Yellow Submarine

Abbey Road

The Beatles

3 Let It Be

1969

1968

1970

id title year

1969

1968

1970

quantity

SELECT year, count(year)FROM

albumsGROUP BY year;

3 Let It Be 1970

dataset

sorted

Duplicate elimination and grouping can be achieved not only via hashing but

also through sorting. In this case, a dataset must be pre-sorted by the

grouping fields.

The Unique node removes duplicates by appending the next value to the

result when it differs from the previous one

The GroupAggregate node outputs aggregated data In the example on the

slide, the node creates a list of unique year values and counts how often

each value occurs in the dataset.

As discussed in the previous section, sorted rows can be obtained via index

access or through sorting in the Sort node. In the first case, the Unique and

GroupAggregate nodes are typically more efficient than HashAggregate

because they run quickly and use less memory. However, in the second

case, hash table construction is generally more efficient than sorting.

Unlike HashAggregate, both the Unique and GroupAggregate nodes return

sorted data.

In parallel execution plans

Gather

Partial

HashAggregate

Parallel

SeqScan

Finalize

HashAggregate

Partial

HashAggregate

Parallel

SeqScan

Gather

Merge

Finalize

GroupAggregate

Partial

HashAggregate

Parallel

SeqScan

Sort

Partial

HashAggregate

Parallel

SeqScan

Sort

Aggregation doesn't have a dedicated parallel implementation, but can be

executed in parallel execution plans. In this case, each worker process

aggregates its portion of the data and passes the results to a Gather node,

which combines all the data into a single set. The next node handles the

aggregation of the combined dataset and computes the result of the

aggregate function.

An example of this plan is shown on the left. This approach is typically more

effective when there are a sufficiently large number of groups.

If there are only a few groups, sorting the grouped data within each worker

process can be more efficient. Leverage the sorted data and perform a more

efficient grouping via sorting at the final stage.

An example of this plan is shown on the right slide.

Window functions

Grouping always relies on sorting

1 2 3 4 5 1 2 3 4 5

OVER (PARTITION BY) OVER (PARTITION BY ORDER BY)

Grouping is also used in window functions.

If the window definition includes the PARTITION BY clause, the window

function operates on groups of rows, similar to GROUP BY grouping.

For each group of rows, the frame boundaries vary within their respective

ranges. If no additional instructions are provided, the function will return the

same value for all rows within a group.

Similar to the example in the "Sorting" section, the ORDER BY clause sorts

the rows within each group; in this case, the frame is considered to include

rows from the first to the current one.

Unlike standard GROUP BY grouping, window definition grouping always

relies on sorting: the execution plan is designed to ensure that the

WindowAgg node receives a set of rows sorted first by the PARTITION BY

keys and then by the ORDER BY keys. Hashing is not used even without an

ORDER BY clause.

Takeaways

Grouping occurs not only during aggregation, but also during

duplicate elimination.

Performed using hashing or sorting

Practice

1. How does the query calculate the number of seats per category in

the seats table?Try using the default parameter values, and then

disable hash-based grouping.

2. Examine the query using a window function and the

PARTITION BY clause:

SELECT status, count(*) OVER (PARTITION BY

status)FROM flightsWHERE flight_no = 'PG0007'AND

departure_airport = 'VKO'AND flight_id BETWEEN

24104 AND 24115;

Add a row_number function call using the same window and

compare the results and execution plans.

1. Query

GROUP BY fare_conditions;

To disable hash-based grouping, set the enable_hashagg parameter to off.