Spec: Materialized View Spec by JanKaul · Pull Request #11041 · apache/iceberg

JanKaul · 2024-08-29T12:25:26Z

This PR implements the Iceberg Materialized View Proposal #10043 by adding a section for Materialized Views to the View spec. It follows the design of the proposal document.

The idea is to resolve any remaining questions before starting the voting process on the dev list.

JanKaul · 2024-08-29T12:25:36Z

CC @rdblue @danielcweeks @RussellSpitzer @szehon-ho @wmoustafa @jackye1995 @emkornfield @findepi @stevenzwu

github-actions · 2024-11-12T00:15:04Z

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

github-actions · 2024-11-19T00:15:53Z

This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.

szehon-ho

I think this is the right direction, thanks @JanKaul. I think we should reopen. Also left some suggestions.

szehon-ho

Thanks, one more minor suggestion

devozerov · 2024-11-25T08:00:56Z

When materialized view is created, two entities would be added to the catalog - a view and a storage table. From the engine perspective it is important to expose it as a single object during listing. Are there any rules how catalog implementors should deal with these objects? E.g., shall we expose the view via listViews, but filter-out the storage table for listTables?

JanKaul · 2024-11-25T09:42:23Z

Yes correct, the idea is to filter-out the storage-table for the listTables operation. But I would regard this as not part of the table/view spec but should rather be included in the REST catalog specification.

Co-authored-by: Daniel Weeks <daniel.weeks@databricks.com>

- Define freshness as a universal property: storage table equals the result of the current view query (at the MV's current view-version-id) over the current state of dependencies. - Define dependencies objectively by parsing the SQL: base tables, views (transitively expanded), and intermediate materialized views (treated as their storage tables with recursive freshness via their own refresh-state). Drop the producer-relative dependency framing. - Reorganize the freshness section into producer flexibility and consumer options. Producer chooses what to record; consumer chooses what to verify. Producer-first ordering matches the chronological flow. - Tighten the schema: intermediate materialized views are recorded as a single table entry referencing the storage table; recording as a view entry is not permitted. - Crisp read-decision rule: if a consumer's assessment passes, it reads from the storage table; otherwise it evaluates the view query in place of the storage table. - Replace the "consumer/producer behavior" prose under Freshness with Producer flexibility and Consumer options subsections that match the three reference verification paths (recency policy, trust, verify). - Add Appendix B "What counts as a dependency" with the SQL-derived dependency rules and a worked example showing the recursive boundary for intermediate materialized views.

Co-authored-by: Daniel Weeks <daniel.weeks@databricks.com>

- make source table snapshot-id optional (empty source tables) - define ref semantics for branch/tag tracking vs pinned snapshots - note storage-table field is backward compatible for MV-unaware readers - fix C/D materialized view example and spec/typo nits

findinpath · 2026-10-15T13:12:36Z

+    } ],
+    "storage-table" : {
+      "namespace" : [ "default" ],
+      "name" : "event_agg_mv__storage"


it is worthwhile to point out explicitly in the text that it is the writer's responsibility to create the storage table event_agg_mv__storage.

findinpath · 2026-10-15T13:13:34Z

+
+### Strategy 2: Treat nested materialized views as tables
+
+Same as Strategy 1, but the query reads from materialized views. The producer stops at each MV boundary and records the MV's storage table snapshot ID and view version ID. No expansion beyond the MV.


the query reads from materialized views

the query reads from materialized views storage tables.

findinpath · 2026-10-15T13:15:27Z

+
+### Strategy 3: Treat nested materialized views as views
+
+Same as Strategy 1, but the query reads from materialized views. The producer treats each materialized view as a regular view: expand through the MV's view definition and record the underlying tables and views. The MV's storage table snapshot ID is **not** recorded.


but the query reads from materialized views

but the query expands recursively the view queries corresponding to the materialized views.

Generated-by: Claude Code Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

bennychow · 2026-10-15T16:05:12Z

+| _optional_  | `snapshot-id` | The snapshot-id of the source table that was read during the refresh operation. Omitted when the source table had no snapshots at refresh time (e.g. a newly created, empty table) |
+| _optional_  | `ref`         | Branch name of the source table being referenced in the view query |
+
+When `ref` is set, the source table is tracked against that branch or tag, and the materialized view is stale once the ref advances past the recorded `snapshot-id`. When `ref` is not set, the source table is pinned to the recorded `snapshot-id` (e.g. `FOR VERSION AS OF <snapshot-id>`); the materialized view remains fresh with respect to this source as long as that snapshot exists. Producers that read a branch should record the branch name in `ref`.


the materialized view remains fresh with respect to this source as long as that snapshot exists.

I think this is overly strong statement. It again tries to define "freshness".

findinpath · 2026-10-15T16:06:01Z

+
+### Strategy 5: Skip non-Iceberg dependencies
+
+The producer records only Iceberg sources and omits non-Iceberg dependencies entirely. Useful when the view query reads from a mix of Iceberg and non-Iceberg sources and the producer chooses to track only the Iceberg side.


producer -> writer

Generated-by: Claude Code

github-actions Bot added the Specification Issues that may introduce spec changes. label Aug 29, 2024

stevenzwu reviewed Aug 29, 2024

View reviewed changes

Comment thread format/view-spec.md Resolved

Comment thread format/view-spec.md Resolved

bennychow reviewed Sep 3, 2024

View reviewed changes

Comment thread format/view-spec.md Resolved

bennychow reviewed Sep 5, 2024

View reviewed changes

Comment thread format/view-spec.md Resolved

Comment thread format/view-spec.md Resolved

JanKaul force-pushed the materialized-view-spec branch from 62ba549 to b986cf0 Compare September 19, 2024 09:08

karuppayya reviewed Oct 2, 2024

View reviewed changes

This comment was marked as outdated.

Sign in to view

JanKaul force-pushed the materialized-view-spec branch from 90f517e to 2790b0d Compare October 4, 2024 14:13

JanKaul mentioned this pull request Nov 8, 2024

Iceberg Materialized Views #10043

Closed

3 of 6 tasks

github-actions Bot added the stale label Nov 12, 2024

github-actions Bot closed this Nov 19, 2024

szehon-ho reviewed Nov 21, 2024

View reviewed changes

szehon-ho reopened this Nov 21, 2024

JanKaul force-pushed the materialized-view-spec branch from 2790b0d to af76391 Compare November 21, 2024 10:53

szehon-ho reviewed Nov 21, 2024

View reviewed changes

Comment thread format/view-spec.md Resolved

github-actions Bot removed the stale label Nov 22, 2024

JanKaul added 8 commits November 25, 2024 08:33

add materialzied view to view spec

8c5d276

add uuid to source table

afc4a0d

Aktualisieren von view-spec.md

b2d0b68

improve refresh-state description

27783c7

remove identifier from refresh-state

e85ab16

fix comments

cff9596

incorporate comments

a8b52b2

fix MV introduction

521477f

JanKaul force-pushed the materialized-view-spec branch from af76391 to 521477f Compare November 25, 2024 07:33

JanKaul and others added 20 commits October 15, 2026 08:29

Update format/view-spec.md

62bc95d

Co-authored-by: Daniel Weeks <daniel.weeks@databricks.com>

fix language

71d4a18

fix optional refresh-state

8d1f9b3

fix freshness

3bdbc03

update diamond pattern

d4a7573

add non-deterministic functions

963f749

update dependency definition

a4b5bca

fixes

5bda6b3

update refresh-state

71e5574

Update format/view-spec.md

b945dd3

Co-authored-by: Daniel Weeks <daniel.weeks@databricks.com>

Update format/view-spec.md

77de26d

Co-authored-by: Daniel Weeks <daniel.weeks@databricks.com>

fixes

5fcd9ae

add wmoustafas appendix

9305cdb

fix appendix

c067582

fix source table definition

bb9ff16

make source table/view catalog field "required"

84537a8

add catalog field to refresh-state example

ef7e95d

remove freshness

ba28913

JanKaul force-pushed the materialized-view-spec branch from f14886b to ba28913 Compare October 15, 2026 06:29

findinpath reviewed Oct 15, 2026

View reviewed changes

clarify how nested MV strategies read materialized views

746fde4

Generated-by: Claude Code Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

bennychow reviewed Oct 15, 2026

View reviewed changes

findinpath reviewed Oct 15, 2026

View reviewed changes

JanKaul added 2 commits October 29, 2026 16:59

update appendix

a63c8a5

fix truncated sentences in materialized view appendix

75cb171

Generated-by: Claude Code

JanKaul force-pushed the materialized-view-spec branch from f1ef761 to 75cb171 Compare October 29, 2026 19:26


		### Strategy 2: Treat nested materialized views as tables

		Same as Strategy 1, but the query reads from materialized views. The producer stops at each MV boundary and records the MV's storage table snapshot ID and view version ID. No expansion beyond the MV.


		### Strategy 3: Treat nested materialized views as views

		Same as Strategy 1, but the query reads from materialized views. The producer treats each materialized view as a regular view: expand through the MV's view definition and record the underlying tables and views. The MV's storage table snapshot ID is not recorded.


		### Strategy 5: Skip non-Iceberg dependencies

		The producer records only Iceberg sources and omits non-Iceberg dependencies entirely. Useful when the view query reads from a mix of Iceberg and non-Iceberg sources and the producer chooses to track only the Iceberg side.

Conversation

JanKaul commented Aug 29, 2024

Uh oh!

JanKaul commented Aug 29, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

github-actions Bot commented Nov 12, 2024

Uh oh!

github-actions Bot commented Nov 19, 2024

Uh oh!

szehon-ho left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

szehon-ho left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

devozerov commented Nov 25, 2024

Uh oh!

JanKaul commented Nov 25, 2024

Uh oh!

findinpath Oct 15, 2026

Choose a reason for hiding this comment

Uh oh!

findinpath Oct 15, 2026

Choose a reason for hiding this comment

Uh oh!

findinpath Oct 15, 2026

Choose a reason for hiding this comment

Uh oh!

bennychow Oct 15, 2026

Choose a reason for hiding this comment

Uh oh!

findinpath Oct 15, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants