[refactor](paimon) Per-catalog Paimon metadata cache with two-level table+snapshot structure#60478
Open
suxiaogang223 wants to merge 11 commits intoapache:masterfrom
Open
[refactor](paimon) Per-catalog Paimon metadata cache with two-level table+snapshot structure#60478suxiaogang223 wants to merge 11 commits intoapache:masterfrom
suxiaogang223 wants to merge 11 commits intoapache:masterfrom
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run external |
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
Author
|
run external |
Contributor
FE Regression Coverage ReportIncrement line coverage |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Summary
Refactor Paimon metadata cache from a single global instance to per-catalog instances,
introduce a two-level Table+Snapshot cache structure, and unify TTL resolution logic
across Iceberg/Paimon/Schema caches.
Motivation
The previous design shared a single
PaimonMetadataCacheinstance and a single globalsnapshotCacheacross all Paimon catalogs. This caused:catalogIdfor isolation; invalidation required scanning allkeys and filtering.
PaimonExternalTableeagerly fetched theTableobject at construction time, incurringremote calls even when the table was never subsequently accessed.
Changes
Per-catalog cache instantiation
CatalogScopedCacheMgr<T>, a generic catalog-keyed cache manager backed byConcurrentHashMap.computeIfAbsent.PaimonMetadataCacheMgrnow extendsCatalogScopedCacheMgr<PaimonMetadataCache>;each catalog owns an independent
PaimonMetadataCache.icebergCacheMap(previously hand-rolled double-checked locking)in
ExternalMetaCacheMgrto the sameCatalogScopedCacheMgr.Two-level table + snapshot cache
snapshotCachewithtableCache(key:
PaimonTableCacheKey, value:PaimonTableCacheValue).PaimonTableCacheValueholds the PaimonTableobject and lazy-loadsPaimonSnapshotCacheValuevia double-checked locking.Tableobject is managed by CaffeineLoadingCache, subject to TTL/maxSize.When TTL expires, Caffeine creates a new
PaimonTableCacheValue, and the snapshotis re-lazily-loaded on next access.
tableCachedirectly; MTMV queries go through the explicitsnapshot path; branch queries call the Paimon catalog directly (branches have
independent schemas, not suitable for the main table cache).
Unified TTL resolution
ExternalCatalog.resolveCacheTtlSpec()to centralize TTL property parsing:null→ use global default (external_cache_expire_time_seconds_after_access)-1→ no expiry (Caffeine does not setexpireAfterAccess)0→ disable cache (maxSize=0, Caffeine evicts immediately)>0→ use asexpireAfterAccessIcebergMetadataCache,PaimonMetadataCache, andExternalSchemaCache.paimon.table.meta.cache.ttl-secondcatalog property with validation incheckProperties().ALTER CATALOG SETtriggers cache reinitialization vianotifyPropertiesUpdated(), consistent with Iceberg's existing pattern.Lazy table access in PaimonExternalTable
paimonTablefield from the constructor.Tableobject access now goes throughPaimonUtils→PaimonMetadataCache.tableCache, making it lazy and cache-aware.PaimonUtilsas the centralized static accessor for Paimon cacheoperations, simplifying call sites.
Iceberg invalidation fast path
IcebergMetadataCache.invalidateTableCache()now attempts a direct key lookup(
getIfPresent) first. On hit, invalidate immediately; on miss, fall back to fullcache scan matching by local name. Avoids unnecessary iteration on the common path.
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)