Skip to content

[feat](Iceberg)Rest & S3Table Support Iam-role#60498

Open
CalvinKirs wants to merge 1 commit intoapache:masterfrom
CalvinKirs:master-rest-iceberg-iam-role
Open

[feat](Iceberg)Rest & S3Table Support Iam-role#60498
CalvinKirs wants to merge 1 commit intoapache:masterfrom
CalvinKirs:master-rest-iceberg-iam-role

Conversation

@CalvinKirs
Copy link
Member

@CalvinKirs CalvinKirs commented Feb 4, 2026

FYI #59893

The initial implementation was contributed by https://github.com/Sbaia . Many thanks for his contributions.

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@CalvinKirs
Copy link
Member Author

run buildall

@hello-stephen
Copy link
Contributor

FE UT Coverage Report

Increment line coverage 69.39% (34/49) 🎉
Increment coverage report
Complete coverage report

@doris-robot
Copy link

TPC-H: Total hot run time: 31840 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e3ae91676c5ddb5f65020c808185ce5b0f76cee1, data reload: false

------ Round 1 ----------------------------------
q1	17616	5306	5074	5074
q2	1993	306	210	210
q3	10226	1287	756	756
q4	10214	873	316	316
q5	7560	2191	1914	1914
q6	203	194	150	150
q7	883	741	607	607
q8	9276	1400	1089	1089
q9	5356	4843	4898	4843
q10	6841	1942	1573	1573
q11	493	303	287	287
q12	382	382	234	234
q13	17768	4096	3234	3234
q14	248	241	219	219
q15	938	833	811	811
q16	698	671	615	615
q17	649	864	424	424
q18	6737	6488	6403	6403
q19	1291	1140	652	652
q20	404	350	240	240
q21	2631	2124	1912	1912
q22	360	322	277	277
Total cold run time: 102767 ms
Total hot run time: 31840 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5351	5293	5261	5261
q2	277	332	250	250
q3	2149	2732	2272	2272
q4	1361	1734	1301	1301
q5	4286	4160	4226	4160
q6	225	181	137	137
q7	2335	2080	1847	1847
q8	2592	2551	2409	2409
q9	7463	7381	7613	7381
q10	2884	3064	2681	2681
q11	551	488	442	442
q12	680	761	620	620
q13	3938	4544	3609	3609
q14	299	341	286	286
q15	882	821	825	821
q16	692	714	656	656
q17	1205	1312	1335	1312
q18	8007	7818	7906	7818
q19	905	878	880	878
q20	2122	2237	2018	2018
q21	4720	4160	4081	4081
q22	563	546	494	494
Total cold run time: 53487 ms
Total hot run time: 50734 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 28.31 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e3ae91676c5ddb5f65020c808185ce5b0f76cee1, data reload: false

query1	0.05	0.04	0.05
query2	0.09	0.05	0.05
query3	0.25	0.08	0.09
query4	1.61	0.11	0.11
query5	0.27	0.25	0.24
query6	1.16	0.66	0.67
query7	0.03	0.02	0.03
query8	0.05	0.04	0.05
query9	0.55	0.50	0.48
query10	0.55	0.55	0.54
query11	0.13	0.10	0.10
query12	0.14	0.10	0.11
query13	0.64	0.61	0.62
query14	1.07	1.06	1.05
query15	0.87	0.85	0.87
query16	0.40	0.39	0.40
query17	1.14	1.14	1.14
query18	0.23	0.22	0.21
query19	2.01	1.93	1.96
query20	0.02	0.01	0.01
query21	15.40	0.26	0.15
query22	4.95	0.06	0.05
query23	15.81	0.30	0.12
query24	1.10	0.67	0.34
query25	0.08	0.12	0.07
query26	0.15	0.14	0.14
query27	0.06	0.06	0.06
query28	3.82	1.16	0.97
query29	12.58	3.95	3.18
query30	0.29	0.13	0.12
query31	2.82	0.64	0.40
query32	3.23	0.60	0.49
query33	3.28	3.27	3.21
query34	16.05	5.37	4.77
query35	4.81	4.84	4.75
query36	0.65	0.50	0.48
query37	0.11	0.07	0.06
query38	0.07	0.05	0.04
query39	0.04	0.03	0.03
query40	0.20	0.16	0.14
query41	0.08	0.04	0.03
query42	0.04	0.02	0.02
query43	0.05	0.04	0.04
Total cold run time: 96.93 s
Total hot run time: 28.31 s

@hello-stephen
Copy link
Contributor

FE Regression Coverage Report

Increment line coverage 0.00% (0/49) 🎉
Increment coverage report
Complete coverage report

@Sbaia
Copy link

Sbaia commented Feb 5, 2026

Thanks for picking this up and building on #59893, @CalvinKirs.

I want to clarify an important distinction about what this PR actually enables vs. what users on EKS/GKE typically need.

What this PR adds: cross-account assume-role support (AssumeRoleAwsClientFactory), where a role_arn is explicitly provided in the catalog properties.

What this PR does NOT add: support for IRSA (IAM Roles for Service Accounts) or, more generally, the AWS default credentials chain without any explicit credential configuration.

The validation in IcebergRestProperties (lines 128-137) explicitly rejects the case where neither access-key/secret-key nor role_arn is provided:

boolean hasAccessKeyAndSecret = ...;
boolean hasIamRole = ...;
return !hasAccessKeyAndSecret && !hasIamRole;  // fails validation

With IRSA (and future equivalents like GKE Workload Identity or ECS task roles), the pod already has credentials injected via the service account. The user doesn't have a role_arn to specify in the catalog DDL — the role is attached to the pod's ServiceAccount via the eks.amazonaws.com/role-arn annotation. The correct behavior is to let Iceberg use the AWS SDK's DefaultCredentialsProvider, which picks up the IRSA token automatically. This means accepting a catalog definition with no credential properties at all.

Regarding the assume-role path itself: AssumeRoleAwsClientFactory uses the AWS SDK's default credentials chain internally to obtain the base credentials needed to call sts:AssumeRole. So who actually performs the AssumeRole call depends on the environment:

  • EKS with IRSA configured: the pod's ServiceAccount role (via web identity token) is used as the base identity to assume the target role. This requires a trust policy on the target role that allows the SA role to assume it.
  • EKS without IRSA: falls back to the EC2 instance profile (node role via IMDS). The node role needs sts:AssumeRole permission on the target role. This is less secure since all pods on the node share the same base identity.
  • EC2 (non-Kubernetes): the instance profile role is used directly.

So the assume-role feature works, but it's solving a different use case (cross-account access) than what most EKS users need (direct IRSA credentials without specifying any role_arn).

Our original PR #59893 takes a simpler approach: when no explicit credentials are provided, we just don't set any credential properties, allowing Iceberg/AWS SDK to resolve credentials through the default chain. This covers IRSA, instance profiles, environment variables, and any future pod identity mechanism — with zero configuration in the catalog DDL.

I'd suggest either:

  1. Relaxing the validation to allow no credentials at all (supporting both IRSA and assume-role), or
  2. Keeping assume-role as an addition on top of [Enhancement](iceberg) Support AWS default credentials chain for S3 Tables and REST catalogs #59893's default-chain support

Happy to collaborate on merging the two approaches.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants