Codecov Report

Patch coverage has no change and project coverage change: +23.35 :tada:

Comparison is base (1a5e0af) 29.30% compared to head (6b985c6) 52.65%.

@@              Coverage Diff              @@
##             master    #1287       +/-   ##
=============================================
+ Coverage     29.30%   52.65%   +23.35%     
+ Complexity     5385      522     -4863     
=============================================
  Files           695       43      -652     
  Lines         70823     3705    -67118     
  Branches       8182      354     -7828     
=============================================
- Hits          20752     1951    -18801     
+ Misses        48065     1623    -46442     
+ Partials       2006      131     -1875     

| Flag | Coverage Δ | | |---|---|---| | core | ? | | | trino | 52.65% <ø> (-0.11%) | :arrow_down: |

Flags with carried forward coverage won't be shown. Click here to find out more.

see 653 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Why are the changes needed?

fix #1145 , fix #1057

Brief change log

• create and shutdown ExecutorService in OptimizeService for each operation separately

How was this patch tested?

• [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

• [ ] Add screenshots for manual tests if appropriate

• [x] Run test locally before making a pull request

Documentation

• Does this pull request introduces a new feature? (yes / no)
• If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

Why are the changes needed?

fix bugs in PR: #1145

Brief change log

avoid memroy leak

How was this patch tested?

• [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

• [ ] Add screenshots for manual tests if appropriate

• [x] Run test locally before making a pull request

Documentation

• Does this pull request introduces a new feature? (no)
• If yes, how is the feature documented? (not documented)

Why are the changes needed?

fix, #1242

Brief change log

• optimizing-check thread pool skips pending tables and only changes optimizing status from idle to pending
• optimizing-planner thread skips idle tables and only plans tables in pending

How was this patch tested?

• [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

• [ ] Add screenshots for manual tests if appropriate

• [x] Run test locally before making a pull request

Documentation

• Does this pull request introduces a new feature? (yes / no)
• If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

[ARCTIC-1242][ams] Optimizing plan thread should only plan tables in pending status (#1284)

• plan thread only handle pending tables

• remove useless log

After troubleshooting, gc found no exceptions.

The most time consuming methods is com.netease.arctic.ams.server.optimize.AbstractIcebergOptimizePlan#buildOptimizeTask

There is access to HDFS within the method.

Why are the changes needed?

This PR was inspired by related issue #1057 , but was not created for implementation.

AMS can not handle large scale Iceberg tables which has a certain number of small files/delete files. Some of tables in iceberg catalog are not optimizing enabled, so these tables are not nessessary for AMS to scan or load into memory. I assume that self-optimizing disabled tables allocate a mount of memory and compare the GC times after remove them:

• Load all valid tables:
• Only load self-optimizing tables:

The below sreenshot lists number of instances and retained heap per class:

I also dump it and find that ConcurrentHashMap in OptimizeService retained too much heap

• case 1: Too many snapshots entry

• case 2: Too many delete files

Brief change log

• Reduce AMS memory
• Avoid load/display useless tables for AMS optimize
• Clean the tasks when self-optimizing disabled

How was this patch tested?

• [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

• [ ] Add screenshots for manual tests if appropriate

• [x] Run test locally before making a pull request

Documentation

• Does this pull request introduces a new feature? (no)
• If yes, how is the feature documented? (not documented)

[ARCTIC-1057][AMS] Only load self-optimizing enabled tables into cache (#1145)

• [Optimize] Only load self-optimizing enabled tables into cache

• checkstyle

• fix log level

• refactor code base on the review

• load un-optimized table into db

• [Optimize] Only load self-optimizing enabled tables into cache

• checkstyle

• fix log level

• refactor code base on the review

• import CatalogLoader

• checkstyle

• avoid insert duplicate record into sysdb

• [WAP] remove table from unOptimizeTables when clear table

• rewrite base on code review

• rewrite base on code review

Co-authored-by: Xavier Bai xuba@cisco.com Co-authored-by: luting 1004611953@qq.com Co-authored-by: ZhouJinsong zhoujinsong0505@163.com

Verified in a local scenario, running from 17:00 ~ 17:50 for 50 minutes, the table in Pending status went from 70+ to 20+. The utilization rate is now ~36%.

Codecov Report

Patch coverage has no change and project coverage change: +23.51 :tada:

Comparison is base (0e5140e) 29.27% compared to head (a18030a) 52.79%.

@@              Coverage Diff              @@
##             master    #1285       +/-   ##
=============================================
+ Coverage     29.27%   52.79%   +23.51%     
+ Complexity     5374      525     -4849     
=============================================
  Files           695       43      -652     
  Lines         70786     3705    -67081     
  Branches       8180      354     -7826     
=============================================
- Hits          20723     1956    -18767     
+ Misses        48058     1621    -46437     
+ Partials       2005      128     -1877     

| Flag | Coverage Δ | | |---|---|---| | core | ? | | | trino | 52.79% <ø> (+0.10%) | :arrow_up: |

Flags with carried forward coverage won't be shown. Click here to find out more.

see 652 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Why are the changes needed?

Add a scala code style check rule. fix #1059

Brief change log

• Add scala code checkstyle rule
• Fix scala code to meet checkstyle rule

How was this patch tested?

• [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

• [ ] Add screenshots for manual tests if appropriate

• [X] Run test locally before making a pull request

Codecov Report

Patch coverage has no change and project coverage change: +23.49 :tada:

Comparison is base (0e5140e) 29.27% compared to head (4ab7273) 52.76%.

@@              Coverage Diff              @@
##             master    #1284       +/-   ##
=============================================
+ Coverage     29.27%   52.76%   +23.49%     
+ Complexity     5374      524     -4850     
=============================================
  Files           695       43      -652     
  Lines         70786     3705    -67081     
  Branches       8180      354     -7826     
=============================================
- Hits          20723     1955    -18768     
+ Misses        48058     1621    -46437     
+ Partials       2005      129     -1876     

| Flag | Coverage Δ | | |---|---|---| | core | ? | | | trino | 52.76% <ø> (+0.08%) | :arrow_up: |

Flags with carried forward coverage won't be shown. Click here to find out more.

see 653 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Why are the changes needed?

fix, #1242

Brief change log

• optimizing-check thread pool skips pending tables and only changes optimizing status from idle to pending
• optimizing-planner thread skips idle tables and only plans tables in pending

How was this patch tested?

• [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

• [ ] Add screenshots for manual tests if appropriate

• [x] Run test locally before making a pull request

Documentation

• Does this pull request introduces a new feature? (yes / no)
• If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

What happened?

1、 we define a kafka source with canal-json format, and add a sink to arctic. 2、The task failed with a exception thrown.

imo, The parallelism of Flinksink Operator depends on the parallelism of previous operator. In the scene, parallelism of previsous operator is -1, so we encounted the exception.

Affects Versions

0.3.2

What engines are you seeing the problem on?

Flink

How to reproduce

1、create a sql task like 2、if we run the task, we got the exception:

Relevant log output

No response

Anything else

No response

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

Codecov Report

Patch coverage has no change and project coverage change: +23.38 :tada:

Comparison is base (8822476) 29.27% compared to head (7d422a3) 52.65%.

@@              Coverage Diff              @@
##             master    #1282       +/-   ##
=============================================
+ Coverage     29.27%   52.65%   +23.38%     
+ Complexity     5369      522     -4847     
=============================================
  Files           695       43      -652     
  Lines         70769     3705    -67064     
  Branches       8177      354     -7823     
=============================================
- Hits          20720     1951    -18769     
+ Misses        48047     1623    -46424     
+ Partials       2002      131     -1871     

| Flag | Coverage Δ | | |---|---|---| | core | ? | | | trino | 52.65% <ø> (-0.11%) | :arrow_down: |

Flags with carried forward coverage won't be shown. Click here to find out more.

see 653 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Brief change log

1.Tables menu - view the tables under the database from the hover event to the click event 2.Tables menu background color modification and adaptation

How was this patch tested?

• [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

• [ ] Add screenshots for manual tests if appropriate

• [ ] Run test locally before making a pull request

Documentation

• Does this pull request introduces a new feature? (no)
• If yes, how is the feature documented? (not documented)

Search before asking

• [X] I have searched in the issues and found no similar issues.

What would you like to be improved?

1.Tables menu - view the tables under the database from the hover event to the click event 2.Tables menu background color modification and adaptation

How should we improve?

1.Improve the experience of viewing table information

Are you willing to submit PR?

• [X] Yes I am willing to submit a PR!

Subtasks

No response

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

I suggest to change the title to

Support terminal access hive table when switch to a catalog with HMS metastore .

Because that is the real problem that we want to solve, and using ArcticSparkSessionCatalog is a feasible way to do this.