NetEase
Repos
78

Arctic is a streaming lake warehouse service open sourced by NetEase

285
77

Events

closed issue
[Feature][AMS]:support add/modify by AMS Dashboard

Description

support add/modify setting by AMS Dashboard

Use case/motivation

now ams just support add/modify setting by config file, users cannot flexibly observe and modify the configuration, so we should support add/modify setting by AMS Dashboard

Describe the solution

support add/modify setting by AMS Dashboard

Related issues

none

Are you willing to submit a PR?

  • [X] Yes I am willing to submit a PR!

Code of Conduct

Created at 3 hours ago
started
Created at 3 hours ago
started
Created at 4 hours ago
Created at 11 hours ago
Created at 12 hours ago
started
Created at 14 hours ago
pull request opened
[Spark] Feature spark 2.3 double flout write

Why are the changes needed?

Feature spark 2.3 double flout write

Brief change log

(For example:)

  • Add spark 2.3 insert logical plan
  • Add cast types in insert plan
  • Add complex type test

How was this patch tested?

  • [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

  • [ ] Add screenshots for manual tests if appropriate

  • [ ] Run test locally before making a pull request

Documentation

  • Does this pull request introduces a new feature? (yes )
  • If yes, how is the feature documented? (not documented)
Created at 16 hours ago
opened issue
[Improvement]: Relax part of the check in AMS-Server

Search before asking

  • [X] I have searched in the issues and found no similar issues.

What would you like to be improved?

A request carrying a token only needs to determine whether the token is empty

How should we improve?

change code

Are you willing to submit PR?

  • [X] Yes I am willing to submit a PR!

Code of Conduct

Created at 19 hours ago
issue comment
[ARCTIC-360][AMS] Refine table optimize_task_history from partition level to task level

LGTM

Created at 20 hours ago
issue comment
[Feature]: Automatic toggle switch to enable double write

After discussing offline with @zhoujinsong, we will save the flag whether the logstore data has caught up in the arctic table properties. log-store.catch-up and log-store.catch-up-timestamp

Created at 20 hours ago
opened issue
[Improvement]: Integrate the Flink table propereties when the Arctic table refresh method is invoked

Search before asking

  • [X] I have searched in the issues and found no similar issues.

What would you like to be improved?

As of now, the Flink table properties should combine the flink properties and arctic table properties. And the flink properties will replace the same key of the arctic table properties.

tableLoader = ArcticTableLoader.of(PK_TABLE_ID, catalogBuilder);

    tableLoader.open();
    ArcticTable arcticTable = tableLoader.loadArcticTable();

    UpdateProperties updateProperties = arcticTable.updateProperties();
    updateProperties.set(LOG_STORE_CATCH_UP, String.valueOf(LOG_STORE_CATCH_UP_DEFAULT));
    updateProperties.set(LOG_STORE_CATCH_UP_TIMESTAMP, String.valueOf(LOG_STORE_CATCH_UP_TIMESTAMP_DEFAULT));
    updateProperties.commit();

    arcticTable.refresh();
    Map<String, String> properties = arcticTable.properties();
    Assert.assertSame("" + LOG_STORE_CATCH_UP_DEFAULT, properties.get(LOG_STORE_CATCH_UP));
    Assert.assertSame("" + LOG_STORE_CATCH_UP_TIMESTAMP_DEFAULT, properties.get(LOG_STORE_CATCH_UP_TIMESTAMP));

How should we improve?

Add a new proxy method(refresh) of the arctic table in the FlinkTablePropertiesInvocationHandler

Are you willing to submit PR?

  • [X] Yes I am willing to submit a PR!

Code of Conduct

Created at 20 hours ago
issue comment
[ARCTIC-360][AMS] Refine table optimize_task_history from partition level to task level

we could add expire and clean logic for both optimize_history and optimize_task_history, in order to fix #268.

The retention time, 30 day fro optimize_history, 7 day for optimize_task_history may be appropriate.

ok.

Created at 21 hours ago