stigsb
Repos
36
Followers
40
Following
14

Kubernetes Continuous Deployment Toolkit

33
4

Varnish Request Timing Exporter for Prometheus

10
4

Events

started
Created at 1 week ago
issue comment
Roadmap 2022 (discussion)

intend to contribute most of it by the end of the year.

@stigsb I hope it is promise, not just declaration

It's our clear intent, things are just taking longer than planned!

Created at 1 week ago
issue comment
Roadmap 2022 (discussion)

@rafael81 MaterializedMySQL is implemented for ~90%, not 100%, and it is still experimental. There are many complaints about this feature and today no one is working on it. (Previous contributors have vanished)

The status is unclear. We will try to find a way to revive it, but there is no guarantee that it will happen.

My team has several engineers working on MaterializedMySQL full time, we have made substantial performance and robustness improvements and intend to contribute most of it by the end of the year.

Created at 1 week ago
issue comment
Add SETTING update_allow_materialized_columns

@tavplubix I believe everything should be as you requested now, care to take a final look?

Created at 4 weeks ago
create tag
stigsb create tag v1.0.4
Created at 1 month ago

CI: build docker release with --network=host

Created at 1 month ago

Fix darwin arm64 build

Created at 1 month ago
create tag
stigsb create tag v1.0.3
Created at 1 month ago

Update .goreleaser.yml

Created at 1 month ago
create tag
stigsb create tag v1.0.2
Created at 1 month ago

Update gopkg.in/yaml.v2

Created at 1 month ago
create tag
stigsb create tag v1.0.1
Created at 1 month ago

Moved ReplicatedMergeTree tests to a file with "zookeeper" in the name

Created at 1 month ago

Allow updating MATERIALIZED columns with update_allow_materialized_columns setting

This setting (if different from the default value) is persisted with mutations, to ensure they are picked up when the mutation is executed asynchronously or in replicated tables.

Compatibility warning: downgrading from a version of ClickHouse with "settings: " in mutations_*.txt to one that does not support it may cause mutations to fail loading. This happens if the newer version persisted active mutations using update_allow_materialized_columns=1 and the old version tries to load them.

Created at 1 month ago

better support of read_in_order in case of fixed prefix of sorting key

better tests and comments

NuRaft fix for install snapshot

Improve preprocessing performance

Fix style

Cache deltas for path

Add the %z format descriptor to formatDateTime

Add docs

Small fixes

Merge branch 'master' into fix-read-in-order-fixed-prefix

Merge branch 'master' into fix-install-snapshot

Fix uncommitted auth

avoid logical errors at some common operation with ColumnObject

Remove unnecessary backslashes in markdown sql blocks

Add tests

Metric for the number of detached parts

Fix timezone offset formatting

Minor: remove unnecessary ContextAccess::make() method

Add extra message about startup preprocessing

Don't remove from cache if it wasn't added during rollback

Created at 1 month ago
issue comment
Add SETTING update_allow_materialized_columns

But it's also possible to cause the same data consistency problems using INSERT query with insert_allow_materialized_columns and we don't have special INSERT_MATERIALIZED_COLUMN access privilege. Moreover, most users usually do not have ALTER_TABLE access, it's mostly for admins and other users who know what they are doing. So I think we don't need ALTER_UPDATE_MATERIALIZED_COLUMN access right.

Okay, I buy that argument :) Will remove.

Created at 1 month ago

Use GITHUB_TOKEN for docker login

Created at 1 month ago
create tag
stigsb create tag v1.0.0
Created at 1 month ago
delete tag
stigsb delete tag v1.1.0
Created at 1 month ago
delete tag
stigsb delete tag v1.2.0
Created at 1 month ago
create tag
stigsb create tag v1.2.0
Created at 1 month ago

Update libraries, include darwin/arm64 build

Created at 1 month ago
create tag
stigsb create tag v1.1.0
Created at 1 month ago

fix columnObject crash with array of variadic dimension elems

fix fold dimensions with null type

Merge branch 'master' into ch_canh_fix_json

wip

add selects

added config for backup dest

Merge branch 'master' into add-backup

add text

Merge branch 'master' into ch_canh_fix_json

Use index when row_policy_filter is always false

Fix possible error Attempt to read after eof in CSV schema inference

Fix

Merge branch 'master' into add-backup

feat: implement catboost in library-bridge

This commit moves the catboost model evaluation out of the server process into the library-bridge binary. This serves two goals: On the one hand, crashes / memory corruptions of the catboost library no longer affect the server. On the other hand, we can forbid loading dynamic libraries in the server (catboost was the last consumer of this functionality), thus improving security.

SQL syntax:

SELECT catboostEvaluate('/path/to/model.bin', FEAT_1, ..., FEAT_N) > 0 AS prediction, ACTION AS target FROM amazon_train LIMIT 10

Required configuration:

<catboost_lib_path>/path/to/libcatboostmodel.so</catboost_lib_path>

*** Implementation Details ***

The internal protocol between the server and the library-bridge is simple:

  • HTTP GET on path "/extdict_ping": A ping, used during the handshake to check if the library-bridge runs.

  • HTTP POST on path "extdict_request" (1) Send a "catboost_GetTreeCount" request from the server to the bridge, containing a library path (e.g /home/user/libcatboost.so) and a model path (e.g. /home/user/model.bin). Rirst, this unloads the catboost library handler associated to the model path (if it was loaded), then loads the catboost library handler associated to the model path, then executes GetTreeCount() on the library handler and finally sends the result back to the server. Step (1) is called once by the server from FunctionCatBoostEvaluate::getReturnTypeImpl(). The library path handler is unloaded in the beginning because it contains state which may no longer be valid if the user runs catboost("/path/to/model.bin", ...) more than once and if "model.bin" was updated in between. (2) Send "catboost_Evaluate" from the server to the bridge, containing the model path and the features to run the interference on. Step (2) is called multiple times (once per chunk) by the server from function FunctionCatBoostEvaluate::executeImpl(). The library handler for the given model path is expected to be already loaded by Step (1).

Fixes #27870

chore: delete obsolete modelEvaluate() function

  • superseded by catboostEvaluate() which no longer uses the internal repository for external models

  • also removed was statement SYSTEM RELOAD MODELS and the monitoring view SYSTEM.SYSTEMMODELS

style: fix style

fix: broken links in documentation (hopefully)

fix: test 02117_show_create_table_system

chore: incorporate review feedback

perf: don't copy data around unnecessarily

Created at 1 month ago

Get defaults from Settings internals, add test for ReplicatedMergeTree

Add docs for update_allow_materialized_columns setting

Created at 1 month ago

Add docs for update_allow_materialized_columns setting

Created at 1 month ago