prometheus
Repos
44

The Prometheus monitoring system and time series database.

45708
7394

Exporter for machine metrics

8200
1885

Prometheus Alertmanager

5343
1812

Events

Created at 1 minute ago
issue comment
PromQL label matcher not working for remote server's externalLabel when using remote read

The promblem still occur when i use regex matcher, but equalitiy matcher works as expected

Created at 10 minutes ago
opened issue
PromQL label matcher not working for remote server's externalLabel when using remote read

What did you do?

Main Prometheus config remote read to remote server

global:
  remote_read:
  - url: remote_prometheus

Remote server config:

global:
  external_labels:
    role: remote

What did you expect to see?

When i run promql from main prometheus, ex: some_metric{role != "remote"} ,the data of remote server still shows:

some_metric{role="remote"} 1

What did you see instead? Under which circumstances?

No data should be returned from remote prometheus

System information

No response

Prometheus version

prometheus, version 2.24.1 (branch: HEAD, revision: e4487274853c587717006eeda8804e597d120340)
  build user:       root@0b5231a0de0f
  build date:       20210120-00:09:36
  go version:       go1.15.6
  platform:         linux/amd64

Prometheus configuration file

No response

Alertmanager version

No response

Alertmanager configuration file

No response

Logs

No response

Created at 12 minutes ago
Created at 14 minutes ago
Created at 1 hour ago
Cert-related problems with TLS cluster setup

I'm trying to deploy two alertmanagers in a cluster configuration and I'm running into weird certificate problems I can't explain: for some reason alertmanager expects its own certificate while connecting to its peer.

We have a standardized process to issue certificates using Vault and Ansible, non cluster parts of Alertmanager (and many other services) seemingly use them just fine.

It feels like I'm missing something obvious here. Please help. :)

Peer 01, error message:

Dec 02 05:20:00 test-gl-gcpew1b-vic-vmalert01 alertmanager[42828]: ts=2022-12-02T05:20:00.284Z caller=cluster.go:470 level=warn component=cluster msg=refresh result=failure addr=10.157.192.197:9094 err="1 error occurred:\n\t* Failed to join 10.157.192.197:9094: failed to dial: x509: certificate is valid for localhost, test-gl-gcpew3c-vic-vmalert02.gl.mx, not test-gl-gcpew1b-vic-vmalert01.gl.mx\n\n"

Peer 01, command line arguments & IP address:

/usr/local/bin/alertmanager --cluster.listen-address=10.157.191.196:9094 --cluster.peer=test-gl-gcpew3c-vic-vmalert02.gl.mx:9094 --config.file=/etc/alertmanager/alertmanager.yml --storage.path=/var/lib/alertmanager --web.listen-address=0.0.0.0:9093 --web.config.file=/etc/alertmanager/web.yml --cluster.tls-config=/etc/alertmanager/cluster-tls.yml --web.external-url=https://test-gl-gcpew1b-vic-vmalert01.gl.mx:9093/

root@test-gl-gcpew1b-vic-vmalert01:~# ip -4 a | grep 10.157
    inet 10.157.191.196/32 brd 10.157.191.196 scope global dynamic ens4

Peer 01, cluster-tls config:

tls_server_config:
  cert_file: /etc/alertmanager/ssl/test-gl-gcpew1b-vic-vmalert01.gl.mx.pem
  client_auth_type: RequireAndVerifyClientCert
  client_ca_file: /etc/alertmanager/ssl/ca.pem
  key_file: /etc/alertmanager/ssl/test-gl-gcpew1b-vic-vmalert01.gl.mx.key


tls_client_config:
  cert_file: /etc/alertmanager/ssl/test-gl-gcpew1b-vic-vmalert01.gl.mx.pem
  key_file: /etc/alertmanager/ssl/test-gl-gcpew1b-vic-vmalert01.gl.mx.key
  server_name: test-gl-gcpew1b-vic-vmalert01.gl.mx

Peer 02:

Dec 02 05:27:16 test-gl-gcpew3c-vic-vmalert02 alertmanager[41231]: ts=2022-12-02T05:27:16.483Z caller=cluster.go:470 level=warn component=cluster msg=refresh result=failure addr=10.157.191.196:9094 err="1 error occurred:\n\t* Failed to join 10.157.191.196:9094: failed to dial: x509: certificate is valid for localhost, test-gl-gcpew1b-vic-vmalert01.gl.mx, not test-gl-gcpew3c-vic-vmalert02.gl.mx\n\n"
alertma+   41231  0.4  0.4 728616 36788 ?        Ssl  05:18   0:01 /usr/local/bin/alertmanager --cluster.listen-address=10.157.192.197:9094 --cluster.peer=test-gl-gcpew1b-vic-vmalert01.gl.mx:9094 --config.file=/etc/alertmanager/alertmanager.yml --storage.path=/var/lib/alertmanager --web.listen-address=0.0.0.0:9093 --web.config.file=/etc/alertmanager/web.yml --cluster.tls-config=/etc/alertmanager/cluster-tls.yml --web.external-url=https://test-gl-gcpew3c-vic-vmalert02.gl.mx:9093/

root@test-gl-gcpew3c-vic-vmalert02:~#  ip -4 a | grep 10.157
    inet 10.157.192.197/32 brd 10.157.192.197 scope global dynamic ens4
root@test-gl-gcpew3c-vic-vmalert02:~# cat /etc/alertmanager/cluster-tls.yml
tls_server_config:
  cert_file: /etc/alertmanager/ssl/test-gl-gcpew3c-vic-vmalert02.gl.mx.pem
  client_auth_type: RequireAndVerifyClientCert
  client_ca_file: /etc/alertmanager/ssl/ca.pem
  key_file: /etc/alertmanager/ssl/test-gl-gcpew3c-vic-vmalert02.gl.mx.key


tls_client_config:
  cert_file: /etc/alertmanager/ssl/test-gl-gcpew3c-vic-vmalert02.gl.mx.pem
  key_file: /etc/alertmanager/ssl/test-gl-gcpew3c-vic-vmalert02.gl.mx.key
  server_name: test-gl-gcpew3c-vic-vmalert02.gl.mx

Just in case, the certs, on both nodes:

root@test-gl-gcpew1b-vic-vmalert01:~# openssl x509 -in /etc/alertmanager/ssl/test-gl-gcpew1b-vic-vmalert01.gl.mx.pem -text | grep vmalert
        Subject: CN = test-gl-gcpew1b-vic-vmalert01.gl.mx
                DNS:localhost, DNS:test-gl-gcpew1b-vic-vmalert01.gl.mx, IP Address:10.157.191.196, IP Address:127.0.0.1

root@test-gl-gcpew3c-vic-vmalert02:~# openssl x509 -in /etc/alertmanager/ssl/test-gl-gcpew3c-vic-vmalert02.gl.mx.pem -text | grep vmalert
        Subject: CN = test-gl-gcpew3c-vic-vmalert02.gl.mx
                DNS:localhost, DNS:test-gl-gcpew3c-vic-vmalert02.gl.mx, IP Address:10.157.192.197, IP Address:127.0.0.1

Our environment is Debian 11 running in GCP.

root@test-gl-gcpew3c-vic-vmalert02:~# uname -srm
Linux 5.10.0-19-cloud-amd64 x86_64

root@test-gl-gcpew1b-vic-vmalert01:~# /usr/local/bin/alertmanager --version
alertmanager, version 0.24.0 (branch: HEAD, revision: f484b17fa3c583ed1b2c8bbcec20ba1db2aa5f11)
  build user:       root@265f14f5c6fc
  build date:       20220325-09:31:33
  go version:       go1.17.8
  platform:         linux/amd64

If I need to provide more information, please tell me I'll gather everything that's needed. Thanks in advance for any help.

Created at 1 hour ago
Created at 2 hours ago
pull request opened
Merge back release-2.40 branch again

2.40.5 was released

Created at 3 hours ago
Created at 3 hours ago
Created at 3 hours ago
Created at 3 hours ago
issue comment
Unnecessary ipv6 dns lookup (AAAA)

I'm interesting in this proposal and would like to work on it if it's being accepted. Thanks

Created at 3 hours ago
issue comment
Unnecessary ipv6 dns lookup (AAAA)

Per this similar issue: https://github.com/prometheus/blackbox_exporter/pull/728

Suggest to change this issue to a proposal that adding an option in Prometheus config to disable ipv6 dns lookup if users want.

Created at 3 hours ago
Created at 4 hours ago
Created at 4 hours ago
Created at 4 hours ago
Created at 4 hours ago
Created at 4 hours ago
Created at 4 hours ago
Created at 5 hours ago
Created at 5 hours ago
Created at 6 hours ago
Created at 6 hours ago
Created at 7 hours ago
create branch
dependabot[bot] create branch dependabot/go_modules/documentation/examples/remote_storage/github.com/prometheus/client_golang-1.14.0
Created at 7 hours ago
pull request opened
build(deps): bump github.com/prometheus/client_golang from 1.13.1 to 1.14.0 in /documentation/examples/remote_storage

Bumps github.com/prometheus/client_golang from 1.13.1 to 1.14.0.

Dependabot compatibility score

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


You can trigger Dependabot actions by commenting on this PR:

  • @dependabot rebase will rebase this PR
  • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
  • @dependabot merge will merge this PR after your CI passes on it
  • @dependabot squash and merge will squash and merge this PR after your CI passes on it
  • @dependabot cancel merge will cancel a previously requested merge and block automerging
  • @dependabot reopen will reopen this PR if it is closed
  • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
  • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
  • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
  • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Created at 7 hours ago
create branch
dependabot[bot] create branch dependabot/npm_and_yarn/web/ui/fortawesome/fontawesome-svg-core-6.2.1
Created at 7 hours ago