Prometheus instrumentation library for Go applications
Apache License 2.0
4267
73
1022

This might not even be considered a bug, but I think this is a very annoying behavior.

When using NewMetricWithTimestamp in a Collector as in:

// {
//     Name: "metric_test_1", Description: "metric_test_1 description",
//     Values: [ { value: 1,  timestamp: 1663713926158 }, { value: 2,  timestamp: 1663713937491 } ] 
// } 

desc := prometheus.NewDesc(m.Name, m.Description, nil, nil)

for _, value := range m.Values {
  ch <- prometheus.NewMetricWithTimestamp(
    value.timestamp,
    prometheus.MustNewConstMetric(desc, prometheus.GaugeValue, value.value),
  )
}

The client is running checkMetricConsistency for each of the pushed metrics which is throwing the following error

...
* collected metric "metric_test_1" { gauge:<value:2 > timestamp_ms:1663713937491 } was collected before with the same name and label values
...

I think the validation should be aware of the metrics timestamps and not throw an error in this case because the metrics belong to different point in times, and there are use cases where it makes sense to allow multiple ones to be pushed in the same scrape call.

Using exemplars is quite painful currently. There is a common pattern you want to do such as:

image

I would propose something more natively supported. Some options:

A) Bring ObserveWithExemplar to Observer, why not?
B) Create a separate package that creates natively observers with exemplars (sounds brittle and more to maintain for not a good reason)
C) Create a helper that does something like:

func ExemplarObserve(obs prometheus.Observer, val float64, traceID string) {
	if traceID != "" {
		obs.(prometheus.ExemplarObserver).ObserveWithExemplar(
			val, map[string]string{"trace-id": traceID})
	} else {
		obs.Observe(val)
	}
}

The opinionated state is simple:

  • We want to observe with a trace ID.
  • We want a consistent trace ID label.
  • We want trace ID which is sampled, no point in any other (ofc won't work with tail sampling)
  • Assumption: Empty Trace ID means it's not sampled.

Ideally we have opinionated helper as part of method to Observe itself with configurable trace ID label... 🙈

Having if else when histogram is used is making client instrumentation too verbose IMO. Let's brainstorm this (:

Thoughts? @kakkoyun @juliusv @beorn7

Whilst updating the golang-github-prometheus-client-golang package in Debian, autopkgtest revealed that the TestSparseHistogram test panics with an integer divide by zero on ppc64el (ppc64le as GOARCH).

=== RUN   TestSparseHistogram
=== RUN   TestSparseHistogram/no_sparse_buckets
=== RUN   TestSparseHistogram/factor_1.1_results_in_schema_3
=== RUN   TestSparseHistogram/factor_1.2_results_in_schema_2
=== RUN   TestSparseHistogram/factor_4_results_in_schema_-1
--- FAIL: TestSparseHistogram (0.00s)
    --- PASS: TestSparseHistogram/no_sparse_buckets (0.00s)
    --- PASS: TestSparseHistogram/factor_1.1_results_in_schema_3 (0.00s)
    --- PASS: TestSparseHistogram/factor_1.2_results_in_schema_2 (0.00s)
    --- FAIL: TestSparseHistogram/factor_4_results_in_schema_-1 (0.00s)
panic: runtime error: integer divide by zero [recovered]
	panic: runtime error: integer divide by zero

goroutine 3103 [running]:
testing.tRunner.func1.2({0x434f00, 0x8244b0})
	/usr/lib/go-1.19/src/testing/testing.go:1396 +0x1fc
testing.tRunner.func1()
	/usr/lib/go-1.19/src/testing/testing.go:1399 +0x364
panic({0x434f00, 0x8244b0})
	/usr/lib/go-1.19/src/runtime/panic.go:884 +0x240
github.com/prometheus/client_golang/prometheus.(*histogramCounts).observe(0xc0001b4750, 0xc000061d18?, 0xb0710?, 0x0?)
	/tmp/autopkgtest-lxc.4sj9alys/downtmp/autopkgtest_tmp/_build/src/github.com/prometheus/client_golang/prometheus/histogram.go:636 +0x324
github.com/prometheus/client_golang/prometheus.(*histogram).observe(0xc0003d8000, 0x3fe0000000000000, 0x10?)
	/tmp/autopkgtest-lxc.4sj9alys/downtmp/autopkgtest_tmp/_build/src/github.com/prometheus/client_golang/prometheus/histogram.go:807 +0xa4
github.com/prometheus/client_golang/prometheus.(*histogram).Observe(0xc0003d8000, 0xc95125dbd?)
	/tmp/autopkgtest-lxc.4sj9alys/downtmp/autopkgtest_tmp/_build/src/github.com/prometheus/client_golang/prometheus/histogram.go:707 +0x5c
github.com/prometheus/client_golang/prometheus.TestSparseHistogram.func1(0xc0003ae820)
	/tmp/autopkgtest-lxc.4sj9alys/downtmp/autopkgtest_tmp/_build/src/github.com/prometheus/client_golang/prometheus/histogram_test.go:675 +0x35c
testing.tRunner(0xc0003ae820, 0xc0003aa470)
	/usr/lib/go-1.19/src/testing/testing.go:1446 +0x11c
created by testing.(*T).Run
	/usr/lib/go-1.19/src/testing/testing.go:1493 +0x36c
FAIL	github.com/prometheus/client_golang/prometheus	24.050s

All other tested platforms (amd64, arm64, arm, i386, s390x) pass the test.

Since this may in fact be a Go compiler bug, I will disable that test on ppc64le, but thought it would be prudent to at least raise the issue here, in case there is an easy fix.

cc: @beorn7

My setup

We run a Grafana which queries Prometheus. Inbetween Grafana and Prometheus, there is a Google Cloud Load Balancer L7 which terminates SSL:

Grafana --> GCLB --> Prometheus

Issue

Grafana user opens a dashboard and Grafana opens a long running TCP connection. Rarely we observe a TCP Reset sent by the Load Balancer, which Grafana displays to user as 5xx error. The request is not retried.

Background

Grafana utilizes client_golang library and in particular its httpapi.QueryRange method which relies on apiClientImpl.DoGetFallback. This method tries to perform the query using POST request and in case of specific issues (405) it fallbacks to GET.

Prometheus' client prefers to use POST over GET, but this causes inconsistencies as POST is not consider idempotent.

Quote from net/http

Transport only retries a request upon encountering a network error if the request is idempotent and either has no body or has its Request.GetBody defined. HTTP requests are considered idempotent if they have HTTP methods GET, HEAD, OPTIONS, or TRACE; or if their Header map contains an "Idempotency-Key" or "X-Idempotency-Key" entry. If the idempotency key value is a zero-length slice, the request is treated as idempotent but the header is not sent on the wire.

More information about this retry feature can be found in https://go-review.googlesource.com/c/go/+/3210/9//COMMIT_MSG .
I found it really confusing, that when the implementation of apiClientImpl.DoGetFallback fallbacks to GET request it might recover from some connectivity issues while with default POST request, it will just error.

Expectations

For me ideally I would prefer making the POST request idempotent (as I believe they are; using one of the Idempotency-* header?). I am not sure if this would be acceptable as this changes the behavior of the client a bit.

Reproducer

None yet.

import (
proclientApi "github.com/prometheus/client_golang/api"
)

var ProClientMap = make(map[string]proclientApi.Client, 0)

func PromQueryRange(){
    ProClient := proclientV1.NewAPI(ProClientMap[dataSource])
    value, _, err := ProClient.QueryRange(ctx, sql, r)
    if err != nil {
         logger.Errorf("[proQuery]error,[%v]", err)
         return PromMonitorData, err
    }
}

when concurrent PromQueryRange,ProClient.QueryRange panic,why?

[2022-11-21 02:02:51]
[2022-11-21 02:02:51 goroutine 638 [sleep]:]
[2022-11-21 02:02:51 time.Sleep(0x3b9aca00)]
[2022-11-21 02:02:51 /usr/local/go/src/runtime/time.go:194 +0x12e]
[2022-11-21 02:02:51 github.com/valyala/fasthttp.(*TCPDialer).tcpAddrsClean(0x1493740)]
[2022-11-21 02:02:51 /build/vendor/github.com/valyala/fasthttp/tcpdialer.go:372 +0x33]
[2022-11-21 02:02:51 created by github.com/valyala/fasthttp.(*TCPDialer).dial.func1]
[2022-11-21 02:02:51 /build/vendor/github.com/valyala/fasthttp/tcpdialer.go:282 +0xaa]
[2022-11-21 02:02:51]
[2022-11-21 02:02:51 goroutine 1187 [IO wait, 2 minutes]:]
[2022-11-21 02:02:51 internal/poll.runtime_pollWait(0x7efc942d7e88, 0x72)]
[2022-11-21 02:02:51 /usr/local/go/src/runtime/netpoll.go:302 +0x89]
[2022-11-21 02:02:51 internal/poll.(*pollDesc).wait(0xc001611a80?, 0xc000b95000?, 0x0)]
[2022-11-21 02:02:51 /usr/local/go/src/internal/poll/fd_poll_runtime.go:83 +0x32]
[2022-11-21 02:02:51 internal/poll.(*pollDesc).waitRead(...)]
[2022-11-21 02:02:51 /usr/local/go/src/internal/poll/fd_poll_runtime.go:88]
[2022-11-21 02:02:51 internal/poll.(*FD).Read(0xc001611a80, {0xc000b95000, 0x1000, 0x1000})]
[2022-11-21 02:02:51 /usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a]
[2022-11-21 02:02:51 net.(*netFD).Read(0xc001611a80, {0xc000b95000?, 0x405ea9?, 0x4?})]
[2022-11-21 02:02:51 /usr/local/go/src/net/fd_posix.go:55 +0x29]
[2022-11-21 02:02:51 net.(*conn).Read(0xc000222ae0, {0xc000b95000?, 0xc0003d16b0?, 0x1?})]
[2022-11-21 02:02:51 /usr/local/go/src/net/net.go:183 +0x45]
[2022-11-21 02:02:51 net/http.(*persistConn).Read(0xc00164ac60, {0xc000b95000?, 0xc0012d4540?, 0xc000a5fd30?})]
[2022-11-21 02:02:51 /usr/local/go/src/net/http/transport.go:1929 +0x4e]
[2022-11-21 02:02:51 bufio.(*Reader).fill(0xc0004c8ae0)]
[2022-11-21 02:02:51 /usr/local/go/src/bufio/bufio.go:106 +0x103]
[2022-11-21 02:02:51 bufio.(*Reader).Peek(0xc0004c8ae0, 0x1)]
[2022-11-21 02:02:51 /usr/local/go/src/bufio/bufio.go:144 +0x5d]
[2022-11-21 02:02:51 net/http.(*persistConn).readLoop(0xc00164ac60)]
[2022-11-21 02:02:51 /usr/local/go/src/net/http/transport.go:2093 +0x1ac]
[2022-11-21 02:02:51 created by net/http.(*Transport).dialConn]
[2022-11-21 02:02:51 /usr/local/go/src/net/http/transport.go:1750 +0x173e]
[2022-11-21 02:02:51]
[2022-11-21 02:02:51 goroutine 3169 [select]:]
[2022-11-21 02:02:51 net/http.(*persistConn).writeLoop(0xc0017eed80)]
[2022-11-21 02:02:51 /usr/local/go/src/net/http/transport.go:2392 +0xf5]
[2022-11-21 02:02:51 created by net/http.(*Transport).dialConn]
[2022-11-21 02:02:51 /usr/local/go/src/net/http/transport.go:1751 +0x1791]
[2022-11-21 02:02:51]
[2022-11-21 02:02:51 goroutine 683 [IO wait, 4 minutes]:]
[2022-11-21 02:02:51 internal/poll.runtime_pollWait(0x7efc942d7ca8, 0x72)]
[2022-11-21 02:02:51 /usr/local/go/src/runtime/netpoll.go:302 +0x89]
[2022-11-21 02:02:51 internal/poll.(*pollDesc).wait(0xc000601d80?, 0xc000a30000?, 0x0)]
[2022-11-21 02:02:51 /usr/local/go/src/internal/poll/fd_poll_runtime.go:83 +0x32]
[2022-11-21 02:02:51 internal/poll.(*pollDesc).waitRead(...)]
[2022-11-21 02:02:51 /usr/local/go/src/internal/poll/fd_poll_runtime.go:88]
[2022-11-21 02:02:51 internal/poll.(*FD).Read(0xc000601d80, {0xc000a30000, 0x1000, 0x1000})]
[2022-11-21 02:02:51 /usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a]
[2022-11-21 02:02:51 net.(*netFD).Read(0xc000601d80, {0xc000a30000?, 0x405ea9?, 0x4?})]
[2022-11-21 02:02:51 /usr/local/go/src/net/fd_posix.go:55 +0x29]
[2022-11-21 02:02:51 net.(*conn).Read(0xc0003d0640, {0xc000a30000?, 0x0?, 0x0?})]
[2022-11-21 02:02:51 /usr/local/go/src/net/net.go:183 +0x45]
[2022-11-21 02:02:51 net/http.(*persistConn).Read(0xc001185b00, {0xc000a30000?, 0xc0005c7620?, 0xc000a5cd30?})]
[2022-11-21 02:02:51 /usr/local/go/src/net/http/transport.go:1929 +0x4e]
[2022-11-21 02:02:51 bufio.(*Reader).fill(0xc000cd6f00)]
[2022-11-21 02:02:51 /usr/local/go/src/bufio/bufio.go:106 +0x103]
[2022-11-21 02:02:51 bufio.(*Reader).Peek(0xc000cd6f00, 0x1)]
[2022-11-21 02:02:51 /usr/local/go/src/bufio/bufio.go:144 +0x5d]
[2022-11-21 02:02:51 net/http.(*persistConn).readLoop(0xc001185b00)]
[2022-11-21 02:02:51 /usr/local/go/src/net/http/transport.go:2093 +0x1ac]
[2022-11-21 02:02:51 created by net/http.(*Transport).dialConn]
[2022-11-21 02:02:51 /usr/local/go/src/net/http/transport.go:1750 +0x173e]
[2022-11-21 02:02:51]
[2022-11-21 02:02:51 goroutine 738 [IO wait, 2 minutes]:]
[2022-11-21 02:02:51 internal/poll.runtime_pollWait(0x7efc942d7d98, 0x72)]
[2022-11-21 02:02:51 /usr/local/go/src/runtime/netpoll.go:302 +0x89]
[2022-11-21 02:02:51 internal/poll.(*pollDesc).wait(0xc000410880?, 0xc000e42000?, 0x0)]
[2022-11-21 02:02:51 /usr/local/go/src/internal/poll/fd_poll_runtime.go:83 +0x32]
[2022-11-21 02:02:51 internal/poll.(*pollDesc).waitRead(...)]
[2022-11-21 02:02:51 /usr/local/go/src/internal/poll/fd_poll_runtime.go:88]
[2022-11-21 02:02:51 internal/poll.(*FD).Read(0xc000410880, {0xc000e42000, 0x1000, 0x1000})]
[2022-11-21 02:02:51 /usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a]
[2022-11-21 02:02:51 net.(*netFD).Read(0xc000410880, {0xc000e42000?, 0x405ea9?, 0x4?})]
[2022-11-21 02:02:51 /usr/local/go/src/net/fd_posix.go:55 +0x29]
[2022-11-21 02:02:51 net.(*conn).Read(0xc000306888, {0xc000e42000?, 0x0?, 0x0?})]
[2022-11-21 02:02:51 /usr/local/go/src/net/net.go:183 +0x45]
[2022-11-21 02:02:51 net/http.(*persistConn).Read(0xc000d14c60, {0xc000e42000?, 0xc000043b60?, 0xc000a59d30?})]
[2022-11-21 02:02:51 /usr/local/go/src/net/http/transport.go:1929 +0x4e]
[2022-11-21 02:02:51 bufio.(*Reader).fill(0xc000cd7a40)]
[2022-11-21 02:02:51 /usr/local/go/src/bufio/bufio.go:106 +0x103]
[2022-11-21 02:02:51 bufio.(*Reader).Peek(0xc000cd7a40, 0x1)]
[2022-11-21 02:02:51 /usr/local/go/src/bufio/bufio.go:144 +0x5d]
[2022-11-21 02:02:51 net/http.(*persistConn).readLoop(0xc000d14c60)]
[2022-11-21 02:02:51 /usr/local/go/src/net/http/transport.go:2093 +0x1ac]
[2022-11-21 02:02:51 created by net/http.(*Transport).dialConn]
[2022-11-21 02:02:51 /usr/local/go/src/net/http/transport.go:1750 +0x173e]
[2022-11-21 02:02:51]
[2022-11-21 02:02:51 goroutine 739 [select, 2 minutes]:]
[2022-11-21 02:02:51 net/http.(*persistConn).writeLoop(0xc000d14c60)]
[2022-11-21 02:02:51 /usr/local/go/src/net/http/transport.go:2392 +0xf5]
[2022-11-21 02:02:51 created by net/http.(*Transport).dialConn]
[2022-11-21 02:02:51 /usr/local/go/src/net/http/transport.go:1751 +0x1791]
[2022-11-21 02:02:51]
[2022-11-21 02:02:51 goroutine 5497 [select]:]
[2022-11-21 02:02:51 net/http.(*persistConn).writeLoop(0xc001253c20)]
[2022-11-21 02:02:51 /usr/local/go/src/net/http/transport.go:2392 +0xf5]
[2022-11-21 02:02:51 created by net/http.(*Transport).dialConn]
[2022-11-21 02:02:51 /usr/local/go/src/net/http/transport.go:1751 +0x1791]
[2022-11-21 02:02:51]
[2022-11-21 02:02:51 goroutine 6315 [select]:]
[2022-11-21 02:02:51 github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher.func1()]
[2022-11-21 02:02:51 /build/vendor/github.com/go-sql-driver/mysql/connection.go:614 +0xaa]
[2022-11-21 02:02:51 created by github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher]
[2022-11-21 02:02:51 /build/vendor/github.com/go-sql-driver/mysql/connection.go:611 +0xfe]
[2022-11-21 02:02:51]
[2022-11-21 02:02:51 goroutine 1188 [select, 4 minutes]:]
[2022-11-21 02:02:51 net/http.(*persistConn).writeLoop(0xc00164ac60)]
[2022-11-21 02:02:51 /usr/local/go/src/net/http/transport.go:2392 +0xf5]
[2022-11-21 02:02:51 created by net/http.(*Transport).dialConn]
[2022-11-21 02:02:51 /usr/local/go/src/net/http/transport.go:1751 +0x1791]
[2022-11-21 02:02:51]
[2022-11-21 02:02:51 goroutine 6370 [select]:]
[2022-11-21 02:02:51 net/http.(*persistConn).roundTrip(0xc000bbf0e0, 0xc000d27240)]
[2022-11-21 02:02:51 /usr/local/go/src/net/http/transport.go:2620 +0x974]
[2022-11-21 02:02:51 net/http.(*Transport).roundTrip(0x1495240, 0xc000dc9200)]
[2022-11-21 02:02:51 /usr/local/go/src/net/http/transport.go:594 +0x7c9]
[2022-11-21 02:02:51 net/http.(*Transport).RoundTrip(0x40d325?, 0xeeb400?)]
[2022-11-21 02:02:51 /usr/local/go/src/net/http/roundtrip.go:17 +0x19]
[2022-11-21 02:02:51 net/http.send(0xc000dc9200, {0xeeb400, 0x1495240}, {0xd71040?, 0x7efc9477df01?, 0x0?})]
[2022-11-21 02:02:51 /usr/local/go/src/net/http/client.go:252 +0x5d8]
[2022-11-21 02:02:51 net/http.(*Client).send(0xc0000444c8, 0xc000dc9200, {0x7efc9477dfff?, 0x7efc945f0c00?, 0x0?})]
[2022-11-21 02:02:51 /usr/local/go/src/net/http/client.go:176 +0x9b]
[2022-11-21 02:02:51 net/http.(*Client).do(0xc0000444c8, 0xc000dc9200)]
[2022-11-21 02:02:51 /usr/local/go/src/net/http/client.go:725 +0x8f5]
[2022-11-21 02:02:51 net/http.(*Client).Do(...)]
[2022-11-21 02:02:51 /usr/local/go/src/net/http/client.go:593]
[2022-11-21 02:02:51 github.com/prometheus/client_golang/api.(*httpClient).Do(0xc0000444c0, {0xef08d0, 0xc00003a068}, 0xc000dc9100)]
[2022-11-21 02:02:51 /build/vendor/github.com/prometheus/client_golang/api/client.go:100 +0x285]
[2022-11-21 02:02:51 github.com/prometheus/client_golang/api/prometheus/v1.(*apiClientImpl).Do(0xd1d0c0?, {0xef08d0?, 0xc00003a068?}, 0xc?)]
[2022-11-21 02:02:51 /build/vendor/github.com/prometheus/client_golang/api/prometheus/v1/api.go:1088 +0x43]
[2022-11-21 02:02:51 github.com/prometheus/client_golang/api/prometheus/v1.(*apiClientImpl).DoGetFallback(0xd2b1c0?, {0xef08d0, 0xc00003a068}, 0xc00103a630, 0xc0003a019b?)]
[2022-11-21 02:02:51 /build/vendor/github.com/prometheus/client_golang/api/prometheus/v1/api.go:1142 +0x1d4]
[2022-11-21 02:02:51 github.com/prometheus/client_golang/api/prometheus/v1.(*httpAPI).QueryRange(0xc000c49b78, {0xef08d0, 0xc00003a068}, {0xc00076b880, 0x66}, {{0x0, 0xedb0cd297, 0x14ab420}, {0x0, 0xedb0cd34b, ...}, ...})]
[2022-11-21 02:02:51 /build/vendor/github.com/prometheus/client_golang/api/prometheus/v1/api.go:848 +0x3e2]
[2022-11-21 02:02:51 ***/pkg/util/prometheus.PromQueryRange({0xc00076b880, 0x66}, {0xc001237dfe?, 0xa?}, {0xc001237e30, 0xc})]

go version: 1.19.2

i receive the following error message on command:
go env -w GO111MODULE=off; go get -v github.com/prometheus/client_golang/prometheus

#15 28.71 ../github.com/prometheus/common/expfmt/decode.go:89:38: cannot use v (variable of type *io_prometheus_client.MetricFamily) as type protoreflect.ProtoMessage in argument to pbutil.ReadDelimited:
#15 28.71 	*io_prometheus_client.MetricFamily does not implement protoreflect.ProtoMessage (missing ProtoReflect method)
#15 28.71 ../github.com/prometheus/common/expfmt/encode.go:120:40: cannot use v (variable of type *io_prometheus_client.MetricFamily) as type protoreflect.ProtoMessage in argument to pbutil.WriteDelimited:
#15 28.71 	*io_prometheus_client.MetricFamily does not implement protoreflect.ProtoMessage (missing ProtoReflect method)

here the full jenkins job output:

#15 3.226 + go env -w GO111MODULE=off
#15 3.229 + go get -v github.com/prometheus/client_golang/prometheus
#15 3.232 github.com/prometheus/client_golang (download)
#15 4.839 github.com/beorn7/perks (download)
#15 5.520 github.com/cespare/xxhash (download)
#15 6.170 github.com/golang/protobuf (download)
#15 8.937 get "google.golang.org/protobuf/encoding/prototext": found meta tag vcs.metaImport{Prefix:"google.golang.org/protobuf", VCS:"git", RepoRoot:"https://go.googlesource.com/protobuf"} at //google.golang.org/protobuf/encoding/prototext?go-get=1
#15 8.937 get "google.golang.org/protobuf/encoding/prototext": verifying non-authoritative meta tag
#15 9.080 google.golang.org/protobuf (download)
#15 10.87 get "google.golang.org/protobuf/encoding/protowire": found meta tag vcs.metaImport{Prefix:"google.golang.org/protobuf", VCS:"git", RepoRoot:"https://go.googlesource.com/protobuf"} at //google.golang.org/protobuf/encoding/protowire?go-get=1
#15 10.87 get "google.golang.org/protobuf/encoding/protowire": verifying non-authoritative meta tag
#15 11.00 get "google.golang.org/protobuf/reflect/protoreflect": found meta tag vcs.metaImport{Prefix:"google.golang.org/protobuf", VCS:"git", RepoRoot:"https://go.googlesource.com/protobuf"} at //google.golang.org/protobuf/reflect/protoreflect?go-get=1
#15 11.00 get "google.golang.org/protobuf/reflect/protoreflect": verifying non-authoritative meta tag
#15 11.14 get "google.golang.org/protobuf/proto": found meta tag vcs.metaImport{Prefix:"google.golang.org/protobuf", VCS:"git", RepoRoot:"https://go.googlesource.com/protobuf"} at //google.golang.org/protobuf/proto?go-get=1
#15 11.14 get "google.golang.org/protobuf/proto": verifying non-authoritative meta tag
#15 11.28 get "google.golang.org/protobuf/reflect/protoregistry": found meta tag vcs.metaImport{Prefix:"google.golang.org/protobuf", VCS:"git", RepoRoot:"https://go.googlesource.com/protobuf"} at //google.golang.org/protobuf/reflect/protoregistry?go-get=1
#15 11.28 get "google.golang.org/protobuf/reflect/protoregistry": verifying non-authoritative meta tag
#15 11.41 get "google.golang.org/protobuf/runtime/protoiface": found meta tag vcs.metaImport{Prefix:"google.golang.org/protobuf", VCS:"git", RepoRoot:"https://go.googlesource.com/protobuf"} at //google.golang.org/protobuf/runtime/protoiface?go-get=1
#15 11.41 get "google.golang.org/protobuf/runtime/protoiface": verifying non-authoritative meta tag
#15 11.55 get "google.golang.org/protobuf/reflect/protodesc": found meta tag vcs.metaImport{Prefix:"google.golang.org/protobuf", VCS:"git", RepoRoot:"https://go.googlesource.com/protobuf"} at //google.golang.org/protobuf/reflect/protodesc?go-get=1
#15 11.55 get "google.golang.org/protobuf/reflect/protodesc": verifying non-authoritative meta tag
#15 11.68 get "google.golang.org/protobuf/runtime/protoimpl": found meta tag vcs.metaImport{Prefix:"google.golang.org/protobuf", VCS:"git", RepoRoot:"https://go.googlesource.com/protobuf"} at //google.golang.org/protobuf/runtime/protoimpl?go-get=1
#15 11.68 get "google.golang.org/protobuf/runtime/protoimpl": verifying non-authoritative meta tag
#15 11.69 github.com/prometheus/client_model (download)
#15 12.44 get "google.golang.org/protobuf/types/known/timestamppb": found meta tag vcs.metaImport{Prefix:"google.golang.org/protobuf", VCS:"git", RepoRoot:"https://go.googlesource.com/protobuf"} at //google.golang.org/protobuf/types/known/timestamppb?go-get=1
#15 12.44 get "google.golang.org/protobuf/types/known/timestamppb": verifying non-authoritative meta tag
#15 12.45 github.com/prometheus/common (download)
#15 13.63 github.com/matttproud/golang_protobuf_extensions (download)
#15 14.27 github.com/prometheus/procfs (download)
#15 15.65 get "golang.org/x/sys/unix": found meta tag vcs.metaImport{Prefix:"golang.org/x/sys", VCS:"git", RepoRoot:"https://go.googlesource.com/sys"} at //golang.org/x/sys/unix?go-get=1
#15 15.65 get "golang.org/x/sys/unix": verifying non-authoritative meta tag
#15 15.79 golang.org/x/sys (download)
#15 18.46 google.golang.org/protobuf/internal/set
#15 18.46 google.golang.org/protobuf/internal/flags
#15 18.48 github.com/beorn7/perks/quantile
#15 18.48 google.golang.org/protobuf/internal/pragma
#15 18.48 github.com/prometheus/common/internal/bitbucket.org/ww/goautoneg
#15 18.48 github.com/cespare/xxhash
#15 18.48 crypto/sha1
#15 18.56 google.golang.org/protobuf/internal/detrand
#15 18.56 github.com/prometheus/procfs/internal/util
#15 18.56 google.golang.org/protobuf/internal/version
#15 18.57 github.com/prometheus/procfs/internal/fs
#15 18.57 github.com/prometheus/common/model
#15 18.66 golang.org/x/sys/unix
#15 18.66 net
#15 18.76 google.golang.org/protobuf/internal/errors
#15 18.87 google.golang.org/protobuf/encoding/protowire
#15 19.06 google.golang.org/protobuf/reflect/protoreflect
#15 19.58 google.golang.org/protobuf/internal/descopts
#15 19.58 google.golang.org/protobuf/internal/encoding/messageset
#15 19.58 google.golang.org/protobuf/internal/descfmt
#15 19.58 google.golang.org/protobuf/runtime/protoiface
#15 19.58 google.golang.org/protobuf/internal/strs
#15 19.58 google.golang.org/protobuf/internal/order
#15 19.58 google.golang.org/protobuf/internal/genid
#15 19.77 google.golang.org/protobuf/reflect/protoregistry
#15 19.86 google.golang.org/protobuf/internal/encoding/text
#15 20.57 google.golang.org/protobuf/proto
#15 20.77 google.golang.org/protobuf/internal/encoding/defval
#15 21.38 google.golang.org/protobuf/encoding/prototext
#15 21.38 google.golang.org/protobuf/internal/filedesc
#15 21.38 github.com/matttproud/golang_protobuf_extensions/pbutil
#15 21.78 vendor/golang.org/x/net/http/httpproxy
#15 21.78 net/textproto
#15 21.78 crypto/x509
#15 21.86 github.com/prometheus/procfs
#15 22.36 vendor/golang.org/x/net/http/httpguts
#15 22.36 mime/multipart
#15 22.67 google.golang.org/protobuf/internal/encoding/tag
#15 22.86 google.golang.org/protobuf/internal/impl
#15 23.57 crypto/tls
#15 25.78 net/http/httptrace
#15 25.81 net/http
#15 25.92 google.golang.org/protobuf/internal/filetype
#15 26.06 google.golang.org/protobuf/runtime/protoimpl
#15 26.10 google.golang.org/protobuf/types/known/timestamppb
#15 26.10 google.golang.org/protobuf/types/descriptorpb
#15 26.18 github.com/golang/protobuf/ptypes/timestamp
#15 26.50 google.golang.org/protobuf/reflect/protodesc
#15 26.88 github.com/golang/protobuf/proto
#15 27.67 github.com/prometheus/client_model/go
#15 28.07 github.com/prometheus/client_golang/prometheus/internal
#15 28.68 expvar
#15 28.68 github.com/prometheus/common/expfmt
#15 28.71 # github.com/prometheus/common/expfmt
#15 28.71 ../github.com/prometheus/common/expfmt/decode.go:89:38: cannot use v (variable of type *io_prometheus_client.MetricFamily) as type protoreflect.ProtoMessage in argument to pbutil.ReadDelimited:
#15 28.71 	*io_prometheus_client.MetricFamily does not implement protoreflect.ProtoMessage (missing ProtoReflect method)
#15 28.71 ../github.com/prometheus/common/expfmt/encode.go:120:40: cannot use v (variable of type *io_prometheus_client.MetricFamily) as type protoreflect.ProtoMessage in argument to pbutil.WriteDelimited:
#15 28.71 	*io_prometheus_client.MetricFamily does not implement protoreflect.ProtoMessage (missing ProtoReflect method)

i have tried it also as recommended from go but this also not work:

#15 2.146 + go env -w GO111MODULE=on
#15 2.149 + go install github.com/prometheus/client_golang/prometheus@latest
#15 2.226 go: downloading github.com/prometheus/client_golang v1.13.0
#15 2.275 go: github.com/prometheus/client_golang/prometheus@latest (in github.com/prometheus/client_golang@v1.13.0):
#15 2.275 	The go.mod file for the module providing named packages contains one or
#15 2.275 	more exclude directives. It must not contain directives that would cause
#15 2.275 	it to be interpreted differently than if it were the main module.

Go Version:
go version go1.14.13 darwin/amd64

Error Message seen is:
go get github.com/prometheus/client_golang/prometheus@v1.12.2
go get github.com/prometheus/client_golang/prometheus@v1.12.2: github.com/prometheus/client_golang/prometheus@v1.12.2: reading https://gonexus.dev/github.com/prometheus/client_golang/prometheus/@v/v1.12.2.info: 503 Service Unavailable

go version: 1.19.2

i receive the following error message on command:
go env -w GO111MODULE=off; go get -v github.com/prometheus/client_golang/prometheus

#15 28.71 ../github.com/prometheus/common/expfmt/decode.go:89:38: cannot use v (variable of type *io_prometheus_client.MetricFamily) as type protoreflect.ProtoMessage in argument to pbutil.ReadDelimited:
#15 28.71 	*io_prometheus_client.MetricFamily does not implement protoreflect.ProtoMessage (missing ProtoReflect method)
#15 28.71 ../github.com/prometheus/common/expfmt/encode.go:120:40: cannot use v (variable of type *io_prometheus_client.MetricFamily) as type protoreflect.ProtoMessage in argument to pbutil.WriteDelimited:
#15 28.71 	*io_prometheus_client.MetricFamily does not implement protoreflect.ProtoMessage (missing ProtoReflect method)

here the full jenkins job output:

#15 3.226 + go env -w GO111MODULE=off
#15 3.229 + go get -v github.com/prometheus/client_golang/prometheus
#15 3.232 github.com/prometheus/client_golang (download)
#15 4.839 github.com/beorn7/perks (download)
#15 5.520 github.com/cespare/xxhash (download)
#15 6.170 github.com/golang/protobuf (download)
#15 8.937 get "google.golang.org/protobuf/encoding/prototext": found meta tag vcs.metaImport{Prefix:"google.golang.org/protobuf", VCS:"git", RepoRoot:"https://go.googlesource.com/protobuf"} at //google.golang.org/protobuf/encoding/prototext?go-get=1
#15 8.937 get "google.golang.org/protobuf/encoding/prototext": verifying non-authoritative meta tag
#15 9.080 google.golang.org/protobuf (download)
#15 10.87 get "google.golang.org/protobuf/encoding/protowire": found meta tag vcs.metaImport{Prefix:"google.golang.org/protobuf", VCS:"git", RepoRoot:"https://go.googlesource.com/protobuf"} at //google.golang.org/protobuf/encoding/protowire?go-get=1
#15 10.87 get "google.golang.org/protobuf/encoding/protowire": verifying non-authoritative meta tag
#15 11.00 get "google.golang.org/protobuf/reflect/protoreflect": found meta tag vcs.metaImport{Prefix:"google.golang.org/protobuf", VCS:"git", RepoRoot:"https://go.googlesource.com/protobuf"} at //google.golang.org/protobuf/reflect/protoreflect?go-get=1
#15 11.00 get "google.golang.org/protobuf/reflect/protoreflect": verifying non-authoritative meta tag
#15 11.14 get "google.golang.org/protobuf/proto": found meta tag vcs.metaImport{Prefix:"google.golang.org/protobuf", VCS:"git", RepoRoot:"https://go.googlesource.com/protobuf"} at //google.golang.org/protobuf/proto?go-get=1
#15 11.14 get "google.golang.org/protobuf/proto": verifying non-authoritative meta tag
#15 11.28 get "google.golang.org/protobuf/reflect/protoregistry": found meta tag vcs.metaImport{Prefix:"google.golang.org/protobuf", VCS:"git", RepoRoot:"https://go.googlesource.com/protobuf"} at //google.golang.org/protobuf/reflect/protoregistry?go-get=1
#15 11.28 get "google.golang.org/protobuf/reflect/protoregistry": verifying non-authoritative meta tag
#15 11.41 get "google.golang.org/protobuf/runtime/protoiface": found meta tag vcs.metaImport{Prefix:"google.golang.org/protobuf", VCS:"git", RepoRoot:"https://go.googlesource.com/protobuf"} at //google.golang.org/protobuf/runtime/protoiface?go-get=1
#15 11.41 get "google.golang.org/protobuf/runtime/protoiface": verifying non-authoritative meta tag
#15 11.55 get "google.golang.org/protobuf/reflect/protodesc": found meta tag vcs.metaImport{Prefix:"google.golang.org/protobuf", VCS:"git", RepoRoot:"https://go.googlesource.com/protobuf"} at //google.golang.org/protobuf/reflect/protodesc?go-get=1
#15 11.55 get "google.golang.org/protobuf/reflect/protodesc": verifying non-authoritative meta tag
#15 11.68 get "google.golang.org/protobuf/runtime/protoimpl": found meta tag vcs.metaImport{Prefix:"google.golang.org/protobuf", VCS:"git", RepoRoot:"https://go.googlesource.com/protobuf"} at //google.golang.org/protobuf/runtime/protoimpl?go-get=1
#15 11.68 get "google.golang.org/protobuf/runtime/protoimpl": verifying non-authoritative meta tag
#15 11.69 github.com/prometheus/client_model (download)
#15 12.44 get "google.golang.org/protobuf/types/known/timestamppb": found meta tag vcs.metaImport{Prefix:"google.golang.org/protobuf", VCS:"git", RepoRoot:"https://go.googlesource.com/protobuf"} at //google.golang.org/protobuf/types/known/timestamppb?go-get=1
#15 12.44 get "google.golang.org/protobuf/types/known/timestamppb": verifying non-authoritative meta tag
#15 12.45 github.com/prometheus/common (download)
#15 13.63 github.com/matttproud/golang_protobuf_extensions (download)
#15 14.27 github.com/prometheus/procfs (download)
#15 15.65 get "golang.org/x/sys/unix": found meta tag vcs.metaImport{Prefix:"golang.org/x/sys", VCS:"git", RepoRoot:"https://go.googlesource.com/sys"} at //golang.org/x/sys/unix?go-get=1
#15 15.65 get "golang.org/x/sys/unix": verifying non-authoritative meta tag
#15 15.79 golang.org/x/sys (download)
#15 18.46 google.golang.org/protobuf/internal/set
#15 18.46 google.golang.org/protobuf/internal/flags
#15 18.48 github.com/beorn7/perks/quantile
#15 18.48 google.golang.org/protobuf/internal/pragma
#15 18.48 github.com/prometheus/common/internal/bitbucket.org/ww/goautoneg
#15 18.48 github.com/cespare/xxhash
#15 18.48 crypto/sha1
#15 18.56 google.golang.org/protobuf/internal/detrand
#15 18.56 github.com/prometheus/procfs/internal/util
#15 18.56 google.golang.org/protobuf/internal/version
#15 18.57 github.com/prometheus/procfs/internal/fs
#15 18.57 github.com/prometheus/common/model
#15 18.66 golang.org/x/sys/unix
#15 18.66 net
#15 18.76 google.golang.org/protobuf/internal/errors
#15 18.87 google.golang.org/protobuf/encoding/protowire
#15 19.06 google.golang.org/protobuf/reflect/protoreflect
#15 19.58 google.golang.org/protobuf/internal/descopts
#15 19.58 google.golang.org/protobuf/internal/encoding/messageset
#15 19.58 google.golang.org/protobuf/internal/descfmt
#15 19.58 google.golang.org/protobuf/runtime/protoiface
#15 19.58 google.golang.org/protobuf/internal/strs
#15 19.58 google.golang.org/protobuf/internal/order
#15 19.58 google.golang.org/protobuf/internal/genid
#15 19.77 google.golang.org/protobuf/reflect/protoregistry
#15 19.86 google.golang.org/protobuf/internal/encoding/text
#15 20.57 google.golang.org/protobuf/proto
#15 20.77 google.golang.org/protobuf/internal/encoding/defval
#15 21.38 google.golang.org/protobuf/encoding/prototext
#15 21.38 google.golang.org/protobuf/internal/filedesc
#15 21.38 github.com/matttproud/golang_protobuf_extensions/pbutil
#15 21.78 vendor/golang.org/x/net/http/httpproxy
#15 21.78 net/textproto
#15 21.78 crypto/x509
#15 21.86 github.com/prometheus/procfs
#15 22.36 vendor/golang.org/x/net/http/httpguts
#15 22.36 mime/multipart
#15 22.67 google.golang.org/protobuf/internal/encoding/tag
#15 22.86 google.golang.org/protobuf/internal/impl
#15 23.57 crypto/tls
#15 25.78 net/http/httptrace
#15 25.81 net/http
#15 25.92 google.golang.org/protobuf/internal/filetype
#15 26.06 google.golang.org/protobuf/runtime/protoimpl
#15 26.10 google.golang.org/protobuf/types/known/timestamppb
#15 26.10 google.golang.org/protobuf/types/descriptorpb
#15 26.18 github.com/golang/protobuf/ptypes/timestamp
#15 26.50 google.golang.org/protobuf/reflect/protodesc
#15 26.88 github.com/golang/protobuf/proto
#15 27.67 github.com/prometheus/client_model/go
#15 28.07 github.com/prometheus/client_golang/prometheus/internal
#15 28.68 expvar
#15 28.68 github.com/prometheus/common/expfmt
#15 28.71 # github.com/prometheus/common/expfmt
#15 28.71 ../github.com/prometheus/common/expfmt/decode.go:89:38: cannot use v (variable of type *io_prometheus_client.MetricFamily) as type protoreflect.ProtoMessage in argument to pbutil.ReadDelimited:
#15 28.71 	*io_prometheus_client.MetricFamily does not implement protoreflect.ProtoMessage (missing ProtoReflect method)
#15 28.71 ../github.com/prometheus/common/expfmt/encode.go:120:40: cannot use v (variable of type *io_prometheus_client.MetricFamily) as type protoreflect.ProtoMessage in argument to pbutil.WriteDelimited:
#15 28.71 	*io_prometheus_client.MetricFamily does not implement protoreflect.ProtoMessage (missing ProtoReflect method)

i have tried it also as recommended from go but this also not work:

#15 2.146 + go env -w GO111MODULE=on
#15 2.149 + go install github.com/prometheus/client_golang/prometheus@latest
#15 2.226 go: downloading github.com/prometheus/client_golang v1.13.0
#15 2.275 go: github.com/prometheus/client_golang/prometheus@latest (in github.com/prometheus/client_golang@v1.13.0):
#15 2.275 	The go.mod file for the module providing named packages contains one or
#15 2.275 	more exclude directives. It must not contain directives that would cause
#15 2.275 	it to be interpreted differently than if it were the main module.

and just in case you came again with this solution: #1154

this doesn`t help:

root@866a2fe8fad6:/go/src/camunda-engine-prometheus-exporter# GO111MODULE=off go get -v github.com/prometheus/client_golang
github.com/prometheus/client_golang (download)
package github.com/prometheus/client_golang: no Go files in /go/src/github.com/prometheus/client_golang

Currently, group and labels cannot have the same name. If any key is displayed, an error message is displayed. label:xxx already contains grouping label xxx

The implementation here is very inflexible, for example I want to dynamically group by some label

There is no such judgment on the prometheus-python client; I recommend using the value of group if it exists

It would be helpful if we the caller had access to canCollectProcess to know in advance if they can use the NewProcessCollector in the OS.

The following tests fail in v1.13.0 and main branch (as of writing).

=== RUN   TestGoCollectorAllowList
=== RUN   TestGoCollectorAllowList/Without_any_rules
=== RUN   TestGoCollectorAllowList/allow_all
    go_collector_latest_test.go:193: got [go_cgo_go_to_c_calls_calls_total go_gc_cycles_automatic_gc_cycles_total go_gc_cycles_forced_gc_cycles_total go_gc_cycles_total_gc_cycles_total go_gc_duration_seconds go_gc_heap_allocs_by_size_bytes go_gc_heap_allocs_bytes_total go_gc_heap_allocs_objects_total go_gc_heap_frees_by_size_bytes go_gc_heap_frees_bytes_total go_gc_heap_frees_objects_total go_gc_heap_goal_bytes go_gc_heap_objects_objects go_gc_heap_tiny_allocs_objects_total go_gc_limiter_last_enabled_gc_cycle go_gc_pauses_seconds go_gc_stack_starting_size_bytes go_goroutines go_info go_memory_classes_heap_free_bytes go_memory_classes_heap_objects_bytes go_memory_classes_heap_released_bytes go_memory_classes_heap_stacks_bytes go_memory_classes_heap_unused_bytes go_memory_classes_metadata_mcache_free_bytes go_memory_classes_metadata_mcache_inuse_bytes go_memory_classes_metadata_mspan_free_bytes go_memory_classes_metadata_mspan_inuse_bytes go_memory_classes_metadata_other_bytes go_memory_classes_os_stacks_bytes go_memory_classes_other_bytes go_memory_classes_profiling_buckets_bytes go_memory_classes_total_bytes go_memstats_last_gc_time_seconds go_sched_gomaxprocs_threads go_sched_goroutines_goroutines go_sched_latencies_seconds go_threads],
                                    want [go_gc_cycles_automatic_gc_cycles_total go_gc_cycles_forced_gc_cycles_total go_gc_cycles_total_gc_cycles_total go_gc_duration_seconds go_gc_heap_allocs_by_size_bytes go_gc_heap_allocs_bytes_total go_gc_heap_allocs_objects_total go_gc_heap_frees_by_size_bytes go_gc_heap_frees_bytes_total go_gc_heap_frees_objects_total go_gc_heap_goal_bytes go_gc_heap_objects_objects go_gc_heap_tiny_allocs_objects_total go_gc_pauses_seconds go_goroutines go_info go_memory_classes_heap_free_bytes go_memory_classes_heap_objects_bytes go_memory_classes_heap_released_bytes go_memory_classes_heap_stacks_bytes go_memory_classes_heap_unused_bytes go_memory_classes_metadata_mcache_free_bytes go_memory_classes_metadata_mcache_inuse_bytes go_memory_classes_metadata_mspan_free_bytes go_memory_classes_metadata_mspan_inuse_bytes go_memory_classes_metadata_other_bytes go_memory_classes_os_stacks_bytes go_memory_classes_other_bytes go_memory_classes_profiling_buckets_bytes go_memory_classes_total_bytes go_memstats_last_gc_time_seconds go_sched_goroutines_goroutines go_sched_latencies_seconds go_threads]
=== RUN   TestGoCollectorAllowList/allow_GC
    go_collector_latest_test.go:193: got [go_gc_cycles_automatic_gc_cycles_total go_gc_cycles_forced_gc_cycles_total go_gc_cycles_total_gc_cycles_total go_gc_duration_seconds go_gc_heap_allocs_by_size_bytes go_gc_heap_allocs_bytes_total go_gc_heap_allocs_objects_total go_gc_heap_frees_by_size_bytes go_gc_heap_frees_bytes_total go_gc_heap_frees_objects_total go_gc_heap_goal_bytes go_gc_heap_objects_objects go_gc_heap_tiny_allocs_objects_total go_gc_limiter_last_enabled_gc_cycle go_gc_pauses_seconds go_gc_stack_starting_size_bytes go_goroutines go_info go_memstats_last_gc_time_seconds go_threads],
                                    want [go_gc_cycles_automatic_gc_cycles_total go_gc_cycles_forced_gc_cycles_total go_gc_cycles_total_gc_cycles_total go_gc_duration_seconds go_gc_heap_allocs_by_size_bytes go_gc_heap_allocs_bytes_total go_gc_heap_allocs_objects_total go_gc_heap_frees_by_size_bytes go_gc_heap_frees_bytes_total go_gc_heap_frees_objects_total go_gc_heap_goal_bytes go_gc_heap_objects_objects go_gc_heap_tiny_allocs_objects_total go_gc_pauses_seconds go_goroutines go_info go_memstats_last_gc_time_seconds go_threads]
=== RUN   TestGoCollectorAllowList/allow_Memory
=== RUN   TestGoCollectorAllowList/allow_Scheduler
    go_collector_latest_test.go:193: got [go_gc_duration_seconds go_goroutines go_info go_memstats_last_gc_time_seconds go_sched_gomaxprocs_threads go_sched_goroutines_goroutines go_sched_latencies_seconds go_threads],
                                    want [go_gc_duration_seconds go_goroutines go_info go_memstats_last_gc_time_seconds go_sched_goroutines_goroutines go_sched_latencies_seconds go_threads]
--- FAIL: TestGoCollectorAllowList (0.00s)
    --- PASS: TestGoCollectorAllowList/Without_any_rules (0.00s)
    --- FAIL: TestGoCollectorAllowList/allow_all (0.00s)
    --- FAIL: TestGoCollectorAllowList/allow_GC (0.00s)
    --- PASS: TestGoCollectorAllowList/allow_Memory (0.00s)
    --- FAIL: TestGoCollectorAllowList/allow_Scheduler (0.00s)

Tested with Go 1.19.1 on linux/amd64, running kernel 5.19.

I'm not sure how this got past Circle CI, assuming that it runs tests. The release notes state "we also test client_golang against the new 1.19 version", but this doesn't appear to be true:

workflows:
  version: 2
  client_golang:
    jobs:
      # Refer to README.md for the currently supported versions.
      - test:
          name: go-1-17
          go_version: "1.17"
          run_lint: true
      - test:
          name: go-1-18
          go_version: "1.18"
          run_lint: true
          # Style and unused/missing packages are only checked against
          # the latest supported Go version.
          run_style_and_unused: true

// Following comment originates from https://pkg.go.dev/net/http#Transport

This depends on prometheus/client_model#61 .

Exemplars can be added to the buckets of conventional histograms but not to native histograms. So far, the way of sending exemplars is by also rendering a conventional histogram. With the issue above fixed, we have a way of sending exemplars independently of conventional buckets. However, without conventional buckets, we don't have a heuristics which exemplars to pick. Native histograms can have so many buckets that adding an examplar to each is infeasible.

This is P2 as explained in the milestone's description.

It is a good thing if the format can expose the conventional and the native version of a histogram side by side. But if a native histogram has a zero threshold of zero and zero observations yet, it might easily look like it’s not a native histogram at all, so Prometheus might ingest it assuming it’s a conventional histogram, which gets weird once it has actually received observations and Prometheus starts to ingest it as a native histogram. See code for details.

This is P2 as explained in the milestone's description.

The prototype already has strategies to limit the bucket count of a Native Histograms. Vet them, and consider more strategies. For example, it might come handy to limit the overall bucket count in a HistogramVec or the overall bucket count in a registry.

This is P2 as explained in the milestone's description. Once we are confident the existing strategies are stable and we can add more later without a breaking change, this can be considered P3.

A Native Histogram can easily be exposed as a conventional Histogram, if need be in a lower resolution, maybe with configurable bucket selection.

Currently, conventional and native histograms are tracked separately from each other, which might be fine for some use cases. However, it also takes resources for each. Rendering a conventional histogram from the native histogram would have no added cost compared to a pure native histogram. The tricky part is what boundaries to pick for the conventional buckets. Changes in the bucket layout should be avoided, so we could have some reasonable default or make it configurable for the user.

Probably we should only use boundaries exactly representable by the Native Histogram. Interpolation might come with implications, but maybe we should consider it.

The code below panics for duplicate registration because by default golang runs tests in parallel.
The actual code would work since New() is called only once so wondering if there could be any workaround that can use for the testing.
I really want to avoid having to use a custom register.

type TT struct {
	a prometheus.Gauge
}

func New() *TT {
	return &TT{
		a: promauto.NewGauge(prometheus.GaugeOpts{
			Namespace: "a",
			Subsystem: "a",
			Name:      "a",
			Help:      "a",
		}),
	}
}

func Test1(t *testing.T) {
	New()
}
func Test2(t *testing.T) {
	New()
}

I use main branch code with commit hash 5f202ee to test NativeHistogram feature,I set the NativeHistogramBucketFactor to math.Exp2(0.5), I really want use Schema 1 https://github.com/prometheus/client_golang/blob/main/prometheus/histogram.go#L44 buckets, but when I use Prometheus to collect data, find it use Schema 2 buckets.

When I change the NativeHistogramBucketFactor to math.Exp2(0.5001), it use Schema 1 buckets.

I've upgraded from Go 1.17.7 to 1.17.8 and I've noticed that exported metrics labels changed as a result.
le label on some histogram uses now different values than it used to:

--- a/metrics.txt
+++ b/metrics.txt
@@ -105,13 +105,13 @@ go_gc_heap_tiny_allocs_objects_total
 go_gc_pauses_seconds_total_bucket{le="-5e-324"}
 go_gc_pauses_seconds_total_bucket{le="9.999999999999999e-10"}
 go_gc_pauses_seconds_total_bucket{le="9.999999999999999e-09"}
-go_gc_pauses_seconds_total_bucket{le="1.2799999999999998e-07"}
-go_gc_pauses_seconds_total_bucket{le="1.2799999999999998e-06"}
-go_gc_pauses_seconds_total_bucket{le="1.6383999999999998e-05"}
-go_gc_pauses_seconds_total_bucket{le="0.00016383999999999998"}
-go_gc_pauses_seconds_total_bucket{le="0.0020971519999999997"}
-go_gc_pauses_seconds_total_bucket{le="0.020971519999999997"}
-go_gc_pauses_seconds_total_bucket{le="0.26843545599999996"}
+go_gc_pauses_seconds_total_bucket{le="9.999999999999998e-08"}
+go_gc_pauses_seconds_total_bucket{le="1.0239999999999999e-06"}
+go_gc_pauses_seconds_total_bucket{le="1.0239999999999999e-05"}
+go_gc_pauses_seconds_total_bucket{le="0.00010239999999999998"}
+go_gc_pauses_seconds_total_bucket{le="0.0010485759999999998"}
+go_gc_pauses_seconds_total_bucket{le="0.010485759999999998"}
+go_gc_pauses_seconds_total_bucket{le="0.10485759999999998"}
 go_gc_pauses_seconds_total_bucket{le="+Inf"}
 go_gc_pauses_seconds_total_sum NaN
 go_gc_pauses_seconds_total_count
@@ -240,13 +240,13 @@ go_sched_goroutines_goroutines
 go_sched_latencies_seconds_bucket{le="-5e-324"}
 go_sched_latencies_seconds_bucket{le="9.999999999999999e-10"}
 go_sched_latencies_seconds_bucket{le="9.999999999999999e-09"}
-go_sched_latencies_seconds_bucket{le="1.2799999999999998e-07"}
-go_sched_latencies_seconds_bucket{le="1.2799999999999998e-06"}
-go_sched_latencies_seconds_bucket{le="1.6383999999999998e-05"}
-go_sched_latencies_seconds_bucket{le="0.00016383999999999998"}
-go_sched_latencies_seconds_bucket{le="0.0020971519999999997"}
-go_sched_latencies_seconds_bucket{le="0.020971519999999997"}
-go_sched_latencies_seconds_bucket{le="0.26843545599999996"}
+go_sched_latencies_seconds_bucket{le="9.999999999999998e-08"}
+go_sched_latencies_seconds_bucket{le="1.0239999999999999e-06"}
+go_sched_latencies_seconds_bucket{le="1.0239999999999999e-05"}
+go_sched_latencies_seconds_bucket{le="0.00010239999999999998"}
+go_sched_latencies_seconds_bucket{le="0.0010485759999999998"}
+go_sched_latencies_seconds_bucket{le="0.010485759999999998"}
+go_sched_latencies_seconds_bucket{le="0.10485759999999998"}
 go_sched_latencies_seconds_bucket{le="+Inf"}
 go_sched_latencies_seconds_sum NaN
 go_sched_latencies_seconds_count

It's not a huge problem but it makes it harder to compare metrics between 2 instances of the same service compiled with different Go version. This is likely related to #967

When writing histogram metrics, and there is an exemplar for the +Inf bucket, then the +Inf bucket is added with the value of the previous bucket:
https://github.com/prometheus/client_golang/blob/main/prometheus/metric.go#L188

This is incorrect, as the cumulative count of the +Inf bucket should instead be added with the total count of the histogram datapoint.

The above behaviour results in invalid values being reported for the +Inf bucket.