fstab
Repos
116
Followers
133
Following
5

Events

Attribute to indicate whether a Span is used as an Exemplar

Would love a better holistic analysis/proposal for tail-sampling.

I can spend a couple of hours next week to write a design doc. Then we have something to start with. It would be great to find a way to make exemplars work with tail-sampling.

Created at 3 days ago
Attribute to indicate whether a Span is used as an Exemplar

the Exemplar specification defines a pluggable reservoir component

Is it possible to change this and allow other implementations? I understand the issue with the reservoir is that at the point in time when you put a span in the reservoir you don't know yet whether you will actually use it as an Exemplar.

I think there could be other implementations that decide directly whether or not to use a span as an Exemplar, without using a reservoir. For example, you could just select a new Exemplar every n seconds.

Created at 5 days ago
issue comment
Package JPProf as a Java agent

Hi Alexander, I agree that having a profiler that you can just attach to any application without requiring code change would be great. We have this on our roadmap, but it might take a while.

Created at 6 days ago
issue comment
How to add exemplars to pure Native Histograms

OpenTelemetry solved this by just adding a list of Exemplars. Sounds like this would work for Prometheus as well? https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/metrics/v1/metrics.proto#L557

Created at 1 week ago
issue comment
why in this place use System.exit(1)?

@dhoard thanks a lot for looking into this. I agree the agent is not written with dynamic attachment in mind. There are some other issues as well, like the agent does not detach cleanly as some classes will still be referenced after detaching the agent (see https://github.com/prometheus/client_java/issues/809). The recommended usage is either to attach the agent at startup and leave it attached, or to run jmx_exporter_httpserver in standalone mode and scrape JMX beans remotely via the network.

Created at 1 week ago

Histogram exemplar test

Created at 2 weeks ago

first half of explicit buckets histogram tests

Created at 2 weeks ago
closed issue
Jakarta Servlet 5.0

Looks like servlet-api are moving to jakarta package names. Any plans for upgrading client_java to support it? It will probably take several years before everyone is over, so perhaps we need parallel releases?

https://jakarta.ee/specifications/servlet/5.0/

Created at 2 weeks ago
issue comment
Jakarta Servlet 5.0

Isn't it this one? Or am I confusing things? https://github.com/prometheus/client_java/tree/main/simpleclient_servlet_jakarta

Created at 2 weeks ago
issue comment
HTTP server responds with metrics on any url

I think it makes sense to expose metrics on only one path, defaulting to /metrics. However, it would be a breaking change.

We can do breaking changes with rel 1.0 because of the major version jump. So instead of implementing this now in a backwards-compatible way, it might be better to do it with 1.0 in a breaking way.

Created at 2 weeks ago

Add counter exemplar test

Created at 2 weeks ago
Attribute to indicate whether a Span is used as an Exemplar

For reference: The Prometheus Java metrics library has a retention time for Exemplars, which defaults to 7s. Exemplars are kept for min 7s before the next Exemplar is sampled. This should work fine with marking Spans as "used as an Exemplar", as these spans would only be produced once every 7s.

Is there any chance to change the Spec to allow something like this in the OpenTelemetry SDK? And if that's not possible, is there any chance to introduce a "used as exemplar" attribute anyway for 3rd party metric libraries like Prometheus?

Created at 2 weeks ago
Attribute to indicate whether a Span is used as an Exemplar

Thanks Cijo, that's interesting. Tbh I don't understand why the SDKs implement it that way. You can only have one Exemplar per metric per collection cycle, so I don't see why Exemplars should be updated more frequently.

My own background is with Prometheus metrics, I'm maintainer of the Prometheus Java client library. Updating the Exemplar only once per collection cycle should be straightforward to implement in Prometheus libraries, and I guess also in other metrics libraries out there.

I'm wondering whether this could still be considered in the interest of making OpenTelemetry tracing (and sampling) work well with Prometheus metric instrumentation?

Created at 2 weeks ago
issue comment
-javaagent failed, processJavaStart failed

Do you have any other agent attached in addition to jmx_exporter?

Created at 2 weeks ago
pull request opened
Add fstab
Created at 3 weeks ago

Add fstab

Created at 3 weeks ago
Created at 3 weeks ago
Attribute to indicate whether a Span is used as an Exemplar

What are you trying to achieve?

Exemplars are used to navigate from metrics to example traces. For example, the screenshot shows latencies from the http.server.duration metric as visualized by Grafana. Each little dot represents an Exemplar. If you click on an Exemplar, you can navigate to the corresponding trace.

screenshot_2022-10-05_16:23:20_457465619

However, this does not work well if traces are sampled in the collector: In many cases the metric library will choose a trace as an exemplar that will be discarded in the collector during sampling.

It would be good if metric libraries could add a standard Span attribute to indicate that a Span has been selected as an Exemplar. This attribute could be considered by the collector's sampling algorithm to ensure that Exemplars aren't discarded.

A simple way to implement this is adding a boolean attribute, like exemplar="true".

A better way to implement this would be an attribute containing the metric name, like `exemplar.metric="http.server.duration". If the attribute is present, it means that the Span was used as an Exemplar. As an additional benefit, the metric name could be used by vendors to allow navigation from a trace to the dashboard showing the corresponding metric.

What do you think?

P.S.: If a Github issue here is the wrong place for this let me know where else to put it.

Created at 3 weeks ago

Counter name handling

Created at 3 weeks ago

Add initial counter test

Created at 3 weeks ago
issue comment
why in this place use System.exit(1)?

Yes. If you start the application with an illegal command line, the application exits immediately.

Created at 3 weeks ago
issue comment
why in this place use System.exit(1)?

What else would you expect to happen if you run it with illegal command line parameters?

Created at 3 weeks ago

First test successfully ported

Created at 3 weeks ago
Metrics Annotations

Frameworks like Spring Boot, MicroProfile, and Dropwizard offer annotations like @Timed or @Counted for creating metrics from method calls.

The benefit is that developers can provide metrics for their business logic while still separating business code from metrics tracking.

@Timed
@Counted
public void processPayment() {
    // business code here
}

The OpenTelemetry Java auto instrumentation supports the @WithSpan annotation for tracing, but has nothing equivalent for metrics yet.

Brainstorming of a few ideas:

  • For @Timed the type of histogram should be configurable, like @Timed(histogramType = EXPONENTIAL). Other histogram types would be explicit buckets and default buckets.
  • Explicit buckets should be configurable: @Timed(buckets = { 0.001, 0.002, 0.003, 0.004 })
  • For exponential histograms the resolution should be configurable.
  • Static attributes should be configurable: @Counted(attributes = { "key", "value" })
  • Dynamic attributes should be configurable similar to OpenTelemetry's @SpanAttribute.
  • Dynamic attributes based on the return value and on an exception thrown would be great.

Moreover, it would be awesome to integrate this with @WithSpan so that develpers can say "I want to time this method AND have a Span". However, it should also be possible to use tracing without metrics, because histograms may introduce a lot of cardinality, and users might just want to create a Span without creating a histogram.

What do you think, is it worthwhile to put some more thoughts into this?

Created at 3 weeks ago

Update README.md

Merge pull request #2 from grafana/fstab-patch-1

Add linke to the Faro OSS page

Created at 3 weeks ago
pull request closed
Add linke to the Faro OSS page

Link to the Faro OSS page as the current README doesn't have a lot of information.

Created at 3 weeks ago
pull request opened
Update README.md

Link to the Faro OSS page as the current README doesn't have a lot of information.

Created at 3 weeks ago
create branch
fstab create branch fstab-patch-1
Created at 3 weeks ago

temporary backup commit

Created at 3 weeks ago
issue comment
What is best practice to query datas use java?

Prometheus is a time series database. That means it does not only have the current value of metrics, but also the history of previous values.

A good example why this is needed is a request count for an HTTP service, something like http_requests_total. If you just know the current value, it tells you nothing. However, if you know how much the count has increased over the past 5 minutes, this is valuable information, because it gives you the the request rate based on a 5 minute time window.

For many cases, you need the time series data. You can only get that when querying Prometheus, not from a client library.

Created at 4 weeks ago