kakkoyun
Repos
149
Followers
458
Following
417

eBPF based always-on profiler auto-discovering targets in Kubernetes and systemd, zero code changes or restarts needed!

238
37

Prometheus instrumentation library for Go applications

4275
983

Highly available Prometheus setup with long term storage capabilities. A CNCF Incubating project.

11230
1688

Continuous profiling for analysis of CPU and memory usage, down to the line number and throughout time. Saving infrastructure cost, improving performance, and increasing reliability.

2742
147

The Prometheus monitoring system and time series database.

45714
7394

❄️ Coolest database around 🧊 Embeddable column database written in Go.

834
45

Events

Updated waka-readme graph with new metrics

Created at 11 hours ago
pull request opened
Update README.md

Remove out-dated information.

Created at 19 hours ago
create branch
kakkoyun create branch kakkoyun-update-readme
Created at 19 hours ago
delete branch
kakkoyun delete branch merge_release_0101
Created at 20 hours ago
pull request closed
Merge release v0.10.1

⚠️ Do not squash and merge ⚠️

There is a merge conflict with release-0.10; thus, I have created a new branch for the merge.

Created at 20 hours ago

Merge pull request #1053 from jnsgruk/fix-snap-failures

ci: deconflict docker/lxd iptables rule for snapcraft builds

Conditionally set unwind tables map size (#1074)

Right now, we set the unwind tables map size to 100 elements. This means that we havce to allocate and lock roughly 100 elements * 250k entries * 16 bytes per entry ~= 400 MB, just for this map.

We are currently working on some optimisations that will result in lower memory required for the unwind tables, but this will take some weeks to land, so in the meantime, this commit reduces the default unwind tables map size to 2, which would be around 8MB, and only bump it to a 100 elements if dwarf based unwinding is enabled.

We still want to allocate and lock some memory even if we know that dwarf unwinding is disabled, as in the future this would be transparently enabled and it's highly likely that at least few processes (or mappings, soon) will use the dwarf based unwind approach. This way we will at least ensure that we can already allocate enough for 500k unwind rows.

Test Plan

by default:

[javierhonduco@fedora parca-agent]$ sudo bpftool map | grep unwind_tables -A2
70: hash  name unwind_tables  flags 0x0
        key 8B  value 4000032B  max_entries 2  memlock 8003584B
        btf_id 254

with --experimental-enable-dwarf-unwinding

[javierhonduco@fedora parca-agent]$ sudo bpftool map | grep unwind_tables -A2
47: hash  name unwind_tables  flags 0x0
        key 8B  value 4000032B  max_entries 100  memlock 400007168B
        btf_id 230

Signed-off-by: Francisco Javier Honduvilla Coto javierhonduco@gmail.com

Signed-off-by: Francisco Javier Honduvilla Coto javierhonduco@gmail.com

Bump unwind table shards from 3 to 6

Going from 0.75M to 1.5M unwind table rows in total

Merge pull request #1076 from javierhonduco/increase-unwind-table-shards

Bump unwind table shards from 3 to 6

Make sure we handle errors when config file path is specified (#1064)

Signed-off-by: Kemal Akkoyun kakkoyun@gmail.com

Signed-off-by: Kemal Akkoyun kakkoyun@gmail.com

Reduce the updates for unwind tables (#1083)

Signed-off-by: Kemal Akkoyun kakkoyun@gmail.com

Signed-off-by: Kemal Akkoyun kakkoyun@gmail.com

Lower the required memlock rlimit if dwarf unwinding is not enabled (#1080)

  • Lower the required memlock rlimit if dwarf unwinding is not enabled

Signed-off-by: Kemal Akkoyun kakkoyun@gmail.com

  • Lower the value for DWARF unwinding

Signed-off-by: Kemal Akkoyun kakkoyun@gmail.com

  • Lower the memory lock limit when DWARF unwinding is not enabled

Signed-off-by: Kemal Akkoyun kakkoyun@gmail.com

  • Address review issues

Signed-off-by: Kemal Akkoyun kakkoyun@gmail.com

Signed-off-by: Kemal Akkoyun kakkoyun@gmail.com

Revert binary path changes

Signed-off-by: Kemal Akkoyun kakkoyun@gmail.com

Regenerate README

Signed-off-by: Kemal Akkoyun kakkoyun@gmail.com

Remove required rlimit for memlock (#1085)

  • Increase the lower rlimit for memlock

16MB limit did not work for Kubernetes on Minikube

Signed-off-by: Kemal Akkoyun kakkoyun@gmail.com

  • Remove the limits by default

Signed-off-by: Kemal Akkoyun kakkoyun@gmail.com

Signed-off-by: Kemal Akkoyun kakkoyun@gmail.com

Re-generate README

Signed-off-by: Kemal Akkoyun kakkoyun@gmail.com

Merge branch 'main' into merge_release_0101

Merge pull request #1087 from parca-dev/merge_release_0101

Merge release v0.10.1

Created at 20 hours ago

Add a make action to build debug binaries (#1035)

Signed-off-by: Kemal Akkoyun kakkoyun@gmail.com

Signed-off-by: Kemal Akkoyun kakkoyun@gmail.com

Add more local debug information search patterns (#1036)

Signed-off-by: Kemal Akkoyun kakkoyun@gmail.com

Signed-off-by: Kemal Akkoyun kakkoyun@gmail.com

bpf: Reduce BPF map memory usage

Reduce BPF map memory usage by using one BPF map for the multiple shards.

When sharding was introduced, multiple BPF maps were added for each of the shards. This is very wasteful as the kernel must preallocate and lock the memory, so it can never be paged out.

The other issue is that machines with low memory count would not have enough memory to create all the maps.

Before this commit we were using 3 shards * 256 (MAX_PID_MAP_SIZE ) * 64B * 2 (size of each row) * 250k MAX_UNWIND_TABLE_SIZE + other metadata that's not too big ~= 24.6GB. This is obviously a ridiculous amount of memory.

This commit changes the sharded table scheme to just use one BPF map where they key is (pid, shard_id) with the same size, so it will roughly lock 8GB. This number is still rather large. I've reduced MAX_PID_MAP_SIZE to 100, which brings the locked memory now to 3.2GB.

The good news is that by using a single BPF map we can easily fine-tune the shard size to a more reasonable value in the future.

Memory usage should be reduced in the userspace code, too, as before we were naively adding all the shards to their buffer and then writting them. This commits reuses the created buffer and never holds the whole, BPF-friendly unwind table in memory.

(See test plan in the PR)

Signed-off-by: Francisco Javier Honduvilla Coto javierhonduco@gmail.com

Merge pull request #1042 from parca-dev/improve-unwind-table-sharding

bpf: Reduce BPF map memory usage

Merge branch 'main' into merge_release_branch_010

Merge pull request #1049 from parca-dev/merge_release_branch_010

Merge Release 0.10 branch to main

build(deps): update docker.io/library/busybox:1.35.0 docker digest to ec6dc15 (#1044)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Kemal Akkoyun kakkoyun@users.noreply.github.com

chore(deps): pin kylemayes/install-llvm-action action to 665aaf9 (#1050)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Maxime Brunet max@brnt.mx

ci: deconflict docker/lxd iptables rule for snapcraft builds

docs/eh_frame: Add features and limitations doc

Signed-off-by: Francisco Javier Honduvilla Coto javierhonduco@gmail.com

Merge pull request #1053 from jnsgruk/fix-snap-failures

ci: deconflict docker/lxd iptables rule for snapcraft builds

Merge pull request #1054 from parca-dev/add-dwarf-based-stack-walker-docs

docs/eh_frame: Add features and limitations doc

testdata: Update

Signed-off-by: Francisco Javier Honduvilla Coto javierhonduco@gmail.com

.eh_frame: Initial DWARF expression support

This commits adds partial support for DWARF expressions, in particular, for unwind ranges that cover PLTs for the CFA calculation (DW_CFA_def_cfa_expression).

Arbitrary expressions require a VM to evaluate them, which is not easy to do in BPF, so after doing some quick data analysis (see below) it was clear that by adding some hardcoded expressions to increase the success ratio of the stack walker without bloating it too much. I don't think we should add every "common" expression, but the PLT ones seemed like good bang for the buck.

On my Fedora machine, these were the most common expressions for binaries, and libraries, as well as for processes running:

[javierhonduco@fedora parca-agent]$ sudo readelf -wf /bin/* 2>/dev/null | grep DW_CFA_def_cfa_expression | sort | uniq -c | sort -k1h
      1   DW_CFA_def_cfa_expression (DW_OP_breg7 (rsp): 160; DW_OP_deref)
      5   DW_CFA_def_cfa_expression (DW_OP_breg4 (esp): 4; DW_OP_breg8 (eip): 0; DW_OP_lit15; DW_OP_and; DW_OP_lit9; DW_OP_ge; DW_OP_lit2; DW_OP_shl; DW_OP_plus)
      5   DW_CFA_def_cfa_expression (DW_OP_breg5 (ebp): -16; DW_OP_deref)
     12   DW_CFA_def_cfa_expression (DW_OP_breg6 (rbp): -40; DW_OP_deref)
     62   DW_CFA_def_cfa_expression (DW_OP_breg7 (rsp): 8; DW_OP_breg16 (rip): 0; DW_OP_lit15; DW_OP_and; DW_OP_lit11; DW_OP_ge; DW_OP_lit3; DW_OP_shl; DW_OP_plus)
   1673   DW_CFA_def_cfa_expression (DW_OP_breg7 (rsp): 8; DW_OP_breg16 (rip): 0; DW_OP_lit15; DW_OP_and; DW_OP_lit10; DW_OP_ge; DW_OP_lit3; DW_OP_shl; DW_OP_plus)
[javierhonduco@fedora parca-agent]$ sudo readelf -wf /lib64/* 2>/dev/null | grep DW_CFA_def_cfa_expression | sort | uniq -c | sort -k1h
[...]
     47   DW_CFA_def_cfa_expression (DW_OP_breg7 (rsp): 8; DW_OP_breg16 (rip): 0; DW_OP_lit15; DW_OP_and; DW_OP_lit11; DW_OP_ge; DW_OP_lit3; DW_OP_shl; DW_OP_plus)
    237   DW_CFA_def_cfa_expression (DW_OP_breg6 (rbp): -24; DW_OP_deref)
    910   DW_CFA_def_cfa_expression (DW_OP_breg6 (rbp): -8; DW_OP_deref)
   1006   DW_CFA_def_cfa_expression (DW_OP_breg6 (rbp): -40; DW_OP_deref)
   2563   DW_CFA_def_cfa_expression (DW_OP_breg7 (rsp): 8; DW_OP_breg16 (rip): 0; DW_OP_lit15; DW_OP_and; DW_OP_lit10; DW_OP_ge; DW_OP_lit3; DW_OP_shl; DW_OP_plus)
[javierhonduco@fedora parca-agent]$ sudo readelf -wf /proc/*/exe 2>/dev/null | grep DW_CFA_def_cfa_expression | sort | uniq -c | sort -k1h
[...]
     25   DW_CFA_def_cfa_expression (DW_OP_breg7 (rsp): 120; DW_OP_deref; DW_OP_plus_uconst: 8)
     25   DW_CFA_def_cfa_expression (DW_OP_breg7 (rsp): 152; DW_OP_deref; DW_OP_plus_uconst: 8)
     25   DW_CFA_def_cfa_expression (DW_OP_breg7 (rsp): 88; DW_OP_deref; DW_OP_plus_uconst: 8)
     30   DW_CFA_def_cfa_expression (DW_OP_breg7 (rsp): 0; DW_OP_deref; DW_OP_plus_uconst: 8)
     35   DW_CFA_def_cfa_expression (DW_OP_breg7 (rsp): 40; DW_OP_deref; DW_OP_plus_uconst: 8)
     45   DW_CFA_def_cfa_expression (DW_OP_breg7 (rsp): -8; DW_OP_deref; DW_OP_plus_uconst: 8)
    130   DW_CFA_def_cfa_expression (DW_OP_breg7 (rsp): 8; DW_OP_breg16 (rip): 0; DW_OP_lit15; DW_OP_and; DW_OP_lit10; DW_OP_ge; DW_OP_lit3; DW_OP_shl; DW_OP_plus)

As it can be seen above, by adding PLT expression support, we can get ~50% more unwind sections working.

PLT sections are particularly important as libc is typically dynamically linked. In this case, most libc calls go through the Procedure Linkage Table, and before this commit, we will stop walking the stack.

See test plan in the PR.

Signed-off-by: Francisco Javier Honduvilla Coto javierhonduco@gmail.com

Merge pull request #1058 from parca-dev/dwarf-expressions-plt

Add support for dwarf expressions for PLTs

ci(e2e): pass container image as artifact between jobs (#1061)

ci(pre-commit): apply automatic fixes (#1059)

ci: fix snap build for tags (#1062)

Reduce elf.Open() calls (#1001)

  • pkg/buildid: use elf file instead of filepath

  • pkg/objectfile: reduce elf.Open() calls

  • pass elf file as a field to Objectfile struct type
  • update tests accordingly
  • fix golang-ci lints

  • pkg/buildid: Refactor buildid api

  • unify path and elf file into an ElfFile struct type
  • update references to and tests for BuildID() accordingly

Signed-off-by: Sumera Priyadarsini sylphrenadin@gmail.com

  • Update pkg/buildid/buildid_test.go

Signed-off-by: Sumera Priyadarsini sylphrenadin@gmail.com

Replace gzip lib to reduce allocs in profile writer (#1065)

Created at 21 hours ago
create branch
kakkoyun create branch merge_release_0101
Created at 21 hours ago
pull request opened
Merge release v0.10.0
Created at 21 hours ago
kakkoyun create tag v0.10.1
Created at 21 hours ago

Re-generate README

Signed-off-by: Kemal Akkoyun kakkoyun@gmail.com

Created at 21 hours ago
delete branch
kakkoyun delete branch increase_lower_rlimit
Created at 23 hours ago

Remove required rlimit for memlock (#1085)

  • Increase the lower rlimit for memlock

16MB limit did not work for Kubernetes on Minikube

Signed-off-by: Kemal Akkoyun kakkoyun@gmail.com

  • Remove the limits by default

Signed-off-by: Kemal Akkoyun kakkoyun@gmail.com

Signed-off-by: Kemal Akkoyun kakkoyun@gmail.com

Created at 23 hours ago
pull request closed
Remove required rlimit for memlock

Remove rlimit memlock by default. This shouldn't create a problem for the kernel version >=5.11 We make sure it's removed for older kernel versions by default.

Signed-off-by: Kemal Akkoyun kakkoyun@gmail.com

cc @javierhonduco

Created at 23 hours ago
Created at 23 hours ago

Remove the limits by default

Signed-off-by: Kemal Akkoyun kakkoyun@gmail.com

Created at 1 day ago
started
Created at 1 day ago
opened issue
Launch the program as root (with sudo)

I'm trying to debug a program that needs root privileges. Is there a way to achieve this?

Created at 1 day ago
Created at 1 day ago
issue comment
fix: re-introduce memory bumping

Hey @rafaeldtinoco, what was the reason for this revert? I couldn't find it.

Would you also be open to making it configurable? We'd love to disable it or find a way to specify the value of the lock. I'd be happy to contribute. I want to understand the reasoning and possible downsides.

Created at 1 day ago
opened issue
Add shellcheck

https://github.com/parca-dev/parca-agent/blob/f5d49fbe70bc96d2d37e543bf419181e5c53a619/.pre-commit-config.yaml

Created at 1 day ago
issue comment
Fix/debuginfofile

@zdyj3170101136如果您需要我们这边的进一步帮助来推动这件事,请告诉我。

which ubuntu version are you using? i try to run 22.01 in mac, and the output is wrong:

docker run -it --name ubuntu ubuntu:22.04 bash

In CI, we use ubuntu-latest, so v20.04 https://github.com/actions/runner-images#available-images

Created at 1 day ago

Updated waka-readme graph with new metrics

Created at 1 day ago

Regenerate README

Signed-off-by: Kemal Akkoyun kakkoyun@gmail.com

Created at 1 day ago

Revert binary path changes

Signed-off-by: Kemal Akkoyun kakkoyun@gmail.com

Created at 1 day ago
closed issue
docs: Update design.md with the new architecture

Introduced in system-wide (https://github.com/parca-dev/parca-agent/pull/627)

Created at 1 day ago
issue comment
docs: Update design.md with the new architecture

Fixed by https://github.com/parca-dev/docs/pull/202 and https://github.com/parca-dev/parca-agent/pull/1084

Created at 1 day ago