We should bring over Caddy's updated algorithm into this package. :+1:

@GitRowin Ah, yes that's #174 -- I think we can implement this as well. Hopefully the 'ask' endpoint is very fast, so as to minimize latency on that first handshake.

As for the timeouts, let me look closer when I have a chance.

By reading the git history, I found that caddy has changed the hashing algorithm func hostByHashing(pool []*Upstream, s string) *Upstream) which brings some troubles to the joint use of caddy and caddy-l4, please refer to this issue for details. Is it possible to keep the old and new hash algorithms at the same time, instead of replacing the old with the new?

The old algorithm was flawed (see this also); the new one is much better. I don't see any compelling reason to keep the old one.

You mean this order? https://caddyserver.com/docs/caddyfile/directives#directive-order

That's why I'm thinking maybe we could keep a single ephemeral key in memory. The cert is pretty cheap after that.

@francislavoie What if the self-signed cert was discarded after being used for that handshake?

This might be unacceptably slow, however... unless maybe we keep a single ephemeral key in memory.

Code LGTM -- I think there's some good discussion points in here.

The matcher idea is interesting. In the future, maybe we could use this to "feed" a matcher for a trusted proxy? I.e. it can pull from this global option and use it to determine whether a requests matches or not.

I thought about passing the req to this interface, but then it's more push than pull, and acts more like a request matcher.

True, it does feel like a matcher -- however I have not regretted passing the *Request in, because it does open a lot more possibilities and still grants the same functionality at no higher cost or complexity. (We just have to trust that the request being passed in isn't being changed inadvertently.)

And actually, the "push" model feels more enticing to me because of the possibilities for dynamic answers. Kind of like how we do for reverse proxy upstreams, being able to get them dynamically at every request.

Would it be so bad for the trusted proxies implementation to actually be a set of request matchers instead? That way we're not limited to just IP ranges in the future.

This is a valid Caddy Configuration?

  reverse_proxy http://localhost:3000 {
         buffer_requests
         buffer_responses
         flush_interval -1
         max_buffer_size 35MiB
        }

https://github.com/openspeedtest/Speed-Test/issues/67

I am running OpenSpeedTest Container

docker run --restart=unless-stopped --name openspeedtest -d -p 3000:3000 -p 3001:3001 openspeedtest/latest

Caddy Performance

• 9500 for Download and 9600 for Download when using HTTP

• 5400 for Download and 1200 for Upload when using HTTPS

Why I am getting very slow performance when I use HTTPS?

Test without using Caddy

• 9500 for Download and 9600 for Download when using HTTP (3000 Docker Port)

• 9500 for Download and 9600 for Download when using HTTPS (3001 Docker Port)

This is a valid Caddy Configuration?

No, it's missing the site address at the top.

So far, this doesn't look like a bug in Caddy. It looks more like a question about OpenSpeedTest rather than a bug report or feature request for Caddy.

Since this issue tracker is reserved for actionable development items, I'm going to close this, but we have a community forum at caddy.community where more people will be exposed to your question, including people who may be more expert or experienced than I am with the specific issue you're facing. I hope you'll ask your question there. The Help category in the forum even provides a template you can fill out.

If I'm wrong, and this does turn out to be a bug after further investigation, we can reopen this. Thanks for understanding!

If it's a bug, I'll need the minimal instructions to reproduce the bug on my machine, including code.

But the reality is, I think it is some sort of external or network issue. "i/o timeout" means packets weren't being delivered quickly enough to/from the nameserver.

@powerman Could you please open an issue and fill out this bug report template? It will help us identify whether it is a bug or simply a configuration issue.

Template

Instructions -- please heed otherwise we cannot help you (help us help you!)

1. Environment: Please fill out your OS and Caddy versions, even if you don't think they are relevant. (They are always relevant.) If you built Caddy from source, provide the commit SHA and specify your exact Go version.

2. Description: Describe at a high level what the bug is. What happens? Why is it a bug? Not all bugs are obvious, so convince readers that it's actually a bug.

   • 2c) Log output: Paste terminal output and/or complete logs in a code block. DO NOT REDACT INFORMATION except for credentials. Please enable debug and access logs.
   • 2d) Workaround: What are you doing to work around the problem in the meantime? This can help others who encounter the same problem, until we implement a fix.
   • 2e) Relevant links: Please link to any related issues, pull requests, docs, and/or discussion. This can add crucial context to your report.

3. Tutorial: What are the minimum required specific steps someone needs to take in order to experience the same bug? Your goal here is to make sure that anyone else can have the same experience with the bug as you do. You are writing a tutorial, so make sure to carry it out yourself before posting it. Please:

   • Start with an empty config. Add only the lines/parameters that are absolutely required to reproduce the bug.
   • Do not run Caddy inside containers.
   • Run Caddy manually in your terminal; do not use systemd or other init systems.
   • If making HTTP requests, avoid web browsers. Use a simpler HTTP client instead, like curl.
   • Do not redact any information from your config (except credentials). Domain names are public knowledge and often necessary for quick resolution of an issue!
   • Note that ignoring this advice may result in delays, or even in your issue being closed. 😞 Only actionable issues are kept open, and if there is not enough information or clarity to reproduce the bug, then the report is not actionable.

Example of a tutorial:

{ ... }

Open terminal and run Caddy:

$ caddy ...

Make an HTTP request:

$ curl ...

Notice that the result is ___ but it should be ___.

Ah, thanks Mohammed.

@JensErat After a brief few days vacation, I've returned to the office and am ready to do a final review and merge this, whenever you're ready! :+1: You might be more qualified to resolve the merge conflicts. Let me know if I can help though.

It looks like something is failing to connect. Likely a nameserver misconfiguration or an issue with the environment's local DNS configuration. Make sure DNS lookups on that nameserver are able to succeed on your server. :+1:

1.6746582486760027e+09  error   acme_client     cleaning up solver      {"identifier": "*.51pwn.com", "challenge_type": "dns-01", "error": "no memory of presenting a DNS record for \"_acme-challenge.51pwn.com\" (usually OK if presenting also failed)"}
1.6746582487523222e+09  error   obtain  could not get certificate from issuer   {"identifier": "*.51pwn.com", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "[*.51pwn.com] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of \"_acme-challenge.51pwn.com\": dial tcp: lookup ns1.51pwn.com.: i/o timeout (order=https://acme-v02.api.letsencrypt.org/acme/order/933097807/160934339777) (ca=https://acme-v02.api.letsencrypt.org/directory)"}
1.674658248753018e+09   info    obtain  releasing lock  {"identifier": "*.51pwn.com"}
[ERR] An error occurred while applying for a certificate, error: [*.51pwn.com] Obtain: [*.51pwn.com] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of "_acme-challenge.51pwn.com": dial tcp: lookup ns1.51pwn.com.: i/o timeout (order=https://acme-v02.api.letsencrypt.org/acme/order/933097807/160934339777) (ca=https://acme-v02.api.letsencrypt.org/directory)

error   acme_client     cleaning up solver      {"identifier": "*.exploit-poc.com", "challenge_type": "dns-01", "error": "no memory of presenting a DNS record for \"_acme-challenge.exploit-poc.com\" (usually OK if presenting also failed)"}
1.6746582711675534e+09  error   obtain  could not get certificate from issuer   {"identifier": "*.exploit-poc.com", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "[*.exploit-poc.com] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of \"_acme-challenge.exploit-poc.com\": dial tcp: lookup ns1.51pwn.com.: i/o timeout (order=https://acme-v02.api.letsencrypt.org/acme/order/933097807/160934403287) (ca=https://acme-v02.api.letsencrypt.org/directory)"}
1.6746582711682625e+09  info    obtain  releasing lock  {"identifier": "*.exploit-poc.com"}

An error occurred while applying for a certificate, error: [*.exploit-poc.com] Obtain: [*.exploit-poc.com] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of "_acme-challenge.exploit-poc.com": dial tcp: lookup ns1.51pwn.com.: i/o timeout (order=https://acme-v02.api.letsencrypt.org/acme/order/933097807/160934403287) (ca=https://acme-v02.api.letsencrypt.org/directory)

ns1\ns2 is ok

https://github.com/projectdiscovery/interactsh/issues/447

No worries; just run go get github.com/caddyserver/caddy/v2@latest and then run go mod tidy. It should update go.mod and go.sum.

Sure -- I'm pretty busy this week though, however I could click a button to merge a PR if you're interested in that!

The module compiles correctly for Caddy 2.6.2, but NTLM passthrough does not seem to work. In the previous version we used (2.4.6), it worked perfectly. Upstream server is IIS 7.5.

Closing due to inactivity. If we get more information we can re-open!

Hey Brad, thanks! I'm returning to the office tomorrow so I'll check this closer right away then.

cmd: caddy fmt return code is 1 if not formatted (#5297)

• cmd: Fix caddy fmt if input isn't formatted

• Fixes #5294

• return exit 1 with an error message

• cmd: Use formattingDifference for caddy fmt

• #5294

• expose caddyfile.formattingDifference

Right now we have

Usage:
  caddy fmt [flags]

Flags:
      --diff        Print the differences between the input file and the formatted output
  -h, --help        help for fmt
      --overwrite   Overwrite the input file with the results

I originally thought caddy fmt will return 1 if failed like most other linters. It turns out not. It returns formatted content instead. If we don't want to change the behavior of caddy fmt, it would be great to provide another flag to allow returning 1 for failure. Thanks! 😃

• Fixes #5294
• return exit 1 with an error message

@krystian-panek-wttech

@mholt not really but ok ;) your lib is rather providing a facade that does not offer to customize format-specific options.

Sorry, but huh? You linked to the exported struct you can set the options on:

https://github.com/mholt/archiver/blob/f1ef6ebbdf2a7b10af9c44b4540fe9b953b7fba1/zstd.go#L15-L19

That lets you customize Zstd with any of the Zstandard options in the klauspost/compress package. What more do you need?

simply saying I am not sure if this extension checking is needed here: https://github.com/mholt/archiver/blob/v3.5.1/tarzst.go#L31

That's for Archiver v3... FYI I'm no longer maintaining that as v4 is the future.

I am not, go for it! 👍