This is basically ready for review, but lets land https://github.com/freedomofpress/securedrop-builder/pull/427 first and then I'll rebase this.

Add SecureDrop-specific metadata to buildinfo files

dpkg will only export specific known environment variables into the buildinfo file[1], but we want to track at least two more things:

1. the Git commit of the package being built
2. the Git commit of the securedrop-builder repository being used, as the bootstrap and wheels used affect the package output.

We can track those by capturing the values before the build process and then manually adding them to the end of the buildinfo file. This also means that builds must be done from a Git checkout, and cannot be from a tarball; so that path now errors out.

It would be too complicated to have this script respect the new $SD_BUILDER_GITREF by checking out a different version of the repository, re-initializing the bootstrap, etc., so we leave that as a responsiblity of the caller. But as a failsafe we still double-check that the desired version of the builder repository is being used.

While we're at it, look up the package filename by "parsing" debian/files instead of trying to find it via find.

[1] https://git.dpkg.org/cgit/dpkg/dpkg.git/tree/scripts/Dpkg/BuildInfo.pm

Update documentation for build-debianpackage

The kernel version parsing code is https://github.com/canonical/ubuntu-pro-client/blob/main/uaclient/system.py#L27 - this is basically the same problem as https://github.com/freedomofpress/securedrop/issues/6762 and will eventually be fixed by https://github.com/freedomofpress/kernel-builder/pull/33

in which case we should consider removing ubuntu-advantage-tools during installation.

this seems sensible regardless.

Status

Ready for review

Description of Changes

Updates mod_wsgi from 4.6.7 to 4.9.4, addressing the X-Client-IP issue described in https://modwsgi.readthedocs.io/en/latest/release-notes/version-4.9.3.html.

Testing

• [ ] CI is passing
• [ ] make safety is passing locally.

Deployment

n/a

Checklist

If you made changes to the server application code:

• [ ] Linting (make lint) and tests (make test) pass in the development container

If you made non-trivial code changes:

• [x] I have written a test plan and validated it for this PR

Choose one of the following:

• [ ] I have opened a PR in the docs repo for these changes, or will do so later
• [ ] I would appreciate help with the documentation
• [x] These changes do not require documentation

If you added or updated a production code dependency:

Production code dependencies are defined in:

• admin/requirements.in
• admin/requirements-ansible.in
• securedrop/requirements/python3/requirements.in

If you changed another requirements.in file that applies only to development or testing environments, then no diff review is required, and you can skip (remove) this section.

Choose one of the following:

• [x] I have performed a diff review and pasted the contents to the packaging wiki: https://github.com/freedomofpress/securedrop-builder/wiki/mod_wsgi-4.6.7-to-4.9.4
• [ ] I would like someone else to do the diff review

Updated mod-wsgi from 4.6.7 to 4.9.4

Merge pull request #6775 from freedomofpress/update-mod_wsgi

Updated mod-wsgi from 4.6.7 to 4.9.4

To close the loop for those following along here and not IRC, Both Lucas and @bd808 have access to this GH repo and PyPI - leaving it in their good hands now :-)

This would need to be tested on a staging or prod SD right? Since mod_wsgi/apache isn't used in dev?

Thank you, but I would like to stop maintaining this repo, I don't really use it anymore. I could transfer it to you (+pypi access) if you're interested, but also maybe we should make it collaboratively maintained on Wikimedia GitLab or something.

@Pathoschild the event page and rules now exist :-)

One open question that I kind of discussed in https://github.com/freedomofpress/securedrop-builder/pull/433#issuecomment-1483382710 is at what layer should SD_BUILDER_GITREF be handled. It seems absurdly complex to try and have the build-debianpackage script re-clone the builder repo if it's the wrong version, my current thinking is we do it one layer higher and have the rebuild script take care of it.

But then should the build-debianpackage script error out if SD_BUILDER_GITREF is set and we're using the wrong builder? I think that would be a reasonable fail-safe.

Add SecureDrop-specific metadata to buildinfo files

dpkg will only export specific known environment variables into the buildinfo file[1], but we want to track at least two more things:

1. the Git commit of the package being built
2. the Git commit of the securedrop-builder repository being used, as the bootstrap and wheels used affect the package output.

We can track those by capturing the values before the build process and then manually adding them to the end of the buildinfo file. This also means that builds must be done from a Git checkout, and cannot be from a tarball, so that path now errors out.

While we're at it, look up the package filename by "parsing" debian/files instead of trying to find it via find.

[1] https://git.dpkg.org/cgit/dpkg/dpkg.git/tree/scripts/Dpkg/BuildInfo.pm

dpkg will only export specific known environment variables into the buildinfo file[1], but we want to track at least two more things:

1. the Git commit of the package being built
2. the Git commit of the securedrop-builder repository being used, as the bootstrap and wheels used affect the package output.

We can track those by capturing the values before the build process and then manually adding them to the end of the buildinfo file. This also means that builds must be done from a Git checkout, and cannot be from a tarball, so that path now errors out.

While we're at it, look up the package filename by "parsing" debian/files instead of trying to find it via find.

[1] https://git.dpkg.org/cgit/dpkg/dpkg.git/tree/scripts/Dpkg/BuildInfo.pm

I am also renaming PKG_GITREF to SD_PKG_GITREF, as this will be exported in the .buildinfo file so let's prefix it with "SD" to make it abundantly clear this is our environment variable and reduce the risk of collision.

There don't appear to be any other users of this besides humans, so I have not added in any backwards-compatibility support for the old name.

Test plan

• [ ] Check out this PR. Run SD_PKG_GITREF=main make securedrop-proxy, verify the buildinfo file contains SD_BUILDER_GITREF and SD_PKG_GITREF with commits that correspond to this builder PR you just checked out and the current main branch of sd-proxy.
• [ ] Add a dummy commit to your sd-builder checkout. Re-run SD_PKG_GITREF=main make securedrop-proxy, verify SD_BUILDER_GITREF changed to correspond to the new commit you just added.
• [ ] Run git -C /tmp/securedrop-proxy checkout HEAD~1, note which commit you're now on. Run PKG_PATH=/tmp/securedrop-proxy make securedrop-proxy, verify SD_PKG_GITREF changed to the new commit you switched to.

Error out if multiple checksums are found for a package

Co-authored-by: Cory Francis Myers cory@freedom.press

OK, we're all set on the infra side so @cfm this is ready for your review :)

We also need to land https://github.com/freedomofpress/securedrop-yum-test/pull/44 - I think that will require admin access (@eloquence?) to remove the project from CircleCI first.

Refactor clean-old-nightlies so it works for all package folders

The cleanup script can now be used for the non-nightly directories too. Instead of trying to parse a timestamp out of the filename, it sorts the packages by version and only keeps the specified number.

It uses the proper dpkg version comparing function, and properly supports the Linux packages that have a different naming scheme.

Automatically clean up non-nightly packages

Instead of relying on humans to delete the older packages when they add new ones, let's automate it.

CI will now keep the 4 latest versions of a package, deleting the rest as part of the nightly run.

While we're at it, lower the number of nightlies to be kept from 14 days to 7 days. The 14 days amount was rather conservative when we first implemented automation, we can probably be slightly more aggressive now.

Add script to simplify promoting packages from dev to prod

The new ./scripts/promote-suite simplifies the process of promoting packages from the dev repository to the prod one.

The primary use case currently is for promoting Tor and Linux packages.

For each package in the dev repo, it compares the highest version to the highest version in the prod repo, copying over the dev package if they don't match. There are some edge cases where this doesn't work (which is annotated with a code comment), but it should get the majority correct.

As a side-effect, it could reveal when a package in dev was forgotten for promotion to prod.

In the future this script could be fancier by showing debdiffs and diffoscopes of the newly promoted packages, or a dry-run mode that shows which packages are pending promotion.

[update-safety-alerts] silence 2 safety alerts

Merge pull request #405 from freedomofpress/clean-old-packages

Automatically clean up non-nightly packages

Merge pull request #410 from freedomofpress/update-safety-alerts

Silence 2 safety alerts

Create issues when a new Tor release is fetched

CI will now create a new issue (or update an existing one) whenever it fetches new Tor packages. This is something that I have been doing manually for a while now whenever I see a new Tor release announcement.

The generated issue contains the checklist as well as the diff of the new debs so you can see the version and checksums of the packages. An example issue (with the wrong patch) is https://github.com/freedomofpress/securedrop/issues/6723.

Some potential future improvements that I deferred for now is to include the version number in the issue title, and the correct Tor Project forum link.

The script wraps around the official gh CLI tool, it needs a GITHUB_TOKEN to be set in the environment to work properly. gh is only available in bullseye-backports, so I had to adjust the image so it would be installable.

Of course, if this works out well, I'd like to expand this to other things like kernel and dependency updates.

Merge pull request #408 from freedomofpress/auto-issue

Create issues when a new Tor release is fetched

Only run new-tor-issue when we have a new package (for real)

Follow-up to 1862e178a1abb08.

I wrongly understood how bash grouped || and &&. We wanted diff-index || (commit && push && new-tor-issue), but instead bash gave us (diff-index || commit) && push && new-tor-issue, which meant that regardless of whether there is a new package, we were pushing (harmless) and running the new-tor-issue script (disruptive!).

By adding explicit parenthesis it should do what we actually want.

Merge pull request #411 from freedomofpress/fix-auto-issue

Only run new-tor-issue when we have a new package (for real)

Merge pull request #406 from freedomofpress/promote-suite

Add script to simplify promoting packages from dev to prod

Remove buster-only (3.7) wheels

Upgrade Cython in workstation-bootstrap to 0.29.33 and rebuild

Bookworm has switched to Python 3.11, so we need to rebuild the wheel anyways. But our pinned verison of Cython is incompatible, so upgrade it to 0.29.33 and rebuild it on both Bullseye (3.9) and Bookworm (3.11).

Rebuild securedrop-client wheels for Bookworm / Python 3.11

Rebuild securedrop-proxy wheels for Bookworm / Python 3.11

Rebuild PyYAML wheel after Cython upgrade

The Cython upgrade changed some of the generated C code, so the PyYAML wheel output is different and needs to be rebuilt so it matches a fresh build. diffoscope of the two wheels showed nothing useful/interesting.

Patch dh_virtualenv for Python 3.11 support during install-deps

Merge pull request #412 from freedomofpress/cython-bump

Rebuild Bookworm wheels for Python 3.11

Update apt LFS repo names

Update README.md

Have CI build nightlies for securedrop-updater

The job increments the version number in the specfile, runs make rpm-build and commits the result to the securedrop-workstation-dev-rpm-packages-lfs repository.

For now this is specific to securedrop-updater, in the future we expect to use the same build commands for securedrop-workstation and will set up nightlies for that too.

Fixes https://github.com/freedomofpress/securedrop-updater/issues/10.

Also, would I need to add something in debian/rules (or somewhere else) to ensure the postinst is ran?

Nope, as long as it's called postinst in the debian/ folder and executable it should automatically get picked up. See https://manpages.debian.org/bullseye/debhelper/dh_installdeb.1.en.html