legoktm
Repos
159
Followers
122
Following
75

🌻 The collaborative editing software that runs Wikipedia. Mirror from https://gerrit.wikimedia.org/g/mediawiki/core. See https://mediawiki.org/wiki/Developer_access for contributing.

3139
1098

GitHub repository for the SecureDrop whistleblower platform. Do not submit tips here!

3326
616

A Python library that interfaces with the MediaWiki API. This is a mirror from gerrit.wikimedia.org. Do not submit any patches here. See https://www.mediawiki.org/wiki/Developer_account for contributing.

509
164

A webapp for accessing information about the FIRST Robotics Competition.

324
149

A Python parser for MediaWiki wikicode

557
65

Kiwix for Windows and GNU/Linux desktops

365
65

Events

Remove sh dependency (issue #6580)

Merge pull request #6580 from lb803/6547-remove-sh-dependency

Remove sh dependency

Created at 10 hours ago
Remove `sh` dependency

This is a good first issue for new contributors to take on, if you have any questions, please ask on the task or in our Gitter room!

Description

sh is a pretty cool project, it allows you to shell out to programs as if they were native Python programs. This comes with a downside, it obfuscates the underlying command being run, and it's an extra dependency.

Here are all the places we use it:

Targets
    Occurrences of 'from sh import' in Project
Found Occurrences in Project  (4 usages found)
    Unclassified  (4 usages found)
        securedrop  (4 usages found)
            securedrop  (1 usage found)
                i18n_tool.py  (1 usage found)
                    19 from sh import git, msgfmt, msgmerge, pybabel, sed, xgettext
            securedrop/tests  (3 usages found)
                test_i18n.py  (1 usage found)
                    34 from sh import pybabel, sed
                test_i18n_tool.py  (1 usage found)
                    13 from sh import git, msginit, pybabel, sed, touch
                test_template_filters.py  (1 usage found)
                    13 from sh import pybabel

Given how minimal its usage is (just i18n_tool.py plus tests), I propose we replace it with explicit subprocess.check_output(...) calls. The biggest advantage of this removal is that pybabel will be the only dependency needed to compile translations at package build time.

Created at 10 hours ago
pull request closed
Remove sh dependency

Status

Ready for review

Description of Changes

Fixes #6547.

I removed all the sh dependencies with the exception of sh.sed, as this has been addressed in another contribution (https://github.com/freedomofpress/securedrop/issues/6547#issuecomment-1256930073).

Testing

Small changes, limited to two files: $ securedrop/bin/dev-shell bin/run-test tests/test_i18n_tool.py $ securedrop/bin/dev-shell bin/run-test tests/test_i18n.py

Deployment

Ready

Checklist

If you made changes to the server application code:

  • [x] Linting (make lint) and tests (make test) pass in the development container
  • [x] These changes do not require documentation
Created at 10 hours ago
Pre-build Python bytecode, don't run alembic as root

@legoktm with this change, can we update the apparmor profile to make the pycache dirs readonly for apache?

Done but untested.

Created at 16 hours ago

Byte-compile Python files at install time

Generate all the Python bytecode at install time so it doesn't need to be lazily created upon first use. This ensures that all the files are consistently owned by root:root and matches what Debian packaging traditionally does[1].

We can now also make the __pycache__ files read-only in AppArmor, though they shouldn't be writable by www-data anyways.

In the future we will investigate shipping the bytecode in the package itself, keeping reproducibility in mind[2].

[1] https://salsa.debian.org/python-team/tools/dh-python/-/blob/d0bd4cf7e04aeba40ed11bbb9e903b971a12f602/autoscripts/postinst-py3compile [2] https://vulns.xyz/2021/08/reproducible-python-bytecode/

Run alembic and other postinst commands as www-data

There's no need for us to run this as root, all the files it needs to touch (primarily the SQLite database) are writable by www-data.

For mod_wsgi-express module-config, it works fine as www-data as the ouput redirection will still be run as root.

Created at 16 hours ago
refactor(scripts): clean manual `sys.path` additions

Code looks good, I'll finish up the testing in a little bit!

Created at 17 hours ago

Add Linux kernel testing steps

Mostly based on what we did in the most recent release.

Refs https://github.com/freedomofpress/securedrop/issues/6514.

Remove kernel mentions from the release process

Kernels are now released independently of SecureDrop server releases.

Refs https://github.com/freedomofpress/securedrop/issues/6328.

Created at 17 hours ago
test_deb_package_contains_expected_conffiles fails on macOS

Description

Unexpected test failure when building debs on mac

Steps to Reproduce

make build-debs

Expected Behavior

All tests pass

Actual Behavior

     test_deb_package_contains_expected_conffiles[docker://xenial-sd-dpkg-verification-/tmp/build/securedrop-app-code_0.12.0~rc1+xenial_amd64.deb]
    [gw2] darwin -- Python 2.7.14 /Users/redshiftzero/.virtualenvs/securedrop/bin/python

    host = <testinfra.host.Host object at 0x110c6ea90>
    deb = '/tmp/build/securedrop-app-code_0.12.0~rc1+xenial_amd64.deb'

        @pytest.mark.parametrize("deb", deb_packages)
        def test_deb_package_contains_expected_conffiles(host, deb):
            """
            Ensures the `securedrop-app-code` package declares only whitelisted
            `conffiles`. Several files in `/etc/` would automatically be marked
            conffiles, which would break unattended updates to critical package
            functionality such as AppArmor profiles. This test validates overrides
            in the build logic to unset those conffiles.
            """
            deb_package = host.file(deb.format(
                securedrop_test_vars.securedrop_version))

            # Only relevant for the securedrop-app-code package:
            if "securedrop-app-code" in deb_package.path:
                tmpdir = tempfile.mkdtemp()
                # The `--raw-extract` flag includes `DEBIAN/` dir with control files
                host.run("dpkg-deb --raw-extract {} {}".format(deb, tmpdir))
                conffiles_path = os.path.join(tmpdir, "DEBIAN", "conffiles")
                f = host.file(conffiles_path)

    >           assert f.is_file
    E           assert False
    E            +  where False = <file /var/folders/gv/2r10zvfj303fyd5_6yn_p0b80000gn/T/tmp2wh6VC/DEBIAN/conffiles>.is_file

Comments

Introduced after merge of #4167, needs investigation

Created at 17 hours ago
test_deb_package_contains_expected_conffiles fails on macOS

The conclusion from today's server dev hangout was that for package building we will document that Debian stable (currently bullseye) should be used (ticket to be filed). We're not opposed to making whatever fails on macOS to be disabled with xfails, but someone else would need to contribute that and keep it up to date.

Created at 17 hours ago
Pre-build Python bytecode, don't run alembic as root

I tested everything except for logging into the JI, because the onion service keeps timing out for me :/

Created at 18 hours ago
Harden Python package download process

pip does not encourage use of or check Python package signatures. That doesn't mean we can't do better than just running pip wheel in the build_securedrop_app_code_deb role. Here are a few things we could do to harden the package download process:

  • [ ] Check signatures on the packages that provide them (last time I checked a while back 4/19 upstream projects signed their packages).
  • [x] Pin SHA256sums of packages. This is something pip supports enforcing via the requirements.txt file.
  • [ ] Pin PyPi's cert. It hasn't changed in years.
  • [ ] Use Tor.
  • [ ] Try to reduce the number of dependencies SD uses. There are numerous packages we use that have barely if at all changed in years, and that we only use tiny snippets from.
  • [ ] File issues with upstream package developers encouraging them to sign their packages.
Created at 18 hours ago
Harden Python package download process

No objections in today's hangout, so closing.

Created at 18 hours ago

Run alembic and other postinst commands as www-data

There's no need for us to run this as root, all the files it needs to touch (primarily the SQLite database) are writable by www-data.

For mod_wsgi-express module-config, it works fine as www-data as the ouput redirection will still be run as root.

Created at 18 hours ago
pull request opened
Pre-build Python bytecode, don't run alembic as root

Status

Ready for review

Description of Changes

  • Generate Python bytecode at install time, ensures they're all owned by root:root and that it doesn't need to be generated on the fly.
  • Run alembic and mod_wsgi-express as www-data rather than root for additional hardening.

Refs https://github.com/freedomofpress/securedrop-security/issues/84.

Testing

  • [ ] Build new packages, try installing them, make sure nothing breaks. Then try a submission and log into the JI (should exercise enough code paths that the pyc isn't obviously broken)

Deployment

Any special considerations for deployment? Not particularly.

Checklist

  • [ ] Linting (make lint) and tests (make test) pass in the development container
  • [ ] Configuration tests pass
  • [ ] I have written a test plan and validated it for this PR
  • [ ] These changes do not require documentation
Created at 18 hours ago
legoktm create branch pyc-permissions
Created at 18 hours ago
Upgrade to flask 2.0.3

Unfortunately we don't have a lot of guidance for using Windows, do you have the Windows Subsystem for Linux (WSL) set up? You'll also need to install Docker.

Created at 19 hours ago
legoktm delete branch pylint-scripts
Created at 19 hours ago