The PHP Unit Testing framework.
BSD 3-Clause "New" or "Revised" License
18667
334
2111

When the tested code thrown an object that can do something more (like contain some additional data), there is no possibility to test it. Unfortunately the \PHPUnit\Framework\TestCase::checkExceptionExpectations is marked as private so there is no possibility to extend the TestCase functionalities.

It will be nice, that the Exception that was caught in runTest will be available somewhere for postCondition methods.

Q A
PHPUnit version 9.5.25
PHP version 8.1.10
Installation Method Composer

Summary

If you enable beStrictAboutResourceUsageDuringSmallTests=true, PHPUnit should at least warn about xdebug being unavailable and the flag ignored.

Current behavior

Currently it just does nothing, the flag is behaving as if ignored.

How to reproduce

Enable the flag and run tests without xdebug being available.

Expected behavior

It would warn the user about xdebug (and the function exec tracking feature) being unavailable.

Sometimes we need to change the behavior of a method, that was already defined earlier in the code, without recreating the full mocked object from scratch, something like this:

  public function myTest() {
    $myStub = $this->createStub(SomeClass::class);
    /* A lot of other mocks and method matches here */

    // Testing behavior with access allowed.
    $myStub->method('checkAccess')
      ->willReturn(true);
    $testingClass->testingFunction($myStub);
    // Assertions for access allowed.

    // Testing behavior with access denied.
    $myStub->method('checkAccess')
      ->willReturn(false);
    $testingClass->testingFunction($myStub);
    // Assertions for access denied.
  }

But this will not work as expected because the second method will not overwrite the previous one.

So will be good to have some ways to get the current matcher for a method by method name, and change its return value.

If it already possible now with the current API? If yes, could you please show me an example?

Q A
PHPUnit version 9.5.10
PHP version 7.4
Installation Method Composer

Summary

Currently test case like:

 $this->assertSame(8.20234376757473, 8.20234376757474);

pass even if the numbers are different. They are really stored differently, test case:

$this->assertSame(bin2hex(pack('e', 8.20234376757473)), bin2hex(pack('e', 8.20234376757474)));

fails correctly.

It seems phpunit converts float numbers using default php float to string conversion which depends on php.ini precision configuration, test case like:

ini_set('precision', '20');
$this->assertSame(8.20234376757473, 8.20234376757474);

fails as expected.

Also, the float diff output has currently limited precision.

To fix exact comparisons & diff output, I propose to convert float numbers to string using function like:

/**
 * Cast float to string with lossless precision.
 */
final public static function castFloatToString(float $value): string
{
    $precisionBackup = ini_get('precision');
    ini_set('precision', '-1');
    try {
        return (string) $value;
    } finally {
        ini_set('precision', $precisionBackup);
    }
}
Q A
PHPUnit version 9.5.20
PHP version 7.4.24
Installation Method Composer

Summary

when running phpunit without defining a source directory at the end of your command it does not display phpunit's (deprecation) warnings

Current behavior

this does not display phpunit's (deprecation) warnings

vendor/bin/phpunit --testsuite=Foo

while this does (e.g. Warning: Test case class not matching filename is deprecated)

vendor/bin/phpunit --testsuite=Foo tests/

How to reproduce

rename one of your class names of an existing test without changing the filename itself

Expected behavior

i'm not quite sure what is to be expected (i.e. is this expected behaviour or a bug). personally i'd expect the (deprecation) warnings to be shown regardless or perhaps have a flag to have them displayed

composer info | sort output

out.txt

Q A
PHPUnit version 9.5.24
PHP version n/a
Installation Method Composer

Summary

Identified in #4983.

When a AssertionFailedError is thrown as a re-throw of a previous exception (for instance as part of a onNotSuccessfulTest() callback), and the previous exception is chained in, the failure reported by PHPUnit traces to the part where AssertionFailedError was thrown, and the original trace is lost. So there is no information about where the test failed.

Current behavior

PHPUnit 9.5.24 #StandWithUkraine

Runtime:       PHP 8.1.10
Configuration: /home/runner/work/d8-unit/d8-unit/core/phpunit.xml.dist

Testing Drupal\Tests\node\Functional\NodeTitleTest
F                                                                   1 / 1 (100%)

Time: 00:06.500, Memory: 10.00 MB

There was 1 failure:

1) Drupal\Tests\node\Functional\NodeTitleTest::testNodeTitle
Title found

/home/runner/work/d8-unit/d8-unit/core/tests/Drupal/Tests/BrowserTestBase.php:441
/home/runner/work/d8-unit/d8-unit/core/tests/Drupal/Tests/Traits/UnsuccessfulTestHandlerTrait.php:16
/home/runner/work/d8-unit/d8-unit/vendor/phpunit/phpunit/src/Framework/TestResult.php:726

FAILURES!
Tests: 1, Assertions: 17, Failures: 1

How to reproduce

Implement a onNotSuccessfulTest() callback in a test, and add logic so that a Throwable is re-thrown as a AssertionFailedError with the Throwable chained in. For example,

    if ($t instanceof MinkException) {
      throw new AssertionFailedError($t->getMessage(), $t->getCode(), $t);
    }

Expected behavior

PHPUnit 9.5.24 #StandWithUkraine

Runtime:       PHP 8.1.10
Configuration: /home/runner/work/d8-unit/d8-unit/core/phpunit.xml.dist

Testing Drupal\Tests\node\Functional\NodeTitleTest
F                                                                   1 / 1 (100%)

Time: 00:06.500, Memory: 10.00 MB

There was 1 failure:

1) Drupal\Tests\node\Functional\NodeTitleTest::testNodeTitle
Title found

/home/runner/work/d8-unit/d8-unit/core/tests/Drupal/Tests/BrowserTestBase.php:441
/home/runner/work/d8-unit/d8-unit/core/tests/Drupal/Tests/Traits/UnsuccessfulTestHandlerTrait.php:16
/home/runner/work/d8-unit/d8-unit/vendor/phpunit/phpunit/src/Framework/TestResult.php:726

Caused by Behat\Mink\Exception\ExpectationException: Title found
/home/runner/work/d8-unit/d8-unit/core/tests/Drupal/Tests/WebAssert.php:538
/home/runner/work/d8-unit/d8-unit/core/tests/Drupal/Tests/WebAssert.php:281
/home/runner/work/d8-unit/d8-unit/core/modules/node/tests/src/Functional/NodeTitleTest.php:94
/home/runner/work/d8-unit/d8-unit/vendor/phpunit/phpunit/src/Framework/TestResult.php:726

FAILURES!
Tests: 1, Assertions: 17, Failures: 1.

In our project's testRoutes_shouldDenyCustomersAccessToAdminPages() test, we have a for-loop that iterates each and every "admin" prefixed route of our PHP App.

Now sometimes an error is thrown, or the assertion fails, and we would like to know for which route exactly, for that, said for-loop stores the context in lastRoute instance-variable of test-case, and TestListener prefixes the logged message with said context, like:

public function addError(Test $test, \Throwable $t, float $time): void
{
    $this->onThrow($test, $t);
}
public function addFailure(Test $test, AssertionFailedError $error, float $time): void
{
    $this->onThrow($test, $error);
}

private function onThrow(Test $test, \Throwable $error)
{
    if ($test instanceof MyTest) {
        if ($test->lastRoute !== null) {
            self::prefixMessage($error, 'While testing route = ' . $test->lastRoute . ":\n");
        }
    }
}

private static function prefixMessage($obj, $prefix)
{
    try {
        $reflection = new \ReflectionClass($obj);
        $property = $reflection->getProperty('message');
        $property->setAccessible(true);
        $property->setValue($obj, $prefix . $property->getValue($obj));
    } catch (\ReflectionException $e) {
    }
}

@sebastianbergmann How can we achive above with the new TestHook API?
Which looks like:

interface AfterTestErrorHook extends TestHook
{
    public function executeAfterTestError(string $test, string $message, float $time): void;
}

interface AfterTestFailureHook extends TestHook
{
    public function executeAfterTestFailure(string $test, string $message, float $time): void;
}

I mean, we could Make the lastRoute a static-variable (instead of instance-variable).
But how can we alter the $message?

Solutions

As you deprecated our side of handling this,
please consider implementing one of below solutions into PHPUnit API.

# 1 You could change your existing API's $message to &$message, I mean, to a reference, then use reflection like above (to update message).

# 2 Or, like some other frameworks, provide withContext(...) method, and the next failure (no matter if error or assert) should log the context before actual message, then clear context.

Also, test-runner should clear context after each test-method (of course).

Right now, test methods are identified by the prefix test, but data providers are specified by the annotation dataProvider. It would save a lot of broiler plate code if we could specify data providers by a provider prefix too. eg)

public function testAdd(int $a, int $b, int $expected): void
{
    $this->assertSame($expected, $a + $b);
}
public function providerAdd(): array
{
    return [
        [0, 0, 0]
    ];
}

For backwards compatibility, we could keep the dataProvider annotation functionality as-is, allowing for an arbitrary name for the data provider. But if that annotation does not exist as in the example above, then assume the name of the data provider is providerMethod because the test name is testMethod. If such a method does not exist and the test method has parameters, then emit an error as is currently done.

Q A
PHPUnit version 8.5.8
PHP version 7.4.13
Installation Method Composer

composer info | sort output:

[dev@ip-189-31-5-156  project]$ composer info | sort
appstract/laravel-opcache             4.0.1     PHP OPcache Artisan commands for Laravel.
asm89/stack-cors                      v2.0.1    Cross-origin resource sharing library and stack middleware
aws/aws-sdk-php                       3.166.1   AWS SDK for PHP - Use Amazon Web Services in your PHP project
aws/aws-sdk-php-laravel               3.6.0     A simple Laravel 5/6/7/8 service provider for including the AW...
brick/math                            0.9.1     Arbitrary-precision arithmetic library
composer/ca-bundle                    1.2.10    Lets you find a path to the system CA bundle, and includes a f...
composer/composer                     2.1.5     Composer helps you declare, manage and install dependencies of...
composer/metadata-minifier            1.0.0     Small utility library that handles metadata minification and e...
composer/semver                       3.2.5     Semver library that offers utilities, version constraint parsi...
composer/spdx-licenses                1.5.5     SPDX licenses list and validation library.
composer/xdebug-handler               2.0.2     Restarts a process without Xdebug.
dnoegel/php-xdg-base-dir              v0.1.1    implementation of xdg base directory specification for php
doctrine/cache                        1.10.2    PHP Doctrine Cache library is a popular cache implementation t...
doctrine/dbal                         2.10.3    Powerful PHP database abstraction layer (DBAL) with many featu...
doctrine/event-manager                1.1.1     The Doctrine Event Manager is a simple PHP event system that w...
doctrine/inflector                    2.0.3     PHP Doctrine Inflector is a small library that can perform str...
doctrine/instantiator                 1.3.1     A small, lightweight utility to instantiate objects in PHP wit...
doctrine/lexer                        1.2.1     PHP Doctrine Lexer parser library that can be used in Top-Down...
dragonmantank/cron-expression         v2.3.0    CRON for PHP: Calculate the next or previous run date and dete...
egulias/email-validator               2.1.19    A library for validating emails against several RFCs
facade/flare-client-php               1.3.5     Send PHP errors to Flare
facade/ignition                       2.3.6     A beautiful error page for Laravel applications.
facade/ignition-contracts             1.0.1     Solution contracts for Ignition
fideloper/proxy                       4.4.0     Set trusted proxies for Laravel
filp/whoops                           2.7.3     php error handling for cool kids
fruitcake/laravel-cors                v2.0.1    Adds CORS (Cross-Origin Resource Sharing) headers support in y...
fzaninotto/faker                      v1.9.1    Faker is a PHP library that generates fake data for you.
guzzlehttp/guzzle                     6.5.5     Guzzle is a PHP HTTP client library
guzzlehttp/promises                   1.4.0     Guzzle promises library
guzzlehttp/psr7                       1.7.0     PSR-7 message implementation that also provides common utility...
hamcrest/hamcrest-php                 v2.0.1    This is the PHP port of Hamcrest Matchers
intervention/image                    2.5.1     Image handling and manipulation library with support for Larav...
justinrainbow/json-schema             5.2.11    A library to validate a json schema.
kitloong/laravel-migrations-generator 4.4.0     Generates Laravel Migrations from an existing database
krlove/code-generator                 1.0.0     Code Generator
krlove/eloquent-model-generator       1.3.5     Eloquent Model Generator
laravel/framework                     v7.27.0   The Laravel Framework.
laravel/sanctum                       v2.6.0    Laravel Sanctum provides a featherweight authentication system...
laravel/socialite                     v4.4.1    Laravel wrapper around OAuth 1 & OAuth 2 libraries.
laravel/tinker                        v2.4.2    Powerful REPL for the Laravel framework.
league/commonmark                     1.5.4     Highly-extensible PHP Markdown parser which fully supports the...
league/flysystem                      1.1.3     Filesystem abstraction: Many filesystems, one API.
league/mime-type-detection            1.4.0     Mime-type detection for Flysystem
league/oauth1-client                  1.8.0     OAuth 1.0 Client Library
mockery/mockery                       1.4.2     Mockery is a simple yet flexible PHP mock object framework
monolog/monolog                       2.1.1     Sends your logs to files, sockets, inboxes, databases and vari...
mpociot/laravel-test-factory-helper   v2.1.0    Generate Laravel test factories from your existing models
mtdowling/jmespath.php                2.6.0     Declaratively specify how to extract elements from a JSON docu...
myclabs/deep-copy                     1.10.1    Create deep copies (clones) of your objects
nesbot/carbon                         2.39.0    An API extension for DateTime that supports 281 different lang...
nikic/php-parser                      v4.9.1    A PHP parser written in PHP
nunomaduro/collision                  v4.2.0    Cli error handling for console/command-line PHP applications.
nunomaduro/larastan                   v0.7.12   Larastan - Discover bugs in your code without running it. A ph...
opis/closure                          3.5.6     A library that can be used to serialize closures (anonymous fu...
paragonie/random_compat               v9.99.99  PHP 5.x polyfill for random_bytes() and random_int() from PHP 7
phar-io/manifest                      1.0.3     Component for reading phar.io manifest information from a PHP ...
phar-io/version                       2.0.1     Library for handling version information and constraints
phpdocumentor/reflection-common       2.2.0     Common reflection classes used by phpdocumentor to reflect the...
phpdocumentor/reflection-docblock     5.2.1     With this component, a library can provide support for annotat...
phpdocumentor/type-resolver           1.3.0     A PSR-5 based resolver of Class names, Types and Structural El...
phpoption/phpoption                   1.7.5     Option Type for PHP
phpspec/prophecy                      1.11.1    Highly opinionated mocking framework for PHP 5.3+
phpstan/phpstan                       0.12.94   PHPStan - PHP Static Analysis Tool
phpunit/php-code-coverage             7.0.10    Library that provides collection, processing, and rendering fu...
phpunit/php-file-iterator             2.0.2     FilterIterator implementation that filters files based on a li...
phpunit/php-text-template             1.2.1     Simple template engine.
phpunit/php-timer                     2.1.2     Utility class for timing
phpunit/php-token-stream              3.1.1     Wrapper around PHP's tokenizer extension.
phpunit/phpunit                       8.5.8     The PHP Unit Testing framework.
psr/container                         1.0.0     Common Container Interface (PHP FIG PSR-11)
psr/event-dispatcher                  1.0.0     Standard interfaces for event handling.
psr/http-message                      1.0.1     Common interface for HTTP messages
psr/log                               1.1.3     Common interface for logging libraries
psr/simple-cache                      1.0.1     Common interfaces for simple caching
psy/psysh                             v0.10.4   An interactive shell for modern PHP.
ralouphie/getallheaders               3.0.3     A polyfill for getallheaders.
ramsey/collection                     1.1.0     A PHP 7.2+ library for representing and manipulating collections.
ramsey/uuid                           4.1.1     A PHP library for generating and working with universally uniq...
react/promise                         v2.8.0    A lightweight implementation of CommonJS Promises/A for PHP
scrivo/highlight.php                  v9.18.1.2 Server side syntax highlighter that supports 185 languages. It...
sebastian/code-unit-reverse-lookup    1.0.1     Looks up which function or method a line of code belongs to
sebastian/comparator                  3.0.2     Provides the functionality to compare PHP values for equality
sebastian/diff                        3.0.2     Diff implementation
sebastian/environment                 4.2.3     Provides functionality to handle HHVM/PHP environments
sebastian/exporter                    3.1.2     Provides the functionality to export PHP variables for visuali...
sebastian/global-state                3.0.0     Snapshotting of global state
sebastian/object-enumerator           3.0.3     Traverses array structures and object graphs to enumerate all ...
sebastian/object-reflector            1.1.1     Allows reflection of object attributes, including inherited an...
sebastian/recursion-context           3.0.0     Provides functionality to recursively process PHP variables
sebastian/resource-operations         2.0.1     Provides a list of PHP built-in functions that operate on reso...
sebastian/type                        1.1.3     Collection of value objects that represent the types of the PH...
sebastian/version                     2.0.1     Library that helps with managing the version number of Git-hos...
seld/jsonlint                         1.8.3     JSON Linter
seld/phar-utils                       1.1.1     PHAR file format utilities, for when PHP phars you up
socialiteproviders/manager            v3.5      Easily add new or override built-in providers in Laravel Socia...
socialiteproviders/microsoft          v3.0.0    Microsoft OAuth2 Provider for Laravel Socialite
socialiteproviders/yahoo              v3.0.2    Yahoo OAuth2 Provider for Laravel Socialite
swiftmailer/swiftmailer               v6.2.3    Swiftmailer, free feature-rich PHP mailer
symfony/console                       v5.1.5    Symfony Console Component
symfony/css-selector                  v5.1.5    Symfony CssSelector Component
symfony/deprecation-contracts         v2.1.3    A generic function and convention to trigger deprecation notices
symfony/error-handler                 v5.1.5    Symfony ErrorHandler Component
symfony/event-dispatcher-contracts    v2.1.3    Generic abstractions related to dispatching event
symfony/event-dispatcher              v5.1.5    Symfony EventDispatcher Component
symfony/filesystem                    v5.3.4    Provides basic utilities for the filesystem
symfony/finder                        v5.1.5    Symfony Finder Component
symfony/http-foundation               v5.1.5    Symfony HttpFoundation Component
symfony/http-kernel                   v5.1.5    Symfony HttpKernel Component
symfony/mime                          v5.1.5    A library to manipulate MIME messages
symfony/polyfill-ctype                v1.18.1   Symfony polyfill for ctype functions
symfony/polyfill-iconv                v1.18.1   Symfony polyfill for the Iconv extension
symfony/polyfill-intl-grapheme        v1.18.1   Symfony polyfill for intl's grapheme_* functions
symfony/polyfill-intl-idn             v1.18.1   Symfony polyfill for intl's idn_to_ascii and idn_to_utf8 funct...
symfony/polyfill-intl-normalizer      v1.18.1   Symfony polyfill for intl's Normalizer class and related funct...
symfony/polyfill-mbstring             v1.20.0   Symfony polyfill for the Mbstring extension
symfony/polyfill-php70                v1.18.1   Symfony polyfill backporting some PHP 7.0+ features to lower P...
symfony/polyfill-php72                v1.18.1   Symfony polyfill backporting some PHP 7.2+ features to lower P...
symfony/polyfill-php73                v1.18.1   Symfony polyfill backporting some PHP 7.3+ features to lower P...
symfony/polyfill-php80                v1.18.1   Symfony polyfill backporting some PHP 8.0+ features to lower P...
symfony/process                       v5.1.5    Symfony Process Component
symfony/routing                       v5.1.5    Symfony Routing Component
symfony/service-contracts             v2.1.3    Generic abstractions related to writing services
symfony/string                        v5.1.5    Symfony String component
symfony/translation-contracts         v2.1.3    Generic abstractions related to translation
symfony/translation                   v5.1.5    Symfony Translation Component
symfony/var-dumper                    v5.1.5    Symfony mechanism for exploring and dumping PHP variables
theseer/tokenizer                     1.2.0     A small library for converting tokenized PHP source code into ...
tijsverkoyen/css-to-inline-styles     2.2.3     CssToInlineStyles is a class that enables you to convert HTML-...
twilio/sdk                            6.22.0    A PHP wrapper for Twilio's API
vlucas/phpdotenv                      v4.1.8    Loads environment variables from `.env` to `getenv()`, `$_ENV`...
voku/portable-ascii                   1.5.3     Portable ASCII library - performance optimized (ascii) string ...
webmozart/assert                      1.9.1     Assertions to validate method input/output with nice error mes...

Summary

Groups not printed when running the command vendor/bin/phpunit --list-group. Not recognized during test runs as well.

Current behavior

Added test groups for more granular control over test runs. The new test groups added are listed when running the command vendor/bin/phpunit --list-group on my local setup. However, when running this on the server (CentOS Linux 7 (Core)), the added group is not printed.

Output on server:

[dev@ip-189-31-5-156  project]$ vendor/bin/phpunit --list-group
PHPUnit 8.5.8 by Sebastian Bergmann and contributors.

Available test group(s):
 - default

Output on local:

PS D:\xampp\htdocs\project> .\vendor\bin\phpunit --list-groups                              
PHPUnit 8.5.8 by Sebastian Bergmann and contributors.

Available test group(s):
 - default
 - usesQueue

Expected behavior

The group usesQueue should be printed. However, it isn't.

Note
The following has been already verified:

  1. The code is deployed and group name exists for the relevant tests.
  2. All code caches have been flushed (opcache cleared). Even the server has been restarted!
  3. Have used groups successfully in the past.

Thanks!

Q A
PHPUnit version 7.5.2
PHP version 7.2+
Installation Method Composer & PHAR

Given the following test:

DemoTest.php
<?php

declare(strict_types=1);

namespace Acme;

use function file_put_contents;
use Generator;
use const PHP_EOL;
use PHPUnit\Framework\TestCase;
use function spl_object_id;

class DemoTest extends TestCase
{

    /**
     * {@inheritdoc}
     */
    protected function setUp(): void
    {
        file_put_contents('php://stderr', PHP_EOL.'SETUP'.PHP_EOL);
        file_put_contents('php://stderr', spl_object_id($this).PHP_EOL);
    }

    /**
     * @dataProvider provideCallable
     */
    public function test_something(callable $callable): void
    {
        file_put_contents('php://stderr', PHP_EOL.'TEST'.PHP_EOL);
        file_put_contents('php://stderr', spl_object_id($this).PHP_EOL);

        $callable();

        $this->assertTrue(true);
    }

    public function provideCallable(): Generator
    {
        file_put_contents('php://stderr', PHP_EOL.'PROVIDER'.PHP_EOL);
        file_put_contents('php://stderr', spl_object_id($this).PHP_EOL);

        yield [
            function () {
                file_put_contents('php://stderr', PHP_EOL.'YIELDED CALLABLE'.PHP_EOL);
                file_put_contents('php://stderr', spl_object_id($this).PHP_EOL);
            }
        ];
    }
}

Will yield:

PROVIDER
336
SETUP
337
TEST
337
YIELDED CALLABLE
336

So you can see that the $this value reference changes between when the provider is called (hence when the $this is used in a callable as well) and when the test is set-up & executed.

It is worth mentioning that if providers are loaded lazily (not sure if it's gonna be done?) then the bug would be solved by itself. Otherwise maybe it could be worth recommending to write providers as static functions to avoid to reference$this.

As a developer of slower integration tests
I want to see prettified tests results as soon as tests are completed
so that my workflow is both agile and æsthetic

Background

The buffering testdox printer and its friend the spinner are breaking some workflows that people used to good effect.

The example from #3560 is using Testdox to get readable results while running. The actual test order is not that important, set by configuration or implied by test collection structure. My implementation of the buffering printer breaks this specific use case.

Solution

Implement explicit "do not buffer result ouput" option, along the lines of #3513:

  • extend buffering mechanism to explicitly support always-flush
  • add CLI flag
  • add end-to-end tests voor --enable-buffer and --disable-buffer flag
  • add XML-configuration option
  • add unit test coverage for unbuffered Testdox

If a data provider method uses complex logic and loops to build up its array of datasets, then it can be easy to accidentally create duplicate sets.

The presence of duplicate sets wastes developer time when running tests, CI time, and energy.

It would be nice if PHPUnit checked for duplicate sets and produced a warning.

Q A
PHPUnit version 7.0.0 (version designated to restrict Listeners)
PHP version 7.x.x
Installation Method Composer / PHAR

In light of #2477 which would remove the current ability to (sneakily, I know!) manipulate test suites within Listeners, this is an issue to request some alternative means of injecting a class with write access. Currently this can be provided within test classes, however there may be some advantage to doing this indirectly as Listeners currently allow.

To explain my own specific needs, a few of us are bringing Humbug back to life, as a mutation testing framework. This involves generating small file changes, and running PHPUnit. We run phpunit for every single change...potentially hundreds of times. To improve performance, we take a number of basic steps described here: https://github.com/humbug/humbug#performance which can be summarised as:

  1. Only run tests on covered lines.
  2. Eliminate tests not applicable to a line.
  3. Order tests fastest-first to hopefully reach a failure quickly.
  4. Stop on first failure/error/exception (serves no purpose to continue after one).

Technically, we only order (or filter) test suites - I'll need to double check if that was a limitation of using Listeners, or of PHPUnit in general, or a limitation in our analysis of code coverage data. We do not order individual tests within a parent suite (unless the children are themselves suites) which would actually be the preferred approach.

The performance benefits are central to the framework being useful, as we are not running all tests all the time hundreds of times.

TL;DR - One way of summarising this is getting an array of tests (which would be multidimensional, grouping tests by suite), manipulating that array, injecting a revised array, all while assuming execution order is simply determined by array index. It could perhaps be done by exporting/importing a name map where avoiding any deeper write level interference with actual objects is preferable.

Q A
PHPUnit version 9.5
PHP version n/a
Installation Method n/a

Summary

Working on adjusting Drupal for PHPUnit 10, I realized MockBuilder::setMethods is removed from PHPUnit 10. Unlike other methods deprecated in PHPUnit 8 and 9, there's no warning thrown about the deprecation.

Since in Drupal we rely on deprecation errors being triggered, so far this deprecation was missed from any conversion.

Current behavior

n/a

How to reproduce

n/a

Expected behavior

I suppose a Warning should be thrown in PHPUnit 9, on top of the annotated deprecation.

getMockClass() has been removed in PHPUnit 10.

  • expectDeprecation(), expectDeprecationMessage(), and expectDeprecationMessageMatches()
  • expectError(), expectErrorMessage(), and expectErrorMessageMatches()
  • expectNotice(), expectNoticeMessage(), and expectNoticeMessageMatches()
  • expectWarning(), expectWarningMessage(), and expectWarningMessageMatches()

withConsecutive() has been removed in PHPUnit 10.

Q A
PHPUnit version 7.4.3
PHP version 7.2.9
Installation Method PHAR

https://stackoverflow.com/questions/52965511/phpunit-false-positives-when-running-tests-in-separate-processes

/**
 * @runTestsInSeparateProcesses
 */
class ProfileTest extends TestCase
{
    public function testFalsePositive()
    {
        $this->assertFalse(true);
    }
}

In short: thiscode returns all tests passed, when there is annotation @runTestsInSeparateProcesses.
Without the annotation, test correctly fails.
Checked in PHPUnit v.5, with or without annotation, tests fail.

Take this test for example:

<?php

declare(strict_types=1);

namespace Doctrine\DBAL\Tests;

use Exception;
use PHPUnit\Framework\TestCase;

abstract class IntegrationTest extends TestCase
{
    protected $connection;

    /**
     * @before
     */
    protected function connect(): void
    {
        // connect to the database
        $this->connection = ...;
    }

    /**
     * @after
     */
    protected function disconnect(): void
    {
        // disconnect from the database
        $this->connection = null;
    }
}

class TearDownFailureTest extends IntegrationTest
{
    protected function tearDown(): void
    {
        throw new Exception();
    }

    public function testSkipped(): void
    {
        self::markTestSkipped();
    }
}

The code above does the following:

  1. The base class defines the @before and @after methods that will prepare and clean up the test database and the connection.
  2. The specific test cases may define their own setup and teardown procedures.

There is a problem in the TearDownFailureTest that it may get skipped but it doesn't check (and probably can't) if it was skipped, it attempts to perform the teardown logic and throws the exception that gets suppressed by the test runner. Because of that, the disconnect() method in the parent class doesn't get called and the test instance remains connected to the database leaking resources.

It wouldn't be a problem if the test object got destroyed after the run but it doesn't (there are a few reported issues about that, e.g. #3039).

If the teardown methods are meant for the cleanup, should they be called independently on the results of each other? Currently, an exception stops the chain:

foreach ($hookMethods['after'] as $method) {
if ($this->methodDoesNotExistOrIsDeclaredInTestCase($method)) {
continue;
}
$this->{$method}();

Currently, the desired logic could be implemented in a TestListener class which has access to the Test object but it's deprecated:

public function endTest(Test $test, float $time): void;

Neither the deprecated TestHook API, nor the event-based API seem to provide access to the Test instance for reliable cleanup.