sebastianbergmann
Repos
71
Followers
6625

The PHP Unit Testing framework.

18657
2027

Events

pull request closed
Testdox improvements
  • [x] implements #3567 ability to toggle buffering
  • [ ] implements #3900 reduce data provider noise
  • [x] improve unit test coverage in general
    • [x] Testdox buffering mechanism as seperate from colorization
    • [x] colorized Testdox features like the spinner
    • [x] more coverage Configuration value object
  • [x] bunch of small fixes (typos, code style)
Created at 1 day ago
Testdox improvements

Thank you for your contribution. I appreciate the time you invested in preparing this pull request. However, I have decided not to merge it.

Created at 1 day ago
Unbuffered TestDox printer option

The existing TestDox implementation in PHPUnit 8.5 and PHPUnit 9.5 will not be changed. PHPUnit 10 will have an implementation of TestDox that does not have this problem.

Created at 1 day ago
self referencing data providers

Won't fix, yes.

Created at 1 day ago

Fix: Enable required extensions only

Merge branch '8.5' into 9.5

Bump

Update tools

Update tools

Update URL

Merge branch '8.5' into 9.5

Update tools

Merge branch '8.5' into 9.5

Psalm needs the cURL extension for sending data to Shepherd

Merge branch '8.5' into 9.5

Refactor

Do not enforce time limits when a debugging session through DBGp is active

Merge branch '8.5' into 9.5

Update tools

Merge branch '8.5' into 9.5

Remove SKIPIF conditions that have not been relevant for quite some time

Merge branch '8.5' into 9.5

Update tools

Merge branch '8.5' into 9.5

Created at 1 day ago

Add missing deprecation annotation

Merge branch '9.5'

Created at 1 day ago

Add missing deprecation annotation

Created at 1 day ago
Request for ability to order/filter test suites and/or tests without changes to underlying test code.

It was not implemented.

Created at 1 day ago
Deprecate getMockClass()

getMockClass() has been removed in PHPUnit 10.

Created at 1 day ago
Deprecate withConsecutive()

withConsecutive() has been removed in PHPUnit 10.

Created at 1 day ago
Deprecate expect*() methods that have been removed in PHPUnit 10
  • expectDeprecation(), expectDeprecationMessage(), and expectDeprecationMessageMatches()
  • expectError(), expectErrorMessage(), and expectErrorMessageMatches()
  • expectNotice(), expectNoticeMessage(), and expectNoticeMessageMatches()
  • expectWarning(), expectWarningMessage(), and expectWarningMessageMatches()
Created at 1 day ago
Reliable API for freeing resources used by the test

Take this test for example:

<?php

declare(strict_types=1);

namespace Doctrine\DBAL\Tests;

use Exception;
use PHPUnit\Framework\TestCase;

abstract class IntegrationTest extends TestCase
{
    protected $connection;

    /**
     * @before
     */
    protected function connect(): void
    {
        // connect to the database
        $this->connection = ...;
    }

    /**
     * @after
     */
    protected function disconnect(): void
    {
        // disconnect from the database
        $this->connection = null;
    }
}

class TearDownFailureTest extends IntegrationTest
{
    protected function tearDown(): void
    {
        throw new Exception();
    }

    public function testSkipped(): void
    {
        self::markTestSkipped();
    }
}

The code above does the following:

  1. The base class defines the @before and @after methods that will prepare and clean up the test database and the connection.
  2. The specific test cases may define their own setup and teardown procedures.

There is a problem in the TearDownFailureTest that it may get skipped but it doesn't check (and probably can't) if it was skipped, it attempts to perform the teardown logic and throws the exception that gets suppressed by the test runner. Because of that, the disconnect() method in the parent class doesn't get called and the test instance remains connected to the database leaking resources.

It wouldn't be a problem if the test object got destroyed after the run but it doesn't (there are a few reported issues about that, e.g. https://github.com/sebastianbergmann/phpunit/issues/3039).

If the teardown methods are meant for the cleanup, should they be called independently on the results of each other? Currently, an exception stops the chain: https://github.com/sebastianbergmann/phpunit/blob/8053bc6441311c5b8db213cac56f87633b8c14d6/src/Framework/TestCase.php#L919-L924

Currently, the desired logic could be implemented in a TestListener class which has access to the Test object but it's deprecated: https://github.com/sebastianbergmann/phpunit/blob/5a9574812e6f77e376e1e391c1577425523db829/src/Framework/TestListener.php#L37

Neither the deprecated TestHook API, nor the event-based API seem to provide access to the Test instance for reliable cleanup.

Created at 1 day ago
Proxied method calls are not counted when called on $this

Is this the same as or similar to #4586? Thanks!

Created at 1 day ago
Can't run SINGLE test files with namespace in class name

| Q | A | --------------------| --------------- | PHPUnit version | 9.5.9 | PHP version | 7.4.23 | Installation Method | Composer

Summary

Our app has a mix of real namespaces and an older way of namespacing using underscores. Example, the path to this file is tests/Thing/ThingTest.php

<?php

class Thing_ThingTest extends PHPUnit\Framework\TestCase
{
    public function testThisThing()
    {
        $this->assertTrue(true);
    }
}

Current behavior

In phpunit 9.5.9 this happens when trying to run this test by its path.

deploy/vendor/bin/phpunit ./tests/Thing/ThingTest.php 
Class 'ThingTest' could not be found in '/home/me/project_folder/deploy/tests/Thing/ThingTest.php'.

But in phpunit 8.5.22 it works

deploy/vendor/bin/phpunit ./tests/Thing/ThingTest.php 
PHPUnit 8.5.22 by Sebastian Bergmann and contributors.

Runtime:       PHP 7.4.23 with Xdebug 3.0.3
Configuration: /home/me/project_folder/phpunit.xml.dist

.                                                                   1 / 1 (100%)

Time: 740 ms, Memory: 32.00 MB

OK (1 test, 1 assertion)

How to reproduce

mkdir phpunit_test
cd phpunit_test
composer require phpunit/phpunit:^9.5 --dev
mkdir Thing

Edit composer.json to look like this

{
    "autoload-dev": {
        "psr-0": {
            "": "./"
        }
    },
    "require-dev": {
        "phpunit/phpunit": "^9.5"
    }
}

Create Thing/ThingTest.php

<?php

class Thing_ThingTest extends PHPUnit\Framework\TestCase
{
    public function testThisThing()
    {
        $this->assertTrue(true);
    }
}

Run the test

vendor/bin/phpunit Thing/ThingTest.php
Class 'ThingTest' could not be found in '/Users/jtowe/Documents/Projects/phpunit_test/Thing/ThingTest.php'.

Revert back to 8.5 and see that it works

composer require phpunit/phpunit:^8.5 --dev -W
vendor/bin/phpunit Thing/ThingTest.php
PHPUnit 8.5.22 by Sebastian Bergmann and contributors.

.                                                                   1 / 1 (100%)

Time: 14 ms, Memory: 4.00 MB

OK (1 test, 1 assertion)

NOTE: This only seems to happen when I try to run the test by itself. If I make a phpunit.xml of

<?xml version="1.0" encoding="UTF-8"?>
<phpunit
    bootstrap="./vendor/autoload.php"
    >
    <testsuites>
        <testsuite name="Test Suite">
            <directory>./</directory>
        </testsuite>
    </testsuites>
</phpunit>

And run all the tests vendor/bin/phpunit it works

PHPUnit 9.5.11 by Sebastian Bergmann and contributors.

.                                                                   1 / 1 (100%)

Time: 00:00.001, Memory: 6.00 MB

OK (1 test, 1 assertion)

Expected behavior

I'm not sure what's happening but there's a high chance i'm just doing something wrong. We have LOTS of tests namespaced using underscores so if the answer is to namespace them all using Namespace I may have to revert back to 8.5 for now. I think i'd just like to understand what's changed to make this happen so I can revert or fix it.

composer info | sort

composer info | sort                                                                                     1 ✘  12:55:47 PM 
doctrine/instantiator              1.4.0   A small, lightweight utility to instantiate objects in PHP without invoking their constructors
myclabs/deep-copy                  1.10.2  Create deep copies (clones) of your objects
nikic/php-parser                   v4.13.2 A PHP parser written in PHP
phar-io/manifest                   2.0.3   Component for reading phar.io manifest information from a PHP Archive (PHAR)
phar-io/version                    3.1.0   Library for handling version information and constraints
phpdocumentor/reflection-common    2.2.0   Common reflection classes used by phpdocumentor to reflect the code structure
phpdocumentor/reflection-docblock  5.3.0   With this component, a library can provide support for annotations via DocBlocks or otherwise retrieve information that ...
phpdocumentor/type-resolver        1.6.0   A PSR-5 based resolver of Class names, Types and Structural Element Names
phpspec/prophecy                   v1.15.0 Highly opinionated mocking framework for PHP 5.3+
phpunit/php-code-coverage          9.2.10  Library that provides collection, processing, and rendering functionality for PHP code coverage information.
phpunit/php-file-iterator          3.0.6   FilterIterator implementation that filters files based on a list of suffixes.
phpunit/php-invoker                3.1.1   Invoke callables with a timeout
phpunit/php-text-template          2.0.4   Simple template engine.
phpunit/php-timer                  5.0.3   Utility class for timing
phpunit/phpunit                    9.5.11  The PHP Unit Testing framework.
sebastian/cli-parser               1.0.1   Library for parsing CLI options
sebastian/code-unit                1.0.8   Collection of value objects that represent the PHP code units
sebastian/code-unit-reverse-lookup 2.0.3   Looks up which function or method a line of code belongs to
sebastian/comparator               4.0.6   Provides the functionality to compare PHP values for equality
sebastian/complexity               2.0.2   Library for calculating the complexity of PHP code units
sebastian/diff                     4.0.4   Diff implementation
sebastian/environment              5.1.3   Provides functionality to handle HHVM/PHP environments
sebastian/exporter                 4.0.4   Provides the functionality to export PHP variables for visualization
sebastian/global-state             5.0.3   Snapshotting of global state
sebastian/lines-of-code            1.0.3   Library for counting the lines of code in PHP source code
sebastian/object-enumerator        4.0.4   Traverses array structures and object graphs to enumerate all referenced objects
sebastian/object-reflector         2.0.4   Allows reflection of object attributes, including inherited and non-public ones
sebastian/recursion-context        4.0.4   Provides functionality to recursively process PHP variables
sebastian/resource-operations      3.0.3   Provides a list of PHP built-in functions that operate on resources
sebastian/type                     2.3.4   Collection of value objects that represent the types of the PHP type system
sebastian/version                  3.0.2   Library that helps with managing the version number of Git-hosted PHP projects
symfony/polyfill-ctype             v1.24.0 Symfony polyfill for ctype functions
theseer/tokenizer                  1.2.1   A small library for converting tokenized PHP source code into XML and potentially other formats
webmozart/assert                   1.10.0  Assertions to validate method input/output with nice error messages.
Created at 1 day ago
Can't run SINGLE test files with namespace in class name

This should have been fixed by #5021.

Created at 1 day ago
Exporter configuration - allow setting custom exporter implementation or limiting maximum depth

Exporter implementation is right now hardcoded in Constraints and Comparators. This works fine in most cases but it becomes unusable in the case of large objects such as Doctrine entities.

When an assertion fails with such objects, the recursive dump may take a lot of time and memory (EntityManager is referenced by Doctrine entities which means pretty much whole identity map is a part of the result).

We're experiencing delays in order of minutes both locally (in PHPStorm, it seems like the TeamCity listener adds some delay when it generates diff) and on CI with raw output. The memory increase is also significant, our CI pods are sometimes killed due to running out of memory (caused only by such dump) which makes it very hard to debug which test/assertion caused the issue.

Surely tests should not fail and be always reproducible but that's simply not 100% true in real projects and it wastes developers time.

We mitigated the issue for most cases by overriding assertNull and providing our own comparator for entities with custom exporter but it does not cover all cases.

I'd very much appreciate a possibility to provide our own instance of exporter or at least provide a configuration option that would limit exporter depth when traversing objects.

Does any of these two options seem viable to you or do you have any other tips? Thank you

Created at 1 day ago
Exporter configuration - allow setting custom exporter implementation or limiting maximum depth

My suggestion would be to use a custom comparator that specifically handles Doctrine entities.

Created at 1 day ago
set_error_handler in bootstrap with process isolation does hide errors

Hi.

I have a pretty weird case of hidden errors in PHPUnit which i can't find out why.

Lets see following testcase and setup. Running on Windows 10 with PHP 8.1.

Bootstrap is verified to be included in both test results bellow by making a test assertion $this->assertSame('123', $_SERVER['TESTVAR']); to a global variable that has been set only in bootstrap.php.

Command: php phpunit.phar --bootstrap bootstrap.php TestTest.php --process-isolation

TestTest.php

<?php
declare(strict_types=1);
final class TestTest extends PHPUnit\Framework\TestCase
{
    public function testTest(): void
    {
        $this->assertSame('123', $_SERVER['TESTVAR']);
        echo $foo;
        $this->assertTrue(true);
    }

    public function testTestWithDirectErrorHandlerSet(): void
    {
        $this->assertSame('123', $_SERVER['TESTVAR']);
        set_error_handler(function (mixed $errno, mixed $errstr, mixed $errfile, mixed $errline) {
            throw new Exception($errstr);
        }, E_ALL);
        echo $foo;
        $this->assertTrue(true);
    }
}

bootstrap.php

<?php
$_SERVER['TESTVAR'] = '123';
error_reporting(E_ALL);
set_error_handler(function (mixed $errno, mixed $errstr, mixed $errfile, mixed $errline) {
    throw new Exception($errstr);
}, E_ALL);

Running the test with the given command hide the error of the first test. However, when setting the same error handler directly in a test, it does correctly mark the test as failed an throw an exception. The set_error_handler callback in the bootstrap is never called and i can't find out why. Output:


PHPUnit 9.5.11 by Sebastian Bergmann and contributors.

.E                                                                  2 / 2 (100%)

Time: 00:00.439, Memory: 24.00 MB

There was 1 error:

1) TestTest::testTestWithDirectErrorHandlerSet
Exception: Undefined variable $foo

C:\www\ouced\dev-tools\phpunit-tests\TestTest.php:16
C:\www\ouced\dev-tools\phpunit-tests\TestTest.php:18

ERRORS!
Tests: 2, Assertions: 3, Errors: 1.

Running the same test, but without process-isolation does mark both tests as failed, which is correct.

PHPUnit 9.5.11 by Sebastian Bergmann and contributors.

EE                                                                  2 / 2 (100%)

Time: 00:00.001, Memory: 24.00 MB

There were 2 errors:

1) TestTest::testTest
Exception: Undefined variable $foo

C:\www\ouced\dev-tools\phpunit-tests\bootstrap.php:5
C:\www\ouced\dev-tools\phpunit-tests\TestTest.php:8

2) TestTest::testTestWithDirectErrorHandlerSet
Exception: Undefined variable $foo

C:\www\ouced\dev-tools\phpunit-tests\TestTest.php:16
C:\www\ouced\dev-tools\phpunit-tests\TestTest.php:18

ERRORS!
Tests: 2, Assertions: 2, Errors: 2.
Created at 1 day ago
set_error_handler in bootstrap with process isolation does hide errors

Your error handler is deactivated when PHPUnit registers its own error handler. This is not a bug but expected behaviour, sorry.

Created at 1 day ago
Mocking \ReflectionClass throws an error in PHPUnit 8.5 when run with PHP>=8.1

| Q | A | --------------------| --------------- | PHPUnit version | 8.5 | PHP version | 8.1 | Installation Method | Composer

Summary

Mocking \ReflectionClass throws an error in PHPUnit 8.5 when run with PHP>=8.1.

Current behavior

Throws an exception: Call to undefined method ReflectionUnionType::getName().

How to reproduce

// Execute with PHP>=8.1
$this->createMock(\ReflectionClass::class);

Expected behavior

Works like in PHPUnit 9.3.

Created at 1 day ago
Mocking \ReflectionClass throws an error in PHPUnit 8.5 when run with PHP>=8.1

No feedback, closing.

Created at 1 day ago
Setting Env Variables With $_ENV[keyName] Issue

| Q | A | --------------------| --------------- | PHPUnit version | 9.5.10 | PHP version | 8.0.15 | Installation Method | Composer

Summary

Setting env variables with $_ENV[keyName] does not seem to be working.

Current behavior

On a project where setting environment variables using $_ENV[keyName] in unit tests has worked in the past. I've been doing a bit of checking on it and it appears these variables are remaining unchanged. force="true" is being used in the test.xml

How to reproduce

  • Create a project with a single unit test and a single variable in tests.xml with force="true".
  • Try override this by setting the variable with $_ENV[keyName]
  • var_dump the variable

Expected behavior

The variable should be change

Created at 1 day ago
Setting Env Variables With $_ENV[keyName] Issue

I cannot reproduce this.

Created at 1 day ago
Test for dangling error handler registrations?

Hey. We stumbled upon a scenario in our test suites and think it might be a useful feature for phpunit.

When a test registers an own error handler using set_error_handler() and fails to unset this error handler again (eg. in a tearDown()), this registered error handler may "eat" errors raised by subsequent unrelated tests, since they may not bubble up into the default phpunit error handler anymore, depending on implementation of that error handler.

Phpunit already implements a similar sanity check in TestCase stopOutputBuffering() where it verifies a test did not leave a dangling output buffer. It marks a test that fails to do so as 'risky'.

We'd suggest a similar thing for error handlers. Unfortunately, there is no method like ob_get_level() for error handling, but it's still possible to verify if a test added an own error handler, since set_error_handler() returns the previous one from the stack. We came up with code like that in our general tearDown():

            // Register a dummy error handler to retrieve the previous one
            $previousErrorHandler = set_error_handler(function () {});
            // Drop dummy error handler again
            restore_error_handler();
            // Previous *must* be the phpunit ErrorHandler
            if (!$previousErrorHandler instanceof ErrorHandler) {
                throw new \RuntimeException(
                    'tearDown() check: A dangling error handler setup has been found. Use restore_error_handler() to unset it.',
                    1410633510
                );
            }

We register a dummy error handler and see if the 'previous' error handler is the phpunit one. If not, we throw an exception, but we could of course throw RiskyTestError() or similar.

Do you think a feature like that would be generally useful for phpunit? If so, we may try to implement something, hoping it's not too complicated ;)

Created at 1 day ago
Test for dangling error handler registrations?

I do not think that this is worth pursuing, sorry.

Created at 1 day ago
Unsupported Unicode character in XML

| Q | A | --------------------| --------------- | PHPUnit version | master | PHP version | 7.2 | Installation Method | not installed

Summary

I was searching for working solution to escape all unsupported XML symbols and I find Xml::prepareString(). But looks like chars FFFE, FFFF is not get escaped. Can somebody confirm this?

Current behavior

Looks like chars FFFE, FFFF is not get escaped

How to reproduce

$fffe = hex2bin('efbfbe');
Xml::prepareString($fffe);
$ffff = hex2bin('efbfbf');
Xml::prepareString($ffff);

Expected behavior

Any Unicode character is allowed, excluding the surrogate blocks, FFFE, and FFFF (not even as character reference). https://www.w3.org/TR/xml/#charsets

Created at 1 day ago
TestListener object argument loader regression

| Q | A | --------------------| --------------- | PHPUnit version | 9.5.8 | PHP version | 7.4.22 | Installation Method | PHAR distrib from official site https://phar.phpunit.de/

Summary

I've detected a regression on PHPUnit 9.5 when we use an object/class as argument of a testlistener

Current behavior

Class 'MyLoggerObjArg' not found

How to reproduce

Here are code I used to reproduce issue

<?php declare(strict_types=1);

require dirname(__DIR__) . '/vendor/autoload.php';

class MyLoggerObjArg
{
}

class MyLogger implements \PHPUnit\Framework\TestListener
{
    private $target;

    public function __construct($class = null)
    {
        $this->target = __DIR__ . '/testlistener-phpunit-' .  \PHPUnit\Runner\Version::series() . '-' . date('YmdHis') . '.log';

        $data = \PHPUnit\Runner\Version::getVersionString() . PHP_EOL;
        error_log($data, 3, $this->target);

        if (is_object($class)) {
            error_log('class exists(My Logger Object Arg) ? ' . (class_exists('MyLoggerObjArg') ? 'yes': 'no') . PHP_EOL,3, $this->target);
        }
    }

    public function addError(\PHPUnit\Framework\Test $test, Throwable $t, float $time): void
    {
        $data = __FUNCTION__ . ', ' . $test->getName() . PHP_EOL;
        error_log($data, 3, $this->target);
    }

    public function addWarning(\PHPUnit\Framework\Test $test, \PHPUnit\Framework\Warning $e, float $time): void
    {
        $data = __FUNCTION__ . ', ' . $test->getName() . PHP_EOL;
        error_log($data, 3, $this->target);
    }

    public function addFailure(\PHPUnit\Framework\Test $test, \PHPUnit\Framework\AssertionFailedError $e, float $time): void
    {
        $data = __FUNCTION__ . ', ' . $test->getName() . PHP_EOL;
        error_log($data, 3, $this->target);
    }

    public function addIncompleteTest(\PHPUnit\Framework\Test $test, Throwable $t, float $time): void
    {
        $data = __FUNCTION__ . ', ' . $test->getName() . PHP_EOL;
        error_log($data, 3, $this->target);
    }

    public function addRiskyTest(\PHPUnit\Framework\Test $test, Throwable $t, float $time): void
    {
        $data = __FUNCTION__ . ', ' . $test->getName() . PHP_EOL;
        error_log($data, 3, $this->target);
    }

    public function addSkippedTest(\PHPUnit\Framework\Test $test, Throwable $t, float $time): void
    {
        $data = __FUNCTION__ . ', ' . $test->getName() . PHP_EOL;
        error_log($data, 3, $this->target);
    }

    public function startTestSuite(\PHPUnit\Framework\TestSuite $suite): void
    {
        $data = __FUNCTION__ . ', ' . $suite->getName() . PHP_EOL;
        error_log($data, 3, $this->target);
    }

    public function endTestSuite(\PHPUnit\Framework\TestSuite $suite): void
    {
        $data = __FUNCTION__ . ', ' . $suite->getName() . PHP_EOL;
        error_log($data, 3, $this->target);
    }

    public function startTest(\PHPUnit\Framework\Test $test): void
    {
        $data = __FUNCTION__ . ', ' . $test->getName() . PHP_EOL;
        error_log($data, 3, $this->target);
    }

    public function endTest(\PHPUnit\Framework\Test $test, float $time): void
    {
        $data = __FUNCTION__ . ', ' . $test->getName() . PHP_EOL;
        error_log($data, 3, $this->target);
    }
}
<?php declare(strict_types=1);

namespace Your\Name_Space;

use PHPUnit\Framework\TestCase;

class YourTestSuite extends TestCase
{
    public function testIncomplete()
    {
        // Optional: Test anything here, if you want.
        $this->assertTrue(TRUE, 'This should already work.');

        // Stop here and mark this test as incomplete.
        $this->markTestIncomplete(
          'This test has not been implemented yet.'
        );
    }

    public function testRisky()
    {
    }

    public function testSkipped()
    {
        $this->markTestSkipped('This test was skipped for any reason.');
    }

    public function testFailure()
    {
        $this->assertEmpty(array('foo'));
    }

    public function testPass()
    {
        $this->assertTrue(TRUE, 'This should always work.');
    }

    /**
     * @dataProvider additionProvider
     */
    public function testDataProvider($a, $b, $expected)
    {
        $this->assertEquals($expected, $a + $b);
    }

    public function additionProvider()
    {
        return [
            [0, 0, 0],
            [0, 1, 1],
            [1, 0, 1],
            [1, 1, 3]
        ];
    }
}
<phpunit xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:noNamespaceSchemaLocation="https://schema.phpunit.de/9.5/phpunit.xsd"
         backupGlobals="false"
         backupStaticAttributes="false"
         bootstrap="bootstrap.php"
         colors="true"
         stopOnError="false"
         stopOnFailure="false"
         stopOnIncomplete="false"
         stopOnRisky="false"
         stopOnSkipped="false"
         beStrictAboutTestsThatDoNotTestAnything="true"
         verbose="false">
    <listeners>
        <listener class="MyLogger">
        </listener>
    </listeners>
    <testsuites>
        <testsuite name="Demo Test Suite">
            <file>testSuite.php</file>
        </testsuite>
    </testsuites>
</phpunit>
<phpunit xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:noNamespaceSchemaLocation="https://schema.phpunit.de/9.5/phpunit.xsd"
         backupGlobals="false"
         backupStaticAttributes="false"
         bootstrap="bootstrap.php"
         colors="true"
         stopOnError="false"
         stopOnFailure="false"
         stopOnIncomplete="false"
         stopOnRisky="false"
         stopOnSkipped="false"
         beStrictAboutTestsThatDoNotTestAnything="true"
         verbose="false">
    <listeners>
        <listener class="MyLogger">
            <arguments>
                <object class="MyLoggerObjArg" />
            </arguments>
        </listener>
    </listeners>
    <testsuites>
        <testsuite name="Demo Test Suite">
            <file>testSuite.php</file>
        </testsuite>
    </testsuites>
</phpunit>

WARNING I've noticed also that a startTestSuite end endTestSuite with no entry are generated only with PHPUnit 9.5, while it was not on PHPUnit 8.5 and even PHPUnit 7.5

See

PHPUnit 9.5.8 by Sebastian Bergmann and contributors.
startTestSuite, 
startTestSuite, Demo Test Suite
startTestSuite, Your\Name_Space\YourTestSuite
startTest, testIncomplete
addIncompleteTest, testIncomplete
endTest, testIncomplete
startTest, testRisky
addRiskyTest, testRisky
endTest, testRisky
startTest, testSkipped
addSkippedTest, testSkipped
endTest, testSkipped
startTest, testFailure
addFailure, testFailure
endTest, testFailure
startTest, testPass
endTest, testPass
startTestSuite, Your\Name_Space\YourTestSuite::testDataProvider
startTest, testDataProvider with data set #0
endTest, testDataProvider with data set #0
startTest, testDataProvider with data set #1
endTest, testDataProvider with data set #1
startTest, testDataProvider with data set #2
endTest, testDataProvider with data set #2
startTest, testDataProvider with data set #3
addFailure, testDataProvider with data set #3
endTest, testDataProvider with data set #3
endTestSuite, Your\Name_Space\YourTestSuite::testDataProvider
endTestSuite, Your\Name_Space\YourTestSuite
endTestSuite, Demo Test Suite
endTestSuite, 

Expected behavior

With following commands: phpunit -c examples/listener_no_arg.xml or even phpunit -c examples/listener_obj_arg.xml

PHPUnit 9.5.8 by Sebastian Bergmann and contributors.

IRSF....F                                                           9 / 9 (100%)

Time: 00:00.004, Memory: 18.00 MB

There were 2 failures:

1) Your\Name_Space\YourTestSuite::testFailure
Failed asserting that an array is empty.

/shared/backups/bartlett/phpunit-bootstrap/examples/testSuite.php:31

2) Your\Name_Space\YourTestSuite::testDataProvider with data set #3 (1, 1, 3)
Failed asserting that 2 matches expected 3.

/shared/backups/bartlett/phpunit-bootstrap/examples/testSuite.php:44

--

There was 1 risky test:

1) Your\Name_Space\YourTestSuite::testRisky
This test did not perform any assertions

/shared/backups/bartlett/phpunit-bootstrap/examples/testSuite.php:20

FAILURES!
Tests: 9, Assertions: 7, Failures: 2, Skipped: 1, Incomplete: 1, Risky: 1.
Created at 1 day ago
TestListener object argument loader regression

Anything related to TestListener in PHPUnit 8.5 and PHPUnit 9.5 will not be changed anymore and has been removed in PHPUnit 10.

Created at 1 day ago
@xfail annotation

A useful feature of pytest is its @pytest.mark.xfail annotation, which indicates that a test should run but is expected to fail. If a test so annotated does pass, the test run interprets it as a failure.

Generally this is used in TDD and BDD to indicate that a bug or feature is still a work in progress. It's distinct from skipping in that the failure when the test starts passing should prompt the developer to remove the annotation or update the test accordingly.

I did some looking around and was surprised to see that this seems to be a unique feature of pytest (at least by that name). I'd like to add it to PHPUnit, but wanted to get a 👍 👎 before starting work.

Created at 1 day ago