Order of tests which use data from data providers is not affected by test sorting

[sebastianbergmann]

I had a look into it and I think what I see in my testing is, that the order of data-provider tests is not ordered by defects.
meaning: when running a test which uses a dataprovider and the n-th data provided test fails, I would expect this test would get re-ordered to the front.

it seems thats not the case , and I don't know whether thats a oversight or intentional. wdyt?

see

➜  infection git:(master) ✗ '/Users/m.staab/Documents/GitHub/infection/vendor/bin/phpunit' '--configuration' '/var/folders/h7/phd4h7t106v6y8kbxq2hdxhc0000gn/T/infection/phpunitConfiguration.08b04c35617d45e5bac7f7d7b9d61863.infection.xml' --testdox
PHPUnit 11.5.25 by Sebastian Bergmann and contributors.

Runtime:       PHP 8.3.22
Configuration: /private/var/folders/h7/phd4h7t106v6y8kbxq2hdxhc0000gn/T/infection/phpunitConfiguration.08b04c35617d45e5bac7f7d7b9d61863.infection.xml

..................................................F

Time: 00:00.194, Memory: 20.00 MB

Configuration Factory (Infection\Tests\Configuration\ConfigurationFactory)
 ✔ It can create a configuration with data set "minimal"
 ✔ It can create a configuration with data set "null html file log path with existing path from config file"
 ✔ It can create a configuration with data set "absolute html file log path"
 ✔ It can create a configuration with data set "relative html file log path"
 ✔ It can create a configuration with data set "override html file log path from CLI option with existing path from config file"
 ✔ It can create a configuration with data set "set html file log path from CLI option when config file has no setting"
 ✔ It can create a configuration with data set "null html file log path in config and CLI"
 ✔ It can create a configuration with data set "null timeout"
 ✔ It can create a configuration with data set "config timeout"
 ✔ It can create a configuration with data set "null tmp dir"
 ✔ It can create a configuration with data set "empty tmp dir"
 ✔ It can create a configuration with data set "relative tmp dir path"
 ✔ It can create a configuration with data set "absolute tmp dir path"
 ✔ It can create a configuration with data set "no existing base path for code coverage"
 ✔ It can create a configuration with data set "absolute base path for code coverage"
 ✔ It can create a configuration with data set "relative base path for code coverage"
 ✔ It can create a configuration with data set "no PHPUnit config dir"
 ✔ It can create a configuration with data set "relative PHPUnit config dir"
 ✔ It can create a configuration with data set "absolute PHPUnit config dir"
 ✔ It can create a configuration with data set "progress in non-CI environment"
 ✔ It can create a configuration with data set "progress in CI environment"
 ✔ It can create a configuration with data set "no progress in non-CI environment"
 ✔ It can create a configuration with data set "no progress in CI environment"
 ✔ It can create a configuration with data set "Github Actions annotation disabled, not logged in non-Github Actions environment"
 ✔ It can create a configuration with data set "Github Actions annotation disabled, not logged in Github Actions environment"
 ✔ It can create a configuration with data set "Github Actions annotation not provided, not logged in non-Github Actions environment"
 ✔ It can create a configuration with data set "Github Actions annotation not provided, logged in Github Actions environment"
 ✔ It can create a configuration with data set "Github Actions annotation enabled, logged in non-Github Actions environment"
 ✔ It can create a configuration with data set "Github Actions annotation enabled, logged in Github Actions environment"
 ✔ It can create a configuration with data set "null GitLab file log path with existing path from config file"
 ✔ It can create a configuration with data set "absolute GitLab file log path"
 ✔ It can create a configuration with data set "relative GitLab file log path"
 ✔ It can create a configuration with data set "override GitLab file log path from CLI option with existing path from config file"
 ✔ It can create a configuration with data set "set GitLab file log path from CLI option when config file has no setting"
 ✔ It can create a configuration with data set "null GitLab file log path in config and CLI"
 ✔ It can create a configuration with data set "ignoreMsiWithNoMutations not specified in schema and true in input"
 ✔ It can create a configuration with data set "ignoreMsiWithNoMutations not specified in schema and false in input"
 ✔ It can create a configuration with data set "ignoreMsiWithNoMutations true in schema and not specified in input"
 ✔ It can create a configuration with data set "ignoreMsiWithNoMutations false in schema and not specified in input"
 ✔ It can create a configuration with data set "ignoreMsiWithNoMutations true in schema and false in input"
 ✔ It can create a configuration with data set "ignoreMsiWithNoMutations false in schema and true in input"
 ✔ It can create a configuration with data set "minMsi not specified in schema and not specified in input"
 ✔ It can create a configuration with data set "minMsi specified in schema and not specified in input"
 ✔ It can create a configuration with data set "minMsi not specified in schema and specified in input"
 ✔ It can create a configuration with data set "minMsi specified in schema and specified in input"
 ✔ It can create a configuration with data set "minCoveredMsi not specified in schema and not specified in input"
 ✔ It can create a configuration with data set "minCoveredMsi specified in schema and not specified in input"
 ✔ It can create a configuration with data set "minCoveredMsi not specified in schema and specified in input"
 ✔ It can create a configuration with data set "minCoveredMsi specified in schema and specified in input"
 ✔ It can create a configuration with data set "no test framework"
 ✘ It can create a configuration with data set "test framework from config"
   │
   │ Failed asserting that two strings are identical.
   │ --- Expected
   │ +++ Actual
   │ @@ @@
   │ -'phpspec'
   │ +'phpunit'
   │
   │ /Users/m.staab/Documents/GitHub/infection/tests/phpunit/Configuration/ConfigurationAssertions.php:126
   │ /Users/m.staab/Documents/GitHub/infection/tests/phpunit/Configuration/ConfigurationFactoryTest.php:193
   │

FAILURES!
Tests: 51, Assertions: 2423, Failures: 1.

with a config of

<?xml version="1.0" encoding="UTF-8"?>
<phpunit xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="./vendor/phpunit/phpunit/phpunit.xsd"
 bootstrap="/var/folders/h7/phd4h7t106v6y8kbxq2hdxhc0000gn/T/infection/interceptor.autoload.08b04c35617d45e5bac7f7d7b9d61863.infection.php" 
 colors="false" executionOrder="defects" failOnWarning="true" failOnRisky="true" requireCoverageMetadata="true" displayDetailsOnPhpunitDeprecations="true" displayDetailsOnTestsThatTriggerDeprecations="true" displayDetailsOnTestsThatTriggerWarnings="true" cacheDirectory="/var/folders/h7/phd4h7t106v6y8kbxq2hdxhc0000gn/T/infection/.phpunit.result.cache.08b04c35617d45e5bac7f7d7b9d61863" cacheResult="true" stopOnDefect="true" stderr="false">
  <php>
    <env name="COLUMNS" value="100" force="true"/>
  </php>
  <testsuites>
    <testsuite name="Infection testsuite with filtered tests">
      <file>/Users/m.staab/Documents/GitHub/infection/tests/phpunit/Configuration/ConfigurationFactoryTest.php</file>
    </testsuite>
  </testsuites>
  <source>
    <include>
      <directory>/Users/m.staab/Documents/GitHub/infection/src</directory>
    </include>
  </source>
  <groups>
    <exclude>
      <group>e2e</group>
    </exclude>
  </groups>
</phpunit>

reproduces the same ordering all the time, while I feel the failed

✘ It can create a configuration with data set "test framework from config"

test should get re-ordered to the front. wdyt?

Originally posted by @staabm in #6254

[staabm]

looking into it.

[staabm]

I think the root cause of the problem is, that DefaultResultCache is persisting the test-status with keys like
PHPUnit\TestFixture\FaillingDataProviderTest::testWithProvider#good1 in DefaultResultCache::setStatus().

but looks them up later on in the format:
PHPUnit\TestFixture\FaillingDataProviderTest::testWithProvider with data set "good1" in DefaultResultCache::status().

---

A test like

<?php declare(strict_types=1);
/*
 * This file is part of PHPUnit.
 *
 * (c) Sebastian Bergmann <sebastian@phpunit.de>
 *
 * For the full copyright and license information, please view the LICENSE
 * file that was distributed with this source code.
 */
namespace PHPUnit\TestFixture;

use PHPUnit\Framework\Attributes\DataProvider;
use PHPUnit\Framework\TestCase;

class FaillingDataProviderTest extends TestCase {
    public function testOne() {
        $this->assertTrue(true);
    }

    #[DataProvider('provideData')]
    public function testWithProvider($sum, $summand, $summmand2) {
        $this->assertSame($sum, $summand + $summmand2);
    }

    static public function provideData():array {
        return [
            'good1' => [3, 1, 2],
            'good2' => [5, 2, 3],
            'fail1' => [10, 3, 4],
            'good3' => [10, 5, 5],
            'fail2' => [20, 3, 4],
        ];
    }
}

invoked with php phpunit tests/_files/FaillingDataProviderTest.php 2 times re-orders correctly with a hacky fix like

diff --git a/phpunit.xml b/phpunit.xml
index d5a0696b5..3fc5baf65 100644
--- a/phpunit.xml
+++ b/phpunit.xml
@@ -4,6 +4,7 @@
          bootstrap="tests/bootstrap.php"
          cacheDirectory=".phpunit.cache"
          beStrictAboutOutputDuringTests="true"
+         executionOrder="defects"
          failOnRisky="true"
          failOnWarning="true"
          colors="true">

diff --git a/src/Event/Value/Test/TestMethod.php b/src/Event/Value/Test/TestMethod.php
index 4c9726454..99ee3157b 100644
--- a/src/Event/Value/Test/TestMethod.php
+++ b/src/Event/Value/Test/TestMethod.php
@@ -109,7 +109,7 @@ public function id(): string
         $buffer = $this->className . '::' . $this->methodName;
 
         if ($this->testData()->hasDataFromDataProvider()) {
-            $buffer .= '#' . $this->testData->dataFromDataProvider()->dataSetName();
+            $buffer .= ' with data set "' . $this->testData->dataFromDataProvider()->dataSetName() .'"';
         }
 
         return $buffer;