Front page | perl.perl5.changes |
Postings from March 2023
Branch: refs/heads/yves/fix_harness_missing_test_files_in_manifest
Home: https://github.com/Perl/perl5
Commit: 64a8fa3566e20bc81beb33861d60c00e97862759
https://github.com/Perl/perl5/commit/64a8fa3566e20bc81beb33861d60c00e97862759
Author: Yves Orton <demerphq@gmail.com>
Date: 2023-03-16 (Thu, 16 Mar 2023)
Changed paths:
M t/TEST
M t/harness
Log Message:
-----------
t/harness - die if t/harness wont run a test listed in MANIFEST
Historically we used to parse out the tests that we ran in t/harness
from the MANIFEST file. At some point this changed and we started
consulting the disk using globs. However because we do not use a
recursive search over the t/ directory it is quite possible that a new
directory of tests is added which actually never runs.
In https://github.com/Perl/perl5/pull/20637#discussion_r1137878155 Tony
C noticed that I had added a new test file t/op/hook/require.t which is
in a new subdirectory t/op/hook/ which was unknown to t/harness and thus
not actually being run by make test_harness. (This patch does NOT add
t/op/hoop to the list of directories to scan, I will do that in the PR.)
I then took the time to add code to detect if any other test files are
not being run, and it turns out that it is also the case for the new
t/class/ directory of tests and it is also the case for the tests for
test.pl itself, found in the t/test_pl directory.
This patch adds logic to detect if this happens and make t/harness die
if it finds a test file in the manifest which will not be detected by
the custom rules for finding test files that is used in t/harness. It
does not die if t/harness finds tests that are not in MANIFEST, that
should be detected by a different test.
The level of complexity in finding and deciding the tests files that we
should run, and the differences between t/TEST and t/harness is fairly
high. In the past Nicholas put some effort into unifying the logic, but
it seems since then we have drifted apart. Even though t/harness uses
t/TEST and the _tests_from_manifest() function, for some time now it
has only used it to find which extensions to test, not which test
files to run. I have *NOT* dug into whether t/TEST is also missing
test files that are in manifest. That can happen in a follow up patch.
Long term we should unify all of this logic so that t/TEST and t/harness
run the same test files always, and that we will always detect
discrepancies between the MANIFEST and the tests we are running. We do
not for instance test that they test the same things. :-) :-(