RFC: Proper Addition Of New Tests

In One Sentence
---------------

It will be checked that new tests fail (they find the bug) before the fix is
applied.


Synopsis
--------

Its is often forgotten that tests are code.  And code has bugs.  Very often
the test won't actually be testing what you think its testing.  So how do
you test a test?

A very simple way is to apply the test and make sure it fails, apply your
code patch, then re-run the test to make sure it passes.  This can all be
automated and should reduce the number of tests which don't work.


Process
-------

A simple patching wrapper can be written (or extend the existing wrapper
around patch that perl5 uses) which automates this process.

1)  Apply the test.
2)  Run the test.  Make sure it fails.
3)  Apply the code patch.
4)  Run the test again, make sure it passes.

Simple.  Stupid.  And there's no reason this can't be applied to Perl5.


Problems
--------

It is necessary to enforce one patch, one test in order to make this
technique feasible.  So this RFC is linked to my other RFC about A Test For
Every Patch.


Things to do NOW
----------------

- Write the testing patching program and/or extend perl5's existing patch 
  wrapper, whichever is easier.  [Up for grabs]

- Begin using this technique on perl5 (independently, or with p5p's
  blessing) and gather statistics on how often tests fail to test what 
  they're testing.  [Up for grabs, too]


Blame
-----

All blame for this RFC can be transfered to Michael G Schwern
<schwern@pobox.com>.
-- 

Michael G Schwern      http://www.pobox.com/~schwern/      schwern@pobox.com
Just Another Stupid Consultant                      Perl6 Kwalitee Ashuranse
But why?  It's such a well designed cesspool of C++ code.  Why wouldn't
you want to hack mozilla?
                -- Ziggy

• RFC: Proper Addition Of New Tests by Michael G Schwern