[cxtest-users] CxTest refinements - design discussion

[Jeremy White]

Hi folks,

I'm happy to announce that we're going to be making a big
push on CxTest in the next few months to try to bring
it up a notch.

Right now, our top priorities are as follows:

  1.  Make runs a bit more consistent.
      We still get the occassional failure that is
      really more due to a test environment or an
      odd metacity bug, and not a true Wine failure.

      We hope to fix that.


  2.  Improve the reporting section dramatically

      We now have a lot of data being fed into
      www.cxtest.org, but it's really hard to read the
      data.

      Now we are going to make a change to catalog
      failures into failure classes; that way,
      we can review runs and analyze each one.


But I also want to make some other reporting changes,
and I'd like to get feedback on this.  I have this
sense that there is some 'One True Way (TM)'
to do this, and I'm not seeing it.  But here is
my best approximation:

In gestalt, what we want to see is whether or
not things are running okay.

When they're not, we need te ability to drill
down and understand why they're not.

As a first change, I think we need a top level
view that isn't just all the most recent runs,
but is instead a view by time period.

So the report would show how many reports came
in during a given time period, and what the overall
success rate was.

Then, in any of the drill down reports, I think
we need the ability to click on a link to be able
to see that 'thing' projected out over time.

So, for example, if I'm looking at the 'supported_other'
test run, and it has a partial success, I'd like to
be able to click on a link next to it and see what the
trend has been for the 'supported_other' run; has it
been failing across the board, or is this failure an
anomoly?  The same thing would go for any test within
a run, and for any platform.

In other words, when I see an anomolous failure,
I need some tools to probe for what variables may
explain the failure.

Does this make sense?  Does someone else have a sense
of the obvious right way that you can't believe
we're missing?

Cheers,

Jeremy