[cxtest-users] CxTest refinements - design discussion

[Olivier Kaloudoff]

>
>
>Hi folks,
>
>  
>

Hi Jeremy,

>I'm happy to announce that we're going to be making a big
>push on CxTest in the next few months to try to bring
>it up a notch.
>
>  
>
That's really good news !

>Right now, our top priorities are as follows:
>
>  1.  Make runs a bit more consistent.
>      We still get the occassional failure that is
>      really more due to a test environment or an
>      odd metacity bug, and not a true Wine failure.
>
>      We hope to fix that.
>  
>

I don't see exactly which kind of error you mean, but I have to say that
it seems very hard to guess why this particular run or other failed, 
from a user
point of view. (no real error message ...)

http://www.cxtest.org/reports?cxtest=5c91834b7cd13762a9845328f198ad79;run_id=2201

It's a little bit frustrating to hang on the same error during days, and 
not know how
to dig into this and understand. I guess that's your work, not mine ;-)

>
>  2.  Improve the reporting section dramatically
>
>      We now have a lot of data being fed into
>      www.cxtest.org, but it's really hard to read the
>      data.
>
>      Now we are going to make a change to catalog
>      failures into failure classes; that way,
>      we can review runs and analyze each one.
>
>
>But I also want to make some other reporting changes,
>and I'd like to get feedback on this.  I have this
>sense that there is some 'One True Way (TM)'
>to do this, and I'm not seeing it.  But here is
>my best approximation:
>
>In gestalt, what we want to see is whether or
>not things are running okay.
>
>When they're not, we need te ability to drill
>down and understand why they're not.
>
>As a first change, I think we need a top level
>view that isn't just all the most recent runs,
>but is instead a view by time period.
>
>So the report would show how many reports came
>in during a given time period, and what the overall
>success rate was.
>
>Then, in any of the drill down reports, I think
>we need the ability to click on a link to be able
>to see that 'thing' projected out over time.
>
>So, for example, if I'm looking at the 'supported_other'
>test run, and it has a partial success, I'd like to
>be able to click on a link next to it and see what the
>trend has been for the 'supported_other' run; has it
>been failing across the board, or is this failure an
>anomoly?  The same thing would go for any test within
>a run, and for any platform.
>
>In other words, when I see an anomolous failure,
>I need some tools to probe for what variables may
>explain the failure.
>
>Does this make sense?  Does someone else have a sense
>of the obvious right way that you can't believe
>we're missing?
>  
>

Here is  the "True Way" I would propose;


each run has fields like:
- launched by an user (u)
- on a particular distribution (dist),
- at day (d),
- for program (p)
(and any critera that is useful for you)

then, there would be a 'browse' mecanism, much like Freshmeat's one,
that would allow to (add/remove) search critera (including date range,
type of plaform, and so on), and, at each stage, would show the number
of occurences available if I add this critera(89) or if I del this 
critera(500),
or another one, and so on.

Hope that freshmeat users understand my proposition, and that this is 
understandable
for others.

The goal is to have the maximum information in a minimum of click, without
having "empty path", because I know each time, before clicking on a 
critera add
or remove, that there is much or not much data behind.


Olivier