We often test programs which fails or get stuck (e.g. due to racing conditions). Let’s suppose that after a fix the failure rate is reduced – how can we be sure with a high confidence the fix indeed reduces the failure rate? This question is equivalent to very famous statistics problem – tossing of coin. If we toss a coin 100 times and get the head say 63 times. Can we accept with a high confidence that the probability of getting the head is above 0.5?
Let’s suppose you test a SW program and observe that it sometime is stuck, 10 of 100 runs the program is stuck (e.g. due racing conditions). A SW engineer which is responsible for the program executes some fix and now the program is stuck in 5 runs of 100 (i.e. the observed failure rate is reduced twice from 0.1 to 0.05). Can we conclude that the fix indeed improves a reliability of the program and we are in the right way to make the program ‘stuck-free’ or completely reliable?
In language of statistics we re-phrase the above question as follows:
Can we with confidence level say 98% accept that the failure rate after the fix is below P=0.1?
We have two hypotheses:
Null hypothesis H0: P=0.1
Alternative hypothesis HA: P<0.1
The observed failure rate after the fix is Ps=0.05, let’s compute the value Z as follows:
Z = ( Ps – P )/sqrt[P(1-P)/n]
Where n is the number of runs, in our case n=100 and P=0.1 (null hypothesis)
We reject the null hypothesis (or in other words we accept the alternative hypothesis that the fix helps) if Z <-2.05 (since the left-tailed Z for α=0.02 is -2.05).
In our case with Ps=0.05 and n=100 we have Z=-1.6. Hence, we can’t infer with the confidence above 98% that the fix indeed helps.
Let’s suppose that the SW engineer makes another fix and we obtain better results: 3 failures of 100 (i.e. Ps =0.03). In such case Z = -2.3 < -2.05 and hence we can infer with the confidence above 98% that the last fix indeed reduces the failure rate.
23+ years’ programming and theoretical experience in the computer science fields such as video compression, media streaming and artificial intelligence (co-author of several papers and patents).
the author is looking for new job, my resume
Hello, Neat post. There’s a problem with your site in internet explorer, could test this?K IE still is the market chief and a huge part of other people will leave out your magnificent writing due to this problem.
This is the right blog for anyone who wants to find out about this topic. You realize so much its almost hard to argue with you (not that I actually would want…HaHa). You definitely put a new spin on a topic thats been written about for years. Great stuff, just great!
I like this web site its a master peace ! Glad I noticed this on google .
Good write-up, I am normal visitor of one?¦s web site, maintain up the nice operate, and It is going to be a regular visitor for a long time.
You got a very great website, Sword lily I detected it through yahoo.
Hey there just wanted to give you a quick heads up. The words in your post seem to be running off the screen in Internet explorer. I’m not sure if this is a format issue or something to do with web browser compatibility but I figured I’d post to let you know. The design and style look great though! Hope you get the problem solved soon. Cheers
Some genuinely great articles on this site, thankyou for contribution.
I am lucky that I found this web site, exactly the right info that I was looking for! .
It’s the best time to make some plans for the future and it’s time to be happy. I’ve read this post and if I could I want to suggest you few interesting things or tips. Maybe you could write next articles referring to this article. I want to read more things about it!
Cool blog! Is your theme custom made or did you download it from somewhere? A design like yours with a few simple adjustements would really make my blog stand out. Please let me know where you got your theme. Thank you
almost all original, if i use some figure from other source i add the reference
Thanks for your marvelous posting! I certainly enjoyed reading it, you may be a great author.I will always bookmark your blog and will come back sometime soon. I want to encourage continue your great job, have a nice afternoon!
I am really impressed together with your writing skills and also with the layout on your weblog. Is that this a paid subject or did you customize it yourself? Anyway keep up the nice quality writing, it’s uncommon to look a great weblog like this one these days..
Very interesting information!Perfect just what I was looking for!
I’m really loving the theme/design of your web site. Do you ever run into any internet browser compatibility issues? A couple of my blog visitors have complained about my website not operating correctly in Explorer but looks great in Firefox. Do you have any ideas to help fix this issue?
Do you mind if I quote a few of your articles as long as I provide credit and sources back to your webpage? My blog is in the very same niche as yours and my visitors would definitely benefit from some of the information you present here. Please let me know if this okay with you. Thanks a lot!
you are welcome to quote
Hello my friend! I want to say that this article is amazing, nice written and include almost all vital infos. I¦d like to look extra posts like this .
Thank you a lot for sharing this with all folks you actually realize what you are speaking about! Bookmarked. Kindly additionally seek advice from my site =). We will have a link exchange arrangement between us!
I like this post, enjoyed this one regards for putting up.
Thanks for sharing superb informations. Your web site is so cool. I am impressed by the details that you’ve on this web site. It reveals how nicely you perceive this subject. Bookmarked this web page, will come back for extra articles. You, my friend, ROCK! I found simply the information I already searched all over the place and just couldn’t come across. What an ideal web site.