Blogs - Tags - development_analytics Blogsurn:lsid:ibm.com:blogs:entries03022015-05-15T16:43:55-04:00IBM Connections - Blogsurn:lsid:ibm.com:blogs:entry-073fe9f5-307e-4403-802f-d3c07dfcd246A video demonstrating a tool applying predictive methods to Agile effortsmcantor@us.ibm.com110000CH6XactivefalseComment Entriesapplication/atom+xml;type=entryLikestrue2013-03-14T08:33:32-04:002013-03-14T08:33:32-04:00<div>In my last blog, I laid out a vision of how a project lead and her stakeholders might use the predictive analytics to drive to better project outcomes. As I mentioned in the entry, IBM Rational is work on such a tool. A demonstration of this tool is found here: <a href="http://researcher.watson.ibm.com/researcher/files/us-tarr/Public%20ANDES%20demo%20recording.mp4">Agile Development Analytics Demo</a>. The video was created by Peri Tarr, the lead architect on the project.</div><div> </div><div>Some of you might notice that the terminology and development process described in the demo is at odds with your understanding of Agile. We do understand that and currently we are working on making a robust tool that accommodates a wide range of processes for what might some might call 'pure Agile' to various hybrids we are discovering in the market place. </div><div> </div><div>In the next blog, I will explain more on how the tool works. </div><div> </div>In my last blog, I laid out a vision of how a project lead and her stakeholders might use the predictive analytics to drive to better project outcomes. As I mentioned in the entry, IBM Rational is work on such a tool. A demonstration of this tool is found...003605urn:lsid:ibm.com:blogs:entry-ab8f0b07-49e4-4677-a1ee-96fb3b5b5baaNormal and Triangular Distributionsmcantor@us.ibm.com110000CH6XactivefalseComment Entriesapplication/atom+xml;type=entryLikestrue2012-07-15T10:48:09-04:002012-07-15T10:48:09-04:00
<div>In my previous couple of blog entries, I used triangular distributions for examples. For many who suffered through (or maybe enjoyed) their stat classes (what are the odds?), this might be a surprising choice. They were taught the default choice would be a Gaussian distribution. For those more attuned with modern business analytics, they are likely to be familiar with triangular distributions. In this entry, I'll briefly the reasoning beyond each of them. </div><div> </div><div>First, as you hopefully recall, both are distributions associate with random variables (Those who don't recall migh benefit from the series of tutorials at <a href="http://www.khanacademy.org/math/probability/v/basic-probability">The Khan Academy</a> site). Each are non-negative functions with integral (area under the curve) one. (There are fancier mathematical definitions, but no matter.) Each describes the likelihood of each of set of possible outcomes of some random variable. The difference in shape between Gaussian (aka Normal) and triangular distributions reflects the nature and use of the random variables.</div><div><span class="Apple-style-span"><br />Briefly, </span><a href="http://en.wikipedia.org/wiki/Gaussian_distribution">normal distributions</a> are often arise as the <a href="http://en.wikipedia.org/wiki/Histogram">histogram</a> of a set of measurements. They have some central value (called the mean) and some dispersion (called standard deviation) around the mean. Anyone who took a stat class studied these distributions. They show up in a many contexts: <br /><br /><ul><li>The distribution resulting from tabulating the histogram of repeated, but imprecise measures of some quantity and then divided the entries by the sum of the measures is often assumed to be normal. The mean of the distribution is the estimator of the actual measure.<br /></li></ul><ul><li> Measurements of natural sets of populations are assumed to be normal. For example IQ tests and their scoring are designed <a href="http://en.wikipedia.org/wiki/Intelligence_quotient">to have mean 100 and standard deviation 15. </a> Note actual measures cannot be assumed to be normal. For example, <a href="http://www.wolframalpha.com/input/?i=distribution+of+height+american+males">the height of American males</a> is sort of normal but skewed to lower heights. </li></ul>Statisticians like the normal distribution for several reasons. First, it is easy to parameterize. If you know the mean. mu (μ), and the standard deviation, sigma (σ), you have completely characterized the distribution. For example, the likelihood of a measurement occurring is often characterized as being within some number of σ's from the mean. Figure 1 shows how this works.</div><div><br /><br /><a '="" href="https://www.ibm.com/developerworks/mydeveloperworks/blogs/RationalBAO/resource/BLOGS_UPLOADED_IMAGES/ScreenShot2012-04-15at10.43.32AM.png
" target="_blank"><img alt="image" src="https://dw1.s81c.com/developerworks/mydeveloperworks/blogs/RationalBAO/resource/BLOGS_UPLOADED_IMAGES/ScreenShot2012-04-15at10.43.32AM.png" style=" display:block; margin: 0 auto;text-align: center; position:relative;" /></a> <br /><br />The likelihood of a value falling in a range is given by the area under the curve. For example, the probability of a value of the normally distributed random variable falling within one standard deviation of the mean is 68.2%. <br /><br />Normal distributions have one really cool feature called the <a href="http://en.wikipedia.org/wiki/Central_limit_theorem">Central Limit Theorem</a>, which states that under remarkably general conditions, the sum of a set of random variables will be close to normal. Notice, in the previous blog entry, when we added two triangular random variables, the sum appeared smooth and in fact started to look normal. <br /><br />All that said, I do have have a pet peeve. Normal distributions are overused. Most things in nature and economics are not normally distributed. For example, as <a href="http://en.wikipedia.org/wiki/Pareto_distribution">as documented in Wikipedia</a>, these phenomena are nowhere near normal, but are closer to a Pareto distribution:<br /><ul><li>The sizes of human settlements (few cities, many hamlets/villages)</li><li>File size distribution of Internet traffic which uses the TCP protocol (many smaller files, few larger ones)</li><li>Hard disk drive error rates</li><li>The values of oil reserves in oil fields (a few large fields, many small fields)</li><li>The length distribution in jobs assigned supercomputers (a few large ones, many small ones)[citation needed]</li><li>The standardized price returns on individual stocks<br /></li><li>Fitted cumulative Pareto distribution to maximum one-day rainfalls</li><li>Sizes of sand particles<br /></li><li>Sizes of meteorites</li><li>Areas burnt in forest fires </li><li>Severity of large casualty losses for certain lines of business such as general liability, commercial auto, and workers compensation.</li></ul><br />Getting back to our topic, let's turn to triangular distributions. They are not used to describe a set of measured outcomes from an experiment. <b>They are used to describe what we know or believe about some unknown random variable.</b> For example, the sales of a new product one year after delivery generally can not be determined by measuring the sales of a bunch of new products. As pointed out by <a href="http://www.hubbardresearch.com/">Douglas Hubbard</a>, treating the future sales as a single fixed variable is unreasonable (although all too common). What is more reasonable is setting the low (L), high (H) , and most likely (E) values of the future sales. As I wrote <a href="https://www.ibm.com/developerworks/mydeveloperworks/blogs/RationalBAO/entry/distributions_explained117?lang=en">in an earlier entry</a>,<a href="https://www.ibm.com/developerworks/mydeveloperworks/blogs/RationalBAO/entry/distributions_explained117?lang=en"> </a> these are the values that specify a triangular distribution. I.e. triangular distributions are set to zero below a given low value, L, and above the high value, H, and peaks at the expected value E. The distribution is then a describe be a triangular curve so that the total area is 1. Here is the distribution for L = 1, E=6, and H=7.</div><div><br /> <a '="" href="https://www.ibm.com/developerworks/mydeveloperworks/blogs/RationalBAO/resource/BLOGS_UPLOADED_IMAGES/ScreenShot2012-04-08at9.23.34AM.png" target="_blank"><img alt="image" src="https://dw1.s81c.com/developerworks/mydeveloperworks/blogs/RationalBAO/resource/BLOGS_UPLOADED_IMAGES/ScreenShot2012-04-08at9.23.34AM.png" style=" width:100%; display:block; margin: 0 auto;text-align: center; position:relative;" /></a></div><div> </div><div>Some would argue there is a 'real' distribution of the future sales random variable and it is unlikely to be triangular. My response is for all practical purposes, it does not matter. The triangular distribution is a good-enough approximation to whatever the real distribution might be. By 'good enough' I mean they may be used to support decision making: they are a big improvement over using single values. They are also practical as they easy to specify and there is no assumption of symmetry, No wonder they are common in business analytics.</div><div><br />To wrap up, normal distributions are occasionally useful to describe outcomes of measurements while triangular distributions are useful for giving rough estimates of one's belief of the liklihood of outcomes based on the evidence on hand. More generally, normal distributions are useful in <i>frequentist</i> statistics and triangular in <i>Bayesian</i> statistics. See <a href="http://en.wikipedia.org/wiki/Probability_interpretations">this Wikepedia article for a discussion of the kinds of statistics.</a> Much of what we do in development analytics is more Bayesian than frequentist. I hope to write more about that in the near future.</div><div> </div><a href="http://en.wikipedia.org/wiki/Intelligence_quotient">
</a>
In my previous couple of blog entries, I used triangular distributions for examples. For many who suffered through (or maybe enjoyed) their stat classes (what are the odds?), this might be a surprising choice. They were taught the default choice would be a...2014298