Suggesting Statistical Significance
"Many web analytics studies are challenged to produce useful results because there’s often substantially more natural variation in the human behavior being studied and in the resulting data than the size of the effects that the studies are trying to detect."
Practicing What We Preach
One of the rarest of breeds, an affable statistician, Kevin sums up one of the main challenges facing online measurement today. The issue of statistical significance touches all methods of online measurement and influence.
While it is possible to segment the data in potentially innumerable directions, do those cuts add insights or do they add error?
After interviewing statisticians, experienced veterans and academics regarding this topic I wanted to start a conversation on the larger topic of quantitative analysis amongst practictioners. This is the first post of an ongoing series which will, with luck, frame a conversation about the future of online measurement and what it means for the ewb analyst in their day to day work.
For me, the discussion of the understanding of statistics, including econometrics as a closely related discipline, should start with one of the most memorable illustrations of why the basics are always important.
Dr. Stephen Jay Gould wrote his famous ‘The Median Isn’t The Message’ essay after being diagnosed with cancer; the essay is something that I re-read to keep myself from overreacting to this or that variation which appears. Dr. Gould’s allusions add depth to his writing, which is a large reason why I have always found his work to be memorable. However, after recently sharing the essay with a friend of mine I realized that the same allusions may present a barrier to individuals who learned English as a second language or who may have a different cultural experience.
Therefore I have produced an ‘Annotated – The Median Isn’t The Message’ version of the essay with numerous references called out: defining vocabulary; links to articles about the subjects mentioned; and links to the full quotes.
Read the ‘Annotated – The Median Isn’t The Message’ essay here.