hypothesis testing

Introduction

"Can you check if this is significant?"

It was a seemingly innocuous question from a dangerous source: a semi data-literate scientist. The kind who believed, deep in his heart, that small p-values were "good" and large p-values were "erroneous".

On this day, the man in question had come forth with a large, complex multivariate dataset. He'd manually combed the data, visually inspected it, and hand-picked a hypothesis.

"Can you check if this is significant?"

"I can, but I shouldn't."