{"id":2600,"date":"2026-01-01T00:01:43","date_gmt":"2026-01-01T05:01:43","guid":{"rendered":"https:\/\/mathvoices.ams.org\/featurecolumn\/?p=2600"},"modified":"2026-01-02T13:42:42","modified_gmt":"2026-01-02T18:42:42","slug":"information-insight-and-the-problem-with-parameters","status":"publish","type":"post","link":"https:\/\/mathvoices.ams.org\/featurecolumn\/2026\/01\/01\/information-insight-and-the-problem-with-parameters\/","title":{"rendered":"Information, Insight, and the Problem With Parameters"},"content":{"rendered":"

Could we possibly gain more insight by trading away even more information?<\/em><\/span><\/p>\n

Information, Insight, and the Problem With
\nParameters<\/h1>\n

Anil Venkatesh
\nAdelphi University<\/b><\/p>\n

Introduction<\/h1>\n

I have two data sets for us to consider. Both consist of observations
\nof a single variable. Data Set 1<\/a> holds the daily number of positive
\nCOVID-19 tests in Kings County, NY, since March 2020. Data Set 2<\/a> holds
\nthe LYVE1 level of 535 patients\u2014this is the urinalysis concentration of
\na certain protein that may relate to the metastasis of pancreatic
\ncancer.<\/p>\n

\n\"Two
Line plots of Data Sets 1 and 2.<\/figcaption><\/figure>\n

One of these plots is a lot more informative than the other. When we
\nplot observations from left to right, there\u2019s an implied hypothesis that
\nthe order of the observations has an association with the observations
\nthemselves. This is clearly true in the case of positive COVID-19 tests
\nover time, but it\u2019s clearly false in the case of LYVE1 levels
\nvs.\u00a0patient ID. Even if we can identify a trend in the second plot, it
\ncan\u2019t possibly carry any insight.<\/p>\n

While both data sets ostensibly only measure one variable, the
\nCOVID-19 data set has an implicit time-ordering that allows us to
\nvisualize it in two dimensions. With no second variable in the
\nurinalysis data set, we\u2019re stuck building a one-dimensional scatter plot
\n(is that a mathematical koan?) We can certainly mark a number line at
\neach value that\u2019s contained in the data set, but let\u2019s extend these
\nmarks into vertical lines for readability.<\/p>\n

\n\"A
Novelty plot (or one-dimensional scatterplot) of the
\nurinalysis data set.<\/figcaption><\/figure>\n

While this visualization of LYVE1 levels no longer implies a spurious
\nassociation with patient ID, it hasn\u2019t greatly improved our insight
\nabout the data. As you look at this barcode-like plot, you can\u2019t help
\nbut allow some of the lines to run together into blobs. Aggregating like
\nthis does lose some information, but it allows you to gain a little
\ninsight about the ranges of values of LYVE1 that are most commonly
\nrepresented in the data set. In jargon:\u00a0aggregating nearby observations
\ngives an approximate measurement of the density function<\/em> of the
\nLYVE1 variable. Could we possibly gain more insight by trading away even
\nmore information?<\/p>\n

\n\"Histograms
Histogram representations of LYVE1 (various bin
\nwidths).<\/figcaption><\/figure>\n

A histogram<\/em> does exactly this:\u00a0break up the axis into bins
\nof equal width, then count the number of observations in each bin and
\nplot these counts. The choice of bin width has a profound effect on the
\nvisualization and attendant insights (or lack thereof). While widening
\nthe bins can smooth out the random jiggles of the observations, when
\ntaken too far it inevitably smooths out the entire data set into a blob.
\nAs we saw before, narrowing the bins too aggressively results in a
\nnearly useless barcode pattern. Is there such thing as a perfect bin
\nwidth? Students of calculus will recall that any continuous function on
\na closed and bounded interval attains a maximum somewhere in that
\ninterval; here, we wish to know if the \u201cinsightfulness\u201d of a histogram
\n(holistically construed) can be maximized by some certain bin width
\nbetween 0 and the range of the data. But is this \u201cinsightfulness\u201d a
\ncontinuous function of bin width? And if so, can we define it concretely
\nenough to solve the optimization problem?<\/p>\n

Much research has been done on the optimal bin width question. As the
\ndefault option in Microsoft Excel, Scott\u2019s normal reference rule is
\nsurely the most ubiquitous approach. The rule prescribes a bin width
\nproportional to $n^{1\/3}$, where $n$ is the size of the data set, and
\nactually gives an exact constant of proportionality as well. The
\n\u201dinsightfulness\u201d function in Scott\u2019s rule is the integrated mean
\nsquare error<\/em> between the histogram and the true distribution of the
\ndata\u2026or at least, a normal distribution that most closely matches the
\ndata. Other popular rules make similar assumptions about the properties
\nof the data distribution, sometimes even recovering the $n^{1\/3}$ rule but with a
\ndifferent constant of proportionality. In all these cases, the author
\nproposed a working definition of insightfulness and then made enough
\nassumptions about the data in order to reach a closed-form rule for bin
\nwidth. Unfortunately, different assumptions arrive at different
\nformulas! This means that we are still stuck making a choice.<\/p>\n

The Problem With
\nParameters<\/h1>\n

A mathematical model that requires the user to choose a value (like
\nbin width) is called a parametric<\/em> model, i.e.\u00a0involving one or
\nmore undetermined parameters. Every time we choose the value of a
\nparameter, we influence the model and potentially change the nature of
\nthe insight that it provides. This means that every parametric model has
\nthe user\u2019s thumb on the scale\u2014a problem if we want to gain unbiased
\ninsight from data. The fewer parameter values we have to pick, the less
\nour actions will bias the results, right? In what circumstances can we
\nopt out of choosing a parameter value entirely?<\/p>\n

Before we go any further, I have some bad news about the parameter
\nspace of histograms. While we\u2019ve been doing our best to pick a good bin
\nwidth, there\u2019s been another parameter lurking in the shadows. What about
\nsliding all the bins left or right? This \u201cbin offset\u201d parameter can
\nclearly affect the visualization as observations hop from one bin to
\nanother. One option is to place the bins arbitrarily and hope that the
\nresult is good enough, but let\u2019s consider a way of entirely opting out
\nof this parameter choice. With apologies to P\u00f3lya<\/a>, I humbly
\npropose:\u00a0When you have two things to say, instead say infinitely
\nmany things and take the average.<\/em><\/p>\n

\n\"The
Histogram of LYVE1 together with its offset-averaged
\ncurve.<\/figcaption><\/figure>\n

By averaging histograms across all possible bin offsets, we
\nsuccessfully correct for much of the edge effects inherent to the
\nhistogram. If a cluster of observations happens to land right on the
\nedge of a bin, this may substantially distort the histogram but it won\u2019t
\nupset the offset-averaged curve. While we built this curve by thinking
\nabout sliding the histogram bins around, it can be constructed
\nequivalently by defining a triangular function of height 1 and base of 2
\nbin widths, then centering a copy of this function at each data point
\nand adding everything up (that\u2019s a nice exercise to work out!)
\nFormulated this way, we\u2019ve actually described an example of kernel
\ndensity estimation<\/em> (KDE). A kernel<\/em> is a family of
\nfunctions that smoothly transitions from an infinitesimally narrow spike
\nto a wide, flat form. Generally, we want the integral of the kernel to
\nremain constant throughout this transition. This makes the kernel a
\nmodel of diffusion of mass or heat from a point source to the
\nsurrounding environment.<\/p>\n

\n\"Isosceles
Five members of a triangle kernel family of mass
\n1.<\/figcaption><\/figure>\n

In kernel density estimation, we center an identical kernel at each
\ndata point and then allow all the kernels to begin transitioning,
\nintermingling with each other as they collectively disperse their mass.
\nWhen configured well, KDE yields smooth, highly interpretable density
\ncurves. It is implemented by Python\u2019s seaborn library as a standalone
\nfunction and an optional argument when forming a histogram, and by a
\nbuilt-in function in R. As the underlying concept of heat diffusion
\nworks in any ambient dimension, KDE can also be used on multidimensional
\ndata and is particularly good for heatmap visualizations over two
\npredictor variables.<\/p>\n

\n\"Six
Kernel density estimates of LYVE1 (various diffusion
\ntimes).<\/figcaption><\/figure>\n

Depending on how long we allow the diffusion process to run, a
\nvariety of KDE models can result (and they\u2019re suspiciously reminiscent
\nof the variety of histograms we saw before.) Clearly, it all comes down
\nto when we choose to hit \u201cpause.\u201d If we stop too soon, the kernels will
\nnot have had enough time to intermingle and a sparse, jagged form will
\nresult. If we stop too late, the kernels will have distributed their
\nmass into one big blob (or even worse, a big flat pancake). Just as with
\nbin width optimization, there is a body of literature on the optimal
\ndispersion time in KDE. However, another more serious problem
\narises:\u00a0the optimal dispersion time depends on the local density of the
\ndata. In portions of the data set with many closely-packed data points,
\nthe kernels intermingle rapidly to form one dense clump; by comparison,
\nthe kernels of isolated data points take a lot longer to intermingle
\nwith the others. Generally speaking, dispersion time should be longer
\nwherever the data set is sparse, and shorter wherever the data set is
\ndense. But\u2026I thought the whole point of KDE was to estimate the density
\nof the data! If the optimal KDE algorithm requires prior knowledge of
\nthe density of the data, then what are we even doing here?<\/p>\n

\n\"The
Illustration of mixing time depending on local
\ndensity.<\/figcaption><\/figure>\n

Each row of this heatmap is a stage of KDE, normalized by its maximum
\nvalue. Reading from the bottom, we see how the white (high-density)
\nkernels disperse and gradually merge with surrounding regions. Note that
\ntight clusters of kernels merge together much faster than isolated
\nkernels. This effect is particularly pronounced in data sets with many
\nrepeated values, a common feature of medical and sociological data
\nbecause of the prevalence of ordinal, non-numeric attributes
\n(e.g.\u00a0Likert scale, Apgar score).<\/p>\n

In Search of
\nInsight<\/h1>\n

In principle, the optimal KDE model might need a distinct
\ndiffusion-rate parameter for each data instance<\/em>. What started
\nas an effort to remove an unwanted parameter has now saddled us with
\nmore parameters than we know what to do with! This brings us all the way
\nback to the humble histogram with its bin width parameter, together with
\na light touch of KDE to smooth out the offset bias. If there\u2019s a moral
\nhere, it\u2019s this:\u00a0to gain insight, you must lose information.<\/p>\n","protected":false},"excerpt":{"rendered":"

Could we possibly gain more insight by trading away even more information? Information, Insight, and the Problem With Parameters Anil Venkatesh Adelphi University Introduction I have two data sets for us to consider. Both consist of observations of a single variable. Data Set 1 holds the daily number of positiveRead More →<\/a><\/span><\/p>\n","protected":false},"author":2,"featured_media":2037,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"advanced_seo_description":"","jetpack_seo_html_title":"","jetpack_seo_noindex":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_post_was_ever_published":false},"categories":[213,109,25],"tags":[214,215,153],"class_list":["entry","author-uwhitcher","post-2600","post","type-post","status-publish","format-standard","has-post-thumbnail","category-213","category-anil-venkatesh","category-probability-and-statistics","tag-data-visualization","tag-kernel-density-estimation","tag-statistics"],"jetpack_featured_media_url":"https:\/\/i0.wp.com\/mathvoices.ams.org\/featurecolumn\/wp-content\/uploads\/sites\/2\/2024\/07\/cropped-FC1380x500x2.png?fit=1380%2C288&ssl=1","jetpack_sharing_enabled":true,"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/mathvoices.ams.org\/featurecolumn\/wp-json\/wp\/v2\/posts\/2600","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mathvoices.ams.org\/featurecolumn\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mathvoices.ams.org\/featurecolumn\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mathvoices.ams.org\/featurecolumn\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/mathvoices.ams.org\/featurecolumn\/wp-json\/wp\/v2\/comments?post=2600"}],"version-history":[{"count":6,"href":"https:\/\/mathvoices.ams.org\/featurecolumn\/wp-json\/wp\/v2\/posts\/2600\/revisions"}],"predecessor-version":[{"id":2615,"href":"https:\/\/mathvoices.ams.org\/featurecolumn\/wp-json\/wp\/v2\/posts\/2600\/revisions\/2615"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/mathvoices.ams.org\/featurecolumn\/wp-json\/wp\/v2\/media\/2037"}],"wp:attachment":[{"href":"https:\/\/mathvoices.ams.org\/featurecolumn\/wp-json\/wp\/v2\/media?parent=2600"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mathvoices.ams.org\/featurecolumn\/wp-json\/wp\/v2\/categories?post=2600"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mathvoices.ams.org\/featurecolumn\/wp-json\/wp\/v2\/tags?post=2600"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}