Calibration in wine-tasting

A couple months ago, I ran across an article about wine-tasting that I promptly lost and have not been able to find again. But it made some interesting points about calibration, so—as part of the current series on measurement—I'll try to reproduce the gist of it here. Since I can't find the article I can't give you a footnote to substantiate the factual claims I make about wine; but I think you'll agree that they are mostly common sense.

We all know that there is a difference between Good Wine and Bad Wine, and also that Good Wine generally costs more. But this article suggests that we recognize at least three levels: Terrible Wine, Good-Enough Wine, and Great Wine. And the differences between these levels are revealing.

As you climb from Terrible Wine to Good-Enough Wine, the price goes up by a bit but generally not by a lot. At the same time, the overall quality improves dramatically. Most wine drinkers can tell the difference between Terrible Wine and Good-Enough Wine.

But when you then climb from Good-Enough Wine to Great Wine, the variables shift. With this step the price may shoot up much higher. The wine gets a lot better too, but what is interesting is that not all wine-drinkers can taste the difference. More precisely, anyone can tell that the Great Wine doesn't taste quite the same as the Good-Enough wine. But unless you have a trained palate, you may not be able to distinguish the subtleties that make this bottle worth ten times as much as that bottle. Even so, those subtleties really do exist. But it generally takes a trained palate to recognize them.

What does this have to do with calibration? Everything.

In wine-tasting, your palate is the measuring instrument; the wine is the object to be measured; and its quality is the dimension in question. And the point is that the measuring instrument—your palate—has to be calibrated to meet the requirements of the measurement. But this calibration is of two kinds. 

• On the one hand, you want to make sure no one is leaning on the scale; or in other words, that the measuring instrument reads zero when the inputs are (in fact) zero. 
• On the other hand, you want to make sure that your measuring instrument is capable of the readings you need. If you need nanometer precision, don't use a yardstick. But if you are measuring carpet, don't use a nanomeasuring machine.

These principles apply exactly to the measurement of wine. 

• The first requirement—that your palate should read zero when you aren't tasting anything—means that you shouldn't be distracted by other flavors. You can achieve this by taking a bite of something with a neutral flavor before sipping your wine.* 
• The second requirement means that your palate has to be trained to match the use case you have in mind. 
• If all you need is to find a table wine that will complement your hamburger or your Halloween candy,** you have to be able to tell the difference between Terrible Wine and Good-Enough Wine. And for that use case, a greater sensitivity might be wasted. 
• 
  On the other hand, if you are judging premium wines at the highest level—or if you are trying to re-create Alexandre Dumas's experience drinking Montrachet***—well, for that you need both sensitivity and training.

Once again, as always, what you need all depends on what you are trying to do.   

__________

* Note, for example, the care with which the Comte de Rueil offered his guests olives between each course to cleanse their palates before tasting the wine, in Dorothy Sayers, "The Bibulous Business of a Matter of Taste," in Lord Peter (New York: Harper & Row, pp.154-167.)   

** Yes, this is really a thing! See for example this blog post from October 2022.

*** Dumas once declared that Montrachet should be drunk only “on bended knee, with head bared.” It is supposed to be the best white wine in the world, or one of them.

   

Oops, my bad—tools CAN'T always calibrate each other!

I goofed last week. I said you can have two tools calibrate each other, and I didn't put any restrictions around that. I was wrong.

You remember the whole topic was whether two tools can calibrate each other. The question I asked was this: "Suppose you calibrate some of your own tools in-house, instead of sending them out. And suppose that when you calibrate Tool-1, you use Tool-2. Normally there's nothing wrong with that, so long as Tool-2 itself is also correctly calibrated. But back when you calibrated Tool-2, you did it using Tool-1. Is that a problem?"

I argued that you can do this, within the parameters of quality system standards like ISO 9001 and ISO 17025. And right away I got helpful feedback from commenters on LinkedIn telling me, "Not so fast!"

Christopher Paris pointed out a technical issue I had forgotten. When I described the calibration history of Tool-1 and Tool-2, I traced them both back to the day you bought them from the manufacturer. I assumed you got a certification from the manufacturer at that point. But Chris observed "that the original calibration certificate from the manufacturer is rarely traceable to national/international standards. It's typically some basic certificate that doesn't really provide much information. So tracking back to that doesn't get you full compliance to ISO 9001. If the OEM's cert doesn't list traceable standards used to calibrate the device, then the device still has to be subject to a third-party lab or some other traceable calibration."

So yes, I accept that correction. Tool-1 and Tool-2 both have to be calibrated at the beginning in a way that is traceable to the correct international standards.

But what about the part where you then use the tools to support each other? What about the way that they leapfrog one another on and on into the future forever?

Scott Kruger and John Schultz each flagged this as an improper use case, and we had some helpful discussions about pragmatic topics to clarify why. But I wanted chapter-and-verse. If this is a bad practice, it should be forbidden by the relevant standards—either that, or there's a hole in the standards and someone is going to exploit it.

I finally found it, but I had to dig. ISO 17025, clause 6.5.1, states: "The laboratory shall establish and maintain metrological traceability of its measurement results by means of a documented unbroken chain of calibrations, each contributing to the measurement uncertainty, linking them to an appropriate reference." [Emphasis mine.]

Let's apply this to my thought experiment in last week's post. Here's what I wrote then.

Remember that when you calibrate any tool, that measurement typically has a validity period. Maybe it's valid for one year. So let's say you calibrate Tool-1 every January, and that calibration is good from January to December. Then you calibrate Tool-2 every July, and that calibration is good from July to June.

Last month was July 2023. Time to calibrate Tool-2.

So you pull out Tool-1. Is it a valid tool to use? Check the sticker. It was calibrated in January 2023, and is good through December 2023. So it must be good to use.

But wait. Let's check the paperwork to make sure. According to the paperwork, when we calibrated it in January 2023 we used Tool-2. Hold on! Isn't that the same tool we're trying to check right now?

No. It's not.

The tool we are trying to check "right now" (meaning last month, when I've set this story) is "Tool-2-as-of-July-2023." The tool we used last winter back when we were calibrating Tool-1 was "Tool-2-as-of-January-2023." If you look at it right those should count as different tools,....

Stop right there.

What I should have seen is that as soon as I treat "Tool-2-as-of-July-2023" and "Tool-2-as-of-January-2023" as different tools, I've got a problem. What does Tool-2's unbroken chain of calibrations look like?

July 2023: Tool-2-as-of-July-2023 was calibrated by Tool-1-as-of-January-2023.

January 2023: Tool-1-as-of-January-2023 was calibrated by Tool-2-as-of-July-2022.

July 2022: Tool-2-as-of-July-2022 was calibrated by Tool-1-as-of-January-2022.

January 2022: Tool-1-as-of-January-2022 was calibrated by Tool-2-as-of-July-2021.

And so on.

Every single one of those measurements introduced an uncertainty. Maybe it was small, but it was there.

Just to make things simple, let's pretend the additional uncertainty is the same each time. (In real life, it might not be.) Call that uncertainty ε. [That's a Greek epsilon.] Then every year that you play this leapfrog game, the uncertainty for each tool increases by 2ε (adding one in January and one in July). If you've been using Tool-1 and Tool-2 to calibrate each other for ten years, you have added 20ε to the uncertainty of each one. 

When does that accumulated uncertainty become too much? At what point does it make the tool worthless?

It all depends what you normally use your tools for. How small an uncertainty do you need? Maybe if the tools start off a lot better than you actually need, you can get away with it for a while. But you must need some level of precision and accuracy, or you wouldn't bother calibrating your tools at all. And you probably didn't spend the extra money to get tools that were 100x more precise and accurate than you really needed. So you can't really play this game too long. Certainly not forever.

As I say, I goofed. I was wrong, and I'm grateful for the corrections. Thank you, all. 

           

Tools that calibrate each other

Here's a question that puzzled me the first time I saw it: Can you have two tools, each of which is used to calibrate the other? 

More precisely: Suppose you calibrate some of your own tools in-house, instead of sending them out. And suppose that when you calibrate Tool-1, you use Tool-2. Normally there's nothing wrong with that, so long as Tool-2 itself is also correctly calibrated. But back when you calibrated Tool-2, you did it using Tool-1. Is that a problem?

The first time I saw this, it bothered me. It looked like, "I'll tell them you're an expert; and then if they want to know what gives me the standing to say so, you tell them I'm an expert. Since they already know that you're an expert (because I just said so) they should trust your judgement. Right?"

And in a static world, I might have had an argument. But I forgot about time. 

Remember that when you calibrate any tool, that measurement typically has a validity period. Maybe it's valid for one year. So let's say you calibrate Tool-1 every January, and that calibration is good from January to December. Then you calibrate Tool-2 every July, and that calibration is good from July to June.

Last month was July 2023. Time to calibrate Tool-2.

So you pull out Tool-1. Is it a valid tool to use? Check the sticker. It was calibrated in January 2023, and is good through December 2023. So it must be good to use.

But wait. Let's check the paperwork to make sure. According to the paperwork, when we calibrated it in January 2023 we used Tool-2. Hold on! Isn't that the same tool we're trying to check right now?

No. It's not.

The tool we are trying to check "right now" (meaning last month, when I've set this story) is "Tool-2-as-of-July-2023." The tool we used last winter back when we were calibrating Tool-1 was "Tool-2-as-of-January-2023." If you look at it right those should count as different tools, because tools drift over time. We all know that tools drift. That's why calibration has to be repeated. That's why there is a validity period in the first place!

Sure enough, if you keep following the paperwork back in time, you'll find the two tools leapfrogging each other. Each time you use this one to calibrate that one, the one you are using turns out to be a legitimately calibrated tool because it's only six months into its 12 month validity period. And when you trace them far enough back in time you find the day that each tool was purchased from the manufacturer, at which point it came with some kind of certification and guarantee.

It took me a while to figure it out, but I'm sure this is the answer.

          

Accuracy and precision

Calibration is one of the basic methods in any Quality Management System, but for years my understanding of it was not deep. Of course I knew it was important. If you are making a product that requires precision measurements, but your measuring tools aren't calibrated, you have no guarantee that your measurements are right. Concretely, if you need a part to be 0.250"± 0.001" but your tool is off and the part is actually (let's say) 0.234" instead, it's not going to fit. So yes, it matters. 

This is why clause 7.5.1 of ISO 9001:2015 requires that you figure out whether you need calibrated equipment; and then, if you do, that you calibrate it. And in all the years that I did internal audits, I made sure to turn over any measuring equipment that was actually in use to check the calibration stickers.

But then I got a chance to work for a calibration laboratory, and I began to appreciate the huge amount of mathematical theory that underlies the whole job of calibration. I didn't work there long enough to be able to regenerate all the calculations on my own from first principles. But I did learn some of the more important concepts.

One of these is the distinction between accuracy and precision. Those both sound like good things, and of course they are. But they are different things, and the exact nature of the difference matters.

Whenever you measure something, you get a reading of some kind. But you can never be sure that the reading is exactly right. In order to make sure that the reading is as good as possible, you want to ensure two different things:

On the one hand, you don't want your measuring tool to bounce around. You want it to give you the same answer every time you measure the same thing. I've got a scale in my kitchen that isn't very good at this. If I set a weight on it all at once (like a bag of onions) the needle bounces up to a certain reading. If I add the same weight gradually (for example, by pouring in rice until I get the right amount) the needle tends to stick on the way up; then I'll add a little more and it jumps up to a higher number. This scale is good enough to make dinner with, but I would never dream of using it in a production facility to build product.

In calibration terminology, my kitchen scale is not precise: I can put the same weight on it and get two different readings, depending whether I add the weight all at once or gradually. A precise scale would give the same reading for the same weight no matter what.

But precision is only half the battle. I remember a professor of mine once told about visiting the NIST lab that stored the nation's first official atomic clock, which the tour guide called "the most perfect clock in the world." And my prof saw that someone had set the hands to the wrong time. As an atomic clock, it was more precise than any other clock in the country just then. But on that afternoon it wasn't accurate, because the hands were set wrong. It was telling the wrong time, but it told that wrong time with unequaled precision. 

In other words, precision means that you get the same measurement each time. Accuracy means that the measurement you get is correct. You need them both.

You can have accuracy without precision. That's like my kitchen scale: there's a lot of fluctuation in the readings, but when nothing is on the scale it reliably shows zero. It's not set wrong. There's just room for error when you weigh something on it.

And you can have precision without accuracy. That's like the atomic clock on the day my prof joined the tour.

A quick search on the Internet turns up dozens of pictures that all show the difference in basically the same way. Here's one, for example.

Or, if you prefer, Randall Munroe explained the difference this way in his webcomic xkcd.