Our Review Methodology: Whiskey and Spirits

Transparency, full disclosure, and honesty. Those are the values that I brought to hunting and outdoor sporting equipment reviews for the last seven years, and I’m bringing those same values to the whiskey and spirit reviews we post on this site.

We’ve developed a standard format for spirit reviews. I wanted to take a moment to discuss the different sections of that review format, their purpose, and how they impact the final rating.

History

Some whiskey producers have a long and storied past, with master distillers who have passed down their secrets from generation to generation. Other producers are brand new, striking out for the first time and trying to do something new and interesting. Either way, the whiskey they produce is a reflection of the story of their company. By understanding that story, it can sometimes put the whiskey in a better context, allowing us to better appreciate the product.

The goal here is to identify when the whiskey producer started operating and tell an abridged version of their story. Sometimes we’ll even go a little further back, like in the case of Belle Meade where it’s the second incarnation of a historical brand.

Product

Sometimes a whiskey is just a whiskey. There are plenty of mass produced spirits that are delicious and drinkable. But there’s also a strong trend in the spirits industry of “small batch” produced spirits made in a very specific fashion.

In this section of our reviews, the goal is to describe exactly how the spirit is made, whether that’s a well polished machine churning out a consistent alcohol or a small batch production using novel methods.

This is usually more important for the “small batch” labeled spirits, to identify exactly what earns them that “small batch” distinction. Sometimes it’s nothing at all — Colville is a great example of a whiskey that claims to be “small batch” production but actually comes from a large facility and is simply bottled and branded as if it were from a boutique producer.

Not every bottle comes from a single barrel. Sometimes there’s a blend of whiskeys that are hand picked to create the bottled spirit, whether that’s barrels that the distillery themselves produced or ones they have shipped in. In those cases we try to figure out the source materials, mainly so we can better understand where the flavors come from and the quality of the spirit being used.

Here at Thirty One Whiskey, we’re perfectly OK with mass produced whiskies that are bottled by a company that isn’t the distiller. We’re okay with blends of different distilleries. But the one thing that irks us is when a whiskey claims to be something that it isn’t — either it dresses itself as a local craft spirit but is actually produced in a large facility elsewhere without telling the consumer, or it’s claims to be produced somewhere when it reality it’s only bottled there… we could go on with examples, but you get the idea. Own your story, own your ingredients, and you’ll get a thumbs up from us.

Packaging

Your parents probably told you that it’s what’s on the inside that counts. And they’re right, at the end of the day the contents of the bottle are more important than the bottle itself. But it is a factor that influences people’s buying decisions — and with an increasing competitive market, we have to be honest that shelf appeal can be a huge factor in the success of a spirit.

There are also some rare instances where the design of the bottle actually negatively impacts the consumption of product, like the Firestone & Robertson whiskey where the canvas collar actually gets in the way of the whiskey when being poured.

Neat

This is probably the biggest test of the whiskey, and where we spend the most time.

Step one is to take a good whiff of the spirit. Swirl it a bit in the glass, observe the color and consistency, and try to describe the aroma coming off the whiskey.

Then, after much consideration, we take a sip. And a few more. We’re looking for the weight and consistency of the liquid (usually correlated with alcohol content). How smooth the taste is, looking for any bitterness or unpleasant flavors. And we try to identify as many flavors as possible.

Usually we start this process “in a vacuum” — without any tasting notes from the distiller or other locations. We’ll then refer to the notes from others and indicate if there are any flavors that we picked up on after reading those other reviews, or if there’s anything that we didn’t identify but others might have.

On Ice

One of the easiest and most popular presentations for a spirit is “on the rocks” with a little bit of ice. Typically the purpose of the ice is to improve the taste of the spirit through dilution (thanks to the melting ice) and through chilling the liquid (as Alton Brown points out, chilling things tends to remove or mask many unpleasant flavors).

Sometimes ice can save an otherwise terrible spirit. Other times it can destroy or mask the flavors that make a spirit remarkable. We intend to find out.

Other Preparations

What we’re usually looking for here is whether we can improve the spirit. If the flavor wasn’t great when tasted neat or on the rocks, is there anything we can do to make it drinkable?

Another question we’re often trying to answer is whether the spirit is flavorful enough to be heard through the noise of a mixed or built drink. Are you gaining anything by adding this specific spirit compared to a neutral grain spirit or a vodka, or is it just alcohol for the sake of alcohol?

The two preparations that we usually do are an Old Fashioned (muddled sugar and orange bitters with an orange slice garnish) and a Mule (a splash of lime juice and ginger beer). The fact that these are my favorite mixed drinks makes it easy to perform the strenuous testing, but it’s also a good challenge for the spirits. One is a simple cocktail where the spirit is still the star, the other is a bold flavored mixer that tends to overpower the more delicate flavors. A spirit that does both well is one that would work in nearly any other mixed drink.

I’ll make a note here that we don’t do additional preparations for certain spirits where it isn’t necessarily appropriate. For example, we don’t make a fine aged single malt scotch whisky into a Scottish Mule. That’s not what it was designed to do, and we don’t think it’s a fair test. Especially given the way we compile ratings (which we’ll get to in a moment).

Overall Rating

The single question that we are trying to answer in this review:

Is this spirit better or worse than similarly priced and marketed spirits in their category and class?

It’s not fair to compare a rum to a scotch, or a vodka to a cognac. And it’s not necessarily fair to compare a $100 whiskey with a $20 whiskey. They’re built and marketed for obviously different circumstances.

Generally we’ll have a “reference spirit” picked out that embodies our expectations for what a spirit in that price range and style should be. For bourbon, that’s Bulleit Bourbon. That is our benchmark for the very definition of a three star “average” review.

If a bottle sells for more than that reference spirit, then it needs to be proportionately more delicious just to maintain that three star rating. If it maintains that quality for less money, it gets a better rating. Similarly, better tasting spirits at the same price point also get better ratings.

So, the rating scale should be viewed as follows:

  1. Poor quality for the price and category
  2. Below average quality for the price and category
  3. Average quality for the price and category
  4. Improved quality for the price and category
  5. Outstanding quality for the price and category

As you can see, a three star rating is not necessarily a bad thing. It just means that the spirit met expectations and is worth the sticker price.

The final rating is not a mathematical combination of the quality from the tasting segments. A whiskey that does poorly on three of four areas and excels in one specific application can still be an above three star spirit. It just depends how, in our opinion, things balance out.

And, in the end, it really is our opinion. While we will try to present the facts of the spirit as truthfully as possible, and remain as objective as possible, the final rating is based on our subjective (read: human) opinion. You are welcome to have your own, even if it disagrees with ours. Actually, please do, just make sure to comment and tell us your take! Because a diversity of opinions on whiskey is what makes this whole process interesting, after all.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.