Visualizing Nested Loops Joins And Understanding Their Implications

Published on: 2018-12-11

Watch this week’s episode on YouTube.

What Physical Join Operators Tell Us

Everyone has their own method of reading an execution plan when performance tuning a slow SQL query.  One of the first things I like to look at are what kind of join operators are being used:

These three little icons may not seem like the most obvious place to begin troubleshooting a slow query, but with larger plans especially I like starting with a quick glance at the join operators because they allow you to infer a lot about what SQL Server thinks about your data.

This will be a three part series where we’ll learn how each join algorithm works and what they can reveal about our upstream execution plan operators.

Nested Loops Join

Nested loops joins work like this: SQL Server takes the first value from our first table (our “outer” table – by default SQL Server decides for us which table of the two this will be), and compares it to every value in our second “inner” table to see if they match. 

Once every inner value has been checked, SQL Server moves to the next value in the outer table and the process repeats until every value from our outer table has been compared to every value in our inner table.

This description is a worst case example of the performance of a nested loop join.  Several optimizations exist that can make the join more efficient.  For example, if the inner table join values are sorted (because of an index you created or a spool that SQL Server created), SQL Server can process the rows much faster:

In the above animation, SQL Server has the inner input data sorted, allowing it to seek directly to the rows it needs, reducing the total number of comparisons that need to be made.

For more in-depth explanations of the internals and optimizations of nested loops joins, I recommend reading this post by Craig Freedman as well as Hugo Kornelis’s reference on nested loops.

What Do Nested Loops Joins Reveal?

Knowing the internals of how a nested loops join works allows us to infer what the optimizer thinks about our data and the join’s upstream operators, helping us focus our performance tuning efforts. 

Here are a few scenarios to consider the next time you see a nested loops join being used in your execution plan:

  • Nested loops joins are CPU intensive; at worst, every row needs to be compared to every other row and this can take some time.  This means when you see a nested loops join, SQL Server probably thinks that one of the two inputs is relatively small.
    • … and if one of the inputs is relatively small, great!  If instead you see upstream operators that are moving large amounts of data, you may have a estimation problem going on in this area of the plan and may need to update stats/add indexes/refactor the query to have SQL Server provide better estimates (and maybe a more appropriate join).
  • Nested loops sometimes accompany RID or key lookups.  I always check for one of these because they often leave room for some performance improvements:
    • If a RID lookup exists, it’s usually easy enough to add a clustered index to that underlying table to squeeze out some extra performance.
    • If either RID or key lookup exist, I always check what columns are being returned to see if a smaller index could be used instead (by including a column in a key/column of an existing index) or if the query can be refactored to not bring back those columns (eg. get rid of the SELECT *).
  • Nested loops joins do not require data to be sorted on input.  However, performance can improve with a sorted inner data source (see animation above), and SQL Server might choose a more efficient operator if the inputs are both sorted. 
    • At the very least, nested loops joins make me think to check whether the input data isn’t sorted because of some upstream transformations, or because of missing indexes.

So while nested loops in your plans will always require more investigation, looking at them and the operators around them can provide some good insight into what SQL Server thinks about your data.

Thanks for reading. You might also enjoy following me on Twitter.

Want to learn even more SQL?

Sign up for my newsletter to receive weekly SQL tips!

Two Words For Faster Scalar Functions

Published on: 2018-12-04

Watch this week’s episode on YouTube.

WITH SCHEMABINDING optimizations were added all the way back in SQL Server 2005.  So why bother talking about them in 2018?

Because no one is taking advantage of them!

Ok, maybe that’s a little unfair.  I’m sure some people advantage of the optimizations, but most code I see posted online that could benefit doesn’t include it.  So let’s talk about an easy way for some of our non-data-accessing scalar functions to get a performance boost.

WITH SCHEMABINDING

When you create a function or view, you can add the WITH SCHEMABINDING option to prevent any database objects that the view/function uses from being modified.  This is a pretty cool feature which prevents you from making a table or column change that would cause a view/function to break.

And while that’s pretty cool functionality on its own, what’s even better is that the SQL Server optimizer can do some pretty cool things when it knows one of your non-data accessing scalar functions is schema bound.

For example, let’s say we have these two functions.  You’ll notice the second one includes the WITH SCHEMABINDING syntax:

When SQL Server executes a function, by default it checks whether the function has any database objects it needs to access.  For our example functions above, this is a waste of time since neither function accesses any data.

The WITH SCHEMABINDING option forces SQL Server to take note at function creation time whether any data access is occurring.  This allows SQL Server to skip that check at run time and allowing the function to execute significantly faster:

Take a look at those CPU times: 1594ms vs 62ms!  Since SQL Server saves a LOT of time by not having to verify the underlying data sources (for a more in-depth explanation of how SQL Server checks whether a function accesses data, I highly recommend reading this StackOverflow answer by Paul White).

WITH SCHEMABDINING also has performance optimization for queries that would normally need to implement Halloween Protection as well.  Halloween Protection essentially prevents SQL Server from modifying the same records more than once and usually implemented by the addition of an spool operator to the execution plan:

In this example, you’ll notice our non-schema-bound function introduces a Table Spool while our second schema-bound function forgoes this addition since SQL Server knows there won’t be any potential for conflict.

In conclusion, if you have a non-data-accessing scalar function you should always add WITH SCHEMABINDING to reap the benefits of SQL Server’s optimizations when available.

Thanks for reading. You might also enjoy following me on Twitter.

Want to learn even more SQL?

Sign up for my newsletter to receive weekly SQL tips!

Brute Forcing SQL Logins and Passwords

Published on: 2018-11-27

Watch this week’s episode on YouTube.

Following up on last week’s post about the different types of SQL injection, this week I want to show how injection can be used to obtain a SQL login and its password.

My goal with today’s post is to show how easy it is for someone to do this to your server.  My hope is that if your security practices aren’t currently the best, the least you can do is learn and follow the few simple steps at the end of this post to help protect your server.

Iterating the Login

Let’s try to guess the current login by iterating over characters one at a time:

The key to this script is line 13 where we use SUBSTRING() to grab the first letter of SYSTEM_USER and check to see if it equals the current letter we are iterating over (if we could perform a union-based attack and return SYSTEM_USER outright, that of course would be easier).

But having the ability to execute a whole script like that to determine the login in milliseconds is a luxury.  If you are actually injecting code into a dynamic query, a more realistic looking attack might look like this (using our vulnerable stored procedure from last week’s demos):

Now if the query takes 5 second to return results, we know we found the correct first letter (“S” in this example).  We can repeat this for each subsequent character until we have the whole login.  Easy.

Note: A similar process can be performed for SQL Users as well.

Cracking the Password

Passwords are more work to obtain.  You could start guessing common passwords (eg. “admin”, “ilovesql”, “”, etc…) and trying to use them to login, but once you deplete the easy-to-guess list, you will have to try something else.

A more systematic method requires you to first obtain the login’s password hash:

Which returns:

That first column is SQL Server’s hashed version of the login’s password.  And while we were able to determine the SQL login character by character, we won’t be able to do the same for this hashed password. 

To “crack the hash”, what we have to do is guess a password, hash it, and see if it is equal to the hash we found above.  If it doesn’t match, we guess another password, hash, check, and repeat until we find the correct password.

The simplest way to do this is to iterate over the entire password space.  The very poorly T-SQL code written below does that for 4 character passwords.  We basically start with guessing a password “!!!!”, checking to see if it’s hash matches, and if not then moving on to the next character combintation (ie. !!!#”, “!!!$”, “!!!%”, etc…):


After about 9 minutes, SQL Server returns my very easy to guess sa password in clear text:

At this point you might be thinking, “9 minutes for a 4 character password!  Doesn’t that mean it would take years to crack an 8 character, or 12 character password?”

Yes and no.  T-SQL is not the right tool for the job here.  A better tool would be something that is optimized for hash checking, something like hashcat.

Hashcat performs the same type of looping logic to check an entire address space for a matching hash, but it does so very quickly with lots of optimizations (word lists, prefix/suffix mutations, etc..).  It also multithreads the process and can make use of graphics cards to perform computations even faster.

I don’t want to turn this into a “step by step how to use hashcat to crack your SQL logins” post, but be aware that doing so is fairly straightforward.  Also know that GPU optimized cloud machines are easily available, and running hashcat on them easily gets you into the 470 million hashes/second range.

With some back of the napkin calculations, that means your completely random 8 character password would take at most:

Now that may seem like a long time, but remember, that’s one machine doing the processing over the entire character space. 

The cloud is easily abused for solving performance problems by spinning up additional hardware.  The same tools you have to run your company’s cloud workloads are available to criminals as well.

And hardware is always improving.  Could new hardware come out in the next couple years that will speed this up even more?  Almost certainly.

Finally this is the longest possible time if would take – if your password isn’t completely random (uses dictionary words) or the hacker knows something about your password schema (all passwords use only letters and numbers), then it’s pretty much game over.

Don’t Let This Happen To You

Hopefully you are using at least 8 character, random passwords with a full subset of upper and lowercase letters, numbers, and symbols.  That should be the minimum you should be doing today, knowing full well that this is already probably inadequate, if not today then in the very near future. 

If you aren’t using long passwords, then stop reading right now and go figure out how to change this right away.  SQL Server allows you to use passwords up to 128 characters long – make use of  all of those characters!

In addition to using long, random passwords, follow the best practices mentioned in last week’s post as well: use the principle of least privilege to restrict as much access to your logins as possible, and prevent SQL injection from allowing important information from leaving your server in the first place.

Thanks for reading. You might also enjoy following me on Twitter.

Want to learn even more SQL?

Sign up for my newsletter to receive weekly SQL tips!