Data with Bert logo

Building Dynamic Table-Driven Queries

MJ-t-sql-Tuesday

This post is a response to this month's T-SQL Tuesday #104 prompt by me! T-SQL Tuesday is a way for SQL Server bloggers to share ideas about different database and professional topics every month.

This month's topic is asking what code would you hate to live without?


Watch this week's video on YouTube

When given the choice between working on new projects versus maintaining old ones, I'm always more excited to work on something new.

That means that when I build something that is going to used for years to come, I try to build it so that it will require as little maintenance as possible in the future.

One technique I use for minimizing maintenance is making my queries dynamic.  Dynamic queries, while not right for every situation, do one thing really well: they allow you to modify functionality without needing a complete rewrite when your data changes.  The way I look it, it's much easier to add rules and logic to rows in table than having to modify a table's columns or structure.

To show you what I mean,let's say I want to write a query selecting data from model.sys.database_permissions:

SELECT class
      ,class_desc
      ,major_id
      ,minor_id
      ,grantee_principal_id
      ,grantor_principal_id
      ,type
      ,permission_name
      ,state
      ,state_desc
  FROM model.sys.database_permissions

Writing the query as above is pretty simple, but it isn't flexible in case the table structure changes in the future or if we want to programmatically write some conditions.

Instead of hardcoding the query as above, here is a general pattern I use for writing dynamic table-driven queries.  SQL Server has the handy views sys.all_views and sys.all_columns that show information about what columns are stored in each table/view:

2018-07-03_21-00-45

Using these two views, I can use this dynamic SQL pattern to build the same exact query as above:

-- Declare some variables up front
DECLARE 
    @FullQuery nvarchar(max),
    @Columns nvarchar(max),
    @ObjectName nvarchar(128)

-- Build our SELECT statment and schema+table name
SELECT 
    @Columns = COALESCE(@Columns + ', ', '') + '[' + c.[name] + ']',
    @ObjectName = QUOTENAME(s.name) + '.' + QUOTENAME(o.name)
FROM 
    sys.all_views o  
    INNER JOIN sys.schemas s
        ON o.schema_id = s.schema_id
    INNER JOIN sys.all_columns c
        ON o.object_id = c.object_id
WHERE 
    o.[name] = 'database_permissions'
ORDER BY
    c.column_id 

-- Put all of the pieces together an execute
SET @FullQuery = 'SELECT ' + @Columns + ' FROM ' + @ObjectName

EXEC(@FullQuery)

The way building a dynamic statement like this works is that I build my SELECT statement as a string based on the values stored in my all_columns view.  If a column is ever added to this view, my dynamic code will handle it (I wouldn't expect this view to change that much in future versions of SQL, but in other real-world tables I can regularly expect changing data).

Yes, writing certain queries dynamically like this means more up front work.  It also means some queries won't run to their full potential (not necessarily reusing plans, not tuning every individual query, needing to be thoughtful about SQL injection attacks, etc...).  There are A LOT of downsides to building queries dynamically like this.

But dynamically built queries make my systems flexible and drastically reduce the amount of work I have to do down the road.  In the next few weeks I hope to go into this type of dynamically built, table-driven process in more detail (so you should see the pattern in the example above get reused soon!).

Code You Would Hate To Live Without (T-SQL Tuesday #104 Invitation)

MJ-t-sql-Tuesday

The recent news about Microsoft acquiring GitHub has me thinking about how amazing it is for us to be part of today's online code community.

Before modern online programming communities, finding good code samples or sharing your own code was challenging.  Forums and email lists (if searchable) were good, but beyond that you had to rely on books, coworkers, and maybe a local meetup of like-minded individuals to help you work through your programming problems.

Watch this week's video on YouTube

Today, accessing and using code from the internet is second nature - I almost always first look online to see if a good solution already exists.  At the very least, searching blogs, GitHub, and StackOverflow for existing code is a great way to generate ideas.

For this month's T-SQL Tuesday, I want you to write about code you've written that you would hate to live without.

Maybe you built a maintenance script to free up disk space, wrote a query to gather system stats for monitoring, or coded some PowerShell to clean up string data.  Your work doesn't need to be completely original either - maybe you've improved the code in some open source project to better solve the problem for your particular situation.

There's probably someone out there in the world who is experiencing the same problem that you have already solved; let's make their life a little easier by sharing.

And don't worry if your code isn't perfect - just explain how your solution works and if you are aware of any caveats.  If it's not an exact solution for someone else's problem, at the very least it may help them generate some ideas.

Finally, here's a reminder of the official rules for T-SQL Tuesday:

  1. Publish your contribution on Tuesday, July 10, 2018. Let's use the "it's Tuesday somewhere" rule.
  2. Include the T-SQL Tuesday Logo and have it link to this post.
  3. Please comment below with a link to your post (trackbacks/pingbacks should work too but...comments ensure I don't miss your post)
  4. Tweet about your post using #tsql2sday.
  5. If you'd like to host in the future, contact Adam Machanic.

Filtering Tables, Procedures, And Other Objects In SSMS

Watch this week's video on YouTube

While the text of this post contains good information on SSMS object filters, I highly recommend watching this week's video on YouTube - I stretched my creativity with filming this week while I was on vacation.

This week I'm sharing a trick I learned at Jess Pomfret's Ohio North SQL Server User Group presentation on data compression.  Her presentation on data compression was awesome (check it out if you get the chance), but I was shocked that I have been working with SSMS for so many years and have never known about the following trick I watched her perform in her demos.

An "Organized" Nightmare

I'm guessing you've probably worked in a database that has hundreds or thousands of database objects:

lots-of-tables

I often come across this in applications where for one reason or another someone decides that there is no need to separate apps into different databases; why bother creating different databases when you can have lots of different schemas to organize your objects instead!? (*cough* vendor applications *cough*).

The problem with these enormous lists of tables, procedures, functions, etc... is that it can get pretty tiring to scroll through them to find what you need.

For years I wore down my mouse's scroll wheel, scrolling between thousands of objects across multiple server instances.  As it turns out, SSMS has a much better feature for handling this problem.

Filtering Objects in SSMS

You can apply filters to most objects in SSMS by right clicking and choosing "Filter Settings":

filter-settings

You can filter on attributes such as name, schema, create date, etc...:

filter-settings-dialog

For example, if I want to see only tables that are in the Travel schema, all I have to do is create a filter:

FILTER-SETTINGS-DIALOG-FILLED-IN-1

SSMS will even tell me that my list of objects is filtered so I don't go crazy later on wondering where all of my other tables went.

FILTERED-TABLES

Limitations

There are a few limitations with using SSMS object filters though.

For example, the different filter attributes work together as if they had "AND" operators between them, so you can do something like filter on tables in the Travel schema that contain the letter "a":

and-filter

However, there is no way to write multiple conditions with OR logic (eg. you can't filter on the schemas "Travel" OR "Lodging").

And while SSMS will indicate that your objects are filtered, it won't persist that filter after restarting SSMS.

Even with those drawbacks, I've used this filtering feature at least once per week since learning about it; it saves a lot of time and I can't believe I went so long without knowing about it.

Join Elimination: When SQL Server Removes Unnecessary Tables

In this post and video at SQLPerformance.com, I discuss what join elimination is and how it works in SQL Server.  Join elimination is one of the many techniques that the SQL Server query optimizer uses to create efficient query plans. Specifically, join elimination occurs when SQL Server can establish equality by using query logic or trusted database constraints to eliminate unnecessary joins... Read more or watch the video

Prioritizing Index Performance Pains - What I Learned From Brent Ozar's Master Index Tuning Class

Watch this week's video on YouTube

I love continuing my education in SQL Server.

Recently I was fortunate enough to take Brent Ozar's 3-day live online Master Index Tuning training class.

Since I had never taken an online class before, I thought it'd be fun to write a mini-review of what I thought about the class as well as what lesson from the course had the biggest impact on me.


As excited as I was to take the class, I was a little concerned about a couple of aspects about it. My main concerns before taking the online class were:

  • Would I have difficulty focusing in an online class?  YouTube, Facebook, and other distractions are only a few clicks away.
  • Will it feel like a 1-way dialogue since I wouldn't be there in person to ask questions, participate in discussion, and interact with other student?

My first concern was immediately invalidated when we started working on our labs. During the 3 days, Brent does the traditional training of explaining problems, techniques, and solutions (in his own humorous way of course) BUT THEN instead of moving on to the next topic he gives the students time to work through problems on their own in labs he set up.

This was huge. Every student gets an 8 core, 64TB Azure VM running simulated workloads that provide a great real-world representation of an under performing SQL Server.

Instead of taking notes for 3 days and hoping that I would remember to try them out when I returned to work, the labs allowed me to apply those new concepts immediately. I think working through these problems in a real-world scenario is not only practical, but probably the best way of retaining the information you learn. I'm not sure how much work it was for Brent to set up these labs (guess: a ton), but they really payoff as a student.

As for my concern about the course not being interactive, that turned out to not be a problem either. During the duration of the class (as well as for a couple hours before and after the class each day) there was a Slack chatroom where Brent and the students could interact.

The Slack chat had two huge advantages:

  1. Questions were typed out and written down, making it easy to know who asked what and what question Brent was responding to.  It also made it easy to copy and paste the chat for my notes.  It was also nice to have the ability to type a question overnight and get an answer the next morning.
  2. We were asked to post all of our lab queries into the chat when we think we had found a solution to the performance problems. This was AWESOME because not only did I get 1-on-1 code review and advice from Brent, I got to see what everyone else was submitting as an answer.
    Each lab scenario could usually be solved a few different ways, so this meant other students were submitting completely different solutions than what I came up with.  Seeing other people's approach to problem solving was really cool. Even better, I could try out their solutions in my own lab by copying and pasting the queries from Slack into my lab VM, seeing how those different solutions compared to my own.

So while I had some concerns before the class started, it turns out they weren't problems at all and the class was great.

My Favorite Lesson: Prioritizing Performance Solutions

The Master Index Tuning training covered everything needed to practically troubleshoot and solve index performance issues.  While Brent taught many techniques, from everyday solutions to advanced once-in-a-while techniques for certain edge cases, my biggest takeaway was how Brent taught us to prioritize the low-hanging-index-tuning-fruit first.

I'm a query tuner by nature.  Before taking this training, my go-to solution for fixing a performance problem was to start refactoring a query or correcting the indexes for that one particular query.

Instead, Brent recommended that we focus on the actions with the largest performance returns first: at a database or instance level, clean up duplicate (or near duplicate) indexes, drop unused indexes, add missing indexes, etc...

Many individual query performance issues can be fixed by changing indexes alone, without ever needing to touch the queries themselves.  Fixing poorly created indexes on your server has the additional advantage that it can improve performance of MULTIPLE queries running on your server simultaneously.

The takeaway here is that unless your server is already in tip-top shape, you'll probably get more bang for your buck fixing duplicate, inadequate, and missing index issues than you will by tuning individual queries.  This may sound obvious, but from my "every problem is a query tuning problem" standpoint this was a nice reminder that sometimes looking at the bigger picture will produce larger returns than focusing only on the details.

Final Verdict: Is It Worth It?

Yes.  Especially if you are the kind of person who learns best by doing and wants real-world index tuning advice.  By the end of the class I felt confident enough to immediately go back to work and start applying what I had learned from Brent and practiced in the labs.