Rahman Agoro, the frontline SQL DBA

Just another WordPress.com weblog

Archive for the ‘Professional Development’ Category

SQL server DBA development code review checklist

Posted by rahmanagoro on October 4, 2010


As the database administrator, I often have to deal with production problems. Now some of these problems can be classed under different categories, they include the followins.

  • Peformance issues.
  • Issues as a result of a release.
  • Bugs within the SQL engine
  • Faulty SQL code
  • Proactive/reactive issues

The issue of performance problems and faulty SQL code can often be attributed to bad coding, whilst I know that a job of a DBA itself can be hard, the code review process is one which is often ignored in most shops. I don’t share thesame approach, I like to know what code is being written on the database server which I manage, afterall if there are problems, the database administrator will be the one to be called at 02:45AM in the early hours of the morning. I like to avoid such problems and also ensure that my environment is as stable as possible. The art of performance tuning is not one that can be learnt from text books or just reading from the internet, its one that you gain often from experience.  I often gain such experience by dealing with production issues, attenging seminars and of course a lot of reading, but I find that some of these problems can be avoided in the first place. Thats where DBA code reviews comes into play, imagine a situation where a developer was corrected as to how to best write a SQL query, another developer is making thesame mistake, so what do you then do. Do you go to every developer and start correcting them ?

At a previous job, what I used to do is organise presentations once a month with my development team, often speaking about issues that I have either noticed or topics that the development team want me to speak about. As for code reviews, I came up with a checklist of items that I often look out for. They are as follows.

  • Consistent naming convention should be used at all times, there are loads of naming conventions out there, pick one and stick to it.
  • All objects especially stored procedures should have comments, information can include author, date, description of the object, changes etc.
  • Error handling using TRY.. CATCH. TRY..CATCH was released in SQL 2005 and is by far the best error handling method built into SQL.
  • Use of schema names when referencing tables, specify Database.schemaname.objectname. This ensures that the SQL engine doesnt start looking for the schema which holds the object.
  • Avoid the use of Select *, all column names must be explicitly specified. This will avoid unnecessary IO.
  • Ensure that all tables have a primary key (preferably of integer type), and all foreign keys are indexes with a non clustered index.
  • Avoid the placing of duplicate indexes on tables. As simple as it might sound, I have seen this in various places.
  • Developers should make efforts to ensure that databases are normalised, a relationship diagram for the database would also be useful.
  • Avoid server side cursors and maintain the use of SET based operation.  If a cursor really needs to be used, use a WHILE loop instead and specify a primary key in your table, a very easy way to do this is using an identity column.
  • Temporary tables and table variable, as a general rule of thumb use a table variable for relatively small datasets, and use a temporary table for fairly large datasets. Temp tables maintain statistics and can have indexes, but often recompile on the other hand as a downside, table variables don’t maintain statistics but they don’t force a recompilation. This was demonstrated in the SQL BITS 2009 session.
  • When using nested stored procedures, ensure that the temporary table names are unique/different. I have seen this affect query plans in a very catastrophic manner.
  • Avoid the use of functions in Joins within SQL 2005/2008, from experience performance tends to suffer as the dataset grows mianly because the SQL optimizer doesn’t know which index to use. Use a CTE or a derived table instead.
  • Embrace the use functions like OBJECT_ID to check for the existence of an object within stored procedures/script etc. This option is better then querying sysobjects where it can cause locking.
  • When using ORDER BY within SQL, avoid using ORDER BY ORDINALS for example select name,surname from tblStudents order by 1,2. When using order by, ensure that the column named are specified, alias names should also be avoided as this doesn’t work in SQL 2008.
  • Stored procedures that take in different combination of parameters should use local variables in order to avoid parameter sniffing. When having branching IF statements within stored procedures that process complicated logic, each IF statement should call another sub procedure rather than a direct SQL statement. This will allow stored procedures to make use of the procedure cache, and prevent stored procedures generating inconsistent query plans due to parameters.
  • Tables that hold parameter information or lookups should have not null constraints defined.
  • Developers should make efforts in identifying heavy queries within their applications, and review accordingly on how to optimise the query if needed.
  • New tables to be sized appropriately with respect to the use of the table and the nature of the data it will store.
  • Avoid the use of optimiser hints, in SQL 2008 hints should only be used in exceptional circumstances i.e. maintaining a consistent planguide etc. At all times, the optimiser should be left to determine the best plan for all queries.
  • Use of SET NOCOUNT ON at the beginning of stored procedures, this improves performance by reducing network traffic.
  • Developers should be careful with the use of custom data types, native SQL data types should be used where possible at all times.
Advertisements

Posted in Database development, Professional Development, SQL Administration | Leave a Comment »

Field notes from SQL BITS 7 full conference

Posted by rahmanagoro on October 4, 2010


I recently attended the SQL bits 7 full conference in the UK, I will be sharing tips and other information that I was able to pick up on the day. I will divide this into the different days and all the sessions that I was able to attend.

Thursday Session

I attended Rob Farleys Advanced Query Tuning training on Thursday, it was a tough start to the day as my train to York got canceled as soon as I got to the station, ok its not too bad I was offered another ticket to get on the train leaving in about 3 minutes, hooray !! It was only the beginning of a tough start, I mistakenly got into a fancy steam train to Newcastle, it looked like one of the Harry Potter trains, something just didn’t feel right when I got into it. Luckily for me, I was able to get out of the train in time before it departed as it would have been a slow steam train all the way to Newcastle and there are no stops before then.

Ok finally, I missed my train and got on the next one 30 minutes later. I only missed about 30 minutes of the training and to be honest I choose to attend this course because I believe that query tuning is an art that one needs to master by constantly learning and constantly being challenged to tune queries, so my expectations were that I will learn something that I have not seen or heard of before.

Rob Farleys Session

During this session I was expecting to be thought some advanced SQL topics, below are the notes that I took during the session.

  • Investigate the use of a full join
  • Always check on the execution plan properties as it does have some useful information on it.
  • Do some more research on join types, nested loops, merge join, hash join etc.
  • Using a distinct and group by with a scalar function, the thing to look out for here is that if one uses the distinct keyword, all the records are passed to the function but if a group by is used, it reduces the number of records that is passed to the scalar function.
  • One should not use the between clause because of the issue with 23:59:999, one should use < @date and >@date instead. 
  • Indexes can sometimes use a different collation and one should be wary of this.

 

Brian McGeHee

Health Checks for SQL server. 

This session sounded useful as I have a set of checks that I would normally carry out on database servers that I manage every morning. These includes the following. 

  • Checking connectivity to the server.
  • Checking the error log
  • Checking the backups
  • Checking SQL replication
  • Checking waits on the database

The point of attending this session was to see if there are any other checks that I may be missing out and ones that I can easily automate. What I have done is automated this checks using a combination of SSIS for collecting the data from multiple servers and using SSRS for the web presentation for the reports.

Notes that I took down on the day includes the following. 

  • Check Brian McGehess’s website for more information about the daily checks.
  • Extract baseline information about your servers.
  • Create a best practice manual
  • When commissioning a new server, ensure that firmware is always up to date.
  • Look at collecting event log across multiple servers (I want to automate this using powershell)
  • Using PAL (Nice handy tool)
  • Download SQL server tacklebox
  • Download Brent Ozar’s blitz script

Briand McGeHee’s website can be found on: www.BrianMcGehee.com/healthcheck.zip

Brant Ozar’s SQL Server and SAN session

This session was to discuss managing a SQL server instance on a SAN from a storage perspective, things that a DBA should look out for and other tips. I have been to several of Brent’s presentation, and he generally a good speaker. So far its always been fun. 

Notes that I took down on the day includes the following.

  • Windows task manager on virtual machines should not be relied upon, in fact he said its in a mess.
  • Run CPU-Z to get some visibility on physical hardware, check www.cpuid.com
  • Minimise the number of virtual cores
  • Strip out unnecessary virtual hardware, i.e. floppy drive, CD drive etc.
  • Remove all background services
  • Monitor for throttling.
  • Avoid affinity masking

 

Brent also touched on some VMware memory best practices

They include the following

  • Setting memory reservation
  • Set minimum and maximum memory allocation appropriately

HyperV 

  • Avoid dynamic memory for SQL server
  • VMware counters to check out include the following.
  •  % processor time
  • Host processor speed
  • Limit (mhz)
  • Reservation (Mhz) – ask your sysadmins about reservation on vmware hosts.

VMware memory counters

  • Memory limit
  • Emory reservation (MB)
  • Memory ballooned
  • Memory swapped (This should always be 0)

Posted in Professional Development, SQL 2008, Tips | Leave a Comment »

SQL 2008 Installation step by step with pictures including silent unattended installation

Posted by rahmanagoro on August 28, 2010


SQL 2008 Manual Installation Guide

The following guide details how to install SQL 2008 step by step guide with pictures.

One important setting worth mentioning is collation setting, please ensure that you are using the right collation settings, the chosen database collation setting has been detailed is: SQL_Latin1_General_CP1_CS_AS .

Manual SQL Installation Steps

This page is for the SQL 2008 Installation steps.
Prerequisites To install SQL Server 2008

1. Insert the SQL Server installation media. From the root folder, double-click setup.exe. To install from a network share, locate the root folder on the share, and then double-click setup.exe. If the Microsoft SQL Server 2008 Setup dialog box appears, click OK to install the prerequisites, then click Cancel to exit SQL Server 2008 installation.
2. If the .NET Framework 3.5 SP1 installation dialog box appears, select the check box to accept the .NET Framework 3.5 SP1 License Agreement. ClickNext. To exit SQL Server 2008 installation, click Cancel. When installation of .NET Framework 3.5 SP1 is complete, click Finish.
3. Windows Installer 4.5 is also required, and might be installed by the Installation Wizard. If you are prompted to restart your computer, restart it, and then restart SQL Server 2008 setup.exe

Carry on with the following steps.

Click on show details and ensure that all the checks are passed.

Select all the features for installation

Select the root directory for the installation.

Check disk space summary.

Select the account details for provisioning the SQL service and other related services.

Select the collation setting for the installation, note the collation setting in use here.

Select Authentication mode, if mixed authentication is used, select the SA password.

Select a directory for the database.

Select the filestream option if applicable.

Select the Analysis services account configuration.

Select the directory for the Analysis service related files.

Select reporting services configuration.

Select error reporting options.

Ensure that installation rules are all passed.

Installation screen, the process will take some time, this depends on the capacity of the machine on which it is being installed on to.


After a successfull installation, open up Programs >> SQL Server 2008 >> Configuration Tools >> SQL Server Configuration Manager

Check the TCP/IP section and enable it.

Click on Enable.

SQL 2008 Scripted Installation

 

Use the command line options below to install SQL 2008, remember to change the directory to the directory where the installation media is.
Setup.exe /QUIET /QS /ACTION=Install /ERRORREPORTING=1 /SAPWD=”******” /SQLSVCPASSWORD=”****” /AGTSVCPASSWORD=”****” /ASSVCPASSWORD=”****” /ISSVCPASSWORD=”****” /RSSVCPASSWORD=”******” /ConfigurationFile=C:\Downloads\Config.ini
Attention should also be paid to the configuration file as well, see attached configuration file which contains answers for the installation.
Install SQL 2008 SP1 Quietly
You can install SQL 2008 SP1 quietly without the need for a reboot, just ensure that you close all Management Studio windows that are open, this way a reboot will most likely not be required. Use the attached batch file, ensure that you have downloaded the service pack and change the path referenced in the batch file to the correct path.

Configuration.ini file

Below is a sample configuration.ini file for the installation above, copy it and save it to  a file called Config.ini as referenced above in the command line installation.


;SQLSERVER2008 Configuration File
[SQLSERVER2008]

; Specify the Instance ID for the SQL Server features you have specified. SQL Server directory structure, registry structure, and service names will reflect the instance ID of the SQL Server instance.

INSTANCEID="MSSQLSERVER"

; Specifies a Setup work flow, like INSTALL, UNINSTALL, or UPGRADE. This is a required parameter.

ACTION="Install"

; Specifies features to install, uninstall, or upgrade. The list of top-level features include SQL, AS, RS, IS, and Tools. The SQL feature will install the database engine, replication, and full-text. The Tools feature will install Management Tools, Books online, Business Intelligence Development Studio, and other shared components.

FEATURES=SQLENGINE,REPLICATION,FULLTEXT,AS,RS,BIDS,CONN,IS,BC,SDK,BOL,SSMS,ADV_SSMS,SNAC_SDK,OCS

; Displays the command line parameters usage

HELP="False"

; Specifies that the detailed Setup log should be piped to the console.

INDICATEPROGRESS="False"

; Setup will not display any user interface.

QUIET="False"

; Setup will display progress only without any user interaction.

QUIETSIMPLE="False"

; Specifies that Setup should install into WOW64. This command line argument is not supported on an IA64 or a 32-bit system.

X86="False"

; Specifies the path to the installation media folder where setup.exe is located.

MEDIASOURCE="I:\"

; Specify if errors can be reported to Microsoft to improve future SQL Server releases. Specify 1 or True to enable and 0 or False to disable this feature.

ERRORREPORTING="False"

; Specify the root installation directory for native shared components.

INSTALLSHAREDDIR="C:\Program Files\Microsoft SQL Server"

; Specify the installation directory.

INSTANCEDIR="C:\Program Files\Microsoft SQL Server"

; Specify that SQL Server feature usage data can be collected and sent to Microsoft. Specify 1 or True to enable and 0 or False to disable this feature.

SQMREPORTING="False"

; Specify a default or named instance. MSSQLSERVER is the default instance for non-Express editions and SQLExpress for Express editions. This parameter is required when installing the SQL Server Database Engine (SQL), Analysis Services (AS), or Reporting Services (RS).

INSTANCENAME="MSSQLSERVER"

; Agent account name

AGTSVCACCOUNT="NT AUTHORITY\SYSTEM"

; Auto-start service after installation.

AGTSVCSTARTUPTYPE="Manual"

; Startup type for Integration Services.

ISSVCSTARTUPTYPE="Automatic"

; Account for Integration Services: Domain\User or system account.

ISSVCACCOUNT="NT AUTHORITY\SYSTEM"

; The name of the account that the Analysis Services service runs under.

ASSVCACCOUNT="NT AUTHORITY\SYSTEM"

; Controls the service startup type setting after the service has been created.

ASSVCSTARTUPTYPE="Automatic"

; The collation to be used by Analysis Services.

ASCOLLATION="Latin1_General_CS_AS"

; The location for the Analysis Services data files.

ASDATADIR="C:\Program Files\Microsoft SQL Server\MSAS10.MSSQLSERVER\OLAP\Data"

; The location for the Analysis Services log files.

ASLOGDIR="C:\Program Files\Microsoft SQL Server\MSAS10.MSSQLSERVER\OLAP\Log"

; The location for the Analysis Services backup files.

ASBACKUPDIR="C:\Program Files\Microsoft SQL Server\MSAS10.MSSQLSERVER\OLAP\Backup"

; The location for the Analysis Services temporary files.

ASTEMPDIR="C:\Program Files\Microsoft SQL Server\MSAS10.MSSQLSERVER\OLAP\Temp"

; The location for the Analysis Services configuration files.

ASCONFIGDIR="C:\Program Files\Microsoft SQL Server\MSAS10.MSSQLSERVER\OLAP\Config"

; Specifies whether or not the MSOLAP provider is allowed to run in process.

ASPROVIDERMSOLAP="1"

; Specifies the list of administrator accounts that need to be provisioned.

ASSYSADMINACCOUNTS="DOMAIN\username"

; Startup type for the SQL Server service.

SQLSVCSTARTUPTYPE="Automatic"

; Level to enable FILESTREAM feature at (0, 1, 2 or 3).

FILESTREAMLEVEL="3"

; Name of Windows share to be created for FILESTREAM File I/O.

FILESTREAMSHARENAME="MSSQLSERVER"

; Set to "1" to enable RANU for SQL Server Express.

ENABLERANU="False"

; Specifies a Windows collation or an SQL collation to use for the Database Engine.

SQLCOLLATION="SQL_Latin1_General_CP1_CI_AS"

; Account for SQL Server service: Domain\User or system account.

SQLSVCACCOUNT="NT AUTHORITY\SYSTEM"

; Windows account(s) to provision as SQL Server system administrators.

SQLSYSADMINACCOUNTS="Domain\username"

; The Database Engine root data directory.

INSTALLSQLDATADIR="C:\Database"

; Default directory for the Database Engine backup files.

SQLBACKUPDIR="C:\Database"

; Default directory for the Database Engine user databases.

SQLUSERDBDIR="C:\Database"

; Directory for Database Engine TempDB files.

SQLTEMPDBDIR="C:\Database"

; Provision current user as a Database Engine system administrator for SQL Server 2008 Express.

ADDCURRENTUSERASSQLADMIN="False"

; Specify 0 to disable or 1 to enable the TCP/IP protocol.

TCPENABLED="0"

; Specify 0 to disable or 1 to enable the Named Pipes protocol.

NPENABLED="0"

; Startup type for Browser Service.

BROWSERSVCSTARTUPTYPE="Disabled"

; Specifies which account the report server NT service should execute under.  When omitted or when the value is empty string, the default built-in account for the current operating system.
; The username part of RSSVCACCOUNT is a maximum of 20 characters long and
; The domain part of RSSVCACCOUNT is a maximum of 254 characters long.

RSSVCACCOUNT="NT AUTHORITY\SYSTEM"

; Specifies how the startup mode of the report server NT service.  When
; Manual - Service startup is manual mode (default).
; Automatic - Service startup is automatic mode.
; Disabled - Service is disabled

RSSVCSTARTUPTYPE="Automatic"

; Specifies which mode report server is installed in.
; Default value: “FilesOnly”

RSINSTALLMODE="DefaultNativeMode"

Posted in Day to day queries, Professional Development, SQL 2008, SQL Administration, Uncategorized | 1 Comment »

Script out SQL agent jobs from powershell

Posted by rahmanagoro on August 26, 2010


At my shop, we currently use SVN to store database objects, one object type which is often overlooked is SQL agent jobs, I needed a way in which I can generate sql agent jobs into sql files, SQL management studio allows you to script jobs, but you can either do this one at a time, or script all the jobs in one big file. The two options are not really ideal for me, the reason being that I may choose to be selective about which jobs I install on a server, other jobs are installed as part of an installation suite, e.g management data warehouse in SQL 2008.

I have written a simple powershell script that will simple script out all SQL agent jobs, one might have multiple servers, I have made the script such that it reads a text file for a list of servers, simply save this file on a location and call the powershell script, you also need to specify where you want the script to be generated to as well.


#Original
# Load SMO extension

[System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SqlServer.Smo") | Out-Null;
# Get List of sql servers to check

#$sqlservers = Get-Content "$Env:USERPROFILE\sqlservers.txt";
$sqlservers = Get-Content "C:\Servers.txt";

# Loop through each sql server from sqlservers.txt
foreach($sqlserver in $sqlservers)

{

      # Create an SMO Server object
      $srv = New-Object "Microsoft.SqlServer.Management.Smo.Server" $sqlserver;

      # Jobs counts
      $totalJobCount = $srv.JobServer.Jobs.Count;
      $failedCount = 0;
      $successCount = 0;

      # For each jobs on the server
      foreach($job in $srv.JobServer.Jobs)

      {
            # Default write colour
            $colour = "Green";
            $jobName = $job.Name;
            $jobEnabled = $job.IsEnabled;
            $jobLastRunOutcome = $job.LastRunOutcome;
 			$jobNameFile = "C:\SQLJOBS\" + $jobName+".sql"

 			Write-Host $job.Name
			Write-Host "The location of the file is called " $jobNameFile

#			[System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SqlServer.Smo") | Out-Null
#			$srv = New-Object "Microsoft.SqlServer.Management.Smo.Server" $sqlserver
#			#$srv.JobServer.Jobs | foreach {$_.Script()} | out-file -path $path
			$job | foreach {$_.Script()} | out-file $jobNameFile

            # Set write text to red for Failed jobs
            if($jobLastRunOutcome -eq "Failed")

            {

                  $colour = "Red";
                  $failedCount += 1;
            }

            elseif ($jobLastRunOutcome -eq "Succeeded")
            {
                  $successCount += 1;
            }

            Write-Host -ForegroundColor $colour "SERVER = $sqlserver JOB = $jobName ENABLED = $jobEnabled LASTRUN = $jobLastRunOutcome";
      }

      # Writes a summary for each SQL server
      Write-Host -ForegroundColor red "=========================================================================================";
      Write-Host -ForegroundColor red "$sqlserver total jobs = $totalJobCOunt, success count $successCount, failed jobs = $failedCount.";
      Write-Host -ForegroundColor red "=========================================================================================";
}

Posted in Day to day queries, Powershell, Professional Development, Scripts, SQL 2008, Tips | 4 Comments »

DBA and Developer relationship

Posted by rahmanagoro on August 26, 2010


Throughout my career as a DBA, I often find that I need to work with developers more closely than anyone can imagine, DBA’s are often thought about as the middle man between the development team and also system administrators. I find that this is often the case, the DBA is a special position within any organization, responsible for the day to day maintenance and smooth running of the database involves being patient, responsive, being communicative with your customers (developers, business users, management, system administrators and other stake holders). Personally I tend to have a good rapport with developers, as they would often know how the database objects interact with the application, especially within a shop with little or no documentation.

I also find that although, as a DBA one doesn’t really have to know the application or business inside out, but it does help if you do. There was an occasion where an analyst complained about slow performance of the application, if users complain about slow performance within applications, as a DBA there are many lines of enquiries to make, some of which are not limited to the following:

  • Locking on the database
  • Improper use of indexes on the database
  • Statistics which are not up to date.
  • Inappopriate query plans
  • Heavy IO operation on the database, this can be heavy reporting/ad-hoc queries or perhaps backups running on the system.

In my own case, I was able to speak to the developer, a profiler trace was created to see what the problem was. It was later tracked down to a stored procedure using an in appropriate query plan. I was able to speak to the developer who quickly pointed out to me the stored procedure that was being called for the functionality the user was trying to use. This was subsequently fixed and the user was able to carry on using the application.

My tips are as follows

Speak to developers about applications that they are working with, especially ones which interact with the database. This way, you have an idea of what sort of things to watch out for.

Always see your developers as your customers as well, a healthy relationship with your developers means that both sides can work together as a team, and thus improving the service being offered to the organization.

Don’t think your developers will always come and ask you questions, at first the trust may not be there, that’s why its good to have a good report and make sure they are comfortable with you.

Be diplomatic when you see bad SQL code, don’t say to developers “your code is really horrible, I don’t know how you got the job ! “, be diplomatic and try and help them write better queries, this way when they are not comfortable with writing SQL code or a code which is not performing, they are willing to seek your help as the DBA, instead of just releasing it on to the production database server.

Socialize with the team !

Posted in Day to day queries, Professional Development | Leave a Comment »