Saturday, 2 November 2013

Ola Hallengren SQL Server Backup Script adds support for MS Shipped objects - eg merge replication metadata tables

SQL Server Merge replication makes very heavy use of metadata tables, and these are likely to become fragmented quickly. My own totally unscientific findings are that regular maintenance of these tables can give a 10% reduction in sync times.
If you want to know more about the subject read this article from BOL:
http://msdn.microsoft.com/en-us/library/ms151789(v=sql.105).aspx

You should be aware that most maintenance systems including the excellent and very popular Ola Hallengren solution http://ola.hallengren.com and the built in SQL Server Maintenance Plan system did not include the merge replication metadata tables.

Whilst investigating Ola's solution I noted that it did not include merge metadata tables which are marked as MS_Shipped objects, so I contacted Ola and he listened carefully, asked some very good questions and decided to add as a feature. The feature is now available in his build and you need to look at the parameter MSShipped objects.

It is commonplace to say the SQL Server built in Maintenance Plan is poor, as it rebuilds all indexes without regard to levels of fragmentation or size of tables etc. This is true but if you are not running 24 x 7 and the extra log generation is not going to create many problems (remember you will be creating/backing up a very large log file with the the built in maintenance plan in full recovery) and you have some disk IO to burn it is a good start and certainly much better than nothing. Take the option for compression if you have a supported version (SQL Server 2008R2 Standard and higher).

Whatever you are using check fragmentation of merge metadata tables.
You can read more here:
http://www.brentozar.com/blitz/merge-replication-with-infinite-history/


Tuesday, 24 September 2013

Network and online backup problems after upgrading Server 2012 to 2012 R2 upgrade

I recently upgraded my Server 2012 'Workstation' to 2012 R2 RTM.

The system is mainly used for testing changes etc and is an old rack mounted Dell 610 - 48gb RAM with a solid state boot disk and filled with as many 7.2K cheap/large capacity disks as it will hold in RAID 6 to give reasonable disk IO and large capacity. It also has a 4 port Intel network card with a couple of the ports teamed together mainly to test that the new MS NIC teaming works as intended. The NIC team has worked flawlessly for a year laying to rest the Server 2003 scalable networking debacle.

Following upgrade from Server 2012 to Server 2012 R2 I couldn't access it via remote desktop so I went in through the Dell Remote Access Card (DRAC) and looked at the console. After a few minutes poking about it I had worked out that the NIC teaming had been removed and the network set to DHCP. I was able to connect to the DHCP IP address and sorted it out from there. I would say this behavior is very unfortunate in enterprise computing. Normally I would have said it was a one off but the same thing happened a few years ago when upgrading some Server 2008 to Server 2008 R2 Dell Servers, which lost their network card settings during upgrade. If the server is remote and you don't have a remote access tool like the DRAC then you could be in a lot of trouble. Worth watching out for.

I was looking to change the Windows Azure Backup Policy on the machine after upgrade but it wouldn't change, and kept coming up with a  generic error after a timeout period. I deleted it and recreated it from scratch and it is now running fine. I cannot imagine how that backup software changed between versions as it was a manual download from the internet but there you go.

Apart from these upgrade errors the Server 2012 R2 release is looking good with many new features.

Sunday, 25 August 2013

EnableWriteOrderPreservationAcrossDisks not in Hyper-V Replica GUI or set ON by default for new replicas

One of the most frustrating things about Hyper-V Replica (which I believe is the best feature of Hyper-V Server 2012 and truly a game changer to SMBs) is the lack of totally CLEAR and UNAMBIGUOUS up front information relating to the support of  mission critical systems like SQL Server and Exchange.

Both SQL Server and Exchange databases have one thing in common - they write out to a log whenever data is committed before they write out to the main data store. This ensures consistency under all recovery situations with excellent performance and the ability to replay transactions etc etc. Best practice is to put the logs and databases onto 'different disks' to improve performance/recovery. In the context of Hyper-V this is interesting as it means placing a Hyper-V vhd(x) file onto it's own physical disk. Either way that is a different discussion.

However, this is the undoing of Hyper-V Replica since it does not guarantee that writes to different disks will be applied to the replica in the same order. This could create disastrous results where the log and main data store are subtly out of sync whilst fooling the administrators of the system that everything is working perfectly. That is the worst possible scenario. During a planned fail-over when the primary is closed down in an orderly manner it would appear to work beautifully as the disks would all be sync'd but come the day of disaster and the writes have not been applied in the correct sequence it would not work correctly and you are out of business - literally. There should be far more on this subject.

The worst aspect is that you have to know the problem might exist to start hunting down the answer and that means you will be trawling around for a while looking for reasonably definitive information. Those unaware of the foibles of SQL, Exchange et al are waiting to fall at the first unplanned fail-over.

At time of writing I  found that SQL Server is supported
http://support.microsoft.com/kb/956893
and Exchange is not supported
http://blogs.technet.com/b/rmilne/archive/2013/07/29/exchange-and-hyper-v-replica-support.aspx

However, I cannot see any reason why Exchange not using DAGs would not work perfectly. I suspect that DAGs are the supported option so that is what you are forced to use. The Hyper-V replica is certainly much easier to set up and maintain for smaller organisations with say 1 Exchange server than a DAG. I can see Hyper-V replica and DAGs together would be a very poor idea.
When Hyper-V 1 first came out (remember the 180 day promise after Server 2008 RTM!) I ran a Blackberry Enterprise Server 4.1 on it even though it was not supported. I knew it would just work and it did for several years until the BES Server was replaced. I did not expect any support from anyone but that is always the best way so you know in your own mind you are doing it right!

So if you want to use a SQL Server workload you will need to use the EnableWriteOrderPreservationAcrossDisks flag on replica creation using the
Set-VMReplication cmdlet.
This means you must use PowerShell to create the replica as unfortunately this flag is not available in the GUI so the unknowing will not set it or be aware of it.

This flag should be in the GUI, and it should have comments like:
This feature ensures data writes are consistent across disks for Enterprise database systems such as SQL Server
This flag should be set if you are running a SQL Server workload - see KB111111
Exchange workloads are specifically not supported - see KB222222
For other workloads please read KB333333

What I don't understand is that if there is little performance loss or other downsides (I don't know) then setting that flag ON should be the default behavior when you create a replica and not a hidden away afterthought.



Tuesday, 16 July 2013

RDP Remote control deal breaker in Server 2012 fixed in R2

There were many advances in remote desktop in Server 2012, including valiant attempts to make installation simpler. The optimised used of bandwidth is an excellent behind the scenes advantage that you can see working when bandwidth is limited. Very impressive.

However, the initial move to management via Server Manager and PowerShell brought a major casualty - the remote control feature had been removed. When I first could not see the feature I assumed I was doing something wrong! But it dawned on me that it had been removed. A quick search showed that others had the same problem. A few weak workarounds were suggested but this feature is one of the most important features of all out of the feature set! I have lost count of the number of times I have helped out someone by logging in and attaching to their session.

Here is the reason from MS:
Due to the removal of the classic shell and the new architecture of the desktop window manager, in addition, consider to the security, we have removed the Remote Control(Shadow Session) in Windows Server 2012.

This was a deal breaker and Server 2008 R2 is currently in use as I could not use 2012 without this feature. I wondered if it had been removed for good and would be left ultimately looking at higher end solutions with richer feature sets. When 2012 R2 came out recently one of the first things I checked for was remote desktop enhancements and the feature was back under the guise of Session Shadowing. For me this means that Server 2012 R2 will probably be replacing 2008 R2 for terminal services when it is released.
New features in Server 2012 R2:
http://technet.microsoft.com/en-us/library/dn283323.aspx

On a related but separate note I think that the move to GUIs based on PowerShell has not been entirely successful, the new GUIs are pretty clunky, and you can almost feel them running the PowerShell commands under the covers. You can see they really hate doing anything other than manual refreshes where they rerun the PowerShell command to get a dataset back. Incremental search never features in any PowerShell backed GUI I have seen recently due to the same limitation. I am hoping they will improve on this in the next version or two.

Thursday, 13 June 2013

Active Directory Office 365 sync problems

Setup is Directory Sync to Office 365 hybrid (ie write back some AD attributes from Office 365) with password sync.

On initial sync there were some problems not found with the Office 365 pre-check tool.

A few users had inadequate permissions to write-back AD attributes. This was caused by them not inheriting security in AD. Switching inheritance back on fixed the problem immediately. Why they had been set to not inherit many years ago remains a mystery.

Another user who had moved to a different role just would not sync due to a duplicate attribute. The sync tool 'helpfully' omitted which attribute was in conflict.  The user in question had been copied from the original user and then changed and the original user left in place for business reasons. After some time including checking every (visible?) AD attribute no duplicates could be found. The (very useful) idfix tool did not spot any problems either with this record. Google/Bing for idfix.
I added the person to my Outlook to take a look what they had in their mailbox and 2 copies of the same mailbox appeared - very strange- never seen that before. I then removed the person from my Outlook only for 1 of the mailboxes to remain. It seemed obvious that the record had some major problem.The person in question no longer required the account and it was deleted in AD. 2 syncs later the problem had gone. It would have been useful to know exactly what was failing the unique constraint. I am sure it would be pretty easy for the programmers to report this information in the sync failure email you get sent.

I strongly recommend you pin the sync tool UI to the taskbar / desktop as you might be looking at it more than you had hoped at the start!

You can find it here (could change with future versions):
C:\Program Files\Windows Azure Active Directory Sync\SYNCBUS\Synchronization Service\UIShell\miisclient.exe

When you look at the on screen log of what it does automatically it is easy to run manual syncs by mimicking the steps it takes. There is a PowerShell command start-onlinecoexistencesync but you will be needing the GUI to track down problems. This GUI is actually a poor man's very crude AD auditing tool as you can see how many and the nature of changes to your AD and roughly when they happened but not who did it. It is certainly better than nothing, and is a good spin-off benefit of having installed the software.

It can save time when troubleshooting to change the default sync time from 3 hours to 5 minutes so the sync tool keeps attempting to sync whilst you are troubleshooting without you having to run anything:

To do this change the file:
C:\Program Files\Windows Azure Active Directory Sync\Microsoft.Online.DirSync.Scheduler.exe.Config

temporarily from this:
<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <appSettings>
    <!--the interval in hours-->
    <!--refer for valid values:http://msdn2.microsoft.com/en-us/library/system.timespan.parse.aspx-->
    <add key="SyncTimeInterval" value="3:0:0" />
  </appSettings>
</configuration>

to this:
<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <appSettings>
    <!--the interval in hours-->
    <!--refer for valid values:http://msdn2.microsoft.com/en-us/library/system.timespan.parse.aspx-->
    <add key="SyncTimeInterval" value="0:5:0" />
  </appSettings>
</configuration>

Set it back to your selected sync time afterwards - I would not recommend leaving it at 5 mins!

Happy syncing



Tuesday, 4 June 2013

SQL Server 2012 Extended events for error_reported with some merge replication noise removed

One of the main problems with the XE error_reported event is that it includes informational data that most people would not define as an error. Here is my version, which merges various internet versions I have seen with my own additions and is pretty simple. Many of the items filtered have a severity of 10 but I prefer to filter each error type individually having seen them appear in the trace file and made a conscious decision that the error can be ignored.  It might be a good start if you don't have anything else.

With extended events I always save 10 files of 100 meg as you can copy a 100 meg file over relatively low bandwidth comms if the need arises and 1 gig is not very onerous on storage. The files from this script would take several years to rollover as very few genuine errors are now logged.

As with all scripts use on your own servers at your own discretion. It is pretty low impact unless you have lots of errors in which case you will have more or different noise to my applications or some errors that need fixing! I don't need track_causality switched on with this trace but you might do depending on what you are doing. There are many bespoke replication 'noise' and minor errors filtered out which you might be an important error in your system so check before use.


CREATE EVENT SESSION [NM_Error] ON SERVER 
ADD EVENT sqlserver.error_reported(
ACTION(sqlserver.database_name,sqlserver.query_hash,sqlserver.query_plan_hash,sqlserver.session_nt_username,sqlserver.sql_text,sqlserver.tsql_frame,sqlserver.tsql_stack)
    WHERE ([error_number]<>(14108) AND [error_number]<>(20532) AND [error_number]<>(14149) AND [error_number]<>(20556) AND [error_number]<>(20554) AND [error_number]<>(20567) AND [error_number]<>(20568) AND [error_number]<>(3262) AND [error_number]<>(14226) AND [error_number]<>(17177) AND [error_number]<>(14150) AND [error_number]<>(14554) AND [error_number]<>(3197) AND [error_number]<>(3198) AND [error_number]<>(2528) AND [error_number]<>(18264) AND [error_number]<>(3211) AND [error_number]<>(3014) AND [error_number]<>(4035) AND [error_number]<>(5701) AND [error_number]<>(5703) AND [error_number]<>(18265) AND [error_number]<>(14205) AND [error_number]<>(14213) AND [error_number]<>(14214) AND [error_number]<>(14215) AND [error_number]<>(14216) AND [error_number]<>(14549) AND [error_number]<>(14558) AND [error_number]<>(14559) AND [error_number]<>(14560) AND [error_number]<>(14561) AND [error_number]<>(14562) AND [error_number]<>(14563) AND [error_number]<>(14564) AND [error_number]<>(14565) AND [error_number]<>(14566) AND [error_number]<>(14567) AND [error_number]<>(14568) AND [error_number]<>(14569) AND [error_number]<>(14570) AND [error_number]<>(14635) AND [error_number]<>(8153) AND [error_number]<>(14638) AND [error_number]<=(50000))) 
ADD TARGET package0.event_file(SET filename=N'd:\sqlxe\NM_Error',max_file_size=(100),max_rollover_files=(10))
WITH (MAX_MEMORY=4096 KB,EVENT_RETENTION_MODE=ALLOW_SINGLE_EVENT_LOSS,MAX_DISPATCH_LATENCY=30 SECONDS,MAX_EVENT_SIZE=0 KB,MEMORY_PARTITION_MODE=NONE,TRACK_CAUSALITY=OFF,STARTUP_STATE=ON)
GO

ALTER EVENT SESSION [NM_Error] ON SERVER STATE=START

Wednesday, 22 May 2013

Azure backup Start-OBBackup does not reliably emit errors

I noticed that the Windows Azure Online backup command does not always emit an error when the backup does not successfully complete in say a Try / Catch block.

This was most apparent when the job was cancelled/failed at an early stage during the backup.

As such it is easier to look through the output of the backup command and make sure it contains the text "The backup operation completed successfully". The output only contains this string if the backup completes successfully.

eg

$SuccessString = "The backup operation completed successfully"

$OnlineBackupOutput = Get-OBPolicy | Start-OBBackup
$OnlineBackupOutputString = $OnlineBackupOutput | Out-String

Then you can check
if ($OnlineBackupOutputString -match $SuccessString)
# ok
else
error condition...