Saturday 28 December 2013

Where have all the Windows Service Packs gone?

Have you noticed the lack of Service Packs for Windows 7 / Server 2008 R2?

We had Service Pack 1 more than 2 years ago and since then just a constant stream of updates. Windows 7 desktops are going to be around for a very long time replacing XP as the de facto Windows version to use in corporate desktops. By the time they expire in 2020(!) hopefully the newer versions of Windows / Android / IOS + current unknown OSs will provide a new paradigm that is better (I have my doubts).

The number of updates every month has stayed steady meaning there is a myriad of updates for a 'fresh' Windows 7 SP1 install. This was brought home the other week when a newly arrived Dell Laptop installed updates and rebooted for a good half an hour. Keeping baselines of updates is becoming unpractical with this number of updates going on. This is a real problem for servers where you want them all patched to the same level without going through a WSUS high frequency update route.

In the past on internal servers we used to apply Service Packs as the building block of Windows build level. If there was a specific reason to apply a patch we would apply it. More often than not it would more likely be a version of the .Net framework linked to an application like SQL Server. Most internal servers are showing vast lists of individual patches required. When servers can obtain typical uptimes of around a year I don't like putting on non required patches and rebooting them every month or so unless there is a very good reason to do so. Rebooting them once per year for a Service Pack seems ideal to me.

The cadence / patch mechanism of Windows releases has changed dramatically over the years. Remember computer magazines with attached disks of  NT4 SP3, SP4 or SP6a as a mechanism of distributing patches! Since then we had Windows 2000 which had 4 Service Packs, Server 2003 which brought in the R2 OS refresh cycle, 2008, then 2008 R2 and now 2012 / R2.

If you look at the release dates of recent software we have Server 2012 R2 in 2013 just a year after Server 2012. With Google and Apple adding new features and not just patches with almost every update every few months. MS have been forced to follow suit. I am running Windows 8.1 on my laptop and it is more a Service Pack on Windows 8 than it is a new release (I know there are new features, well hidden by the dumbed down GUI), at least the 8.1 gives a clue to the lack of fundamental changes. But with Server 2012 R2 a whole raft of new server features have been added. It could be the new update regime without Service Packs is better suited to Windows Azure with fully automated patching or they just build new base images containing the patches already incorporated for cloning on a schedule. It might be that Windows 7 / 2008 R2 will never see another Service Pack - in which case installing a new Windows 7 machine could install literally hundreds of updates to bring it up to date. Maybe keeping base images with incorporated selected updates will be the way to go. At the moment I would be happy to install a Windows 7 SP1 desktop and let it catch up on updates using WSUS plus a daily reboot over a few days. With mainstream support for Windows 7 ending in 2015 we really could do with Service Pack by then!

SQL Server and Exchange Server do (currently) have a steady and structured release system of updates and Service Packs making patching easier to plan. It is a pity that Exchange updates have been such poor quality of late. I would rather run reliable old software than up to date software with broken features myself.

Maybe the days of Service Packs are gone. Get a 'new' OS every year instead! I have heard little from MS on the subject but it must be discussed frequently. If so it makes life just a little harder for on premise installations...

Friday 8 November 2013

Server 2012 Hyper-V Live migration poor GUI error message

My colleague recently set up the networking on a backup machine that would be used to accept Server 2012 live migrations. The network was teamed using LACP and we always use the naming convention VMLAG for the VM Ethernet switch.

When I tested live migration from the GUI there was a problem - the error message was the virtual machine was not compatible. The target server was significantly older than the source and had never run a VM before so we started investigating hardware.

We looked through the specifications of the (significantly different Dell models) servers in question and they both met all the requirements of live migration. We rebooted and checked the BIOS and everything was switched on for virtualisation / live migration.

After some head scratching I decided to run the same command in Powershell and lo and behold the real error was displayed:
The original GUI message was still there:
The virtual machine 'XNMTEST' is not compatible with physical computer 'BACKUPSERVER'
But Powershell gave some crucial extra information relating to the error:
Could not find Ethernet Switch VMLAG


I immediately looked more closely at the target server. My colleague had put a space in the VM networking name!

Instead of VMLAG he had typed VM<space>LAG. I renamed the switch to match the source server and it worked perfectly.

2 morals to this story:
Double check the obvious first -  looking at processor specifications and BIOS settings was not required.

Use Powershell to get better error messages - the GUI message whilst technically accurate was somewhat misleading! The GUI should be modified to give the same error messages as Powershell instead of (presumably) just the highest level error.



Saturday 2 November 2013

Ola Hallengren SQL Server Backup Script adds support for MS Shipped objects - eg merge replication metadata tables

SQL Server Merge replication makes very heavy use of metadata tables, and these are likely to become fragmented quickly. My own totally unscientific findings are that regular maintenance of these tables can give a 10% reduction in sync times.
If you want to know more about the subject read this article from BOL:
http://msdn.microsoft.com/en-us/library/ms151789(v=sql.105).aspx

You should be aware that most maintenance systems including the excellent and very popular Ola Hallengren solution http://ola.hallengren.com and the built in SQL Server Maintenance Plan system did not include the merge replication metadata tables.

Whilst investigating Ola's solution I noted that it did not include merge metadata tables which are marked as MS_Shipped objects, so I contacted Ola and he listened carefully, asked some very good questions and decided to add as a feature. The feature is now available in his build and you need to look at the parameter MSShipped objects.

It is commonplace to say the SQL Server built in Maintenance Plan is poor, as it rebuilds all indexes without regard to levels of fragmentation or size of tables etc. This is true but if you are not running 24 x 7 and the extra log generation is not going to create many problems (remember you will be creating/backing up a very large log file with the the built in maintenance plan in full recovery) and you have some disk IO to burn it is a good start and certainly much better than nothing. Take the option for compression if you have a supported version (SQL Server 2008R2 Standard and higher).

Whatever you are using check fragmentation of merge metadata tables.
You can read more here:
http://www.brentozar.com/blitz/merge-replication-with-infinite-history/


Tuesday 24 September 2013

Network and online backup problems after upgrading Server 2012 to 2012 R2 upgrade

I recently upgraded my Server 2012 'Workstation' to 2012 R2 RTM.

The system is mainly used for testing changes etc and is an old rack mounted Dell 610 - 48gb RAM with a solid state boot disk and filled with as many 7.2K cheap/large capacity disks as it will hold in RAID 6 to give reasonable disk IO and large capacity. It also has a 4 port Intel network card with a couple of the ports teamed together mainly to test that the new MS NIC teaming works as intended. The NIC team has worked flawlessly for a year laying to rest the Server 2003 scalable networking debacle.

Following upgrade from Server 2012 to Server 2012 R2 I couldn't access it via remote desktop so I went in through the Dell Remote Access Card (DRAC) and looked at the console. After a few minutes poking about it I had worked out that the NIC teaming had been removed and the network set to DHCP. I was able to connect to the DHCP IP address and sorted it out from there. I would say this behavior is very unfortunate in enterprise computing. Normally I would have said it was a one off but the same thing happened a few years ago when upgrading some Server 2008 to Server 2008 R2 Dell Servers, which lost their network card settings during upgrade. If the server is remote and you don't have a remote access tool like the DRAC then you could be in a lot of trouble. Worth watching out for.

I was looking to change the Windows Azure Backup Policy on the machine after upgrade but it wouldn't change, and kept coming up with a  generic error after a timeout period. I deleted it and recreated it from scratch and it is now running fine. I cannot imagine how that backup software changed between versions as it was a manual download from the internet but there you go.

Apart from these upgrade errors the Server 2012 R2 release is looking good with many new features.

Sunday 25 August 2013

EnableWriteOrderPreservationAcrossDisks not in Hyper-V Replica GUI or set ON by default for new replicas

One of the most frustrating things about Hyper-V Replica (which I believe is the best feature of Hyper-V Server 2012 and truly a game changer to SMBs) is the lack of totally CLEAR and UNAMBIGUOUS up front information relating to the support of  mission critical systems like SQL Server and Exchange.

Both SQL Server and Exchange databases have one thing in common - they write out to a log whenever data is committed before they write out to the main data store. This ensures consistency under all recovery situations with excellent performance and the ability to replay transactions etc etc. Best practice is to put the logs and databases onto 'different disks' to improve performance/recovery. In the context of Hyper-V this is interesting as it means placing a Hyper-V vhd(x) file onto it's own physical disk. Either way that is a different discussion.

However, this is the undoing of Hyper-V Replica since it does not guarantee that writes to different disks will be applied to the replica in the same order. This could create disastrous results where the log and main data store are subtly out of sync whilst fooling the administrators of the system that everything is working perfectly. That is the worst possible scenario. During a planned fail-over when the primary is closed down in an orderly manner it would appear to work beautifully as the disks would all be sync'd but come the day of disaster and the writes have not been applied in the correct sequence it would not work correctly and you are out of business - literally. There should be far more on this subject.

The worst aspect is that you have to know the problem might exist to start hunting down the answer and that means you will be trawling around for a while looking for reasonably definitive information. Those unaware of the foibles of SQL, Exchange et al are waiting to fall at the first unplanned fail-over.

At time of writing I  found that SQL Server is supported
http://support.microsoft.com/kb/956893
and Exchange is not supported
http://blogs.technet.com/b/rmilne/archive/2013/07/29/exchange-and-hyper-v-replica-support.aspx

However, I cannot see any reason why Exchange not using DAGs would not work perfectly. I suspect that DAGs are the supported option so that is what you are forced to use. The Hyper-V replica is certainly much easier to set up and maintain for smaller organisations with say 1 Exchange server than a DAG. I can see Hyper-V replica and DAGs together would be a very poor idea.
When Hyper-V 1 first came out (remember the 180 day promise after Server 2008 RTM!) I ran a Blackberry Enterprise Server 4.1 on it even though it was not supported. I knew it would just work and it did for several years until the BES Server was replaced. I did not expect any support from anyone but that is always the best way so you know in your own mind you are doing it right!

So if you want to use a SQL Server workload you will need to use the EnableWriteOrderPreservationAcrossDisks flag on replica creation using the
Set-VMReplication cmdlet.
This means you must use PowerShell to create the replica as unfortunately this flag is not available in the GUI so the unknowing will not set it or be aware of it.

This flag should be in the GUI, and it should have comments like:
This feature ensures data writes are consistent across disks for Enterprise database systems such as SQL Server
This flag should be set if you are running a SQL Server workload - see KB111111
Exchange workloads are specifically not supported - see KB222222
For other workloads please read KB333333

What I don't understand is that if there is little performance loss or other downsides (I don't know) then setting that flag ON should be the default behavior when you create a replica and not a hidden away afterthought.



Tuesday 16 July 2013

RDP Remote control deal breaker in Server 2012 fixed in R2

There were many advances in remote desktop in Server 2012, including valiant attempts to make installation simpler. The optimised used of bandwidth is an excellent behind the scenes advantage that you can see working when bandwidth is limited. Very impressive.

However, the initial move to management via Server Manager and PowerShell brought a major casualty - the remote control feature had been removed. When I first could not see the feature I assumed I was doing something wrong! But it dawned on me that it had been removed. A quick search showed that others had the same problem. A few weak workarounds were suggested but this feature is one of the most important features of all out of the feature set! I have lost count of the number of times I have helped out someone by logging in and attaching to their session.

Here is the reason from MS:
Due to the removal of the classic shell and the new architecture of the desktop window manager, in addition, consider to the security, we have removed the Remote Control(Shadow Session) in Windows Server 2012.

This was a deal breaker and Server 2008 R2 is currently in use as I could not use 2012 without this feature. I wondered if it had been removed for good and would be left ultimately looking at higher end solutions with richer feature sets. When 2012 R2 came out recently one of the first things I checked for was remote desktop enhancements and the feature was back under the guise of Session Shadowing. For me this means that Server 2012 R2 will probably be replacing 2008 R2 for terminal services when it is released.
New features in Server 2012 R2:
http://technet.microsoft.com/en-us/library/dn283323.aspx

On a related but separate note I think that the move to GUIs based on PowerShell has not been entirely successful, the new GUIs are pretty clunky, and you can almost feel them running the PowerShell commands under the covers. You can see they really hate doing anything other than manual refreshes where they rerun the PowerShell command to get a dataset back. Incremental search never features in any PowerShell backed GUI I have seen recently due to the same limitation. I am hoping they will improve on this in the next version or two.

Thursday 13 June 2013

Active Directory Office 365 sync problems

Setup is Directory Sync to Office 365 hybrid (ie write back some AD attributes from Office 365) with password sync.

On initial sync there were some problems not found with the Office 365 pre-check tool.

A few users had inadequate permissions to write-back AD attributes. This was caused by them not inheriting security in AD. Switching inheritance back on fixed the problem immediately. Why they had been set to not inherit many years ago remains a mystery.

Another user who had moved to a different role just would not sync due to a duplicate attribute. The sync tool 'helpfully' omitted which attribute was in conflict.  The user in question had been copied from the original user and then changed and the original user left in place for business reasons. After some time including checking every (visible?) AD attribute no duplicates could be found. The (very useful) idfix tool did not spot any problems either with this record. Google/Bing for idfix.
I added the person to my Outlook to take a look what they had in their mailbox and 2 copies of the same mailbox appeared - very strange- never seen that before. I then removed the person from my Outlook only for 1 of the mailboxes to remain. It seemed obvious that the record had some major problem.The person in question no longer required the account and it was deleted in AD. 2 syncs later the problem had gone. It would have been useful to know exactly what was failing the unique constraint. I am sure it would be pretty easy for the programmers to report this information in the sync failure email you get sent.

I strongly recommend you pin the sync tool UI to the taskbar / desktop as you might be looking at it more than you had hoped at the start!

You can find it here (could change with future versions):
C:\Program Files\Windows Azure Active Directory Sync\SYNCBUS\Synchronization Service\UIShell\miisclient.exe

When you look at the on screen log of what it does automatically it is easy to run manual syncs by mimicking the steps it takes. There is a PowerShell command start-onlinecoexistencesync but you will be needing the GUI to track down problems. This GUI is actually a poor man's very crude AD auditing tool as you can see how many and the nature of changes to your AD and roughly when they happened but not who did it. It is certainly better than nothing, and is a good spin-off benefit of having installed the software.

It can save time when troubleshooting to change the default sync time from 3 hours to 5 minutes so the sync tool keeps attempting to sync whilst you are troubleshooting without you having to run anything:

To do this change the file:
C:\Program Files\Windows Azure Active Directory Sync\Microsoft.Online.DirSync.Scheduler.exe.Config

temporarily from this:
<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <appSettings>
    <!--the interval in hours-->
    <!--refer for valid values:http://msdn2.microsoft.com/en-us/library/system.timespan.parse.aspx-->
    <add key="SyncTimeInterval" value="3:0:0" />
  </appSettings>
</configuration>

to this:
<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <appSettings>
    <!--the interval in hours-->
    <!--refer for valid values:http://msdn2.microsoft.com/en-us/library/system.timespan.parse.aspx-->
    <add key="SyncTimeInterval" value="0:5:0" />
  </appSettings>
</configuration>

Set it back to your selected sync time afterwards - I would not recommend leaving it at 5 mins!

Happy syncing



Tuesday 4 June 2013

SQL Server 2012 Extended events for error_reported with some merge replication noise removed

One of the main problems with the XE error_reported event is that it includes informational data that most people would not define as an error. Here is my version, which merges various internet versions I have seen with my own additions and is pretty simple. Many of the items filtered have a severity of 10 but I prefer to filter each error type individually having seen them appear in the trace file and made a conscious decision that the error can be ignored.  It might be a good start if you don't have anything else.

With extended events I always save 10 files of 100 meg as you can copy a 100 meg file over relatively low bandwidth comms if the need arises and 1 gig is not very onerous on storage. The files from this script would take several years to rollover as very few genuine errors are now logged.

As with all scripts use on your own servers at your own discretion. It is pretty low impact unless you have lots of errors in which case you will have more or different noise to my applications or some errors that need fixing! I don't need track_causality switched on with this trace but you might do depending on what you are doing. There are many bespoke replication 'noise' and minor errors filtered out which you might be an important error in your system so check before use.


CREATE EVENT SESSION [NM_Error] ON SERVER 
ADD EVENT sqlserver.error_reported(
ACTION(sqlserver.database_name,sqlserver.query_hash,sqlserver.query_plan_hash,sqlserver.session_nt_username,sqlserver.sql_text,sqlserver.tsql_frame,sqlserver.tsql_stack)
    WHERE ([error_number]<>(14108) AND [error_number]<>(20532) AND [error_number]<>(14149) AND [error_number]<>(20556) AND [error_number]<>(20554) AND [error_number]<>(20567) AND [error_number]<>(20568) AND [error_number]<>(3262) AND [error_number]<>(14226) AND [error_number]<>(17177) AND [error_number]<>(14150) AND [error_number]<>(14554) AND [error_number]<>(3197) AND [error_number]<>(3198) AND [error_number]<>(2528) AND [error_number]<>(18264) AND [error_number]<>(3211) AND [error_number]<>(3014) AND [error_number]<>(4035) AND [error_number]<>(5701) AND [error_number]<>(5703) AND [error_number]<>(18265) AND [error_number]<>(14205) AND [error_number]<>(14213) AND [error_number]<>(14214) AND [error_number]<>(14215) AND [error_number]<>(14216) AND [error_number]<>(14549) AND [error_number]<>(14558) AND [error_number]<>(14559) AND [error_number]<>(14560) AND [error_number]<>(14561) AND [error_number]<>(14562) AND [error_number]<>(14563) AND [error_number]<>(14564) AND [error_number]<>(14565) AND [error_number]<>(14566) AND [error_number]<>(14567) AND [error_number]<>(14568) AND [error_number]<>(14569) AND [error_number]<>(14570) AND [error_number]<>(14635) AND [error_number]<>(8153) AND [error_number]<>(14638) AND [error_number]<=(50000))) 
ADD TARGET package0.event_file(SET filename=N'd:\sqlxe\NM_Error',max_file_size=(100),max_rollover_files=(10))
WITH (MAX_MEMORY=4096 KB,EVENT_RETENTION_MODE=ALLOW_SINGLE_EVENT_LOSS,MAX_DISPATCH_LATENCY=30 SECONDS,MAX_EVENT_SIZE=0 KB,MEMORY_PARTITION_MODE=NONE,TRACK_CAUSALITY=OFF,STARTUP_STATE=ON)
GO

ALTER EVENT SESSION [NM_Error] ON SERVER STATE=START

Wednesday 22 May 2013

Azure backup Start-OBBackup does not reliably emit errors

I noticed that the Windows Azure Online backup command does not always emit an error when the backup does not successfully complete in say a Try / Catch block.

This was most apparent when the job was cancelled/failed at an early stage during the backup.

As such it is easier to look through the output of the backup command and make sure it contains the text "The backup operation completed successfully". The output only contains this string if the backup completes successfully.

eg

$SuccessString = "The backup operation completed successfully"

$OnlineBackupOutput = Get-OBPolicy | Start-OBBackup
$OnlineBackupOutputString = $OnlineBackupOutput | Out-String

Then you can check
if ($OnlineBackupOutputString -match $SuccessString)
# ok
else
error condition...

Sunday 28 April 2013

Windows Azure Backup Setup via Powershell

This a follow on article from

http://maneffa-it.blogspot.co.uk/2012/05/new-features-in-windows-server-2012-for.html

Windows Azure Backup Vault is a great way to get an incremental and compressed off site backup. In my testing of the preview over several months and 100 gb  it 'just worked', and was superior to all the other 3rd party mechanisms I tried.

Here is the PowerShell needed to set up a Windows Azure Backup policy. It is mostly self explanatory, and assumes you have already saved a certificate into the store on the local machine.

You will need to create and upload a service certificate - here is an example to create a certificate, make sure you use a recent version of makecert as some older versions I had lying about did not like the option for certificate length, but failed silently!

makecert.exe -r -pe -n CN=mybackup -ss my -sr localmachine -eku 1.3.6.1.5.5.7.3.2 -len 2048 -e 04/24/2016 mybackup.cer

Now login to the Azure portal and upload this new certificate to your Backup vault. You are now ready to set up the backup.

Firstly set up the server using the certificate and your chosen encryption password.
$cert = Get-OBCertificateListFromLocalStore
$cert

CertificateThumbprint : <thumprint>
IssuedTo              : backup
IssuedBy              : backup
ExpirationDate        : 24/04/2016 00:00:00
IntendedPurpose       : Client Authentication

$item = Get-OBRecoveryService -Certificate $cert[0]
Start-OBRegistration  -RecoveryService $item[0]

ConvertTo-SecureString -String very_long_password_goes_here -AsPlainText -Force | Set-OBMachineSetting

Now set up the backup. here we are setting a backup of the directory d:\azure_backup, which I am robocopying files into during the day and the backup takes place each night at 2 am. The files are relatively small and static so robocopy ensures we only copy updated versions.

$mon = [System.DayOfWeek]::Monday
$tue = [System.DayOfWeek]::Tuesday
$wed = [System.DayOfWeek]::Wednesday
$thu = [System.DayOfWeek]::Thursday
$fri = [System.DayOfWeek]::Friday
$sat = [System.DayOfWeek]::Saturday
$sun = [System.DayOfWeek]::Sunday

$policy = New-OBPolicy
$filespec = New-OBFileSpec -FileSpec d:\azure_backup
$sched = New-OBSchedule -DaysofWeek Monday,Tuesday,Wednesday,Thursday,Friday,Saturday,Sunday -TimesofDay 02:00
$ret = New-OBRetentionPolicy -RetentionDays 30
Add-OBFileSpec -Policy $policy -FileSpec $filespec
Set-OBSchedule -policy $policy -schedule $sched
Set-OBRetentionPolicy -policy $policy -retentionpolicy $ret
Set-OBPolicy $policy

This next line sets the machine to throttle the uplink bandwidth to 10 meg during working hours and 50 meg outside working hours. This server does not use a proxy, although you can specify a proxy if that is how your system is set up. Remember most ADSL type connections have very limited uplink bandwidth compared to this fibre connection so if you are using ADSL make sure you do not swamp the uplink bandwidth as this will dramatically affect downloading as well!

Set-OBMachineSetting -WorkDay $mon, $tue, $wed, $thu, $fri, $sat, $sun -StartWorkHour "7:00:00" -EndWorkHour "22:00:00" -WorkHourBandwidth (10000*1024) -NonWorkHourBandwidth (50000*1024)

The first backup will take a long time, but subsequent backups will be much shorter.

Wednesday 17 April 2013

Upgraded Server 2008 to 2008R2 Hyper-V backup failure

This setup is Server 2008 UPGRADED to Server 2008R2 + SP1 + updates. Backing up perfectly using WSB admin utility.
Hyper-V was installed and the VMs put on their own volume.
Backup started failing only when this volume was backed up. The error initially showed up as:
image
VSSAdmin list writers showed Hyper-V writer problem.
The source error was one I had never seen before:
No snapshots to revert were found for virtual machine….
After searching around the answer was to enable Automount on new volumes...
image
After that the backup started working perfectly again…
OS upgrades often leave these gotchas behind so it can be more time effective in the longer run for a fresh install.

Wednesday 27 February 2013

Windows 8 users confused over desktop / start menu

In the past few weeks I have helped users with new Windows 8 laptop. In every case they had used used Windows before and were very confused where the start menu had ‘gone to’. In one case they were accused of ‘doing something’ to the laptop when they showed it to a relative as the relative thought they had broken Windows. The more I see Windows 8 the decision to remove the traditional start menu is just wrong on non touch devices. The uncomfortable jump between the desktop and the new touch oriented start menu does not feel correct. Why the old start menu was removed for Windows 8 RTM is still a mystery for me. The whole thing could be fixed so easily with a few lines of code that presumably and deliberately was never written by MS.
If touch screen then go to new start menu
Otherwise
Go to desktop and add Windows 7 Start Menu

Then allow more experienced users to select either of the above as default behaviour. The default behaviour may need to change if the user uses lots of Windows 8 Apps instead of traditional Apps.
The final icing on the cake was I asked a Windows user of 10 years+ to turn the laptop off, and they couldn't work out how to do it. I showed them that the charms bar, ALT F4 and CTRL-ALT-DEL could be used, but they were still amazed at the lack of usability. So here is a vote to do something about it. On my own laptop I have settled on the Classic Start Menu, whilst on my development Server running Server 2012 I have left the original behaviour to always remind myself how it works out of the box. It is not all bad as pressing the start key and typing the program you want to run still works the same, although I don’t see any advantage over Windows 7 Start Menu. If you use the Surface Touch Screen the new system works well, except non touch devices will be in the majority for some time to come. So here is a vote to change the behaviour of Windows 8 start menu on non touch devices. In a couple of years it will be less of a problem as touch screen devices will be ubiquitous. MS no doubt has already decided against this plan, although if business take up of Windows 8 is low they may be forced to rethink...

Friday 8 February 2013

Restore of physical Dell Server 2008R2 with Perc H700 to VM fails with stop 7b inaccessible boot device

I have come across this several times in the last year and decided to document the answer myself so I can find it more quickly next time! The best answer is to be found here if you want full details:
http://www.minasi.com/forum/topic.asp?TOPIC_ID=31980

The scenario is:
Take a windows backup of a Dell (and probably other manufacturers) Server 2008R2 with a PERC (or equivalent) disk controller:

eg wbadmin start backup -backuptarget:\\remoteserver\remoteshare -include:c:,d:,e:  -allcritical -vssfull

This creates a VHD(s).

Then go to Hyper-V and restore this backup to a new VM.

Lo and behold when you boot the VM you will get a stop 7b - inaccesible boot device.

Then mess around looking for the answer for half an hour.

The answer is simple.
Mount the VHD
Load the registry of your newly restored server.
Go to:
 HKLM\System\ControlSet001\Services\Intelide and change Start to 0
 HKLM\System\ControlSet001\Services\Pciide and change Start to 3

Restart - it boots perfectly. Why the bare metal restore cannot sort out the above automatically I just cannot fathom. This type of problem has been going on for ages and plagued moving XP/2003 disks to different hardware. The bizarre thing is that on NT4 you could manually 'preload' another driver (I admit the number of drivers was very limited!) and move to different hardware more simply - so much for plug and play! I should point out that using disk2vhd always works perfectly in the above scenario so it must have code to workaround the problem, although it does have the advantage of guaranteeing the hardware that will be presented to to the VM!

I have not tried to see if it is 'fixed' on Server 2012, but it probably isn't.

Sunday 3 February 2013

Beware SQL Developer/Enterprise feature add column on Standard edition

I always add new columns to SQL Server 2012 on large tables out of hours due to the extended table locks required. Factor in merge replication which I use heavily and is usually impossible to recover from schema failure (mitigated potentially by sp_markpendingschemachange) and I usually take a full backup of publisher and all subscribers (which are fixed servers, not mobile clients making it simple) before making schema changes.

A developer said they didn't understand the problem as schema changes on their 'dev' database only took a second. I had to explain that the Production database was on SQL Server 2012 Standard Edition and they were using SQL Server 2012 Developer Edition, and that Developer Edition is the same as Enterprise Edition, including the feature that allows Enterprise Edition to add columns almost instantly for most standard schema changes. here is an extract from SQL Server 2012 BOL.


Adding NOT NULL Columns as an Online Operation
In SQL Server 2012 Enterprise Edition, adding a NOT NULL column with a default value is an online operation when the default value is a runtime constant. This means that the operation is completed almost instantaneously regardless of the number of rows in the table.


A standard new column line syntax for a merge replicated table might be:
alter table mytable add mycolumn int constraint de_myconstraint default 0 not null
where the new column is defined in 1 DDL operation.

This could easily catch out unwitting users who try out a schema change on their 'dev' database, finds it completes subsecond and then try the same on Standard Edition and wait around for a potentially long time waiting for the schema locks to be dropped.
This Developer Edition vs Standard Edition problem has continued to get worse over the years and the feature difference is now very large.
If only MS would put a trace flag or other feature to allow Developer edition to behave as Standard Edition or other editions things would be much simpler all round.

Thursday 24 January 2013

Install Sysinternals suite in environment path variable to save time

One thing I have been doing for years and saved me lots of troubleshooting time is to install the Sysinternals Suite on every server and put that directory into the path environment variable so you can run any Sysinternals application straight from any CMD or Powershell prompt.
At the time of writing the download link was:
http://download.sysinternals.com/files/SysinternalsSuite.zip

I always uncompress the sysinternals zip file and install them in the same directory (eg c:\sysinternals) for consistency. Once you have installed them add the path of that directory to the Windows path environment variable.

Run this in an elevated Powershell prompt to add your new path to the current path.
[Environment]::SetEnvironmentVariable("Path",$Env:Path + ";C:\sysinternals", "Machine")

Now if you (or anybody else) opens a command prompt or Powershell prompt you can run your favourite Sysinternals commands without navigating to the directory.

Since you always install to the same directory it is easy to update all the servers from a central repository by robocopying a new installation to overwrite the old one. The great thing about Sysinternals tools is that they run without any installation.

It goes without saying that tools of this power so easily run can be a significant security risk so make sure you weigh up the pros and cons in your own environment before 'installing' them.

Tuesday 22 January 2013

Openfiles Maintain Objects List global flag behaviour change in Server 2012?

The very useful command openfiles documented below:
http://technet.microsoft.com/en-us/library/cc732490.aspx
Needs the Maintain Objects List global flag set AND A RESTART to track local file handles in Server 2008R2, and is turned off by default to reduce overhead, although I have never seen documented how much overhead it may generate.
However in Server 2012 the command generates a strange message on checking the status.
Server 2008 R2
image
Server 2012
image
I suspect the flag is pre-set to ON in Server 2012, and the reason it is on by default is that the new Server 2012 Powershell cmdlet Get-SmbOpenFile requires the same code under the covers and MS wanted it to work straight out of the box. If that is the case it does make you wonder how much overhead is really involved by setting it on… If anyone knows any more about this behaviour please comment.

Friday 11 January 2013

Powershell makes AD replication status monitoring trivial

Powershell is making monitoring various components much easier than in previous versions of Windows without third party tools.

A great example of this is Active Directory replication. With 15 sites to keep track of I run these commands on a schedule task every 15 mins to create a handy text file on the desktop of the state of AD replication.
repadmin /replsum > replsum.txt
repadmin /showrepl > showrepl.txt

So if there are any issues I can get a quick overview of the summary and detail state of AD replication.

But what I am really interested in knowing is when there is an active directory replication failure for further investigation. This can be achieved and eg emailed in a timely fashion very easily in a few lines of Powershell.
The commands below run repadmin and put the results into an array, then parse the array looking for lines with errors by seeing if the lines are longer in length than lines having no error text. Not pretty but it works well. Note the code -like *dc* which only checks lines with the letters dc contained in them and ignoring any other lines. If all your domain controllers have some other naming convention you would need to change that.


$replsumerror = $false
$arrayreplsum = repadmin /replsum
for($i = 0; $i -le $arrayreplsum.length -1; $i++) {if(($arrayreplsum[$i].length -gt 57) -and ($arrayreplsum[$i] -like "*dc*" )) {$replsumerror = $true}}

if ($replsumerror -eq $true) ...do some event  - eg email


I add this Powershell script on a 15 minute scheduled task and we get easy AD replication monitoring.


I have been disappointed with new Server 2012 Get-ADReplicationFailure cmdlet as it always returns the last error, which could be seconds or years ago. It could do with an option to show only 'current errors'
in a similar fashion to the repadmin /replsum /errorsonly switch or something similar.