One thing I have been doing for years and saved me lots of troubleshooting time is to install the Sysinternals Suite on every server and put that directory into the path environment variable so you can run any Sysinternals application straight from any CMD or Powershell prompt.
At the time of writing the download link was:
http://download.sysinternals.com/files/SysinternalsSuite.zip
I always uncompress the sysinternals zip file and install them in the same directory (eg c:\sysinternals) for consistency. Once you have installed them add the path of that directory to the Windows path environment variable.
Run this in an elevated Powershell prompt to add your new path to the current path.
[Environment]::SetEnvironmentVariable("Path",$Env:Path + ";C:\sysinternals", "Machine")
Now if you (or anybody else) opens a command prompt or Powershell prompt you can run your favourite Sysinternals commands without navigating to the directory.
Since you always install to the same directory it is easy to update all the servers from a central repository by robocopying a new installation to overwrite the old one. The great thing about Sysinternals tools is that they run without any installation.
It goes without saying that tools of this power so easily run can be a significant security risk so make sure you weigh up the pros and cons in your own environment before 'installing' them.
Thursday, 24 January 2013
Tuesday, 22 January 2013
Openfiles Maintain Objects List global flag behaviour change in Server 2012?
The very useful command openfiles documented below:
http://technet.microsoft.com/en-us/library/cc732490.aspx
Needs the Maintain Objects List global flag set AND A RESTART to track local file handles in Server 2008R2, and is turned off by default to reduce overhead, although I have never seen documented how much overhead it may generate.
However in Server 2012 the command generates a strange message on checking the status.
Server 2008 R2
Server 2012
I suspect the flag is pre-set to ON in Server 2012, and the reason it is on by default is that the new Server 2012 Powershell cmdlet Get-SmbOpenFile requires the same code under the covers and MS wanted it to work straight out of the box. If that is the case it does make you wonder how much overhead is really involved by setting it on… If anyone knows any more about this behaviour please comment.
http://technet.microsoft.com/en-us/library/cc732490.aspx
Needs the Maintain Objects List global flag set AND A RESTART to track local file handles in Server 2008R2, and is turned off by default to reduce overhead, although I have never seen documented how much overhead it may generate.
However in Server 2012 the command generates a strange message on checking the status.
Server 2008 R2
Server 2012
I suspect the flag is pre-set to ON in Server 2012, and the reason it is on by default is that the new Server 2012 Powershell cmdlet Get-SmbOpenFile requires the same code under the covers and MS wanted it to work straight out of the box. If that is the case it does make you wonder how much overhead is really involved by setting it on… If anyone knows any more about this behaviour please comment.
Friday, 11 January 2013
Powershell makes AD replication status monitoring trivial
Powershell is making monitoring various components much easier than in previous versions of Windows without third party tools.
A great example of this is Active Directory replication. With 15 sites to keep track of I run these commands on a schedule task every 15 mins to create a handy text file on the desktop of the state of AD replication.
repadmin /replsum > replsum.txt
repadmin /showrepl > showrepl.txt
So if there are any issues I can get a quick overview of the summary and detail state of AD replication.
But what I am really interested in knowing is when there is an active directory replication failure for further investigation. This can be achieved and eg emailed in a timely fashion very easily in a few lines of Powershell.
The commands below run repadmin and put the results into an array, then parse the array looking for lines with errors by seeing if the lines are longer in length than lines having no error text. Not pretty but it works well. Note the code -like *dc* which only checks lines with the letters dc contained in them and ignoring any other lines. If all your domain controllers have some other naming convention you would need to change that.
$replsumerror = $false
$arrayreplsum = repadmin /replsum
for($i = 0; $i -le $arrayreplsum.length -1; $i++) {if(($arrayreplsum[$i].length -gt 57) -and ($arrayreplsum[$i] -like "*dc*" )) {$replsumerror = $true}}
if ($replsumerror -eq $true) ...do some event - eg email
I add this Powershell script on a 15 minute scheduled task and we get easy AD replication monitoring.
I have been disappointed with new Server 2012 Get-ADReplicationFailure cmdlet as it always returns the last error, which could be seconds or years ago. It could do with an option to show only 'current errors'
in a similar fashion to the repadmin /replsum /errorsonly switch or something similar.
A great example of this is Active Directory replication. With 15 sites to keep track of I run these commands on a schedule task every 15 mins to create a handy text file on the desktop of the state of AD replication.
repadmin /replsum > replsum.txt
repadmin /showrepl > showrepl.txt
So if there are any issues I can get a quick overview of the summary and detail state of AD replication.
But what I am really interested in knowing is when there is an active directory replication failure for further investigation. This can be achieved and eg emailed in a timely fashion very easily in a few lines of Powershell.
The commands below run repadmin and put the results into an array, then parse the array looking for lines with errors by seeing if the lines are longer in length than lines having no error text. Not pretty but it works well. Note the code -like *dc* which only checks lines with the letters dc contained in them and ignoring any other lines. If all your domain controllers have some other naming convention you would need to change that.
$replsumerror = $false
$arrayreplsum = repadmin /replsum
for($i = 0; $i -le $arrayreplsum.length -1; $i++) {if(($arrayreplsum[$i].length -gt 57) -and ($arrayreplsum[$i] -like "*dc*" )) {$replsumerror = $true}}
if ($replsumerror -eq $true) ...do some event - eg email
I add this Powershell script on a 15 minute scheduled task and we get easy AD replication monitoring.
I have been disappointed with new Server 2012 Get-ADReplicationFailure cmdlet as it always returns the last error, which could be seconds or years ago. It could do with an option to show only 'current errors'
in a similar fashion to the repadmin /replsum /errorsonly switch or something similar.
Tuesday, 27 November 2012
Exchange 2010 2 member DAG quorum problem after FSW reboot
Most DAG discussions circle around DAGs with umpteen servers spanning multiple datacentres etc, but I am sure that many DAG’s are much simpler, and are subject to different issues like the one discussed below.
Setup is Exchange 2010 - 2 member DAG with a domain controller (DC) as file share witness (FSW). Whilst the DC was rebooted the DAG would show typical expected error state:
When both Exchange DAG members are online there is no apparent problem and there appears to be nothing amiss, but if you then innocently restart the other DAG member in this state you will hit major problems as the databases on the remaining ACTIVE member will dismount as there is no longer quorum and Active Manager will stop the databases being remounted manually until the other DAG member is back up and everything is back in sync. This behaviour, while logical and ultimately to be expected, is disconcerting to say the least! If you are using one server as a primary and the other as a failover (ie one server has all the active copies of every database) then you have lost all service even though the primary server is 100% up. This scenario could give you more downtime than you might have been led to believe!!!, especially during patching or if you have a design whereby one of the DAG members and the FSW are running as VMs on the same host and the host is rebooted…
But after restart of the DC the cluster still showed an error on file share witness, despite the file share being available.
This was due to the cluster only checking the file share resource once per hour, and I reduced this to 15 minutes. Looking around the internet this article appears to summarise the problem using Exchange 2007 with clusters.
http://blogs.technet.com/b/timmcmic/archive/2009/01/22/exchange-2007-sp1-ccr-windows-2008-clusters-file-share-witness-fsw-failures.aspx
So now the exposure to a loss of quorum was down from 1 hour to 15 minutes after the DC/FSW reboot.
To reduce this further the answer was to manually bring the file share witness resource on line after a reboot. I scripted a ps1 file and put it in scheduled tasks on one of the DAG members to start the cluster resource if was both offline AND the file share witness file share could be accessed, which was the norm after a reboot.
ie IF clusterFSW resource offline AND can access FSW share then try and start cluster resource.
Why the cluster cannot check the FSW resource more frequently is a mystery to me, but I presume there are good reasons.
The motto of the story is to make sure 2 out of the three quorum members are available at all times…
Setup is Exchange 2010 - 2 member DAG with a domain controller (DC) as file share witness (FSW). Whilst the DC was rebooted the DAG would show typical expected error state:
When both Exchange DAG members are online there is no apparent problem and there appears to be nothing amiss, but if you then innocently restart the other DAG member in this state you will hit major problems as the databases on the remaining ACTIVE member will dismount as there is no longer quorum and Active Manager will stop the databases being remounted manually until the other DAG member is back up and everything is back in sync. This behaviour, while logical and ultimately to be expected, is disconcerting to say the least! If you are using one server as a primary and the other as a failover (ie one server has all the active copies of every database) then you have lost all service even though the primary server is 100% up. This scenario could give you more downtime than you might have been led to believe!!!, especially during patching or if you have a design whereby one of the DAG members and the FSW are running as VMs on the same host and the host is rebooted…
But after restart of the DC the cluster still showed an error on file share witness, despite the file share being available.
This was due to the cluster only checking the file share resource once per hour, and I reduced this to 15 minutes. Looking around the internet this article appears to summarise the problem using Exchange 2007 with clusters.
http://blogs.technet.com/b/timmcmic/archive/2009/01/22/exchange-2007-sp1-ccr-windows-2008-clusters-file-share-witness-fsw-failures.aspx
So now the exposure to a loss of quorum was down from 1 hour to 15 minutes after the DC/FSW reboot.
To reduce this further the answer was to manually bring the file share witness resource on line after a reboot. I scripted a ps1 file and put it in scheduled tasks on one of the DAG members to start the cluster resource if was both offline AND the file share witness file share could be accessed, which was the norm after a reboot.
ie IF clusterFSW resource offline AND can access FSW share then try and start cluster resource.
Why the cluster cannot check the FSW resource more frequently is a mystery to me, but I presume there are good reasons.
The motto of the story is to make sure 2 out of the three quorum members are available at all times…
Saturday, 24 November 2012
Windows 8 File History bandwidth stealer!
I recently turned on Windows 8 file history on my laptop and pointed the ‘backup’ to my main server using a UNC share. I wandered at the time if the system had any logic about running on low bandwidth/different subnet connections. I didn't need to wonder for long as next day at home with only an ADSL connection to the same server instead of the gigabit LAN all remote connections were running very slow. The router showed uplink bandwidth maxed out. I immediately suspected File History and turned the service off and the bandwidth was released. Despite this problem I did like the feature so I knocked up a quick and dirty solution so it only worked on the same subnet as the LAN.
First of all I removed the triggers from the File History Service so it couldn't start on any triggers. The triggers are Group Policy Triggers but I just removed them anyway and set the service to manual. In an elevated command window type
sc qtriggerinfo “fhsvc” delete
You should not remove these triggers without discussing with your syasdmins first!
So now the service was set to manual and would never autostart (or so I thought...).
I then created a very simple and crude powershell ps1 file that gets triggered at computer startup and sleeps for 5 minutes to allow everything else to startup. My Laptop has a DHCP reservation so it always gets the same address but you could do something much more sophisticated than below. I always prefer to pipe powershell to strings as you can work with strings more easily. My wired connection has been renamed to Ethernet so don’t copy this code directly.
The powershell looks something like this:
Start-Sleep -s 300
$IP_Info = Get-NetIPAddress | Where-Object {$_.InterfaceAlias -eq "ethernet" -and $_.AddressFamily -eq "ipv4" } | out-string
if ($IP_Info -match "200.1.1.250")
{
start-service "file history service"
}
else
{
stop-service "file history service"
}
Set-Service "fhsvc" -startuptype "manual"
The stop service is not required as it theoretically never starts but does not do any harm. Setting the startuptype to manual is required as I noticed the service always sets itself back to autostart whenever it is started!
This is saved in a .ps1 file and run as a high permissions scheduled task on computer startup, After 5 minutes it starts the service if on the same subnet as the server. I do not need to think any more about manual switching on and off.
I think there should be an option to limit the bandwidth/stop the service/multiple backup targets depending on subnet in certain connection scenarios. Maybe windows 8.5…
First of all I removed the triggers from the File History Service so it couldn't start on any triggers. The triggers are Group Policy Triggers but I just removed them anyway and set the service to manual. In an elevated command window type
sc qtriggerinfo “fhsvc” delete
You should not remove these triggers without discussing with your syasdmins first!
So now the service was set to manual and would never autostart (or so I thought...).
I then created a very simple and crude powershell ps1 file that gets triggered at computer startup and sleeps for 5 minutes to allow everything else to startup. My Laptop has a DHCP reservation so it always gets the same address but you could do something much more sophisticated than below. I always prefer to pipe powershell to strings as you can work with strings more easily. My wired connection has been renamed to Ethernet so don’t copy this code directly.
The powershell looks something like this:
Start-Sleep -s 300
$IP_Info = Get-NetIPAddress | Where-Object {$_.InterfaceAlias -eq "ethernet" -and $_.AddressFamily -eq "ipv4" } | out-string
if ($IP_Info -match "200.1.1.250")
{
start-service "file history service"
}
else
{
stop-service "file history service"
}
Set-Service "fhsvc" -startuptype "manual"
The stop service is not required as it theoretically never starts but does not do any harm. Setting the startuptype to manual is required as I noticed the service always sets itself back to autostart whenever it is started!
This is saved in a .ps1 file and run as a high permissions scheduled task on computer startup, After 5 minutes it starts the service if on the same subnet as the server. I do not need to think any more about manual switching on and off.
I think there should be an option to limit the bandwidth/stop the service/multiple backup targets depending on subnet in certain connection scenarios. Maybe windows 8.5…
Monday, 8 October 2012
Exchange 2010 DAG’s about to be cheaper under Windows Server 2012!
Warning: Exchange 2010 does not run on Windows Server 2012 when this article was first written, but will be engineered to do so.
Running Exchange 2010 DAGs always meant installing Windows Server 2008/R2 ENTERPRISE Edition, as DAGs require the clustering components that are not found in Windows Server 2008/R2 Standard Edition. Enterprise Edition is considerably more expensive than Standard Edition. It also allows much more RAM, as Windows Server 2008R2 Standard Edition is limited to 32 gb.
However, Windows Server 2012 Standard Edition includes unlimited memory and all the components previously reserved for Enterprise Edition. That means that a DAG running on Windows Server 2012 will be significantly cheaper than a DAG running on Server 2008/R2.
Running Exchange 2010 DAGs always meant installing Windows Server 2008/R2 ENTERPRISE Edition, as DAGs require the clustering components that are not found in Windows Server 2008/R2 Standard Edition. Enterprise Edition is considerably more expensive than Standard Edition. It also allows much more RAM, as Windows Server 2008R2 Standard Edition is limited to 32 gb.
However, Windows Server 2012 Standard Edition includes unlimited memory and all the components previously reserved for Enterprise Edition. That means that a DAG running on Windows Server 2012 will be significantly cheaper than a DAG running on Server 2008/R2.
Friday, 17 August 2012
An end to the scalable networking fiasco?
If you run a lot of Windows Servers chances are you will have come across the disgraceful debacle of 'Scalable networking', where the network card takes corrupts some of the load from the OS, thereby improving destroying performance.
This has been a shambles since it first appeared in Server 2003 (service pack 2 as I remember). Instead of realising performance increases you will infrequently but invariably come across some unfathomable glitch that makes a mockery of the whole technology.
When you first come across this 'feature' many hours will be spent trying to track down bizarre problems that just don't add up.
I have come across this on several occasions when I was naive to the problem, most notably when Exchange 2007 OWA clients (running on W2K3 sp2) would not load properly. MS support eventually suggested I turned off scalable networking and everything worked perfectly in an instant.
Other problems include SQL servers with a fraction of the network throughput and sporadic delays. Whenever we replace servers nowadays one of the items to check is the scalable network settings, since you can never tell what will happen with network card A on OS B until you have tried changing all the settings in your own environment. Performance can vary dramatically, and sometimes some settings make your system practically unusable. If in doubt turn it all off, both in Windows and the network card itself. This problem remains in Server 2008R2 running on the latest Dell servers, whereby you need to check which settings give best performance.
There is no logical reason for the sometimes massive performance variations. As I said disgraceful. This story from MS itself perfectly sums up this shameful time wasting technology...
http://blogs.msdn.com/psssql/archive/2010/02/21/tcp-offloading-again.aspx
Here are my own notes on this area when I install 2008R2 on our latest Dell Servers found by trial and error for our environment - mainly SQL and file traffic (we standardise on a model/network hardware as much as possible so we we are not having to do too much retesting in this area)
install broadcom package
netsh int tcp set global chimney=disabled
netsh int tcp set global rss=disabled
****Broadcom - defaults (ie scalable network settings on the NIC are ON - how bizarre is that)
****Check file copy speed etc
For reasons unknown I have always seen more problems in this area with Broadcom - whether this is statistically valid or not I don't know, but I now specify Intel network cards as they have always been my favourite since NT4 days, and disable the Broadcom network cards.
If you run into problems I recommend you change the settings on both the NIC and Windows itself as there is no correct setting - trial and error only I'm afraid.
I know the next time we buy another batch of servers (the NIC etc will be different model by then) we will need to try the settings again to see what works best.
I personally would put this shambles as the most disastrous Windows Server technology in the last decade. It should either be sorted out once and for all or simply be dropped. I have never worked out if it is a bug in Windows itself or the network card manufacturers at fault.
I have seen the potential performance gains that this technology can give but reliability has to take preference over potential performance gains.
Addendum
I am hoping Server 2012 will solve this problem once and for all as it has NIC teaming baked into the OS and it is likely a lot of work has been done fixing any glitches in this area. We will see...
This has been a shambles since it first appeared in Server 2003 (service pack 2 as I remember). Instead of realising performance increases you will infrequently but invariably come across some unfathomable glitch that makes a mockery of the whole technology.
When you first come across this 'feature' many hours will be spent trying to track down bizarre problems that just don't add up.
I have come across this on several occasions when I was naive to the problem, most notably when Exchange 2007 OWA clients (running on W2K3 sp2) would not load properly. MS support eventually suggested I turned off scalable networking and everything worked perfectly in an instant.
Other problems include SQL servers with a fraction of the network throughput and sporadic delays. Whenever we replace servers nowadays one of the items to check is the scalable network settings, since you can never tell what will happen with network card A on OS B until you have tried changing all the settings in your own environment. Performance can vary dramatically, and sometimes some settings make your system practically unusable. If in doubt turn it all off, both in Windows and the network card itself. This problem remains in Server 2008R2 running on the latest Dell servers, whereby you need to check which settings give best performance.
There is no logical reason for the sometimes massive performance variations. As I said disgraceful. This story from MS itself perfectly sums up this shameful time wasting technology...
http://blogs.msdn.com/psssql/archive/2010/02/21/tcp-offloading-again.aspx
Here are my own notes on this area when I install 2008R2 on our latest Dell Servers found by trial and error for our environment - mainly SQL and file traffic (we standardise on a model/network hardware as much as possible so we we are not having to do too much retesting in this area)
install broadcom package
netsh int tcp set global chimney=disabled
netsh int tcp set global rss=disabled
****Broadcom - defaults (ie scalable network settings on the NIC are ON - how bizarre is that)
****Check file copy speed etc
For reasons unknown I have always seen more problems in this area with Broadcom - whether this is statistically valid or not I don't know, but I now specify Intel network cards as they have always been my favourite since NT4 days, and disable the Broadcom network cards.
If you run into problems I recommend you change the settings on both the NIC and Windows itself as there is no correct setting - trial and error only I'm afraid.
I know the next time we buy another batch of servers (the NIC etc will be different model by then) we will need to try the settings again to see what works best.
I personally would put this shambles as the most disastrous Windows Server technology in the last decade. It should either be sorted out once and for all or simply be dropped. I have never worked out if it is a bug in Windows itself or the network card manufacturers at fault.
I have seen the potential performance gains that this technology can give but reliability has to take preference over potential performance gains.
Addendum
I am hoping Server 2012 will solve this problem once and for all as it has NIC teaming baked into the OS and it is likely a lot of work has been done fixing any glitches in this area. We will see...
Subscribe to:
Posts (Atom)