Troubleshooting - Disappering configuration.mof File

Symptoms:

   Recently we noticed an unexpected behavior of Hardware Inventory settings applied in the configuration.mof file on one of our SCCM instances. None of the WMI classes introduced into the file were collected and upon re-checking the file it turned out, that any newly added section were being removed and the file was reverted to it's original form. Interestingly enough, the section added did not contain any errors, as this was happening even in case of adding additional spaces or new lines to the file - they would disappear as well.

Resolution:

    My first suspicion was to verify the extension added by the custom HP inventory module solution, which was only applied on this particular instance and added the following section at the end of the configuration.mof file:

Extension added to configuration.mof file by a custom HP plugin

    Removing this section however didn't take an effect either - the file was restored and the section re-appeared. An action was reflected in the dataldr.log file with the following entries:

MofCompiler failed:
parsing error
Failed to confert MOF class/poperties to policy; a retry will be attemted in 30 seconds...

Errors in the dataldr.log file

    Despite no apparent syntax problems the file refused to get parsed. Even voiding each section one by one didn't fix the problem - an empty file would get removed and restored from backup too. What helped to pinpoint the problem was running file comparison tool FC.exe - against another configuration.mof file from the properly working environment. This revealed the following phenomenon:

Errors in the dataldr.log file

    Apparently the beginning of the file was taken by some special character invisible either notepad or any other editing tool that was available. The problem was fixed by replacing the file with a properly working one and transplanting custom sections from the old faulty one. Looking at the history of the changes it seems that the latest accepted change to the file was accepted after adding the custom HP SCCM plugin

Troubleshooting - Unusual Error during Linux Agent Deployment

Symptoms:

   During the installation of SCOM agent on RHEL servers I encountered the following error message during the process of signing agent's certificate:

Exception message: Unable to create certificate context
; {ASN1 bad tag value met.
}

    The message quite unusual - apart from one previous case I could not find any other reference to this problem associated in any way with SCOM.

Resolution:

    The only suggested solution - a firewall problem has been ruled out in first place. After trying several approaches it turned out, that during certificate signing process, SCOM agent was trying to use the older versions of two particular libraries, that the ones present on the system, and failed due to this. The workaround applied was creation of the symbolic links named as the old library file pointing to the new files with the following commands and manually re-initiating certificate signing process:

cd /usr/lib
sudo ln -s libcrypto.so.1.0.1e libcrypto.so.1.0.0
sudo ln -s libssl.so.1.0.1e  libssl.so.1.0.0
sudo /opt/microsoft/scx/bin/tools/scxsslconfig -f -v

    Following up on the threads suggesting this approach (even though for a different problem) I figured out, that the problems reported to have been fixed with that script were mitigated with the release of next Cumulative Update for Management Pack for UNIX and Linux Operating Systems. After verification it turned out, that the agent binaries were taken from SCOM 2012 R2 Sp1 iso and didn't contain the latest fixes applied to the Management Pack. After downloading the latest version of the binaries the "ASN1 bad tag value" problem disappeared for all the Linux servers

Troubleshooting - Disappearing Run As Profiles Configuration Settings

Symptoms:

   Sometimes you have a general feeling, that there is something wrong with the infrastructure, and by looking around you catch the symptoms one after another until you are able to compose an overall image of the problem. This is what happened in this case I had with one of the customers recently, that has been resolved together with Microsoft Premier Support. It seems very interesting though, and that's why I have decided to share it with you. Here are all the symptoms observed before pinning the problem down, in more or less chronological order:

1. The groups created in the SCOM were not available for choice in the reports. They appeared in the console, but not in the Reporting part of SCOM (which suggests problems with processing data from Ops DB to DataWarehouse DB)
2. Big amount of data stored in the Staging area of the DataWarehouse DB. Running the following T-SQL query revealed hundreds of thousands of rows in the Alert and State parts of the Staging area

SELECT count(*) from Alert.AlertStage
SELECT count(*) from Event.EventStage
SELECT count(*) from Perf.PerformanceStage
SELECT count(*) from State.StateStage

3. Data Warehouse Data Collection State errors showing up in the Health Explorer of Management Servers themselves in SCOM

4. Large amount of 31551 events in SCOM event viewer log informing about failures while storing data into Data Warehouse. They look similar to the following event:

Log Name:      Operations Manager

Source:        Health Service Modules

Date:          27/01/2013 22:00:15

Event ID:      31551

Task Category: Data Warehouse

Level:         Error

Keywords:      Classic

User:          N/A

Computer:      XXX

Description:

Failed to store data in the Data Warehouse. The operation will be retried.

Exception 'SqlException': Management Group with id 'VVVVVVVV-VVVV-VVVV-VVVV-VVVVVVVVVVVV' is not allowed to access Data Warehouse under login 'YYY\WRITER'

 

One or more workflows were affected by this. 

 

Workflow name: Microsoft.SystemCenter.DataWarehouse.CollectPerformanceData

Instance name: XXX

Instance ID: {WWWWWWWW-WWWW-WWWW-WWWW-WWWWWWWWWWWW}

Management group: ZZZ

Reason:

    It turns out, that we suffered from an issue, that Microsoft admitted to be kind of a bug, which seems to randomly occur in different environments. It turns out, that on rare occasions default configuration of SCOM Run As accounts for Data Warehouse created during the installation of SCOM servers might disappear from Run As profiles configuration. The root cause of this behavior unfortunately hasn't been yet identified by Microsoft.

Resolution:

    In order to resolve the problem you have to re-introduce the settings once again. Below you can find the screenshots of properly configured Data Warehouse Account and Data Warehouse Report Deployment Account Run As profiles

Data Warehouse Run As Profiles default configuration

     After re-introducing the configuration everything should get back to normal.

Advanced Troubleshooting - SCCM 2007 WMI permissions issue

Symptoms:

    I decided to share with you one of the old, but also most cryptic cases that I have resolved some time ago due to the fact there wasn't any related trace that I could find in the Internet showcasing this issue.

    The problem has been observed several times in SCCM 2007 infrastructure in the past and it usually occurred after the power outage or any unexpected reboot of an SCCM server. After the system is back up and running users, who actually are configured to have access to SCCM are welcomed with the following view when starting the console:

Configuration Manager Console error

 

    Another interesting aspect of the problem was, that the local admins could still access SCCM without any problem. Issue affected only SCCM admins without the local admin rights on the SCCM server. The temporary workaround was to provide such privileges to all users, but obviously it could not be considered a permanent solution.

Reason:

    After numerous in depth investigations it turned out, that the root cause of the problem were missing permissions to WMI namespaces. The reason for the loss of such permissions remains unknown, but the problem is reproducible.

Resolution:
      In order to fix the problem the following actions have to be performed:

1. Access Start Menu, open Run prompt, and execute mmc command
2. Once the Console is opened press Ctrl+M in order to add the snap-in
3. Navigate down, choose WMI Control and click on Add button
4. Choose to connect to the local computer and Click on Ok button

5. Click again on the Ok button
6. Click on the arrow next to WMI Control (local) Snap-in and when it disappears right-click on the Snap-In and access the properties

 7. Access the Security tab and drill down all the way to root\sms and root\sms\site_XXX namespaces. It's them that are missing the permissions

   

    The permissions have to be set up in a proper way in order to allow SMS Admins group access SCCM Console again and perform delegated actions. The settings can be retrieved from a fresh SCCM 2007 installation by comparison. Or you can use below proper configuration's screenshots as a reference

 

Proper configuration of WMI permissions for the root\sms namespace

Proper configuration of WMI permissions for the root\sms\site_XXX namespace

Advanced Troubleshooting - NLB Address for SCOM Console

Symptoms:

    After the installation of a brand new SCOM environment one of the lasts steps before handing the system over to the production was to ensure, that the SCOM Console is available for all the users via Network Load Balancing address in both Web and thick client versions. The Active Directory team set up appropriate DNS entry and the Network team set up the load balancing....

    After the configuration was completed it turned out, that even though the web console was working properly the application one behaved very strange. It was connecting properly from any place to the direct servers' FQDN addresses without any authentication prompt, but when directed to the NLB address it didn't recognize the user account and was asking for credentials. When provided with the valid ones it was throwing the following error:

Operations Manager Console error when connecting via NLB address

    The whole output of the error message is below:

====================================================

Date: 12/8/2016 5:19:09 PM
Application: Operations Manager
Application Version: 7.0.9538.1136
Severity: Error
Message: Failed to connect to server 'XXX'

Microsoft.EnterpriseManagement.Common.ServerDisconnectedException: The client has been disconnected from the server. Please call ManagementGroup.Reconnect() to reestablish the connection. ---> System.ServiceModel.ProtocolException: You have tried to create a channel to a service that does not support .Net Framing. It is possible that you are encountering an HTTP endpoint. ---> System.IO.InvalidDataException: Expected record type 'PreambleAck', found '72'.
   --- End of inner exception stack trace ---

Server stack trace:
   at System.ServiceModel.Channels.FramingDecoder.ValidatePreambleAck(FramingRecordType foundType)
   at System.ServiceModel.Channels.ClientDuplexDecoder.Decode(Byte[] bytes, Int32 offset, Int32 size)
   at System.ServiceModel.Channels.ConnectionUpgradeHelper.ValidateUpgradeResponse(Byte[] buffer, Int32 count, ClientFramingDecoder decoder)
   at System.ServiceModel.Channels.ConnectionUpgradeHelper.InitiateUpgrade(StreamUpgradeInitiator upgradeInitiator, IConnection& connection, ClientFramingDecoder decoder, IDefaultCommunicationTimeouts defaultTimeouts, TimeoutHelper& timeoutHelper)
   at System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.SendPreamble(IConnection connection, ArraySegment`1 preamble, TimeoutHelper& timeoutHelper)
   at System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.DuplexConnectionPoolHelper.AcceptPooledConnection(IConnection connection, TimeoutHelper& timeoutHelper)
   at System.ServiceModel.Channels.ConnectionPoolHelper.EstablishConnection(TimeSpan timeout)
   at System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.CallOnceManager.CallOnce(TimeSpan timeout, CallOnceManager cascade)
   at System.ServiceModel.Channels.ServiceChannel.EnsureOpened(TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]:
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at Microsoft.EnterpriseManagement.Common.Internal.IDispatcherService.Connect(SdkClientConnectionOptions connectionOptions)
   at Microsoft.EnterpriseManagement.Common.Internal.SdkDataLayerProxyCore.Initialize(EnterpriseManagementConnectionSettings connectionSettings, SdkChannelObject`1 channelObjectDispatcherService)
   at Microsoft.EnterpriseManagement.Common.Internal.SdkDataLayerProxyCore.CreateEndpoint[T](EnterpriseManagementConnectionSettings connectionSettings, SdkChannelObject`1 channelObjectDispatcherService)
   --- End of inner exception stack trace ---
   at Microsoft.EnterpriseManagement.Common.Internal.ExceptionHandlers.HandleChannelExceptions(Exception ex)
   at Microsoft.EnterpriseManagement.Common.Internal.SdkDataLayerProxyCore.CreateEndpoint[T](EnterpriseManagementConnectionSettings connectionSettings, SdkChannelObject`1 channelObjectDispatcherService)
   at Microsoft.EnterpriseManagement.Common.Internal.SdkDataLayerProxyCore.ConstructEnterpriseManagementGroupInternal[T,P](EnterpriseManagementConnectionSettings connectionSettings, ClientDataAccessCore clientCallback)
   at Microsoft.EnterpriseManagement.Common.Internal.SdkDataLayerProxyCore.RetrieveEnterpriseManagementGroupInternal[T,P](EnterpriseManagementConnectionSettings connectionSettings, ClientDataAccessCore callbackDispatcherService)
   at Microsoft.EnterpriseManagement.Common.Internal.SdkDataLayerProxyCore.Connect[T,P](EnterpriseManagementConnectionSettings connectionSettings, ClientDataAccessCore callbackDispatcherService)
   at Microsoft.EnterpriseManagement.ManagementGroup.InternalInitialize(EnterpriseManagementConnectionSettings connectionSettings, ManagementGroupInternal internals)
   at Microsoft.EnterpriseManagement.Mom.Internal.UI.Common.ManagementGroupSessionManager.Connect(String server, String username, SecureString password, String domain)
   at Microsoft.EnterpriseManagement.Monitoring.Console.Internal.ConsoleWindowBase.ConnectWithCredentials(Exception ex, ConsoleJobEventArgs args)
System.ServiceModel.ProtocolException: You have tried to create a channel to a service that does not support .Net Framing. It is possible that you are encountering an HTTP endpoint. ---> System.IO.InvalidDataException: Expected record type 'PreambleAck', found '72'.
   --- End of inner exception stack trace ---

Server stack trace:
   at System.ServiceModel.Channels.FramingDecoder.ValidatePreambleAck(FramingRecordType foundType)
   at System.ServiceModel.Channels.ClientDuplexDecoder.Decode(Byte[] bytes, Int32 offset, Int32 size)
   at System.ServiceModel.Channels.ConnectionUpgradeHelper.ValidateUpgradeResponse(Byte[] buffer, Int32 count, ClientFramingDecoder decoder)
   at System.ServiceModel.Channels.ConnectionUpgradeHelper.InitiateUpgrade(StreamUpgradeInitiator upgradeInitiator, IConnection& connection, ClientFramingDecoder decoder, IDefaultCommunicationTimeouts defaultTimeouts, TimeoutHelper& timeoutHelper)
   at System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.SendPreamble(IConnection connection, ArraySegment`1 preamble, TimeoutHelper& timeoutHelper)
   at System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.DuplexConnectionPoolHelper.AcceptPooledConnection(IConnection connection, TimeoutHelper& timeoutHelper)
   at System.ServiceModel.Channels.ConnectionPoolHelper.EstablishConnection(TimeSpan timeout)
   at System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.CallOnceManager.CallOnce(TimeSpan timeout, CallOnceManager cascade)
   at System.ServiceModel.Channels.ServiceChannel.EnsureOpened(TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]:
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at Microsoft.EnterpriseManagement.Common.Internal.IDispatcherService.Connect(SdkClientConnectionOptions connectionOptions)
   at Microsoft.EnterpriseManagement.Common.Internal.SdkDataLayerProxyCore.Initialize(EnterpriseManagementConnectionSettings connectionSettings, SdkChannelObject`1 channelObjectDispatcherService)
   at Microsoft.EnterpriseManagement.Common.Internal.SdkDataLayerProxyCore.CreateEndpoint[T](EnterpriseManagementConnectionSettings connectionSettings, SdkChannelObject`1 channelObjectDispatcherService)
System.IO.InvalidDataException: Expected record type 'PreambleAck', found '72'.

 ====================================================

Reason:

    The error message is very uncommon for SCOM configuration and I haven't found any trace of the similar error occurring in regards to the SCOM Console application. There is no further need to any configuration on the SCOM side of things in order to set up Console NLB. Neither firewalls seemed to be a problem, as the same behavior occurred locally on the server and moreover there was no trace of dropped traffic in the firewall logs system. However looking up particular parts of the error message provided some traces - apparently in some cases error message was caused by improper port redirection, for instance when outgoing port was different from the incoming one. That lead to the configuration of the NLB settings, which turned out to be a culprit

Resolution:

    After taking a closer look at the configuration of NLB it turned out, that even though:

1. Traffic IP group was set up correctly and was pointing to a proper NLB address
2. The Pool was configured correctly and contained all the SCOM server nodes inside
3. The Virtual Server was configured to forward port 5724

    There was a problem in the setup of Virtual Server. In general this configuration item looks as follows:

Problematic Configuration of NLB Virtual Server for SCOM Console Application

    As you can see, even though the forwarded port is properly setup to 5724, the Protocol is incorrectly configured as HTTP. It turned out this was the source of the problem. It was fixed by setting it to Generic Client First, which caused the error message to go away.

Troubleshooting - Microsoft KB3192391 crashing SCOM console

Symptoms:

    The SCOM console suddenly started crashing across the whole organization and it was not available for any of the users. It happens on all the computers - including end user workstations as well as the consoles ran directly on the Management Servers. In some cases application crashed right away after starting, in some other it was possible to browse a little bit through the menu and it would crash or freeze after a few clicks. When this happens you might observe all or some of the following error messages in the Application Event Viewer logs:

====================================================

Faulting application name: Microsoft.EnterpriseManagement.Monitoring.Console.exe, version: 7.0.9538.1123, time stamp: 0x54314363
Faulting module name: clr.dll, version: 4.0.30319.36365, time stamp: 0x579fd37e
Exception code: 0xc0000005
Fault offset: 0x00000000000919df
Faulting process id: 0x115c4
Faulting application start time: 0x01d24180b6dc8014
Faulting application path: C:\Program Files\System Center Operations Manager 2012\Console\Microsoft.EnterpriseManagement.Monitoring.Console.exe
Faulting module path: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll
Report Id: 35f64cec-ad74-11e6-b4f6-0030846dd8a4
Application: Microsoft.EnterpriseManagement.Monitoring.Console.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an internal error in the .NET Runtime at IP 000007FEF33019DF (000007FEF3270000) with exit code 80131506.

 ====================================================

 The program Microsoft.EnterpriseManagement.Monitoring.Console.exe version 7.0.9538.1123 stopped interacting with Windows and was closed. To see if more information about the problem is available, check the problem history in the Action Center control panel.
 Process ID: 121b0
 Start Time: 01d2418008176db6
 Termination Time: 27
 Application Path: C:\Program Files\System Center Operations Manager 2012\Console\Microsoft.EnterpriseManagement.Monitoring.Console.exe
 Report Id: f104753a-ad73-11e6-b4f6-0030846dd8a4

The protocol host process 21628 did not respond and is being forcibly terminated {filter host process 67228}.

====================================================

 Faulting application name: Microsoft.EnterpriseManagement.Monitoring.Console.exe, version: 7.0.9538.1136, time stamp: 0x559bf219

Faulting module name: KERNELBASE.dll, version: 6.1.7601.23543, time stamp: 0x57d2fe27
Exception code: 0xe0434352
Fault offset: 0x000000000001a06d
Faulting process id: 0x13a2c
Faulting application start time: 0x01d241a75ac7a3f6
Faulting application path: C:\Program Files\System Center Operations Manager 2012\Console\Microsoft.EnterpriseManagement.Monitoring.Console.exe

Faulting module path: C:\Windows\system32\KERNELBASE.dll
Report Id: dedc128e-ad9a-11e6-b4f6-0030846dd8a4

==================================================== 

Reason:

    It turns out, that this issue started approximately at the same moment, at which October 2016 cumulative updates for Windows have been deployed to the servers and workstations. The cause of the error is the installation of Security Update for Microsoft Windows (KB3192391), which has been confirmed by Microsoft here:

https://support.microsoft.com/en-us/kb/3200006


Resolution:

    In order to fix the problem you have to download and install the fix from the Microsoft Update Catalog web page:

http://www.catalog.update.microsoft.com/Search.aspx?q=3200006

Additional Notes:

     The version of the installed fix has to match the version of the operating system on which the SCOM console application is crashing