Friday, November 4, 2011

Instant OCS RealTime MessageSending Service Dies Unexpectedly on Servers with FIPSAlgorithmPolicy Enabled

When deploying Instant OCS RealTime recently for a customer, we encountered an interesting bug.  Installation progressed smoothly, aside from some AD/SQL Server configuration issues.  However, when we attempted to start the services, while the MessageCapture service worked fine, the MessageSending service would silently die after a few seconds.  Examining the log files provided no insight, as there were no logged exceptions.  At the time, there was no global exception handler when the service is run as a service,

However, when running in TEST mode, we do have a global uncaught exception handler registered.  Running the service via the Debug UI enabled us to see the exception which was killing the service, both in our logs and in a MessageBox.
System.TypeInitializationException: The type initializer for 'Microsoft.Rtc.Collaboration.Presence.PresenceCategory' threw an exception. ---> System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.InvalidOperationException: This implementation is not part of the Windows Platform FIPS validated cryptographic algorithms.
   at System.Security.Cryptography.MD5CryptoServiceProvider..ctor()
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle._InvokeConstructor(Object[] args, SignatureStruct& signature, IntPtr declaringType)
   at System.Reflection.RuntimeConstructorInfo.Invoke(BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Security.Cryptography.CryptoConfig.CreateFromName(String name, Object[] args)
   at System.Security.Cryptography.MD5.Create()
   at Microsoft.Rtc.Collaboration.Presence.PresenceCategory..cctor()
   --- End of inner exception stack trace ---
   at Microsoft.Rtc.Collaboration.Presence.PresenceCategory..ctor(String name)
   at Microsoft.Rtc.Collaboration.Presence.CustomPresenceCategory..ctor(String categoryName, String dataXml)
   at Microsoft.Rtc.Collaboration.Presence.PresenceCategoryWithMetaData.InitializeMetaDataProperties(CategoryType category)
   at Microsoft.Rtc.Collaboration.Presence.PresenceCategoryWithMetaData.ParseCategoriesXml(XmlReader reader, String& uri)
   at Microsoft.Rtc.Collaboration.Presence.PresenceBatchSubscription.ParseNotificationData(Byte[] buffer, Int32 offset, Int32 count, SourceNetwork messageSource)
   at Microsoft.Rtc.Collaboration.Presence.PresenceBatchSubscription.ProcessNotification(SipMessageData message)
   at Microsoft.Rtc.Collaboration.Presence.PublishSubscribeSession.SipSubscriptionProcessor.ProcessNotification(SipMessageData message)
   at Microsoft.Rtc.Signaling.SipSubscription.InvokeProcessNotification(SipMessageData messageData)
   at Microsoft.Rtc.Signaling.ISubscriptionProcesorWorkitem`1.Microsoft.Rtc.Signaling.IWorkitem.Process()
   at Microsoft.Rtc.Signaling.WorkitemQueue.ProcessItems()
   at Microsoft.Rtc.Signaling.SerializationQueue`1.ResumeProcessing()
   at Microsoft.Rtc.Signaling.SerializationQueue`1.ResumeProcessingCallback(Object state)
   at Microsoft.Rtc.Signaling.QueueWorkItemState.ExecuteWrappedMethod(WaitCallback method, Object state)
   at System.Threading.ExecutionContext.runTryCode(Object userData)
   at System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode code, CleanupCode backoutCode, Object userData)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading._ThreadPoolWaitCallback.PerformWaitCallbackInternal(_ThreadPoolWaitCallback tpWaitCallBack)
   at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(Object state)
After some searching, we discovered that this is usually the result of having FIPS compliance checking enabled on the server.  See here.  And indeed, when we enabled FIPS checking in our test environment, we experienced the same behavior.

Fortunately, our customer was  not required to have FIPS checking enabled, so it was a simple matter to disable it.

  1. Open regedit on the machine RealTime is installed to.
  2. Navigate to HKEY_LOCAL_MACHINE/SYSTEM/CurrentControlSet/Control/Lsa/FipsAlgorithmPolicy
  3. Edit the Enabled key, and set it to 0.
  4. Close regedit
After disabling FIPS checking, the MessageSending service executed as expected.

Unfortunately, we do not yet have a solution to this problem if FIPS must be enabled.  We are investigating the option of using an updated version of the Unified Communications SDK, in the hopes that Microsoft has addressed this issue.  

No comments: