HCW8064 – HCW unable to perform OAuth | Free-busy issues

As part of an Exchange Server recovery, the Edge Servers were re-subscribed to the AD site and HCW was ran. The HCW did complete, but threw some warnings towards the end as shown below:

The warning HCW8056 was ignored as the receive connectors on both the Edge Transport servers were already setup correctly. At the same time, I was little bit sceptical about the warning HCW8064 and thought of going ahead with the post-checks. The link “learn more” pointed to this MS article.

After the mail flow checks were successful, next part was to perform free-busy checks. This is when I noticed that the free-busy from O365 to Exchange on-premises was failing with below error. The free-busy from Exchange on-premises to O365 worked fine however.

For free-busy to work correctly there are couple of checks performed by the Availability Service on Exchange Online in the following order :

a) Intra-Organization Connector [OAUTH] (for Exchange Server 2013 CU5+)

b) Organization Relationship [DAUTH]

c) Availability Address Space

To understand more about how hybrid free-busy works please read this Microsoft Exchange Team Blog.

Upon checking the Intra-Org connector status on cloud we noticed that it was showing as disabled.

This confirmed why the free-busy was failing and can be fixed by executing the below command on cloud:

Get-IntraOrganizationConnector | Set-IntraOrganizationConnector -Enabled $true

Make sure the Intra-Org connectors are enabled at the On-Prem side as well for On-Prem to cloud free-busy to work correctly.

Note: As a workaround, instead of enabling the Intra-Org connector on cloud you can update the TargetSharingEpr value on the Organization Relationship in cloud with the On-Premises Exchange servers EWS ExternalUrl endpoint.

Mail flow issues in Exchange Hybrid post SMTP certificate renewal

As an IT admin we often need to renew third party certificates in our customer environments. Recently, we had to renew our third party Exchange SMTP certificate installed on the Exchange 2013 Edge Transport servers and Hybrid servers. This certificate is used for the secure hybrid mail transport (we are running on Exchange 2013 hybrid).

Once the certificate renewal on the first Edge server (out of the two) was complete, we re-subscribed the Edge server to the AD site. Post this, the mandatory EdgeSync was also run. We then noticed that the inbound emails to Exchange on-premises from Office 365 started piling up on the problematic Edge server queue. Similarly, this Edge server was not able to send out emails to Office 365 as well. The queue also had an error as shown below:

454 4.7.5 The certificate specified in TlsCertificateName of the SendConnector could not be found.

We noticed another error in the event viewer of the Edge server:

Microsoft Exchange could not find a certificate that contains the domain name <I>CN=DigiCert SHA2 High Assurance Server CA, OU=www.digicert.com, O=DigiCert Inc, C=US<S>CN=***, O=***, L=***, S=***, C=*** in the personal store on the local computer. Therefore, it is unable to support the STARTTLS SMTP verb for the connector Default internal receive connector *** with a FQDN parameter of <I>CN=DigiCert SHA2 High Assurance Server CA, OU=www.digicert.com, O=DigiCert Inc, C=US<S>CN=***, O=***, L=***, S=***, C=***. If the connector’s FQDN is not specified, the computer’s FQDN is used. Verify the connector configuration and the installed certificates to make sure that there is a certificate with a domain name for that FQDN. If this certificate exists, run Enable-ExchangeCertificate -Services SMTP to make sure that the Microsoft Exchange Transport service has access to the certificate key.

Upon noticing these errors we suspected something wrong with the new SSL certificate installation, also comparing the old and new certificates it was identified that the attribute TlsCertificateName on the Edge server’s receive connector “Default internal receive connector” and the send connector “Outbound to office 365“ was still showing the old value while it was different in the new one. So it happened that when the certificate was requested from DigiCert customer portal the certificate  domain name information selected was not the same as the old one. Due to this the Edge server’s send/receive connector was not bound to the new certificate correctly.

This issue is documented in this MS article. So finally to solve this issue, we have to update the new certificate domain name values on the TlsCertificateName attribute of send/receive connector. This can be done as follows:

*Updating the TlsCertificateName attribute on the Office 365 Send Connector “Outbound to Office 365” (on Edge server in my case, MBX for environments without Edge) :-

$Cert = Get-ExchangeCertificate -Thumbprint <New Exchange Certificate>
$TLSCert = (‘<I>’+$cert.issuer+'<S>’+$cert.subject)
Set-SendConnector -Identity <Office 365 send Connector> -TLSCertificateName $TLSCert

*Updating the TlsCertificateName attribute on the Exchange On-Premises(Hybrid) Receive Connector which is receiving email from Office 365 “Default internal receive connector” (on Edge Server in my case, MBX for environments without Edge Server):-

$Cert = Get-ExchangeCertificate -Thumbprint <New Exchange Certificate>
$TLSCert = (‘<I>’+$cert.issuer+'<S>’+$cert.subject)
Set-ReceiveConnector -Identity <Office 365 to On-Prem> -TLSCertificateName $TLSCert

There are other scenarios as well were similar issues can happen due to an update with Microsoft certificate for the attribute TLSDomainCapabilities. These are discussed here. 

Once this is fixed run the Hybrid Configuration Wizard to update the new certificate for Office 365.

Error System.Net.WebException: The request failed with HTTP status 401: Unauthorized

Exchange 2013 on-premises users started complaining that they are not able to view the free/busy information of the office 365 users.

The issue started exactly after the expiry of our Lync OAuth third party certificate, even-though we had the new Lync OAuth certificate installed prior to the certificate expiry. This was identified when I had performed a fiddler trace while reproducing the issue in outlook. The logs found on fiddler is shown below:

Autodiscover failed for email address &lt; ******mail.onmicrosoft.com&gt;SMTP: ******mail.onmicrosoft.com with error System.Net.WebException: The request failed with HTTP status 401: Unauthorized.

I was able to find the same error in the Microsoft free/busy troubleshooting guide (Index 17) attached below as well:

FB_Errors.FixesV6

As suggested in the guide, the first thing we tried was to identify which is the certificate thumbprint being used for AuthConfig.

Get-ExchangeCertificate -Thumbprint (GetAuthConfig).CurrentCertificateThumbprint | fl

We found out that the thumbprint of the old certificate was still being used. Hence, we replaced it with new certificate thumbprint using below commands:

$a=get-date
Set-AuthConfig -NewCertificateThumbprint 1A39741F8DF58D4821567DD8F899B27410F7C096 NewCertificateEffectiveDate $a

Once this was executed, next step was to publish the new certificate

Set-AuthConfig –PublishCertificate

You can also remove any references to the old certificate using the command

Set-AuthConfig -ClearPreviousCertificate

Unfortunately, after performing all these steps the free/busy issue still persisted. We had to run the Hybrid Configuration Wizard to finally update the new Lync OAuth certificate.

Free/busy issue from on-premises to office 365 was fixed after running HCW. However, make sure you backup any customization’s on the send connectors prior to running HCW.

Running the new Office 365 Hybrid Configuration Wizard

As you all know, the latest Hybrid Configuration Wizard now runs from O365 irrespective of the previous versions of Exchange where HCW is embedded with the on-prem product. With this addition you will get the latest wizard every time you download it, which means that  you don’t have to wait for the next CU to resolve issues with the current HCW.

I recently updated my Exchange 2013 SP1 to CU16 and is about to run the new wizard:

First, I have to enable Exchange Hybrid on the on-prem server.

You will be asked to log in to your O365 tenant.

The wizard will redirect to O365 sign in page.

Once logged in, click on Enable again and a new tab will open with the link to download the HCW.

Download and run the HCW tool.

Below is the launch page of the Office 365 HCW. Click Next to proceed.

The HCW detects the optimal on-prem server to be the Hybrid Server (in this case, its the Ex 2013 server). You can also manually select a server of your choice. Also specify the type of O365 Organization. Click Next.

Next you have to provide your windows and Office 365 tenant credentials. Once done, click Next to proceed.

 

In next stage, the wizard will gather and confirm the configuration information. Once the test is successful, click Next.

The wizard will ask how the hybrid environment has to be configured for mail flow. If your organization uses Edge servers, you may have to select the second option.

Click on the Advanced button to list additional features. You will then see a check box that says ‘Enable centralized mail transport‘, the description also provides information about what this feature does. If you enable this, all your e-mail flow will happen through the on-prem environment. Once the options are selected, click on Next.

Choose the on-prem Exchange server that should host the receive connector for secure mail transport. Click Next.

Now, choose the on-prem Mailbox server that should host the send connector for secure mail transport. Click Next.

Select the transport certificate to be used for secured mail flow trusted by an external CA. Click Next.

Specify your Organization FQDN for mail flow. Click Next.

Make sure your external URL’s are configured on all virtual directories prior to running the HCW. Click Next.

Since, I already have an Exchange 2010 SP3 Hybrid in place the wizard detects the same and asks to update the configuration. Click Update.

The configuration starts as shown below. Click Stop to cancel.

The HCW process completes. If any configurations are pending it will be shown as below. In this case, my endpoint (Exchange 2010 hybrid server) is offline due to which the notification was received.

The server was turned on and DNS records were confirmed. On re-running the HCW, no issues were reported indicating that procedure completed successfully.

at

 

Updating hybrid configuration failed with error ‘Subtask CheckPrereqs execution failed:Check Tenant Prerequisites’

I came across this error when running the Hybrid Configuration Wizard on my Exchange Server 2013 SP1 server. Detailed error :

Subtask CheckPrereqs execution failed: Check Tenant Prerequisites
Deserialization fails due to one SerializationException: Microsoft.Exchange.Compliance.Serialization.Formatters.BlockedTypeException: The type to be (de)serialized is not allowed:

Towards the end of the error, it asked to view the Hybrid Configuration log for more information. You can find the log in the following location of your Exchange 2013 server : C:\Program Files\Microsoft\Exchange Server\V15\Logging\Update-HybridConfiguration.

Upon searching on the issue, I came across a Microsoft article that says this issue occurs due to a recent change in Microsoft’s Exchange Online environment that prevents the Exchange 2013 HCW to run correctly. The issue can be resolved by installing the latest cumulative update. In my environment I had to download the CU6 update to resolve this issue.

Reference : https://support.microsoft.com/en-us/help/2988229/-subtask-checkprereqs-execution-failed-error-in-hybrid-configuration-wizard-for-exchange-server-2013

 

Converting an Office 365 Federated domain to Managed

My existing azure lab has an Exchange 2010 Hybrid set up with ADFS for single sign-on. I am planning to remove ADFS from the environment and use password sync instead.

First I should check if password sync is already enabled or not. I can check and confirm this from the Azure AD Connect application. Launch AAD Connect tool and check the current configuration :

To check the status of the domain you can use the following commands, once connected to Exchange Online using powershell:

Connect-MsolService -Credential $cred

Get-MsolDomain

The output will be similar to the below screenshot:

As you can see above, the domain ‘anishjohnes.ga‘ is ‘Federated’.

If you go to ADFS management -> Relaying Party Trust, you will notice a trust already set up with MS Office 365.

Now to convert the domain to ‘Managed’ execute the below command :

Convert-MsolDomainToStandard -DomainName <String> -PasswordFile <String>              -SkipUserConversion <Boolean>  [-Confirm] [-WhatIf] [<CommonParameters>]

Once the domain is converted to ‘Managed’ single sign-on will be no longer applicable, instead same sign-on will be applied. The trust with Microsoft Office 365 will be removed from Relaying Party Trust as well.

Installing Exchange 2013 in an Exchange 2010 SP3 Hybrid Environment

I have an Exchange 2010 SP3 hybrid set up in my lab, and is planning to install and an Exchange 2013 as an Hybrid server.

I directly ran the Exchange 2013 setup without performing any schema preps and received the following error

So, I went on to my DC server and tried executing the commands for Schema preps as shown below only to get another failure notification :

As I am already in Hybrid, Exchange requires me to run the prep command adjacent to the /TenantOrganizationConfig switch. You also have to generate a config xml file by connecting to your Exchange online tenant.

For this, connect to your Exchange online tenant through powershell and execute the below command :

Get-OrganizationConfig | Export-Clixml -Path MyTenantOrganizationConfig.XML

The xml file will be generated as shown above. Copy the xml file to C: of the server where you are running the prep command ie, the DC.

Make a note that instead of /PrepareSchema, we will use /PrepareAD to run the /TenantOrganizationConfig switch adjacent to the setup.

.\Setup.exe /PrepareAD /IAcceptExchangeServerLicenseTerms /TenantOrganizationConfig:C:\MyTenantOrganizationConfig.XML

The Exchange setup will complete successfully now. 🙂

Error “You haven’t configured a target delivery domain. Please choose the appropriate remote domain as the target delivery domain.”

The scenario being discussed here is an Exchange 2010 to Exchange Online (Office 365) migration. A hybrid configuration was set up resulting in the coexistence of Exchange 2010 SP3 and Office 365. The procedure selected for mailbox move was ‘Remote Move Migration’. However, an error was received while trying to move an on-premise Exchange 2010 mailbox to the cloud ie, Office 365.

While performing the remote move request, in the move settings, you will be asked to provide the ‘Target Delivery Domain’. When I click on Browse option all I get is the error above, indicating that no domains are configured.

This error can be resolved by following the below steps:

• In the Exchange Management Console, navigate to Organization Configuration -> Hub Transport   -> Remote Domains
• Add your domain name specifically along with the Default ‘*’ value

Now, right click the domain name entry you just created, and select Properties. Ensure that under the Office 365 Tenant Domain the tick mark is selected as shown below:

Create a new remote move request and confirm that the mailbox move is working fine now.