Exch 2013 EAC | eDiscovery page error “The request failed. The remote server returned an error: (400) Bad Request.”

We are running Exchange Server 2013 CU23 on all servers and recently when trying to perform an eDiscovery search in the Compliance Management section of the Exchange Admin Center it reported an error “The request failed. The remote server returned an error: (400) Bad Request.”

eDiscovery page error

I got this error as soon as I selected the Compliance Management section and even if I manage to create an eDiscovery search it would fail at the last step with the same error. The error was not specific to any server but happened across all the Exchange 2013 servers in the environment.

If I try to execute the cmdlet Get-MailboxSearch or New-MailboxSearch it would also return the same error. The error returned in Event Viewer is as shown below:

Microsoft.Exchange.Data.DataSourceOperationException: The request failed. The remote server returned an error: (400) Bad Request. ---> Microsoft.Exchange.WebServices.Data.ServiceRequestException: The request failed. The remote server returned an error: (400) Bad Request. ---> System.Net.WebException: The remote server returned an error: (400) Bad Request.

Upon researching about this error we came across this MS thread which discussed the same issue. We ran the below command on our multi-role Exchange 2013 server to check the existing values configured on the InternalNLBBypassUrl parameter of the EWS frontend and backend website:

Get-WebServicesVirtualDirectory -ShowMailboxVirtualDirectories -Server ServerName | fl identity,*url*

Incorrect configuration

As you see, the InternalNLBBypassUrl on the EWS Default Web Site and Back End website is populated with a value. However, this is different from the default configuration of the EWS virtual directories as per Microsoft. According to Microsoft, the correct configuration for the Default Web Site’s InternalNLBBypassUrl is “Null”. And since the Back End website listens on TCP 444 the format should be https://serverfqdn:444/ews/exchange.asmx. ie, it should look like below configuration:

Correct configuration

So to fix the issue, first correct the InternalNLBBypassUrl value on the EWS Default Web Site. Launch the Exchange Management Shell and execute the below command

Set-WebservicesVirtualDirectory -Identity “Servername\EWS (Default Web Site)” –InternalNLBBypassUrl $null

Secondly, correct the InternalNLBBypassUrl value on the EWS Back End website. This has to be done using PowerShell by loading the Exchange PowerShell Snapin. This is because running the same command on Exchange Management Shell will fail with an error mentioning that the operation is outside the users’s write scope. Also, keep in mind that modifying the back end website using this method maynot be supported by Microsoft.

Launch Windows Powershell as an Admin and execute the below commands:

Add-PsSnapin *Exch*

Set-WebservicesVirtualDirectory -identity “ServerName\EWS (Exchange Back End)” –InternalNLBBypassUrl https://serverfqdn:444/ews/exchange.asmx

Now run the command to confirm if the changes have been applied:

Get-WebServicesVirtualDirectory -ShowMailboxVirtualDirectories -Server ServerName | fl identity,*url*

If in case the changes are not applied yet, you can Recycle the MSExchangeServicesAppPool from IIS.

Navigate to the Exchange Admin Center -> Compliance Management and make sure that you are no longer receiving the 400 error code.

Repeat the same steps for all the Exchange Servers in your environment. Ideally, if this setting is corrected on the Exchange Server that hosts the Arbitration mailbox or Discovery Search mailbox, the issue should get fixed. However, if this server is not available for any reason the mailbox might move to a different server and then there are chances you might notice the error again. So it is better to make the same changes on all the servers.

To check the location of the Arbitration Mailbox and Discovery search mailbox you can use the below commands:

Get-Mailbox -RecipientTypeDetails DiscoveryMailbox | select name,database

Get-Mailbox *Disc* | fl name,alias,servername

Get-Mailbox -Arbitration | fl Name,Database

In my case, couple of Exchange Servers were recently recovered due to corruption and after the disaster recovery steps, the default values were changed. So if your environment has any Exchange Servers recovered recently, just make sure that these values are configured correctly.

Error “Policy does not allow granting permissions at this level to one or more of the recipient(s). Please select another permission level and send the sharing invite again.” | Exchange Hybrid

I recently had a user complain that he was not able to share his calendar information with his boss in Outlook. Upon checking I noticed that the affected user’s mailbox is hosted on Exchange Server 2013 and the boss’s mailbox is hosted on Office 365 cloud. Our environment is in an Exchange Hybrid scenario with Office 365. The user was able to share just the availability information but whenever he selected limited details or full calendar details the sharing failed.

Calendar sharing in Outlook

The error the user had received when selecting Limited details or Full details was as shown below:

Sharing error

I had already confirmed that free/busy is working cross premises as expected. When researching on this issue I came across this Microsoft article that suggested to select the recipient through Global Address List. However, this did not help me. Then finally I noticed another article here that suggested to check the Sharing Policy assigned on the user mailbox.

I ran the below command and identified that the user is assigned to the Default Sharing Policy.

Get-Mailbox user | fl *sharing*

Checking SharingPolicy

At this point I navigated to the Exchange 2013 Admin Center -> Organization Configuration. The Default Sharing Policy was found in the Individual Sharing section.

Default SharingPolicy

I then edited the Default Sharing Policy configuration and noticed that only free/busy information is allowed to be shared with all domains. This was the reason why the user was able to share Availability information only with his boss and not limited/full calendar details.

Default SharingPolicy configuration

Inorder to fix this issue and if your organization permits, you can either edit the Sharing with all domains sharing rule and select the appropriate permission level under Share your calendar folder and save the settings as shown below:

or

you can create a new sharing policy for the specific domain (in this case your Hybrid/Split domain) and select “All calendar appointment information, including time, subject, location and title”. This is the preferred option.

Custom SharingPolicy

Save the settings. This should fix the sharing issue.

Error “451 4.7.0 Temporary server error. Please try again later. PRX2”

We had to rebuild one of our Exchange Server 2013 recently by performing disaster recovery steps as a result of an OS corruption. Once the server was back in production we started noticing that the server was not accepting any new emails. In the Queue viewer of the Exchange server that was sending mails to this recovered server, we noticed the error “451 4.7.0 Temporary server error. Please try again later. PRX2.”.

We had confirmed that Telnet to the problematic server on port 25 was working fine as expected. The send connector protocol logs of the sending Exchange Server was checked and it showed “Queued mail for delivery”. This indicated that the sending server was trying to send the emails to the recovered problematic server. Next, we tried to send an email using Telnet to the problematic server and received the same error as in the Queue viewer “451 4.7.0 Temporary server error. Please try again later. PRX2.”. The screenshot is shown below:

At this point, we switched to the problematic server for further troubleshooting. The Default Frontend receive connector protocol logs also showed the same PRX2 error along with a DNS error “451 4.4.0 DNS query failed“.

We had a custom receive connector which was actually responsible for receiving the emails from the Exchange server, the protocol logs showed that connection is made but reported proxy and DNS errors:

The connectivity logs also had similar DNS retry errors. The application event logs on the server reported Event ID 16025 that the DNS Servers could not be contacted from the server’s network adapter with its respective adapter guid value. It also suggeted to run the Get-NetworkConnectionInfo command for more information:

The Get-NetworkConnectionInfo command returned the results of different Network adapters on the server and its configurations. To our surprise the Adapter guid value on the server did not match the Adpater guid value shown in the above screenshot. This prompted us to check the DNS Configurations on the FrontEndTransportService and TransportService by running the commands

Get-FrontendTransportService Servername | fl *dns*
Get-TransportService Servername | fl *dns*

It was then identified that the InternalDNSAdapterGuid value was still referencing the old server’s Adapter guid instead of the correct value diplayed when executing the Get-NetworkConnectionInfo. The DNS issues were finally resolved by mapping the InternalDNSAdapterGuid value to the correct guid of the server’s network adapter. This can be done by executing the command

Set-FrontEndTransportService ServerName -InternalDNSAdapterGuid {GUID}

Set-TransportService ServerName -InternalDNSAdapterGuid {GUID}

You can also assign the value of InternalDNSAdapterGuid to “00000000-0000-0000-0000-000000000000” so that internal DNS lookups are performed by using the DNS settings of any available network adapter.

More info on InternalDNSAdapterGuid parameter:

-InternalDNSAdapterGuid

The InternalDNSAdapterGuid parameter specifies the network adapter that has the DNS settings used for DNS lookups of servers that exist inside the Exchange organization. The concept of an internal network adapter and an external network adapter is only applicable in a multi-homed Exchange server environment. When no particular network adapter is specified as the network adapter for external DNS lookups, the value of the InternalDNSAdapterGuid parameter is 00000000-0000-0000-0000-000000000000, and internal DNS lookups are performed by using the DNS settings of any available network adapter. You may enter the GUID of a specific network adapter to use for internal DNS lookups. The default value of the InternalDNSAdapterGuid parameter is 00000000-0000-0000-0000-000000000000.

Reference:

Exchange – Event ID 205 and Event ID 16025

“The EdgeSync Cookie has not been updated as expected” | EdgeSync Error

After rebuilding an Exchange 2013 MBX/CAS server in the environment, the Edge Subscriptions were re-created so that the new Exchange server can also participate in the EdgeSync synchronization. Once the Active Directory information was synchronized to AD LDS and mail flow was confirmed to be working, the next step was to make sure that the EdgeSync was working as expected.

Upon running the Test-EdgeSynchronization command the SyncStatus showed as failed with the reason

“The EdgeSync Cookie has not been updated as expected. It may indicate EdgeSync has encountered synchronization errors. For more information, check the EdgeSync logs.”

The EdgeSync logs in the Exchange installation folder “\Program Files\Microsoft\Exchange Server\V15\TransportRoles\Logs\EdgeSync” were checked but did not report any issues.

Identified this technet article that discussed on the same issue and as per the article, the synchronization errors are due to stale entries in the ADAM database on the Edge Server. Below are the steps on how to clear these entries from ADSIEdit on the Edge Server:

1. Login to the Edge Server and execute the command

Type “dsdbutil” and then type “list instances”, keep a note of the output

2. On the Edge Server, launch ADSIEdit.msc and select Connect to. Now, below the ‘Select a well known Naming Context‘ drop down menu select Configuration. Also update the Edge Server name and LDAP port number (50389) from above below Computer section and click OK.

3. Expand Configuration -> Services -> Microsoft Exchange -> First Organization -> Administrative Groups -> Servers -> Edge Server

4. Right click the Edge Server object and locate the attribute msExchEdgeSyncCookies. Remove any old DC values and other stale entries from here. Click OK.

5. Login back to the internal MBX/CAS server and execute the command to start synchronization of Active Directory data to the Edge Server. This will take some time depending on the number of recipient objects in your organization.

Start-EdgeSynchronization -ForceFullSync -ForceUpdateCookie

More explanation on these switches are available here.

6. Now, re-run Test-EdgeSynchronization and make sure that it completed successfully.

HCW8064 – HCW unable to perform OAuth | Free-busy issues

As part of an Exchange Server recovery, the Edge Servers were re-subscribed to the AD site and HCW was ran. The HCW did complete, but threw some warnings towards the end as shown below:

The warning HCW8056 was ignored as the receive connectors on both the Edge Transport servers were already setup correctly. At the same time, I was little bit sceptical about the warning HCW8064 and thought of going ahead with the post-checks. The link “learn more” pointed to this MS article.

After the mail flow checks were successful, next part was to perform free-busy checks. This is when I noticed that the free-busy from O365 to Exchange on-premises was failing with below error. The free-busy from Exchange on-premises to O365 worked fine however.

For free-busy to work correctly there are couple of checks performed by the Availability Service on Exchange Online in the following order :

a) Intra-Organization Connector [OAUTH] (for Exchange Server 2013 CU5+)

b) Organization Relationship [DAUTH]

c) Availability Address Space

To understand more about how hybrid free-busy works please read this Microsoft Exchange Team Blog.

Upon checking the Intra-Org connector status on cloud we noticed that it was showing as disabled.

This confirmed why the free-busy was failing and can be fixed by executing the below command on cloud:

Get-IntraOrganizationConnector | Set-IntraOrganizationConnector -Enabled $true

Make sure the Intra-Org connectors are enabled at the On-Prem side as well for On-Prem to cloud free-busy to work correctly.

Note: As a workaround, instead of enabling the Intra-Org connector on cloud you can update the TargetSharingEpr value on the Organization Relationship in cloud with the On-Premises Exchange servers EWS ExternalUrl endpoint.

Exchange server 2013 recovery installation failure | [ERROR] The internal transport certificate for the local server was damaged or missing in Active Directory

Just recently, one of our Exchange 2013 MBX/CAS server crashed due to a kernel issue after windows updates installation. The only option ahead was to perform a recovery mode installation of the server. As this server was also a DAG member the steps discussed in this article were followed.

The recovery installation started failing at the Mailbox role stage with the below error:

[ERROR] The internal transport certificate for the local server was damaged or missing in Active Directory. The problem has been fixed. However, if you have existing Edge Subscriptions, you must subscribe all Edge Transport servers again by using the New-EdgeSubscription cmdlet in the Shell.

Our Exchange environment had two Exchange 2013 Edge Transport servers subscribed to a single AD site. As per the output received, it said the issue had been fixed and we had to re-subscribe our Edge Transport servers. However, we were not able continue with the Exchange installation.

Upon researching further on this issue online and with available logs from Event Viewer + Exchange setup logs, we understood that this error occur because the Exchange recovery procedure tries to re-encrypt the EdgeSync credentials used for Edge Synchronization process and since the self-signed certificate on this new Exchange Server 2013 is different from the self-signed certificate on the failed Exchange 2013 server encryption fails.

This issue can be fixed by clearing the old EdgeSync credential values from the Exchange 2013 server using ADSIEdit. You can do this by connecting to the Configuration Partition on your Domain Controller -> Services -> Microsoft Exchange -> Your Organization Name -> Administrative Groups -> Exchange Administrative Group (FYDIBOHF23SPDLT) -> Servers -> Your Hub Transport Server (ie, MBX/CAS server in this case) -> Right click and select “Properties” -> Edit the attribue “msExchEdgeSyncCredential” and clear the EdgeSync credential values. Save the changes.

Now re-run the recovery installation procedure and it will get completed successfully.

Reference article can be found here.

MC227292 – 1:1 Call Recording Policy Introduction | MS Teams

Microsoft had released a new announcement on November 20, 2020 and a subsequent update to it on November 27, 2020 about a change concerning 1:1 (peer to peer) call recording. For people who had missed this announcement can read it from your admin center here.

As per the new announcement, starting December 18, 2020 1:1 call recording will be controlled by a new teams calling policy (-CsTeamsCallingPolicy) and the attribute responsible for the same is AllowCloudRecordingForCalls. Keep in mind that by default this attribute will be set to false and call recording will not work unless it is set to true. You can check the status of new policy by executing the command:

Get-CsTeamsCallingPolicy -Identity Global | select Identity, AllowCloudRecordingForCalls

Currently, 1:1 calling is controlled by the teams meeting policy (-CsTeamsMeetingPolicy) and the responsible attribute is AllowCloudRecording. This can be checked by executing the command:

Get-CsTeamsMeetingpolicy -Identity Global | select Identity, AllowCloudRecording

So, for companies who need to continue to have call recording enabled for 1:1 calls should execute the below command on Teams PowerShell:

Set-CsTeamsCallingPolicy -Identity Global -AllowCloudRecordingForCalls $true

In the above commands, ‘Global’ is the default global teams meeting/calling policy.

Restore deleted public folders | Exchange Server 2013

An admin accidentally deleted couple of public folders recently from the Exchange Admin Center. Now we had to restore those public folders back. The first option most of us would try is to restore the public folder from Outlook.

Launch Outlook, select the Folders view first. Right click the root folder that hosted the deleted public folder and click on Recover Deleted Items. You will notice the deleted Public Folder here, now choose Restore Selected Items and click OK.

It might take some time depending on the size of the data and number of public folders for the folders to get restored.

However, in our case the restore through outlook was not successful as it used to timeout in between. Due to this, the next approach was to restore from the Exchange Management Shell.

First, execute the below command to export all public folder information. This might take some time depending on the size of the public folders.

Get-PublicFolder –Identity “\NON_IPM_SUBTREE” –Recurse | FL >C:\PF_Export.txt

The “\NON_IPM_SUBTREE” is actually the system folder root. These are system folders that are not accessible through clients like Outlook and OWA.

Now, open the exported file and search for the deleted folder name. Find the property Identity and copy the value as shown below

Launch the Exchange Management Shell and execute the below command to restore the deleted folder and its sub folders along with the public folder permissions:

Set-PublicFolder –Identity “\NON_IPM_SUBTREE\DUMPSTER_ROOT\<GUID>\FolderName” –Path “\” –Verbose

The above command restores the public folder to the root public folder hierarchy. You can also restore to the correct parent folder if you know the path.

In case you have multiple Active Directory domains, make sure that the restored public folder AD objects are moved to the correct domain where these existed previously. There are chances these AD objects stay in the root domain after the restore.

You can use this method of restore if the folders were deleted within the deleted item retention period. For cases where the folders were deleted beyond the retention period, you will have to restore the appropriate database full backup to a recovery database.

To find out more about restoring from a recovery database backup please refer here.

O365 Sign in error – AADSTS51004: The user account *** does not exist in the *** directory | O365 Hard match

We had a user who returned to the company and he wanted to have the same AD account and email address. The AD account was restored by the IAM team which somehow populated the Exchange attributes in the on-premises Exchange 2013 and it was displayed as a remote user mailbox as expected in an Exchange Hybrid environment. We could see the mailbox in Exchange Online with all the data as he had left it and did not suspect any issues. The user side IT team started having issues while setting up Outlook for the user and started getting the below error both while configuring Outlook and trying to login to OWA.

Upon checking the mailbox status in Office 365/Azure AD we noticed that the user mailbox is not synced with on-premises Active Directory and is currently acting as a cloud only account. Since our domain is also federated the user authentication seemed to have been failing due to the Sync issue.

To try soft match (or smtp matching) was not an option as the UPN/SMTP address was the same both on-premises and on cloud. The Exchange guid was same in both these forests as well. Next option was to perform a hard match (using ImmutableID). More information on Soft-match/Hard-match can be found here. A good read on ImmutableID can be found here.

Steps to perform hard-match:-

1) Export all the mailbox information from both forests ie, on-premises and cloud for later reference. Take backup of data if required for safer side.

2) In the on-premises side launch powershell window on a server where you have AD management tools installed and run

Import-Module ActiveDirectory.

3) Now find the ObjectGUID of the user account by running the command

Get-ADUser -Identity “user logon name” -Server “domain controller name”

4) Convert the Active Directory ObjectGUID value to Base64 value. The term Base64 is also called SourceAnchor/ImmutableID. This can be achieved many ways for eg: through PowerShell script, 3rd party applications etc. The simplest method is to use this website https://toolslick.com/conversion/data/guid.

5) Connect to Azure Active Directory to update the new ImmuatbleID. First, import the MSOnline module by running

Import-Module MSOnline.

Then run

Connect-MSOLService

and provide your Office 365 admin credentials. Now execute the below command to update the ImmutableID that was converted in step 4.

Set-MsolUser -UserPrincipalName user@contoso.com -ImmutableId **************==

Then either manually run a delta sync on your Azure AD Connect server or wait for the scheduled sync cycle to complete.

For users having a federated domain might get an error like below while running the above Set-MsolUser command:

This issue is discussed here. The solution is to temporarily change the UserPrincipalName to the default onmicrosoft.com domain by running the command:

Set-MsolUserPrincipalName -UserPrincipalName user@contoso.com -NewUserPrincipalName user@contoso.onmicrosoft.com

After the new ImmutableID is updated change the UPN back to the old one. Now wait for the AD Sync to take place. After which the on-premises account will properly sync to the cloud account. You can verify this from Azure Active Directory that the source of the object is Windows Server AD instead of Azure Active Dirctory and the Directory Sync status is showing as enabled.

Fix mailbox showing up in both Exchange Online and on-premises | Exchange Hybrid

Recently, I came across an issue where a user complained that he is not receiving emails from internal applications. Upon checking the mailbox status I noticed that the user had a mailbox hosted on our Exchange on-premises server and also on Exchange online. The on-prem mailbox only had around 100 MB, while the cloud mailbox had around 5GB data.

When the mailbox creation dates were compared, the online mailbox was created first and the on-prem mailbox got created later. In this situation when an office 365 user sends email to user it will be delivered to the cloud mailbox and when an on-prem user send email to will be delivered to the on-prem mailbox. Not sure how this condition happened, but couple of reasons are discussed here.

In this post I will be discussing the steps to recover from this scenario without making any changes to the cloud mailbox as the primary mailbox of my user is hosted in Office 365. If you want to try the other way around ie, recovering by removing the cloud mailbox and not touching the on-prem check this article instead.

1) Run the powershell command to save all mailbox information of both on-prem and cloud mailbox to a file. Make sure the data on the on-prem mailbox is exported to a PST file prior to following the below steps.

Get-Mailbox “mailbox identity” | fl > mailbox.txt

2) On the Exchange 2013 ECP, disable the user mailbox by clicking on the ellipsis (…) -> Disable.

Alternatively, you can run the below powershell cmdlet as well to disable the mailbox

Disable-Mailbox “mailbox identity”

Note : Do not delete the mailbox, only disable/disconnect. You will get an error while disabling the mailbox, if litigation hold is enabled. You can remove litigation hold by running the below command:

Set-Mailbox -Identity “user identity” -LitigationHoldEnabled:$false -DomainController DCName

3) Create a remote mailbox for the user on on-premises by running the command:

Enable-RemoteMailbox “user identity” -RemoteRoutingAddress “user@company.mail.onmicrosoft.com”

Add any additional SMTP addresses to the remote mailbox object from ECP if required. If you are using any custom attributes for AD Sync, update the same as well.

4) Identify the Exchange Online GUID of the cloud mailbox by running the below command in Office 365 Exchange Online shell

Get-Mailbox “mailbox identity” | fl *Exchangeguid*

5) Stamp the Exchange Online GUID on the remote mailbox in on-premises. This is not mandatory but will be helpful during any future mailbox offboarding.

Set-RemoteMailbox “user identity” -ExchangeGuid “Exchange Online guid”

You can see in the above screenshot that using Get-Mailbox we can verify the current Exchange Guid value. In this case its blank.

6) Now either force a directory sync or wait for the delta sync to complete. Restore the on-prem mailbox data to cloud if required.