Troubleshooting web farms

Monitoring web farm synchronization tasks

You can monitor web farm tasks in the Web farm application on the Tasks tab. The tab shows all synchronization tasks that are currently active (waiting to be processed).

In an ideal situation, this tab should not contain any tasks – this means that the web farm tasks are processed soon after they are created. The system automatically removes successfully processed tasks.

The Anonymous tasks tab displays a list of currently active (waiting to be processed) anonymous synchronization tasks. These tasks are logged by external applications (e.g. Windows services) to ensure that changes made by the external application are reflected in the web application cache. Tasks are logged as anonymous only if the application is NOT configured to run in a web farm. If it is configured to run in a web farm, these tasks are logged as standard synchronization tasks on the Tasks tab.

When processing of any web farm synchronization task fails, the system logs an error to the event log (at most once per 30 minutes to avoid flooding the log).

Monitoring web farm servers

You can monitor web farm servers in the Web farm application on the Servers tab. Each server registered in the system can have one of the following statuses.

The statuses represent the ability of web farm servers to connect to the database (and thus process synchronization tasks).

Status	Description	Recommended action
Healthy	The web farm server is reporting to the system at expected intervals, regularly, without any significant delays.	The server is working properly. No action is needed.
Transitioning	The web farm server is initializing or has stopped responding.	We recommend waiting for about 3 minutes before looking for the source of the issues, as the status will probably change during this time.
Not responding	The web farm server is not responding for an extended period of time. Details for manual web farms: After 24 hours in this state, the system stops generating memory tasks for the server and deletes already generated memory tasks. If the server is started again, it obtains the data from the database. However, the web farm continues generating all other types of tasks for the server (for example file tasks). The server stays in the system until you remove it manually. Details for automatic web farms: After 24 hours in this state, the system removes the server from the system along with all its web farms tasks. Generating of synchronization tasks for Not responding servers can cause unnecessary load and performance problems, particularly in hosting environments that dynamically adjust the number of instances (e.g., autoscaling in Azure App Services). If you encounter such problems, you can reduce the default 24 hour interval for which Not responding servers remain in the system. Set the CMSWebFarmNotRespondingInterval configuration key to the required number of minutes, e.g., 60 for 1 hour. appsettings.json `"CMSWebFarmNotRespondingInterval": 60,` web.config `<add key="CMSWebFarmNotRespondingInterval" value="60" />`	The server is down. Check the server physically and check whether it is able to connect to the database. When running on Microsoft Azure, check the server status through the Azure Management Portal.

When a server switches to the Not responding status, the system logs an error to the event log.

Event log email notifications

You can configure the system to automatically send email notifications whenever errors occur in the system, as described in Setting up email notifications for errors.

If you configure the system to send email notifications, the following notifications related to web farms will be sent:

When a server switches to the Not responding status.
When a server switches to the Healthy status.

A notification will be sent once for each status change that happens.

Troubleshooting web farms

Server time zone consistency

In environments with multiple web farm servers hosted across different time zones, certain servers may appear as “Not responding” in the Web farms application. This is caused by time zone inconsistencies across the hosting servers.

Xperience uses an internal timestamp mechanism to determine whether a web farm server is up and running. If a server does not respond within a set timeframe, it is labeled as non-responding and ignored by the rest of the environment. To achieve synchronization, the timestamp mechanism uses server time of the hosting server. For servers with different time zone configurations, these times can differ in the counts of hours, which is magnitudes larger than the slippage tolerated by the synchronization mechanism. This leads to the server being labeled as “Not responding” by the web farm environment.

Solution

Set the same server time zone for all servers hosting your Xperience applications.

Changing server time zones in Azure-hosted applications

Due to the nature of cloud-based services, the server hosting your application changes frequently. These changes are also reflected in event time-stamps and other places where server time is displayed or used, which may be undesirable.

If you wish to persistently configure the time zone for Azure-hosted applications, follow the instructions outlined in Changing the server time zone on Azure.

Incorrect number of web farm servers in the system

If you have configured an automatic web farm and the number of your web farm servers connected to the database does not correspond to the number of servers registered in the system, the web farm will not work properly. This can happen for various reasons. For example, if you tamper with the system time on one of the servers, the system may consider other servers to be down due to the time difference and delete them.

Solution

In most cases, you can solve the problem by restarting the web farm server that is missing in the system or all web farm servers (or at least their IIS). After an IIS restart, Xperience will be able to register the web farm server properly. If you are running your project on Microsoft Azure, you can reboot the instances of your project.

To quickly restart the entire web farm, you can open the System application and click Restart all web farm servers on the General tab.

If the system shows more web farm servers than there should be, the automatic web farm will delete them after some time.

Outdated servers error

Error: [DataConnection.HandleError]: Query: Proc_CMS_WebFarmSync_SetServerTasks Caused exception: The INSERT statement conflicted with the FOREIGN KEY constraint “FK_CMS_WebFarmServerTask_ServerID_CMS_WebFarmServer”.

This error occurs if automatic web farm server generation is enabled and some servers become outdated or new servers are missing in the system.

Solution

Restart the faulty web farm server or all web farm servers (or at least their IIS).

Web farm tasks are not being processed

If the Tasks tab in the Web farm application contains web farm tasks that fail to be processed, the cause can be one of the following:

The system is configured for more web farm servers than there really are.
The worker thread processing the tasks is not running.
- Check if the CMS.WebFarmSync.WebFarmTaskProcessor thread is running in Debug -> Worker threads.

Solution

Check the Web farm application on the Servers tab and make sure that the number of configured web farm servers corresponds to reality.

Restart the web farm servers that are missing from the list or all web farm servers (or at least their IIS). The IIS restart registers the servers back into the web farm and also runs the thread that processes tasks.
If there are more servers than there should be, wait for the system to automatically delete them together with their tasks.