Setting up continuous integration

Kentico provides a continuous integration solution that allows you to serialize the data of objects from the database into XML files on the file system. You can then add the files to a source control system (for example Team Foundation Server or Git) and use them to synchronize database data between team members. The system ensures that the XML data of matching objects is always identical and consistent (including element and attribute order), even when serialized on different instances of Kentico.

The system stores the XML files containing the serialized data of objects in the project's CMS\App_Data\CIRepository folder. For details about the folder and file structure, see Continuous integration repository structure.

If you need to work with the continuous integration files in a different location, use one of the approaches described in Storing the continuous integration files in a custom location.

The continuous integration solution is usable in the following environment:

  • Multiple development instances, each with its own project files and database
    – AND –
  • A central source control system that provides change management and version control for files

To start developing sites using the continuous integration solution:

  1. Set up your development environment and continuous integration according to the instructions in the following sections:
  2. Establish a process for transferring objects back into the database from the CIRepository XML files. See Restoring continuous integration files to the database.

  3. Begin development. We strongly recommend following the best practices described in Using continuous integration with Visual Studio. The page primarily focuses on using Visual Studio in combination with a Team Foundation Version Control or Git source control repository, but many of the recommendations also apply generally to any source control system.

 

Usage and relationship with other features

Continuous integration is designed to allow synchronization and version control of database data during development (through an external source control system).

Continuous integration has a negative impact on site performance. The impact is not significant for development instances with a low number of users, but continuous integration should not be enabled on production sites. For more information about deployment options, see the Deploying data to production sites section.

Additionally, we do not recommend using continuous integration together with the other team development and versioning features in Kentico:

Continuous integration always works with the main version of each object in the database. Older or deleted versions of objects and non-published versions of pages are ignored. For example:

  • If you use the object versioning feature, continuous integration ignores all versions of objects except for the latest. If you roll back an object to an older version, the object is updated in the CIRepository folder.
  • If you delete an object or page to the recycle bin, continuous integration fully removes the corresponding XML file. If you restore the object, an XML file is added as if a completely new object were created.
  • If you edit an object's code externally (e.g. the code of a CSS stylesheet), changes are ignored by continuous integration until you synchronize the code back into the database. You can still synchronize the code without the continuous integration solution by including the external file in your source control.

Limitations

Continuous integration currently has the following limitations:

  • Not all object types in the system are supported. See Object types supported by continuous integration for a complete list.

  • The page workflow feature is not supported when using continuous integration.

  • Continuous integration only tracks object changes made through the Kentico administration interface or API. Changes made directly in the database or by external tools that modify the database are not tracked. After making such changes, you need to manually serialize all objects in the Continuous integration application to synchronize the database with the file system repository. For example, continuous integration does not track roles imported by the AD import external utility.

  • You cannot reliably use continuous integration in environments where the CIRepository folder is located on a shared file system (multiple Kentico instances with separate databases connected to the same file system repository). Common examples of shared file systems are external storage providers, such as Azure storage or Amazon S3.

  • When restoring content from the continuous integration repository on an instance with Staging enabled, the system does not log staging synchronization tasks for changes made to pages.

Preparing the development environment

Before you can use the continuous integration solution, you need to set up Kentico instances for your development team.

Start by preparing an initial instance:

  1. Either install a new instance of Kentico or select an existing instance as a starting point.
  2. Enable continuous integration on the given instance (see Enabling continuous integration for details).
  3. Create a backup of the instance's database.
  4. Add the entire Kentico solution to your source control (adding only the CIRepository folder may lead to problems and is not recommended).

To add development instances into your environment, perform the following steps for each instance:

  1. Connect to the source control from the development machine and load the latest version of the solution files onto the local file system.
  2. If you use IIS to run your development sites, register the project as an application in IIS:
    1. Open Internet Information Services (IIS) Manager.
    2. Add a new application under your IIS web site and map the Physical path to the project's CMS folder.
    3. Create a new application pool for the application (or use an existing application pool).

  3. Create (restore) a copy of the initial database on your SQL server.
  4. Connect the instance to the new copy of the database (set the appropriate connection string in the project's web.config).

After you complete the process, all development machines will contain their own Kentico project files connected to a separate copy of the database. You can start using your source control system to synchronize project files between the instances. The continuous integration solution serializes database data onto the file system and you can include it in your source control, just like any other files.

The process described above ensures that objects have identical GUID values (globally unique identifiers) across your entire development environment. If you perform a separate database installation for each development instance, you will need to manually resolve a large number of GUID conflicts when synchronizing data through the continuous integration solution.

If you need to add further instances later (after development with continuous integration starts), use the same process. However, you need to make sure the database is synchronized with the rest of the development environment. Use one of the following approaches:

  • Maintain a central database connected to the mainline source control and always create an up-to-date backup for new developers.
    – OR –
  • Use the original database backup and then get the latest data:
    1. Restore the current object status from the continuous integration repository into the database. See Restoring continuous integration files to the database.
    2. Manually transfer changes made to the data of object types not supported by continuous integration (maintain documentation of such changes). You can use the export and import feature.

Enabling continuous integration

To configure a Kentico instance to use the continuous integration solution, perform the following steps:

  1. Disable running of scheduled tasks (using the Settings -> System -> Scheduled tasks enabled setting).
  2. Open the Continuous integration application in the Kentico administration interface.
  3. Select the Enable continuous integration checkbox.
  4. Click Save.
  5. Click Serialize all objects.
  6. Wait until the serialization process finishes and then re-enable scheduled tasks.

The project's CMS\App_Data\CIRepository folder now contains XML files storing the serialized data of all supported objects from the database. The system also tracks create, update and delete operations for the given objects and automatically transfers the changes to the serialized data on the file system.

Configuring instances to synchronize macros

To ensure that macro expressions work correctly when synchronizing objects using the continuous integration solution, all development instances must use the same hash salt value.

If your development instances do not all start with the same web.config file, you need to set a matching value for the CMSHashStringSalt key in the appSettings section on all instances. The best option is the hash salt used by the starting instance that you used to create your initial database backup. You can use any string as the value, but the salt should be random and at least 16 characters long. For example, a randomly generated GUID is a strong salt:

<add key="CMSHashStringSalt" value="e68b9ad6-a461-4707-8e3e-ece73f03dd02" />

The salt value is used as part of the input for the hash function that creates the security signatures of macros. Having the same hash salt value on all instances is necessary to ensure that macro signatures are valid when transferring data between instances.

The best option is to set the hash salt value before you start development. Changing the salt causes all current hash values to become invalid. To fix existing macro expressions in the system after changing the hash salt, you need to re-sign the macros. See Working with macro signatures for more information.

Filtering object types

The continuous integration solution allows you to exclude specific types of objects if you do not wish to synchronize all supported types. To configure the system to ignore certain object types within continuous integration:

  1. Edit the repository.config file in the root of the CMS\App_Data\CIRepository folder.
  2. Add <ObjectType> elements into one of the following sections of the config file:
    • <IncludedObjectTypes> - defines a whitelist of object types. If you specify one or more object types, continuous integration only processes objects of the given type. No restrictions apply if the whitelist is empty.
    • <ExcludedObjectTypes> - defines a blacklist of object types. Continuous integration processes all included object types except for the listed types.

  3. Set the values of individual ObjectType elements to the names of object types that support continuous integration.

    Tip: To find the ObjectType values, see Object types supported by continuous integration or check the names of the corresponding folders in the CIRepository folder.

  4. Save the repository.config file.
  5. Run complete serialization for all objects to bring your CIRepository folder into the required state:
    1. Disable running of scheduled tasks (using the Settings -> System -> Scheduled tasks enabled setting).
    2. Open the Continuous integration application in the Kentico administration interface.
    3. Click Serialize all objects.
    4. Wait until the serialization process finishes and then re-enable scheduled tasks.

Excluding an object type in the repository.config file affects all processes related to continuous integration:

  • Serializing of all objects onto the file system
  • Automatic tracking of create, update and delete operations for the given objects and transferring of the changes to the serialized data
  • Restoring of objects from the file system into the database

By default, the <ExcludedObjectTypes> list contains the cms.settingskey object type. Settings may contain sensitive data – only remove the exclusion if you agree to make all setting values available within the file system used by your application and any connected source control systems.

Excluding child and binding object types

The system uses the following rules to automatically exclude child and binding object types:

  • Child object types follow the configuration of their parent object type. For example if you exclude page types, then queries, transformations and alternative forms of page types are also excluded.
  • Binding object types are excluded if all object types within the given relationship are excluded.

The <IncludedObjectTypes> whitelist only allows you to specify the main object types.

In the <ExcludedObjectTypes> blacklist, you can exclude both the main object types and specific binding or child object types.

Important: To maintain consistency, always use the same repository.config settings across your entire development environment. Whenever you make any changes, use your source control system to synchronize the config file along with the other content of the CIRepository folder. When a developer loads a new version of repository.config from the source control, the new settings start applying after restoring objects into the database or a manual restart of the application.

repository.config examples:

<?xml version="1.0" encoding="utf-16"?>
<RepositoryConfiguration xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
	<!-- Continuous integration includes all supported objects except for email templates. -->
	<IncludedObjectTypes>
	</IncludedObjectTypes>
	<ExcludedObjectTypes>
		<ObjectType>cms.emailtemplate</ObjectType>
	</ExcludedObjectTypes>
</RepositoryConfiguration>
<?xml version="1.0" encoding="utf-16"?>
<RepositoryConfiguration xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
	<!-- Continuous integration only includes web parts and web part categories. Web part layouts are excluded. -->
	<IncludedObjectTypes>
		<ObjectType>cms.webpart</ObjectType>
		<ObjectType>cms.webpartcategory</ObjectType>
	</IncludedObjectTypes>
	<ExcludedObjectTypes>
		<ObjectType>cms.webpartlayout</ObjectType>
	</ExcludedObjectTypes>
</RepositoryConfiguration>

Deploying data to production sites

Deployment of content and data to instances that host live websites is not the primary purpose of the continuous integration solution. If possible, we recommend using the following deployment options instead:

  • Set up one of the instances in your continuous integration environment as a source server for Staging (this scenario does not support deployment of page content)
    [Note: Apply hotfix 9.0.3 or newer if you wish to use staging to deploy object changes synchronized through continuous integration]

  • Export and import packages

If you do decide to deploy data to a live production site via the continuous integration restore functionality, use the following process:

Warning

Restoring data through continuous integration deletes any pages and supported objects that are missing in the CIRepository folder. You may lose content created directly on the production instance through standard editing or live site interaction.

For example, the following scenario is possible:

  • You prepare a content update in your development environment and copy the CIRepository folder to the production instance.
  • Live site users meanwhile create new pages via user contributions.
  • You restore the CIRepository content to the database.
  • The pages added by the live site users are deleted, because they are not included in the restored data.

Optionally, you can configure the repository.config file on the production instance to exclude any object types that you do not wish to update and overwrite on the live site.

  1. Leave continuous integration disabled on your production instance of Kentico (the setting in the Continuous integration application). Continuous integration has a negative impact on site performance and should not be enabled on production sites.

    Important: You can restore data even when continuous integration is disabled.

  2. Upload the CIRepository folder containing the required data into the CMS\App_Data folder of your production instance.
  3. Restore the CIRepository content to the production database – see Restoring continuous integration files to the database.
    • We strongly recommend performing the restore process at a time when the site has minimal traffic and no editing is taking place.
    • The restore process may take longer than usual when continuous integration is disabled.

  4. Restart the application (if not done automatically by your restore process). Open the System application and click Restart application on the General tab.

The data is now deployed to the production database. Keep in mind that any changes made in the administration interface or on the live site will not be tracked and reflected in the CIRepository folder.

Upgrading projects that use continuous integration

If you need to apply a hotfix or upgrade to an instance that has continuous integration enabled, use the following procedure:

  1. Restore objects from the continuous integration repository to your database.

    You can skip the restore if you are sure that your database is synchronized with the current status of your CIRepository folder.

  2. Apply the hotfix or upgrade.
  3. Run complete serialization for all objects to recreate the content of the CIRepository folder:
    1. Disable running of scheduled tasks (using the Settings -> System -> Scheduled tasks enabled setting).
    2. Open the Continuous integration application in the Kentico administration interface.
    3. Click Serialize all objects.
    4. Wait until the serialization process finishes and then re-enable scheduled tasks.

This approach ensures that your CIRepository folder contains all object changes made by the hotfix or upgrade and that the serialization process itself runs according to the new version.

For hotfixes, you need to apply the update separately for each development instance. After one developer commits the hotfixed changes to the source control, other developers CANNOT commit or load changes until they apply the hotfix to their own instance.

For major upgrades, it may be more efficient to upgrade only one instance and then set up your environment again according to the process described in Preparing the development environment.

Storing the continuous integration files in a custom location

After you set up and enable continuous integration, the system serializes database data into XML and stores the results on the file system.

By default, the files are created in the Kentico project's CMS\App_Data\CIRepository folder. If your development environment requires a different location for the continuous integration files, you can use one of the following approaches:

Important

We recommend keeping the default CIRepository folder within the App_Data folder of the Kentico web project and having the entire Kentico solution included within your source control. This setup allows you to easily synchronize all types of related project files together with the database data tracked by the continuous integration solution.

If you do decide to change the CIRepository location, do NOT use the new folder as the root of your source control repository. Always add it as a subfolder under a different root folder. The continuous integration solution deletes all folders within CIRepository when storing all objects. If the folder is the root of your source control repository, you may encounter errors or other problems when using source control systems that utilize custom folders in the root.

Creating a junction point

By defining a file system Junction point, you can redirect the CIRepository folder to a different location.

Limitation: The target location must be on a local drive or volume (network locations cannot be used).

  1. If you have already enabled continuous integration and serialized objects, you need to:
    1. (Optional) Back up your repository.config and the data files in CIRepository.
    2. Delete the CIRepository folder in your Kentico project (it is not possible to create a junction point for an existing folder).

  2. Create the target folder where you wish to store the continuous integration files.
  3. Open the Windows command prompt and run the mklink /J command:

    mklink /J <CIRepository folder path> <target folder path>
    
    For example:
    mklink /J C:\inetpub\wwwroot\Kentico\CMS\App_Data\CIRepository C:\ExternalSourceControl\CIRepository

When serializing object data to the file system, the Kentico continuous integration solution now creates the files in the specified target folder. You can add the target folder into your source control system.

Using a custom storage provider

You can use a custom storage provider to change the path of the CIRepository folder. The target location can be a local drive, or any network location for which the application has sufficient permissions.

Limitations:

  • The target location always includes the full ~/App_Data/CIRepository folder structure, starting from a specified root directory.
  • After you define the custom storage provider, the CIRepository folder can no longer be accessed within the original location (unlike when using a junction point).

To register a storage provider, you need to run custom code during the initialization of the application. You can do this by creating a custom module, or via the CMSModuleLoader partial class in the Kentico project's App_Code folder. For example, the following steps demonstrate how to change the CIRepository path by adding a class in App_Code:

  1. Open the Kentico web project in Visual Studio (using the WebSite.sln or WebApp.sln file).
  2. Create a new class in the App_Code folder (or CMSApp_AppCode -> Old_App_Code on web application projects).

  3. Use the following code for the class:

    using CMS.IO;
    using CMS.Base;
    
    [CICustomStorage]
    public partial class CMSModuleLoader
    {
    	private class CICustomStorageAttribute : CMSLoaderAttribute
    	{
    		/// <summary>
    		/// The system executes the Init method of CMSModuleLoader attributes when the application starts.
    		/// </summary>
    		public override void Init()
    		{
    			// Maps the 'App_Data/CIRepository' folder in the Kentico project to a custom location
    			// The 'UseLocalFileSystemForPath' method internally creates a custom StorageProvider instance,
    			// and maps the specified Kentico directory (first parameter) to a different location based on a root path (second parameter)
    			StorageHelper.UseLocalFileSystemForPath("~/App_Data/CIRepository", @"C:\ExternalSourceControl");
    		}
    	}
    }

When serializing object data to the file system, the Kentico continuous integration solution now uses the CIRepository folder located under the mapped root folder. You can add the folder into your source control system.


Was this page helpful?