Sunday, December 20, 2015

Handling deployments of Sitecore Marketplace modules, Sitecore patches and Hotfixes


Part 1 | Part 2 | Part 3 | Part 4

This is my last post in this series and I am writing about how we have handled deployments of other required elements in our Sitecore ecosystem.

As we had chosen our way of auto deployment mechanism using Octopus Deploy, we wanted to be consistent in our approach irrespective of what kind of deployment we are doing. If that is not the case then usually different means involve different processes, tools and operational overhead that incur due to diversity. In Sitecore world, you need lot of additional market place modules, upgrade patches and hotfixes from Sitecore, hotfixes from our own teams, small configuration changes, site configuration and package deployments in a multi-site deployment environment etc. Having a consistent deployment strategy helps in streamlining release processes, consistent deployments across environments and less chances of manual errors

Typically Sitecore deployments has 2 aspects.
  1. Files that you need to deploy in the root directory (configuration files, dlls, js files etc.) of each server in your environment
  2. Sitecore Packages (serialized Items) which needs to be deployed once per cluster (CM / CD / Job etc)

Some customers deploy this as one single package on one server and have a Windows DFS or similar tools (such as Repliweb) to sync files from master to slaves. It adds more complexity if you have different configurations for different roles e.g. for CM, CD and Job (Publishing) server.

We rather chose to do it this way in our build process
  1. Separate file system package as a Nuget package - We deploy this on all servers in an instance, one by one.
  2. Create different Nuget package for Sitecore items alone - we deploy this once on any one server identified as CM-Master in an instance

We also created a utility that will take any zip / update packages as the input and allows you to create it as a Nuget package. This was needed for us to separate out the files and items packaged in Marketplace modules or  Sitecore update packages or Sitecore zip packages. It provides you flexibility to select and configure the folders where each individual files will be deployed and also allows you to publish these packages in the configured Nuget repository to take it off for environment wide deployments.

Long-term through process is also to pre-process required Sitecore modules in Nuget packages and deploy them as pre-requisite whenever we need, using Octopus Deploy to create our own Sitecore instance.


Following screenshot shows different steps in the custom utility we created.

Screen to upload Sitecore Module / Package
Step 1: Sitecore Module / Package Upload
Showing Files and Sitecore Items separately
Step 2 - Segregation of Files and Sitecore Items
Step 3: Name the packages

Step 4 : To upload it into a Nuget Repository (Optional) or download the packages


Happy Deployments !!



Friday, November 27, 2015

Sitecore Deployments using OctopusDeploy - Part 3 (Handling environment specific Configuration)


Part 1 | Part 2 | Part 3 | Part 4

This is my third post in this series and I am going to talk about different ways to handle configuration changes specific to each environment OR even each role within an environment.

While handling deployments in multiple environments, it is always a question with respect to how to handle environment specific configuration changes? This is not a new problem to solve for !!

There are primarily two stages wherein you can handle environment specific configurations in automated way (without relying on error prone manual editing of configurations by logging into specific environments and performing live changes).

Handling transformations during the build phase
Handling it at the time of build phase through your continuous integration platform. Problem with this approach is for each environment, you need to run a separate build to ensure the transformations are run for that specific environment and the resulting packages include relevant configurations. With this approach, you are deploying different packages in different environment (though you would have ensured to take the artifacts from the same tag to ensure you are building the right code and packaging the right configurations.

Handling transformations during the deployment phase
Other way to handle this is during deployments. You create a package once and you promote it to various environments as it is. Package contains all the configuration transformation rules you need for each environment right from the beginning. Platforms like OctopusDeploy has a built in support for doing transformations at the time of deploying a release. Alternatively, for some of the common configurations such as AppSettings and ConnectionStrings you can choose the configuration variable replacement concept at the time of deployment. You can read more about it here.

Approach adopted to handle environment specific configuration differences


  1. Configurations that go under sitecore configuration element, Sitecore already has a patching logic which works similar to configuration transformation and Sitecore handles that pretty well. We did not want to re-invent the wheel to do it using our own / visual studio style, also for a reason that we started on handling environment related configurations in automated way little later. This is not to say that what we did was not right, perhaps you may want to keep it the same way even if you are starting your engagement just now.

  2. Handling configurations other than Sitecore (such as web.config / other configuration files)
    • One of the hurdles and debate we came across is - should we include web.config in our package? Usually this is provided by Sitecore as a product and it’s a bad idea to deploy your own web.config will all changes incorporated in it. We wanted to stay away from that by just keeping the config transform rules on top of web.config and not touching the original web.config. But OctopusDeploy does not run the transformation rules if original web.config is not part of the package (because when Octopus extracts the Nuget package, it first runs the transformations before it copies over to the destination folder).
    • We thought of an alternative. OctopusDeploy raises lot of events as part of Nuget package deployment task and as a pre-deploy event, we thought of picking up the original file from the root directory before OctopusDeploy runs the transformation. We realized that as soon as you transform the web.config, you don’t have original web.config anymore at the root !! You can find more about OctopusDeploy feature ordering here
    • We finally decided to include web.config that comes as part of Sitecore default installation (for the Sitecore version we use) and took a cautious call to change it whenever we upgrade to newer version of Sitecore.
    • Few scenarios where this would be handy is appsetting configurations, logging configurations

  3. Handling connectionstrings
    • Connectionstrings contain sensitive information and you can either encrypt whole connectionstrings section using standard ASP.NET / IIS configuration encryption mechanism. You can find more information here on this
    • Challenge was, someone still needs to maintain it in the source control repository and they have to decrypt, change and encrypt it again.
    • OctopusDeploy, as stated earlier, provides a nice feature to replace connectionstrings by means of variable feature, at the time of deployment.

Few considerations :

    • OctopusDeploy handles transformation at the deployment time but it runs it only at an environment level and not at the role level. In our situation, and probably any Sitecore deployment situation, we had several roles as stated in my previous blog, and it was unfortunate that we could not do transformation rules execution at the role level (such as CM / CD, Job etc.).
    • The way we handled it then was either using Sitecore patching logic OR if certain configuration was not relevant for a specific role (such as switchonrebuild / switchtojobserver etc.) then we had a cleanup task after package deployment which was deleting unnecessary files not relevant for that role. We maintained this list of files in an Octopus variable



Happy Deployments !!


Wednesday, September 23, 2015

Sitecore Deployments using OctopusDeploy - Part 2 (Deployment Process and Considerations using OctopusDeploy)


Part 1 | Part 2 | Part 3 | Part 4

In my first post I had mentioned about typical Continuous Integration and Deployment architecture, considerations for development and Continuous Integration. In this 4 part series, this is the second post and I am talking about Deployment process and various considerations that needs to be made when using OctopusDeploy


OVERVIEW


OctopusDeploy is a nice deployment frameworks which works against Nuget packages. If you are not already using any deployment automation platform then you should definitely consider OctopusDeploy as a platform, not only for Sitecore but also for enterprise wide Microsoft solutions delivery.

OctoupsDeploy has two components. One is OctopusDeploy server and other is group of tentacles (agents running on your instances). The communication is secure by tentacle authenticating the server over X.509 certificate. More on this here



I am listing down few features which would be useful in your Sitecore automated deployments using OctopusDeploy


  • Configuration for high availability. Find out more about this here
  • Rolling deployments - Useful when you are doing web package deployments across CM or CD nodes in a clustered instance. More information about this here 
  • Ability to execute PowerShell commands on the Tentacles
  • In-built Nuget feed which would be enough for small engagements.
  • Deployment dashboards, showing which release is deployed in which environment
  • Ability to configure multiple tentacles on the same server, tag different machines with different roles
  • Ability to do configuration transformations at the time of deployment to specific environment.
  • Ability to listen to any external Nuget feed.
  • Ability to create variables and scope it at each Environment, Role or Machine level or combination of it, to define contextual values. This comes very handy to make your deployment process generic and repeatable.
  • Rich set of script templates allows you to build the deployment process pretty quickly. You can check this library here
  • Supports Windows Azure deployments
  • Ability to create automated releases, which will be helpful if you intend to setup continuous deployment to your pre-production (development / demo) servers.
  • Integration with your existing Active Directory. 
  • In-built roles allows you to restrict access to projects, environment configurations and others. Read more about security in OctopusDeploy here


PRE-CONSIDERATIONS


Before or during the process of deployment automation, consider below mentioned aspects 


  • If you are working in multi-vendor kind of engagement wherein infrastructure is owned and maintained by some other agency, its better to get them involved early in the process
  • You will need few service accounts with specific permissions to folders on each VM, execute PowerShell to deal with IIS, Nuget package deployment, restart app pools, read certificate store etc. 
  • You will also need a Sitecore user who has access to install packages, issue publishing and indexing requests as part of deployment process.
  • You need to open few inbound ports on each VM so that Octopus can communicate with its tentacles for deployments and health checks
  • If you also have steps for database backup / script executions / restorations, consider a SQL Server user having that level of access.
  • Try and script out your tentacle installation. Depending on number of machines and environments this can be error prone if you do this manually. You can get some help here
  • Store your OctopusDeploy master key safely. This will be handy if at all you need to restore the old db backup. Note that Octopus deployment database is encrypted. Without Master key, you will not be able to restore old data
  • Create test Sitecore deployment servers to ensure your deployment process works once its ready and also during the instrumentation phase so that you have some place to do try things out.
  • Plan for testing all the tentacles in all your environments. Even though you tested your deployment process in test environment, you do not want to try deployments first time in production without ensuring if the tentacles and all necessary permissions are granted as expected in those machines in Production.
  • Find out size of your packages and how long it takes to deploy / install your packages. You may need to tweak some timeouts at firewall or app pool level to be able to handle medium / large packages if those take beyond 5 minutes or so.
  • Give due considerations on how would you deploy web packages (not Sitecore). Do you have distributed file sync mechanism? or do you want to rely on Octopus? We got away with file sync mechanism and handled all package deployments using Octopus. 
  • If so, you also need to think about how would you deploy Sitecore Modules or hotfixes received from Sitecore? They usually many not be available in Nuget package format. We addressed this by developing our own tool to convert and package anything on the file system to Nuget package.
  • Consider all your deployment scenarios. In our case we had so many
    • Core Platform packages
    • Brand platform packages 
    • Hotfix packages
    • Sitecore Modules, Patches / Hotfixes in different formats
    • Small Configuration changes (such as tuning Sitecore configs, changing log levels)
    • Site specific configuration changes

I think that's quite a long list :)





DEPLOYMENT PROCESS


Setup

Below pictorial representation shows the kind of environment, roles and machine we have setup for our Sitecore deployment

Sitecore Environments configured in OctopusDeploy
Sitecore Environments Configured in OctopusDeploy



















Below are the role definitions and their purpose (Note: Role in Octopus term is just the textual tag and you can give any friendly name you wish)
  • CM-Master - This is the role to scope our deployment steps for Sitecore package installations only to this machine. As Sitecore package in our build process contains only Sitecore items, they have to be deployed only once. We use the same role to do content publish (not the site content but platform related items such as templates, layouts etc.)
  • CM-Slave - This is just another CM Node and distinguishes itself from the Master
  • CM-Job - We share the same Job server VM across multiple production instances (not exactly how it is shown in the picture above). To be able to do that we run multiple Sitecore instances on the Job Server VM and for each of the Sitecore instance we have a Octopus tentacle service.
  • CD - These are content delivery servers
  • SingleMaster - Where there is a standalone CM / CD server (not in cluster). There are situations such as development / pre-prod servers where its not the similar environment as in higher or production environments
  • OctopusServer - This is Octopus Server itself which is used to execute Database scripts remotely using SQL Management Objects and client library. We did not want to install tentacle on database server.
  • There are few others

Process


We have about 20 step deployment process and some of the steps has child steps within it.

I am putting high level details of important steps but you will get an idea on how to build this



First 4 steps
Step 5


  1. As a first step, we check if the instance is up for us to do the deployment (this is different than tentacle health). We do this using PS command Invoke-WebRequest and checking the status code to match 200 OK.
  2. If the above step successful, we install a TDS service package (to ensure its there - this is converted to nuget package by us and uploaded to Nuget feed) which is needed for Sitecore package installation.
  3. We then notify the recipients, configured in the OctopusDeploy variable (email address of a distribution list), that the deployment has started
  4. We then take the backups, first for CM databases (Master, Core and Web)
  5. Step 5 above takes the file system backup, and this excludes certain directories such as /sitecore /temp /app_data folders 

  6. Sitecore Package Deployment STep which includes Sitecore Items


  7. Then, as shown above, we deploy the Sitecore Packages only on servers which are in Master roles (CM-Master, SingleMaster etc.) and below is how we do it. Package path parameter is the path where Nuget package deployment task has extracted the Sitecore update package.
  8.         
    #Installing sitecore package using TDS service
    & "$TDSInstallationDirectory\Installer\PackageInstaller.exe" -v -sitecoreUrl "http://$SitecoreUrl" -sitecoreDeployFolder $WebsiteFolder -packagePath $Sitecore_Package_Installation_Directory\$contentFolder\$updatePackage
    
    

  9. We do a HTTP health check (hitting a URL and checking status code 200) to ensure that Sitecore instance is still up

  10. Step 7

  11. Next we do a web package deployment using Nuget package deployment step (available in Octopus library) as shown above. We also do a file clean up to delete certain files which are not required on CM Master role e.g configs which are specific to CD or publishing / job server needs to be deleted. Note: The configuration transformation happens at this stage for specific environment where deployment step is being executed.
  12. We have multiple Sitecore packages from different teams so next few steps does the deployment using similar logic stated above (step 7 & 8)
  13. We then publish the specific areas of content tree to the internal publishing target using our custom service
  14. Notify stakeholders via email for them to proceed with sanity testing before we proceed to CD deployments
  15. We have a manual intervention step (available in Octopus) here for someone to manually click a button once sanity testing is complete, to proceed with CD deployments
  16. Once we start CD deployment (post manual intervention and go ahead), a notification is sent to stakeholders that deployment to CD instances has started
  17. First step is to check if CD servers are responding by doing the HTTP health check with expected status code of 200 OK
  18. Take the backup of Pub (publishing target) database
  19. Take the file system backup (excluding list of folders - as mentioned in step 5 above
  20. Publish Sitecore Items by calling our custom service. Octopus task has a parameter which takes values of desired locations to publish from a variable
  21. Once publishing is done, we do a Nuget web package deployment in all CD servers in Rolling Deployment mechanism
  22. Once Nuget package is deployed, we do a clean up using file system clean up task in OctopusDeploy for specific files and configurations not relevant for Content Delivery servers.
  23. Finally do a HTTP health check with expected status code of 200 OK
  24. Notify stakeholders of Deployment completion


Deployment Considerations


Few considerations which may be helpful when you implement your deployment process


  1. Restoration - Since we are taking the backup of databases and file system, restoration can be done on-demand and that process we have kept outside of this process. To handle such scenario, we have automated the restoration part as a separate Octopus project
  2. AppPool Recycle Requests - Ideally you should not need any explicit application pool recycles but sometimes to clear the HTML output cache on priority we may need to have this in place. We created a separate Octopus project to handle this. Project of this nature must be secured and permissions must be granted only to appropriate group of users such as Octopus Administrators.
  3. Pre-requisite deployments - In our engagement, there are lot of other dependencies such as device detection database from 51 degrees or a security roles and permissions package etc., which are relevant whenever we are setting up a new instance or rarely changed and deployed but not as part of regular releases. You can either have these steps executed conditionally (by means of checking if those already exists on the instance) or have a separate project, like we have, to deploy the pre-requisites first before making your releases.
  4. Deployment step generalization - Keep your deployment steps as generic as possible. If parameters for your deployment steps changes from one environment to another, one deployment role to another or from machine to machine, consider using Octopus variables or better even, variable sets (which allows you to group octopus variables logically)
  5. Define Deployment Lifecycle - If you want your releases should get promoted in particular order e.g. From SIT -> QA -> Staging -> Production then define your project life cycle in octopus and configure that to be used in your deployment process

In my next post I will talk about different considerations about handling Configuration Transformation


Happy Deployments !!


Monday, August 17, 2015

Sitecore Deployments using OctopusDeploy - Part 1 (Overview and Architecture)


Part 1 | Part 2 | Part 3 | Part 4

Overview

While working on one of the biggest Sitecore Implementation, few years back there was no guidance nor the defined process for Continuous Integration and Deployment for Sitecore. We have learned our lessons hard way and wanted to ensure that other folks in the community does not go through the same pain.

In this blog series I wanted to put forth my thoughts on specific to area of deployment while briefly touching on the build and Continuous Integration process

In the 4 part series of deployment, I will be focusing on these areas
  1. Continuous Integration and Deployment Architecture
  2. Overall deployment process and considerations using OctopusDeploy
  3. Handling environment specific Configuration Transformation
  4. Handling external packages such as Sitecore support patches, Modules and Hotfixes

Continuous Integration and Deployment Architecture

Strategic Architecture

Strategic Continuous Integration and Deployment Architecture which involves Sitecore and Other Microsoft Technolgoies
Strategic CI And Deployment Platform


Specific to Sitecore Project

Project specific Continuous Integration and Deployment Architecture
Sitecore Project Specific Continuous Integration and Deployment Architecture


















This will be explained in detailed in the subsequent blogs

Tools and Technologies

Tools and Technologies used in Sitecore Continuous Integration and Deployment
Tools Used in Sitecore Continuous Integration and Deployment

Please note: We are using TDS in all three phases. 

  • In Development to work against local Sitecore instance and serialize our items in source control system
  • In Continuous Integration to create Sitecore Packages
  • And In deployment we are using PackageInstaller.exe that comes with TDS to install Sitecore packages using Powershell.


Brief considerations for Development and Continuous Integration

Development Environment Considerations

  1. Use Team Development for Sitecore (TDS) from Hedgehog to work with Sitecore items directly in Visual Studio
  2. If you have multiple teams working on the same Sitecore platform, better separate their responsibilities and areas in Sitecore where they work. TDS allows you to sync the children under the parent but if you have two teams working on the same folder / root item, when you install the package with these settings, you may step over one another and end state will not be what you desire.
  3. Work against your local Sitecore sandbox environment than central Sitecore development server, otherwise, it will add limits to your capabilities to do parallel streams of development without having additional environments for each individual team working in parallel sprints / across multiple branches in source code repository
  4. Consider cloud based source code repository such as Bitbucket OR any other GIT based repositories. This will allow you to do feature based development, facilitate multi-geo development and avoid additional infrastructure to support. This will also depend on your legal and regulatory compliance.
  5. Bring in test driven development right from beginning using NUnit, XUnit and FakeDB for mocking Sitecore
  6. Use SlowCheetah to create configuration transformation files (other than web.config). This way you can also preview transformed configurations right in the visual studio.

Build and Continuous Integration (CI) Considerations

  1. For CI, you can use any of the Continuous Integration platforms as long as it supports MS Build, PowerShell and also has plug-ins to generate Nuget Packages.
  2. In our environment we generate two separate packages. First package contains our web deploy solution such as files (aspx, cshtml, css, js, dll), configurations (*.config including config transformation files) and the other one contains a Sitecore pacakge with all Sitecore items serialized using TDS. Both these pacakges are created as Nuget packages. Sitecore package is wrapped in a Nuget package
  3. We have customized the package creation for Sitecore to generate incremental package compared to the last production release. This is done by customizing Sitecore Courier and using Sitecore.Update dll
  4. In addition you can run unit test cases built on frameworks mentioned above, run static code analysis using FxCop / StyleCop and submit its report to products like SonarQube
  5. Once the packages are built, CI platform publishes these package into the Nuget feed hosted by Nuget servers such as Proget. You can leverage this server to matinain enterprise nuget feeds and keep all your nuget dependencies project / enterprise wide.
In the next blog I will talk about overall deployment process and considerations for the same using OctopusDeploy.