I was recently asked to implement a simple Continuous Integration (CI) and Continuous Deployment (CD) process for a customer. The aim was to set up an automated build and an automated deployment job for a Blazor application, whereby the deployment job includes the execution of the Infrastructure as Code (IaC) changes and EFCore migrations. I have already implemented such scenarios several times and especially in different tools:
- Azure DevOps (YAML pipelines)
- GitHub Actions
- JetBrains TeamCity
- Jenkins
- GitLab
- Atlassian Bamboo
- Bitbucket
The challenging part was, that the tool I needed to implement the pipelines for was JetBrains Space and I had never used it before. JetBrains advertises Space as “The Intelligent Code Collaboration Platform”. The platform provides a pretty huge bunch of features that support the development workflow. From Software Development features like source code management, code reviews, cloud development environments and automation (CI/CD) over project management features to collaboration and communication features. It’s an all in one place, extensible solution which is fully integrated with JetBrains IDEs.
I already worked with JetBrains products like IDE, ReSharper, TeamCity and PyCharm. Except TeamCity, I love all of the before mentioned products as they are reliable, performant and really developer friendly. In my opinion, back then TeamCity was a little awkward to use in comparison to other CI/CD tools – but to be fair, I have to say that TeamCity worked well too.
Curious as I am, I was looking forward to learning something new. As usual, I first took a look at the JetBrains Space documentation. As always with JetBrains, the documentation is very good. I was particularly interested in the automation feature – so the following sections are mainly dedicated to the automation function of JetBrains Space.
I started by creating the .space.kts file in the repository root and used the official .NET and .NET Core example out of the documentation as first version. After two days, many tryouts and more than hundred failed runs I ended up with the following solution.
Of course there is still a lot of room for improvement, such as moving script code to script files, but for now it’s good.
val buildContainerImage = "mcr.microsoft.com/dotnet/sdk:8.0"
val ubuntuImage = "ubuntu:22.04"
job("Continuous integration build") {
startOn {
gitPush {
enabled = true
anyBranch()
}
}
requirements {
// https://www.jetbrains.com/help/space/cloud-workers.html
workerPool = WorkerPools.SPACE_CLOUD
workerType = WorkerTypes.SPACE_CLOUD_UBUNTU_LTS_LARGE
}
container(displayName = "Build and test ExampleApplication", image = buildContainerImage) {
// https://www.jetbrains.com/help/space/using-service-containers.html#setting-service-container-resources
resources {
cpu = 4.cpu
memory = 8.gb
}
shellScript {
content = """
echo Run .NET build...
dotnet build ExampleApplication.sln --configuration Release
if [ $? -ne 0 ]; then
echo "Build failed"
exit 1
fi
echo Run .NET tests...
dotnet test ExampleApplication.sln --configuration Release
if [ $? -ne 0 ]; then
echo "Tests failed"
exit 1
fi
"""
}
}
}
job("Continuous deployment") {
startOn {
gitPush {
enabled = true
anyBranchMatching {
+"dev"
+"test"
+"main"
}
}
}
requirements {
// https://www.jetbrains.com/help/space/cloud-workers.html
workerPool = WorkerPools.SPACE_CLOUD
workerType = WorkerTypes.SPACE_CLOUD_UBUNTU_LTS_LARGE
}
container(displayName = "Build, test and publish ExampleApplication", image = buildContainerImage) {
// https://www.jetbrains.com/help/space/using-service-containers.html#setting-service-container-resources
resources {
cpu = 4.cpu
memory = 8.gb
}
shellScript {
content = """
echo Run .NET build...
dotnet build ExampleApplication.sln --configuration Release
if [ $? -ne 0 ]; then
echo "Build failed"
exit 1
fi
echo Run .NET tests...
dotnet test ExampleApplication.sln --configuration Release
if [ $? -ne 0 ]; then
echo "Tests failed"
exit 1
fi
echo Run .NET publish...
dotnet publish src/Server/ExampleApplication.Server.csproj --configuration Release --output build/publish/app/ --no-build
if [ $? -ne 0 ]; then
echo "Publishing ExampleApplication failed"
exit 1
fi
echo Run .NET publish...
dotnet publish src/Domain.Migrations/ExampleApplication.Domain.Migrations.csproj --configuration Release --output build/publish/migrations/ --no-build
if [ $? -ne 0 ]; then
echo "Publishing ExampleApplication.Domain.Migrations failed"
exit 1
fi
"""
}
fileArtifacts {
localPath = "build/publish/app/"
// Fail job if build/publish/app/ is not found
optional = false
remotePath = "{{ run:number }}/ExampleApplication.gz"
archive = true
onStatus = OnStatus.SUCCESS
}
fileArtifacts {
localPath = "build/publish/migrations/"
// Fail job if build/publish/migrations/ is not found
optional = false
remotePath = "{{ run:number }}/ExampleApplicationMigrations.gz"
archive = true
onStatus = OnStatus.SUCCESS
}
}
host(displayName = "Create job parameters") {
env["DB_CONNECTION_STRING_DEV"] = "{{ project:db_connection_string_dev }}"
env["AZURE_CLIENT_ID_DEV"] = "{{ project:az_clientid_dev }}"
env["AZURE_CLIENT_SECRET_DEV"] = "{{ project:az_client_secret_dev }}"
kotlinScript { api ->
api.parameters["Environment"] = when (api.gitBranch()) {
"refs/heads/main" -> "prod"
"refs/heads/test" -> "test"
"refs/heads/dev" -> "dev"
else -> ""
}
api.parameters["EnvironmentCode"] = when (api.gitBranch()) {
"refs/heads/main" -> "p1"
"refs/heads/test" -> "t1"
"refs/heads/dev" -> "d1"
else -> ""
}
api.parameters["AzureTenant"] = when (api.gitBranch()) {
"refs/heads/main" -> "PROD_TENANT_ID_HERE"
"refs/heads/test" -> "TEST_TENANT_ID_HERE"
"refs/heads/dev" -> "DEV_TENANT_ID_HERE"
else -> ""
}
api.parameters["AzureClientId"] = when (api.gitBranch()) {
"refs/heads/main" -> ""
"refs/heads/test" -> ""
"refs/heads/dev" -> System.getenv("AZURE_CLIENT_ID_DEV")
else -> ""
}
api.parameters["AzureClientSecret"] = when (api.gitBranch()) {
"refs/heads/main" -> ""
"refs/heads/test" -> ""
"refs/heads/dev" -> System.getenv("AZURE_CLIENT_SECRET_DEV")
else -> ""
}
api.parameters["DbConnectionString"] = when (api.gitBranch()) {
"refs/heads/main" -> ""
"refs/heads/test" -> ""
"refs/heads/dev" -> System.getenv("DB_CONNECTION_STRING_DEV")
else -> ""
}
}
}
container(displayName = "Start deployment", image = "amazoncorretto:17-alpine") {
env["TARGET_ENV"] = "{{ Environment }}"
kotlinScript { api ->
api.space().projects.automation.deployments.start(
project = api.projectIdentifier(),
targetIdentifier = TargetIdentifier.Key(System.getenv("TARGET_ENV")),
version = System.getenv("JB_SPACE_EXECUTION_NUMBER"),
syncWithAutomationJob = true
)
}
}
host(displayName = "Run terraform apply and apply EF Core migrations") {
// https://registry.terraform.io/providers/hashicorp/azuread/latest/docs/guides/service_principal_client_secret
env["ARM_CLIENT_ID"] = "{{ AzureClientId }}"
env["ARM_CLIENT_SECRET"] = "{{ AzureClientSecret }}"
env["ARM_TENANT_ID"] = "{{ AzureTenant }}"
env["AZURE_CLIENT_ID"] = "{{ AzureClientId }}"
env["AZURE_CLIENT_SECRET"] = "{{ AzureClientSecret }}"
env["AZURE_TENANT_ID"] = "{{ AzureTenant }}"
env["DB_CONNECTION_STRING"] = "{{ DbConnectionString }}"
env["ENVIRONMENT"] = "{{ Environment }}"
env["ENVIRONMENT_CODE"] = "{{ EnvironmentCode }}"
fileInput {
source = FileSource.FileArtifact(
"{{ run:file-artifacts.default-repository }}/{{ run:file-artifacts.default-base-path }}",
"{{ run:number }}/ExampleApplicationMigrations.gz",
extract = true
)
localPath = "build"
}
shellScript {
content = """
# disable unattended upgrades to avoid unexpected locks
sudo apt remove unattended-upgrade
# Az CLI - https://learn.microsoft.com/en-us/cli/azure/install-azure-cli-linux?pivots=apt#option-1-install-with-one-command
curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash
# PoSH - https://learn.microsoft.com/en-us/powershell/scripting/install/install-ubuntu?view=powershell-7.4
sudo apt-get update
sudo apt-get install -y wget apt-transport-https software-properties-common
source /etc/os-release
wget -q https://packages.microsoft.com/config/ubuntu/22.04/packages-microsoft-prod.deb
sudo dpkg -i packages-microsoft-prod.deb
rm packages-microsoft-prod.deb
sudo apt-get update
sudo apt-get install -y powershell
# Terraform - https://computingforgeeks.com/how-to-install-terraform-on-ubuntu/
sudo apt update
sudo apt install software-properties-common gnupg2 curl
curl https://apt.releases.hashicorp.com/gpg | gpg --dearmor > hashicorp.gpg
sudo install -o root -g root -m 644 hashicorp.gpg /etc/apt/trusted.gpg.d/
sudo apt-add-repository "deb [arch=$(dpkg --print-architecture)] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
sudo apt install terraform=1.6.6-*
cd infra/iac
terraform init -reconfigure --backend-config=./backend/${'$'}ENVIRONMENT.backend.tfvars -no-color
terraform workspace select ${'$'}ENVIRONMENT_CODE -no-color
if [ "${'$'}ENVIRONMENT_CODE" == "d1" ]; then
terraform apply -auto-approve -var-file="./vars/${'$'}ENVIRONMENT.${'$'}ENVIRONMENT_CODE.tfvars" -no-color
else
terraform apply -auto-approve -var-file="./vars/${'$'}ENVIRONMENT.${'$'}ENVIRONMENT_CODE.tfvars" -no-color -replace azuread_application_password.aadapppwd
fi
# .NET Core SDK - https://learn.microsoft.com/en-us/dotnet/core/install/linux-ubuntu-2204#install-the-sdk
wget https://packages.microsoft.com/config/ubuntu/22.04/packages-microsoft-prod.deb -O packages-microsoft-prod.deb
sudo dpkg -i packages-microsoft-prod.deb
rm packages-microsoft-prod.deb
sudo apt-get update
sudo apt-get install -y dotnet-host
sudo apt-get install -y dotnet-sdk-8.0
az login --service-principal -u ${'$'}AZURE_CLIENT_ID -p ${'$'}AZURE_CLIENT_SECRET --tenant ${'$'}AZURE_TENANT_ID
cd ../../build
dotnet "ExampleApplication.Domain.Migrations.dll" --Migration:IsProduction=true --Migration:ConnectionString="${'$'}DB_CONNECTION_STRING"
"""
}
}
host(displayName = "Run deploy script") {
fileInput {
source = FileSource.FileArtifact(
"{{ run:file-artifacts.default-repository }}/{{ run:file-artifacts.default-base-path }}",
"{{ run:number }}/ExampleApplication.gz",
extract = true
)
localPath = "build"
}
env["AZURE_CLIENT_ID"] = "{{ AzureClientId }}"
env["AZURE_CLIENT_SECRET"] = "{{ AzureClientSecret }}"
env["AZURE_TENANT_ID"] = "{{ AzureTenant }}"
env["ENVIRONMENT_CODE"] = "{{ EnvironmentCode }}"
shellScript {
content = """
sudo apt install zip
cd build
zip -r ExampleApplication.zip .
# Az CLI - https://learn.microsoft.com/en-us/cli/azure/install-azure-cli-linux?pivots=apt#option-1-install-with-one-command
curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash
az login --service-principal -u ${'$'}AZURE_CLIENT_ID -p ${'$'}AZURE_CLIENT_SECRET --tenant ${'$'}AZURE_TENANT_ID
az webapp deploy --resource-group cp-${'$'}ENVIRONMENT_CODE-rg-exampleapplication --name cp-${'$'}ENVIRONMENT_CODE-appsrv-exampleapplication --src-path ExampleApplication.zip
"""
}
}
}
Let’s have a look a the most important parts.
To reduce the execution time of host steps, I updated the worker requirements to large (4 vCPU and 15600 MB):
requirements {
// https://www.jetbrains.com/help/space/cloud-workers.html
workerPool = WorkerPools.SPACE_CLOUD
workerType = WorkerTypes.SPACE_CLOUD_UBUNTU_LTS_LARGE
}
Job parameters can be used to pass data between automation steps. The host step “Create job parameters” creates such job parameters and conditionally assigns environment specific values to them. The values (mainly secrets) are stored as project-wide secrets (i.e. `az_client_secret_dev`).
host(displayName = "Create job parameters") {
env["DB_CONNECTION_STRING_DEV"] = "{{ project:db_connection_string_dev }}"
env["AZURE_CLIENT_ID_DEV"] = "{{ project:az_clientid_dev }}"
env["AZURE_CLIENT_SECRET_DEV"] = "{{ project:az_client_secret_dev }}"
kotlinScript { api ->
api.parameters["Environment"] = when (api.gitBranch()) {
"refs/heads/main" -> "prod"
"refs/heads/test" -> "test"
"refs/heads/dev" -> "dev"
else -> ""
}
api.parameters["EnvironmentCode"] = when (api.gitBranch()) {
"refs/heads/main" -> "p1"
"refs/heads/test" -> "t1"
"refs/heads/dev" -> "d1"
else -> ""
}
api.parameters["AzureTenant"] = when (api.gitBranch()) {
"refs/heads/main" -> "PROD_TENANT_ID_HERE"
"refs/heads/test" -> "TEST_TENANT_ID_HERE"
"refs/heads/dev" -> "DEV_TENANT_ID_HERE"
else -> ""
}
api.parameters["AzureClientId"] = when (api.gitBranch()) {
"refs/heads/main" -> ""
"refs/heads/test" -> ""
"refs/heads/dev" -> System.getenv("AZURE_CLIENT_ID_DEV")
else -> ""
}
api.parameters["AzureClientSecret"] = when (api.gitBranch()) {
"refs/heads/main" -> ""
"refs/heads/test" -> ""
"refs/heads/dev" -> System.getenv("AZURE_CLIENT_SECRET_DEV")
else -> ""
}
api.parameters["DbConnectionString"] = when (api.gitBranch()) {
"refs/heads/main" -> ""
"refs/heads/test" -> ""
"refs/heads/dev" -> System.getenv("DB_CONNECTION_STRING_DEV")
else -> ""
}
}
}
The last two steps took quite a lot of time and nerves. The main challenges were to install the various Linux packages correctly, to specify the paths correctly and to create the ZIP file for the deployment without zipping the build folder.
Conclusion
The idea behind JetBrains Space is great and the documentation is good. Nevertheless, I’m a little disappointed with the implementation of the CI/CD features (Automation (CI/CD)). The approach is super flexible but I miss predefined steps, tasks or containers for standard cases in context of popular languages (i.e. build .NET application or deploy .NET solution to Azure App Service) as it’s too much unnecessary effort to set up such a job like the above from scratch. To make matters worse, the community feels small and consequently there are only a few examples.
One thing I don’t like is the fact that project-wide secrets get exposed as plain text in runtime parameters of job runs if they get passed to the host as environment variable as suggested by the official docs.
Leave a Reply