GitOps with ArgoCD

What is GitOps and Argo CD?

GitOps is a modern approach to managing infrastructure and deployments that relies on Git as the single source of truth for code and configuration. This approach enables teams to version infrastructure and deployments, automate the deployment process, promote collaboration and transparency, and ensure consistency across environments. By using tools like Argo CD, teams can streamline their deployment process and ensure that their applications are deployed consistently and reliably.

Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes. It is designed to automate the deployment of applications to Kubernetes clusters, and it provides a centralized view of the state of a cluster's applications and resources. Argo CD also supports rollbacks, can manage multiple clusters, and integrates with other GitOps tools.

Why GitOps and why ArgoCD?

GitOps and Argo CD provide a number of benefits for managing infrastructure and deployments. By using Git as the single source of truth for code and configuration, teams can version infrastructure and deployments, automate the deployment process, promote collaboration and transparency, and ensure consistency across environments. Argo CD, specifically, is a powerful tool for managing Kubernetes deployments, providing a centralized view of application state, support for rollbacks, and a range of tools for managing deployments.

GitOps offers a number of benefits, including versioning, automation, collaboration, and consistency. By using tools like Argo CD and following GitOps practices, teams can streamline their deployment process and ensure that their applications are deployed consistently and reliably.

Argo CD provides a web-based user interface that enables teams to view and manage deployments in a centralized way. The UI displays the state of applications and resources across all clusters, making it easy to see which deployments are up-to-date and which ones need attention.

The UI also provides a number of tools for managing deployments. For example, teams can use the UI to manually trigger a deployment or rollback, view deployment history, and compare different versions of an application.

Argo CD is a Kubernetes controller that continuously monitors running applications and compares the current live state against the desired target state specified in a Git repository. It provides the following features and functionalities:

• Out-of-sync detection: Argo CD identifies deployed applications whose live state deviates from the desired target state. These applications are considered "OutOfSync".
• Reporting and visualization: Argo CD reports and visualizes the differences between the live state and the desired state of applications.
• Automatic and manual synchronization: Argo CD provides facilities to automatically or manually sync the live state back to the desired target state.
• Git repository integration: Any modifications made to the desired target state in the Git repository can be automatically applied and reflected in the specified target environments.

Getting Started

ArgoCD Installation

You can install ArgoCD in your cluster by following the instructions in the official Argo CD documentation. The installation process involves creating a Kubernetes manifest and deploying it to your cluster. Once you have Argo CD installed, you can use the web-based user interface to manage your deployments, or you can use the Argo CD CLI to manage deployments from your terminal. Before you start using Argo CD, it's a good idea to familiarize yourself with GitOps practices and the basic concepts of Kubernetes deployments.

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Before creating an ingress that points to the argocd-server, you need to first edit that resource by running:

kubectl edit deployment -n argocd argocd-server

And adding the --insecure flag in the manifest like so:

55    containers:
56    - command:
57      - argocd-server
58      - --insecure

https://github.com/argoproj/argo-cd/discussions/8520

After editing the argocd-server deployment, the ingress should look something like this:

kind: Ingress
apiVersion: networking.k8s.io/v1
metadata:
  name: argocd-ingress
  namespace: argocd
spec:
  ingressClassName: nginx
  rules:
    - host: <your-host> # argocd.site.dev
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: argocd-server
                port:
                  number: 80
status:
  loadBalancer:
    ingress:
      - ip: 127.0.0.1

At this point, the ArgoCD UI should be up and running, and you can access it by going to the address you specified in your ingress where it will ask you to login.

The username is admin and you can find the password inside the argocd-initial-admin-secret secret by running:

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d

Configure the app resync time

The purpose of configuring the app resync time in Argo CD is to control how frequently applications are synced between the live state and the desired target state. By adjusting the resync time, you can ensure that applications are synchronized according to your desired frequency.

To configure the app resync time, you need to make the following changes:

Add the data.timeout.reconciliation key in the argocd-cm configmap:

# argocd-cm

apiVersion: v1
data:
  accounts.admin: apiKey
  timeout.reconciliation: 3s # Update or add this field to the time you want your applications to be synced
kind: ConfigMap
metadata:
  annotations:
  labels:
    app.kubernetes.io/name: argocd-cm
    app.kubernetes.io/part-of: argocd
  name: argocd-cm
  namespace: argocd

Once you have made these changes, you will need to restart the following deployments to apply the new configuration:

• argocd-repo-server - Deployment,
• argocd-server - Deployment,
• argocd-applications-controller - StatefulSet

Image Updater Installation

The Argo CD Image Updater is a tool that automatically updates the container images of Kubernetes workloads managed by Argo CD. It checks for new versions of the container images deployed with your Kubernetes workloads and automatically updates them to their latest allowed version using Argo CD.

The Argo CD Image Updater works by setting appropriate application parameters for Argo CD applications in a fully automated manner. It uses annotations on Argo CD Application resources to track image versions and updates them by setting parameter overrides using the Argo CD API.

However, it's important to note that the Argo CD Image Updater does not update your manifests. Instead, it re-configures your Application sources to use the new image tag, and control is handed over to Argo CD. Depending on your Automatic Sync Policy for the Application, Argo CD will either automatically deploy the new image version or mark the Application as Out Of Sync, and you can trigger the image update manually by syncing the Application.

kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj-labs/argocd-image-updater/stable/manifests/install.yaml

Configure image updater

Edit the argocd-image-updater-config configmap and add the following keys.

#argocd-image-updater-config.yml 

apiVersion: v1
data:

# Configure the desired log level
  log.level: debug 

# Configure the image registry you want the image updater to check 
  registries.conf: |
    registries:
    - name: DockerHub
      prefix: docker.io
      api_url: https://index.docker.io/
      credentials: secret:<your-namespace>/<secret-name>

kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/name: argocd-image-updater-config
    app.kubernetes.io/part-of: argocd-image-updater
  name: argocd-image-updater-config
  namespace: argocd

You can create the secret using this command:

kubectl create secret -n argocd docker-registry <secret-name> \
--docker-server=index.docker.io/v1/ \
--docker-username=<dockerhub-username> \
--docker-password=<dockerhub-password> \

Configure the image updater sync time

Edit the argocd-image-updater deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/name: argocd-image-updater
    app.kubernetes.io/part-of: argocd-image-updater
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: argocd-image-updater
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app.kubernetes.io/name: argocd-image-updater
    spec:
      containers:
      - args:
        - --interval
        - 0m5s

...

This will do a check on your container registry (dockerhub) every 5 seconds and update the image tag in your version control.

Application deployment

Now that everything is set we’re going to continue with the deployment of our helm charts which I have stored in version control.

We’re going to deploy a backstage image, which is stored in the docker registry that we specified in the argocd-image-updater-config configmap earlier.

Connect the repo

Log in to the Argo CD UI, navigate to /settings/repo and click on “CONNECT REPO”.

Choose your connection method (HTTPS) and fill in the required fields.

Creating the application

My backstage helm chart is simple, it contains a deployment, service and two values files, one for each environment (development and production).

We also need to create an ‘Application.yaml’ manifest that will sync with the specified helm chart.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: backstage-production
  namespace: argocd
  annotations:
    argocd-image-updater.argoproj.io/image-list: docker.io/<docker-repository>/backstage.production.image
    argocd-image-updater.argoproj.io/write-back-method: git
    argocd-image-updater.argoproj.io/git-branch: main 
    argocd-image-updater.argoproj.io/update-strategy: latest
spec:
  destination:
    namespace: spotify
    server: https://kubernetes.default.svc
  source:
    path: Backstage/
    repoURL: 'https://<AZURE_ORG>@dev.azure.com/<AZURE_ORG>/Spotify/_git/Argo-Manifests'
    targetRevision: HEAD
    helm:
      valueFiles:
        - values.yaml
  project: default
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
      allowEmpty: true

Click on the “NEW APP” button in the Applications directory then click on “EDIT AS YAML” and paste the ‘Application.yaml’ there.

Save it and click on “CREATE”. This should create a new application with Continuous Delivery/Deployment which will be visible in the Applications directory. When a new build is created, the image updater will check for changes in you docker registry, get the latest image tag and create a file in your helm chart with the new tag.

This will trigger Argo CD to sync the live state of the application with the desired state in the helm chart which will happen in a matter of seconds.

ArgoCD Notifications

Let’s cover how to send notifications about the state of the applications to google chat. We can do this using the argocd-notifications-controller resource. The argocd-notifications-controller should already be running but we are going to install some default Triggers and Templates from the catalog. Do this by running:

kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/notifications_catalog/install.yaml

This command will create a configmap named argocd-notifications-cm which you can edit to add notification services (in our case googlechat), create custom messages and triggers.

Create a google chat webhook by following this tutorial and add it to the argocd-notifications-cm like so:

data:
  service.googlechat: |
    webhooks:
      <service-name>: <google-chat-webhook>

Let’s also edit the templates in this cm to send card messages instead of plain text and configure the triggers so that this message is sent only if the Application has been OutOfSync for more than 5 minutes and is not older than 20 minutes (sometimes argocd sends really old messages - this will prevent that)..

data:
  service.googlechat: |
    webhooks:
      argoGChat: "https://chat.googleapis.com/v1/spaces/AAAANVw1AhQ/messages?key=AIzaSyDdI0hCZtE6vySjMm-WEfRq3CPzqKqqsHI&token=5Hcn6EHtB2TzGxxo4x3pLSeCFoXawvbUqM6QYUt1t5o"
  template.app-health-progressing: |
    googlechat:
      cards: |
        - header:
            title: 🔥 Deployment Not Running
          sections:
            - widgets:
                - keyValue:
                    topLabel: Deployment
                    content: {{ $.app.metadata.name }}
                - keyValue:
                    topLabel: Status
                    content: {{ .app.status.sync.status }}
                - keyValue:
                    topLabel: Time (GMT+2)
                    content: {{ (call .time.Parse .app.status.operationState.startedAt).Local.Format "15:04 - Mon Jan 2 2006" }}
                    contentMultiline: true
                    button:
                      textButton:
                        text: OPEN
                        onClick:
                          openLink:
                            url: https://argocd.datadrift.com/applications/argocd/{{ $.app.metadata.name }}?resource=
  template.app-out-of-sync: |
    googlechat:
      cards: |
        - header:
            title: ⚠️ Deployment Out Of Sync 
          sections:
            - widgets:
                - keyValue:
                    topLabel: Deployment
                    content: {{ $.app.metadata.name }}
                - keyValue:
                    topLabel: Status
                    content: {{ .app.status.sync.status }}
                - keyValue:
                    topLabel: Time (GMT+2)
                    content: {{ (call .time.Parse .app.status.operationState.startedAt).Local.Format "15:04 - Mon Jan 2 2006" }}
                    contentMultiline: true
                    button:
                      textButton:
                        text: OPEN
                        onClick:
                          openLink:
                            url: https://argocd.datadrift.com/applications/argocd/{{ $.app.metadata.name }}?resource=
  template.app-sync-status-unknown: |
    googlechat:
      cards: |
        - header:
            title: ⚪ Deployment Sync Status Unknown 
          sections:
            - widgets:
                - keyValue:
                    topLabel: Deployment
                    content: {{ $.app.metadata.name }}
                - keyValue:
                    topLabel: Status
                    content: {{ .app.status.sync.status }}
                - keyValue:
                    topLabel: Time (GMT+2)
                    content: {{ (call .time.Parse .app.status.operationState.startedAt).Local.Format "15:04 - Mon Jan 2 2006" }}
                    contentMultiline: true
                    button:
                      textButton:
                        text: OPEN
                        onClick:
                          openLink:
                            url: https://argocd.datadrift.com/applications/argocd/{{ $.app.metadata.name }}?resource=
  template.app-health-degraded: |
    googlechat:
      cards: |
        - header:
            title: 💔 Application Health Degraded
          sections:
            - widgets:
                - keyValue:
                    topLabel: Deployment
                    content: {{ $.app.metadata.name }}
                - keyValue:
                    topLabel: Status
                    content: {{ .app.status.sync.status }}
                - keyValue:
                    topLabel: Time (GMT+2)
                    content: {{ (call .time.Parse .app.status.operationState.startedAt).Local.Format "15:04 - Mon Jan 2 2006" }}
                    contentMultiline: true
                    button:
                      textButton:
                        text: OPEN
                        onClick:
                          openLink:
                            url: https://argocd.datadrift.com/applications/argocd/{{ $.app.metadata.name }}?resource= 
  trigger.on-health-progressing: |
    - description: Application is in Progressing state
      oncePer: app.status.operationState.syncResult.revision    
      send:
      - app-health-progressing
      when: app.status.health.status == 'Progressing' and time.Now().Sub(time.Parse(app.status.operationState.startedAt)).Minutes() >= 5 and time.Now().Sub(time.Parse(app.status.operationState.finishedAt)).Minutes() <= 20
  trigger.on-out-of-sync: |
    - description: Application is out of sync
      oncePer: app.status.operationState.syncResult.revision
      send:
      - app-out-of-sync
      when: app.status.sync.status == 'OutOfSync' and time.Now().Sub(time.Parse(app.status.operationState.startedAt)).Minutes() >= 5 and time.Now().Sub(time.Parse(app.status.operationState.finishedAt)).Minutes() <= 20
  trigger.on-sync-status-unknown: |
    - description: Application status is 'Unknown'
      oncePer: app.status.operationState.syncResult.revision
      send:
      - app-sync-status-unknown
      when: app.status.sync.status == 'Unknown' and time.Now().Sub(time.Parse(app.status.operationState.startedAt)).Minutes() >= 10 and time.Now().Sub(time.Parse(app.status.operationState.finishedAt)).Minutes() <= 15
  trigger.on-health-degraded: |
    - description: Application is in Degraded state
      oncePer: app.status.operationState.syncResult.revision    
      send:
      - app-health-degraded
      when: app.status.health.status == 'Degraded' and time.Now().Sub(time.Parse(app.status.operationState.startedAt)).Minutes() >= 1 and time.Now().Sub(time.Parse(app.status.operationState.finishedAt)).Minutes() <= 5

After saving the changes all that is left is to update the Application manifests to be subscribed to these events.

Open the ArgoCD UI and add the Notification Subscriptions to your desired Application like so:

You should replace argoGChat with the <service-name> you specified in configMap.

Users and RBAC

In the provided YAML snippet, I am configuring Users and Role-Based Access Control (RBAC) for Argo CD. RBAC is a method of regulating access to resources based on the roles of individual users within your organization. In Argo CD, RBAC enables the restriction of access to Argo CD resources.

The argocd-rbac-cm ConfigMap contains user-defined RBAC policies and role definitions. In this ConfigMap, I am defining permissions for two roles: role:org-admin and role:backend-team. The role:org-admin has permissions to perform various actions on applications, clusters, and repositories, including creating, updating, and deleting repositories. The role:backend-team has permissions to get information about applications, clusters, repositories, and projects.

The g, user_one, role:backend-team and similar lines map users to their respective roles. In this example, the users user_one, user_two, user_three, and user_four are assigned the role role:backend-team.

The policy.default field is used to set the default role which Argo CD falls back to when authorizing API requests. If this field is omitted or empty, users may still be able to log in, but they will not see any apps, projects, etc.

# argocd-rbac-cm

data:
  policy.csv: |
    p, role:org-admin, applications, *, */*, allow
    p, role:org-admin, clusters, get, *, allow
    p, role:org-admin, repositories, get, *, allow
    p, role:org-admin, repositories, create, *, allow
    p, role:org-admin, repositories, update, *, allow
    p, role:org-admin, repositories, delete, *, allow
    p, role:org-admin, logs, get, *, allow
    p, role:org-admin, exec, create, */*, allow    

    p, role:backend-team, applications, get, *, allow
    p, role:backend-team, clusters, get, *, allow
    p, role:backend-team, repositories, get, *, allow
    p, role:backend-team, projects, get, *, allow

    g, user_one, role:backend-team
    g, user_two, role:backend-team
    g, user_three, role:backend-team
    g, user_four , role:backend-team
  policy.default: role:''

Conclusion

This document has introduced the concepts and benefits of GitOps and Argo CD, and provided a step-by-step guide on how to install and use these tools to manage Kubernetes deployments. By following the GitOps principles and using Argo CD as a continuous delivery tool, teams can achieve faster, safer, and more reliable deployments of their applications. Argo CD also provides a user-friendly interface and various features to monitor and control the deployment process. Additionally, Argo CD Image Updater and Argo CD Notifications are useful extensions that can enhance the functionality and usability of Argo CD. Finally, this document has also shown how to configure users and RBAC for Argo CD, which allows for fine-grained access control and security of Argo CD resources.