Since Argo CD is so user-friendly, we tend to use it to keep our Homelab maintained. Besides just this main purpose, it can also be useful for inspecting issues that may arise when we try something new. Flux CD is another popular solution that handles the similar issue of GitOps-ing your cluster, but we prefer this one.

Kustomize and Helm 

Not even a Helm chart can have everything you need nicely templated, so you might look to another solution to keep things clean and organized. ArgoCD supports Helm and Kustomize out of the box, but not both at the same time. Before verifying the personalized manifest into Git, you could fully render the Helm template and manually change it, but that would be a far less elegant method that would also be more difficult to maintain.

To achieve your goals, it would be preferable to combine the might of Helm and Kustomize.

The Old Ways

Patching the argocd-cm ConfigMap was one method of achieving Kustomized Helm before Argo CD v2.8.

<pre class="codeWrap"><code>apiVersion: v1
kind: ConfigMap
metadata:
 name: argocd-cm
data:
 configManagementPlugins: |
   - name: kustomize-build-with-helm
     generate:
       command: [ "sh", "-c" ]
       args: [ "kustomize build --enable-helm" ]
</code></pre>

This could also be done by using Kustomize.

<pre class="codeWrap"><code>apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: argocd
...
patches:
 - path: patches/argocd-cm-kustomize-helm-patch.yaml
</code></pre>

As we did not receive the notification that it had been deprecated since Argo CD v2.4, this is what we have been doing. We did think this was the right method to accomplish it because the official Argo CD sample apps repo showed a similar technique. But, alas we were wrong. So, let's explore other venues.

Config Management Plugin

Our initial effort (commit) was only adhering to the Config Management Plugins (CMP) documentation found on the Argo CD, which resulted in a ConfigMap containing the CMP-config (CMP is not a CRD, but it does follow the Kubernetes-style standard convention)

<pre class="codeWrap"><code>apiVersion: v1
kind: ConfigMap
metadata:
 name: argocd-cm-cmp-kustomize-build-with-helm
data:
 plugin.yaml: |
   apiVersion: argoproj.io/v1alpha1
   kind: ConfigManagementPlugin
   metadata:
     name: kustomize-build-with-helm
   spec:
     generate:
       command: [ "sh", "-c" ]
       args: [ "kustomize build --enable-helm" ]
</code></pre>

We also added a patch to start a CMP-sidecar with the following setup by deploying the argocd-repo-server which allows the upgrading to Argo CD v2.8 without any issues.

<pre class="codeWrap"><code>apiVersion: apps/v1
kind: Deployment
metadata:
 name: argocd-repo-server
spec:
 template:
   spec:
     containers:
       - name: kustomize-build-with-helm
         command: [ /var/run/argocd/argocd-cmp-server ]
         image: quay.io/argoproj/argocd:v2.7.11
         securityContext:
           runAsNonRoot: true
           runAsUser: 999
         volumeMounts:
           - name: var-files
             mountPath: /var/run/argocd
           - name: plugins
             mountPath: /home/argocd/cmp-server/plugins
           - name: argocd-plugin-config
             mountPath: /home/argocd/cmp-server/config/plugin.yaml
             subPath: plugin.yaml
           - mountPath: /tmp
             name: cmp-tmp
     volumes:
       - name: argocd-plugin-config
         configMap:
           name: argocd-cm-cmp-kustomize-build-with-helm
       - name: cmp-tmp
         emptyDir: { }
</code></pre>

But there's so much more we can do with this.

Taking It a Step Further

Although the prior quick repair is effective, it has the bothersome drawback that we have to maintain the alignment of the main image and the patched Argo CD image tag. This section aims to develop a relatively simple solution that requires less manual upkeep. Take a look at the final commit if you want to skip the somewhat long explanation.

‍

It may have caught your attention that the patch references volumes in the containers section that don't appear to be defined in the volumes section. What's the deal? Taking a step back, the answer is already in the question's formulation: this is a patch, and the complete manifest is required to determine the true cause of the issue.

‍

If we take a closer look at the argocd-repo-server deployment, we can see an initContainer that copies the argocd-binary to the mystery var-files. We edited it here, for the sake of brevity.

<pre class="codeWrap"><code>apiVersion: apps/v1
kind: Deployment
metadata:
 name: argocd-repo-server
spec:
 ...
 template:
   ...
   spec:
     ...
     initContainers:
         - command:
           - /bin/cp
           - -n
           - /usr/local/bin/argocd
           - /var/run/argocd/argocd-cmp-server
           image: quay.io/argoproj/argocd:v2.8.2
           name: copyutil
           volumeMounts:
             - mountPath: /var/run/argocd
               name: var-files
           ...
     volumes:
       - emptyDir: { }
          name: var-files
       - emptyDir: { }
         name: plugins
       - ...
</code></pre>

This seems useless on its own, and if you're using the stock Argo CD, we would contend that it is. Without going into too much detail, We'll assume that it's done to simplify the use of sidecar plugins.

‍

Further investigation reveals that the argocd-binary is soft-linked multiple times in the argocd-repo-server container, each time with names that allude to distinct Argo CD components.

<pre class="codeWrap"><code>argocd@argocd-repo-server-6589ffb4b6-4rdvp:/usr/local/bin$ ls -lah
total 205M
drwxr-xr-x 1 root root 4.0K Aug 24 20:34 .
drwxr-xr-x 1 root root 4.0K Jun 24 02:02 ..
-rwxr-xr-x 1 root root 142M Aug 24 20:34 argocd
lrwxrwxrwx 1 root root   21 Aug 24 20:34 argocd-application-controller -> /usr/local/bin/argocd
lrwxrwxrwx 1 root root   21 Aug 24 20:34 argocd-applicationset-controller -> /usr/local/bin/argocd
lrwxrwxrwx 1 root root   21 Aug 24 20:34 argocd-cmp-server -> /usr/local/bin/argocd
lrwxrwxrwx 1 root root   21 Aug 24 20:34 argocd-dex -> /usr/local/bin/argocd
lrwxrwxrwx 1 root root   21 Aug 24 20:34 argocd-k8s-auth -> /usr/local/bin/argocd
lrwxrwxrwx 1 root root   21 Aug 24 20:34 argocd-notifications -> /usr/local/bin/argocd
lrwxrwxrwx 1 root root   21 Aug 24 20:34 argocd-repo-server -> /usr/local/bin/argocd
lrwxrwxrwx 1 root root   21 Aug 24 20:34 argocd-server -> /usr/local/bin/argocd
-rwxr-xr-x 1 root root  203 Aug 24 20:05 entrypoint.sh
-rwxr-xr-x 1 root root  934 Aug 24 20:05 git-verify-wrapper.sh
-rwxr-xr-x 1 root root  215 Aug 24 20:05 gpg-wrapper.sh
-rwxr-xr-x 1 root root  49M Aug 24 20:08 helm
-rwxr-xr-x 1 root root  15M Aug 24 20:08 kustomize
lrwxrwxrwx 1 root root   28 Aug 24 20:08 uid_entrypoint.sh -> /usr/local/bin/entrypoint.sh
</code></pre>

Even though it seems fascinating, and we may take our time to write another article about it, let's not get derailed, and stay on topic. Argo recommends that you use all the tools necessary to make your image, which is undoubtedly a better option than the impending business environment hack, but this is a homelab after all.

‍

Why not attempt to clone the helm and kustomize binaries to a minimum base image, given that we know we need them both and assume they are standalone? After that, we'll use a patch to take control of the copy util-initContainer and, implicitly, the var-files volume.

<pre class="codeWrap"><code>apiVersion: apps/v1
kind: Deployment
metadata:
 name: argocd-repo-server
spec:
 template:
   spec:
     initContainers:
       - name: copyutil
         command: [ /bin/bash ]
         args:
           - -c
           - >-
             /bin/cp -n /usr/local/bin/argocd /var/run/argocd/argocd-cmp-server &&
             /bin/cp -n /usr/local/bin/kustomize /var/run/argocd/kustomize &&
             /bin/cp -n /usr/local/bin/helm /var/run/argocd/helm
</code></pre>

After that, we attempted to use a busybox-image, inspired by the Argo CD CMP example, but this lead to a missing git problem. Initially, we thought to attempt copying the git binary as well, however, that resulted in an error stating that the libpcre2-8.so.0 library was missing. Alright, so that only sort of sort of worked. We might attempt to locate every library that git would require, but at this point, it would be simpler to simply install git in a separate image along with Helm and Kustomize, but that would require additional maintenance.

‍

In the end, we located a minimal image with git already installed and copied the helm and Kustomize binaries from the Argo CD image, just like in the prior patch. Alpine/git was chosen as the minimal-image-with-git-search solution since it appeared to be independent and well-maintained.

‍

The location of the var-files volume has to be added to the PATH variable to complete the work at hand. We foolishly tried /var/run/argocd:$(PATH) at first, thinking $PATH would resolve to the one in the image. However, looking back, we realize this would never work because there is no way to see inside the container in advance and because, as the Kubernetes documentation states, you can only resolve already defined env-variable.

‍

After realizing the error, we chose to define the PATH variable explicitly, doing away with any entry point tricks. The following atomic fix for an Argo CD CMP-sidecar was the consequence of this.

<pre class="codeWrap"><code>apiVersion: apps/v1
kind: Deployment
metadata:
 name: argocd-repo-server
spec:
 template:
   spec:
     containers:
       - name: kustomize-build-with-helm
         env:
           - name: PATH
              value: "/var/run/argocd:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
         command:
           - argocd-cmp-server
         image: alpine/git:2.40.1
         securityContext:
           runAsNonRoot: true
           runAsUser: 999
         volumeMounts:
           - name: var-files
             mountPath: /var/run/argocd
           - name: plugins
            mountPath: /home/argocd/cmp-server/plugins
           - name: cmp-kustomize-build-with-helm
             mountPath: /home/argocd/cmp-server/config/plugin.yaml
             subPath: kustomize-build-with-helm.yaml
           - mountPath: /tmp
              name: cmp-tmp
     volumes:
       - name: cmp-kustomize-build-with-helm
         configMap:
           name: argocd-cm-cmp-kustomize-build-with-helm
       - name: cmp-tmp
         emptyDir: { }
</code></pre>

This is our current solution for using Argo CD with Kustomized Helm in conjunction with the previously specified ConfigMap and initContainer patch.