CI / CD

This section is the operational map of how 6grain FrontEnd applications are delivered: where the build servers, registries, and clusters live, what the pipeline does step by step, and which knobs you touch to run it.

Every FrontEnd application follows one path — its git repository is built by TeamCity from a docker-compose.yml, pushed to AWS ECR, rolled onto EKS with kubectl set image, and served through Traefik, with a notification posted to Slack.

The delivery path

git repo  →  TeamCity  →  AWS ECR  →  EKS + Traefik  →  Slack
(compose)    (build)      (push tag)   (set image)       (notify)

The same five stages run for every app on both environments. What differs between apps is values, not the shape of the path — those values are the six parameters described below.

Where things live

Staging and production are fully separate stacks — separate TeamCity servers, separate clusters, separate ECR tag prefixes. When something is missing, this table says which layer to open.

Layer	Staging	Production	Notes
CI server	`teamcity.stg.ap.6grain.com`	`teamcity.agroplatform.6grain.com`	Runs inside the cluster: namespace `ci-di`, Deployment `teamcity-server-deployment`, data directory on a PVC.
Build configs	TeamCity project `FrontEnd`	same project on both servers	Configs e.g. `FrontEnd_Agroplatform`, `FrontEnd_Rwanda`. Stored on the server PVC, not in git, no Kotlin DSL — edit only through the UI or REST API.
Build agents	`agent-admin` (baseline node)	`agent-admin` + `mapi-agent-1`	Agents run either on the baseline EKS node (host install, checkout `/home/ec2-user/BuildAgent/work`) or inside a container (e.g. `mapi-agent-1`, checkout `/opt/buildagent/work`). Every agent must have `yq` available — see Agent requirement below. A build lands on any free agent, so a missing tool fails intermittently.
App source	git repo per app	`docker-compose.yml` under `app.compose.dir`	Compose dir varies: `./src/deployment`, `./deployment`, `./Core/deployment`, or `.`
Image registry	AWS ECR — account `678991479208`, region `us-east-1`	same	Repos e.g. `agro-platform/core`, `agro-platform/spa`, `6grain/code-quality`.
Image tags	`stg_<build>`, moving `stg_latest`	`prod_<build>`, moving `prod_latest`	Lifecycle policies keep the last N images per repo; the moving `_latest` tags must survive pruning.
Cluster	`staging` — EKS v1.36	`production` — EKS v1.36	Namespace `front-end`. Deployments named `<name>-deployment`.
Ingress	`<app>.stg.ap.6grain.com`	`<app>.agroplatform.6grain.com`	Traefik v3. This documentation is served at `code-quality.stg.ap.6grain.com`.

The compose pipeline

Multi-container apps run this five-step shell pipeline. The order is the execution order. (Application-specific Tests and Restore database steps are not part of the unified template — they were dropped when the pipelines were unified.)

Get containers list — reads docker-compose.yml with yq (a real YAML parser, so any indentation, anchors, flow style, quoting, and profiles are handled correctly — not a regex/indent guess) and pairs each container_name with its image, then narrows the set in three ordered stages — containers_filter (the run-dialog checkboxes) → app.select (static include, migrator mode) → app.container.exclude (infra) — prints the resolved container → image pairs so the build is auditable, and writes the survivors into env.containers/env.images. Requires yq on the agent (see Agent requirement). (The former separate "List containers" step is folded in here.)
Build containers — builds each resolved container. Working directory is the compose directory; runs under set -e, so any build failure stops the pipeline immediately. Kept a step of its own so a parse failure and a compile failure never blur together. docker-compose &percnt;app.compose.buildfiles&percnt; build &percnt;app.build.nocache&percnt; "$container"
Publish to ECR — one step that logs in (aws ecr get-login-password | docker login), ensures a repository exists for every image, pushes each image plus its moving <prefix>_latest tag, and then drops the built images off the agent to free the EC2 disk. It fails loudly if an expected image is not in the local store — the guard that stops a non-existent tag from being "deployed". (Merges the former Refresh-token / Create-repos / Push / Remove-local steps.)
Deploy to Kubernetes — points kubeconfig at the target cluster, then delivers each container according to deploy.kind:
- app (default) — rolls the app by patching its Deployment's image, then gates on readiness: the build stays red until the new pod is actually Ready (observedGeneration caught up and readyReplicas matches the spec).
  aws eks --region us-east-1 update-kubeconfig --name %env.cluster% kubectl set image deployment/$c%app.deploy.suffix%-deployment \ $c%app.deploy.suffix%=${images[$i]} -n front-end
- job (migrators) — runs the image as a one-shot Job rather than a standing Deployment, streams its log, gates on the migration finishing, and removes the Job. See Migrators run as Jobs below.
Notify deployed (Slack) — posts a service message through the Slack connection PROJECT_EXT_5 to #qa and #k8s, listing the Jira keys found in the build's commits.

Agent requirements

A build lands on whichever agent is free, so every agent must carry the same toolchain — a tool missing on one agent makes builds fail intermittently, only when they happen to land there. The compose pipeline needs, on every agent:

Tool	Used by	Note
`yq` (mikefarah v4)	Get containers list	Reads the compose YAML. Missing → exit 127, or exit 1 from the step's own "yq not found" guard.
`docker-compose` (v2)	Build containers	Builds each service image.
`docker`	Publish to ECR	Login, tag, push, prune local.
`aws` CLI	Publish / Deploy	ECR login + repo create; EKS `update-kubeconfig`.
`kubectl`	Deploy to Kubernetes	`set image`, readiness/job gate.

Agents come in two shapes, and the tools go in different places:

Host agent (runs on a node, e.g. agent-admin on the baseline EKS node) — install into the node bootstrap so tools survive a rebuild, e.g. for yq:
sudo curl -fsSL https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 \ -o /usr/bin/yq && sudo chmod +x /usr/bin/yq
Container agent (runs in a Docker image, e.g. mapi-agent-1, checkout /opt/buildagent/work) — the tools belong in the agent image, not on the host (the host filesystem is invisible inside the container). The maintained agent image lives in the kubernetes repo at scripts/build-agent/Dockerfile; keep it in sync when the pipeline gains a new tool dependency. It builds on jetbrains/teamcity-agent and adds exactly the pipeline toolchain:
FROM jetbrains/teamcity-agent:latest USER root RUN apt-get update && apt-get install -y --no-install-recommends curl unzip ca-certificates \ && rm -rf /var/lib/apt/lists/* \ && curl -fsSL -o /usr/local/bin/kubectl https://dl.k8s.io/release/v1.36.2/bin/linux/amd64/kubectl \ && chmod +x /usr/local/bin/kubectl \ && curl -fsSL -o /usr/local/bin/yq https://github.com/mikefarah/yq/releases/download/v4.53.3/yq_linux_amd64 \ && chmod +x /usr/local/bin/yq \ && curl -fsSL https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip -o /tmp/awscliv2.zip \ && cd /tmp && unzip -q awscliv2.zip && ./aws/install && rm -rf /tmp/awscliv2.zip /tmp/aws # `docker-compose` (hyphen) as a shim over the bundled compose v2 plugin — some configs still call it. RUN printf '%s\n' '#!/bin/sh' 'exec docker compose "$@"' > /usr/local/bin/docker-compose \ && chmod +x /usr/local/bin/docker-compose

Install binaries to /usr/bin or /usr/local/bin (a systemd agent service often has a minimal PATH). The Get step additionally probes /usr/local/bin, /usr/bin, /opt/bin, and /snap/bin directly, so a yq in any of those is found regardless of the agent's PATH.

The knobs: per-app parameters

Apps do not differ in pipeline logic — they differ in these values, set per build config. Two more values are constants pinned at the project/server level.

Parameter	Purpose	Seen values
`app.compose.dir`	Directory holding `docker-compose.yml`; also the working dir for the build	`./src/deployment`, `./deployment`, `./Core/deployment`, `.`
`app.compose.buildfiles`	The `-f` arguments for `docker-compose build`	`-f docker-compose.yml -f docker-compose.staging.yml`, `-f docker-compose.yml`
`app.build.nocache`	Extra build flags	(empty), `--no-cache`
`app.select`	Which services to ship — empty = all, one container name = only that one (migrator mode)	(empty), `<container-name>`
`app.container.exclude`	Infra services never deployed	`postgres,rabbitmq,kibana,elasticsearch,es01,portainer,adminer,migrator`
`app.deploy.suffix`	Infix in the Deployment name: `deployment/<container><suffix>-deployment`	(empty), `-data-terminal`, `-weather`, `-vigne`, `-bank-of-agriculture`
`deploy.kind`	How the deploy step delivers the image: `app` = roll a Deployment, `job` = run a one-shot Job (migrators)	`app` (default), `job`
`env.COMPOSE_PROJECT_NAME`	Docker Compose project name — the standard compose variable, set unique per app (lowercase letters, digits, hyphens, underscores). Empty = compose falls back to the compose-file directory basename, which collides when different apps build on the same agent	(empty), `rwanda-application`, `demo-sso-application`, `cnh-application`

Project-level constants: env.cluster (staging/production) and env.tagprefix (stg_/prod_). A migrator is an ordinary config with app.select=<name> and deploy.kind=job — see Migrators run as Jobs.

Two build templates

No FrontEnd config carries its own copy of the pipeline — each is a TeamCity Build Configuration Template with per-config parameters filled in. There are two templates, split only by how the image is built: a multi-container compose build, or a single Dockerfile. The delivery half — push to ECR, roll onto EKS, notify Slack — is the same idea in both.

`FrontEnd_ComposePipeline` — Compose Pipeline

Multi-container applications assembled from a docker-compose.yml. Images are built by shell steps that shell out to docker-compose. Each app's compose file declares services as image: ${DOCKER_REGISTRY}/<repo>:${TAG} with a container_name; infra services are present but excluded. 13 configs on staging.

Steps — the five shell steps described above: Get containers list (parses + lists) → Build containers → Publish to ECR (login + create repos + push + drop local) → Deploy to Kubernetes → Notify (Slack).

Parameters — per config (values in The knobs above):

Parameter	Purpose
`app.compose.dir`	Directory holding `docker-compose.yml`; also the build working dir
`app.compose.buildfiles`	The `-f` arguments for `docker-compose build`
`app.build.nocache`	Extra build flags (`--no-cache` or empty)
`app.select`	Which services to ship — empty = all, one name = migrator mode
`app.container.exclude`	Infra services never deployed
`app.deploy.suffix`	Infix in the Deployment name
`deploy.kind`	`app` (roll a Deployment) or `job` (run a one-shot Job — migrators)
`env.COMPOSE_PROJECT_NAME`	Compose project name, unique per app (e.g. `rwanda-application`); empty falls back to the compose-dir basename

`FrontEnd_NativePipeline` — Native Pipeline

Single-image applications built from one Dockerfile by TeamCity's built-in DockerCommand runner — no compose, no shell parsing. A shorter, five-step shape; the deploy step carries a rollout-status gate.

Steps — five: Create ECR repository → Docker build application (DockerCommand) → Docker push (DockerCommand) → Deploy to Kubernetes (+ rollout-status gate) → Notify (Slack).

Parameters — per config:

Parameter	Purpose	Seen values
`ImageName`	ECR repository / image for this app	`agro-platform/loan-apply`, `agro-platform/loan-portfolio`, `agro-platform/dspspa`, `6grain/code-quality`
`app.dockerfile`	Path to the `Dockerfile` to build	`src/banking/onboarding/Dockerfile`, `src/banking/dashboard/Dockerfile`, `src/WebSpa/Dockerfile`, `Dockerfile`
`env.COMPOSE_PROJECT_NAME`	Declared for parity with the compose template; the native build itself does not use compose	(empty)

The Deployment and container name are not a separate parameter — they are derived as basename(ImageName) (loan-apply, loan-portfolio, dspspa, code-quality), giving deployment/<name>-deployment in front-end.

Members: LoanApplication, LoanPortfolio, DataSharingPlatform, CodeQuality (plus SixGrainSite on production).

Migrators run as Jobs

A migrator image applies its EF Core migrations on startup and then keeps a web host running — it never exits. Run as a standing Deployment it re-applies migrations on every pod restart, which crashes on non-idempotent migrations. So a migrator config sets deploy.kind=job and the deploy step delivers it as a one-shot Job instead:

the Job runs the freshly built image (restartPolicy: Never, backoffLimit: 0, ttlSecondsAfterFinished: 600) — one run, no automatic retry, no re-migration on restart;
the step streams the pod log into the build log, and the build goes green only when the migration marker appears (the app starts listening) — red on ImagePullBackOff, a crash, or timeout;
the Job and its pod are removed at the end; the migrator's ConfigMap and Secret stay.

Job name / ConfigMap / Secret default to <container>-job/-configmap/-secrets. When those names do not follow the convention (e.g. global-presentation), override job.configmap/job.secret on the config.

RBAC: the teamcity user has a narrow front-end Role (teamcity-migrator) granting jobs create/get/list/watch/delete and pods + pods/log read — scoped to front-end so build-time log access cannot reach other namespaces.

Conventions

Thing	Rule	Example
Build number	`1.0.&percnt;build.counter&percnt;`	`1.0.842`
Image tag	`<prefix><build>`	`stg_1.0.842`, moving `stg_latest`
Repository	`<product>/<service>`	`agro-platform/core`, `6grain/code-quality`
Deployment	`<container><suffix>-deployment`	`webspa-deployment`, `spa-data-terminal-deployment`
Namespace	`front-end`	all FrontEnd workloads, both clusters
Ingress host	`<app>.stg.ap.6grain.com`/`<app>.agroplatform.6grain.com`	`rwanda.stg.ap.6grain.com`
Slack notice	connection `PROJECT_EXT_5` → `#qa #k8s`	`"RW-344 is deployed"`

Operations

Deploy an app

Push to the app's git repo, then run its FrontEnd_<App> config on the target server. The pipeline builds, pushes the new <prefix><build> tag to ECR, patches the Deployments with kubectl set image, and posts the Jira keys to #qa #k8s when it is live. To ship a migrator only, set app.select to that container name; a migrator config also runs its image as a Job — see Run a migrator.

Run a migrator

A migrator config (app.select=<name>, deploy.kind=job) runs like any other build: trigger FrontEnd_<App>Migrator; it builds the image, then runs it as a one-shot Job and shows the migration log inline, going green only when the migration completes. Off-pipeline, run it straight from a bastion (ssh kube/ssh kube-prod), where kubectl is admin: scripts/teamcity/run-migrator.sh <job-manifest> -i <image> in the kubernetes repo applies the Job, tails its log, and tears it down.

When a build is red — where to look

Symptom	Where it fails	Most likely cause	First move
build error	`Build containers`	Image or app fails to compile	Read the `Build containers` step log.
"no local image"	`Publish to ECR`	The build never produced / stored the image — the push guard caught it	Go back to the Build step; the image was not in the local store.
`ImagePullBackOff`	pod, after deploy	The tag never reached ECR (or was pruned)	Check the `Publish to ECR` step and confirm the tag exists in ECR.
`CrashLoopBackOff`	pod, after deploy	The app itself crashes at startup — not infra	`kubectl logs` the pod in `front-end`.

The build and push guards make the two build-time rows unambiguous: a red build never leaves a phantom tag behind, and a missing tag surfaces at push, not at rollout.

Onboard a new app

Add a docker-compose.yml to the app repo. Services use image: ${DOCKER_REGISTRY}/<repo>:${TAG} with a container_name; any infra services go in the exclude list.
Create a build config under the FrontEnd project (via UI or REST) and set the per-app parameters — compose dir, build files, no-cache, select, exclude, deploy suffix, and a unique env.COMPOSE_PROJECT_NAME (e.g. <app>-application).
Ensure the target Deployments exist in the front-end namespace, named <container><suffix>-deployment.
Add the Traefik ingress host — <app>.stg.ap.6grain.com on staging, <app>.agroplatform.6grain.com on production.
Run the first build. It creates the ECR repositories, pushes the initial tags, and rolls out.

10 August 2026