CI / CD

The delivery path

The same five stages run for every app on both environments. What differs between apps is values, not the shape of the path — those values are the six parameters described below.

Where things live

Staging and production are fully separate stacks — separate TeamCity servers, separate clusters, separate ECR tag prefixes. When something is missing, this table says which layer to open.

The compose pipeline

Multi-container apps run this shell-script pipeline. Steps 1–2 are optional and usually off; the rest always run. The order is the execution order.

1. Tests (optional) — runs the app's test suite; commonly disabled.

2. Restore database from backup (optional) — seeds a fresh database before build/deploy.

3. Get containers list — parses docker-compose.yml and pairs each container_name with its image within the same service block, drops the infra services from the exclude list, and writes the survivors into env.containers/env.images.

4. List containers — prints the resolved container → image pairs so the build is auditable before anything is built.

5. Build containers — builds each resolved container. Working directory is the compose directory; runs under set -e, so any build failure stops the pipeline immediately. docker-compose %app.compose.buildfiles% build %app.build.nocache% "$container"

6. Refresh ECR token — aws ecr get-login-password | docker login ….

7. Create ECR repositories if not exist — ensures a repository exists for every image so a brand-new service does not fail the push.

8. Push containers to ECR — pushes each image, and fails loudly if an expected image is not in the local store. This guard is what stops a non-existent tag from being "deployed".

9. Remove image from local registry — cleans built images off the agent so the EC2 disk does not fill up across builds.

10. Deploy to Kubernetes — points kubeconfig at the target cluster, then delivers each container according to deploy.kind:

    • app (default) — rolls the app by patching its Deployment's image, then gates on readiness: the build stays red until the new pod is actually Ready (observedGeneration caught up and readyReplicas matches the spec).

      aws eks --region us-east-1 update-kubeconfig --name %env.cluster% kubectl set image deployment/$c%app.deploy.suffix%-deployment \ $c%app.deploy.suffix%=${images[$i]} -n front-end

    • job (migrators) — runs the image as a one-shot Job rather than a standing Deployment, streams its log, gates on the migration finishing, and removes the Job. See Migrators run as Jobs below.

11. Notify deployed (Slack) — posts a service message through the Slack connection PROJECT_EXT_5 to #qa and #k8s, listing the Jira keys found in the build's commits.

The knobs: six parameters

Apps do not differ in pipeline logic — they differ in these six values, set per build config. Two more values are constants pinned at the project/server level.

Project-level constants: env.cluster (staging/production) and env.tagprefix (stg_/prod_). A migrator is an ordinary config with app.select=<name> and deploy.kind=job — see Migrators run as Jobs.

Two build templates

No FrontEnd config carries its own copy of the pipeline — each is a TeamCity Build Configuration Template with per-config parameters filled in. There are two templates, split only by how the image is built: a multi-container compose build, or a single Dockerfile. The delivery half — push to ECR, roll onto EKS, notify Slack — is the same idea in both.

Migrators run as Jobs

A migrator image applies its EF Core migrations on startup and then keeps a web host running — it never exits. Run as a standing Deployment it re-applies migrations on every pod restart, which crashes on non-idempotent migrations. So a migrator config sets deploy.kind=job and the deploy step delivers it as a one-shot Job instead:

• the Job runs the freshly built image (restartPolicy: Never, backoffLimit: 0, ttlSecondsAfterFinished: 600) — one run, no automatic retry, no re-migration on restart;

• the step streams the pod log into the build log, and the build goes green only when the migration marker appears (the app starts listening) — red on ImagePullBackOff, a crash, or timeout;

• the Job and its pod are removed at the end; the migrator's ConfigMap and Secret stay.

Job name / ConfigMap / Secret default to <container>-job/-configmap/-secrets. When those names do not follow the convention (e.g. global-presentation), override job.configmap/job.secret on the config.

RBAC: the teamcity user has a narrow front-end Role (teamcity-migrator) granting jobs create/get/list/watch/delete and pods + pods/log read — scoped to front-end so build-time log access cannot reach other namespaces.

Conventions

Operations