Repository navigation
Conversation
Every API service's execution role, and the reuploader's role, could read any Secrets Manager secret and any SSM parameter in the account (GetSecretValue on *, GetParameter* on arn:aws:ssm:*), although ECS only uses it to inject the task's task_secrets. The reuploader's is also its task role, so its own code had that access too. The policies are now built in HCL instead of templates: the secret statements grant secretsmanager:GetSecretValue and ssm:GetParameters, the two actions ECS uses to inject secrets, on the ARNs in task_secrets only (every task_secrets value in dev and prod is an SSM parameter or Secrets Manager secret ARN). A task without secrets, like the reuploader, gets no secret statement; its code only uses S3. The other statements are unchanged. Planned with fake inputs as prod's ooniauth (SSM and Secrets Manager secrets), ooniprobe, reverseproxy and the reuploader, from main and from this change: the same resources, task definitions unchanged, and the policies differ only in those statements. terraform validate passes for dev and prod. oonith_service and ooni_dataapi have the same policy but are not used by either environment.
ooniprobe uploads failed reports and reads its private config and the anonymous credentials manifests from S3. Its tasks have no task role, so the AWS SDK falls back to the container host's instance credentials, where those permissions are granted. Every container on the host can get those credentials, and they can't be taken away from the containers while ooniprobe depends on them. ooniapi_service gains an optional task_role_policy: given, the task gets a task role with it (ECS_ENABLE_TASK_IAM_ROLE is already on), whose credentials the SDK prefers over the host's. ooniprobe and ooniprobe_legacy get the S3 policy, now declared once as local.ooniprobe_s3_policy. The instance role keeps the same policy for now, so tasks still running the previous revision keep working during the rollout; the next commit removes it. Planned with fake inputs: only ooniprobe gains a role, its policy and a task_role_arn; the other services are unchanged. The instance role's policy is the same statements as before. terraform validate passes for dev and prod. The other services' code makes no AWS calls of its own, apart from ooniauth's SES calls with its own IAM user's keys and the reuploader, which already has a task role.
The tasks run in bridge mode, so any container could fetch the container host's instance credentials from the metadata service. Those could read every secret in Secrets Manager and ooniprobe's S3 objects, whatever the container's own task was given. - The launch template requires IMDSv2 with a hop limit of 1. The ECS agent (host network) and docker's awslogs driver still reach it; the containers, a hop further on the bridge network, can't. Tasks get AWS credentials from their task role instead, through the ECS agent. - The instance role no longer reads secrets: ECS injects each task's secrets with that task's execution role, and nothing on the host (ecs-setup.sh) reads any. - ooniprobe's S3 policy is removed from the instance role; ooniprobe and ooniprobe_legacy have it as their task role since the previous commit. Planned with fake inputs from main and from this branch: the same cluster resources; only the instance role policy (the two secret statements) and the launch template's metadata_options change. terraform validate passes for dev and prod. Rollout, per environment: 1. Apply the previous commit, and let ooniprobe and ooniprobe_legacy deploy the task definition with the task role (check that failed report uploads and the manifest and config reads keep working). 2. Apply this commit. The autoscaling group launches new hosts from the new template ($Latest) but doesn't refresh running ones on a template change; start an instance refresh, or run aws ec2 modify-instance-metadata-options --http-tokens required --http-put-response-hop-limit 1 on each running host.
Terraform Run Output 🤖Format and Style 🖌
|
| Pusher | @aagbsn |
| Action | pull_request |
| Environment | dev |
| Workflow | .github/workflows/check_terraform.yml |
| Last updated | Sun, 04 Oct 2026 06:48:07 GMT |
…oken The build roles of ooniapi_service_deployer, ooni_docker_build and oonith_service_deployer had the AWS managed SecretsManagerReadWrite policy. Every buildspec they run (the API services, fastpath, testlists and the reuploader, on master and on the branches dev builds) reads one secret, oonidevops/dockerhub/access_token, to log in to Docker Hub. The roles now get secretsmanager:GetSecretValue on that secret only (its ARN plus the 6 character suffix Secrets Manager adds) instead of the managed policy. The make db-migration targets that read the Postgres URL run on an operator's machine, not in CodeBuild. Planned ooniapi_service_deployer from main and from this change with fake inputs: the same resources, and the build policy differs by that one statement. terraform validate passes for dev and prod.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scope ECS credentials to what each task uses
Every API service's execution role, the reuploader's role and the ECS hosts' instance role could read every
Secrets Manager secret and SSM parameter in the account. The tasks run in bridge mode, so any container could also
take the host's instance credentials from the metadata service, including ooniprobe's S3 permissions, which were
granted there. This PR limits each role to what its task uses.
Changes (one commit each)
Execution roles read only their own task's secrets (
ooniapi_service,scheduled_service)The secret statements now grant
secretsmanager:GetSecretValue/ssm:GetParameters, the two actions ECS usesto inject secrets, on the ARNs in that task's
task_secretsonly. Everytask_secretsvalue in dev and prod is anARN. A task without secrets gets no secret statement. The reuploader, whose role is also its task role, no longer
reads any secret; its code only uses S3. Other statements are unchanged. The policies are built in HCL instead of
JSON templates, so a statement can be omitted when its ARN list is empty, which IAM requires.
ooniprobe gets a task role for its S3 access
ooniapi_servicegains an optionaltask_role_policy. ooniprobe and ooniprobe_legacy get the existing S3 policy(failed reports upload, private config and anonymous credentials manifest reads), now declared once as
local.ooniprobe_s3_policy. The instance role keeps the same policy in this commit, so tasks on the previousrevision keep working during the rollout.
Containers can't reach the hosts' instance credentials
The ECS launch template requires IMDSv2 with a hop limit of 1: the ECS agent (host network) and the awslogs
driver still reach it, containers on the bridge network can't. The instance role loses its secret access (ECS
injects secrets with each task's execution role; nothing on the host reads any) and ooniprobe's S3 policy.
Rollout (per environment, in order)
failed report uploads and the config and manifest reads still work.
$Latestbut doesn't refresh running ones on a template change:start an instance refresh, or run
aws ec2 modify-instance-metadata-options --http-tokens required --http-put-response-hop-limit 1on each running host.Verification
ooniapi_service(set up like prod's ooniauth, ooniprobe and reverseproxy) andscheduled_service(likethe reuploader) with fake inputs, from
mainand from this branch. Same resources; task definitions unchangedexcept ooniprobe's new
task_role_arn; policies differ only in the secret statements, each listing exactly thattask's secrets.
ecs_clusterthe same way: only the instance policy (two secret statements removed) and the launchtemplate's
metadata_optionschange.terraform validateandfmtpass for dev and prod.SES with its own IAM user's keys, and the reuploader already has a task role (S3 list/get/delete, granted by
reuploader_role, unchanged). No other service calls AWS.Not changed
oonith_serviceandooni_dataapihave the same read-everything policy but aren't used by any environment.logs:*statements they likely don't need.oonidevops_githubCI user can still read all secrets and parameters (it plans with them). It should get adev-only role behind a required-approval environment, or OIDC.
ooni-pipelinekeys from the Ansible vault; they'reunaffected here.