In this article we discuss on how to increase the disk space of a AWS ECS Fargate Container using AWS CLI as this feature is not yet supported on the AWS Console or UI.
This is part 2 of multipart article. Please checkout Part 1 of this article for better continuity and understanding if not done already.
Part 1 – Discusses the prerequisites and the steps to access the interactive shell of a docker container running on AWS ECS Fargate.
Part 2 – Discusses the steps to find and increase the disk space of a docker container on AWS ECS Fargate and also the common issues and troubleshooting steps.
Access the Interactive shell of a docker container and find the diskspace
Assuming you have the access to the bash shell of the container (from Part 1), run the below command to get the disk space.
[ec2-user@ip-10-x-x-xxx]$ aws ecs execute-command \
> --region us-east-2 \
> --cluster test-cluster \
> --task fac2a0afb35f4be38e983f826aef6717 \
> --container test-container \
> --command "/bin/bash" \
> --interactive
The Session Manager plugin was installed successfully. Use the AWS CLI to start a session.
Starting session with SessionId: ecs-execute-command-02938fdaae0e071ba
bash-4.2#
bash-4.2# df -h
Filesystem Size Used Avail Use% Mounted on
overlay 109G 15G 89G 15% /
tmpfs 64M 0 64M 0% /dev
shm 3.9G 0 3.9G 0% /dev/shm
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/xvdcz 109G 15G 89G 15% /etc/hosts
/dev/xvda1 4.9G 1.6G 3.2G 34% /managed-agents/execute-command
tmpfs 3.9G 0 3.9G 0% /proc/acpi
tmpfs 3.9G 0 3.9G 0% /sys/firmware
tmpfs 3.9G 0 3.9G 0% /proc/scsi
bash-4.2#
bash-4.2#
Increase Disk space of AWS ECS Fargate Container
To increase the disk space of the Fargate container, we need to add “Ephemeral Storage” parameter to the earlier run-task command and specify the storage size.
Again replace the CLUSTER_NAME, TASK_DEFINITION, PUBLIC_SUBNET_ID and SECURITY_GROUP_ID with the appropriate values.
aws ecs run-task
--cluster <CLUSTER_NAME>
--task-definition <TASK_DEFINITION>:LATEST
--network-configuration '{"awsvpcConfiguration":{"subnets":["<PUBLIC_SUBNET_ID>","<PUBLIC_SUBNET_ID>"],"securityGroups":["<SECURITY_GROUP_ID>"], "assignPublicIp":"ENABLED"}}'
--launch-type FARGATE
--overrides '{"ephemeralStorage":{"sizeInGiB":100}}'
--enable-execute-command
If everything goes well, we get the output similar to below.
{
"tasks":[
{
"attachments":[
{
"id":"9211ec7e-7953-419a-a62d-**********",
"type":"ElasticNetworkInterface",
"status":"PRECREATED",
"details":[
{
"name":"subnetId",
"value":"subnet-026**********928"
}
]
}
],
"attributes":[
{
"name":"ecs.cpu-architecture",
"value":"x86_64"
}
],
"availabilityZone":"us-east-2a",
"clusterArn":"arn:aws:ecs:us-east-2:3**********0:cluster/MyCluster-UI",
"containers":[
{
"containerArn":"arn:aws:ecs:us-east-2:3**********0:container/MyCluster-UI/fac2a0af**********3f826aef6717/b0272b34-85a3-457f-9e8e-**********",
"taskArn":"arn:aws:ecs:us-east-2:3**********0:task/MyTask-UI/fac2a0afb**********3f826aef6717",
"name":"MyECS-UI",
"image":"3**********0.dkr.ecr.us-east-2.amazonaws.com/MyECS-ui:latest",
"lastStatus":"PENDING",
"networkInterfaces":[
],
"managedAgents":[
{
"name":"ExecuteCommandAgent",
"lastStatus":"PENDING"
}
],
"cpu":"0",
"memoryReservation":"1024"
}
],
"cpu":"1024",
"createdAt":"2023-04-22T08:34:29.019000+00:00",
"desiredStatus":"RUNNING",
"enableExecuteCommand":true,
"group":"family:MyECS-Ui",
"lastStatus":"PROVISIONING",
"launchType":"FARGATE",
"memory":"2048",
"overrides":{
"containerOverrides":[
{
"name":"MyECS-UI"
}
],
"inferenceAcceleratorOverrides":[
]
},
"platformVersion":"1.4.0",
"platformFamily":"Linux",
"tags":[
],
"taskArn":"arn:aws:ecs:us-east-2:3**********0:task/MyECS-UI/fac2a0afb35f4be38e983f826aef6717",
"taskDefinitionArn":"arn:aws:ecs:us-east-2:3**********0:task-definition/MyECS-Ui:2",
"version":1,
"ephemeralStorage":{
"sizeInGiB":100
}
}
],
"failures":[
]
}Note: As per the documentation (at the time of writing this article), ECS Fargate supports max of 200GB of storage disk space for both Linux and Windows containers.
Execute-Command
Now access the shell of the running container with the earlier execute-command. Replace the CLUSTER_NAME, TASK_ID with appropriate values.
aws ecs execute-command \
--region us-east-2 \
--cluster CLUSTER_NAME \
--task TASK_ID \
--container linux_container \
--command "/bin/bash" \
--interactiveWe can now run the command to verify if the container has the new disk space.
[ec2-user@ip-10-x-x-xxx]$ aws ecs execute-command \
> --region us-east-2 \
> --cluster test-cluster \
> --task fac2a0afb35f4be38e983f826aef6717 \
> --container test-container \
> --command "/bin/bash" \
> --interactive
The Session Manager plugin was installed successfully. Use the AWS CLI to start a session.
Starting session with SessionId: ecs-execute-command-02938fdaae0e071ba
bash-4.2#
bash-4.2# df -h
Filesystem Size Used Avail Use% Mounted on
overlay 109G 15G 89G 15% /
tmpfs 64M 0 64M 0% /dev
shm 3.9G 0 3.9G 0% /dev/shm
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/xvdcz 109G 15G 89G 15% /etc/hosts
/dev/xvda1 4.9G 1.6G 3.2G 34% /managed-agents/execute-command
tmpfs 3.9G 0 3.9G 0% /proc/acpi
tmpfs 3.9G 0 3.9G 0% /sys/firmware
tmpfs 3.9G 0 3.9G 0% /proc/scsi
bash-4.2#Common Errors and Troubleshooting
ResourceInitializationError: unable to pull secrets or registry auth
ResourceInitializationError: unable to pull secrets or registry auth: execution resource retrieval failed: unable to retrieve ecr registry auth: service call has been retried 3 time(s): RequestError: send request failed caused by: Post https://api.ecr.us-east-2.amazonaws.com/: dial tcp 3.17.137.97:443: i/o timeout. Please check your task network configuration.Try with the Platform Version 1.4.0.
If you have the obligation to use Platform Version 1.3.0 then the issue could be because the secondary network interface used by Fargate platform is unable to fetch authentication credentials and fetch secrets over the internet gateway.
To resolve this try below steps,
- Launch tasks in a Public subnet
- Launch tasks in Private subnet with NAT Gateway
- Launch tasks in Private subnet with AWS Privatelink endpoints configured with the VPC
SessionManagerPlugin is not found
SessionManagerPlugin is not found. Please refer to SessionManager Documentation here: http://docs.aws.amazon.com/console/systems-manager/session-manager-plugin-not-foundInstall the Session Manager Plugin.
Please refer this article for detailed steps.
An error occurred (InvalidParameterException)
An error occurred (InvalidParameterException) when calling the ExecuteCommand operation: The execute command failed because execute command was not enabled when the task was run or the execute command agent isn’t running. Wait and try again or run a new task with execute command enabled and try again.This could possibly occur because of 2 reasons,
1) Parameters passed incorrectly. Please check the AWS ECS CLI documentation. You could also try “aws help” for possible suggestions.
2) You could be using older version of AWS CLI and the parameters used might not be supported. For example the “ephemeral storage” parameter for windows container is only supported with version 2.11.14 and above.
Use below command to check your AWS CLI Version.
aws --versionIf your AWS CLI version is lower than the latest version, then please upgrade it and try again.
An error occurred (TargetNotConnectedException)
An error occurred (TargetNotConnectedException) when calling the ExecuteCommand operation: The execute command failed due to an internal error. Try again later.Missing IAM permissions on SSMMessages and ExecuteCommand.
Please refer the Pre-requisites section of this post.
Conclusion
In this post we’ve seen the steps to find and increase the disk space of a container running on ECS Fargate. Also we’ve seen common errors and possible solutions.
For more information on AWS ECS Fargate containers, please refer this documentation.
Hope this post was helpful. Please leave a comment & share your thoughts.
