In this series of blogposts we’ll be taking an in-depth look at the security of AWS CodeConnections and their use in several different AWS Services. As CodeConnections become supported in more AWS services, it is important for us to understand exactly how CodeConnections work, what their limitations are and what security controls can be applied to ensure our code repositories and infrastructure stays secure.
This series of blog posts aims to answer the question, can we significantly escalate our privileges via the source code provider permissions granted to AWS if we can compromise a single AWS account or single AWS service such as CodePipeline. You can view all the posts in this series by visiting my AWS CodeConnection project page.
In the first post of the series we covered a primer on AWS CodeConnections and the Apps that are installed into the source code providers. We focused on what permissions AWS CodeConnection gets and the limitations on restricting these permissions.
In the second post we dived into CodePipeline and the UseConnection CodeConnection IAM permission. We explored the “Full Clone” option and looked at what damage can be done when the IAM role granting UseConnection permissions doesn’t have restricting conditions.
In this post we are going to look at a selection of other services that work with AWS CodeConnections.
SageMaker Unified Studio
SageMaker Unified Studio (often titled simply SageMaker) is a suite of tools from AWS that are combined to provide a single data and AI development environment. This service provides it’s own dedicated web frontend that users (e.g. data analysts and data scientists) authenticate to to use the services within SageMaker Unified Studio. The users will typically have no access to the AWS Console or AWS account and so they are restricted to only performing operations within this SageMaker Unified Studio environment.
One of the main services provided within SageMaker Unified Studio is Jupyter Notebook which the users will use to perform experimentation on data stored within the platform. As part of this experimentation they will often need to load/save their work from/to a source code provider such as GitHub. This is where SageMaker Unified Studio makes use of CodeConnections.
The AWS Sagemaker Unified Studio git-connection documentation provides details around using the CodeConnection within your SageMaker domain and supports GitLab, GitHub and BitBucket at time of writing.
Once configured we can explore how it works by experimenting within the Jupyter Notebook environment as a standard user. The image and snippet shown below shows the initial experimentation I performed.
sagemaker-user@default:~/src$ git remote -v
origin https://codeconnections.AWS_REGION.amazonaws.com/git-http/AWS_ACCOUNT/AWS_REGION/CODECONNECTION_UUID/GH_ORG/GH_REPO.git (fetch)
origin https://codeconnections.AWS_REGION.amazonaws.com/git-http/AWS_ACCOUNT/AWS_REGION/CODECONNECTION_UUID/GH_ORG/GH_REPO.git (push)
sagemaker-user@default:~/src$ git config --list
credential.helper=!aws --profile DomainExecutionRoleCreds --region eu-west-1 codecommit credential-helper --ignore-host-check
credential.usehttppath=true
user.email=arn:aws:iam::AWS_ACCOUNT:user/ThomasPreece
user.name=ThomasPreece
core.repositoryformatversion=0
core.filemode=true
core.bare=false
core.logallrefupdates=true
remote.origin.url=https://codeconnections.AWS_REGION.amazonaws.com/git-http/AWS_ACCOUNT/AWS_REGION/CODECONNECTION_UUID/GH_ORG/GH_REPO.git
remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*
branch.SMUS-1.remote=origin
branch.SMUS-1.merge=refs/heads/main
As can be seen, SageMaker Unified Studio uses the exact same mechanism as the “Full clone” option in AWS CodePipeline that we discussed in the previous blog post.
We can dive further into the AWS credentials the user has access to by some basic interaction with the AWS CLI:
sagemaker-user@default:~/src$ cat ~/.aws/config
[default]
credential_source = EcsContainer
region = eu-west-1
[profile DomainExecutionRoleCreds]
credential_process = sagemaker-studio credentials get-domain-execution-role-credential-in-space --domain-id dzd-6cpiejh60o40nq --profile default
sagemaker-user@default:~/src$ aws sts get-caller-identity
{
"UserId": "AROA4MTWKF3E3KPEFTZ2G:SageMaker",
"Account": "851725397705",
"Arn": "arn:aws:sts::851725397705:assumed-role/datazone_usr_role_asf7g1dd4j1ifq_5yojwebfab96l2/SageMaker"
}
sagemaker-user@default:~/src$ aws sts get-caller-identity --profile=DomainExecutionRoleCreds
{
"UserId": "AROA4MTWKF3E5EBU7KDIP:user-f4ed1106-c84d-4700-89e9-3d05974df5a0",
"Account": "851725397705",
"Arn": "arn:aws:sts::851725397705:assumed-role/AmazonSageMakerDomainExecution/user-f4ed1106-c84d-4700-89e9-3d05974df5a0"
}As can be seen above, the user has access to two different AWS roles – datazone_usr_role_xxxxxxxx and AmazonSageMakerDomainExecution (as detailed in the SageMaker Unified Studio documentation) with the later role being the one that is used to access git via the credential helper. Looking at the IAM role we see the following snippet which shows that the UseConnection permission is not restrained with any CodeConnection conditions other than a tag:
{
"Sid": "UseCodeConnectionsPermissionsStatement",
"Effect": "Allow",
"Action": [
"codeconnections:UseConnection",
"codestar-connections:UseConnection"
],
"Resource": "*",
"Condition": {
"Null": {
"aws:ResourceTag/for-use-with-all-datazone-projects": "false"
},
"StringEquals": {
"aws:ResourceTag/for-use-with-all-datazone-projects": "true"
}
}
}
Therefore, all the issues detailed in the previous post apply here too and an attacker with access to the Jypter Notebook environment (e.g. the user or any package/code they install) could cause:
- Information Disclosure – Exfiltrate repository code
- DOS/Ransomware – Copy then delete all repositories in organisation
- Reputation Damage – Create new public repositories with questionable content in
- Lateral movement – If repositories are not using branch protection then writing to them could allow tampering of further build pipelines and AWS environments.
The mitigations detailed in the previous post should be applied to this AmazonSageMakerDomainExecution role, i.e. ensure you add the FullRepositoryId, ProviderPermissionsRequired and ProviderAction conditions to this role.
SageMaker AI
SageMaker AI is the rebranded product originally called SageMaker. SageMaker AI is described by AWS as a comprehensive, managed service for building, training, and deploying machine learning (ML) models at scale.
AWS documentation lists SageMaker AI as being supported for use with CodeConnections. Following the CodeConnection documentation gets us to the Walk Through a SageMaker AI MLOps Project Using Third-party Git Repos page. From this page we can see that the CodeConnection support is via a MLOps template that deploys CodePipelines so again we are probably looking at the same concerns we detailed in the previous CodePipeline post.
Unfortunately, I wasn’t able to get the project template to work fully and ran out of time to debug further. From my testing I found the following:
- The IAM role accessible by user from inside the Jupyter notebook environment doesn’t have any permissions to access the CodeConnection.
- The IAM role used during deployment of the project template is “AmazonSageMakerServiceCatalogProductsCodeBuildRole” and presumably the IAM role used for pipelines would be “AmazonSageMakerServiceCatalogProductsExecutionRole”.
Looking at both of the AmazonSagemakerServiceCatalog roles we see they have the following permissions:
{
"Effect": "Allow",
"Action": [
"codestar-connections:UseConnection",
"codeconnections:UseConnection"
],
"Resource": [
"arn:aws:codestar-connections:*:*:connection/*",
"arn:aws:codeconnections:*:*:connection/*"
],
"Condition": {
"StringEqualsIgnoreCase": {
"aws:ResourceTag/sagemaker": "true"
}
}
}As we saw with the other SageMaker, the permissions again are not restricted other than a condition around a resource tag on the CodeConnection. So an attacker which gains access to the model build/train/deploy pipelines could again cause:
- Information Disclosure – Exfiltrate repository code
- DOS/Ransomware – Copy then delete all repositories in organisation
- Reputation Damage – Create new public repositories with questionable content in
- Lateral movement – If repositories are not using branch protection then writing to them could allow tampering of further build pipelines and AWS environments.
The mitigations detailed in the previous post should be applied to these AmazonSagemakerServiceCatalog roles, i.e. ensure you add the FullRepositoryId, ProviderPermissionsRequired and ProviderAction conditions.
App Runner
AWS App Runner is listed as compatible with CodeConnections in the CodeConnections documentation. However, on testing I found that when setting up an App Runner you cannot use the existing CodeConnections in your account. Instead you have to setup new “App Runner Connection” that leverages the same source code provider apps (i.e. the “AWS CodeConnections on GitHub” GitHub app and “AWS CodeStar application” BitBucket) as CodeConnections uses. To set these up you need to re-authenticate to the source code provider so you cannot use previously created CodeConnections.
The newly created “App Runner Connections” get their own ARN arn:aws:apprunner:AWS_REGION:AWS_ACCOUNT:connection/APPRUNNER_NAME/CONNECTION_ID and are then used via the AppRunner API (also see unlisted AppRunner API operations documentation). Reviewing the CloudTrail logs after starting an App Runner, we see only apprunner.amazonaws.com operations such as ListOwners, StartOAuthHandshake and GetIndividualAccessToken. We don’t see any codeconnection.amazonaws.com operations.
Therefore, as App Runner uses a completely different API, different connections to CodeConnections and requires a re-authentication to connect to the source code provider to setup, I consider it as a separate thing to CodeConnections and further experimentation into how it works will be saved for another time.