AWSTemplateFormatVersion: '2010-09-09' Description: | The stack deployment for RDFox. This will start up an ECS Cluster with all the required permissions and networking, with RDFox running as a task and persisting access control and data stores into an EBS volume. RDFox access control is initialized with the user-supplied role name and a generated SecretManager secret. In addition, a single RDFox data store is created with the user-supplied name. To test the RDFox endpoint, SSH to the bastion host and use curl to query the SPARQL endpoint. Doing so requires the bastion host name, SPARQL endpoint URL (resolvable from within the VPC) and the ARN for the secret holding the first role password, all of which are given as outputs from the template. Note that deleting stacks created with this template will not delete the EBS volumes provisioned for the RDFox server directory. Unless such volumes are manually deleted at that point, recreating the stack with the same stack name will fail. Parameters: RDFoxFirstRoleName: Description: "The name of the first role for the RDFox server. This is used to initialize access control." Type: String Default: admin RDFoxDataStoreName: Description: "The name of the data store." Type: String Default: default KeyName: Description: "Name of an existing EC2 KeyPair to enable SSH access to the ECS instances" Type: AWS::EC2::KeyPair::KeyName ECSAMI: Description: AMI ID Type: AWS::SSM::Parameter::Value Default: "/aws/service/ecs/optimized-ami/amazon-linux/recommended/image_id" InstanceType: Description: EC2 instance type for RDFox Type: String Default: t2.large AllowedValues: - t2.micro - t2.small - t2.medium - t2.large - m4.large - m4.xlarge - m4.2xlarge - m4.4xlarge - m4.10xlarge - c4.large - c4.xlarge - c4.2xlarge - c4.4xlarge - c4.8xlarge ConstraintDescription: Please choose a valid instance type. Mappings: InstanceTypeToMemoryLimit: t2.micro: Memory: 768 t2.small: Memory: 1536 t2.medium: Memory: 3072 t2.large: Memory: 6144 m4.large: Memory: 6144 m4.xlarge: Memory: 12288 m4.2xlarge: Memory: 24576 m4.4xlarge: Memory: 49152 m4.10xlarge: Memory: 122880 c4.large: Memory: 2880 c4.xlarge: Memory: 5760 c4.2xlarge: Memory: 11520 c4.4xlarge: Memory: 23040 c4.8xlarge: Memory: 46080 Resources: #vpc resources VPC: Type: 'AWS::EC2::VPC' Properties: CidrBlock: '10.0.0.0/16' EnableDnsSupport: true EnableDnsHostnames: true InstanceTenancy: default PublicSubnet1: Type: AWS::EC2::Subnet Properties: VpcId: !Ref VPC AvailabilityZone: !Select [0, !GetAZs ''] CidrBlock: !Sub '10.0.1.0/24' MapPublicIpOnLaunch: true RDFoxPrivateSubnet: Type: AWS::EC2::Subnet Properties: VpcId: !Ref VPC AvailabilityZone: !Select [0, !GetAZs ''] CidrBlock: !Sub '10.0.3.0/24' # Creating a load balancer requires a minimum of two subnets. Even though we only want # a single instance of RDFox, we therefore still need two private subnets. This one is # named accordingly. SuperfluousPrivateSubnet: Type: AWS::EC2::Subnet Properties: VpcId: !Ref VPC AvailabilityZone: !Select [1, !GetAZs ''] CidrBlock: !Sub '10.0.4.0/24' InternetGateway: Type: 'AWS::EC2::InternetGateway' VPCGatewayAttachment: Type: 'AWS::EC2::VPCGatewayAttachment' Properties: InternetGatewayId: !Ref InternetGateway VpcId: !Ref VPC PublicRouteTable: Type: 'AWS::EC2::RouteTable' Properties: VpcId: !Ref VPC PublicSubnet1RouteTableAssociation: Type: 'AWS::EC2::SubnetRouteTableAssociation' Properties: SubnetId: !Ref PublicSubnet1 RouteTableId: !Ref PublicRouteTable PublicRoute: Type: 'AWS::EC2::Route' DependsOn: VPCGatewayAttachment Properties: GatewayId: !Ref InternetGateway RouteTableId: !Ref PublicRouteTable DestinationCidrBlock: '0.0.0.0/0' NatGatewayOneAttachment: Type: AWS::EC2::EIP DependsOn: VPCGatewayAttachment Properties: Domain: vpc NatGatewayOne: Type: AWS::EC2::NatGateway Properties: AllocationId: !GetAtt NatGatewayOneAttachment.AllocationId SubnetId: !Ref PublicSubnet1 PrivateRouteTableOne: Type: AWS::EC2::RouteTable Properties: VpcId: !Ref 'VPC' PrivateRouteOne: Type: AWS::EC2::Route Properties: RouteTableId: !Ref PrivateRouteTableOne DestinationCidrBlock: 0.0.0.0/0 NatGatewayId: !Ref NatGatewayOne PrivateRouteTableOneRDFoxSubnetAssociation: Type: AWS::EC2::SubnetRouteTableAssociation Properties: RouteTableId: !Ref PrivateRouteTableOne SubnetId: !Ref RDFoxPrivateSubnet # iam resources TaskRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Statement: - Effect: Allow Principal: Service: [ecs-tasks.amazonaws.com] Action: ['sts:AssumeRole'] Path: / Policies: - PolicyName: AWSMarketplaceMeteringRegisterUsage PolicyDocument: Statement: - Effect: Allow Action: # ECS Tasks to access hourly Metered Usage - 'aws-marketplace:RegisterUsage' Resource: '*' - PolicyName: AmazonS3FullAccess PolicyDocument: Statement: - Effect: Allow Action: # ECS Tasks to access uploading and downloading from S3 - 's3:*' Resource: '*' ECSTaskExecutionRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Statement: - Effect: Allow Principal: Service: [ecs-tasks.amazonaws.com] Action: ['sts:AssumeRole'] Path: / Policies: - PolicyName: AmazonECSTaskExecutionRolePolicy PolicyDocument: Statement: - Effect: Allow Action: # ECS Tasks to download images from ECR - 'ecr:GetAuthorizationToken' - 'ecr:BatchCheckLayerAvailability' - 'ecr:GetDownloadUrlForLayer' - 'ecr:BatchGetImage' # ECS tasks to upload logs to CloudWatch - 'logs:CreateLogStream' - 'logs:PutLogEvents' Resource: '*' - Effect: Allow Action: - secretsmanager:GetSecretValue Resource: !Ref RDFoxFirstRolePassword EC2Role: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Statement: - Effect: Allow Principal: Service: [ec2.amazonaws.com] Action: ['sts:AssumeRole'] Path: / Policies: - PolicyName: ecs-service PolicyDocument: Statement: - Effect: Allow Action: - 'ecs:CreateCluster' - 'ecs:DeregisterContainerInstance' - 'ecs:DiscoverPollEndpoint' - 'ecs:Poll' - 'ecs:RegisterContainerInstance' - 'ecs:StartTelemetrySession' - 'ecs:Submit*' - 'logs:CreateLogStream' - 'logs:PutLogEvents' - 'ecr:GetAuthorizationToken' - 'ecr:BatchGetImage' - 'ecr:GetDownloadUrlForLayer' Resource: '*' - PolicyName: RexrayPolicy PolicyDocument: Statement: - Effect: Allow Action: - 'ec2:AttachVolume' - 'ec2:CreateVolume' - 'ec2:CreateSnapshot' - 'ec2:CreateTags' - 'ec2:DeleteVolume' - 'ec2:DeleteSnapshot' - 'ec2:DescribeAvailabilityZones' - 'ec2:DescribeInstances' - 'ec2:DescribeVolumes' - 'ec2:DescribeVolumeAttribute' - 'ec2:DescribeVolumeStatus' - 'ec2:DescribeSnapshots' - 'ec2:CopySnapshot' - 'ec2:DescribeSnapshotAttribute' - 'ec2:DetachVolume' - 'ec2:ModifySnapshotAttribute' - 'ec2:ModifyVolumeAttribute' - 'ec2:DescribeTags' Resource: '*' DataStoreLambdaExecutionRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Statement: - Effect: Allow Principal: Service: [lambda.amazonaws.com] Action: ['sts:AssumeRole'] ManagedPolicyArns: - arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole Path: / Policies: - PolicyName: AmazonDataStoreLambdaExecutionRolePolicy PolicyDocument: Statement: - Effect: Allow Action: - "logs:*" Resource: !Sub "arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:*:*" - Effect: Allow Action: - secretsmanager:GetSecretValue Resource: !Ref RDFoxFirstRolePassword # cluster resources ECSCluster: Type: AWS::ECS::Cluster Properties: ClusterName: !Sub "${AWS::StackName}" CloudWatchLogsGroup: Type: AWS::Logs::LogGroup Properties: LogGroupName: !Sub "${AWS::StackName}" ContainerSecurityGroup: Type: AWS::EC2::SecurityGroup Properties: VpcId: !Ref VPC GroupDescription: for ecs containers SecurityGroupIngress: - SourceSecurityGroupId: !GetAtt RDFoxLoadBalancerSG.GroupId IpProtocol: -1 EC2InstanceProfile: Type: AWS::IAM::InstanceProfile Properties: Path: / Roles: [!Ref 'EC2Role'] ECSAutoScalingGroup: Type: AWS::AutoScaling::AutoScalingGroup Properties: VPCZoneIdentifier: - !Ref RDFoxPrivateSubnet LaunchConfigurationName: !Ref 'ContainerInstances' MinSize: '1' MaxSize: '3' DesiredCapacity: '1' Tags: - Key: Name Value: !Sub "${AWS::StackName}" PropagateAtLaunch: true CreationPolicy: ResourceSignal: Timeout: PT15M UpdatePolicy: AutoScalingReplacingUpdate: WillReplace: 'true' ContainerInstances: Type: AWS::AutoScaling::LaunchConfiguration Properties: ImageId: !Ref ECSAMI SecurityGroups: [!Ref 'ContainerSecurityGroup'] InstanceType: !Ref InstanceType IamInstanceProfile: !Ref 'EC2InstanceProfile' KeyName: !Ref KeyName UserData: Fn::Base64: Fn::Sub: "#!/bin/bash\nsleep 60\nyum install -y aws-cfn-bootstrap\n/opt/aws/bin/cfn-init -v --region ${AWS::Region} --stack ${AWS::StackName} --resource ContainerInstances\n/opt/aws/bin/cfn-signal -e $? --region ${AWS::Region} --stack ${AWS::StackName} --resource ECSAutoScalingGroup\n\nexec 2>>/var/log/ecs/ecs-agent-install.log\nset -x\nuntil curl -s http://localhost:51678/v1/metadata\ndo\n \ sleep 1\ndone\ndocker plugin install rexray/ebs REXRAY_PREEMPT=true EBS_REGION=${AWS::Region} --grant-all-permissions\nstop ecs \nstart ecs\n" Metadata: AWS::CloudFormation::Init: config: packages: yum: aws-cli: [] jq: [] ecs-init: [] commands: 01_add_instance_to_cluster: command: Fn::Sub: echo ECS_CLUSTER=${ECSCluster} >> /etc/ecs/ecs.config 02_start_ecs_agent: command: start ecs files: "/etc/cfn/cfn-hup.conf": mode: 256 owner: root group: root content: Fn::Sub: | [main] stack=${AWS::StackId} region=${AWS::Region} "/etc/cfn/hooks.d/cfn-auto-reloader.conf": content: Fn::Sub: | [cfn-auto-reloader-hook] triggers=post.update path=Resources.ContainerInstances.Metadata.AWS::CloudFormation::Init action=/opt/aws/bin/cfn-init -v --region ${AWS::Region} --stack ${AWS::StackName} --resource ContainerInstances services: sysvinit: cfn-hup: enabled: 'true' ensureRunning: 'true' files: - "/etc/cfn/cfn-hup.conf" - "/etc/cfn/hooks.d/cfn-auto-reloader.conf" TaskDefinition: Type: AWS::ECS::TaskDefinition Properties: Family: apis ExecutionRoleArn: !Ref ECSTaskExecutionRole TaskRoleArn: !Ref TaskRole Memory: !FindInMap [InstanceTypeToMemoryLimit, !Ref InstanceType, Memory] ContainerDefinitions: - Name: rdfox Image: 709825985650.dkr.ecr.us-east-1.amazonaws.com/data-lens/rdfox:7.0 DependsOn: - Condition: SUCCESS ContainerName: rdfox-init MountPoints: - SourceVolume: !Join ['-', [!Ref 'AWS::StackName', 'rdfox-server-directory']] ContainerPath: /home/rdfox/.RDFox PortMappings: - ContainerPort: 12110 LogConfiguration: LogDriver: awslogs Options: awslogs-group: !Sub "${AWS::StackName}" awslogs-region: !Ref AWS::Region awslogs-stream-prefix: 'rdfox' - Name: rdfox-init Command: ["-subscription-product", "0b284baa-81fb-4a1e-827b-5e1021f45c4f", "-persistence", "file", "-request-logger", "elf"] Image: 709825985650.dkr.ecr.us-east-1.amazonaws.com/data-lens/rdfox-init:7.0 Essential: false MountPoints: - SourceVolume: !Join ['-', [!Ref 'AWS::StackName', 'rdfox-server-directory']] ContainerPath: /home/rdfox/.RDFox Environment: - Name: RDFOX_ROLE Value: !Ref RDFoxFirstRoleName Secrets: - Name: RDFOX_PASSWORD ValueFrom: !Ref RDFoxFirstRolePassword LogConfiguration: LogDriver: awslogs Options: awslogs-group: !Sub "${AWS::StackName}" awslogs-region: !Ref AWS::Region awslogs-stream-prefix: 'rdfox-init' Volumes: - Name: !Join ['-', [!Ref 'AWS::StackName', 'rdfox-server-directory']] DockerVolumeConfiguration: Autoprovision: true Scope: shared Driver: rexray/ebs DriverOpts: volumetype: gp2 size: 5 RDFoxClientSG: Type: AWS::EC2::SecurityGroup Properties: GroupDescription: Security group for instances that should be able to call the load balancer for the RDFox service VpcId: !Ref VPC RDFoxLoadBalancerSG: Type: AWS::EC2::SecurityGroup Properties: GroupDescription: Security group for the RDFox service load balancer VpcId: !Ref VPC SecurityGroupIngress: - IpProtocol: tcp SourceSecurityGroupId: !Ref RDFoxClientSG FromPort: 80 ToPort: 80 RDFoxLoadBalancer: Type: AWS::ElasticLoadBalancingV2::LoadBalancer Properties: Name: !Sub "${AWS::StackName}" Subnets: - !Ref RDFoxPrivateSubnet - !Ref SuperfluousPrivateSubnet Scheme: internal SecurityGroups: - !Ref RDFoxLoadBalancerSG LoadBalancerListener: Type: AWS::ElasticLoadBalancingV2::Listener Properties: LoadBalancerArn: !Ref RDFoxLoadBalancer Protocol: HTTP Port: 80 DefaultActions: - Type: fixed-response FixedResponseConfig: ContentType: "text/plain" MessageBody: "Not found" StatusCode: 404 TargetGroup: Type: AWS::ElasticLoadBalancingV2::TargetGroup Properties: Name: !Sub "${AWS::StackName}" VpcId: !Ref VPC Port: 80 Protocol: HTTP HealthCheckPath: /health HealthCheckProtocol: HTTP Matcher: HttpCode: "204" ListenerRule: Type: AWS::ElasticLoadBalancingV2::ListenerRule Properties: ListenerArn: !Ref LoadBalancerListener Priority: 1 Conditions: - Field: source-ip SourceIpConfig: Values: - !GetAtt VPC.CidrBlock Actions: - TargetGroupArn: !Ref TargetGroup Type: forward Service: Type: AWS::ECS::Service DependsOn: - ListenerRule - ECSAutoScalingGroup Properties: ServiceName: !Sub "${AWS::StackName}" TaskDefinition: !Ref TaskDefinition Cluster: !Ref ECSCluster DesiredCount: 1 DeploymentConfiguration: MaximumPercent: 200 MinimumHealthyPercent: 100 LoadBalancers: - ContainerName: rdfox ContainerPort: 12110 TargetGroupArn: !Ref TargetGroup # Bastion resources BastionInstance: Type: AWS::EC2::Instance Properties: ImageId: !Ref ECSAMI InstanceType: t2.micro KeyName: !Ref KeyName SubnetId: !Ref PublicSubnet1 SecurityGroupIds: - !Ref BastionSecurityGroup - !Ref RDFoxClientSG Tags: - Key: Name Value: !Sub '${AWS::StackName}-BastionInstance' BastionSecurityGroup: Type: AWS::EC2::SecurityGroup Properties: GroupDescription: !Sub 'Security groups for ${AWS::StackName} bastion host' VpcId: !Ref VPC BastionAllowInboundSSHFromInternet: Type: AWS::EC2::SecurityGroupIngress Properties: GroupId: !Ref BastionSecurityGroup IpProtocol: tcp FromPort: 22 ToPort: 22 CidrIp: 0.0.0.0/0 BastionAllowOutboundSSHToApplication: Type: AWS::EC2::SecurityGroupEgress Properties: GroupId: !Ref BastionSecurityGroup IpProtocol: tcp FromPort: 22 ToPort: 22 DestinationSecurityGroupId: !Ref ContainerSecurityGroup RDFoxFirstRolePassword: Type: 'AWS::SecretsManager::Secret' Properties: Name: !Join ['-', [ !Ref AWS::StackName, RDFoxFirstRolePassword]] Description: "Password for the first server role." GenerateSecretString: PasswordLength: 40 ExcludeCharacters: "'" DataStoreLambda: Type: AWS::Lambda::Function Properties: Code: ZipFile: | import boto3 import botocore import json import os import sys import urllib3 NUM_RETRIES=3 TIMEOUT=3.0 http = urllib3.PoolManager() def get_password(password_secret_ARN): session = boto3.session.Session() config = botocore.config.Config(connect_timeout=TIMEOUT, read_timeout=TIMEOUT, retries={'max_attempts': NUM_RETRIES}) client = session.client(service_name='secretsmanager', region_name=session.region_name, config=config) try: get_secret_value_response = client.get_secret_value(SecretId=password_secret_ARN) except botocore.exceptions.ClientError as e: raise Exception(f'Failed to get password secret due to: {e}.') else: if not 'SecretString' in get_secret_value_response: raise Exception("Failed to get password because the response from the secretsmanager service did not contain a 'SecretString' property.") return get_secret_value_response['SecretString'] def send_response(event, context, response_status, reason, physicalResourceId): url = event['ResponseURL'] body = {} body['Status'] = response_status body['Reason'] = reason body['PhysicalResourceId'] = physicalResourceId body['StackId'] = event['StackId'] body['RequestId'] = event['RequestId'] body['LogicalResourceId'] = event['LogicalResourceId'] body['Data'] = {} json_responseBody = json.dumps(body) print(f'Sending response: {json_responseBody}') try: response = http.request("PUT", url, body=json_responseBody, headers={'Content-Type': 'application/json'}, timeout=TIMEOUT, retries=NUM_RETRIES) if response.status < 200 or response.status > 299: printf('Sending the response failed due to: {response.data.decode("utf-8")}.') except Exception as e: print(f'Failed to send response to the specified response URL due to: {str(e)}') def lambda_handler(event, context): response_status = "SUCCESS" reason = "" physical_resource_id = "" try: print(f'Received event:\n{json.dumps(event)}') role_name = event["ResourceProperties"]["RoleName"] data_store_name = event["ResourceProperties"]["DataStoreName"] physical_resource_id = event["ResourceProperties"]["RDFoxOrigin"] + "/datastores/" + data_store_name print("Fetching password...") password = get_password(event["ResourceProperties"]["PasswordSecretARN"]) print("Successfully fetched password.") headers = urllib3.make_headers(basic_auth=f'{role_name}:{password}') method = None url = physical_resource_id successful_change_code = successful_change_phrase = successful_no_change_code = successful_no_change_phrase = None if event['RequestType'] == 'Create': method = "POST" successful_change_code = 201 successful_change_phrase = "created" successful_no_change_code = 409 successful_no_change_phrase = "already exists" elif event['RequestType'] == 'Delete': method = "DELETE" successful_change_code = 204 successful_change_phrase = "deleted" successful_no_change_code = 404 successful_no_change_phrase = "already did not exist" else: response_status = "FAILED" reason = "This function only supports 'Create' and 'Delete' requests." return True i = NUM_RETRIES transient_failure = True while transient_failure and i > 0: r = e = None try: r = http.request(method, url, headers=headers, timeout=TIMEOUT, retries=False) except urllib3.exceptions.ReadTimeoutError as exc: e = exc transient_failure = e != None or r.status == 503 i -= 1 if transient_failure: print(f'RDFox request failed. {i} retries remaining...') if r.status == successful_change_code: reason = f'Data store "{data_store_name}" {successful_change_phrase}.' elif r.status == successful_no_change_code: reason = f'Data store "{data_store_name}" {successful_no_change_phrase}.' else: response_status = "FAILED" reason = r.data.decode("utf-8") except Exception as e: reason = f'Request failed due to: {e}' response_status = "FAILED" finally: send_response(event, context, response_status, reason, physical_resource_id) return True Handler: index.lambda_handler Role: !GetAtt DataStoreLambdaExecutionRole.Arn Runtime: python3.8 Timeout: 40 VpcConfig: SecurityGroupIds: - !Ref RDFoxClientSG SubnetIds: - !Ref RDFoxPrivateSubnet DependsOn: # The following dependencies are important to ensure that the custom resource can # be deleted cleanly. If CloudFormation initiates the delete of any resource needed # for a successful invocation of the lambda before this occurs, deleting the stack # will be unsuccessful and take an hour or more. - Service - PrivateRouteTableOneRDFoxSubnetAssociation - PrivateRouteOne - PublicRoute - PublicSubnet1RouteTableAssociation DataStore: Type: Custom::RDFoxDataStore Properties: ServiceToken: !GetAtt DataStoreLambda.Arn RDFoxOrigin: !Join ['', ['http://', !GetAtt RDFoxLoadBalancer.DNSName]] RoleName: !Ref RDFoxFirstRoleName PasswordSecretARN: !Ref RDFoxFirstRolePassword DataStoreName: !Ref RDFoxDataStoreName Outputs: RDFoxLoadBalancerDNSName: Description: The DNS name for the RDFox load balance resolvable from hosts within the VPC. Value: !GetAtt RDFoxLoadBalancer.DNSName SSHTunnel: Description: SSH command to establish secure tunnel to RDFox. Assumes chosen SSH key exists in .pem format in the working directory and that no other process is listening on port 8080. Value: !Join ['', ['ssh -N -L 8080:', !GetAtt RDFoxLoadBalancer.DNSName, ':80 ec2-user@', !GetAtt BastionInstance.PublicDnsName, ' -i ', !Ref KeyName, '.pem']] RDFoxConsoleURL: Description: URL for the RDFox instance's web console. Requires an SSH tunnel to be opened (see SSHTunnel output). Value: http://localhost:8080/console/ RDFoxConsoleRESTAPI: Description: Base URL for the RDFox instance's REST API. Requires an SSH tunnel to be opened (see SSHTunnel output). Value: http://localhost:8080/ RDFoxFirstRoleName: Description: The role name used to initialize RDFox access. Use this in combination with RDFoxFirstRolePassword to authenticate in the console or against the REST API. Value: !Sub "${RDFoxFirstRoleName}" RDFoxFirstRolePassword: Description: Link to the AWS Secrets Manager secret holding the password used to initialize RDFox access control. Use this in combination with RDFoxFirstRoleName to authenticate in the console or against the REST API. Value: !Join ['', ['https://', !Ref "AWS::Region", '.console.aws.amazon.com/secretsmanager/home?region=', !Ref "AWS::Region", '#!/secret?name=', !Ref "AWS::StackName", '-RDFoxFirstRolePassword']]