DolphinScheduler/deploy/terraform/aws

Prerequisites

• Packer
• Terraform

Build AMI

Set necessary variables by creating a file ds-ami.pkrvars.hcl and adding the following variables according to your own usage.

cat <<EOF > ds-ami.pkrvars.hcl
aws_access_key = ""
aws_secret_key = ""
aws_region     = "cn-north-1"


ds_ami_name = "my-test-ds-2"

# If you want to use the official distribution tar, just set the `ds_version` to the one you want.
ds_version  = 3.1.1

# If you want to use a locally built distribution tar, set the `ds_tar` to the tar file location.
ds_tar      = "~/workspace/dolphinscheduler/dolphinscheduler-dist/target/apache-dolphinscheduler-3.1.3-SNAPSHOT-bin.tar.gz"
EOF

Then run the following command to initialize and build a custom AMI.

• If you want to use the official distribution tar.

packer init --var-file=ds-ami.pkrvars.hcl packer/ds-ami-official.pkr.hcl
packer build --var-file=ds-ami.pkrvars.hcl packer/ds-ami-official.pkr.hcl

• If you want to use the locally built distribution tar.

packer init --var-file=ds-ami.pkrvars.hcl packer/ds-ami-local.pkr.hcl
packer build --var-file=ds-ami.pkrvars.hcl packer/ds-ami-local.pkr.hcl

Create resources

Set necessary variables by creating a file terraform.tfvars and adding the following variables according to your own usage.

Make sure ds_ami_name is the same as the one in ds-ami.pkrvars.hcl above.

cat <<EOF > terraform.tfvars
aws_access_key = ""
aws_secret_key = ""

name_prefix = "test-ds-terraform"
ds_ami_name = "my-test-ds"

ds_component_replicas = {
  master            = 1
  worker            = 1
  alert             = 1
  api               = 1
  standalone_server = 0
}
EOF

Then run the following commands to apply necessary resources.

terraform init -var-file=terraform.tfvars
terraform apply -var-file=terraform.tfvars -auto-approve

Open DolphinScheduler UI

open http://$(terraform output -json api_server_instance_public_dns | jq -r '.[0]'):12345/dolphinscheduler/ui

Inputs

Name	Description	Type	Default	Required
aws_access_key	AWS access key	string	n/a	yes
aws_region	AWS region	string	"cn-north-1"	no
aws_secret_key	AWS secret key	string	n/a	yes
db_instance_class	Database instance class	string	"db.t3.micro"	no
db_password	Database password	string	n/a	yes
db_username	Database username	string	"dolphinscheduler"	no
ds_ami_name	Name of DolphinScheduler AMI	string	"dolphinscheduler-ami"	no
ds_component_replicas	Replicas of the DolphinScheduler Components	map(number)	{ "alert": 1, "api": 1, "master": 1, "standalone_server": 0, "worker": 1 }	no
ds_version	DolphinScheduler Version	string	"3.1.1"	no
name_prefix	Name prefix for all resources	string	"dolphinscheduler"	no
private_subnet_cidr_blocks	Available CIDR blocks for private subnets	list(string)	[ "10.0.101.0/24", "10.0.102.0/24", "10.0.103.0/24", "10.0.104.0/24" ]	no
public_subnet_cidr_blocks	CIDR blocks for the public subnets	list(string)	[ "10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24", "10.0.4.0/24" ]	no
s3_bucket_prefix	n/a	string	"dolphinscheduler-test-"	no
subnet_count	Number of subnets	map(number)	{ "private": 2, "public": 1 }	no
tags	Tags to apply to all resources	map(string)	{ "Deployment": "Test" }	no
vm_associate_public_ip_address	Associate a public IP address to the EC2 instance	map(bool)	{ "alert": true, "api": true, "master": true, "standalone_server": true, "worker": true }	no
vm_data_volume_size	Data volume size of the EC2 Instance	map(number)	{ "alert": 10, "api": 10, "master": 10, "standalone_server": 10, "worker": 10 }	no
vm_data_volume_type	Data volume type of the EC2 Instance	map(string)	{ "alert": "gp2", "api": "gp2", "master": "gp2", "standalone_server": "gp2", "worker": "gp2" }	no
vm_instance_type	EC2 instance type	map(string)	{ "alert": "t2.micro", "api": "t2.small", "master": "t2.medium", "standalone_server": "t2.small", "worker": "t2.medium" }	no
vm_root_volume_size	Root Volume size of the EC2 Instance	map(number)	{ "alert": 30, "api": 30, "master": 30, "standalone_server": 30, "worker": 30 }	no
vm_root_volume_type	Root volume type of the EC2 Instance	map(string)	{ "alert": "gp2", "api": "gp2", "master": "gp2", "standalone_server": "gp2", "worker": "gp2" }	no
vpc_cidr	CIDR for the VPC	string	"10.0.0.0/16"	no
zookeeper_connect_string	Zookeeper connect string, if empty, will create a single-node zookeeper for demonstration, don't use this in production	string	""	no

Outputs

Name	Description
alert_server_instance_id	Instance IDs of alert instances
alert_server_instance_private_ip	Private IPs of alert instances
alert_server_instance_public_dns	Public domain names of alert instances
alert_server_instance_public_ip	Public IPs of alert instances
api_server_instance_id	Instance IDs of api instances
api_server_instance_private_ip	Private IPs of api instances
api_server_instance_public_dns	Public domain names of api instances
api_server_instance_public_ip	Public IPs of api instances
db_address	Database address
db_name	Database name
db_port	Database port
master_server_instance_id	Instance IDs of master instances
master_server_instance_private_ip	Private IPs of master instances
master_server_instance_public_dns	Public domain names of master instances
master_server_instance_public_ip	Public IPs of master instances
s3_access_key	S3 access key
s3_address	S3 address
s3_bucket	S3 bucket name
s3_regional_domain_name	S3 regional domain name
s3_secret	S3 access secret
vm_server_instance_id	Instance IDs of standalone instances
vm_server_instance_private_ip	Private IPs of standalone instances
vm_server_instance_public_dns	Public domain names of standalone instances
vm_server_instance_public_ip	Public IPs of standalone instances
worker_server_instance_id	Instance IDs of worker instances
worker_server_instance_private_ip	Private IPs of worker instances
worker_server_instance_public_dns	Public domain names of worker instances
worker_server_instance_public_ip	Public IPs of worker instances
zookeeper_server_instance_id	Instance IDs of zookeeper instances
zookeeper_server_instance_private_ip	Private IPs of zookeeper instances
zookeeper_server_instance_public_dns	Public domain names of zookeeper instances
zookeeper_server_instance_public_ip	Public IPs of zookeeper instances