terraform-aws-arc-kinesis-stream

Overview

For more information about this repository and its usage, please see Terraform AWS ARC GitHub Kinesis Module Usage Guide.

Introduction

SourceFuse's AWS Reference Architecture (ARC) Terraform module simplifies the creation and management of essential AWS infrastructure components. It is designed to provision and configure the following resources:

1. Amazon Kinesis Data Stream (aws_kinesis_stream): - Name: A unique name for the Kinesis stream, generated using a local variable. - Shard Count: The number of shards provisioned for the stream. This can be dynamically set based on the stream mode (PROVISIONED or ON_DEMAND). - Retention Period: The length of time that data records are accessible after they are added to the stream. - Shard Level Metrics: Metrics to be enabled for monitoring the stream at the shard level. - Enforce Consumer Deletion: A flag to enforce consumer deletion when the stream is deleted. - Encryption Type: The encryption type to be used for the stream. - KMS Key ID: The KMS key ID for encrypting data records in the stream. - Stream Mode Details: Dynamic configuration block to set the stream mode (PROVISIONED or ON_DEMAND). - Tags: Key-value pairs to tag the stream for identification and management.

2. Amazon Kinesis Stream Consumer (aws_kinesis_stream_consumer):
   - Count: The number of consumer instances to create, determined by the var.consumer_count variables. - Name: A unique name for each consumer, generated using a local variable and the count index - Stream ARN: The Amazon Resource Name (ARN) of the Kinesis stream to which the consumer is attached.

Prerequisites

Before using this module, ensure you have the following:

• AWS credentials configured.
• Terraform installed.
• A working knowledge of Terraform.

Getting Started

1. Define the Module

Initially, it's essential to define a Terraform module, which is organized as a distinct directory encompassing Terraform configuration files. Within this module directory, input variables and output values must be defined in the variables.tf and outputs.tf files, respectively. The following illustrates an example directory structure:

ec2/
|-- main.tf
|-- variables.tf
|-- outputs.tf

1. Define Input Variables

Inside the variables.tf or in *.tfvars file, you should define values for the variables that the module requires.

1. Use the Module in Your Main Configuration In your main Terraform configuration file (e.g., main.tf), you can use the module. Specify the source of the module, and version, For Example

module "kinesis" {
  source         = "sourcefuse/arc-kinesis-stream/aws"
  version        = "0.0.1"
  name           = "${var.namespace}-${var.environment}-kinesis"
  shard_count    = var.shard_count
  consumer_count = var.consumer_count

  tags = module.tags.tags
}

1. Output Values

Inside the outputs.tf file of the module, you can define output values that can be referenced in the main configuration. For example:

output "name" {
  description = "Name of the Kinesis stream."
  value       = module.kinesis.name
}

output "shard_count" {
  description = "Number of shards provisioned."
  value       = module.kinesis.shard_count
}

output "stream_arn" {
  description = "ARN of the Kinesis stream."
  value       = module.kinesis.stream_arn
}

First Time Usage

uncomment the backend block in main.tf

terraform init -backend-config=config.dev.hcl

If testing locally, terraform init should be fine

Create a dev workspace

terraform workspace new dev

Plan Terraform

terraform plan -var-file terraform.tfvars

Apply Terraform

terraform apply -var-file terraform.tfvars

Production Setup

terraform init -backend-config=config.prod.hcl

Create a prod workspace

terraform workspace new prod

Plan Terraform

terraform plan -var-file prod.tfvars

Apply Terraform

terraform apply -var-file prod.tfvars  

Cleanup

Destroy Terraform

terraform destroy -var-file dev.tfvars

Requirements

Name	Version
terraform	~> 1.5, < 2.0.0
aws	>= 4.0, < 6.0

Providers

Name	Version
aws	5.58.0

Modules

No modules.

Resources

Name	Type
aws_kinesis_stream.this	resource
aws_kinesis_stream_consumer.this	resource

Inputs

Name	Description	Type	Default	Required
consumer_count	Number of consumers to register with Kinesis stream	number	0	no
encryption_type	The encryption type to use. Acceptable values are NONE and KMS.	string	"KMS"	no
enforce_consumer_deletion	A boolean that indicates all registered consumers should be deregistered from the stream so that the stream can be destroyed without error.	bool	true	no
kms_key_id	The GUID for the customer-managed KMS key to use for encryption.	string	"alias/aws/kinesis"	no
name	Name of kinesis-stream	string	n/a	yes
retention_period	Length of time data records are accessible after they are added to the stream. The maximum value of a stream's retention period is 168 hours. Minimum value is 24. Default is 24.	number	24	no
shard_count	The number of shards that the stream will use	number	1	no
shard_level_metrics	A list of shard-level CloudWatch metrics which can be enabled for the stream.	list(string)	[ "IncomingBytes", "OutgoingBytes" ]	no
stream_mode	Specifies the capacity mode of the stream. Must be either PROVISIONED or ON_DEMAND. If ON_DEMAND is used, then shard_count is ignored.	string	null	no
tags	Tags for AWS elasticache redis	map(string)	n/a	yes

Outputs

Name	Description
consumers	List of consumers registered with Kinesis stream.
name	Name of the Kinesis stream.
shard_count	Number of shards provisioned.
stream_arn	ARN of the Kinesis stream.

Git commits

while Contributing or doing git commit please specify the breaking change in your commit message whether its major,minor or patch

For Example

git commit -m "your commit message #major"

By specifying this , it will bump the version and if you dont specify this in your commit message then by default it will consider patch and will bump that accordingly

Development

Prerequisites

• terraform
• terraform-docs
• pre-commit
• golang
• golint

Configurations

• Configure pre-commit hooks

  pre-commit install
  

Tests

• Tests are available in test directory
• Configure the dependencies

  cd test/
  go mod init github.com/sourcefuse/terraform-aws-refarch-<module_name>
  go get github.com/gruntwork-io/terratest/modules/terraform
  

• Now execute the test
  

  go test -timeout  30m
  

Authors

This project is authored by: - SourceFuse

Contributors ✨

Thanks goes to these wonderful people (emoji key):

Mayank Sharma 🚇 🚧 🧑‍🏫 💻

This project follows the all-contributors specification. Contributions of any kind welcome!