AWS Transit Gateway

AWS Transit Gateway is a service that enables customers to connect their Amazon Virtual Private Clouds (VPCs) and their on-premises networks to a single gateway. As you grow the number of workloads running on AWS, you need to be able to scale your networks across multiple accounts and Amazon VPCs to keep up with the growth. Today, you can connect pairs of Amazon VPCs using peering. However, managing point-to-point connectivity across many Amazon VPCs, without the ability to centrally manage the connectivity policies, can be operationally costly and cumbersome. For on-premises connectivity, you need to attach your AWS VPN to each individual Amazon VPC. This solution can be time consuming to build and hard to manage when the number of VPCs grows into the hundreds.

With AWS Transit Gateway, you only have to create and manage a single connection from the central gateway in to each Amazon VPC, on-premises data center, or remote office across your network. Transit Gateway acts as a hub that controls how traffic is routed among all the connected networks which act like spokes. This hub and spoke model significantly simplifies management and reduces operational costs because each network only has to connect to the Transit Gateway and not to every other network. Any new VPC is simply connected to the Transit Gateway and is then automatically available to every other network that is connected to the Transit Gateway. This ease of connectivity makes it easy to scale your network as you grow.

Without AWS Transit Gateway

With AWS Transit Gateway

https://aws.amazon.com/transit-gateway/

AWS Enabling Enhanced Networking with ENA on Linux Instances

In order to change the instance type on EC2 Linux instances ENA needs to be enabled. In most cases this is already set, however if you have some older EC2’s running they will need to be ENA enabled for this change to occur. Otherwise you will need to create a new Instance from a fresh snapshot or detach and re-attach the EBS volume.

ENA is a custom network interface optimized to deliver high throughput and packet per second (PPS) performance, and consistently low latencies on EC2 instances. Using ENA, customers can utilize up to 20 Gbps of network bandwidth on certain EC2 instance types. ENA-based Enhanced Networking is currently supported on X1 instances, and will be available on other new EC2 instance types in the future.

Open Source licensed ENA drivers are currently available for Linux and Intel® Data Plane Development Kit (Intel® DPDK), and we will soon be releasing an ENA driver for Microsoft Windows® operating systems. The latest Amazon Linux AMI includes the ENA Linux driver support by default. ENA Linux driver source code is also available on github.com for developers to integrate in their AMIs. There is no additional fee to use ENA. For more information, read the Enhanced Networking documentation.

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced-networking-ena.html

Testing Whether Enhanced Networking Is Enabled
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced-networking-ena.html#test-enhanced-networking-ena

ubuntu:~$ modinfo ena
 ERROR: modinfo: could not find module ena

Enabling Enhanced Networking on Ubuntu
The latest Ubuntu HVM AMIs have the module required for enhanced networking with ENA installed and have the required enaSupport attribute set. Therefore, if you launch an instance with the latest Ubuntu HVM AMI on a supported instance type, enhanced networking is already enabled for your instance.

modify-instance-attribute (AWS CLI):

aws ec2 modify-instance-attribute --instance-id instance_id --ena-support

AWS Cloudformation – Resources, Tips and Tricks

Recently been creating CF templates for a job and just learning more in this area of AWS. Listed below are resources, tips and tricks that were helpful.

Resources

AWS User Guide:
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html

AWS Github Sample Templates:
https://github.com/awslabs/aws-cloudformation-templates

AWS Sample Templates:
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-sample-templates.html

Bogotobogo excellent post on Cloudformation, Templates, Change Sets and CLI:
https://www.bogotobogo.com/DevOps/AWS/aws-CloudFormation-Templates.php

Stelligent Cloudformation Templates
https://github.com/stelligent/cloudformation_templates

AWS::CloudFormation::Init
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-init.html

Tips and Tricks

yamllint – A linter for YAML files.

yamllint does not only check for syntax validity, but for weirdnesses like key repetition and cosmetic problems such as lines length, trailing spaces, indentation, etc.
https://github.com/adrienverge/yamllint

CloudFormation Linter

Validate CloudFormation yaml/json templates against the CloudFormation spec and additional checks. Includes checking valid values for resource properties and best practices.
https://github.com/aws-cloudformation/cfn-python-lint

validate-template

Validates a specified template. AWS CloudFormation first checks if the template is valid JSON. If it isn’t, AWS CloudFormation checks if the template is valid YAML. If both these checks fail, AWS CloudFormation returns a template validation error.
https://docs.aws.amazon.com/cli/latest/reference/cloudformation/validate-template.html

CreationPolicy Attribute

Associate the CreationPolicy attribute with a resource to prevent its status from reaching create complete until AWS CloudFormation receives a specified number of success signals or the timeout period is exceeded. To signal a resource, you can use the cfn-signal helper script or SignalResource API. AWS CloudFormation publishes valid signals to the stack events so that you track the number of signals sent.

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-attribute-creationpolicy.html

UpdatePolicy Attribute

Use the UpdatePolicy attribute to specify how AWS CloudFormation handles updates to the AWS::AutoScaling::AutoScalingGroup, AWS::Lambda::Alias, or AWS::ElastiCache::ReplicationGroup resources.

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-attribute-updatepolicy.html

DeletionPolicy Attribute

With the DeletionPolicy attribute you can preserve or (in some cases) backup a resource when its stack is deleted. You specify a DeletionPolicy attribute for each resource that you want to control. If a resource has no DeletionPolicy attribute, AWS CloudFormation deletes the resource by default.

Note that this capability also applies to stack update operations that lead to resources being deleted from stacks. For example, if you remove the resource from the stack template, and then update the stack with the template. This capability does not apply to resources whose physical instance is replaced during stack update operations. For example, if you edit a resource’s properties such that AWS CloudFormation replaces that resource during a stack update.

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-attribute-deletionpolicy.html


Using Nextcloud’s command line

Here is a great reference for using Nextcloud’s command options. Some of these command line switches are very handy as you can’t always do these tasks from the GUI.

Cleanup Nextclouds filecache:

sudo -u www-data php occ files:cleanup

Re-Scan your Nextcloud data:

sudo -u www-data php occ files:scan --all -v

More info and the full listing at:
https://www.c-rieger.de/using-nextclouds-command-line

AWS cli Builder

AWS cli builder is a very handy site for doing just that…. building AWS cli commands. Choose from 148 services and input your required options and it will build out the command to use.

For example to start an EC2 instance, choose Computer > Elastic Computer Cloud and type start. It should provide an option for starting instances. Now enter your details such as region, instance id and output type.

More info and to access the site go here: https://awsclibuilder.com/home

SSH Audit

Ideally you should whitelist your SSH Servers to your IPs or don’t even expose the service to the Internet and use a VPN to login. In some cases this is not an option and/or you may need to create a Bastion Host / Jump Box server with SSH exposed.

If this is the case you should do or at least consider these options:

  • Change the SSH port (May reduce the noise, but not the security)
  • Only use Key Based Authentication i.e. disable password and use keys only
  • Prevent root user from logging in via SSH
  • Limit user access via SSH

Once the above has been completed, use SSH Audit to check that everything from an SSH configuration is setup correctly and passes the security checks. SSH Audit is a free online tool allows you to audit the configuration of an SSH server.

Reference:
https://www.sshaudit.com

Zstandard compression

Zstandard, or zstd as short version, is a fast lossless compression algorithm, targeting real-time compression scenarios at zlib-level and better compression ratios. It’s backed by a very fast entropy stage, provided by Huff0 and FSE library.

The project is provided as an open-source dual BSD and GPLv2 licensed C library, and a command line utility producing and decoding .zst.gz.xz and .lz4 files. Should your project require another programming language, a list of known ports and bindings is provided on Zstandard homepage.

Installation is from repos:
sudo apt install zstd

Reference: https://facebook.github.io/zstd/

grepcidr

grepcidr is a 

grepcidr can be used to filter a list of IP addresses against one or more Classless Inter-Domain Routing (CIDR) specifications. As with grep, there are options to invert matching and load patterns from a file. grepcidr is capable of efficiently processing large numbers of IPs and networks.

grepcidr has endless uses in network software, including: mail filtering and processing, network security, log analysis, and many custom applications.

For detailed instructions and examples, please see the README file or man page. A couple examples of usage:

	grepcidr 2001:db8::/32 logfile
	grepcidr 66.249.64.0/19 access.log

Reference: http://www.pc-tools.net/unix/grepcidr/

Install grepcidr with your package manager:

 sudo apt install grepcidr 

Linux du command alternatives

For years I’ve used ncdu a NCurses Disk Usage utility for Linux. Recently someone alerted me to some other options as well as ncdu:

Dust:
du + rust = dust. Like du but more intuitive, Dust is meant to give you an instant overview of which directories are using disk space without requiring sort or head. Dust will print a maximum of 1 ‘Did not have permissions message’. Dust will list the 20 biggest sub directories or files and will smartly recurse down the tree to find the larger ones. There is no need for a ‘-d’ flag or a ‘-h’ flag. The largest sub directory will have its size shown in red.

https://github.com/bootandy/dust

The Tin Summer:
sn is a replacement for du. It has nicer output, saner commands and defaults, and it even runs faster on big directories thanks to multithreading.

https://github.com/vmchale/tin-summer

NCDU:
Ncdu is a disk usage analyzer with an ncurses interface. It is designed to find space hogs on a remote server where you don’t have an entire graphical setup available, but it is a useful tool even on regular desktop systems. Ncdu aims to be fast, simple and easy to use, and should be able to run in any minimal POSIX-like environment with ncurses installed.
Install from repos: sudo apt install ncdu

man page: https://dev.yorhel.nl/ncdu/man