Setup Nginx, SSL , Firewall | Moving micro-services into AWS EC2 instance – Part 4

Install Nginx proxy server. Nginx also act like a load-balacer which is helpful for the balancing of network traffic.

sudo apt-get update
sudo apt-get install nginx

Commands to stop, start, restart, check status

sudo systemctl stop nginx
sudo systemctl start nginx
sudo systemctl restart nginx

# after making configuration changes
sudo systemctl reload nginx
sudo systemctl disable nginx
sudo systemctl enable nginx

Install SSL – Letsencrypt

Install packages needed for ssl

sudo add-apt-repository ppa:certbot/certbot
sudo apt-get update
sudo apt-get install python-certbot-nginx

Install the SSL Certificate:

certbot -d '*.domain.com' -d domain.com --manual --preferred-challenges dns certonly

Your certificate and chain have been saved at:
   /etc/letsencrypt/live/domain.com/fullchain.pem

Your key file has been saved at:
   /etc/letsencrypt/live/domain.com/privkey.pem
SSL certificate auto renewal

Let’s Encrypt’s certificates are valid for 90 days. To automatically renew the certificates before they expire, the certbot package creates a cronjob which will run twice a day and will automatically renew any certificate 30 days before its expiration.

Since we are using the certbot webroot plug-in once the certificate is renewed we also have to reload the nginx service. To do so append –renew-hook “systemctl reload nginx” to the /etc/cron.d/certbot file so as it looks like this:

/etc/cron.d/certbot
0 */12 * * * root test -x /usr/bin/certbot -a \! -d /run/systemd/system && perl -e 'sleep int(rand(3600))' && certbot -q renew --renew-hook "systemctl reload nginx"

To test the renewal process, use the certbot –dry-run switch:

sudo certbot renew --dry-run

Renew your EXPIRED certificate this way:

sudo certbot --force-renewal -d '*.domain.com' -d domain.com --manual --preferred-challenges dns certonly

Are you OK with your IP being logged?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(Y)es/(N)o: Y

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Please deploy a DNS TXT record under the name
_acme-challenge.<domain>.com with the following value:

O3bpxxxxxxxxxxxxxxxxxxxxxxxxxxY4TnNo

Before continuing, verify the record is deployed.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Press Enter to Continue

You need to update the DNS txt record for _acme-challenge.<domain>.com

sudo systemctl restart nginx # restart nginx to take effect

Configure the Firewall

Next, we’ll update our firewall to allow HTTPS traffic.

Check firewall status in the system. If it is inactive enable firewall.

sudo ufw status # check status

# enable firewall
sudo ufw enable
sudo ufw allow ssh
sudo ufw allow OpenSSH

Enable particular ports where your micro-services are running. Example:

sudo ufw allow 4031/tcp # Authentication service
sudo ufw allow 4131/tcp # File service
sudo ufw allow 4232/tcp # Search service

You can delete the ‘Authentication service’ firewall rule by:

sudo ufw delete allow 4031/tcp

Basic Software installation| Moving micro-services into AWS EC2 instance – Part 1

As I mentioned in the previous post, I should move out of micro-services. Solving this I am taking an AWS EC2 instance and configuring every micro-services in this instance. Here I use Ubuntu 16.04 machine, because my application setup is little old. You can take Ubuntu 20.04 if you have newer versions of Rails, Ruby etc.

Our setup includes Ruby on Rails (5.2.1) micro-services (5-10 nos), NodeJS application, Sinatra Application, Angular 9.1 Front-End Application.

Go to AWS EC2 home page and select ubuntu 16.04 machine with default configurations and ssh enabled.

https://ap-south-1.console.aws.amazon.com/ec2/v2/home

Now login to this new instance and install all the packages we needed for our setup.

Software Installation

Update the package list.

sudo apt-get update

Install Ruby dependencies.

sudo apt-get install ruby-dev
sudo apt-get install libxml2-dev
sudo apt-get install libxslt-dev
sudo apt-get install graphviz

Install NodeJS

curl -sL https://deb.nodesource.com/setup_10.x | sudo -E bash -
sudo apt-get install -y nodejs
node -v

Install yarn and other dependencies.

curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list
sudo apt-get update
sudo apt-get install git-core zlib1g-dev build-essential libssl-dev libreadline-dev libyaml-dev libsqlite3-dev sqlite3 libxml2-dev libxslt1-dev libcurl4-openssl-dev software-properties-common libffi-dev nodejs yarn

Install Mysql 5.7 (Remember this is for Ubuntu 16.04, 18.04 versions)

sudo apt-get install mysql-server-5.7 mysql-client-core-5.7 libmysqlclient-dev
sudo service mysql status # or
systemctl status mysql
username: <your-username>, password: <your-password>

You can also try
mysql_secure_installation, if you use other mysql version.

Note: If you are setting up Ubuntu 20.04 note that there is a big change in mysql and the version of mysql is 8.0 instead of 5.7. If you have applications running in mysql 5.7 then setup and use Ubuntu 16.04 or 18.04.

Let’s continue the installation in our next post.

Our problem with micro-services using AWS ECS

We are into a startup and as a trending technology our predecessors used micro-services for our new website.

It is a good move if we see the benefits of micro-services like

  • scaling a website gets much easier
  • loosely coupled
  • easy for developing
  • deployment can focus on each service
  • can be used different technologies for each service
  • testing can be concentrated on one service

Here we are considered the problems we may face in future and developed the application thinking about the benefits of micro-services.

Point to be noted: We also have to think about – does these problems will be a big issue for us compared to the disadvantages of the micro-services?

NOTE: Our Website is a less traffic website and we are getting our clients one by one. It is not a heavily used website. Keep this in mind while reading this article.

We faced issues following issues with micro-services:

  • Complexity – more complex for deployment and maintenance
  • Cost – AWS ECS costs more for all the micro-service hosting for a less traffic website
  • Even though these are micro-services it has some dependency between them that we cannot avoid. Be ready to deal with this.
  • Sometimes one service (AWS Container) goes offline and starting over! So we get the No Service issue in the website.
  • Very difficult to go back to monolith application even clubbing 3-4 services together, as they use different software or software versions.

So be careful if you are planning to use micro-service architecture for a less used website or as a starting of your business. Take your time and think if micro-service architecture suits your business and your current situation.

The fact: It should not take that much cost for a website which has less traffic. Be ready to pay minimum amount for hosting with micro-services.

Set up:

  • Used AWS ECS (ec2 launch type) with services and task definitions defined
  • 11 Micro-services, 11 containers are spinning
  • Cost: Rs.12k ($160) per month

Workaround:

  • Consider using AWS Fargate type but not sure these issues get resolved
  • Deploy all the services in one EC2 Instance without using ECS

AWS: How to push your docker images to ECR

Try to do the docker login from your terminal. If you face the following error:

Error: Cannot perform an interactive login from a non TTY device

For fixing this update your aws cli version 1.x.x to 2

If you don’t have aws-cli installed, please install it.

Install AWS-CLI v2

$ curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"

$ unzip awscliv2.zip

$ sudo ./aws/install --update -b /home/abhilash/.local/bin

$ aws --version

NOTE: Add/Update access and secret key with aws configure of your new I AM role that has ECR access.

Goto https://ap-south-1.console.aws.amazon.com/ecr/repositories and create a repository.

Login and push to this repository URI:

$ aws ecr get-login-password --region ap-south-1 | docker login --username AWS --password-stdin xxxxxxxx.dkr.ecr.ap-south-1.amazonaws.com

If this cmd not works for some reason, you can try another way:

$ docker login -u AWS -p $(aws ecr get-login-password --region ap-south-1) xxxxxxxx.dkr.ecr.ap-south-1.amazonaws.com

Tag the local image:

$ docker tag test-api:latest xxxxxxxx.dkr.ecr.ap-south-1.amazonaws.com/test-api-container:latest

Push the image:

$ docker push xxxxxxxx.dkr.ecr.ap-south-1.amazonaws.com/test-api-container:latest

AWS doc url: https://docs.aws.amazon.com/AmazonECR/latest/userguide/docker-push-ecr-image.html

AWS SNS: How to send SMS to a topic, OTP SMS, promotional or transactional SMS

You can send SMS in 2 ways using AWS SNS.

  1. Subscribe to a topic that is already created in AWS SNS and send sms to all numbers who has the subscription.
  2. Send SMS directly to a mobile number.

You can find the following aws doc as a starting point from web and it describes how to create a topic, subscribe a topic and sending sms to the mobile numbers.

https://docs.aws.amazon.com/code-samples/latest/catalog/code-catalog-ruby-example_code-sns.html

For the demonstration purpose, I use Ruby here.

  1. Subscribe to a topic and send SMS
require 'aws-sdk-sns'  # v2: require 'aws-sdk'

sns = Aws::SNS::Resource.new(region: 'us-west-2')

topic = sns.topic('arn:aws:sns:us-west-2:123456789:MySampleTextTopic')

topic.publish({
  message: 'Hello!'
})

This assumes you already created a topic inside your aws console:

Goto AWS Console

  1. Open AWS SNS
  2. Goto Left side -> Mobile -> text messaging sms
  3. Create a topic

You can also follow the above link to perform an API to create topic, subscribe to a topic and send sms

2. Send SMS directly to a mobile number (eg: Send OTP SMS)

Either you can use aws console to send the sms or SNS API.

AWS console:

  1. Open AWS SNS
  2. Goto Left side -> Mobile -> text messaging sms
  3. We are using transactional text messages
  4. Goto publish text message
  5. Select transactional, add mobile number and publish it

Almost all cases we use an API for sending OTP SMS. For that follow the steps.

SNS API – Steps

gem install aws-sdk-sns 
require 'aws-sdk-sns'
otp = generate_otp
set_sns_client
set_sns_client_attrs
response = publish_sms(otp)
  • Generate an OTP
def generate_otp
    (1000..9999).to_a.sample
end
  • Set SNS client
def set_sns_client
    @sns_client = Aws::SNS::Client.new(
      region: ENV['AWS_SNS_REGION'],
      access_key_id: ENV['AWS_SNS_ACCESS_KEY'],
      secret_access_key: ENV['AWS_SNS_SECRET_KEY']
    )
end

  • Set SNS client attributes
 def set_sns_client_attrs
    @sns_client.set_sms_attributes({
                                     attributes: {
                                       'DefaultSenderID' => SENDER_ID,
                                       'DefaultSMSType' => SMS_TYPE
                                     }
                                   })
end

  • Publish SMS
def publish_sms(otp)
    @sns_client.publish({
                          phone_number: @mobile_no,
                          message: "#{OTM_MSG} #{otp}"
                        })
  end

Here ,

OTM_MSG: ‘Your OTP for login is’

SMS_TYPE : ‘Transactional’ or ‘Promotional’

If you want to send OTP, then it is ‘Transactional’. Else if you want to send some promotional sms of your product then it is ‘Promotional’

SENDER_ID: is the sms header that you already registered with TRAI.

The Steps to add Sender ID in AWS is given below:

For example suppose your sender id is: Zomato

Follow the steps to add our SENDER ID – Zomato to AWS SNS

  1. Sign in to the Amazon SNS console – https://console.aws.amazon.com/sns/home
  2. On the navigation panel, choose Mobile, Text messaging (SMS).
  3. On the Mobile text messaging (SMS) page, in the Text messaging preferences section, choose Edit.
  4. On the Edit text messaging preferences page, in the Details section, do the following:
  5. For Default sender ID , enter the provided sender ID to be used (Zomato) as the default for all messages from your account.
  6. Choose Save changes.

If you don’t want to register sender id, then skip this method: set_sns_client_attrs and publish the sms. It take the sms as ‘Promotional’ and sender id will be 8 character random number. Amazon use this type of sms from International route and it costs you almost $0.02 (Rs. 1.5) per sms. Very high rate. So I recommend to register any sender id that resembles your product or company name, from Jio trueconnect (that is free, link given below) and use it in SNS.

If you don’t know how to register sender id, follow this:

For AWS SNS service, there is 2 way of sending sms.

  1. Local route
  2. International route

For local route the price is Rs. 0.20 per sms
For international route the price will be Rs 1.58 per sms – too high

by default AWS SNS use International route

If you are from India follow the TRAI registration
For considering local route we have to register our use case and message templates with TRAI .

So first register here:
https://www.vilpower.in/
as an enterprise / company with all company details and our purpose

These registration requirements are designed to reduce the number of unsolicited messages that Indian consumers receive, and to protect consumers from potentially harmful messages

You can Register in Jio for free:

The link to register:
https://trueconnect.jio.com/#/home/entity-registration
Select Principal entity and continue

Recently Indian Govt made DLT Registration mandatory for sending sms.

Example:
Take Msg from AD-ZOMATO , here ZOMATO is 6 char sender id that we can give in the service provider and send sms before. But now we have to register this in DLT then only our service provider can use this.

After registering DLT we get an ENTITY ID. This entity id need to be attached in our’s otp service provider for sending otp msgs.

If you are using SNS service for the first time you should increase your SMS quota:

AWS says:

If you're new to SMS messaging with Amazon SNS, request a monthly SMS spending threshold that meets the expected demands of your SMS use case. By default, your monthly spending threshold is $1.00 (USD). You can request to increase your spending threshold in the same support case that includes your request for a sender ID

Because Amazon SNS is a distributed system, it stops sending SMS messages within minutes of the spending quota being exceeded. During this period, if you continue to send SMS messages, you might incur costs that exceed your quota.

https://docs.aws.amazon.com/sns/latest/dg/channels-sms-awssupport-sender-id.html

Requesting increases to your monthly SMS spending quota for Amazon SNS:

https://docs.aws.amazon.com/sns/latest/dg/channels-sms-awssupport-spend-threshold.html

Currently, Amazon SNS supports SMS messaging in the following AWS Regions:

https://docs.aws.amazon.com/sns/latest/dg/sns-supported-regions-countries.html

Reference:

https://docs.aws.amazon.com/sdk-for-ruby/v2/api/Aws/SNS/Client.html