Understanding OpenStack Heat Auto Scaling

Heat Autoscaling

OpenStack Heat can deploy and configure multiple instances in one command using resources we have in OpenStack. That’s called a Heat Stack.

Heat will create instances from images using existing flavors and networks. It can configure LBaaS and provide VIPs for our load-balanced instances. It can also use the metadata service to inject files, scripts or variables after instance deployment. It can even use Ceilometer to create alarms based on instance CPU usage and associate actions like spinning up or terminating instances based on CPU load.

All the above is done by Heat to provide autoscaling capabilities to our applications. In this post I explain how to do this in RHEL 7 instances. If you want to reproduce this in another OS it’s as simple as replacing how the example webapp packages are installed.

Steps to have Heat autoscaling

1. Create a WordPress repo in a RHEL 7 box. Make sure it’s a basic installation so that all the dependencies are downloaded along with WordPress:

# Install EPEL and Remi repos first, then create a repo
yum -y install http://dl.fedoraproject.org/pub/epel/beta/7/x86_64/epel-release-7-0.2.noarch.rpm
yum -y install http://rpms.famillecollet.com/enterprise/remi-release-7.rpm
yum -y --enablerepo=remi install wordpress --downloadonly --downloaddir=/var/www/html/repos/wordpress
createrepo /var/www/html/repos/wordpress

2. Create a repo for rhel-7-server-rpms with something like:

# First register to Red Hat's CDN with subscription-manager register
# Then subscribe to the channels to be synchronised
reposync -p /var/www/html/repos/ rhel-7-server-rpms
createrepo /var/www/html/repos/rhel-7-server-rpms

3. Download the Heat template which consists on two files: autoscaling.yaml and lb_server.yaml 

Note: The template autoscaling.yaml uses lb_server.yaml (nested stacks?) and it can’t be deployed from Horizon right now due to a bug. It works fine from the command line as described below.

[Update] Note: I made it work in horizon by:

  • Publishing the two templates on a web server.
  • Modifying the autoscaling.yaml template published in the web server to call the nested template like this:
type: http://172.16.0.129:81/repos/heat-templates/lb_server.yaml

4. Modify the Heat template so the first thing it does when cloud-init executes the script passed via user_data by Heat is installing the WordPress repos.

a. Right before yum -y install httpd wordpress add the repos making it look like this:

 
[...]     
      user_data_format: RAW
      user_data:
        str_replace:
          template: |
            #!/bin/bash -v
            #Add local repos for wordpress and rhel7
            cat << EOF >> /etc/yum.repos.d/rhel.repo
            [rhel-7-server-rpms]
            name=rhel-7-server-rpms
            baseurl=http://172.16.0.129:81/repos/rhel-7-server-rpms
            gpgcheck=0
            enabled=1

            [wordpress]
            name=wordpress
            baseurl=http://172.16.0.129:81/repos/wordpress
            gpgcheck=0
            enabled=1
            EOF

            yum -y install httpd wordpress
[...]

b. And right before yum -y install mariadb mariadb-server do exactly the same.

Note: I’m assuming that your two repos are accessible via http from the instances.

Note: All of these steps are optional. If your instances pull packages directly from the Internet and/or another repository you can skip or adapt this to your environment.

5. Take note of:

  • The glance image you will use: nova image-list
    • Note: I’m using the RHEL 7 image available in the Red Hat Customer Portal  rhel-guest-image-7.0-20140618.1.x86_64.qcow2
  • The ssh key pair you want to use: nova keypair-list
  • The flavor you want to use with them: nova flavor-list
  • The subnet where the instances of the Heat stack will be launched on.

6. Create the Heat stack:

heat stack-create AutoscalingWordpress -f autoscaling.yaml \
-P image=rhel7 \
-P key=ramon \
-P flavor=m1.small \
-P database_flavor=m1.small \
-P subnet_id=44908b41-ce16-4f8c-ba6c-9bb4303e6d3f \
-P database_name=wordpress \
-P database_user=wordpress

Note: Here we use all the parameters from the template downloaded before. The are found in the parameters: section of the YAML file. We could add default: value within the template alternatively.

Now, what I do right after is a tail -f /var/log/heat/*log in the controller node, where I have Heat installed, just to make sure everything is fine with the creation of the heat stack.

7. Verify Heat created a LBaaS pool and VIP:

[root@racedo-rhel7-1 heat(keystone_demo)]# neutron lb-pool-list  
+--------------------------------------+----------------------------------------+----------+-------------+----------+----------------+--------+  
| id                                  | name                                  | provider | lb_method  | protocol | admin_state_up | status |  
+--------------------------------------+----------------------------------------+----------+-------------+----------+----------------+--------+  
| 78f02e89-aa07-40fd-917b-1481175b43e8 | AutoscalingWordpress-pool-46zb7elgzamo | haproxy  | ROUND_ROBIN | HTTP    | True          | ACTIVE |  
+--------------------------------------+----------------------------------------+----------+-------------+----------+----------------+--------+
[root@racedo-rhel7-1 heat(keystone_demo)]# neutron lb-vip-list  
+--------------------------------------+----------+-----------+----------+----------------+--------+  
| id                                  | name    | address  | protocol | admin_state_up | status |  
+--------------------------------------+----------+-----------+----------+----------------+--------+  
| 8da663cb-43d7-49af-9343-360431e02655 | pool.vip | 10.1.1.14 | HTTP    | True          | ACTIVE |  
+--------------------------------------+----------+-----------+----------+----------------+--------+  

8. Associate a floating IP to the VIP: neutron floatingip-associate FLOATING_IP_ID VIP_NEUTRON_PORT_ID. In my case I need a floating IP:

[root@racedo-rhel7-1 heat(keystone_demo)]# neutron lb-vip-show pool.vip | grep port_id
| port_id             | 13c01599-23f1-4e1e-96d9-72f2775e6183 |
[root@racedo-rhel7-1 heat(keystone_demo)]# neutron floatingip-list
+--------------------------------------+------------------+---------------------+--------------------------------------+
| id                                   | fixed_ip_address | floating_ip_address | port_id                              |
+--------------------------------------+------------------+---------------------+--------------------------------------+
| 0525f959-5213-4291-a1f0-a2ea2b40e11c |                  | 172.16.0.53         |                                      |
| 09f1bdc9-228b-4057-a5d1-3327ccc0bfc8 |                  | 172.16.0.54         |                                      |
| 5538961a-3423-46a3-9744-aba699e722c5 |                  | 172.16.0.52         |                                      |
+--------------------------------------+------------------+---------------------+-------------------------------------
root@racedo-rhel7-1 heat(keystone_demo)]# neutron floatingip-associate 0525f959-5213-4291-a1f0-a2ea2b40e11c 13c01599-23f1-4e1e-96d9-72f2775e6183

Note: This is optional if your instances are connected to a provider network where you can access directly instead of to a tenant network like in this example.

9. Verify that Heat created the two Ceilometer alarms; one to scale out on high CPU usage and another one to scale down on low CPU:

root@racedo-rhel7-1 heat(keystone_demo)]# ceilometer alarm-list
+--------------------------------------+--------------------------------------------------+-------+---------+------------+---------------------------------+------------------+
| Alarm ID                            | Name                                            | State | Enabled | Continuous | Alarm condition                | Time constraints |
+--------------------------------------+--------------------------------------------------+-------+---------+------------+---------------------------------+------------------+
| 1610f404-8df7-46ed-b131-6d3797fc9e4e | AutoscalingWordpress-cpu_alarm_low-vinrbn2rdjpx  | alarm | True    | False      | cpu_util < 15.0 during 1 x 600s | None            |
| 53c124bd-db57-4909-af55-009f5a635937 | AutoscalingWordpress-cpu_alarm_high-42dc5funjeds | ok    | True    | False      | cpu_util > 50.0 during 1 x 60s  | None            |
+--------------------------------------+--------------------------------------------------+-------+---------+------------+---------------------------------+------------------+

10. Verify you can access the WordPress using the VIP:

Wordpress

11. Now ssh into the WordPress web instance (not the DB one) and put some CPU load, just a couple of dd commands will suffice. Add a floating IP to the instance first if necessary.

[cloud-user@au-g6hl-ye4uglqb5t7r-ylpghgnzyck3-server-nobvg6ftaoe7 ~]$ dd if=/dev/zero of=/dev/null  &  
[1] 908  
[cloud-user@au-g6hl-ye4uglqb5t7r-ylpghgnzyck3-server-nobvg6ftaoe7 ~]$ dd if=/dev/zero of=/dev/null  &  
[2] 909  
[cloud-user@au-g6hl-ye4uglqb5t7r-ylpghgnzyck3-server-nobvg6ftaoe7 ~]$ dd if=/dev/zero of=/dev/null  &  
[3] 910  
[cloud-user@au-g6hl-ye4uglqb5t7r-ylpghgnzyck3-server-nobvg6ftaoe7 ~]$ dd if=/dev/zero of=/dev/null  &  
[4] 911  
[cloud-user@au-g6hl-ye4uglqb5t7r-ylpghgnzyck3-server-nobvg6ftaoe7 ~]$ dd if=/dev/zero of=/dev/null  &  
[5] 912  
[cloud-user@au-g6hl-ye4uglqb5t7r-ylpghgnzyck3-server-nobvg6ftaoe7 ~]$ top  
top - 11:01:05 up 10 min,  1 user,  load average: 6.81, 1.12, 0.71  
Tasks:  90 total,  8 running,  82 sleeping,  0 stopped,  0 zombie  
%Cpu(s): 24.3 us, 75.7 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st  
KiB Mem:  1018312 total,  235068 used,  783244 free,      688 buffers  
KiB Swap:        0 total,        0 used,        0 free.    95480 cached Mem  
  
  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM    TIME+ COMMAND  
  908 cloud-u+  20  0  107920    620    528 R 15.8  0.1  0:09.80 dd  
  909 cloud-u+  20  0  107920    616    528 R 15.5  0.1  0:08.49 dd  
  911 cloud-u+  20  0  107920    620    528 R 15.5  0.1  0:07.88 dd  
  912 cloud-u+  20  0  107920    616    528 R 15.5  0.1  0:07.71 dd  
  910 cloud-u+  20  0  107920    620    528 R 15.2  0.1  0:08.11 dd  

12. Observe how Ceilometer triggers an alarm (State goes to alarm) and how a new instance is launched:

[root@racedo-rhel7-1 heat(keystone_demo)]# ceilometer alarm-list  
+--------------------------------------+--------------------------------------------------+-------+---------+------------+---------------------------------+------------------+  
| Alarm ID                            | Name                                            | State | Enabled | Continuous | Alarm condition                | Time constraints |  
+--------------------------------------+--------------------------------------------------+-------+---------+------------+---------------------------------+------------------+  
| 1610f404-8df7-46ed-b131-6d3797fc9e4e | AutoscalingWordpress-cpu_alarm_low-vinrbn2rdjpx  | ok    | True    | False      | cpu_util < 15.0 during 1 x 600s | None            |  
| 53c124bd-db57-4909-af55-009f5a635937 | AutoscalingWordpress-cpu_alarm_high-42dc5funjeds | alarm | True    | False      | cpu_util > 50.0 during 1 x 60s  | None            |  
+--------------------------------------+--------------------------------------------------+-------+---------+------------+---------------------------------+------------------+

Wordpress Autoscaling

13. Kill the dd processes (directly from top press k and kill all of them)

14. Wait about 10 minutes, which is the duration of the scale down alarm by default in our template. The state of the alarm in Ceilometer will go to alarm just like before but now due to lack of CPU load. You’ll see how one of the two instances is deleted:

[root@racedo-rhel7-1 heat(keystone_demo)]# ceilometer alarm-list  
+--------------------------------------+--------------------------------------------------+-------+---------+------------+---------------------------------+------------------+  
| Alarm ID                             | Name                                             | State | Enabled | Continuous | Alarm condition                 | Time constraints |  
+--------------------------------------+--------------------------------------------------+-------+---------+------------+---------------------------------+------------------+  
| 1610f404-8df7-46ed-b131-6d3797fc9e4e | AutoscalingWordpress-cpu_alarm_low-vinrbn2rdjpx  | alarm | True    | False      | cpu_util < 15.0 during 1 x 600s | None             |  
| 53c124bd-db57-4909-af55-009f5a635937 | AutoscalingWordpress-cpu_alarm_high-42dc5funjeds | ok    | True    | False      | cpu_util > 50.0 during 1 x 60s  | None             |  
+--------------------------------------+--------------------------------------------------+-------+---------+------------+---------------------------------+------------------+  

That’s all.

One thought on “Understanding OpenStack Heat Auto Scaling

  1. Pingback: OpenStack Community Weekly Newsletter (Aug 1 – 8) » The OpenStack Blog

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s