Resizing OpenStack Volumes

Hard Drive

Resizing a Volume with Cinder in Havana

Cinder has the extend functionality in Havana which allows an easy resizing procedure of volumes. It works as expected on the volumes. On the OS I have found it less reliable when using resize2fs on the extended volume. Maybe I haven’t done enough tests yet but in any case the method below works in Havana and in Grizzly.

Resizing a Volume in Grizzly and Havana

The following method works in both Grizzly and Havana. It can be entirely done by the tenant with the nova command (the cinder client is not needed).

1. Identify the volume to be resized

$ nova volume-list
+--------------------------------------+-----------+--------------+------+-------------+--------------------------------------+
| ID | Status | Display Name | Size | Volume Type | Attached to |
+--------------------------------------+-----------+--------------+------+-------------+--------------------------------------+
| 44bcd404-8a6e-41d8-9d56-2ac4e0c1e97c | in-use | None | 10 | None | 438cdb78-5573-4ab0-9f89-79cad806286c |
| 010ea497-98d5-4ace-a6aa-bdc847628cee | available | None | 1 | None | |
+--------------------------------------+-----------+--------------+------+-------------+--------------------------------------+

2. Detach the volume from its instance. It is recommended to ssh into the instance and to unmount it first.

$ nova volume-detach VM1 44bcd404-8a6e-41d8-9d56-2ac4e0c1e97c

3. Create a snapshot of the volume:

$ nova volume-snapshot-create 44bcd404-8a6e-41d8-9d56-2ac4e0c1e97c
+---------------------+--------------------------------------+
| Property            | Value                                |
+---------------------+--------------------------------------+
| status              | creating                             |
| display_name        | None                                 |
| created_at          | 2014-01-16T16:20:17.739982           |
| display_description | None                                 |
| volume_id           | 44bcd404-8a6e-41d8-9d56-2ac4e0c1e97c |
| size                | 10                                   |
| id                  | ea8a1c24-982e-4d63-809f-38f0ad974604 |
| metadata            | {}                                   |
+---------------------+--------------------------------------+

4. Create a new volume from the the snapshot of the volume we are resizing specifying the new desired size:

$ nova volume-create --snapshot-id ea8a1c24-982e-4d63-809f-38f0ad974604 15
+---------------------+--------------------------------------+
| Property            | Value                                |
+---------------------+--------------------------------------+
| status              | creating                             |
| display_name        | None                                 |
| attachments         | []                                   |
| availability_zone   | nova                                 |
| bootable            | false                                |
| created_at          | 2014-01-16T16:22:10.634404           |
| display_description | None                                 |
| volume_type         | None                                 |
| snapshot_id         | ea8a1c24-982e-4d63-809f-38f0ad974604 |
| source_volid        | None                                 |
| size                | 15                                   |
| id                  | 408a9d90-6498-4e87-a26a-43fe506b1b1d |
| metadata            | {}                                   |
+---------------------+--------------------------------------+

5. Wait until the status of the newly created volume is available and attach it to the instance using another device name (if it originally was /dev/vdc then use /dev/vdd for example):

$ nova volume-attach VM1 408a9d90-6498-4e87-a26a-43fe506b1b1d  /dev/vdd
+----------+--------------------------------------+
| Property | Value                                |
+----------+--------------------------------------+
| device   | /dev/vdd                             |
| serverId | 438cdb78-5573-4ab0-9f89-79cad806286c |
| id       | 408a9d90-6498-4e87-a26a-43fe506b1b1d |
| volumeId | 408a9d90-6498-4e87-a26a-43fe506b1b1d |
+----------+--------------------------------------+

Note that the snapshot created in step 3 can now be deleted as well as the original volume.

From within the instance OS, assuming it’s a Linux VM, we need to make the OS aware of the new size.

ubuntu@vm1:/$ sudo e2fsck -f /dev/vdc
e2fsck 1.42 (29-Nov-2011)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/vdc: 13/65536 files (0.0% non-contiguous), 12637/262144 blocks
ubuntu@vm1:/$ sudo resize2fs /dev/vdc
resize2fs 1.42 (29-Nov-2011)
Resizing the filesystem on /dev/vdc to 1310720 (4k) blocks.
The filesystem on /dev/vdc is now 1310720 blocks long.

ubuntu@vm1:/$ sudo mount /dev/vdc /mnt2
ubuntu@vm1:/$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1       9.9G  828M  8.6G   9% /
udev            494M  8.0K  494M   1% /dev
tmpfs           200M  220K  199M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            498M     0  498M   0% /run/shm
/dev/vdb         20G  173M   19G   1% /mnt
/dev/vdc       15.0G   34M 14.7G   1% /mnt2

Notes

  • If resize2fs does not work, try rebooting the instance first. The kernel will come back up fresh and may pick it up fine after reboot.
  • Make sure you changed the block device name (i.e. from vdc to vdd for instance)
  • Do not use partitions (i.e. /dev/vdc1). Check resize2fs for details. It can be done anyway but rebuilding the partition table is needed.

Set up iSCSI Storage for ESXi Hosts From The Command Line

VMware Command Line Interface

The esxcli command line tool can be extremely useful to set up an ESXi host, including iSCSI storage.

1. Enable iSCSI:

~ # esxcli iscsi software set -e true
Software iSCSI Enabled

2. Check the adapter name, usually vmhba32, vmhba33, vmhba34 and so on.

~ # esxcli iscsi adapter list
Adapter Driver State UID Description
------- --------- ------ ------------- ----------------------
vmhba32 iscsi_vmk online iscsi.vmhba32 iSCSI Software Adapter

3. Connect your ESXi iSCSI adapter to your iSCSI target

~ # esxcli iscsi adapter discovery sendtarget add -A vmhba32 -a 10.230.5.60:3260

~ # esxcli iscsi adapter get -A vmhba32
vmhba32
Name: iqn.1998-01.com.vmware:ch02b03-65834587
Alias:
Vendor: VMware
Model: iSCSI Software Adapter
Description: iSCSI Software Adapter
Serial Number:
Hardware Version:
Asic Version:
Firmware Version:
Option Rom Version:
Driver Name: iscsi_vmk
Driver Version:
TCP Protocol Supported: false
Bidirectional Transfers Supported: false
Maximum Cdb Length: 64
Can Be NIC: false
Is NIC: false
Is Initiator: true
Is Target: false
Using TCP Offload Engine: false
Using ISCSI Offload Engine: false

4. Now, on your iSCSI server, assign a volume of the SAN to the IQN of your ESXi host, for example, for a HP StorageWorks:

CLIQ>assignVolume volumeName=racedo-vSphereVolume initiator=iqn.1998-01.com.vmware:ch02b01-01e26a74;iqn.1998-01.com.vmware:ch02b02-20d3e33b;iqn.1998-01.com.vmware:ch02b03-65834587

The above command assigned three IQNs to the volume, two that we already had and the new one we are setting up. This is just for HP StorageWorks CLI, other storage arrays work differently.

5. Back on the ESXi host, discover the targets:

~ # esxcli iscsi adapter discovery rediscover -A vmhba32

Finally, check with the df command that the datastore has been added. If not, try rediscovering again.

This is the simplest configuration possible from the command line. NIC teaming or other more complex setups can also be done from the command line of the ESXi hosts.

Deploying vSphere Remotely From the Command Line

VMware Command Line Interface

If all we have is remote access (ssh) to an ESXi host and we still need install vCenter Server to get vSphere up and running, we can do it with ovftool. The ovftool comes with VMware Workstation and can also be downloaded separately if needed.

The idea is simple: download the vCenter Server OVA appliance to the ESXi host datastore, copy the ovftool to the ESXi host datastore and use it to install the appliance.

This process assumes that the ESXi host has the ssh service enabled.

1. Download the vCenter Server OVA appliance. For example, from a Dropbox folder we share. ESXi comes with wget installed so we can do:

# cd /vmfs/volumes/datastore1
# wget https://dropbox-url/VMware-vCenter-Server-Appliance-5.1.0.5300-947940_OVF10.ova

2. Copy the ovftool (with its libraries) to the ESXi server. Just replace bash by sh as the interpreter and copy it from any Linux box with Workstation installed in it:

# vi /usr/lib/vmware-ovftool/ovftool

Replace:

#!/bin/bash

by:

#!/bin/sh

Copy its directory to the ESXi host:

# scp -r /usr/lib/vmware-ovftool/ root@esxi-ip:/vmfs/volumes/datastore1

3. Now, all it’s left is installing the vCenter Server appliance with it:

Find the actual path to the datastore where the appliance and the ovftool were copied:

# ls -l /vmfs/volumes/datastore1
lrwxr-xr-x    1 root     root            35 Nov 26 03:29 /vmfs/volumes/datastore1 -> 52900c65-73c1bf38-6469-001f29e4ff20

Using the full path, run the ovftool to install the vCenter Server appliance:

# /vmfs/volumes/52900c65-73c1bf38-6469-001f29e4ff20/vmware-ovftool/ovftool -dm=thin /vmfs/vol
umes/52900c65-73c1bf38-6469-001f29e4ff20/VMware-vCenter-Server-Appliance-5.1.0.5300-947940_OVF10.ova "vi://root:password@localhost"

The output should be similar to this:

Opening OVA source: /vmfs/volumes/52900c65-73c1bf38-6469-001f29e4ff20/VMware-vCenter-Server-Appliance-5.1.0.5300-947940_OVF10.ova
The manifest validates
Source is signed and the certificate validates
Accept SSL fingerprint (B3:DC:DF:58:00:68:A3:92:A9:A4:65:41:B2:F6:FF:CF:99:2A:3E:71) for host localhost as target type.
Fingerprint will be added to the known host file
Write 'yes' or 'no'
yes
Opening VI target: vi://root@localhost:443/
Deploying to VI: vi://root@localhost:443/
Transfer Completed
Completed successfully

When this is finished, the vCenter Server VM is created in the ESXi host.

4. Power on the vCenter Server VM.

First, find out its VM ID:

# vim-cmd vmsvc/getallvms

Assuming the VM ID is 1, power it on with:

# vim-cmd vmsvc/power.on 1

And once it’s up and running, find its IP running:

# vim-cmd vmsvc/get.summary 1|grep ipAddress

Point your browser to that IP on port 5480 to start configuring the the vCenter Server you just deployed.

Add a Swap Partition to Nodes Deployed by MAAS

MAASMAAS or Metal as a Service is a service that, combined with Juju, allows to provision physical machines and even VMs extremely easily via PXE. Setting up MAAS on an Ubuntu Server (I recommend 12.04) takes literally 5 minutes. I recommend to install the version from the Tools pocket in the Cloud Archive, and configure maas-dns and maas-dhcp from the Web UI.

MAAS deploys machines without a swap partition by default. This may cause problems when deploying some services, especially if we use multiple LXC containers or Docker on the same machine.

To have MAAS configuring a swap partition all we need to do is to edit the main preseed file:

vi /etc/maas/preseeds/preseed_master

And right after this line that says:

d-i     partman-auto/method string regular

We add the following:

d-i	partman-auto/text/atomic_scheme :: \
		500 10000 1000000 ext3 \
		        $primary{ } \
       			 $bootable{ } \
        		method{ format } \
        		format{ } \
        		use_filesystem{ } \
        		filesystem{ ext3 } \
        		mountpoint{ / } . \
		64 512 300% linux-swap \
        		method{ swap } \
        		format{ } . \

References: Altering the Preseed File

Setting Up a Flat Network with Neutron

Data Network

This setup will allow the VMs to use an existing network. In this example, eth2 is connected to this pre-existing network (192.168.2.0/24) that we want to use for the OpenStack VMs.

All the configuration is done in the node dedicated to Nova Networking.

1. Set up the Open vSwitch bridge:

# ovs-vsctl add-br br-eth2
# ovs-vsctl add-port br-eth2 eth2

2. Set up /etc/network/interfaces (node’s IP is 192.168.2.7):

auto eth2
iface eth2 inet manual
up ifconfig $IFACE 0.0.0.0 up
up ip link set $IFACE promisc on
down ip link set $IFACE promisc off
down ifconfig $IFACE down

auto br-eth2
iface br-eth2 inet static
address 192.168.2.7
netmask 255.255.255.0

3. Tell Open vSwitch to use the bridge connected to eth2 (br-eth2) and map physnet1 to it /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini:

[ovs]
network_vlan_ranges = physnet1
bridge_mappings = physnet1:br-eth2

4. Tell Neutron to not to use the metadata proxy in /etc/nova/nova.conf (otherwise you get a HTTP 400 when querying the metadata service):

service_neutron_metadata_proxy=false

Note that the metadata service is managed by Neutron as usual via the neutron-metadata-agent service anyway.

5. Create the network telling to use the physnet1 mapped above to br-eth2:

# neutron net-create flat-provider-network --shared  --provider:network_type flat --provider:physical_network physnet1

6. Create the subnet:

# neutron subnet-create --name flat-provider-subnet --gateway 192.168.2.5 --dns-nameserver 192.168.1.254  --allocation-pool start=192.168.2.100,end=192.168.2.150  flat-provider-network 192.168.2.0/24

That’s it. Now VMs will get an IP of the specified range and will be directly connected to our network via Open vSwitch.

Default User and Password in Ubuntu Cloud Images

Backdoor

Ubuntu Cloud Images don’t have any password for the root or ubuntu users by default, relying only on using ssh keys. It is convenient for many tenants to just having a user and a password to log in rather than only a ssh key.

This is how to set a password for the ubuntu user:

1. Download the image:

$ wget http://cloud-images.ubuntu.com/precise/current/precise-server-cloudimg-amd64-disk1.img

2. Download backdoor-image:

$ bzr branch lp:~smoser/+junk/backdoor-image

3. Set up the user and password:

$ sh backdoor-image --user ubuntu --password ubuntu --password-auth ubuntu-12.04-server-cloudimg-amd64-disk1.img

Now we can deploy the image, for example in our OpenStack infrastructure, and we can ssh into it with ubuntu as user and password or we can login via the console.

Enable Console Access to vSphere instances in OpenStack

vSphere Instance Console

Instance consoles are not working by default and require configuration in both, ESXi hosts and Nova Compute / Nova API nodes.

1. Nova API and Nova Compute nodes (usually the same node when using OpenStack and vSphere as compute) have the following in  /etc/nova/nova.conf (this assumes its IP is 192.168.2.7):

vncserver_listen=0.0.0.0
vncserver_proxyclient_address=192.168.2.7
novncproxy_base_url=http://192.168.2.7:6080/vnc_auto.html
vnc_enabled=True

Restart the services:

$ sudo restart nova-compute
$ sudo restart nova-api
$ sudo restart nova-console
$ sudo restart nova-consoleauth
$ sudo restart nova-novncproxy

2. ESXi setup.

ssh the ESXi host and check what ports the launched instances are listening on, these ports are where the embedded VNC listens on:

~ # esxcli network ip connection list|grep vmx
tcp         0       0  192.168.2.200:6111  192.168.2.7:50754   ESTABLISHED    434739  vmx-mks:92901823-a03c-4cdd-bbb6-616a8742388a
tcp         0       0  0.0.0.0:6111        0.0.0.0:0           LISTEN         434735  vmx
tcp         0       0  0.0.0.0:6102        0.0.0.0:0           LISTEN         250526  vmx
tcp         0       0  0.0.0.0:6101        0.0.0.0:0           LISTEN          11204  vmx

This can be confirmed by checking the .vmx file for the instances (this is set up by VMwareVCDriver):

~ # grep vnc.port /vmfs/volumes/datastore1/*/*vmx
/vmfs/volumes/datastore1/52c84203-ce3d-47b4-ab22-1d30b2816298/52c84203-ce3d-47b4-ab22-1d30b2816298.vmx:RemoteDisplay.vnc.port = "6102"
/vmfs/volumes/datastore1/92901823-a03c-4cdd-bbb6-616a8742388a/92901823-a03c-4cdd-bbb6-616a8742388a.vmx:RemoteDisplay.vnc.port = "6111"
/vmfs/volumes/datastore1/c4e7264e-a4f7-4dea-87c2-6561b86fb85d/c4e7264e-a4f7-4dea-87c2-6561b86fb85d.vmx:RemoteDisplay.vnc.port = "6101"

In general, you will notice these two config flags in the .vmx files:

RemoteDisplay.vnc.enabled = TRUE
RemoteDisplay.vnc.port = port_number

Now you need to open these ports:

~ # chmod 644 /etc/vmware/firewall/service.xml
~ # chmod +t /etc/vmware/firewall/service.xml
~ # vi /etc/vmware/firewall/service.xml

And append this:

<service id='0033'>
<id>VNC</id>
<rule id='0000'>
<direction>inbound</direction>
<protocol>tcp</protocol>
<porttype>dst</porttype>
<port>
<begin>5900</begin>
<end>6199</end>
</port>
</rule>
</service>

Close vi with:

:x!

Refresh the firewall rules:

~ # esxcli network firewall refresh
~ # esxcli network firewall ruleset set --ruleset-id VNC --enabled true

Done.

Note: there are multiple ways to keep the firewall configuration after ESXi reboots, please review them and chose one of them to make this change permanent.