Archive for HOWTO

OpenStack DevStack Configuration on CentOS 6.3

I ran into some quirks when trying to get DevStack to run out-of-the-box on CentOS 6.3. Even though it’s supported, there were a few additional steps that were needed which were not performed by stack.sh. I’m not contributing these back because I don’t know to what extent they are “bugs”.

Installing

The following steps were performed on a clean minimal install of CentOS 6.3. I highly recommend not running DevStack on any production systems or any other shared systems.

# Install some prereqs / utilities
yum -y install mlocate vim openssl-devel euca2ools telnet Django14
# EPEL is required for all the additional packages required by OpenStack
rpm --import http://dl.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-6
yum -y install http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm

# If you can't find a python-pip package the below can be used instead
#easy_install pip
#ln -s /usr/bin/pip /usr/bin/pip-python

# Horizon requires older version of Kombu. Not sure how old, but 1.0.4 seems to do the trick
# 'RabbitStrategy' object has no attribute 'connection_errors'
pip install kombu==1.0.4

# Clear out any IP table rules
iptables -F
iptables -X
iptables -t nat -F
iptables -t nat -X
iptables -t mangle -F
iptables -t mangle -X
iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT
service iptables save

# Enable SSH keys on the server for easy remote access
sed -i -r 's/^#(AllowAgentForwarding)/1/g' /etc/ssh/sshd_config

# Apply the above configuration change
service sshd restart

# Fix SSH SE-Linux permissions
restorecon -R -v /root/.ssh

# Make default policy permissive
setenforce permissive
sed -i 's/SELINUX=.*/SELINUX=permissive/' /etc/selinux/config

# Setup DevStack
cd /opt
git clone https://github.com/openstack-dev/devstack.git
cd devstack

# ImproperlyConfigured: Error importing middleware horizon.middleware: "cannot import name SafeExceptionReporterFilter"
perl -p -i -e 's/^Django$/Django14/' files/rpms/horizon
# pip install version 1.0.4 instead
perl -p -i -e 's/^(python-kombu)/#$1/' files/rpms/*

# Write a default localrc configuration
cat<<__EOF__>localrc
FLOATING_RANGE=192.168.1.0/24
FIXED_RANGE=192.168.2.0/24
FIXED_NETWORK_SIZE=256
FLAT_INTERFACE=eth0
ADMIN_PASSWORD=openadmin
MYSQL_PASSWORD=openmysql
RABBIT_PASSWORD=openrabbit
SERVICE_PASSWORD=openservice
SERVICE_TOKEN=$(uuidgen)
__EOF__

# Start the installation
rm -f nohup.out; FORCE_PREREQ=true ./stack.sh | tee nohup.out

# Fix for "Permission denied" apache error
chmod 755 /opt/stack/

Once the above completes, you’ll have your new devstack running out of /opt/stack. To reconnect to the screen session controlling the stack, run:

screen -x stack

**WARNING*** If you ever re-run the stack.sh script, you will lose all state and rebuild a fresh stack.

Stopping

To destroy the stack, run:
su – stack -c /opt/devstack/unstack.sh

Resuming

To resume a stack that has been “unstacked”, run the following command:

su - stack -c /opt/devstack/rejoin-stack.sh
# In my experience, it does not start apache automatically
service https start

Destroying

To completely start from scratch, this is what I ran:

WARNING: this blows aways your MySQL & RabbitMQ installations!

su - stack -c /opt/devstack/unstack.sh
./clean.sh

find / -name rabbitmq -type d | xargs rm -rf
yum -y remove mysql-server
rm -rf /var/lib/mysql/
rm -f /opt/stack/devstack/.prereqs

Images

You might also want to test out some more images. Fortunately, RackSpace provides some that you can use.

RackSpace OpenStack Images: https://github.com/rackerjoe/oz-image-build

Here is how you can load them into DevStack (note the URLs are subject to change):

# Load the admin credentials
. /opt/stack/devstack/accrc/admin/admin

# Download & load images
wget http://c250663.r63.cf1.rackcdn.com/centos60_x86_64.qcow2
glance image-create --name centos60_x86_64 --disk-format=raw --container-format=bare --file /opt/centos60_x86_64.qcow2

wget http://c250663.r63.cf1.rackcdn.com/ubuntu-precise_x86_64_60G.qcow2
glance image-create --name ubuntu-precise_x86_64_60G --disk-format=raw --container-format=bare --file /opt/ubuntu-precise_x86_64_60G.qcow2

CloudStack Metadata API Compatibility with EC2

Recently, I needed to support backwards compatibility for the ec2-run-user-data and ec2-ssh-host-key-gen scripts inside of CloudStack to work with cloud-init. CloudStack exposes the metadata API, but not on the same IP as in EC2 (169.254.169.254). In CloudStack, you instead need to query the address of the virtual router which is different depending on the host your on. There’s a very easy way to accomplish this using iptables to rewrite the destination IP (DNAT) on locally-generated packets to 169.254.169.254 by rewriting the packets in the OUTPUT chain. Just make sure to add this to the boot process of the base image/template. Remember that the --to-destination address will change depending on the host/vm, thus the output of iptables-save will not work universally.

You can read more about the iptable rules used here: http://www.karlrupp.net/en/computer/nat_tutorial

Here’s the one-liner that works for me:

iptables -t nat -A OUTPUT --dst 169.254.169.254 -p tcp --dport 80 -j DNAT 
  --to-destination $(cat /var/lib/dhclient/dhclient-eth0.leases | grep dhcp-server-identifier |tail -1|cut -d' ' -f5|cut -d';' -f1):80

Use Command Output for Jenkins Parameters

Sometimes it’s nice to be able to pull parameters from the output of commands or scripts. No plugin in Jenkins directly supports this, however, using the Jenkins Dynamic Parameter Plug-in it is possible.

Here’s a simple Groovy script that will convert each line of output from a command (e.g. ls -1 /tmp/) into an option

def command = """/bin/ls -1 /tmp/"""
def proc = command.execute()
proc.waitFor()
def list = proc.in.text.readLines()

Download the plugin here: https://wiki.jenkins-ci.org/display/JENKINS/Jenkins+Dynamic+Parameter+Plug-in

Generic RPM Spec File for RubyGems

I find myself building a lot of RPMs for rubygems in order to satisfy requirements for non-ruby applications. With very little modification, you should be able to use the below to create an RPM of many RubyGems.

Update the sample .spec file replacing value of gemname with the name of the gem. Also update the version in the value of gemversion with the actual version of the gem.

Usage

Download a gem:

gem fetch gemname-1.0.0.gem

Then build an RPM from it:

rpmbuild -ba rubygem-gemname.spec

Gotchas:

  • You might not need an initrc script additions that I have in the file. Comment out what you don’t need using (#).
  • The sysconfig related lines are only really useful if using an initrc script

[gist id=3191882]

Route53 DNS Round-Robin CNAMEs with Weighted Sets

According to RFC1034, CNAMES cannot reference multiple canonical hosts in the RDATA section. In other words, you can’t have the CNAME “www.domain.com” alias (or resolve to) “www1.domain.com” and “www2.domain.com”.

This sucks because you may want to use DNS for round-robin (RR) load-balancing across CNAME records much like you would for A records. The good news is that Route53 supports “Weighted Sets” which allow you to do round-robin-eque load balancing between multiple CNAMEs. For example, you might want to distribute your load evenly between multiple AWS regions like us-west-1 and us-west-2 that each contain an Elastic Load Balancer (ELB).

The other possibility is you cannot use an ELB because a particular service is private. Remember, services behind an ELB are internet accessible from 0.0.0.0/0; ELBs themselves cannot be controlled by security groups.  Using A records, one can do DNS-RR to either the public or private IP address on an instance, but not both (it just wouldn’t make sense). The drawback here is that you cannot use the same hostname for services that are inside Ec2 as services that are outside of Ec2. If instead, you use a Weighted-Set with a short TTL (e.g. 10 seconds), you can create a CNAME that points to the CNAMEs of each of the EC2 instances. Because the CNAME points to another CNAME, when it’s resolved inside of EC2 a private ip address is returned; conversely when it’s resolved from outside of EC2 a public IP address is returned.

The reason it’s important to return either a public or private IP address is due to the way EC2 security groups work. When an ingress rule is granted from a particular security group (e.g. allow “Group A” to access “Group B”), it only applies to the private IP addresses of instances in “Group A”. This means that when instances in “Group A” need to access instances in “Group B”, they should use the private IP address of instances in “Group B”. Using the CNAME of an EC2 instance ensures you’re always connecting to the most appropriate IP address whether your inside or outside of EC2.

 

 

 

 

Building Varnish Debian/Ubuntu Package From Souce

Varnish-cache.org is nice about providing a lot of binary packages for most distros/releases, but not for all architectures. As a result, you may find yourself needing to build the packages by hand. Here’s how I went about it for Varnish-2.1 on an old Ubuntu release.

Add the Varnish apt repo to your {{{/etc/apt/sources.list}}}:
{{{
deb http://repo.varnish-cache.org/ubuntu/ hardy varnish-2.1
deb-src http://repo.varnish-cache.org/ubuntu/ hardy varnish-2.1
}}}

Add the Varnish distributors GPG key:
{{{
curl http://repo.varnish-cache.org/debian/GPG-key.txt | apt-key add –
}}}

Update your repo cache:
{{{
sudo apt-get update
}}}

Download & compile source:
{{{
sudo apt-get source varnish
cd /usr/src
dpkg-source -x varnish_2.1.2-1*.dsc
cd varnish-2.1.2 && dpkg-buildpackage
}}}

Then, in {{{/usr/src/}}} you’ll find your new .deb packages ready for installation.

Using MacPorts on OSX Lion

After upgrading to OSX Lion, my MacPorts installation failed. I upgraded to OSX XCode 4.1, and then console compilation stopped working. Apparently, by default the “UNIX Development” tools and “System Development” tools are not installed when installing via the App Store. Fortunately, there’s an easy (time consuming) fix. Simply re-run the “Install XCode” app in your Applications directory. It’s easily located by using Spotlight.

Warning: port definitions are more than two weeks old, consider using selfupdate
Warning: Xcode does not appear to be installed; most ports will likely fail to build.
Error: Unable to open port: can’t read "build.cmd": Failed to locate ‘make’ in path: '/opt/local/bin:/opt/local/sbin:/bin:/sbin:/usr/bin:/usr/sbin' or at its MacPorts configuration time location, did you move it?
Error: Unable to execute port: upgrade gettext failed

'xterm-256color': unknown terminal type.

After upgrading to OSX Lion, I started getting this error on certain (Debian/Ubuntu) servers. The fix is simply to install the “ncurses-term” package which provides the file {{{/usr/share/terminfo/x/xterm-256color}}}.

Porting Fedora RPM Spec File to CentOS Error: find: invalid predicate

I’ve been porting many spec files from fc15 to el5. In the process, a common error that is encountered is:
{{{find: invalid predicate `’
error: Bad exit status from /var/tmp/rpm-tmp.63066 (%install)
RPM build errors:
Bad exit status from /var/tmp/rpm-tmp.63066 (%install)
}}}

The fix is almost always that {{{BuildRoot}}} is not defined in the spec file. Defining one near the top of the file will resolve the issue.

{{{BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-buildroot }}}

Symfony: Configuration file does not have a registered handler

I was getting the following error in an older Symfony 1.2 project. Running the standard {{{symfony cc}}} didn’t resolve the issue as the symfony command itself relied on files in {{{cache/}}}. Turns out that after some files had been moved around on the filesystem, which made symfony unhappy. A simple {{{rm -rf cache/*}}} fixed the problem.

{{{
Configuration file “/opt/symfony/1.2/lib/config/config/settings.yml, /opt/symfony/1.2/lib/plugins/sfProtoculousPlugin/config/settings.yml, /vol/svn/cms/plugins/sfJqueryReloadedPlugin/config/settings.yml, /vol/svn/cms/apps/frontend/config/settings.yml” does not have a registered handler.}}}