Loggly

How to use RightScale APIs with Python

Posted 17 Mar, 2010 by raffy@loggly.com in Code and Log Management

I have been quiet for long enough on this blog. It’s time for me to share some things that I learned in the last few months while I was working on Loggly’s Application layer. Lately, I spent some quality time with Django and consequentially Python.

What I want to focus on today is our integration with RightScale. At Loggly, we use RightScale to manage our AWS instances. Loggly runs three types of servers. (Well, I am simplifying). We have a proxy tier which receives your log messages. The proxy tier, which is basically a bank of machines, forwards the messages to the indexing back end that runs Solr. The third group of machines are the Web or application servers. When a new proxy box comes online, the RightScale management interface knows about the box. I had to know about thse proxies on the application tier (i.e., within Django) as well. How do you do that?

The first solution would be to have the proxies register with Django, as soon as they get online. What happens though when they go down or are taken offline? Seems complicated to keep track of that. Another solution would be to periodically poll the proxies from Django. Not very nice either.

My solution is much more elegant. RightScale has two features that helped me out. The first one is machine tags. Each proxy server is labeled as such. (See Machine Tagging). Secondly, I am using the RightScale API to figure out how many proxies I have and what their IPs are. (As a side note, the RightScale APIs are in Beta right now. There might be changes or improvements coming down the pipe.)

I struggled for quite a bit with using the RightScale APIs out of Python. Here are some things that I learned the hard way and you might find helpful:

Using the API to query all your machines in a specific deployment:

curl -H 'X-API-VERSION: 1.0' -u [user@domain.com]:[password] \
https://my.rightscale.com/api/acct/[account]/deployments/[deployment_number]

Note how you have to add the extra header to request version 1.0 of the API.

Here is how you get all the machines that have a specific tag. Note the structure of my tag! I set role:proxy=true. You need to use this hierarchical model!

curl -H 'X-API-VERSION: 1.0' -u [user@domain.com]:[password] -d'resource_type=server' \
-d 'tags[]=role:proxy' https://my.rightscale.com/api/acct/[account]/tags/search.js

Want JSON output instead of XML, add “&format=js” at the end of your request!

Now, from the response, you would think you could just use that HREF to query an individual server. Wrong. That doesn’t work. You have to add “/settings” in order to make that work:

curl -H 'X-API-VERSION: 1.0' -u [user@domain.com]:[password] \
https://my.rightscale.com/api/acct/20184/instances/[instance_id]/status

Here is how you set a tag on a server: (Note: If you change the tag in the user interface for a running server, it will not take effect. Only if you start a new server of that type, will the tag be there. Unlike the API call, where you can set a tag on a running machine).

curl -H 'X-API-VERSION: 1.0' -u [user@domain.com]:[password] \
-d 'resource_href=https://my.rightscale.com/api/acct/[account]/servers/[server_id]' \
-d tags[]=role:proxy=true https://my.rightscale.com/api/acct/[account]/tags/set

The part I struggled with most was how to call the API from within Python. Turns out httplib2 expects the Web server to respond slightly different than the RightScale server is. If you are using the following code, you will not be able to connect:

h = httplib2.Http()
h.add_credentials(user,password)
response, content = h.request(url, headers=headers)

httplib2 will connect to the Web server without sending the credentials. Only if the server challenges the client to use auth, it will then send the authentication headers. And this is precisely what RightScale is not doing. Therefore, you have to do the following in order to include the authentication headers in the first request already:

h = httplib2.Http()
import base64
base64string = base64.encodestring('%s:%s' % (user, password))[:-1]
headers['Authorization'] = "Basic %s" % base64string
response, content = h.request(url, headers=headers)

Credentials are an interesting topic. I ended up creating a separate user in the RightScale interface that I am using for the APIs. Don’t be fooled though. These credentials still let that user log into the Web interface. I hope that RightScale will add a capability such that I can have a user that can only use the API.

I hope this helps you getting off the ground a bit quicker when using RightScale. Let me know how it goes. You can also find me on Twitter: @zrlram

Tweet