Posts tagged python

How to Mask Command Line Output from the Python run() Method

June 10, 2009 · Filed under Point2 - Technical · Tagged , , , , , , , , , , , ,

Recently, We were tasked with creating automated deployment for our Python Django project. For our purposes, this involved creating python modules for programatically logging onto servers and carrying out all the necessary deployment tasks. We decided to use Fabric to make this work. We encountered difficulty, however, when using the run() method to execute commands in remote environments.

The problem is, with every command that Fabric executes on the remote machine, the command itself is always echoed to standard-out when the run() method executes, displaying (in plain text!) any password arguments you may be passing to the command being executed.

We worked around this by “monkey-patching” stdout itself with a wrapper class that checks specifically for the password provided, and replacing any output to standard out matching the password with something else (like “****”).

The wrapper class looks something like this:

class StdoutWrapper:

  def __init__(self, stdout, mask):
    self.stdout = stdout
    self.mask = mask

  def __getattr__(self, attr_name):
    return getattr(self.stdout, attr_name)

  def write(self, text):
    if text.find(self.mask) > -1:
      self.stdout.write(text.replace(self.mask, '****'))
    else:
      self.stdout.write(text)

Then, when you want to execute the command whose output you wish to mask, you simply use the StdoutWrapper class like so:

import getpass.getpass, sys

def some_fabric_method():
  config.username = prompt('Enter Username:')
  config.passwd = getpass('Enter Password:')
  config.app_path = '/path/to/your/app'
  config.svn_url = 'http://some.svn.checkout/'
  out = StdoutWrapper(sys.stdout, config.passwd)
  sys.stdout = out
  try:
    run("svn revert -R $(app_path")
    run("svn switch --username $(username) --password $(svn_password) $(svn_url) $(app_path)")
  finally:
    sys.stdout = sys.__stdout__

Obviously, this is a less-than-ideal solution; it accomplishes what we want, but having to resort to temporarily monkey-patching standard out itself seems like overkill. Has anyone else out there found a more preferable solution for suppressing plain-text password output while using Fabric for Python?

By: Brett McClelland

Leave a comment »

High Ceremony doesn’t have to mean High Cycle Time

April 27, 2009 · Filed under Point2 - Technical · Tagged , , , , , ,

Oftentimes languages such as Java are called “high ceremony” languages compared to languages like Ruby or Python. This refers to the fact that there’s generally a bit more plumbing involved in firing up a Java application – particularly a web application – than there is with the scripting languages.

Of course, Java is compiled (to byte-code at least), so it’s not quite a 1 to 1 comparison with a more interpreted language such as Ruby, but still, even in a “high ceremony” language it’s important not to get too high a “cycle time” for developers, IMO.

By “cycle time” I mean the time between making a change and seeing it working – either in a test, or, ideally, in a running application. Most modern IDEs made the cycle time for tests pretty darn low (and great tools like Inifinitest can take all the manual work out of it, no less), but to see a running application and be able to exercise your changes deployed in a container is a bit more of a grind.

That’s where a tool like Jetty can come in handy. Jetty is a lightweight web app container that can be easily added to your development cycle in place of a heavier-weight solution to allow you a faster cycle time, and, often, greater productivity and interactivity.

Especially in combination with it’s integration with Maven, Jetty can get your app deployed far faster than with other solutions. For most webapps, it’s just a matter of saying:

mvn jetty:run

And you’ve got a container up and running with your app in it within a few seconds.

Jetty can even do a certain amount of “hot update”: modify a JSP (or even some code – although there are limits) and the running webapp is updated, and you’re able to test, edit… cycle away without the painful wait for a deployment any more often than necessary.

You can pass required system properties to your app via maven’s -D mechanism, and they’ll be available to your app:

mvn -Dsome.property=someValue jetty:run

And even control the port your application binds to on the fly (or via the handy jetty.xml file if you want to set it more permanently).

Jetty and maven also give you the ability to easily script, for example, if you need to run a test utility on your running webapp to ping a series of REST calls, for example, you can:

mvn clean package # Build the webapp
mvn jetty:run & # start jetty, spawning it in the background
java -jar mytestutility.jar # Run my test jar, which pings the URLs for all my rest services, maybe does performance checks, etc
mvn jetty:stop # Stop the jetty instance we fired up in the background

Lightweight containers such as Jetty are just one way to help crank down the “cycle time” for developers, of course. Some other possibilities I’ll leave for a later entry.

By: Mike Nash

Leave a comment »

Diving into Python

March 9, 2009 · Filed under Point2 - Technical · Tagged , , , ,

Our company is currently ‘diving into Python‘ which as a developer makes me extremely happy. I can see diversity is emerging in our development arena.

But this post is not so much about Python as it is about disseminating knowledge across our company, i.e. how do we make sure that our organization learns technology X or methodology Y ? If you have been in this industry for a while you must have heard this sort of answer from your boss quite often: “Make them learn that stuff at home!” or “Buy them a few books so that they can read about it at their own pace!” or the classic: “Let’s get a Knowledge Base” or “Let’s get an expert on the topic”.

Although these approaches may work for you, they have never worked for me as well as what my new company has in place. At Point2, one way to disseminate knowledge and make sure that everyone is on the same boat, is using a workshop schema whereby every Friday we run two or three workshops in parallel that last for 1 or 2 hours depending on  presenter and topic. Most of them are so good that presenters have to repeat them the following week for the audience who could not attend some workshops. To give you a taste of what these are, this month we are running the Python series. The following four workshops are aligned:

  • Python 101
  • Django 101
  • Django to interface REST services
  • PyGame

Hopefully out of this series we’ll get some more Pythonistas in our company ;-)

It would be interesting to know what other things worked for our readers, besides workshops.

By: Marcos Tarruella

Leave a comment »

Stubbing a Test Mail Server

February 17, 2009 · Filed under Point2 - Technical · Tagged , , , , , , , , , , ,

Django has built in support for sending email. We make use of this in our app, but when testing, we wanted to be able to access the emails sent so we could assert on their content, and pull data out of the body. In the unit tests, that’s easily solved by mocking the email client call, but we wanted to do this as a black-box regression test. That’s where the Python built in smtpd.SMTPServer comes in handy:

import smtpd, asyncore, threading

email_server = None

class FakeServer(smtpd.SMTPServer):

def __init__(self, localaddr, remoteaddr):
self.server = smtpd.SMTPServer.__init__(self, localaddr, remoteaddr)
self.emails = {}

def process_message(self, peer, mailfrom, rcpttos, data):
for recipient in rcpttos:
idx = recipient.replace(‘@’,'_at_’).replace(‘.’,'_dot_’)
existing_emails = emails.get(idx, [])
existing_emails.append(data)
emails[idx] = existing_emails

def stop(self):
self.server.close()

def main()
email_server = FakeServer((‘localhost’, 10920), None)
def start_email():
asyncore.loop()
thread = threading.Thread(target=start_email)
thread.setDaemon(True)
thread.start()

if __name__==’__main__’:
main()

This starts an email server on port 10910, which on receipt of a new email, stores it in a dictionary keyed by a modified version of the email address. Multiple emails to the same address get appended to the list. There is no reason in this context for replacing the ‘.’ and ‘@’ symbols from the email address in this context, but it was appropriate for our usage.

We run the stub mailserver on a separate box, and make the emails available via a get request on a test HTTP server, which allows us to run tests scripted entirely written in Selenium. I will blog again with details of how we manage our test HTTP server. You can use the this code as the basis for embedding the mail server in a unittest setup too.

By: Chris Tarttelin

Comments (1) »

Creating Fixtures from Within Tests

February 3, 2009 · Filed under Point2 - Technical · Tagged , , , , , , , ,

Django gives you a ‘dumpdata’ target which will create a fixture from all the records in your schema. For what we wanted, this was overkill. We had an existing unit test which was creating data in one test, and it was just the right amount of data for what we wanted. After searching through the Django codebase, it became clear that we could pass the objects we had in that test straight to the JSON serializer, and write the output to file. This ended up looking something like this:-

from django.core.serializers import json
serializer = json.Serializer()
objectsToSerialize =
MyModel.objects.filter(column='restriction')
with open('my_fixture.json','w') as f:
. f.write(serializer.serialize(objectsToSerialize, indent=4))

And that’s it! You can also spin up a shell, using:-

python manage.py shell

and load the data you want to create fixtures from if you don’t have any tests that create the data you want already.

By: Chris Tarttelin

Leave a comment »

Web Application Scalability

February 2, 2009 · Filed under Point2 - Technical · Tagged , , , , , ,

For most small applications, scalability is usually not something that recieves much consideration. For applications that have potential to grow to tens of thousands of users and up, however; scalability may eventually become a concern.

Many web application success stories owe their scalability to the fact that they are written in Python/Django. It is extremely light-weight, as web application architectures go, and allows for much flexibility. Some of the things that we’ve kept in mind while coding that help with scaling are:

• Minimize external dependencies
• Replace/refactor/migrate components and modules as they become problematic (Python components help to streamline this process)
Emphasize ‘low coupling’ of code bases
• Attempt to localize and modularize failures (try and prevent them from spilling into other modules/applications)
• Limit the number of queries within loops (object.get(), object.filter(), etc.)
• It is much more efficient to fetch all records necessary in one query, and work with the retrieved dataset within a loop, rather than querying once per loop.
• Get rid of all obviously unnecessary leaf services.
• Customize reliable open-source software – bend it to your will.
• ‘psyco’ compiler – specialized Python compiler – extremely optimized
• Processor-heavy functions, or highly-executed functions, can be ‘psyco-ized’
• This compiler is something we have not yet tried using, however some development companies report as much as a 400% performance boost by using it properly.

A few rabbit-holes that developers should avoid running down if at all possible:

• Always be aware of the difference between “fast” and “fast enough”
• Python/Django has many scalability optimizations built right in; implement your functionality and test it under appropriate load-testing environments first, before spending too much time optimizing manually.
• Strive for hardware efficiency, but do not obsess over it
• Do not make the assumption that a technique for code optimization for one language will work for the language you are working in.
• Keep in mind that eventually you will have ‘no cards left to play’.

By: Brett McClelland

Leave a comment »