Developing Code, Connecting Industry

For most small applications, scalability is usually not something that recieves much consideration. For applications that have potential to grow to tens of thousands of users and up, however; scalability may eventually become a concern.

Many web application success stories owe their scalability to the fact that they are written in Python/Django. It is extremely light-weight, as web application architectures go, and allows for much flexibility. Some of the things that we’ve kept in mind while coding that help with scaling are:

• Minimize external dependencies
• Replace/refactor/migrate components and modules as they become problematic (Python components help to streamline this process)
Emphasize ‘low coupling’ of code bases
• Attempt to localize and modularize failures (try and prevent them from spilling into other modules/applications)
• Limit the number of queries within loops (object.get(), object.filter(), etc.)
• It is much more efficient to fetch all records necessary in one query, and work with the retrieved dataset within a loop, rather than querying once per loop.
• Get rid of all obviously unnecessary leaf services.
• Customize reliable open-source software – bend it to your will.
• ‘psyco’ compiler – specialized Python compiler – extremely optimized
• Processor-heavy functions, or highly-executed functions, can be ‘psyco-ized’
• This compiler is something we have not yet tried using, however some development companies report as much as a 400% performance boost by using it properly.

A few rabbit-holes that developers should avoid running down if at all possible:

• Always be aware of the difference between “fast” and “fast enough”
• Python/Django has many scalability optimizations built right in; implement your functionality and test it under appropriate load-testing environments first, before spending too much time optimizing manually.
• Strive for hardware efficiency, but do not obsess over it
• Do not make the assumption that a technique for code optimization for one language will work for the language you are working in.
• Keep in mind that eventually you will have ‘no cards left to play’.

By: Brett McClelland