The Streema Blog

At Streema we like to develop at a fast pace.  One of the things that allows us to do that is to have an automatically tested codebase. We want to always be shipping and having our apps tested gives us some degree of confidence that we won’t break it all each time we commit a change.

But running tests takes time. A few months ago our unit tests were taking over 12 minutes to run. Each time someone wanted to run the tests, the situation was pretty much like this:

As you might recall, we use Python/Django. And our problem was the use of fixtures in our unit/functional tests. As @carljm said on his “Testing and Django” talk for PyCon 2012, they are slow to load, hard to maintain and increase test interdependence. We had huge fixtures, 1mb fixtures, which loaded thousands of models and they were used on dozens of tests!  With a little bit of patience we were able to remove them all and replace them by using factory boy.

The result:  we reduced the test run time from over 12 minutes to under a minute.  Yes, a single minute.  =)

Stay tuned for more news and insight on the technical details of Streema. We might have an interesting thing or two to share!

Stay tuned!

The Streema Team

Starting late last year, at Streema we’ve been working hard to improve the website’s uptime and performance – nothing is more annoying than a tech service or device when it doesn’t respond, right?

So we wanted to give you guys a quick update on this, but before we start, though, two important definitions:

Response Time: The interval between a request that enters and the answer that is received. In this case it’s the page load time, i.e. the time to retrieve the HTML page (without images, stylesheets, JavaScript, etc.). So, although it could be below a hundred ms (milliseconds), as a user, it could translate into seconds to load a complete page.

Downtime: It is the period when the system is unavailable, i.e. the period you wait, mostly doing nothing, and if it goes above what you expected, you start hitting your computer or anything around you, in addition to shouting out phrases such as “come on!”, “ahhhh” or others that we can’t publish right here. :)

So, we’re happy to share with you that we’ve reduced our Response Time almost by a half and the number of Downtimes by about 75%. Even better, some of the Downtimes we had before took hours, now they are less than 10 minutes – additionally, now they only happen because of our own updates to our site and not because of infrastructure problems!

For those techies out there, here’s a quick summary of what we’ve been doing to get results:

• Increase the number of servers and balance the load with Nginx.
• Improve the database setup to take advantage of the servers memory.
• Rewrite many SQL queries, specially the ones that requested temporal tables on the hard drive.
• Change the configuration of the Apache modwsgi and increase the number of processes to serve requests.

Finally, regarding the tools we’ve been using, the main ones were: Apache’s Server Status, Munin, New Relic, Percona Tools and Pingdom.

We’re planning to continue developing our performance infrastructure – and we hope you continue visiting Streema. :)

Stay tuned!

The Streema Team