What I Learned This Summer

Back in June at the beginning of the summer I was performing some significant load testing on my companies major MMO to discover a lot of issues with Microsoft's Windows Azure.  Specifically relating to the shared services and their throttling/limiting!

This was quite interesting – when I performed steady growth load testing, I didn't encounter any issues, a part from some requests starting to take a bit longer.  This was due to the fact that my database queries started to queue up a little bit.  However, most cases did not cause requests to fail, just respond slightly slower.

Burst testing was a much difference story.  In the steady growth load testing, I was slowly growing my traffic to over 5,000 concurrent users by having 2 users join per second (separated by 0.5 seconds).  While doing my burst testing, I was trying to send over 500 users per minute (only 4 times the steady growth testing) and immediately found that the database queue grew so quickly that it could never recover and many requests began to timeout.

This is of course is not too surprising.  Databases are a typical bottleneck in many applications.  Now that I had my benchmarks and knew the first bottleneck, I began looking to implement a caching solution.  At the time on Azure, this only meant a shared AppFabric instance.

Once the code was done, I performed the same load testing.  This time the results were much, much worse!  Yes, not better, but worse!  I was quite surprised by this.  Unlike the database, when I reached the limit with AppFabric, my requests were simply blocked and returned an immediate error.  This caused significant problems because we have multiple systems talking to each other and if one layer has failures with the caching, the edge cases to safely handle this quickly became out of control.

At the end of the day, we ended up dropping the idea of using Window's Azure AppFabric caching!

Shortly after this, I thought it was imperative to perform these similar tests on Azure's biggest rival – Amazon.  The results were quite interesting.  Because the Amazon RDS database is not shared, but dedicated, I was immediately able to see a huge increase.  The steady growth testing was able to handle over 6 users per second without issue, compared to the 2 users per second that eventually began to cause slow responses as the database queue slowly built up.

The burst testing was a much different story as well.  I'm pretty sure that I read that SQL Azure is only single threaded causing a huge performance drop compared to a standalone version of SQL Server.  Amazon's RDS instance out performed SQL Azure significantly.  I was able to increase from 500 users a minute to over 1,000 users per minute and even reach over 10,000 concurrent users without any significant speed drops.

Due to resource limitations, we are still exploring moving from Azure to Amazon, but based on these results, I see no reasons not too.  Furthermore, when testing SQL Azure directly against Amazon RDS we found that one minute SQL Azure would be able to handle over 100 requests per minute, then the next minute only handle 25.  Meanwhile, Amazon was able to consistently handle over 1,000 requests per minute!

 

Next came July, while July was actually a bit of a right off for me and my family.  Unfortunately the whole family got sick and we were out of commission for 2 weeks, then it took almost another 2 weeks to recover.

The one thing that did come out of July is how difficult it is to find good employees to hire.  In the month of July I must have had over 30 interviews and was only able to hire 1 junior.  I think my favorite worst answer was when I asked what experience with No SQL databases do you have and the answer was "Oh I'm quite familiar with No SQL".  What No SQL database is your favorite or most used?  "No SQL".  Wow, you mine as well just leave right now!

 

Finally August came.  This was an interesting month for me.  I personally spent a lot of time working on an on-going side project.  Before I got started, I was really stuck on which language and framework to choose.  There are so many options and within those options so many things I like and dislike about each…

The big thing I learned at work was, looking back somewhat obvious, but worth the research is, that threading just for the sake of "threading" isn't always going to improve performance!  There are of course many benefits to threading, but sometimes the cost of creating a thread can be worse than just running the processes one after the other.

In the scenario I was exploring, it was about improving the response time of a JSON REST service.  The function was performing a variety of different actions – in of course a linear form – that didn't relate to each other.  This of course seemed like a good idea to thread and run them all at the same time.  However, once implemented, it was actually slower…not all of the time, but sometimes, depending on the time required to actually create the new thread to perform the process!

The lesson learned here of course is obvious; only thread long running processes not simple processes even if there are several that execute one after the other.

While that was happening at work, at home I decided that I didn't want to invest the time to learn something new, so I went back to PHP as I've being focusing a lot on .NET recently.  Boy what a treat this was!  I forgot how nice and fast it is to work with a weakly typed language while developing.  Of course, I'm sincerely missing the ability to create a static class and have it actually live in memory past the lifetime of a single request!  There are ways around this, but for the time being, leveraging the session will be sufficient.  If I ever need to scale across servers, it's extremely simple to configure PHP sessions to store to a shared environment like Memcache…

As to what framework I chose, I actually decided to go Vanilla PHP.  Why you ask?  The reasons are quite simple; the project has simple UI elements that only get updated when a user performs a specific action.  This action is completed via AJAX and rather than redrawing the whole UI, 9 times out of 10, I'm returning JSON data.  This JSON data is then parsed in Javascript and I'm strongly leveraging jQuery template to display the data in a pretty format.

Rather than choosing my old faithful – CakePHP – I thought that as part of my learning curve I could explore the complexities of creating my own simple framework.  I'm still strongly leveraging the MVC paradigm as I've always been a big fan of it.  The only difference is that I'm removing a lot of the overhead of a fully built framework meant for the average website opposed to the highly optimized and efficient results that I am looking for.  Don't get me wrong, I still would use CakePHP in a heartbeat for any standard website…

As you may have seen from recent blog posts, this has been extremely useful and I have been able to leverage removing pesky require or include functions to both speed up development and performance.  As I continue to progress with this side project I will hopefully have more tidbits to share.  Don't ever look for me to release a framework though, I'm not sure I would ever want to take on such a monumental feat as that – much respect to the ones that have successfully done and are still doing it!

The remainder of the year should hopefully be equally as interesting as my family is about to grow in November.  Hopefully I will continue to have time to learn and grow as a developer while my kids learn and grow as well.  Only time will tell of course.

Cheers.

Photo courtesy of CubaGallery

Published on Sep 7, 2012

Tags: Optimization | cache | ASP.NET MVC and Web API Tutorial | Theory | PHP