RockeTier Agile Development Methodology

Hi,

We at RockeTier, believe that almost any software system can do better by changing and modifying only a small portion of existing code base.
We proved in several cases that using this methodology you can gain major business value in short time. or in other words Agile.

Therefore, it was only a matter of time before we migrate our development team to Agile methodology (you guess right, our software team develops high load software systems which process hundreds of millions of events per day for our clients). This change enabled us providing business value in shorter time frames.

Our development methodology is based on the following basics:

1. System wide design - the product owner is responsible to provide long term road map for the system, including software architecture, database architecture, middleware, non functional requirements, timeline and so on based on inputs from the client or internal product manager. This is an important component which sometimes Agile evalgelists tends to neglect.

2. Product backlog - product owner is responsible for breaking the long term architecture into business processes. Each business process is analysed, confirmed by the client and used as an input for the sprint backlog. Many times business processes are not complete, but describing current days needs in order to gain business value, knowing that these requirements will be changed in the future.

3. Sprint backlog - each business process is broken into a list of tasks (usually by the programmer). This list is documented in a central repository and being written on the team whiteboard.

4. Daily Cycle Status Review- each day a peer meeting is being handled on 10AM, syncronizing open issues with closed issued, adding high priority tasks (usually bugs)

5. Daily Version - each day on 4PM we upload a new "for review" version which includes all the commited new features added to the software in the last 24 hours.

6. Weekly Version - based on the last week "for review" versions we upload a new version on Wednesday 12:00, enabling our team and the client examine the system before weekend.

7. Quality - We believe that people achieve best results when they are responsible for their products. Therefore, most QA is done by the programmers themselves and the team leader:
- Peer review and pair programming - we often use these methods when developing sensitive components and when time is short, to achieve best quality in short time, and in order to reduce the risks.
- Code review - done by the team leader before a new feature is committed to the SVN
- Code management - we use SVN as company standard, but also consider to use GIT, in order commit (locally on your computer) every change you make that way you can return to any point in time but do not break everybody's code if your changes are not complete. and only when code is ready, commit your changes in svn.
- TDD - we use test driven development to make sure that code is not broken between cycles, and making sure that new changes will not harm current business processes

8. Knowledge management - knowing that sharing a knowledge will help our people gain best results, we opened few weeks ago a new code snippets and best methodologies blog. This is an open blog by nature so the community can enjoy our insights and experience.

We seen a great improvment in our development products since implementing this method including: reducing the needed time to get new people into productivity, reducing error and misunderstandings, reducing time to market and reducing bugs,

Keep agile,
Moshe, RockeTier. The performance experts.

Amazon Web Services Meetup Tel Aviv

Hi,

There is still room at the next Amazon Web Services Meetup Tel Aviv. It will be held in the Tel Aviv univerisity

The schedule is:
5:00 - 5:30 Networking
5:30 - 5:45 Opening remarks, Prof. Mel Rosenberg
5:45 - 6:15 Amazon Web Services Overview, Martin Buhr
6:15 - 7:00 Customer Panel
7:00 - 7:15 Open question and answer session
7:15 - 7:30 Closing Remarks, Martin Buhr
7:30 - 8:00 Networking


See you there
Moshe,
RockeTier

Secure your Amazon EC2 Database

Hi,

An important issue that must be addressed in every system, and for sure if it is an internet based, is security. When we build high scalable systems, we face this issue time after time.

Therefore, when you install a new system in Amazon EC2, with a MySQL in it, you should consider how to secure and restrict the access to the system and the database.

Amazon provides set of solutions in order to support these needs:
1. Firewall: Amazon EC2 provides a firewall solution. This inbound firewall is configured in a default deny mode and you must explicitly open any ports to allow inbound traffic. The traffic may be restricted by protocol, by service port, as well as by source IP address (individual IP or CIDR block). More can be found here.
2. Security Groups: Security group is the easiest way to aggregate several EC2 servers based on rule, access needs and functionality: Database, application, web, etc (if you are familiar with VLAN segments and FW legs, it's very similar indeed).
Every server is associate at start up with a security group, so access to this server is restricted based on the group definitions. This can be done by defining FW rules between Amazon EC2 security groups and between a security group and the world (for example HTTP may be enabled from the world to the web servers security groups, while the database will enable access in 3306 to the application servers security group and your company static IP)
3. Anti DDoS, IP spoofing and other issues are addressed here

So how should I start?
1. Define you servers roles and the security groups. Tip: usually it should be the same as your AMI
2. Define the FW rules between the groups and the logical rules between each other, the world, your secured office connection (static IP is recommanded) and every other relevant 3rd party.
3. Define the security groups in Amazon (see ec2-...-group RTFM)
4. Define you FW rules. see more here to learn the exact syntax and in the ec2-authorize RTFM

Best,
Moshe. RockeTier, the performance experts

Online Games Industry Software Perfromance Boosting Case Study

Hi,

My colleague, Romi Kuntsman, refered me to Big Fish Games software performance boosting case study that was published not long ago by MySQL (now SUN).

MySQL is a significant player in the internet based systems niche which includes the online advertisement, SaaS, games and gaming and many others. The internet based systems were not long ago considered as not so complex systsms as the enterprise systems.
This assumption is not longer true. Internet based system and mostly the top players in the market are facing challenges that require non conservative solutions. Players such as Facebook, Google, Big Fish and CJ are facing the challenge to handle millions of events per second. This is significantly larger than most enterprise software and n-tier architectures based systems were designed to handle with.

Therefore, MySQL based systems that were considered to be a low-mid range systems, are facing these days the most challenging requirements of traffic and load.

MySQL just released a good example for that (I will present several other cases we dealt and dealing with in future posts): BigFishGames.com is a fast-growing website with over 25 million unique customer accounts and over 2.5 million visitors per month and it well financed (over $80M in the last financial round).

Due to their high growth rate (100%-220% per year in last few years accurding to TechCrunch replys), Big Fish faced performance issues, although they used over 40 different MySQL servers...

What did Big Fish do?

1. Database Profiling and Tuning:
Big Fish DBA team used MySQL Query Analyzer. This product provides a consolidated view of query activities and execution details, and it enables quick identification of poorly running queries and tackle the root causes directly in the SQL code. The DBA team caught a "bad" query running 400K times overnight which never showed up in query logs. Since the Query Analyzer uses a Service Agent listening to application queries and performances metrics, the MySQL servers can always be live and operational when being analyzed. There is no need to switch the servers back and forth between on-line and off-line, which eliminates unnecessary risks to server availability and reliability. Moreover, it enables the DBA team gain a real time statistics, rather than make the analysis based on test environment simulations.
After deploying the MySQL Query Analyzer, Big Fish Games tripled (X3) its database performance within three days.

2. In Memory Database and Distributed Cache Layer
Big Fish Games used Memcached as an in memory database soluton and distributed caching layer. By using Memcached, all online queries are performed in the application server memory, and no online queries are processed in the database. Once in a while the Memcached is being syncronized with the database. This solution improved the end user response time, system scalability and database performance.

3. High Performance Servers
Big Fish Games chosen Sun Fire x64 Servers (not so surprising since MySQL is owned by SUN) in three different architectures:

• Sun Fire X2100: for applications which require lots of local disk space but less I/O or CPU speed
• Sun Fire X4100: for applications which demand fast processors but don't need speedy local disk I/O
  
• Sun Fire X4140: for applications where faster local disk I/O via RAID 10 and battery backed up write cache is essential (Memcached servers)
  

Right selection of the hardware enabled 20x in performance by merely replacing an X4100 server with an X4140 machine (and probably they paided for this selection)

I think that this case study is a great example of how a poor performance system can be boosted in factors of X100 and more. Moreover, these cases are not so rare. Many players in the market gain success, and require better performing systems at the fraction of the cost of legacy enterprise systems, without major redesigning their existing servers, and while keeping their current code base.

Best Regards,
Moshe,
RockeTier, the performance experts

The World Summit of Cloud Computing - Presentations and Impressions

Hi,

The IGT just released the presentations and videos of all the two days lectures taken place in the IGT2008 - World Summit of Cloud Computing. It was a great event with great lecturers and presentations.

You can find my lecture, both slides and video. My focus in the lecture was presenting several aspects from our cloud computing field experience and strategic aspects we have in this fields including: market current status, market niches (IaaS, PaaS and SaaS), how to avoid vendor lock-in and how to establish SLA.
We also discussed why enterprise should get into the market (Who will be the next financial giant after Citi age? maybe Paypal or Google?)
We presented how cloud computing enables enterprises gain equity and Op-Ex reduction, and why software performance boosting is a major requirement for organizations that move into the cloud and want to keep their Op-Ex low.

I would like to recommand also several other facinating lecutres like:
* Stevie Clifton from Animoto with their amazing cloud case study: scaling from 50 CPUs to 3500 CPU to meet the digged effect (we have a client who is going to be the next story in this field, a story for another time)
* Dr Owen O'Malley from Yahoo! and Apache Hadoop with a great lecture on Hadoop and test cases (it is really a great tool, and we already using it)
* Paul Strong with a not so short presentation, but with a great discussion about eBay (did you know that eBay reaches 150 Billion events per day) and the requirements of large enterprise from the cloud.

It was a great event and we hope to see you next year in the IGT 2009,

Best,
Moshe,
RockeTier. The performance experts.

GPU Cloud - VMware moves ahead?

Hi,

Since I wrote about GPU cloud a week ago, I met another two software companies which are interested in this issue.

Is there any change in the foreseen future? The following post by Michael Larabel reveals a possible answer: "Tungsten Graphics has been acquired by VMware. Open source graphics technology development will continue as part of VMware's engineering team". VMware is the leading virtualization player in the market (but weaker in the cloud market) and this move will probably lead to "improving the 3D support within virtualized environments".

Probably, a move of a key technology player will enable cloud service providers provide better GPU support in the future.

Moshe,
RockeTier

RockeTier Annual Brunch

PLEASE NOTICE THAT DUE TO WEATHER CONDITIONS, THE EVENT WAS POSTPONED

Hi,

Performance and Cloud Computing posts will be updated soon, in the meanwhile:

RockeTier is having its annual brunch next week on Friday (26/12), and you are welcome to join us. The event will take place Tel Aviv, Israel on Friday the 26/12 at 11:00. Look forward for good food, good booze and great company, More details at Facebook. The favor of a reply is requested by 21/12/2008

P.S. It's also Moshe's birthday :-)

Yours Truly.

Moshe and the RockeTiers