You are hereBlogs / moshe weitzman's blog / Speed up Drupal 7 - MongoDB Field Storage

Speed up Drupal 7 - MongoDB Field Storage


By moshe weitzman - Posted on 23 March 2011

This weekend, I started a project to standardize on a LAMP and Drupal stack that migrates data as fast as possible. Cyrve's customers tend to have large data sets (e.g. Examiner.com, The Economist, Martha Stewart, World Economic Forum, ...), so insertion speed is crucial. Early throughput observations ...

Environment Throughput
My Laptop. Untuned Mysql. 900 nodes/minute
My Laptop. MongoDB field storage. 4800 nodes/minute
EC2 with RDS hosted MySQL. 1900 nodes/minute
EC2 with RDS hosted MySQL and MongoDB field storage. 5300 nodes/minute

As you can see, adding mongoDB field storage boosted my laptop by 5x and boosted ECS+RDS by almost 3x. Wow! This is consistent with improvements we experienced with Examiner.com.

Dataset

The dataset I migrated can be found in the migrate_example_baseball module. Here you will find migrate module classes which import a box score from every Major League Baseball game from 2000-2009. All the little bits of the box score (e.g. attendance, batting orders, winning pitcher, etc.) are saved in Fields so this dataset exercises field storage a lot. I think this is typical of larger D7 sites. This dataset should be useful to lots of Drupal benchmarking projects.

Environments

All imports were executed using migrate-import drush command. Like all drush commands, the web server plays no part at all. This is all CLI PHP 5.3, MySQL 5.1, and MongoDB 1.8.

RDS is Amazon's hosted MySQL offerring. I chose a Large instance for all tests. I'm not sure how Amazon configures these, but i would hope that they consider the write-heavy usage scenario is considered.

Caveat: These are not formal benchmarks. I did not try to optimize these environments.

Summary

We already know that MongoDB field storage massively speeds up web page views (i.e. reads). I'm seeing here that it massively speeds up writes as well. Please consider using MongoDB for your large Drupal projects. MongoDb compatibility on its own is a compelling reason to choose Drupal 7 over Drupal 6. Learn more about MongoDB and Drupal from this Drupalcon session.

Tags

Everything about mongodb seems quite compelling but let's say that we want to do our development work in a simple environment and down the line, we want to add capabilities of MongoDB into the site. Is it possible to do an easy switchover to using mongodb or does it require migration of content to get it all working correctly?

You were on target with require migration of content to get it all working correctly. Its not rocket science for simple field types, but I know of no documentation on this topic.

if it's plain field sql storage then the conversion is straightforward. For a generic field, full blown migration might be necessary. Look out for hook_field_attach_load it is almost a sure sign of trouble.

Your numbers make sense. Mongo only has to insert one record, whereas for MySQL a node will span multiple tables and thus require multiple inserts. Transactions might lessen the problem in MySQL.

Interesting benchmarks, compelling reason to move the site I am working on to D7.

Also you have shown that Drupal is possible with RDS. I had assumed it would be but had not found anyone who had actually claimed to have done it. Did you have any issues with it or is it as simple as a standalone mysql server?

It is as simple as a standalone server. Actually simpler, since it is tuned and does automatic backups.

Enlightening article ! However, if you encourage people to use MongoDB over MySQL for large projects, why not, but I have three questions then :

1. Why not use MongoDB for every project ? If it's good for large projects, isn't it also good for small ones ?
2. Correct me if I'm wrong but I've understood that you only use MongoDB for field storage. Do you choose to keep MySQL for the rest because it's better than MongoDB ?
3. Last but not least, isn't MySQL tuning possible like Apache tuning VS nginx ?

Thank you

1. Sure, it would speed up any site to use MongoDB. It is just one additional server to manage so thats why I recommend for sites with an IT staff.

2. Sessions and watchdog and cache in MongoDB might make good sense. This article is only about data migration and there are very few watchdog calls and only one session insert.

3. Of course. That would be a prerequisite for any real benchmarks.