Don't Repeat Yourself: Enterprise Data is Hard

I usually like what Ted Dziuba has to say but his rant, The Most Important Concept in Systems Design, missed the mark on a few points and ultimately misunderstands the problem on a fundamental level. Go read Ted's article if you haven't already, then we'll jump into some counterarguments and finally why the whole rant is misguided.


1. Using Solr to Search a Database
Certainly there are challenges to setting up a functional Solr installation and keeping it updated. Most of us have probably seen Solr Installs From Hell where the index stops updating for no good reason, and this creates a problem because now your users are searching a database that's drifted from your actual production database. The thing to do, Ted says, is:

The Solution: Don't do that. PostgreSQL actually has pretty awesome full-text search capabilities.

Say what now? Postgres does have impressive FTS capabilities in comparison to MySQL, but last I checked tsearch doesn't even support stuff like phrase search (note that Oleg is one of the authors of tsearch).

Let me back up for a second. It very well could be that you need to do searches directly on the production database. eBay immediately comes to mind as a business that might need to do that in order to present up-to-the-second auction listings. If that's the case for you, I hope you've got a team of really experienced search engineers on hand to figure this out. Chances are, though, this isn't the case for you. Think carefully about how search plays into your content strategy -- do you need instantly-fresh results? I doubt it. Do you even need to manage search yourself? Probably not. Think about using a service like Google Site Search. There's no sense in adding complexity when you can just drop a textfield on your site and let a bunch of really smart folks who do search as their day job manage this for you.

If you're a big enterprise-y place, though, you might need to format search results in a way that Site Search doesn't support (to keep it "on-brand"), or you need to really tweak the index, or you're doing geospatial queries or something. There's nothing wrong with Lucene and Solr, and in fact it handles those use cases quite adroitly. You can keep the Point of Truth drift to a minimum by doing two things: manage your Solr box properly (heartbeat, disk space, connectivity to the app monitoring, etc) and making index updates an automatic part of the content management workflow.

2. Precaching SQL Results in a NoSQL System
Eventually your data will get big enough that it hurts MySQL, sure. Ted's example is a little contrived:

You've got all your data in MySQL with a great object-relational model, but querying it from your web app involves a 3 or 4 way JOIN, which causes MySQL to choke, as it hasn't yet learn how to open its throat all the way. So, to fix it, you precompute some data structures and store them in something like Memcached or Redis.

No, to fix it I set up some proper indexes, do capacity planning, and cache at the edge as much as I can. I might set up Memcached with several bins, knowing that I'd get fast (but stale) results and flush them appropriately.

3. Frankendatabases
This is my favorite one, because I'm fighting this battling at work. I imagine most enterprise-level businesses are. While I'd love to use Ted's solution to the Frankendatabase problem, it doesn't really play out this way in practice:

[Y]ou have information from multiple databases that you want to query in one database [...] Do your best to canonically consolidate the databases. Sometimes this isn't practical in legacy systems, so avoid the aggregation DB if you can - query both A and B directly. If you can't that, then hide your shame somewhere out of the way.

Query both A and B directly... sounds great, except A is an MSSQL server and B is an AS/400. Then roll in some compliance issues: C is PCI-compliant, D is an LDAP server, and A, B, and D are HIPAA-compliant. Let's connect and give write access (!) to our webapp, what could go wrong? If you're in the financial services, healthcare, or insurance industry, the combined force of your heads nodding in agreement is turning the Earth off its axis, so stop it.

What now?
I sympathize with Ted's rant because it hits close to home, but what he's solving with application architecture has to be solved with data architecture. The webapp (and mobile app and desktop app) do need to point to a Single Point of Truth which can be provided by an MDM, enterprise service bus, or data warehouse. Once that's available, the SPOT problem is either gone (because your DW rocks) or is pushed off somewhere else (because your DW sucks). A real complicating matter is that in enterprise business -- especially those with retail arms -- the webapp isn't the only consumer or producer of data. It's easy to get myopic and forget that a whole mess of other people are interacting with this data, and they might not even know the webapp exists.

The lesson? Enterprise data is hard.