Should Database Manage The Meaning?
By Alex Bunardzic under Talk, Theory on 29. September 2005I couldn’t resist jumping into the Choose a single layer of cleverness discussion, that is raging on David Heinemeier Hansson’s blog. The majority of the challenges to David’s thesis were so widly off mark, that it had left me completely bewildered. What’s even more bewildering, to me at least, is that many of the misplaced comments seem to be coming from established Ruby and Rails practitioners.
Anywho, the comment that got my particular attention is quoted below; my full reply is reproduced below the quoted comment:
“Just like I’d expect my operating system to respond if I try to write to a file I don’t have permission on, I want my database server to manage the basic rules of the DATA, ie, what relates to what, and which columns should be unique. This is to prevent anything out of the ordinary from affecting the consistancy of the database. The minute you let bad data get in there is the minute any maintainability you love in your application tier goes to hell.”
Bzzzt! Right here, we have the crux of the problem.
I think the cognitive discrepancy lies in equating RDBMS with an operating system. Nothing justifies that parallel.
If we step back and look at what RDBMS is, we’ll no doubt be able to conclude that, as its name suggests (i.e. Relational Database Management System), it is a system that specializes in managing the data in a relational fashion. Nothing more.
Folks, it’s important to keep in mind that it manages the data, not the MEANING of the data!
And if you really need a parallel, RDBMS is much more akin to a word processor than to an operating system.
A word processor (such as the much maligned MS Word, or a much nicer WordPress, for example) specializes in managing words. It does not specialize in managing the meaning of the words.
So who is then responsible for managing the meaning of the words? It’s the author, who else?
Same is with Rails. Rails is the author of the data. As an author, it uses the RDBMS to manage that data in a relational fashion. But, just as we, as the authors of the words, do not expect WordPress to manage the meaning of our words, Rails does not expect the RDBMS to manage the meaning of its data.
As a matter of fact, it would be really terrible if those tools would assume the management of the meaning of the information that is being fed into them. Imagine typing up a letter, only to be jolted when your favorite editor refuses to take the word you’ve just typed, deeming it ‘incoherent’, or ‘not complying with certain constraints’. You’d toss that piece of junk out the window in no time.
Why should Rails developers be any different? Why should we tolerate RDBMS opinions on our data? We’re the masters, RDBMS is the servant, it should shut up and serve. End of discussion.
As for the ongoing ‘the sky is falling’ discussion about what if some other device accesses the RDBMS, it’s the same dilemma as ‘what if some other person accesses our document, and starts changing it?’ There are ways to manage that. Yes, we’re always exposed, always vulnerable to all kinds of attacks, but that’s how life is. You should start getting used to it by now.
Masklinn:
Actually, recent word processors are not that unaware (if one calls that awareness) of the meanings of the texts typed in them, as long as the author himself is using them well…
Following the legacy of TeX/LaTeX and HTML itself, recent versions of word processors (Word 2000+, maybe as early as 97; OpenOffice 1&2 - 2 being quite an improvement - , …) are actually using some kind of semantic marking with styles applied on top, meaning that your section title is not “Bold 24pt underlined”, it’s “Heading 1″ and up to you to generate the style that you’ll see fitting.
The text processor doesn’t, of course, know the meaning of the data, but it gives you a way to specify it for you to undersrand and him to keep.
comment at 29. September 2005
Paul:
Perhaps instead of Word Processor you mean text editor. Because modern word processors do in fact check grammar and spelling. Which is pretty much what this whole argument comes down to. Spell check would be like type checking and grammar checking would equate to DB constraints. That user’s comment isn’t saying he wants the text editor to enforce “meaning” of the words, just that he doesn’t ever want it to allow him to write sentence fragments. Of course, word processors usually just flag the “bad” data… the still allow it. Still I think there’s more to it than just syntax and semantics, text and meaning.
I think it is clearest in DHH’s original post where he points us to Martin Fowler’s distinction between the 2 database styles. DHH is saying he prefer’s one style over the other for his apps. I actually tend to agree with you both but for different reasons.
comment at 29. September 2005
Ian Bicking:
When word processors rewrite the text I type for my benefit, it annoys me. So yes, it’s like word processors, and that’s the point. It’s even a bit like operating systems. File permissions suck. They are stupid and limited and can’t express many restrictions at all; plus they are complex and hard to manage. Not the operating system’s fault, it has no idea what those files are, where they came from, or what I might do with them — it’s just piles of bits. In the same way, the RDMS gets its tuples of data, but it doesn’t know what they are about. It’s just a pile of simple data types.
Unlike the operating system, I can put code in the RDBMS, which is what makes it rather ambiguous. I can give the RDBMS behavior instead of just data. And, given a DRY principle, that can be the best place for the behavior. But if only one app accesses the database it doesn’t really matter; and the opposite issue can happen, because database constraints often have to be repeated in code, because the database isn’t the most expressive or easily introspected medium.
comment at 29. September 2005
Andriy:
I think it might be an oversimplification. Yes, RDBMS are built to manage data in a relational fashion. But then any application runtime environment along with its programming language of choice is designed to manage data by means of an essentially sequential logical flow (and logics is just is abstract as the concept of relationships in an RDBMS and has nothing to do with meaning). Therefore the meaning is something one attributes to the relationships and logical flows at one’s own will. If the meaning is sufficiently complex it often requires advanced means to be expressed. And so if one decides it would be great to handle part of the meaning on the database server side any sort of prebuilt means in the database engine to help express the meaning should come in handy. It is simply that traditionally we expect the application layer to do the most part of “meaning heavylifting” and so in our consciousness the stress is shifted toward it.
comment at 29. September 2005
Withheld to protect the ignorant:
“If we step back and look at what RDBMS is, we’ll no doubt be able to conclude that, as its name suggests (i.e. Relational Database Management System), it is a system that specializes in managing the data in a relational fashion. Nothing more.”
This was the case a awhile ago, but nowadays in the 21st century, they do much more–whether you think they should or not is a different thing (try telling the developers of Oracle, DB2, Postgresql, etc that their additional functionality is “nothing more”). The definition of some things change as they evolve. Remember when all cell phones did was act as a wireless telephone?
If you are using differing column types, you agree that the RDBMS should be enforcing some level of integrity, otherwise you’d might just use one datatype for storage and have your application casting data to the appropriate type when needed.
The database (many of them nowadays) is another place that you can put code. If you have multiple applications accessing the same database, it makes sense to use DRY and put the domain logic somewhere, whether it’s in a web app like Rails with an exposed webservice, an application server, or a datbase.
comment at 29. September 2005
WattsM:
A better comparison with word processors is that database-level integrity checks are like templates that “constrain” you to proper manuscript formatting. Once the templates (constraints) are set up, you can just worry about getting the words (data) right. Most professional writers would consider this kind of restriction to be a Very Good Thing rather than some kind of terrible hardship. Your magazine editor will not be impressed by your creativity in writing your submission in purple crayon. And your marketing VP will not be impressed by your “creativity” in letting his survey database be corrupted in some way that would have been caught if all your integrity checking wasn’t taking place in some high-level module that, for some reason, that update that Bob in accounting just did failed to call. :)
The ironic thing here is that the original assertion that sparked this, that DHH was responding to, was about whether Rails should be able to derive relationships between tables in databases when the database is powerful enough to let Rails do so. The arguments for Rails not doing so haven’t been on practical grounds as much they’ve been about how philosophically evil it is to use integrity checks at the database level, as if by convincing us of that we’ll stop asking whether Rails should have this feature down the road. And that makes this into a completely different discussion. It’s as if we’ve moved from a question about upgrading traffic signals in a town to an argument against using traffic signals at all, because we strongly feel that it shouldn’t be up to a mechanical system at street level to tell people how to drive.
comment at 29. September 2005
Valentino Volonghi:
This is totally nonsense to me.
When I read this post it seemed to me that databases are set upon us like a giant rock upon our heads.
What does ‘RDBMS is the servant’ means?
YOU are the programmer of the RDBMS, the one that explicitly says that there are some restrictions in the data that will be inserted in it.
There is no difference between putting the logic in the database or in the application from the programmer perspective. But there is a VERY BIG difference from the maintanance perspective. When you change programming language for the application (because there is the super new shiny framework and the comany really needs to change the 10 years old application but doesn’t want to change the database because it works) what do you foresee? That you have to rewrite all the data integrity logic. What if you forget to write some of that logic (quite likely when you have to deal with 40-50 tables)? To put it simply: YOU ARE SCREWED.
What if you put everything in the database instead (we are only talking about data coherence here)? That you can change programming language all the times you want without having to worry so much about the data layer because it is already there. Applications are written once and maintaned for all their life. There’s also another issue: rules for data integrity in the application level are scattered throughout the application (wherever you delete something or modify something etc) and if there is a need to change something you have to walk ALL your codebase to change ALL the parameters, instead if you have the data logic in the database (where it really belongs) you only have to modify it once, in the database.
RDBMS concepts are pretty standard (now that MySQL 5 is finally coming out there is even less excuse to not learn Relational Systems and Transactions theories) and are somewhat consistently applied in ALL the Database Systems.
If you want to move the data consistency layer at application level it’s ok, be ready to suffer from your errors though.
And the fact that database are bad and hard to use is not an excuse. Instead of moving the problem in a place where it’s even harder to fix we should think about how to make it simple to deal with the problem at the right layer.
It’s like having a flat ISO-OSI standard instead of the 9 layers. I don’t even want to understand how an RDBMS could be compared to a Word Processor or an Operating System. It’s a data layer full stop. You tell it how it should manage the data you should put in it. There is no magic in RDBMS programming.
comment at 29. September 2005
Alister:
You are answering a bad analogy with another bad analogy. Put the strawmen away.
comment at 29. September 2005
Chris Carter:
First of all, thanks for the honor of picking my post to respond to! Yes, that was my original assertion up there in those fancy quotation marks.
I think we’re getting into symantic arguments and chasing our own tails here. Let me see if I can clarify what we’re talking about. When you say “meaning”, I assume you are talking about business rules and how the data INTERACTS with itself. When I say “rules”, I’m talking about those most basic rules of the road, such as data types (string, integer, boolean, what have you). I believe the database should cover these basic ground rules, and I believe uniqueness constraints and relational integrity are included in these basic ground rules. As someone posted in DHH’s blog thread, would you want a database without data types in it? So you can just put any old data value in a field? That’s a form of constraint right there.
As you said, an RDBMS is an “Relational” system. As such, it should probably do something relational, shouldn’t it? That means relational integrity. Being able to use SQL to SELECT rows from one table where a columns value equals another tables column value is not relational, that’s just set mathematics. Relational comes into effect when you say one table’s column depends on its value from another table’s column. Thus, if you try to put an invalid key into a foreign key, you are prevented. This keeps things relational, and keeps things from becoming frustrating when, 3 years from now, you’re trying to build a reporting system on top of that transactional inventory system you created in Whiz Bang Framework #459.
Once again - I am NOT advocating putting business logic in the database, but if you develop for a company of any size (either as a contractor or in-house developer), I just can’t understand the philosophy that your database is used only unto your application. If that’s the case, then why use a database at all? Why not use a specialized flat file that provides more performance?
Finally, I should have said “file system” and not operating system. With that, how can you possibly assert that you’d be better off without file permissions? How can you have any security at all without file permissions? I don’t understand that sentiment either.
Anyway, hopefully this illustrates or clarifies my point a little better. I have a feeling though, that this is one of those holy wars that neither side will ever really agree on :)
comment at 29. September 2005
Larry:
Well, sort of. The problem with this analogy is that in most situations, the data has multiple authors. In a small shop (such as David’s), you don’t often have a dedicated DBA, but in a larger shop multiple people might need to manipulate the data in a database. A DBA is most likely not going to use a ruby script to do the data manipulation, nor should he. The thought of doing an update on a million or so records using ruby and ActiveRecord is a bit of a scary thought.
comment at 29. September 2005
anton:
think of context and perspective: small apps - your app is the only app that updates the db. you manage it, all is fine. this is where rails is, so its ideology is a good fit.
now take any considerably complex system where multiple platforms/apps/users manage the data in a single database. in this case the answer is different.
…and none of the flowery analogies.
comment at 29. September 2005
Aristotle Pagaltzis:
I am confused.
We rail (no pun intended) on the SOAP/WXS/WS-* because they use an OO mapping on top of XML.
But we are suppsoed to applaud the ORM guys because they use an OO mapping on top of a relational database?
comment at 29. September 2005
Zachary Hartley:
Hrm, I would have thought that a relational database is also supposed to make sure the data going into satisfies the relationships specified, so that you can not accidentally put the data into a state that it was not meant by the designer’s of the schema to be in….
comment at 29. September 2005
Scott Yang:
Besides putting the business logic closer to the data as safe guard to its integrity, there are many uses for store procedures and triggers.
Performance is one. It really reduce the traffic between your RDBMS and your app server. There’s less latency. And your DB probably knows better how to optimise on its indices. I think we are all pretty much influenced by the n-tier architecture today, designing systems to have long pipelines from end user all the way to data storage, having each tier doing just one simple thing. However when you start to scale, the “less code” soon becomes “mooooooore code” and completely kills the performance. I am all for off-loading business logic inside DB for performance sake.
Then you have data encapsulation in integration. Not everyone in the world uses REST style web services. We have clients constantly asking us to provide them ODBC access to the database, so they can do their customised reporting in Access or Excel. Obviously we cannot tell them, “hey, this is the URL for the web service end-point. You shall get back an XML with these parameters”. But we can give them views or selectable store procedures with restricted access.
comment at 30. September 2005
Alex Bunardzic:
Just a couple of quick general observations (I’ll delve a bit deeper into the more probing questions tomorrow, because I need to go to bed soon):
Current information theory sentiment prescribes that databases are a necessary evil. If we close our eyes for a moment and imagine a world with unlimited main memory that never goes down, why would anyone need a database in the first place? If you gave me the means to keep all the objects instantiated while my app is running in the main memory, I would never bother to persist them. The only reason I ever bother to persist them right now is because I’m nervous that either a.) I’ll run out of memory or, b.) the system will crash for whatever reason, and my poor objects would then vanish in a cloud of thick smoke.
There was a time way back when when RDBMS technology was considered daring, adventurous, ‘pushing the envelope’. Those were the days when auxiliary storage was much more expensive than main memory is today.
Nevertheless, E. Codd pushed his agenda, made people buy into the vision of the distant future, and voila! today we live in the world where auxiliary memory is considered practically unlimited. We stopped worrying about our storage capacity long time ago.
This state of affairs caused us to be very wasteful with the databases. But, we’re now very stingy with the in-memory objects. Kind of the way we were stingy with the disk storage 20 - 30 years ago.
But the day will come when terrabytes of main memory will be the norm. At that point, will we still be this obsessed with persistence?
Getting back to the database mentality that most people are exhibiting here — this is the pre-9/11 thinking. Yes, there were times when every budding programmer was drilled into being obsessed with every byte his/her code stores in the auxiliary memory. Witness the Y2K phenomenon which was entirely caused by such anal-retentive mentality. There was a time when it was unthinkable to even suggest that a full 4 digit year gets stored on disk. What, you expect us to sacrifice two additional bytes of disk storage for every date you write? What an ungodly thought!
Today, no one gives a hoot as to how big the data on the disk grows. But the database obsession still remains, as is painfully visible from the sentimental outpour in this thread.
But why are people still so obsessed with databases, today, in the twenty first century? There is actually a very simple explanation: databases are still the only way most programmers are able to express themselves declarativelly, using a piece of mainstream technology.
Declarative programming always gives this unmistakeable sense of solidity and robustness, over algorithmic twiddling. It is no wonder that people reserve the most critical, most robust information to be declared to the database.
But, again — boring! We now have better declarative tools to do a more proper job in ensuring that the most vital business logic is secured. And databases are definitely not a way to go when it comes to ensuring this robustness.
Yet, almost everybody here pretty much seems to feel that databases are a heaven-sent that could never be replaced by a more human-friendly servant. What a curiously retro way of thinking.
Historically speaking, we went through basically two phases in software development before we hit the current state (I’m talking about the state symbolized by such enlightened products as Rails):
Database driven (starting from the database and pushing the processing logic outwards, toward the screens)
Screen driven (starting from the forms, be it VB forms of HTML forms, and pushing the processing inwards, towards the final destination — the database)
Today, the approach has fundamentally changed. Today, we start right smack in the middle, from the object layer (the model/controller tier). We then push the processing simultaneously in two directions: outward towards the view, and inward towards the persistence.
If we now bastardize this discipline and allow the old fashioned ways cloud our thinking, and let the database-centric thinking rule the rooster, we won’t gain anything. We might as well go back to any of the old paradigms and metaphors.
It is important to commit to and observe the discipline. The discipline is tough and demanding. It’s very easy to slip along the way. But we must persist, because only those who persist will be around to tell the story.
comment at 30. September 2005
Aristotle Pagaltzis:
So why are you using an ORM in the first place? Considering this tirade, shouldn’t you be the first one to be scuttling away from ActiveRecord?
SQL is lesscode vis-a-vis keeping all your data in memory and querying it programmatically by writing code to climb through the data structures.
Much like XPath is lesscode vis-a-vis climbing up and down in a DOM.
comment at 30. September 2005
Valentino Volonghi:
Persistent is opposed to Volatile not to available memory.
Some databases never get bigger than 20 or 30GB, a memory space that is easily affordable on a server nowadays. Yet nobody ever dares to drop the database because:
a) Sharing a database is easy.
b) Problems with hardware won’t make you lose any data.
c) You can easily (there are tons of tools for this) replicate and load balance a database among many computers.
d) Rebooting is safe, and you need rebooting when upgrading the operating system because of the latest vulnerability in the kernel or in a fundamental library for your application.
c) Sharing between different applications.
d) Easier maintanance path (upgrading live objects in a system is even harder than persistence)
No, object persistence is not the right answer because it can’t be shared among different computers at the same time in an efficient way (sharing 30GB of data through a socket would, at best, kill performance).
In the end: no, memory availability won’t allow you to avoid database, which, as has been stated, actually reduce the code needed to provide a sensible data layer to an application. And also allows all the data to be used by many other applications without worrying to much that one of them might be doing the wrong thing when inserting data, or you have to replicate the data coherency code in all of those applications (also the ones you didn’t write) but this is morecode, not lesscode.
comment at 30. September 2005
Lorenzo Gatti:
Whether complex constraints and stored procedures are evil, as well as many other architectural decisions about databases, is mostly a matter of perspective.
If the stored procedure XY that retrieves the records of screen X of application Y is considered a part of the database that evolves on its own, we have mixed implementation details with the more abstract and more permanent database schema.
On the other hand, if procedure XY is treated as a part of the application that is written in a different language and is deployed in a different place, the database definition maintains its integrity (with application scripts and permanent tables in different logical containers).
If we add constraints casually because they would spare us some work without breaking any use case, or just because we assume they should be verified, we are reducing the clarity and integrity of our database.
If constraints are explicitly designed to describe what values and references are definitely unacceptable, or at least unsupported by any existing application, they become useful.
” Why should we tolerate RDBMS opinions on our data? We’re the masters, RDBMS is the servant.”
“RDBMS” here means a conflation of three or four different things: the RDBMS proper, which is as empty and generic as an operating system; our definitions of important and permanent data; our definitions used by specific implementations of specific applications; maybe also the SQL queries issued by our non-SQL programs.
Obviously the “servant” consists of the first layer only; everything else is our code, and it is expected to represent and enforce our opinion of our data.
Maybe this confrontational master/servant attitude can arise from social factors: in many situations “we” are the agile OO application programmers and the “RDBMS” is a different slow-acting or competing group or an unpleasant legacy we want to improve.
We want to be in control but we are unwilling, unable or unprepared to conquer the enemy territory of the database, so we put all complex logic on our home ground and then we try to reduce the usefulness and importance of the opposition.
From a technical point of view, it is unquestionable that stored procedures and complex SQL queries should be used: compared with the same thing done in memory in a procedural language, they usually offer a better performance with a fraction of the effort.
Constraints are, like other SQL, easier to write than application code, they avoid repetitions better than well-factored code and they offer an added benefit: they ensure that nobody ever, not only places in our application where we didn’t check correctly, but also unforeseen new applications or someone issuing arbitrary statements, can corrupt the database.
It is a matter of design and organization: if we are not using our tools properly, either we don’t know the correct way or someone is forcing us to do something different.
comment at 30. September 2005
Valentino Volonghi:
Totally agreed.
It is a trend that I’ve seen so many times in the last years. Developers tend to forget about databases and their enormous usefulness. Instead new frameworks come out with the only purpose of making the initial development extremely fast without worrying too much about future development and maintanance. While this has positive effects in the short term, it also sounds a lot like creating use-once applications.
Programming is all about data managing. Ensuring that your data is always coherent is the bare minimum that a programmer can do, no matter what application or language you are using to access that data. And here comes the sentence that I quoted to which I totally agree.
comment at 30. September 2005
Alex Bunardzic:
Aristotle wrote:
“So why are you using an ORM in the first place? Considering this tirade, shouldn’t you be the first one to be scuttling away from ActiveRecord?”
I’m surprised to see how many people don’t seem to understand what ActiveRecord, and the entire Rails is all about. Rails is so phenomenally good precisely because it addresses both the ideal and the pragmatic side of software development.
On the idealized side, concepts such as ActiveRecord are put in place to act as a Smart Servant. What this means is that they are capable of hearing a few short commands and are then capable of materializing these simple commands into a set of intricate and elaborate actions. They act on our behalf. They perform the legwork for us. You gotta love them!
On the pragmatic side, such Smart Servants are capable of making a concession to the imperfections of the happenstance, and adapt themselves to the present constraints.
In this case, ActiveRecord is a Smart Servant who realizes that the majority of information infrastructure is still based around RDBMSs.
However, SmartServant is not hung up on RDBMSs. It can drive the entire app from a non-RDBMS perspective. It is possible to teach ActiveRecord to drive the app from an in-memory object model, or from a pile of flat files, or even from some proprietary legacy data store (such as Lotus Notes). Or from any combination of these beasts thereof. All these ugly things could be abstracted by the SmartServant, shielding us from the aggravating aspects of the legacy stuff.
So you see, I don’t view ActiveRecord as an ORM. Yes, it can act as the ORM, but it doesn’t have to.
For example, I can tell ActiveRecord to find me something. Being a Smart Servant, he will know how to follow my command. But I’m not assuming that he will necessarily attempt to find things in an RDBMS source. Frankly, I absolutely don’t care where is he going to go and look for things, just as long as he finds them for me, within the acceptable time frame.
Similarly, I can tell him to create an object. Now, where is he going to attempt to persist that object is none of my business. I simply don’t want to know. And the less I know, the better (worse is better). And on and on — I can tell it to save with validation, for example, not caring about the implementation details. All I know is that he will attempt to save some state after validating it.
With Smart Servants such as ActiveRecord, I don’t have to be a control freak anymore. I’m better off relinquishing the control. I’m not breathing down anyone’s neck anymore.
As you can see, I love having Smart Servants, I love ActiveRecord, and I don’t see it as painting me into the RDBMS corner. I can always teach it new tricks.
“SQL is lesscode vis-a-vis keeping all your data in memory and querying it programmatically by writing code to climb through the data structures.”
There is nothing in the world stopping anyone from writing an in-memory querying language that wold be a lesscode solution. Why do you think that SQL is suddenly this silver bullet? It’s just a Smart Servant, standing between our code and the RDBMS. Nothing magical about that.
“Much like XPath is lesscode vis-a-vis climbing up and down in a DOM.”
Yeah, I know. You see, Rails also comes with helpers. Hint, hint, nudge, nudge, say no more…
comment at 30. September 2005
Chris Carter:
“Current information theory sentiment prescribes that databases are a necessary evil.”
You’ve lost me on this one. I am REALLY interested in where you’re getting that idea from. It’s great to dream that we’ll have “unlimited main memory”, but out here in the world where most mid to large companies are ensconced with crappy software and are completely aversive to ripping it out (thus risking a failure of operations), the new “pragmatic” developers will have to learn to integrate with the existing systems, and build systems that themselves integrate. It’s a little beyond ambitious to think that you can bust into the “Enterprise” (as this whole business with the Burton Group and DHH talking trash about Java seems to insinuate), when you blithly assume that you will 1. be the only development team ever to work on a project and 2. that Rails will be the only framework ever to access it.
Your opinion fits well if you’re developing on the AS400, where the application framework and the database were one, but that ISN’T how it works anymore. You have two tiers now, and regardless of what clever name you come up with for Active Record, it IS a bridge between those two tiers (ORM, Persistance Layer, whatever you want to call it). When you have two distinctly separate tiers, you MUST suck it up and take the 5 minutes to add some simple checks on your database to cover the event that some numbskull is going to access your database outside your control and screw things up. Don’t assume that just because your application is “small”, that it somebody won’t want to tap it’s data.
Let me repeat - it’s great to dream that one day we’ll be back to the days of a unified application and database, but that won’t be here for quite awhile. In the meantime, out in the real world, losing the whole “relational” side of a “Relational Database Management System” and ignoring the performance gains you can make with indexes and such is a road to grief.
I’m down with Agile development, I’m down with Rails in the “Enterprise”, and I’m down with slapping the database around. However, to ignore the most basic rules of the road when they are so trivially easy to implement is just silly.
comment at 30. September 2005
Alex Bunardzic:
Chris Carter wrote:
“When you say “meaning”, I assume you are talking about business rules and how the data INTERACTS with itself. When I say “rules”, I’m talking about those most basic rules of the road, such as data types (string, integer, boolean, what have you). I believe the database should cover these basic ground rules, and I believe uniqueness constraints and relational integrity are included in these basic ground rules. As someone posted in DHH’s blog thread, would you want a database without data types in it? So you can just put any old data value in a field? That’s a form of constraint right there.”
Actually, I would prefer if all the values entered in the database would be of a single data type — text. This is because, at the end of the day, the data is mostly meant for human consumtion, and we consume it as text.
Regardless of what fancy data type may be sitting in the database, once it hits your eyes, it is plain text (whether displayed on the screen, or printed on paper). Also, even if it’s supposed not to be consumed by humans, but only by machines, it still gets converted to plain text (typically, XML).
So I really see no other point in having any type, other than plain string, defined in the database, other than to leverage the power of declarative programming.
But as I’ve already said above, today there are better ways to do declarative programming (cf. the ActiveRecord). Database is meant strictly for persistence, not for behavior.
“As you said, an RDBMS is an “Relational” system. As such, it should probably do something relational, shouldn’t it? That means relational integrity. Being able to use SQL to SELECT rows from one table where a columns value equals another tables column value is not relational, that’s just set mathematics.”
I think you are confusing ‘relational’ with ‘normalized’. No one ever said that a relational database must be normalized to the 5th normal form. As a matter of fact, it almost never gets that far. Even the third normal form tends to be a bit of a stretch in most production systems I’ve seen.
<reportFromTheTrenches>
And then don’t forget that most systems tend to get severely denormalized for performance reasons too. So, it’s a moot point, to say the least.
</reportFromTheTrenches>
“Relational comes into effect when you say one table’s column depends on its value from another table’s column. Thus, if you try to put an invalid key into a foreign key, you are prevented.”
This is one of the key reasons why general public think that enterprise software sucks. Real-life systems are fudgeable, but software is moronic precisely because of these idiotic constraints.
“This keeps things relational, and keeps things from becoming frustrating when, 3 years from now, you’re trying to build a reporting system on top of that transactional inventory system you created in Whiz Bang Framework #459.”
FUD.
“Once again - I am NOT advocating putting business logic in the database, but if you develop for a company of any size (either as a contractor or in-house developer), I just can’t understand the philosophy that your database is used only unto your application. If that’s the case, then why use a database at all? Why not use a specialized flat file that provides more performance?”
Precisely my point. No one claims that Rails is married to an RDBMS. And frankly, I couldn’t care less about the underlying implementation. As far as I’m concerned, it could as well be a huge army of little green men with hammers, locked inside the engine room, hammering out the data.
“Finally, I should have said “file system” and not operating system. With that, how can you possibly assert that you’d be better off without file permissions? How can you have any security at all without file permissions? I don’t understand that sentiment either.”
When did I assert that I’d be better off without file permissions?
comment at 30. September 2005
Alex Bunardzic:
WattsM wrote:
“A better comparison with word processors is that database-level integrity checks are like templates that “constrain” you to proper manuscript formatting. Once the templates (constraints) are set up, you can just worry about getting the words (data) right. Most professional writers would consider this kind of restriction to be a Very Good Thing rather than some kind of terrible hardship. Your magazine editor will not be impressed by your creativity in writing your submission in purple crayon. And your marketing VP will not be impressed by your “creativity” in letting his survey database be corrupted in some way that would have been caught if all your integrity checking wasn’t taking place in some high-level module that, for some reason, that update that Bob in accounting just did failed to call. :)”
A database controlling the meaning of my data would be equivalent to the word processor who insists that, for example, any document containing the words “George W. Bush” must associate with it words such as “integrity, responsibility, leadership, success, good will, tolerance, open-mindedness”. Or, the word processor who is calibrated to insist that no words such as “incompetence” ever get associated with the word “George W. Bush”. I’m not so sure I’d be so keen on using such a product.
“The ironic thing here is that the original assertion that sparked this, that DHH was responding to, was about whether Rails should be able to derive relationships between tables in databases when the database is powerful enough to let Rails do so.”
No, the original assertion was that Rails philosophy and architecture should be, nay, MUST BE influenced and affected by the awesome capabilities that modern databases offer. All David said in response to that was (and I’m paraphrasing here): “Not on my watch!” To which, all I can say is: my hat goes off to you, David!
“The arguments for Rails not doing so haven’t been on practical grounds as much they’ve been about how philosophically evil it is to use integrity checks at the database level, as if by convincing us of that we’ll stop asking whether Rails should have this feature down the road.”
David simply refuses to be governed in his ideas by the big-ass vendor ideas about what sort of processing should an RDBMS embody. Why should he be compelled to follow someone else’s hidden agendas? The man has a back bone (the first one in this industry with a back bone in the long time). I applaud that.
“And that makes this into a completely different discussion. It’s as if we’ve moved from a question about upgrading traffic signals in a town to an argument against using traffic signals at all, because we strongly feel that it shouldn’t be up to a mechanical system at street level to tell people how to drive.”
No, this is more like a debate on whether it should be mandatory to wear protective gear (helmets etc.) while walking down the street. Governments would like to impose such regulation, in order to avoid footing the bill for potential injuries. Us, common-sense citizens, oppose that, as we feel it encroaches on our personal freedoms.
And if you think I’m kidding, come to Canada and try riding your bike without a helmet!
comment at 30. September 2005
Chris Carter:
I’m going to step out of this discussion, because, as I predicted, I don’t think there will be concensus here :)
You can have your completely ambiguous “data store”, I’ll have my structured database, and if it works for you in the long run, then it works for you in the long run.
However, I’ll stick to my structured database with my data types, enforced relations, and unique constraints. I can guarantee (with minimal effort) that some third party won’t sneak data in behind my application, and that’s what I like (and what, apparently, a good chunk of the people posting here and on DHH’s blog like too).
Two things before I sign out of the debate (because I can’t help myself :)) -
How can you store binary data as text (and don’t say the filesystem, that’s just another database)?
Since I’ve never met you, and can’t find any background on your site, what is your experience? Not meaning to say your opinion doesn’t matter, or that you’re wrong, but I’m just interested in how you’ve come to your conclusions.
comment at 30. September 2005
Robert Church:
What’s so great about riding a bicycle without a helmet anyway?
comment at 30. September 2005
Ryan Tomayko:
I just wanted to thank everyone for the discussion. This has been one of the most lively on lesscode.org to date.
comment at 30. September 2005
Chris Carter:
I should add two quick corrections to myself:
I have no idea where I got the file permissions thing from. That’s what I get for trying to read 5 different blogs at the same time.
After using the mighty power of Google, I think I’ve got a decent idea of your background, Alex. But now that I do, I’m REALLY curious about how you developed your current opinion of what a database should be!
And yes, I agree, great discussion. Regardless of the outcome, it’s always refreshing to see different opinions on ideas the industry takes for granted.
Go Rails!
comment at 30. September 2005
Alex Bunardzic:
Chris Carter wrote:
“After using the mighty power of Google, I think I’ve got a decent idea of your background, Alex. But now that I do, I’m REALLY curious about how you developed your current opinion of what a database should be!”
First off, I thought this is supposed to be the place where we obsess about the philosophy and practicalities of software development, not about personalities:-)
But to answer your plea, I must explain that I came to the field of software development from the field of philosophy of science (Karl Popper, Ludwig Wittgenstein, Derida, Quinne, and the like; I won’t bother you with the links, you know how to google things around). Some 20 years ago I got involved in studying the principles of AI, after reading Dougla Hofstader’s seminal book “Godel, Escher, Bach: The Eternal Golden Braid”.
Fast forward to 1995. IBM was in the midst of a big project they code-named “San Francisco Shareable Frameworks”. The goal of the project was to produce the platform that would deliver to the discipline of business app development what AS/400 delivered to the businesses seven years prior to that (in case you don’t know, AS/400 was launched in 1988 and offered an ‘operator in the box’ — a marvelous piece of technology).
To cut the long story short, what IBM introduced with the San Francisco frameworks was the focus on the middle tier, or the business logic layer. Around that time they’ve also released Visual Age for Java, an interesting little development platform that was not based on the idea of files. Magically, the source code would appear from its repository in the editor, and any little change you make to it would be versioned so that you could traverse the undo/redo stack to your heart’s content.
But the main attraction was their persistence layer. It was, again, magic, in the sense that the instantiated objects would be somehow persisted and resurrected on demand.
Fast forward to 1998 — San Francisco framework ended up being too bulky and too clunky for real life deployment, so IBM scraped the efforts. But, the core philosophy, and the core code base, survived, and migrated to — yes, you’ve guessed it — Enterprise Java!
So back in 1998, Enterprise Java was a platform that proposed that we ignore the database, and focus on the object layer.
.NET followed suit. The rest is history.
These guys didn’t really know how to make it work. But the idea persists. Databases are being viewed as a necessary evil by the majority of the incentives in the enterprise computing.
So, mystery solved, and it wasn’t really my idea at all.
But my way of thinking really is that RDBMS is a technology that shouldn’t be driving our development efforts, nor should it comandeer our design. Given the volatility of the main memory, it’s nice to have a reliable storage, but there are many apps springing today that don’t necessarily sit on top of an RDBMS.
comment at 30. September 2005
Bill de hÓra:
I think it’s worth keeping a strong split between a database and an RDBMS. If you don’t want to manage data in an RDBMS, that’s fine, but you still have to manage the data somewhere. RoR chooses to do that in the application layer, but if you read enough Fowler or Evans or Booch you’ll see that viewpoint is standard in the OO world.
The great lie of enterprise development is that busines logic is nothing of the sort - it’s business illogic. If it was logic, we wouldn’t be having this discussion, because relational databases would be sufficient. Another issue is that business logic across applications tends to be inconsistent and/or contradictory and that makes supprting the constraints in a single databse difficult. Finally it does appear that upfront commitments about the state of the world (which is what domain models are) are becoming less important relative to the ability to react to external data; i.e. it’s more important to know that 60% of customers loved some new feature than having a good domain model of a Customer. I think this favours OO hackery over realtional elegance.
Btw, everyone who cares about this stuff should read Bill Kent’s Data and Reality. Great book.
comment at 30. September 2005
WattsM:
Chris is probably right — there isn’t going to be consensus on this. :)
I’m going to put out just a bit of philosophy myself, though. The pitfall of “opinionated software” is that it carries a risk of blowing off what the “big name vendors” do simply because they’re the big name vendors, whether or not what they’re doing actually has value. And there are times when “tried and true” trumps “innovative and experimental.”
My suspicion, honestly, is that Active Record will start taking advantage of more power in database back ends when that’s appropriate, and there will be times we’ll learn it is indeed appropriate. I’m simply not swayed by the argument that the Smart Servant is made smarter by limiting his knowledge of the available toolset, even if said toolset is philosophically impure. :)
comment at 30. September 2005
ugly DBA:
Chris Carter :
Bhu-gha-gha-ha-aa!!!
What a nuisance! Chris, that must be joke or you don’t have a clue, what relational theory is all about. (mistakenly you name “relational integrity”, talking about referential integrity, that is only one level for constraints in relational DB).
Folks, RM is an application of set theory and predicatte logic to data management, i.e. it is indeed applied mathematics. And mathematics does not manage the meaning and (probably) never will. Meaning and semantics are (yet?) out of scope of maths. Meaning is still a exclusive prerogative of human brain :)
And for each math system (and properly designed concrete relational model is such a system) there always are constraints and a formal system to enforce them. That’s what RDBMS is for, there is no notion of meaning in RDBMS - only set of formal constraints.
OK, ActiveRecord gives good way to use declarative constraints in OO programs, dependent associations, callbacks and observers - but all that is already in modern SQL DBMS’s for years. DHH doesn’t want his database to be clever, instead reimplementing that stuff in pure OO way.
Alex Bunardzic:
I don’t get that. Yes, there are “big-ass vendor ideas”, but there’s also one sound theory for data management and it isn’t clever to ignore it.
comment at 01. October 2005
Laurent Szyster:
If a computer system was to manage “meaning” (semantic and data, “signifiant et signifié”), I would not call it a database, but a metabase.
So, rephrasing your original question as
“Should Semantic Applications use RDBMS”
my answer is no. And the question is actually a rather old hat:
The semantic data model: a modelling mechanism for data base applications
http://portal.acm.org/citation.cfm?id=509264
“Proceedings of the 1978 ACM SIGMOD international conference on
management of data”
You can develop a metabase application “on top” of a relational database system, but you should consider something else.
The application of an RDBMS is to store on large sets of data according to a relational schema and produce statistics using a relational algebra. Since:
data + structure = algorithm
each relational database schema reflects its application, which is seldom applicable to other purposes. But what schema would you use to store a web of information? If you pick a simple relational schema that suites to your specific application, it will most probably hamper other applications to access the same information.
The obvious thing to do, to chose a single data structure, indicates clearly that the relational model itself has no semantic applications.
comment at 01. October 2005
Shared Data and Mobile Data [@lesscode.org]:
[…] about lesscode.org « Should Database Manage The Meaning? […]
pingback at 02. October 2005
Alex Bunardzic:
WattsM wrote:
If consensus is what you’re after when visiting these discussions, then I’m afraid you’ve come to the wrong place:-)
Opinionated software is not opinionated against the big vendors. It is opinionated against anything that is contrary to its underlying philosophy (I’ll write more about this on a future post).
If the big vendors approach is in violation of the principles that the opinionated software upholds, than the opinionated software is going to shun the big vendors approach. But anything that the big vendors support that agrees with the opinionated software’s agenda, will be embraced wholeheartedly.
Database back end (and by that I’m assuming you mean RDBMS, such as Oracle, DB2, SQL Server, Postgress, MySQL etc.) is just one of the many devices that ActiveRecord deals with. ActiveRecord cannot afford to favorize one over the other. In the future, the dominion of RDBMSs will be on the wane, so it isn’t very likely that ActiveRecord would expend a lot of effort into a dwindling community.
I could be wrong on this, of course.
comment at 02. October 2005
Alex Bunardzic:
Robert Church:
You mean, what’s so great about having sex without a condom?
comment at 02. October 2005
Robert Church:
To the extent that relational databases are like sex, I agree that the use of bicycle helmets is generally advisable.
comment at 03. October 2005
Leandro GFC DUTRA:
I haven’t read all the comments, because frankly I found so many of them disinformed about the relational model that it wasn’t worthwhile.
You are incurring in two basic mistakes here.
First, you are mistaking SQL for an RDBMS language. It is not. Anything SQL can’t possibly be relational, it is not powerful enough, it is way too complicated, it violates the basic rules of the relational model.
Second, all business rules should be declared at the RDBMS level as integrity constraints. Yes, all of them. Read Date’s writings on that, ideally An Introduction to Database Systems but What Not How is a good introduction on the specific integrity constraints-as-business rules approach.
Unfortunately, I am aware of only two RDBMS engines available nowadays: the proprietary, MS.Net-based, SQL-federated, NULLs-corrupted Alphora Dataphor, and the pure, free, immature, Java-based dbAppBuilder Rel. Dataphor is outstanding in that one can derive most of the application directly from the relational model of the database, leaving only UI polish to finish it up.
All that said, it is still possible to create an almost relational system on SQL, although with quite some grief. And it is still better than procedural coding of business rules in the application, specially for large, shared databases.
comment at 08. October 2005
The real problem:
I don’t feel like covering every issue in why you will pay dearly for not properly maintaining meaning the database. But consider, Structure and constraint provide meaning, otherwise the data is just noise…. that is what relational integrity is all about.
Do you really think its coincidence that a) all values in a database are scalar, b) attributes are NAMED, and c)Keys are matched by VALUE and not by memory location. I suggest you read about the death Navigational and Network data Models.
But half this argument seems to come about because the rails ActiveRecord using name matching to derive data instead of the catalog. Oh yeah, did i mention that the logical model of a proper RDBMS is self describing. You could really have all of those constraints and validation derived from the RDBMS, without adding it to your code. Not only do you not have to code those constraints now, but you can also allow for the data to be accessed and used in more, meaningful ways (i.e. other clients of “your” database) without breaking your application.
The programming side is getting there, with dynamic typing; the RDBMS side still needs to catch up on implementing the Relational Model as it had been described 30 years ago…. Views should be logically indistinguishable from tables, join speed needs to be seemless, Domains need a lot of improvement, and ditto for the Catalog. (though, that is no excuse for ActiveRecord to not use what it already offers).
comment at 11. October 2005
Ethical Software Alex Bunardzic:
[…] Not that long ago I was involved in a discussion revolving around software-imposed control. More pointedly, I wrote a little piece entitled Should Database Manage The Meaning?, and it created a burst of heated discussion over at the lesscode.org blog. I am now going to revisit one or two salient points presented throughout that discussion, as I feel they reveal the tacit underlying assumption about the role of software in our lives. No Comments so far Leave a comment Line and paragraph breaks automatic, e-mail address never displayed, HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong> […]
pingback at 11. October 2005
Pigsaw Blog:
Domain driven design, and different kinds of databases
While reading Eric Evans’ excellent Domain Driven Design, I was put in mind of one of Dave’s recent blog entries and what is being called The Great Application/Integration Database War of Attrition (by absolutely no-one).
The debate has been spark…
trackback at 18. October 2005
A random pixel » Use stored procedure and triggers sparingly:
[…] A thought similar to this was expresses by DHH here. Also read the follow-up article on the same topic. […]
pingback at 25. October 2005
Ethical Software by Alex Bunardzic » The Myth Of Data Integrity:
[…] David is, of course, right. I don’t have to go any further than point out the reaction to my posts on the same subject, of which the most notoriously acute was Should Database Manage The Meaning? As one of the programmers (Valentino Volonghi) who indulged in the ensuing discussion had pointed out, “This is totally nonsense to me.” […]
pingback at 01. November 2005
wjv:
Oh, lookee, this post made Fabian Pascal’s “to laugh or cry” links. And to think that the smart money was on the original DHH post to be quoted there.
Frankly, I’ve settled for laughing.
comment at 01. November 2005
Ethical Software by Alex Bunardzic » Confessions Of A Former Data Integrity Addict:
[…] But the confusion crept in regarding the boundaries of the applicability of this theory. I wrote about these boundaries in several previous articles (Should Database Manage The Meaning?, Shared Data and Mobile Data, and so on). Needless to say, such soul-searching writings provoked an avalanche of emotional outburst from the database crowd. […]
pingback at 09. November 2005
Bob B:
Let’s compare databases to word processors the right way. Database constraints map nicely to word processor formatting. I can tell the word processor to use double spacing, to use the arial font, to use 14 pt, to underline something, etc etc. I can apply it to as little or as much of the meaningful data as I want and the word processor makes sure those rules apply. The same is true of database constraints. This column is not null, that one is a number, this one must exist in that table.column.
I establish the rules of my data formatting and the database or word processor ensures that all data conforms to those rules. The database is just an application at heart, one which is built to store, retrieve, recover, constrain, and manage gobs of data.
On a side note, there is one huge advantage to a well-designed database. It can change its access path based on the amount and type of data put into it. It can switch between brute force (full table scan), indexed access (unique location and index scanning), and hashing. Furthermore, the databases are created by tons of talented developers with a huge base of users testing. Not only are the database owners testing it, but so are the database users. There are few applications that can say they have had that many users test their system - EVER.
Databases are not bit boxes. While they do seem to serve that purpose, they are much more complex on the inside than is outwardly noticeable. If developers do not understand what it is doing, why it is doing it, and how to take advantage of it, I would not want them to implement data management in my applications.
comment at 14. November 2005
Anonymous:
“Managing the meaning of the data” is a just a straw man! The developmer manages the meaning of the data by specifying constraints and FKs, the database simply ENFORCES those rules.
comment at 21. December 2005
Martin:
You are all talking about different apps hitting on your database, and that being the only problem with “application databases”.
Now, what if you do load-balancing, and have several apps of the same kind hitting your database at the same time?
you’ll need some central integrity checking in your database then.
regards,
Martin
comment at 07. January 2006
Shawn Oster:
I watch this debate ping-pong back and forth across time but one thing I don’t see a lot of is Real World situations. I’ve worked with a lot of different databases and more often I’ve found contraints limiting rather than helpful, especially when dealing with multiple apps written in multiple languages accessing the same DB.
I’ve seen address tables locked down because a bad schema design required a relationship to an account table. A duplicate table had to be created for non-account addresses.
I watched a 5 minute “I’ll just change that in the DB” turn into a 2 hour frustration session by the DB due to the mountain of RI constraints.
I’ve spent hours trying to track down unclear error messages when I tried to delete a record from a table that had a RI because the DB abstraction layer I was using didn’t understand a new error code.
I’ve been stuck twiddling my thumbs because the DBA was on vacation or was out sick that week and I needed a change made.
I’ve gone drinking because I found out that there was no source control on a database’s stored procs and it was either drink or stab someone.
I’ve watched programs do no validation in their apps or pages because they assumed the database was going to catch any errors. You know, because the DBA said so.
In many years and many companies of many sizes I’ve rarely if ever seen apps that were less stable when backed into databases that lacked RI. I can’t ever remember a time I spent hours tracking down a bug due to invalid data in the database but I’ve wasted days and weeks trying to work around constraints.
Milage varies but that’s what I’ve seen.
comment at 02. March 2006
Madeblog » Blog Archive » ActiveRecord Angst… ReActivated.:
[…] It’s LAMP right? Not LAPP, LAOP, LASSP or LADP. Despite the accusations regularly levelled at MySQL, I think that’s what most of us use and if you believe in getting everything through your ORM then there’s a good argument that you really want your database to play dumb. […]
pingback at 04. March 2006
Madeblog » ActiveRecord Angst… ReActivated.:
[…] It’s LAMP right? Not LAPP, LAOP, LASSP or LADP. Despite the accusations regularly levelled at MySQL, I think that’s what most of us use and if you believe in getting everything through your ORM then there’s a good argument that you really want your database to play dumb. […]
pingback at 08. November 2006
Balance On Rails : David Hansson e Opinionated Software:
[…] Atualização: Alex Bunardzic tem uma excelente sequência entitulada Devem os Bancos de Dados Gerenciar o Significado? – fique ligado. […]
pingback at 05. December 2006