Prologue
So I was working for a company a few years ago. The company had been around for a while, and had a bunch of genuinely intelligent senior developers working for them.
Over the course of several years, inevitably a set of common problems arose. Writing data access code was repetitive and error-prone. Many teams were sharing the same database but using different code, so there were inconsistencies in how similar business data was treated. Each application’s log files where all over the place, and each had their own approach for error handling. Some teams would go off on their own and unnecessarily reinvent the wheel. Moving a developer from one team to another required as much ramp-up time as a new hire. Each team used a different build and versioning strategy, with the most common strategy being “none.” Setting up a test environment with multiple applications took days. Recreating the production environment was virtually impossible. Chaos. Dogs and cats living together. Mass hysteria.
To address these issues, some of the more senior architects took it up themselves to build a framework that would greatly simplify everyone’s life. By putting in a little design upfront, they could build a framework layer that would solve many of the problems that the developers had been muddling through over the years, while at the same time homogenizing the code base.
Of course, the company had attempted this before. In fact, there were several previous frameworks over the years. But those previous frameworks were not as good as they could be, either because of design flaws, or changes in the way that the company’s applications work, or because they used outdated technology, or because the previous no-longer-with-the-company designers were now generally considered to be idiots. Anyhow, the new architects had learned from these mistakes, and were designing a new framework that would do a much better job of solving the problems. Due to the scope of such a project, it had been a work-in-progress for about a year. Sure, they were still working out some kinks, and it was not completely finalized yet, but this is a technology investment, and some growing pains were to be expected.
Déjà vu
Well, OK, I lied. This wasn’t one company I worked for. It was several different companies, all with the same story. In fact, it’s a little eerie how much you see this exact scenario play out at companies all over the industry.
Some senior developers have identified some recurring pain points for the developers, and they want to do something about it. As the company has grown, more and more developers have come on board, each with less and less experience, and things need to be brought back under control. By providing a framework, you can lay out the boundaries for developers to operating in, which will encourage consistency, will encourage code reuse, and in the end will allow the company to produce higher-quality software in less time with fewer developers, which will require less maintenance cost over time. In other words, it will pursue the single ultimate goal that should be at the center of every design decision, namely that which will advance the overall long-term profitability of the company more than any other option.
It sounds like a brilliant idea. And if it were to be accomplished, it would be great. But the unfortunate truth is that it doesn’t work. Without exception, I have only seen this result in more work for the developers, longer development cycles, more bugs, poorly compromised designs, and (worst of all) excessive, unhealthily conflict between the development teams.
Admit it, you’ve seen it too. Maybe you were even involved.
So What’s The Problem?
So why does this go wrong?
The problem is usually not intent. I don’t want to make it sound like the people involved are morons or that they are have any desire to do harm to their company. In fact, usually they are excellent developers who are trying to do the best they can to solve a problem. I can’t fault them for that. I just think the approach is a little misguided.
And the problem is not the people down in the trenches, pushing back on every change the framework team wants to introduce. No, these people are trying to get a job done. Their marching orders are not to solve the whole company’s crosscutting problems, but to ship their product on time and in budget, and many of them believe, perhaps rightfully so, that the framework keeps them from doing that as efficiently as they could.
Again, the problem is the approach.
The Challenge Of Frameworks
So what is a framework? Generally, people think of a framework as something that helps you get your job done by providing access to new functionality that you didn’t have before. This is usually the selling point, used when convincing a manager or developer that this is all such a great idea, but the reality is that the true nature of a framework lies not in what it helps you to do, but rather in how it limits you.
For example, the Slightly-Almighty, Moderately-Informative, Usually-Reliable Wikipedia says:
A software framework, in computer programming, is an abstraction in which common code providing generic functionality can be selectively overridden or specialized by user code providing specific functionality…
Software frameworks have these distinguishing features that separate them from libraries or normal user applications:
1. inversion of control – In a framework, unlike in libraries or normal user applications, the overall program’s flow of control is not dictated by the caller, but by the framework.[1]
2. default behavior – A framework has a default behavior. This default behavior must actually be some useful behavior and not a series of no-ops.
3. extensibility – A framework can be extended by the user usually by selective overriding or specialized by user code providing specific functionality.
4. non-modifiable framework code – The framework code, in general, is not allowed to be modified. Users can extend the framework, but not modify its code.
One of the key points here is that the framework is dictating the application flow, rather than the developer who is using it. This is what the Martin Fowler (who literally wrote the book on refactoring) would describe as a Foundation Framework:
A Foundation Framework is … built prior to any application that are built on top of it. The idea is that you analyze the needs of the various applications that need the framework, then you build the framework. Once the framework is complete you then build applications on top of it. The point is that the framework really needs to have a stable API before you start work on the applications, otherwise changes to the framework will be hard to manage due to their knock-on effects with the applications.
While this sounds reasonable in theory, I’ve always seen this work badly in practice. The problem is that it’s very hard to understand the real needs of the framework. As a result the framework ends up with far more capabilities that are really needed. Often its capabilities don’t really match what that the applications really need.
He recommends instead a Harvested Framework:
To build a framework by harvesting, you start by not trying to build a framework, but by building an application. While you build the application you don’t try to develop generic code, but you do work hard to build a well-factored and well designed application.
With one application built you then build another application which has at least some similar needs to the first one. While you do this you pay attention to any duplication between the second and first application. As you find duplication you factor out into a common area, this common area is the proto-framework.
As you develop further applications each one further refines the framework area of the code. During the first couple of applications you’d keep everything in a single code base. After a few rounds of this the framework should begin to stabilize and you can separate out the code bases.
While this sounds harder and less efficient than FoundationFramework it seems to work better in practice.
I’m not sure I would even call this a framework, because all of the things that make it work best are the parts that take it further and further from being a conventional “framework”.
So Are All Frameworks Bad?
Sweet suffering succotash, no. In my humble opinion, the .NET Framework is a thing of beauty. Back in my Win32 C++ days, MFC was not perfect, but worked serviceably well for what it was intended for, namely abstracting away the Win32 API. CMS frameworks like DotNetNuke and Drupal and Joomla have been become very popular. Apparently there is a subset of people who don’t hate Flash with a passion, and apparently those people love it. MVC frameworks like Rails and Django have caught on like wildfire, with ASP.NET MVC picking up a lot of momentum as well. Microsoft Azure and Google AppEngine are in the process of changing how we will build scalable cloud-based applications into the next decade.
Have you noticed a pattern here? None of them were built by you or anyone you know. They were not built to solve a business need, they were built to reinvent a platform. They were not built to get everyone using a “Customer” object the same way, they were build to make it easier for you to do whatever you want with whatever data you need. They were not built by 3 architects for 20 developers, they were built by 30 or 300 architects for 20,000 or 200,000 developers or more. They were not designed and built and delivered and completed in a few months, they were talked design and dog-fooded and tested and tweaked and redesigned over years by some of the smartest computer science experts in the business. Any yet, despite all that, most of them still sucked, and the ones we use today are the select few that survived.
The thing is this: you and your internal development team of architects are not going to build the next great framework. You’re not going to build a good one. You’re not even going to build an acceptable one.
And the other thing is this: If a framework is not great, it is awful, counterproductive, and destructive.
Get In Line
By definition, most frameworks try to define a new way for your developers to develop software. The keep you from doing certain things that have been seen as problematic, and require you do things the “right way”, assuming of course that the architects have actually thought through the right way to do things.
The problem is that there are plenty of good ways already, ways that those developers are already trained in and have spend years mastering, and you are not really as clever as you think you are. You can’t think of everything. Even if you could, you can’t design for everything. And even if you could, you shouldn’t. Trying to shoehorn them into an incomplete, shoddy, and unnecessarily restrictive framework will only breed resentment, at which point you are bleeding money that you’ll never see on an expense report. The productivity difference between a happy developer and a disgruntled developer is enormous, and constantly underestimated. You will also alienate and drive away all of your good developers, leaving you only with the not-so-great developers that really don’t have any better options.
Atlas Shrugged
In order to create a framework, you are taking on a massive responsibility. It’s not a case of adding a few features. You are building an entirely new layer of abstraction. In doing so, it is your responsibility to ensure that your framework provides the developer with every possible thing he will need, otherwise he will be stuck. If you create a data access framework, but never quite could figure out how to get BLOBs working, you’re really leaving the developer up a creek when he needs to store BLOBs. Sure, it’s a growth process, and there are always bugs to be fixed and features to be added, but when you are forcing a development team to use this framework, and in the 11th hour they realize it doesn’t have a critical feature that they need, you are introducing more obstacles then you are removing.
But We Have To Create Reusable Code!
No you don’t. Reusable code is bad.
So What?
So do we give up? Maybe. It depends.
So how do you boil the ocean? There are two answers:
1. You don’t. It’s too big of a problem.
2. One pot at a time.
It all depends on your goal. Is it critical that you actually boil then entire ocean, or do you benefit from every little bit?
Ask yourself, do you REALLY need a framework? Do you REALLY have a problem here that needs to be solved? Do you REALLY think you will save your company time and money by pursuing this? Be honest. Try to be objective. If you find yourself getting the slightest bit excited about the idea of building a framework, recuse yourself from the decision because you are not thinking clearly.
Sure, a well-design framework may save time and money once it is complete, but it may never be complete, and it may never be any good, and the small improvement may not save your company enough to justify the huge expense. As awful as it may seem, the honest answer may be that it is in your company’s best interest to plow ahead with the same busted up junk you’ve had all along. It may not be the most rewarding thing in the world, but you are probably not getting paid to fulfill your dreams, you are getting paid to write the damn code.
So what do we do? Are we doomed to mediocrity? Not necessarily. The other option is to get your head out of the clouds and solve a few small problems at a time. Keep your eye out for redundancies and duplicated code, make note of them, but don’t do anything right away. When you are building a new component, don’t pull your hair out if it slightly resembles some other code, you don’t have to reuse use everything. Once you’ve identified a few low-hanging redundancies, go back and build some small libraries to consolidate that code. Don’t think big picture. Keep it simple. Keep it low-impact. Keep it clean. Put the big guns away. Keep constantly refactoring to make it a little better every day, and before you know it you’ll have system that doesn’t completely suck.
Too much legacy code never gets cleaned up because everyone thinks it is too hard to throw it all out and rewrite it, and no project manager will allow the developers to waste time refactoring code that already works. They are all probably right. A huge refactoring project is probably a waste of money, and it will almost certainly fail. But small, steady, incremental improvements will almost certainly make your world a better place.
Many years ago I harvest a framework, except I didn’t know I was doing it, I thought I was just generalizing common tasks.
Twice I have attempted full blown frameworks based on prior experience and have lost money both times. No more of that.
But it sure is tempting…
Well said. Particularly liked “So Are All Frameworks Bad?”
These buzz-phrases are usually good ideas:
* Build products, not frameworks.
* Buy don’t build.
* Don’t re-invent the wheel.
Pingback: Tweets that mention The Mooney Project » The Framework Myth -- Topsy.com
Well put. Reusable code is overrated a majority of the time. The focus should be on creating well designed software that is easily maintainable. Much more software can be built in a shorter time frame this way.
Taking extreme positions on software topics is very rarely useful in my opinion.
Both this and previous blog posts are mostly silly in their outlook – some frameworks are excellent, some are good, some are bad and some are ugly.
I’ve worked in teams of six/seven people where there is a lack of communication over what is being written, and the duplication (and minor variations on a theme) gets counter-productive very quickly. Having someone step in and produce a framework that people can unify there work within, forces communication to happen, speeds and smooths development.
It’s almost irresponsible to make blogposts this unnuanced :/
Hi Andrew, thanks for the feedback, I’m glad you enjoyed it! 🙂
My experience has been different, the more extreme my positions, usually the more succesful I’ve been, both my own productivity, and in convincing others to rethink their positions on the same old conventional wisdom.
Wonderful article, it articulates very well what I’ve experienced in the last decade. Too often developers ( especially young ones) are looking to create systems instead of solving the problem.
Too often I’ve seen these pointlessly complicated frameworks take way too much overhead time to develop and maintain and then don’t allow the flexibility to solve the “next great problem” and you’re left either changing the framework or trying to figure out how to get around it.
“I’ve worked in teams of six/seven people where there is a lack of communication over what is being written, and the duplication (and minor variations on a theme) gets counter-productive very quickly.”
A skilled project manager sounds a lot more useful than a framework with team that small.
@Andrew, a group of 6/7 people is different from 10 groups of 6/7 people in a single company. A single group in companies like this have no business creating frameworks for everyone else. I think this post makes that point quite well.
Maybe it only works when the framework is opensourced and used, then you get enough eyes and people using it.
Pingback: myninjaplease
A M E N
.NET is not a library not a framework. Compare it to the Wikipedia description: it doesn’t meet any of the four criteria.
My experience has taught me to be extremely wary of using or creating anything called a framework. Luckily for me I’ve realised this soon enough to have embarked on making any serious frame works.
I prefer modular systems, that would be acceptably insular, unobtrusive packages. For example, when programming for a web front end, I prefer jQuery over mootools. Many frameworks claim to have some OO, flexibility, modularity, but what your really have is a huge web of intense interconnectedness that is inhibiting. You can’t rip it apart and take only what you need. Subunits of frameworks tend to be about as independent as an embryo. There is far too much glue. Do you know how it feels when people use these? It feels like when you are watching someone walking through a machine room, and they have unwittingly caught themselves on a cable, and you’re calling them to stop, but it’s too late, they are pulling everything down, things are smashing, its a disaster.
I’ve rarely come across something called a framework that isn’t monolithic and that doesn’t do far too much often practically trying to rewrite the language providing something entirely new and unnecessarily alien.
My point here, is that using a third party framework can be even worse than making your own.
You’ve written the article I was going to write for the last 5 years, which, coincidently, is when I stopped using my own framework as was forced to use someone else’s.
Thank-you for that.
You say:
“They were not built by 3 architects for 20 developers, they were built by 30 or 300 architects for 20,000 or 200,000 developers or more”
With regards to Django, which you mentioned, it was originally built by about 3 developers, for themselves to use. It may have been refined by dozens of core developers since, but it was already useful before that.
Django is much closer to an example of a harvested framework than a foundation framework – it was pulled out of existing applications, and still bears some scars from that process. It is the foundation frameworks like .NET which have the bigger teams, AFAICS. Harvested frameworks can be just as successful if they come from small teams.
These topic can be applied to A.P.I.’s (Application Program Interfaces).
Pingback: Mythos Framework » Bananas Development Blog
You left out a huge component as to why using a (third party) framework can be really good – if a framework can restrict programmers to a general pattern of development, then you are creating an external standard by which developers can agree on using, rather than bickering about trivial things and personal preferences. It also helps when you are hiring to find someone, for similar reasons. But you really have to make sure your pick of framework is a smart choice and that you just stick with it, once adopted.
Our flagship product was initially inherited with a mess of different libraries, frameworks and patterns. We made the choice of developing our own internal framework and I have to say, while it was not ideal and had many of the flaws you mention, it was worth it, because it allowed us to prototype our apps quickly in an envirnment of uncertain requirements. Eventually, we figured out which real framework would suite us best and we adopted that to much success. Sometimes it’s good to write lots of code and then throw much of it away. Code is cheap but solid design is expensive.
Not sure I agree with the post. The approach I think works best is:-
– do I have a problem I can identify?
– is there an existing and active open source project which does what I need?
– can I hire/re-train developers to use it?
– if nothing open source, look to buy
– if nothing buyable that saves us money in the medium to long term, then build application without a framework
– As application is build look for parts that can be harvested.
– When application is finished, evaluate what you have and whether it can be made into a framework.
– Harvest framework as per Martin Fowler’s article if that’s what is appropriate for you.
There are some glaring problems with this analysis.
MFC does not abstract the Win32 API. MFC is the best example of an anti-library, i.e. a library that does not hide any of the details of the underlying API.
Code reuse is extremely good. If we could not reuse code, then we would have to spend ages coding everything over and over. Thanks to code reuse, I can now built an application quite easily, due to the available libraries.
I don’t completely agree to this post, but I think there are very good points.. I have mixed experiences developing frameworks. IMHO, if everybody who is using/modifying/working with the framework has to genuinely understand the philosophy and the intent behind the framework. This has to be preached to all the developers and had to be documented well. If it is done properly, frameworks are not always a bad idea.
This seems eerily familiar, like we worked on the same team in the past.
Good article. Thanks the Fowler sections, I didn’t know where the idea of a Harvested Framework came from but I should have figured.
Pingback: links for 2010-12-10 « dstelow notes…
Pingback: The Drupal and WordPress showdown « Web Weekly
Pingback: The Drupal and WordPress showdown | Web Weekly
We use a simple rule to decide whether we should factorize code and build some reusable piece of software or not: if number of potential or actual reuse is less than a hundred then drop the idea.
This rule comes from the fact that writing a clean, documented, intuitive, robust, maintenable, scalable library is 100 times more complex and costly than hardcoding it.
Of course, the one hundrerd threshold is to be adjusted upon various factors, but we found it reasonably true in average.
Laurent
Actually, I’ve found that “harvesting” or, as I’ve more often heard it mentioned, “extracted” frameworks work extremely well. They do so partly because only the stuff that is actually useful to all the common applications gets extracted, e.g. if the applications you’re developing use very similar patterns to process incoming mail, extracting the best implementation and make the other apps use that will mean you only have one code base to maintain. The important part is that you limit the scale of the “frameworks” to just the things that would obviously benefit.
I’m from the Ruby world, but I’ve worked a lot with .NET as well, and I think the difference in culture is important to this subject. In the Ruby community, if you publish your microframeworks as open source software on GitHub, you’ll get an amazing level of contribution and feedback. This adds to the other benefits of factoring out code into small, reusable frameworks, as improvements and testing is greatly enhanced.
Pingback: Kaliatech 42 » Blog Archive » Weekly Random: #20
You’ve identified the underlying cause quite accurately in paragraph 2 of the “Deja Vu” section:
“As the company has grown, more and more developers have come on board, each with less and less experience…”
I have a nagging suspicion that the rest of the article, interesting as the “framework question” always is, speaks to issues that are second-order to this one.
Pingback: Your Framework Sucks « CodeHouse of Horrors
Pingback: You Arent Gonna Need It – Code Reuse — Numiko Labs
Great points and well written. You’ve clearly been around. Thx
Very inspiring!
Trick question: what do you do with a foundation framework that is both awful in terms of usability and performance, but where zillions have been already invested in using it (all in an environment without proper tool support for refactoring and dependency management)?
Invest years into refactoring?
Build something new from scratch?
Kill yourself?
Any non-radical ideas?
“You’re not even going to build an acceptable one”. I love it. All CTOs should read this so much time, effort and money is wasted writing “our own killer framework”.