The Middle Tier Manifesto: An Alternative Approach to Development with Microsoft SharePoint
Author: Marc D. Anderson
http://mdasblog.wordpress.com
For some time now, I’ve felt the need to set down my thoughts on the power of development in the Middle Tier for SharePoint. Today, I’m publishing the first edition of my white paper The Middle Tier Manifesto: An Alternative Approach to Development with Microsoft SharePoint. I say ‘first edition’ because the days of spending a lot of money to print a white paper and distribute it widely are long gone. I expect to and want to develop these ideas over time based on input from you, the SharePoint community.
In this white paper, I lay out the methods and rationale for preferring to develop for SharePoint using SharePoint Designer and a combination of the Data View Web Part, scripting, and CSS over managed code. For quite a long time this is where I’ve focused my development efforts for clients and was part of the genesis for the jQuery Library for SharePoint Web Services (SPServices). I find that developing in the Middle Tier using SharePoint Designer can be faster, more reliable, and cheaper than the managed code approach.
I expect that some of you may well disagree with this premise and I know that others will absolutely agree with it; I welcome the debate. Take a read of the white paper and let me know what you think.
Thanks to Michael Greene (@webdes03) and Jim Bob Howard (@jbhoward) for their input on early drafts.
Author: Marc D. Anderson
http://mdasblog.wordpress.com
Marc D. Anderson is a Co-Founder and the President of Sympraxis Consulting LLC, based in Newton, MA. He has over 25 years of experience as a technology consultant and line manager across a wide spectrum of industries and organizational sizes. Marc has done extensive consulting on knowledge management and collaboration and what makes them actually work in practice. Marc is a very frequent “answerer” on the MSDN SharePoint – Design and Customization forum.
The “Middle Tier” definitely offers opportunities to be able to swiftly roll out “quick-wins” for a project.
The tools used (SPD) work remotely as well, so there’s something for remote users who may have a multitude of development issues working remotely – a “Sharepoint Designer” can use SPD over a VPN with a little lag but is very usable given a good network connection.
Very cool Marc! There’s a strong business case here in the high ratio of value to effort. Full server side work can be slow with multi-party with delays and change approvals (+costs). However, the middle can often be instant (click save) and more agile.
Keep the good ideas coming! We’re all listening and appreciate the documentation help.
I agree that there is value to be gained from what you call ‘Middle tier development’, which from a developer’s point of view is nothing new technology wise (it is after all CSS, JavaScript and html).
I do take issue that this is the solution to managing important business data. You touch on some of my main concerns:
1. SharePoint designer cannot produce reusable components (at least not in MOSS 2007). By this I mean you need to manually reproduce your steps through your test environments, all the way to production. Any time tedious processes need to be followed by a human, mistakes happen.
2. Code management, storing code in CEWP or a list is not the same as version controlled software such as TFS, a proper development process needs to have change tracking, not so much copies of files. Like the steps above, a manual process to deploy to each environment needs to be followed, a whole bunch of moving parts. On the flip side a well developed wsp needs only some features to be activated .. very quick and easy to repeat by anyone. Don’t under estimate the tacit knowledge needed to deploy ‘middle tier’ components.
I have to completely disagree with this statement in your document:
“Middle Tier solutions are far more adaptable in a large corporate environment
than the deployment of managed code solutions.”
So we are taking about a large corporate environment, given the concerns above, I can’t see this being true at all. ‘Middle tier’ solutions might cut it for keeping a list of lunch orders, it’s simply not going to work for anything more serious for all the reasons mentioned above.
I also disagree with how you trivialize testing:
“Because nothing is deployed to the server, testing can be done in a much more ‘unit testing’ mode”.
Stuff is deployed to the server, all that JavaScript you’ve pasted in a CEWP. Or maybe you’ve added it to the master page? In that case have you tested it across your whole site? You may have pasted code that pollutes the JS global namespace and causes errors on some pages. The fact is your trying to say “look I’m really not doing any development, so I don’t want to waste time testing”, which is fine for your lunch order site, but not for a large corporate install. Your still doing development, so follow development practices. I won’t touch on your ‘Unit test” misconception, I figure your not a developer anyway (or you wouldn’t have called it ‘middle tier’ development, it’s not, its client side development).
Your whole argument revolves around ‘Middle tier’ development being quicker, therefore cheaper.
Headings like ‘Faster Development lifecycles’ under the context of ‘Middle tier’ development sends off warning bells to me. Its says you don’t want to follow best practice around deployment and testing. Lets face it, you’ve already read the stuff above and dismissed it as baseless. You want to cut corners. If everyone does this in an organisation, you will soon end up with half baked solutions that become brittle and hard to maintain (if the organisation even knows about it), a proper governance model shouldn’t allow it.
To me it seems like people are preferring these approaches because they haven’t been exposed to good SharePoint developers. Touching system files shouldn’t be a concern with a good developer, so concerns around this are baseless. So isn’t the big issue the fact that good SharePoint devs are hard to find?
I really do understand what your trying to do, you feel empowered but don’t want the responsibility of deployment and proper testing. Its fine for little ‘toy’ applications, but just doesn’t cut it for serious applications. Overall this is a good place to start a proper dialog around ‘Middle tier’ solutions.
Jay:
Thanks for your comments, and no, I don’t find them baseless at all. Every situation is different, and what I am trying to point out is that there is a hugely useful set of options that many business people and developers are unaware of. I’m not trying to say that these methods are new, just that they tend to be overlooked. When I’m using the term Middle Tier, I’m capitalizing it for a reason. I don’t intend it as the generic ‘middle tier’ you may learn about in university, but as something specific to SharePoint.
Every organization has different needs, and no one approach ought to work for everyone. I’ve consulted to organizations using SharePoint which are anywhere from a one person services firm using WSS in the cloud to a tens of thousands person global bank running the Enterprise CAL. The approach in those two extremes damn well better not be the same or no one is getting anything that they want.
The Middle Tier methods ought to be one set of methods in your portfolio, as I point out in the white paper. Every business requirement should be looked at against the set of available approaches in the portfolio and the optimal approach should be used in each case. Sometimes the answer ought to be notecards, other times SharePoint, and other times it ought to be some other toolset altogether. When it *is* SharePoint, there should be more options than “We have to write managed code.”
Speaking of cloud computing, the Middle Tier is going to be even more important as more and more organizations move their use of SharePoint to the cloud. With SharePoint 2007 (WSS *or* MOSS) and cloud hosting at the low end, there is zero capability to touch the server on the back end. The Middle Tier methods are all that there are to do real customizations. I expect that this will increasingly be the case. In these situations, the Middle Tier ought to have extreme usefulness.
SharePoint Designer cannot produce reusable components (nor does Visual Studio or Notepad, for that matter), but the people using it can. In all cases (and yes, I’ll be absolute here), it’s the people using the tools that either get it right or wrong. With the right mindset, tools, and practices, anyone can create reusable objects with SharePoint Designer. I do it all the time.
When we deploy script in a CEWP or in an aspx page, we aren’t deploying it to the server, per se, but into the database. That script is therefore ‘content’ as opposed to ‘code’ in many ways. I can write script that damages SharePoint sites and content (assuming that I have the permissions to do so), true, but I cannot damage the server file structure itself if the server is set up and administered correctly. (There’s another place where people doing what they ought to is critical.)
As for my credentials, they are available in many places (see the About page on my blog), but let me give you a synopsis. It’s very possible that I was writing code before you were born. My first paying programmer job was in 1983. I’ve worked for quite a few world-class companies, and I’ve probably spent time working in the IT departments of far upwards of 100 different companies at one time or another through my consulting. I’ve also managed real-time 24×7 mission critical systems. The best example of this is my role at Staples in the early 1990’s when I managed (and developed on a daily basis) the systems that ran the delivery business, which included everything from catalog management, order taking, warehousing, fulfillment, to billing, etc. In short, the whole ‘ball of wax’. Through all of this experience, I’ve seen the good, the bad, and the ugly when it comes to software development.
The only reason I’ve raised to the bait on my credentials is to say that I believe that I know what I am talking about. In my experience, overly formalized systems (the generic term ’systems’) lead to stagnation. Having a great, formal SDLC *can* mean that nothing of use ever really gets done. I’ve seen this time and time again in large organizations with their approach to SharePoint development. I’ve also seen solutions that went through the full, formal SDLC that were absolute crap. I’ve seen real business needs go unmet because the SDLC made it impossible to produce a useful result in any sort of reasonable timeframe or at a reasonable cost.
Finally, you are absolutely right that good ‘SharePoint developers’ are hard to find. Good *developers* are hard to find. People who are truly good at *anything* are hard to find. All skill distributions follow a bell curve, and everyone wants to think that they sit in the top tail. That’s statistically impossible; the large majority of people will be clustered around average. We all want to hire the people in the top tail and pay what we’d pay someone in the bottom tail. Too often, the reverse ends up being the case.
Again, thanks for your comments, and I hope my additional comments help to clarify what I’m trying to say in the white paper. There’s a reason I wanted it to be the first edition. I wanted to hear from folks like you to get your reactions so that I can hone the messages. Then people who want to really get valuable work done can take the white paper to their IT people and see if they can’t come up with a way to make good things happen for their organizations.
M.
I totally agree with this Middle Tier way of thinking.
Developing everything using C++ or other such methods is not alway the best or fastest way to do things, especially in SharePoint.
One of the key things about SharePoint is that a well trained administrator, or high end user can do much for themselves before they must rely on IT. And that’s good for IT, as well.
I wanted to have a way to color a calendar, and color a line in a list. I went to our IT developers for a solution. I was put in a que and waited for 4 months before they accomodated my request. Other requests are still waiting after 2 years.
Lo and behold, I found a way to do what I wanted on my own by reading info from this site and others – all Middle Tier solutions.
I put them in place without using SP Designer. What code there was, was in Jquery, or other client side coding I could copy and deploy without harm to SharePoint, and without involving IT.
When I showed the developers what I’d been able to accomplish, they brushed it off and left me with the feeling it was beneath them. Yet, they never came back with their solution to help me.
Is this a common attitude with the developers community, or is it just isolated?
All I know is I’m thankfull to the true (felxible) developers on sites like this one that seem to truely care about the thousands of us non-developers out here who have to fend for themselves much of the time.
Thanks, Mark for all you do.
Tom
I think Marc has addressed all of your comments very well, but I’d like to expand on the corporate environment portion, as that was one area that I did suggest Marc include in the white paper, and I’ll back up my point with an example, as I very much disagree with your point that middle tier = not wanting to follow best practices or proper testing.
I was recently building a portal for an organization in a large corporate environment where one of the requirements was an expanding and collapsing navigation (ie: click the parent topic, topic expands and gives me links to everything under it); I think we’d all agree, very simple.
Now, I could go upstairs to one of our .NET developers with a charge code, and pay the burdened overhead rates of a corporate .NET developer to create a web part, then fight with the corporate server admins to get that web part deployed, but why do that when I can open SharePoint Designer and using out of the box capability offered by Microsoft, create my own DVWP with the custom XLST to perform that task? To suggest that those capabilities are any less stable or secure than a WSP packaged .NET solution is totally wrong. I can spend 20 hours of burdened overhead for a .NET solution, or I can spend 2 hours of standard labor for myself and be done, with a solution that meets spec, and is no more or less stable or secure than an out of the box list view.
The same thing applies to dashboards. Could you create a custom web part or feature to dashboard a few metrics? Absolutely, but it’s far less adaptable and far more expensive than using XLST and a DVWP to do your own analysis; and when my director calls and says “add this metric before my 3:00 meeting”, I don’t have to wait for the .NET developer to update, repackage and redeploy. The nature of middle tier solutions is that they’re adaptable. I practice change management, I log all of my changes, and I test my code application wide in all supported browsers prior to deployment.
I see lots of references to CEWPs, but lets remember that the middle tier encompasses much more than putting JavaScript inside CEWPs. We’re talking about the use of SharePoint Designer as a catalyst to solution development, the use of web services, and (a hint of) a paradigm shift in the way we look at operating. jQuery and JavaScript, as wonderful as it is isn’t all we’re talking about here.
To suggestion that important data shouldn’t be handled by the middle tier approach has no validity what so ever. If my environment is secure, my code doesn’t have vulnerabilities, and I’m good at what I do then there is no issue. Microsoft themselves could have prevented DVWPs, web services, etc. and forced everyone to deploy .NET solutions from Visual Studio for everything, but they didn’t–because it’s just not practical.
Now every developer is different and I’m not going to say I’ve seen it all, but I’ve seen a lot and I’m very good at what I do. Others aren’t fortunate enough to have the background that many of us do; and there is the risk. The Middle Tier is a movement, and hopefully we can show some of our seasoned .NET developers that despite the fact that it’s not .NET there is some validity to what we’re doing, while educating those less experienced about it and how to properly apply the techniques.
To suggest that “[it's] nothing new technology wise (it is after all CSS, JavaScript and html)”, I would flip that right back around and say that your .NET is nothing new, after all it’s .NET. At the end of the day, it’s not the language, or the toolset, or the mindset as a single node, it’s how the skilled developer can utilize each particular element to create the best overall solution for the customer; following the necessary testing, the change management, the budget, and the business rhythm.
It’s always good to have alternatives or options. At the end of the day, it’s what meets a clients needs. They are not concerned about what’s under the hood.
We as developers tend to get or be pidgeon holed – he’s a .net guy , she’s a vb.net person etc. I prefer to have many tools in my toolbox. This brief manifesto was well presented and provided a ‘thinking out of the box’ approach.
I can just say these methods have been highly effective for me. I implemented a number of solutions in a 40′ employee corporation SP environment and it has worked fine so far. Integrating non SP webservices/databases/Sharepoint lists in a DVWP or with SPServices works really smooth. Most solutions/web parts have been made highly reusable with html, js and xsl-files centrally stored and then used in hundreds of sites/site collections. I’m also using my (still very ugly) admin tool to push out webparts to a large number of sites http://spmsaui.codeplex.com/
One big advantage has been the time it takes to get updates to different solutions/webparts. When working with the SP environment used by 40’ people it’s very strict what can be deployed directly to the servers. Everything works in monthly cycles. Development of “Mid tier” solutions has a much lower probability (at least I think/hope so…) to screw up the whole environment. Because of that a bigger number of people can be allowed to “deploy” their own solutions independently of the normal one month cycles.
…just hope this approach won’t stab me in the back one day…
Again I say that Jay is barking down the wrong community. While your points are valid, you are coming in with an obvious structured developer approach to Sharepoint customization. This community is not about stereotypical development. If that’s your game, there are tons of other outlets for that.
Back in January there were some heated debates about the use of jQuery and whether it had any place in a solution provider’s toolkit. I wrote an article about SharePoint Dev and Customization Paths (http://nextconnect.blogspot.com/2010/01/sharepoint-dev-and-customization-paths.html) and I think it applies very well to this discussion. I believe in having options and applying the correct path for a given problem. Not every problem should be built with managed code, and in some cases or environments ad-hoc customizations are not appropriate. More than anything the architect needs to be educated in the decision points and needs to work to help educate the decision makers.
I have been in a few environments where there was a strict rule that there can be no server solutions. To me that is just as wrong as an environment that promotes a strict rule of no SharePoint Designer customizations.
Understand the problem, and select an appropriate solution. Don’t let your preferred approach dictate a solution.
While I cannot speak on development cycles since I’m not a developer, I can add my 2 cents in about the “Middle Tier”. Plain and simple; Unlocking the mysteries of the DVWP series provided my Marc has simply changed the game for me. Ever since following and referring to the series, I have been able to jam out solutions that are running my company’s project workflows. Since this company isn’t considered an enterprise, then you may say they are “toy applications”; I say “They provided a solution.”
From what I’ve read over the internet and experienced myself, I’d say the Enterprise makes up of about 20% and the rest of it being small “toy applications” (Granted 73% of all statistics are made up on the spot, ohhhhh, let’s say about 89% of the time ^_^) If the market only needs small applications, then it would make sense to make use the great tools available for client side development. Again, that’s my opinion and I’m only referring to my experience. So far, I’ve been able to *NOT* deploy any custom code to my server and use all OOTB tools and I’ve managed to build all of our business intelligence into SharePoint. My point really is, if you are a developer, you have to realize what is available to you already. I feel like Marc’s Manifesto brings up some great points. So many, I’ve had to read it a few times…
I may be wrong but I think using a DVWP, you can make any data you have stored in SharePoint, do anything you want. Why would you not want to use that approach?
Oh it would be so cool to be able to use the middle tier more often! Unfortunately for me, I am stuck in an environment where we are not allowed to use Sharepoint Designer on our production network because it was deemed a security risk with no real details as to why. We also have to ‘develop’ packages that can be used by other remote organizations. It is much easier to build most of what I do in Designer but then to try to import that into Visual Studio just seems wrong in so many ways. I am not the best .Net coder by any stretch of my imagination, but I try to keep things simple as much as possible and so far that has worked for the most part. Luckily in my case, we have access to the server farm and we try our best to make solutions that have a small footprint on the server. Unfortunately this model leads us to create solutions that we have to stress to the customers are not very changeable once deployed. Not that changing the solution is not possible but redeploying solutions with your fingers crossed is just not fun!
Never tried, but isn’t it possible to create data view webparts on your test environment and then import them to your production site in the web interface even if SPD isn’t allowed?
You can import if you’ve followed some rules with your DVWP development. It’s not a perfect solution, but it can be a helpful one.
M.
As long as you refer to ListName instead of ListId (or figure out the list id manually) it should be fine?
Generally, yes. Since you can do just about anything in a DVWP, you could cause yourself other problems, too, e.g., absolute vs. relative links.
M.
Yes, I have done this a few times but it still has the same issue of having to redeploy whenever a change is needed. For the most part I just have to write custom webparts if I need them. Mostly I am trying to learn how to override the default form templates which is working well so far.
Marc you have put together what no one has tried to do and I and I know others appreciate it. There is a time and a place for everything and you showed that. Thanks for taking the time to write this. I am even passing this along to our SP Governance team.