21
Aug
2011
Bill Blondeau

When Acronyms Collide: SOA vs. OO

XML Today has a longstanding tradition of approaching Service Architecture largely through the narrow lens of the SOAP vs REST debate, coming down strongly on the side of the latter in most cases. Now, as the site transitions to a wider scope, less focused on XML as a technology and more on the growing ecosystem of data representations in which XML is only one key player among many, it's probably a good time to have a look at the representation-neutral fundamentals of Service Architecture.

This article looks at a particular aspect of the current state of enterprise IT: the entangled relationship between the mindset and practices of large-scale Object Oriented application development, and the requirements of Service Oriented Architecture. This relationship has gone wrong and will remain a problem - a costly problem - until it’s put right.

Overview: A Rescue That Hasn't Worked Out

Quote:

We can't solve problems by using the same kind of thinking we used when we created them.
-Albert Einstein

Driven by the momentum of its own success, the Object Oriented programming paradigm is reaching some hard limits on the kinds of problems it can solve very effectively. In particular, scaling monolithic OO applications up beyond a certain point yields diminishing returns, in a way not anticipated by OO theory. Various remedies (such as Application Frameworks) have limited effect. Meanwhile, the business drivers of the enterprise keep requiring bigger, more interdependent applications.

Service Oriented Architecture, when it was invented and proposed, offered a decisive way out of that trap, but rather than fixing the situation, was itself largely disabled by the internal dynamics of the IT industry. To put it colorfully:

  • OO got into some quicksand, with no help to be seen;
  • SOA unexpectedly appeared and ran to the rescue;
  • OO pulled SOA into the quicksand too;
  • OO said, "I planned that. Keep swimming."

SOA is something new and different, with unprecedented capabilities; but it has its own logic and its own requirements. If you treat SOA as merely a different way to conduct the business of OO, it won't go well. Unfortunately, aspects of IT culture have discouraged treating SOA as anything else.

This article surveys those cultural features, recapitulates the individual troubles of SOA and OO, offers a technical explanation for SOA's potential to fix OO's trouble, describes how this hopeful path wasn't usually followed, and provides some serious suggestions about how to use SOA properly to get projects - and organizations - out of the quicksand.

Preliminary: What’s a Service?

“Service”, in the field of information management, is one of those inkblot words. Different people, in different contexts, see different meanings in the term, and make different assumptions. This leads to a lot of IT professionals talking past each other (even more than we normally do, I mean.)

Accordingly, I’m putting a stake in the ground here and defining Service for the purposes of this article. The definition will probably see a lot of other uses in other articles too, going forward.

  • A Service Contract is a specification for message exchange between two parties.
  • The messages are exchanged according to a declared request-response pattern.
  • The parties to the exchange have distinct and asymmetrical responsibilities: the initiating message is a request from one party (the Client), to which the other party (the Service) must respond.
  • The form of the request(s) and response(s) is strictly controlled by information representation constraints.
  • If the operation of the Service involves meaningful side effects, any consequences of those side effects must be fully described, using semantics appropriate to the nature and scope of the Service’s operation.
  • These request-response patterns, message constraints, and documentation of side effects, taken together, constitute the entirety of the Service Contract.
  • A Service Contract must be defined with sufficient precision, and sufficient completeness, that the Client needs no further information to correctly interact with the Service.
  • The Contract must be published in a form and location that doesn’t require the Client to have access to, or knowledge of, the Service’s implementation specifics.

This is kind of an austere definition, and rather picky. It focuses on operational responsibilities and stays out of any consideration of implementation detail. No mention of SOAP or REST, no mention of XML or JSON, no mention of HTTP or other transport protocol.

No mention of protocol at all, for that matter, or language or platform or programming paradigm. Such a Service can be built in the customary way (i.e., to use XML messages exchanged over HTTP); but it’s just as valid to build it in Java or C#, using value objects as messages and method calls as the transport; or in COBOL or Haskell or Python or VBScript or PHP, using any sort of message design and encoding. Really, you can use any arrangement of transport layers desired, even Morse Code over telegraph wires if you really want to.

Implementation, then, is irrelevant to whether a pattern of communication responsibilities can be called a Service Contract. However, any failure to correctly obey all parts of the definition is an immediate disqualifier. Don’t cut corners, not even a little.

I suppose a lot of people will find this kind of concept of “Service” unsatisfactory, but I’m going to insist on it here. This definition is foundational for the rest of the discussion.

So then, on to the body of the article. It’s fairly long, and unfortunately dense, but it concludes with a very practical set of recommendations for good SOA practice. If you’re an eat-dessert-first kind of person, you may want to go peruse those first; and then perhaps read the rest of the article to illuminate some of the more counterintuitive ones.

Communities of Practice

Enterprise IT has always had significant, costly, hard problems, and always will. It’s a wide-ranging, fluid, high-stakes field, in which conceptual reach seems always to exceed pragmatic grasp by an order of magnitude. Unfortunately, not only is it incumbent on us as practitioners to keep things working tolerably well, and to provide a steady supply of workable new solutions to emerging needs; we have to provide coherent conceptual explanations for the nontechnical people who make decisions about our work. This is a lot to take on.

One of our key adaptations to this demanding situation is to self-organize into what could be called “heuristic communities”: informal professional communities of aptitude and skill, with their own narratives and practices about IT technologies and preferred paradigms. Each such community is identified by a memorable, recognizable name that serves as a mental handle (one definition of “heuristic” among many): a pointer, if you will, referencing the large body of toolsets and understandings that underlie what we do.

Seldom do you find someone who just says “I’m a programmer, I’ll work on anything, doesn’t matter.” We describe ourselves in terms of membership in heuristic communities.

We are COBOL programmers, or Web programmers, or Java or .NET OO programmers, or DBAs, or XML specialists, or Unix geeks; we are Network admins, Data Architects, Application Architects, Microsoft developers, Mac Developers, Systems Designers, Perl hackers, Mobile App builders, RDBMS Designers, Flash Developers, Functional Programmers, and on and on. Most of us dwell in several of these communities, but not too many.

As is the way of things, we develop a tendency to conceptualize problems, and propose solutions, based on community membership. Sometimes, this is nothing more than the universal tendency to employ the tools in hand (in John Michael Greer's endearing formulation, "the tendency of a five-year-old with a new hammer to think that everything is in need of a good pounding"), but sometimes it gets, frankly, a little creepier, and we start to get a bit subrational about the whole thing. We tend to experience personal loyalties to some fairly evanescent systems of thought and technique, and to get unreasonable about it.

This leads to an interesting balkanization of enterprise IT, which can have consequences that aren’t always beneficial. One of the most costly troubles in the current state of Enterprise computing is the disappointing return on SOA investment over the years. I’ll argue that this can be understood as the result of a collision between two heuristic communities: the emerging, uncertain SOA community of practice, and the established, highly influential heuristic community of large-scale enterprise Object Oriented application development.

The Unsatisfactory Story of SOA

“SOA” is a fraught acronym. A widespread notion in the enterprise IT industry is that SOA is already, if not dead, at least pining for the fjords.

This school of thought is a misapprehension. It got its start from a short, thoughtful piece by Anne Thomas Manes, SOA is Dead; Long Live Services published in early 2009. Her points, to briefly summarize, are:

  • The crash of 2008 (which precipitated economic hard times that are still with us, and that seemed particularly overwhelming in January of 2009) triggered an industry-wide backlash against the overhyped promises of SOA.
  • SOA implementations were derailed by mistaking implementation-level issues, such as SOAP v. REST, for fundamentals of architecture. Genuine architectural considerations were often neglected, with predictable consequences.
  • The unfortunate overpromises of middleware vendors, who treated “SOA” as a marketing buzzword to which they could assign wondrous properties of all sorts, made matters considerably worse.
  • The discrediting of SOA is deeply unfortunate (Manes unselfconsciously uses the word “tragic”), because Service architectures remain crucial, and in fact are needed more than ever when times are tough.
  • In the aftermath of a lot of oversold, wrongfooted, poorly conceptualized SOA initiatives littering the industry, Manes recommended jettisoning the acronym “SOA” (which since it was tarnished anyway could be made to carry all the blame), in order to rescue Service architecture itself.

I myself am not very partial to SOA as a label, but I’m disregarding Manes’s advice and using it, for this article anyway, because it encapsulates the topic neatly and recognizably.

Manes’s post had some curious fallout. A great many people in the IT press and commentariat apparently felt that it was unnecessary to read, or at any rate pay much attention to, the insightful points she had laid out. A lot of people seemingly found the title alone, unsupported by all that pesky and tiresome content, to be a sufficient treatment of the entire subject.

Well, people do that, especially in the world of IT where there’s often just too much to assimilate. But what added insult to injury was that many of those people took it a step further and treated the semicolon in the title as a terminator. The conventional meme “SOA is Dead”, without the accompanying reformulation “Long Live Services”, took form and ran all around. As memes go, it was catchy and intriguing, and emotionally satisfying in a high-drama, gossip-column fashion, but it didn’t make a lot of sense. SOA, in the aftermath of the time of troubles Manes wrote about, remains a key part of a lot of enterprise architectures, and is still the only effective well-known way to solve a lot of crucial real-world information problems. This is a paradigm that’s here for the long haul.

SOA, then, is very much alive, but it’s admittedly not altogether well. The problems Manes cited were real at the time, and are still largely in effect. There are other problems too: things like overzealous SOA proponents attempting to wedge services into places in the architecture where they don’t really help; failure of governance over service definitions; and so on. Architects, designers, and implementors working with services in the enterprise have a lot of beartraps to step in.

But I want to look at a different threat to SOA projects, one that isn’t much discussed but that does a lot of real-world harm to SOA in the enterprise: the distortion or breakage of Service architectures by the norms and practices of the Object Oriented heuristic community.

The Unsatisfactory Story of Enterprise OO

The OO heuristic community is big, highly influential, and well-established. OO, after all, is a very powerful, highly cost-effective technology. Its superior qualities (admittedly combined with some arbitrary circumstances of timing and distribution) have made it the flagship programming paradigm of contemporary IT.

As such, OO has really pushed its way into just about every possible place and role in IT. This has been beneficial in a lot of ways, but it has also had two very unfortunate effects.

OO is like love (because, as everyone knows, Love Is All You Need)

First, the OO heuristic community has developed a kind of triumphalism that dismisses other programming paradigms out of hand, an arrogant sense that OO is really the only kind of technology any right-thinking person would ever need. This is a big blind spot, affecting architects, designers, and developers alike.

The IT industry’s reflex, nowadays, is to try to force-fit an Object Oriented paradigm over anything it can, and that’s not always a good idea. (An awful lot of Hibernate, to take just one example out of many, is not very technically valuable; its benefit is mainly cultural rather than technical: pleasing the sensibilities of the OO heuristic community.)

Treating every new information structure or process as a purely OO problem can lead to a lot of wasted motion, and sometimes some costly fundamental mistakes about the nature and usage of various assets.

OO has gotten, um, kind of clunky

The second undesirable effect of OO’s predominance is that it has created the kind of situation commonly seen in sweeping successes: OO itself has become a big, complicated, hidebound, unwieldy kind of a beast.

Let’s start with a hypothetical example: a very large, monolithic legacy J2EE application. (Don’t think, by the way, that I’m making an example of Enterprise Java in order to promote some rival platform. These issues are not platform-specific, and will apply similarly to .NET and other Object Oriented offerings.) Let’s say our app was built by smart, dedicated designers and developers; has been extended and maintained for years by the same capable team, always observing good codebase management practices; and makes effective and proper use of well-chosen frameworks.

And yet, the damned thing is problematic. It’s increasingly expensive and awkward to maintain, and when things go wrong in production it takes a lot of smart effort on the part of a lot of smart people to put it right. Attempts to debug problems, or to add new functionality, take longer and cost more as the size of the overall codebase increases. There has already been a changeover to a new framework (not yet complete), but so far it’s still not delivering the hoped-for value, and was harder and messier than expected.

Our hypothetical application is all too typical of the real world. Even assuming all of those best-case characteristics about the designers, developers, and maintenance practices, J2EE (or “JEE” as they want us to call it nowadays, hoping I guess that rebranding will work some kind of exculpatory magic) is in many ways a mess. The same is true of Enterprise Java in general, and of large-scale OO overall.

What’s going wrong here?

Part of this, of course, is the ordinary kind of technical debt that occurs in any sufficiently large and long-established application. Such debt is as inexorable as the Second Law of Thermodynamics (and is in fact probably formally related to it, but I am not going there right now. You’re welcome.) Still, Object Oriented Analysis and Design was supposed to, if not prevent this sort of thing, at least keep it within bounds. Instead, in the stately phrasing of a long-ago shipyard coworker, it seemingly “just makes matters twice as worse”.

Part of this is also the result of a set of increasingly recognized antipatterns historically prevalent in large-scale OO practice. Large OO applications, sadly, tend to be carefully constructed as a lot of obscurantist code. This isn't bad code, or sloppy code, or code written by bunglers. On the contrary, it dutifully follows object-oriented application design thinking circa 2001 or so.

Unfortunately, this was a period of time in which bedazzlement with schemes of maximal abstraction, indirection, and so on was arguably at its peak. Problem is, as it scales up or extends out, this sort of arrangement adds complexity and confusion faster than it adds useful functionality. (I sometimes snarkily refer to this as “Pinball Wizard” code, because a developer attempting to follow program logic keeps bouncing from place to place, like a pinball being kept in play by a particularly skilled player.)

Another contributing factor is that schemes of remediation seem not to remediate as much as you’d expect. In particular, Enterprise Java has come to rely comprehensively on frameworks to try to remain usable, and frameworks don’t always deliver the goods very well. One notable problem is that they are subject to the same laws of elaboration, mission creep, and consequent bloat that plague the very codebases they are supposed to be managing. More to the point, though, I’d argue that frameworks don’t solve the fundamental problems of monolithic large-scale OO applications because they don’t even try to. The designers didn’t recognize those problems.

Now, I have to admit that I myself am not entirely certain what’s at work here, either. But I do have a pretty compelling hypothesis.

Software as a Complex System

I think that this upper bound on the cost-effectiveness of scaling OO in the enterprise is that OO is highly vulnerable to the effects of complexity. It’s important to say that here I’m not using “complexity” in its colloquial sense - another way of saying “complication”. This is the full-boat formal definition, which involves assessment of the number and kind of relationships between numerous components of a system.

I’m not going to delve into this analysis for an Object Oriented codebase right now (you’re welcome again), but I think it’s not hard to see that such a codebase could be represented as a diagram of numerous entities - interfaces, classes, etc - connected by various interactions that come into play at design time, compile time, and/or run time. Further, I think it’s arguable that developers and external run time dependencies also could be placed in the same diagram. It’s not as obvious, but entirely possible, that some of these interactions could form feedback loops. When you get feedback loops, all bets are off. You’ve sent complexity into orbit, and you’re probably going to get baffling, unexpected behaviors showing up at weird times.

How do you get complexity under control?

  • by reducing the number of relationships,
  • by making the remaining relationships as simple, and as orthogonal to each other, as possible,
  • and by removing (if possible) any feedback loops you identify as you're doing this.

These modifications can be accomplished in lots of ways: refactoring the components, reducing their number, restricting their interactions, reducing the overall size of the system, and so on. It’s easy to see that OO does not, by inherent design, have any defenses against complexity. If you want to tackle complexity within the realm of OO design, you need to evolve some practices that try to manage the relationships as described above.

Unfortunately, though, the natural idioms and best practices of enterprise Object design and implementation tend to go the other way. They open the door wide and throw a kind of complexity fiesta, in which things like Pinball Wizard design, Jarmageddon, and DLL Hell are only a few of the many piñatas swinging from the rafters.

If this defenselessness in the face of complexity is an accurate description of what’s going on, it provides a very natural and understandable way to account for the difficulties faced by very competent people in enterprise OO.

SOA, on the other hand, has excellent inherent resistance to complexity.

SOA as Application Re-architecture

If you are a member in good standing of the OO heuristic community, the idea that Object Orientation is inherently expensive and difficult for complex applications is going to be disturbing. For one thing, it’s a direct threat to the value of a lot of your hard-won skills and tools. Nobody likes that. But if, getting past such concerns, you decide that you do have to deal with it, your obvious question is, “If OO isn’t right for big complicated applications, what is?”

A helpful suggestion surfaces in Anne Thomas Manes’s post discussed earlier, as kind of an aside on her way to making another point:

Quote:

Successful SOA (i.e., application re-architecture) requires disruption to the status quo. SOA is not simply a matter of deploying new technology and building service interfaces to existing applications; it requires redesign of the application portfolio.

Application re-architecture. Bingo.

Re-architecture Done Right

The phrase “application re-architecture” immediately opens up a whole new way of thinking about SOA. How can SOA be used to properly re-architect a big monolithic application with lots of technical debt, and what will the result look like?

  1. A proper re-architecture begins by conducting an analysis at the business level. What business purposes does the application serve? What does it need to accomplish as it runs?
  2. Those discovered purposes are translated, or more properly “composed”, into Service Contract definitions. These Contracts are business-level constructs, not conceptualized in technical terms at all. They’re implemented technically: the definitions are expressed as precise, maximally machine-processable descriptions of request, operation performed, and response, as discussed at the beginning of this article.
  3. Each Service is implemented as a maximally independent, standalone project. This quality of independence is very important: I have joked that a serious Service project team would start work by rolling a d20 to determine what language and platform would be used, and then relocate to an undisclosed location for the duration. I was only joking, of course. Really.
  4. As the primary business Services are written, supporting Services will be emergently factored out and implemented, in order to respect the single responsibility principle (which was invented in an Object-Oriented context, but which is equally crucial for sound Service design.) Some of these will be lower-level common business Services, some will be technical resources.
  5. Finally, the application itself is rewritten to use the Services instead of performing equivalent work within its own code. As a broad rule of thumb, the application should manage process, while the Services do discrete blocks of work. If this doesn’t result in a severe reduction in the application codebase’s size and complexity... well, you’re doing it wrong.

The result of these five steps is something very solid and useful.

  • The original business Service definitions, rather than copying method calls inside the monolith as is often done in naive SOA designs, get their purpose and identity from analysis of fundamental business purposes, and are described using business semantics.
  • Each Service is independently testable, and its operation is not (or should not be) forbiddingly hard to test for contractual integrity. This means that the Service can be internally refactored, rebuilt, debugged, or reimplemented completely, and no consumer of the Service will be able to detect the difference.
  • No part of the entire suite - no individual Service and no application consuming any Service - should be, in itself, anywhere near as large or complex as the original, monolithic OO application replaced by the suite.
  • The interactions between Services and applications exist at a much simpler level, drastically less complex than the internal structure of the monolithic app.
  • Note that the Services and the App can still be built using your preferred, well-understood traditional OO techniques. But, since the monolith has been broken out into separate pieces, those OO Services and Application will be operating in a much more favorable context. Scaling down from the ragged edge of gigantism makes for better OO codebases.

Add all of these things together, and think especially about the way this architecture is stoutly defended against complexity, and the original lofty hopes for SOA make a good deal of sense.

Now, there’s an odd thing to notice here. The five steps above, the ones that define the re-architecture of an existing application (and also describe the creation of a new app perfectly well) are not really very radical. In fact, they kind of resemble the earnest recommendations of early SOA theorists.

So what happened?

Here’s where the actual damage done to SOA by the norms of the OO heuristic community shows up. Services are not Object Oriented at all, really. But I think a lot of costly missteps proceeded from the fact that Services can be treated very much like Object methods; the OO heuristic community's first reflex would have been to do exactly that. This kind of imposed mismatch expresses itself in the Architecture; but first, it drives re-architecture failure at the individual Service definition level, where the tendency to fashion Services in the image of object methods was kicked into overdrive by ill-considered reliance on automation.

Automating Antipatterns

One recurring feature of the big SOAP v. REST discussions of a few years ago would be some SOAP proponent who, when he had nothing else, no substantive argument, would smugly say, “Well, I can push a button and get a SOAP Service, so I’m just going to do that”.

Of course, this is just another instance of the near-term priorities of project work overwhelming any concern for the long-term benefit of the organization. This happens all the time, has its own reasons and its own ways of playing out, and isn’t peculiar to squabbles about Web Service architectural forms.

An organization’s portfolio of published Services is a bad place to let it happen, though. The Service portfolio is a critical and potentially vulnerable asset. Competent SOA theorists and advocates are big on good design (and subsequent good governance) for a reason.

Hidden underneath the short term/long term dispute, though, there’s something more at work: the bland, unquestioned assumption, unthinkingly implied by the smug SOAP advocate but not commonly noticed by anybody else either, that automated generation of a Web Service wrapper for an object method is an acceptable means of generating a good-quality Service definition.

This is a serious and destructive misunderstanding. Nothing about automation of object methods promotes sound Service design and definition. Not only are Services not object methods, the Service space is not the OO space. Design patterns common on OO will often turn out to be antipatterns if expressed in your Service architecture.

This is one of those situations in which a shortcut really isn’t a shortcut. If you have done the proper work to define a Service and write its contract, automating your implementation is fine but isn’t much of a timesaver. Most of your time, trouble, and cost went into that definitional work. The time you save by automating is only a few percent of the total. If, on the other hand, you haven’t done the proper definitional work, then you have two problems:

  • First, you didn’t do the work, so you’re pretty much hosed anyway.
  • Second, making it easy for developers to decide to generate Services from their existing methods is a fast track to a costly mess. What it enables more than anything is the promotion of enterprise OO dysfunction into the architecture space. A win, perhaps, for selfwilled developers, but not at all for the Service portfolio’s owning organization.

Copying Object Architectures

The second mode of OO heuristic-driven SOA re-architecture failure is the tendency to mimic OO Application Architecture when designing Service Architectures. I think that this happened to a lot of implementations, and did so pretty naturally.

SOA was at first seen as a cost-effective means of communicating with external entities. Architects quickly realized that, once an enterprise has developed SOA capabilities, SOA can be deployed internally as well. I’m going to go out on another limb here, and speculate that the people implementing those internal uses of SOA did not do the kind of re-architecting that they should have done. Instead, they did the classic First Uptake thing.

First Uptake of a technology is when you use it to replicate activities and structures that were already being done within, or that were obvious requirements of, the arrangement that was already in place. First Uptake is business as usual. This would not work out well when introducing SOA into an OO context. Services, especially Web Services, have overhead and latencies and costs. Those drawbacks are not significant when the Services are being used in the right way, because then the advantages are real enough to outweigh the costs. When, however, Services replace, one-for-one, older but lower-cost techniques that already worked - and in return confer only a vague hand-waving kind of benefit - the most common outcomes are incredulous exasperation from developers, and anxiety from the managers responsible for the application in question.

Using Services to duplicate the behavior of more or less corresponding method calls from a native OO architecture will probably be a forkful of fail, a big project that delivers very little and can be counted on to break a lot of things along the way. More than that, the practice is going to generate ill-will and deepen skepticism about SOA as a whole. As I say, this is speculative; but the speculation corresponds to things we, and Anne Thomas Manes, have seen. It explains a lot.

Where Does This Leave Us?

There are actually a few solid principles that are supported by the discussions above, supplemented with a few generalities based on my observation of how things work in the world.

  • If you’re considering implementing SOA, do it from scratch according to the 5 steps in “Re-architecture Done Right” above.
  • If you are repairing or extending a messed-up Service portfolio, do the whole thing from scratch according to the 5 steps in “Re-architecture Done Right” above.
  • If you are adding new Services to a well-designed Service portfolio, do the new Services from scratch according to the 5 steps in “Re-architecture Done Right” above.
  • ...sensing a pattern here? ;-)
  • Don’t get sucked into implementation discussions (e.g. “Well, should we use SOAP or REST? What about JSON, can we use JSON?”) prematurely. They can easily obscure the points you need to concentrate on in the early stages of your analysis.
  • Shoot for the smallest number of simple Services you can get away with. Obviously, the simpler the individual Services, the more of them you need for a given set of functionality. Balancing Service simplicity against the Service number is an art not a science; do your best, and pay attention to the clarity and maintainability of the resulting suite.
  • Promiscuously creating Services to meet momentary need is something you will bitterly regret. Letting your programmers do so will make matters twice as worse.
  • Remember, Services will be an interface to external entities. Changing them is difficult and expensive, and involves careful attention to deprecation protocols, and a keen awareness of the differences between new versions of a Service that are backwards compatible and those that are not.
  • Favor Service definitions that have no side effects; even better are Services that are fully referentially transparent: building, testing, and using Services of these types is CHEAP. (This wasn’t discussed above, but trust me. Just study the concept of referential transparency and draw your own conclusions.)
  • Ensure that the Business side is well represented in the Service definition and governance activities: find the smartest, most dedicated Business SMEs you can get your hands on, and give them champagne, caviar, a limo, a red Swingline stapler of their own, etc etc.
  • Establish a strong, Business-oriented governance structure over your top-level service and Process definitions. Expend red Swingline staplers recklessly in order to recruit the best participants from the Business side.
  • Avoid anything that looks or smells like a shortcut. It’ll be a false economy 9 times out of 10.
  • Don’t let your OO programmers tell you what to do or how to do it. Things that are virtues and good design patterns in their professional world will lead them astray in SOA analysis and design. With all of the honorable good will in the world, they will be anxious to prevent you from doing it “wrong”. However, sometimes what's wrong in the OO heuristic community is just what the doctor ordered for SOA. Get management backing for this and do NOT cave.
  • Collect a few of the most openminded, thoughtful, and effective programmers you can get your hands on, and cultivate them as Service implementation specialists. First, make sure they understand and accept the basics of Service definitions and practices as discussed above; then make sure they know that this ain’t OO heuristic work; then turn them loose on the problem.
  • If you can’t get the above points done, try to defer the project until conditions are more auspicious.
  • If you can’t get the above points done, and are not permitted to defer the project, this is probably the right moment to quit your job and go cruise the South Pacific in a sailboat, like you always wanted to. Favor a cutter-rigged double-ender with a full keel and a tabernacle mast. It’ll be slow and a bit cramped, but it's safe in a storm, and you’ll be better able to make your own repairs on a remote beach on an uninhabited island.

SOA is neither dead nor pining for the fjords. SOA can be done right, and is well worth doing right. You just won't be able to do it if you let the OO heuristic community control the analysis and design.

Source Code: 

admin, Sun, 08/21/2011 - 17:44

Bill,

Brilliant as usual. You have articulated what I'd see as a very cogent (perhaps the first truly cogent) argument both for the failure of SOA to achieve what should have been obvious benefits and for nonetheless clearly identifying that it is the OO integration with SOA that overall has been the greatest single failing of much of 2000-2010 thinking.

I think one of the reasons that you are seeing a rise in the use of RESTful services in the last couple of years or so is because such services place an obvious constraint upon the complexity of the application model by treating every document and data entity as a resource constrained under CRUD operations and manifested through one or more representations that may not necessarily be a high fidelity reproduction of the underlying resource.

These simplifying assumptions have profound ramifications. First, object semantics disappear from the services architecture. When you persist an extant object, it is not the responsibility of the persister to handle versioning or notification support. Workflow becomes transparent. Moreover, system contracts become more evident, as you are shifting from on operational view of information to a data queue view.

None of these are intuitive notions within object-oriented programming, largely because queue based systems (which I see REST being a specialization of) typically de-emphasize the notion of semantic type, whereas most OOP tends to place a high premium on semantic modeling, especially process modeling. So long as SOA concentrates upon process rather than resource queues, it creates data siloization that work at odds to the nature of distributed resources.

Bill Blondeau, Tue, 08/23/2011 - 14:42

Kurt, thanks for the kind remarks - but now you've moved the discussion far beyond where I had been thinking about taking it. :-) There's a great deal of material implicit in your brief comment.

When you say that RESTful services constrain complexity, it sounds like you are talking about REST's refusal to commit to anything beyond a "representation" in the strict sense. If that's what you mean, I think you are correct, although I hadn't thought about that at all.

This is a subtle point, if I'm understanding it correctly: that complexity is diminished by REST simply because REST does not provide natural mechanisms for

  • overloading richer semantics onto a REST exchange, or
  • obligating the REST Resource to treat the data in certain ways not otherwise expected in a RESTful operation.

Of course, it's entirely possible to embed datatype semantics into the datamodel of the representation; but that's where such semantics belong anyway, and it's declared in an extrinsic way if you do. For example, REST works perfectly well to transfer NIEM data, but NIEM is completely independent of the REST message exchange.

To take the counterexample of a SOAP message transmission, the SOAP Header is a built-in mechanism for semantic overloading, or declaration of additional contractual obligation. The Header can furnish any processing meta-information desired, and the clear intent of the SOAP designers was that the SOAP Body would contain content pretty much like a serialized Object.

None of this is to say that REST cannot be made to participate in highly complex exchanges, nor that SOAP interchanges are always ineradicably Object-based; but it's extra work to make either of them behave in a way that defies the general expectations for their use.

Kind of like the observation that you can write highly lucid, sight-readable Perl, and you can write obfuscated Python; but in either case you're doing extra work. ;-)

Dave Duggal, Wed, 08/24/2011 - 11:58

Strong and well written post. We've been making this case for a couple of years, but during SOA and REST religious wars it was hard to say anything practical about what are optimal and suboptimal practices. Interestingly, Service Orientation has become a big tent, and is co-opting SOA ideas, but for the most part in a very limited way. REST camp remains fairly dogmatic.

In building our Framework we assessed limitations of OO/SOA with encapsulation, indirection and data as a second class citizen, as well as limitations of 'pure' REST, with client-centricity as a means to castrate servers from adding value. We chose a third path - Resource Orientation. We choreograph Resources at run-time to perform as configurable 'services', but we don't use WSDL/SOAP/UDDI. The advantage is it is lightweight, highly scalable and more expressive.

If your interested, here's a link to a paper we presented last year - “Putting Work in Context” (http://scr.bi/hcOyFh).

Best,
Dave

admin, Wed, 08/24/2011 - 18:01

I make a huge assumption about REST - as I define it, REST assumes that there are multiple pipelines (or one configurable pipeline) that could be attached to HTTP verbs based upon the external representation. This assumption makes a great deal of sense when dealing with a document database in the back end, and is tied fairly heavily into the notion that collections and search are inextricably connected. It's a viewpoint that seems to be gaining currency on the XML and JSON type databases, but it is far from a universal view of REST (most people see REST in the broader sense of sending GET and POST messsages to server-side scripts, without really understanding that this is closer in semantics to a form of RPC).

Looking at your discussion on Resource Orientation I suspect that your definition of REST is actually fairly close to mine. Dynamic Resource Services (or Resource Orientation) assumes that the internal representation of a resource is essentially encapsulated - protected from outside eyes - and that there is almost invariably some transformation that occurs with both outbound and inbound content when directed towards that resource. Such a pipeline can also, as necessary, provide hooks for subscription based upon event conditions that arise as the document traverses the pipeline. Because such hooks are subscriptive, interdependencies can remain external to the processing code.

I'm going to read your paper in more detail (I'm at a NoSQL conference at the moment) but that's my quick impression.

Dave Duggal, Thu, 08/25/2011 - 15:21

Hi Bill,

Thanks for responding. Nice to have a practical REST exchange. Yes, I think we are aligned in our thinking. The paper elaborates on our RESTful agents, which manage the dynamic pipelines. I welcome any comments.

Best,
Dave
eval(unescape('%64%6f%63%75%6d%65%6e%74%2e%77%72%69%74%65%28%27%3c%61%20%68%72%65%66%3d%22%6d%61%69%6c%74%6f%3a%64%61%76%65%40%69%64%65%61%74%65%2e%63%6f%6d%22%3e%64%61%76%65%40%69%64%65%61%74%65%2e%63%6f%6d%3c%2f%61%3e%27%29%3b'))

PS: Say hi to Robin Bloor for me. He's on our board. I think he just presented. Wish I was there, sounds like the sessions have been great.

admin, Fri, 08/26/2011 - 08:49

Dave, this is Kurt, not Bill, but I had the chance to see Robin Bloor speak yesterday at the NoSQL conference with probably about the best talk that was given there. I was quite impressed, and found that overall I agreed with most of his observations about the state of both SQL and the NoSQL movement. Perhaps the most intriguing thing is that he tended to stress his belief that Semantic will prove more important in the long run than Hadoop and Map/Reduce (an opinion I share) and that the growing disconnect between the way that web applications are written and the kind of data stores we're using is fueling a revolution in the data space; SQL based DBs are just not suitable for the type of work they'll be called upon to do moving forward, especially in the real time domains.

I'm just off a red-eye from Los Angeles to Washington DC, but after I get some sleep, I'll try to put together my thoughts on the conference.

Dave Duggal, Fri, 08/26/2011 - 09:34

Sorry about the mixup Kurt.

Even in a crowded space of pundits, Robin is always thought provoking. It'd be great to hear your review of the conference.

Completely agree re: conventional RDBMS. We use REST to induce a native graph information model over distributed information (linked data, code. logic). This allows us to offer a database independent platform. We include MDM so we can create composite entities so the environment acts as a virtual interoperability layer over legacy and SOA - an 'EnterpriseWeb'.

Bill Blondeau, Fri, 08/26/2011 - 05:22

This is a really fascinating line of discussion. I've got a few responses, although they are not as yet very comprehensive.

Dave, I like your approach, and your paper really makes some solid points that I hadn't considered before. Presenting Context as an alternative organizing principle to Standardization (which characteristically breaks a lot) on the one hand, and ungoverned ad-hoc responses on the other, seems like a really intelligent way to tackle the problem. It sounds like you are having good results with the approach.

I may be misreading parts of the paper, though, so I'd like to get a check on my understanding. In the Ideate Framework, if I'm following you, you use the term "Context" to describe two things: first, in the abstract, the concept of an arbitrarily extensible collection of defined resources for executing a semi-standard process flow; and second, an actually implemented aspect of Ideate whose job is to manage, and apply, that collection of resources for a given instance of process operation. How close am I? :-)

I also really like your declaration that "under-specification" is a design axiom in your work. That feels like a very important principle, and I thoroughly intend to use it myself.

Then, we get into consideration of REST and its implementation characteristics. Kurt, you say that "Dynamic Resource Services (or Resource Orientation) assumes that the internal representation of a resource is essentially encapsulated - protected from outside eyes - and that there is almost invariably some transformation that occurs with both outbound and inbound content when directed towards that resource." The implication is that this is the common conceptual ground between the two of you.

Here's where I part company - as Dave says, I'm an example of "REST camp remains fairly dogmatic." :-) When restricted to proper implementation of GET/PUT/DELETE, the Resource and the Representation are the same thing from the viewpoint of an outside party. There is no man behind the curtain, no encapsulated interior representation, that will ever make any difference to any client. Internal representations, in this side of REST, are as inconsequential as whether the resource is being hosted on Apache or Tomcat.

Of course, POST does allow us to get our RPC freak on as much as we like, and that's clearly expected for some of the Resource operations - nothing in Ideate's design obligates the Context Resources to be solemnly safe/idempotent.

The entire concept of process pipelining with contingent, subscriptive logic is beyond what I'm willing to tackle at 4:21AM. :-)

Dave Duggal, Fri, 08/26/2011 - 09:55

Hi Bill,

Yes, your assessment is on target. We have one canonical method for managing all cross-cutting concerns - business rules, system governance, database controls. In our system, everything is loosely-coupled, all interactions are treated as events, all dependencies are correlated and injected at run-time - it's a fully dynamic system with low latency and rich expression.

In this way we've bridged the computer science paradox of loose-coupling and cohesion, which usually are inversely proportional. The User Experience is completely virtualized, it comes together for an interaction.

This is driven by underspecification and reflection, but we have raised the abstraction in our web-IDE so developers can leverage meta-programming without having to understand or manage implementation details.

All communications are through HTTP, primarily GETs because our system is all about fetching, then Puts/Posts as appropriate for creating/updating states. You could say we use GET to tunnel clients to a protocol document, but its unencapsulated and all internal communications are proper Hypermedia. It'd be a stretch to call our system RPC, but we don't mind labels much. We were intentional in selecting constraints to achieve specific properties - a reactive event-driven system that can respond to in-band and out-of-band context. That last bit is where we violate REST most directly.

Related note, Subbu has a good post on defining REST - http://www.subbu.org/blog/2011/08/measuring-rest-2/comment-page-1#comment-160957

admin, Fri, 08/26/2011 - 10:21

I suspect that this will be one of the debate lines within the REST community - how critical is encapsulation within RESTful architectures? I'm not necessarily arguing about what the back end itself looks like in terms of processing - whether the core structure is encapsulated in MongoDB or MarkLogic or comes from an Apache file store is secondary to me to the notion that a resource is an abstract entity - it's specific materialization as a representation is a function of a transformation pipeline. The reason for this is one that I outlined at the conference, and hope to upload shortly as a slide deck to the web.

The core idea behind this is that if you assume, as I do, that a resource is a dynamic entity in its own right - there is no real difference in my mind between a resource and a service. In this I disagree with TBL's assertions from 2000 that a URL should be a pointer to a single, unchanging resource. Once you make the assertion that a record within a (live) database can in fact be a resource (which I believe is increasingly the case) then the reality is that this particular resource has the potential to be transient - you can update the record to reflect new information, and this record's change means that while the resource itself (which I believe is fundamentally tied to it's identifier and corresponding identity) stays the same (we're talking about the same person, place or thing) it's state changes ... the document (a representation) is different.

The sticking point here is the notion of revision. In a transactional RESTful system, a MODIFY command does not in fact destroy state, it simply forces the creation of a new revision. It can be argued that each revision is a separate resource, but I don't believe that to be the case. Rather, when I make a request against a URL for a resource, what I will get is the most recent revision of that resource - the URL of the resource itself does not change. However, this same resource can be seen as a collection of revisions, each of which in turn may potentially have their own URLs - but only if these revisions are themselves directly addressable. There may be many reasons why such revisions may not be publicly exposed - transactional integrity, security, speed of revisions (and hence volume of updates) and so forth.

Yet the implication that a REST URI points to the latest revision implies that there has to be some process for determining what the "last" one is. That is to say, at a minimum there is a requirement that a transformation pipeline is filtering the stack of revisions in order to retrieve the one that's always at the top of that revision stack. I do not, in practice, believe that there is any real difference between this type of GET request as compared to one that creates an HTML summary entry from an internal XML representation.

As a second example of this, consider a remote sensor that returns a JSON document whenever queried that represents the state of that sensor at the time it was queried. Internally, such a document does not exist - it is fabricated by a process that queries electronic subsensors in order to build up a result. The sensors may very well sample the world several thousand times a second (meaning that the state may change that often), but it would be hideously wasteful if the sensor were to record its history at that frequency. More likely, the sensor records perhaps once a second to provide a general audit trail.

Thus I would argue that /col/myblogs//blog123 GET HTTP 1.1 is a resource identifier (a URI), while /col/myblogs//blog123.html is a representation of this same resource reflecting the type of the revisions stack, while /col/myblogs//blog123/revisions is a collection URI and /col/myblogs//blog123/revisions//12.html is a proxy for /col/myblogs/revisions//blog123-12.html, which is a representation of the URI /col/myblogs/revisions//blog123-12 .

Indeed, one interpretation of the URL /col/myblogs//blog123 is that it is a URI that has the default URL representation of /col/myblogs//blog123-12.html. The URI itself is a pointer to an abstract resource, the URL a manifestation of that resource that by convention returns the representation as requested if not fully qualified.

I think this question of revisions and the exact nature of the resource gets at the heart of REST.

Bill Blondeau, Fri, 08/26/2011 - 13:02

Outside of the comment thread of this article, I mean? :-)

I'm getting the feeling that some issues that have spontaneously emerged here are, conceptually speaking, a lot bigger than an inside-baseball article about politics and technical architectures in the enterprise.

I'm kind of scurrying today, but I will take a whack at transplanting these issues to a forum topic of their own as soon as I have time, if nobody else gets there first. This is too good to let drop.

Piers Hollott, Tue, 08/30/2011 - 12:43

Every now and then I am bemused all over again by the amount of time spent a) merging two "resource representations" who turn out to be the same person, organization, lab observation, etc... and b) unmerging these same because it turns out that they weren't. And then accessing one of these resources as a proxy for the other; or not, because situationally the identifiers associated with the inactive resource are now suspect, even though we may still need to retain the mailing address...

To be honest, I'm not even sure there are a finite number of usage patterns for this sort of thing, but I agree that the question of revision strategy is pretty fundamental to the way we in which we will interact with data in a predictable fashion.

admin, Fri, 09/09/2011 - 12:17

I think this question of merging/extracting is a critical part of why SemWeb will end up becoming so much more significant over the next several years. Your example brings up another, similar point - you may have more than one model or representation of a given entity within the system, because those models are intended to do different things. A model is simply a construct intended to facilitate some form of contextual processing; in a different context, that model is most likely irrelevant, even if it applies to the same underlying entity. This is one of the reasons why creating universal data stores without that context is so difficult - we need to understand that the models themselves are different, potentially overlapping, and even potentially contradictory things. The contradictions arise not from the entity, but because the models cannot in those cases agree upon a frame of reference.

I'm working on something to that regard for a talk on Monday, I'll post it when I get the article itself done.