A reader has posted an especially complimentary review of Enterprise Integration Patterns on Amazon.
The review of Enterprise Integration Patterns, Like the Ragu Spaghetti Sauce Commercial said ... "It's in there", says, "After reading this book, I am actively trying to construct a proposal for a new course based on its contents ... it's that good." Nice to be well thought of.
Bobby Woolf: WebSphere SOA and JEE in Practice
with Tags: patterns X
Design patterns for dynamically typed languages.
In Another Take on Design Patterns, Rick DeNatale has posted a friendly discussion of The Design Patterns Smalltalk Companion (I'm a co-author). He points out that the book explains the patterns from the original Design Patterns book with the perspective of a dynamically typed language. The language used in the book is Smalltalk, but the principles also apply to other dynamically typed languages like Ruby. Thanks, Rick.
I'll also point out that the book talks about development of business applications (whereas the original book mostly talks about frameworks). So if you're looking to apply the original design patterns in business applications or using dynamically typed languages, The Smalltalk Companion may help.
I've worked with a couple of clients who use standby queues. This is supposed to solve problems, but I think it really creates more.
The problem is the same as described in The Message Consumer Rollback Pattern: A consumer successfully receives a valid message but cannot process it because a resource it needs is down.
We know the message is valid because it passes the message validator.
Options: Throw away the message, retry, put it on an invalid message queue. All bad.
If only there were someplace to queue up these messages that cannot be processed, so that they can be processed later.
Therefore, use a standby queue. When a message cannot be processed, move it to the standby queue.
This seems great. The message is out of the way but not lost. Processing can continue on other messages on the queue.
But wait: If some of the messages fail, won't they all fail? The queue should be a datatype channel, so the messages should all be doing the same sort of thing, thus all requiring the unavailable resource; so why keep reading from it? Let's say it's not a datatype channel, that some of the messages need database A and others need database B. Then you need two standby queues, one for messages that need database A and one for B messages; so that when one database is available again, you'll know which messages to retry. With two standby queues, A and B, you've now created datatype channels, so you should have just done that in the first place.
Anyway, once you get the messages on the standby queue, how do you get them off again? You don't want to do it until the resource is available again, but how do you know when that is? When it is, what do you do with the messages? Move them from the standby queue back to the input queue? How?
Standby queues also make transactions very difficult. When you move a message from one queue to another, you need to do so in a single transaction so that the message cannot be lost. But if the transaction tries to use the resource and fails (which is the whole premise of this (anti)pattern), then the transaction is shot, so it can't be used to commit the message onto the standby channel. So separate transactions need to be used to read the message off the input queue and update the resource, which is a lousy transaction model that can easily duplicate or loose data.
So standby queues are bad. I wouldn't use them.
What should you do instead? Use The Message Consumer Rollback Pattern. If the message is valid, but you can't process it right now, don't bother putting it on a dead letter queue or an invalid message queue or a standby queue, just roll back the transaction and the message will roll back onto the input queue. It's the best way to put things back the way they were and wait until the resource is available again.[Read More]
So how do you tell if something is wrong with a message? Here's a pattern.
A messaging consumer receives a message.
How can a consumer make sure it has received a valid message?
A consumer can only process a message if the message is valid, meaning that the message's format and contents fit the consumer's expectations. Otherwise, the consumer won't be able to process the message and make sense out of it. If the message contents are supposed to conform to a particular XML schema, the contents had better be valid for the schema. If the message is supposed to contain a message/request ID to use as a reply's correlation identifier, the message had better contain that ID.
Part of the value of detecting invalid messages is so that they can be put on an invalid message channel, kind of an error log for invalid messages. This gets the bad messages off the main queues and on to a side queue where error handling code can try to do something with them.
Some consumers can assume that all messages are valid, that only valid messages are put on the queue. But that's often not a safe assumption.
For data validation, the consumer could try to commit the data to the database and see if that fails; but if the commit does fail, then the transaction is ruined. The consumer should be a transactional consumer so that messages are not lost, but if the transaction fails, then it can only be rolled back, after which the message will just be consumed again and fail again. To put the message on the invalid message channel, the transaction must be able to commit successfully.
So invalid messages need to be detected, and must do so without invalidating the transaction.
Use a message validator, which parses the message format and checks the data, making sure that the message is valid.
A message validator can be a separate object--you pass in the message, it answers whether or not the message is valid. Or the validator can be the message processing code which errors out if a problem is encountered--for example, parsing XML data using a validating parser. In any event, the validation must be contained in the consumer's transactional context, so that once the consumer detects an invalid message, it can move the message to an invalid message channel and commit successfully.
If the message is valid, but processing it fails, then the cause must be a problem with the resources being used to process the message. As long as the message is valid, save the message to retry it again when the resource is fixed by using message consumer rollback to put the message back on the queue.[Read More]
What happens when messaging systems go bad? Specifically, when a consumer can receive and read a message, but cannot do what it says to do? Here's the answer in a pattern-ish sort of form.
Here's the pattern:
A messaging consumer (either event-driven or polling) receives a message and successfully parses it. But when the consumer tries to process the contents--such as performing an action, storing some data, or reacting to an event--the processing fails; probably because the app doing the processing uses a database and the database is down, inaccessible, overloaded, etc. So the app gets an error that the processing cannot be done.
When a consumer cannot process a valid message, what should it do with the message?
The app could throw the message away, but that'd be bad. It could try to process the message again, but that'll probably fail again. (The definition of insanity is ...) It could put the message on an invalid message channel, but that's misleading because there's nothing wrong with the message; the problem is with the consumer's app that can't successfully processes the message.
This is what transactions are for. Hopefully the consumer is a transactional consumer.
The consumer should be transactional and should rollback the transaction, therefore putting the message back on the queue.
This way, the message remains on the queue as if it were never consumed in the first place. It waits on the queue until it's consumed again, and hopefully when that retry happens, the app will succeed in processing the message this time. If the consumer is a message-driven bean (MDB), transactions are part of the EJB component model, so the rollback basically happens for free (because the EJB container does it for you).
Moral of this pattern: Use transactions, they're your friends. For more info, see "Configuring and using XA distributed transactions in WebSphere Studio," one of the articles I've written (also listed in my author's spotlight).[Read More]
Enterprise Integration Patterns is #2 on a Top 10 list of enterprise books.
eWeek Magazine has a series of articles called "Pings and Packets." It seems to be kind of a blogish recap of recent news tidbits. In any event, it has started to include a list called "Amazon.com's Top 10 Enterprise Books." There are lists for December, November, and October.
Looking at the list, it seems to include any computer books with "enterprise" in the title. So an architect study guide and an EJB book make the list, but Core J2EE Patterns does not. In any event, Patterns of Enterprise Application Architecture is #1 on the list and EIP is #2. Pretty cool.
Rational has a Pattern Solutions web site, from which you can download tools to generate implementations of common software patterns. It includes new patterns by IBM for SOA and ESB.
The tools are packaged as Reusable Asset Specifications (RAS) (an OMG standard for what are basically executable templates) for Rational Software Architect (RSA). For more info about RAS, check out Bill's RSA mentoring.
Here are some developerWorks articles with more information:
Here are some related resources that may be helpful:
Some patterns don't seem to really be patterns, they just seem to define common
What reminded me of this issue is a post concerning Enterprise Integration Patterns. The author, Mark Carlson, states:
So far, I've found that many of the "patterns" in the book fall into what I would have normally called definitions. Examples of these foundational patterns include Message (66), Message Channel (60), Message Endpoints (95), Point-to-Point Channels (103), Publish-Subscribe Channels (106) and others.
However, I don't want to pick on Mark. Since Gregor and I started working on the book, I've heard several people express that concern. I'm just quoting Mark as an example.
This debate has occurred for years, and I probably won't settle it today, but here's my $0.02 worth.
Not too surprisingly, I contend that these items are patterns, not just definitions. What's the difference? A definition tells you what an item is. A pattern tells you not only what it is, but what to use it for and how. The definition of a Message Channel tells you that it transmits messages from senders to receivers. OK, that's nice to know, but so what? The Message Channel pattern tells you when to use separate channels, so that you start to get some idea of how many you need and why you can't use just one. The definition of a Message tells you that it contains data to be transmitted between a sender and receiver. The Message pattern tells you why you can't simply dump a data structure or a bunch of bytes into a message channel and hope to have it come out the other end properly, how you need to design your messages to have a format that the sender and receiver agree on. As pattern documents, these sets of prose should (and hopefully do) have some Therefore, BOOM! to them, which helps you understand why the problem is difficult to solve and then solves it anyway.
I often feel like one person's pattern is another person's "obvious solution that we use all the time." This discussion comes up a lot at PLoP. The sentiment Mark expressed seems similar; one person's pattern is another person's "definition of the term." It probably seems more like a definition to readers who already know the term, but it's new news to readers who are still learning these terms.
If it's a language or product feature, is it still a pattern? The JMS names for Message Channel and Message are Destination and Message. A Point-to-Point Channel is what JMS calls a Queue, as do most messaging products such as WebSphere MQ. But these are still patterns because you need to know not just that they are features, but when to use those features. Sean Neville addresses this issue when he says, "When a pattern's tactic or implementation strategy is standardized, the pattern does not wane in usefulness, replaced by that standard; instead, quite the opposite occurs." A pattern that is a product feature is a pattern that's much easier to apply!
Patterns are decisions: What should you do and when should you do it? Many definitions do not make good patterns because they don't represent decisions. As discussed, Message is a pattern because you need to decide when to use one and how many. Our pattern document explains that a message is composed of two parts, a header and a body, thereby defining what those terms mean. Header and body are not good patterns because you don't decide when to use them; you get them whenever you decide to use a message. OTOH, a good pattern might be "header field," because sometimes you have to decide whether and how to put a value in the message header. You might need your own custom header fields for a request ID to be used as a Correlation Identifier, or as a flag to be used by a Content-Based Router. Notice that I'm not talking so much about what a header field is--that's a definition; I'm talking about what a header field is used for--that's a pattern.
So don't think of patterns as definitions. A pattern document that just defines a term shouldn't call itself a pattern. Think of them as the fundamental building blocks of a pattern language, the parts we've all got to agree on before we can get to the harder stuff.
BTW, I wonder whatever happened to the webMethods paper Mark was working on? It sounds interesting; I'd like to take a look at it.
Antipatterns should be ameliorative. For that matter, patterns should be too. For that matter, so should you, at least in the way you do your job.
Ameliorate (verb): to make a situation better or more tolerable
I've been talking about antipatterns. It's not just enough for an antipattern to give a common solution, to say "Bet you did this, didn't you? D'oh!" It then needs to offer a preferred solution. This makes the document ameliorative; it helps the reader understand not just why their situation sucks, but also figure out how to make their situation better.
Any pattern should be ameliorative. It tells the reader what to do to make their situation a good one, perhaps better than it was, in any event better than it would be if they applied a less-than-best practice. If a pattern documents an approach which is not ameliorative, whatever it's documenting is not much of a pattern.
When you're doing your job, are you ameliorative? Are you making the situation better? Or worse? Perhaps you're trying to make it better but the situation fights back; that's just a bad (but common) situation. But in a reasonable situation, if you're not helping to make things better, why not? See the importance of leadership. Leaders ameliorate.
So, be ameliorative. Get out there and ameliorate.
A well written pattern has a style we call "Therefore, BOOM!" We've now decided that a well written antipattern has a style we've christened "Therefore, D'OH!"
Last week, I helped run a conference on antipatterns. We ran it very much like the PLoP conferences. It's given me the opportunity to teach a new group of people about patterns and pattern languages, writer's workshops and shepherding--all good stuff that I learned at PLoP. It's always fun to be a member of a smart group of people who are eager to learn.
A lot of what I talked about at the conference was my ideas and experience with how to write good patterns, that is, how to write patterns well. I've learned this over several years working with many knowledgeable people at PLoP, and through practice, practice, practice.
One practice we've found for writing a pattern well is "Therefore, BOOM!" I wasn't sure how well this idea would go over in an antipatterns conference, but it took quite well indeed. Being that an antipattern is the opposite of a pattern, we found the opposite of "BOOM!" as well.
We call this quality of an antipattern "Therefore, D'OH!" (And no, I didn't coin the term, but I loved it as soon as I heard it.) It's where the antipattern paper makes you, the reader, realize how stupid (i.e. uninformed, misguided, naive) you were to apply this common but misguided solution. The writing style makes you realize, "Darn, why did I do that?!"
Where does the name come from? "D'oh!" (listen to 32 Dohs (WAV file)) is the sound Homer Simpson makes when he realizes that he's made a mistake. It signifies his sudden realization of his own stupidity.
Much like suddenly realizing that in your actions, you've followed an antipattern.
One quality a well written pattern has, a style that separates it from not-so-well written patterns, is what we call "Therefore, BOOM!"
We figured this out at the first couple of PLoP conferences--writing patterns, reading each others' patterns, and discussing what seemed to work best. Ken Auer has documented this well in "Therefore, BOOM!" It's already described in Sun's Java Patterns Community, so I'll let you read about it there.
Simply put, as a pattern author, to the extent you can write your document to embody this "BOOM!" quality, the better your pattern is. I believe that it makes your pattern better in at least two respects; it makes your pattern explanation:
"BOOM!" is one of, if not the, major reasons why well-written patterns work so much better than just using standard prose. A pattern is not an experience report, and it's not an account of a cleaver trick; it's a mind-meld from an expert to a novice of what works well. It's a best practice that you know not just in your head, but in your heart and your gut as well. "BOOM!" gets that gut feeling across in a way that words alone cannot.
Think about a pattern you've read that really resonated with you, where you knew as soon as you read it that the author knew exactly what he was talking about. (And if you've never had this experience, you need to read more and better patterns.) Remember how you felt (not what you thought, but how you felt) when you read that pattern? That's "BOOM!"
I've recently been reminded of what a great set of conferences the PLoP conferences are. Let me take this opportunity to explain why.
The PLoP conferences were started by the Hillside Group, a collection of visionary computer guys. They wanted to take Christopher Alexander's ideas about patterns and pattern languages and figure out a way to apply them to our jobs of developing computer software. They decided a good way to do it was to start a conference. The focus of the conference would be not to talk about patterns, not to present patterns, but to write them. And the focus would be not to talk about the patterns they'd written, but to help each other improve the patterns they'd written. The focus would be on the writing, and the writing would produce a library of patterns, literature on how to do software development well.
So how does PLoP work? First, you don't submit an abstract of an idea, or a presentation (a la PowerPoint) of an idea, or a paper that discusses your idea; you submit a paper that is the idea. It's not "This is a paper about what you should do," it's "You should do this!" Second, when you submit a paper, it's neither accepted nor rejected, it's shepherded. An experienced author helps you improve you paper to make it good enough for the conference. If a submitter is undedicated, uncommitted, or simply lacks the necessary time and effort, this gets rid of him pretty quickly. But for those dedicated to the task, good shepherding can take an idea expressed awkwardly and make the expression (the paper) much better.
Third, the conference itself is run as a set of writer's workshops. In these workshops, the participants are all authors who have submitted papers they want feedback on for improvement. Non-authors and non-submitters are generally not welcome; experience as a writer is what you bring to the group; you're committed to the group knowing that your paper will be one of those reviewed. There's a workshop per paper, whose purpose it to review the paper (see "How to Hold a Writer's Workshop" and a Writer's Workshop Pattern Language). The paper's author listens silently while the readers discuss what's good about the paper and make suggestions for improvement. As an author, you often wonder what readers are thinking when they read your paper; this workshop lets you know. The focus of the workshop is on helping the author improve his paper.
Notice that so far, I haven't said anything about patterns. Yet the purpose of PLoP is to help with writing patterns (about patterns) and pattern languages. This is the purpose of PLoP. Shepherding and writer's workshops are the method. The focus is on helping interested authors write good patterns papers.
It's a great process, one I must admit that I'd grown accustomed to and bored with over the years. But I've recently become more interested again, as I'll talk about next.