My friend and sometimes co-author/editor Mark Little has a new post talking about recent work in extended transactions, by which he means transactions that relax the strict ACID requirements of traditional transaction management (of course, these are never really ACID to begin with, but it's a useful fiction that we can live with). In fact, our transaction book reviews different transaction models that helped inform the work on more general frameworks including the work in the OMG Activity Service and WS-CAF, as Mark notes.
For a long time, I thought that extended transactions would be very important. Actually, I thought they would be in wide use by now. In 2001, I was very interested to see what would become of BTP; the answer, at this point, is some very interesting additions to the literature and a few failed attempts at commercialization. The devil was in the details: with BTP, the application may subsume the role of the coordinator at which point it becomes less and less clear why the protocol is needed at all above and beyond the business logic itself. This led us to focus more on choreographies, with the notion of defining the business protocols through some formal structure that may or may not be automated by "transactional" infrastructure. The experience in recent years has led me to believe that transactions may be the wrong paradigm altogether for broadly distributed systems. The trouble is, we don't know what the right paradigm is yet.
Mark notes a recent paper by Pat Helland as an important read. I agree, but my conclusion is also increasingly heterodox with respect to the transactions community: that is, Pat's paper should be taken at face value to be about writing application logic. There are patterns here that need to be understood by application developers. Perhaps supported by frameworks. But it may simply be that transaction management won't play a broad role in that context.