I was reading the latest Forrester BPM report on eBizQ and found it to be quite an interesting  read - especially for the endorsement it seemed to give human process management as a required extension to BPM - without actually naming it as such. That was my only quarrel with the authors - they expect BPM suites to be extended to handle unstructured, ad-hoc, chaotic (their term) human processes. That makes it sound like handling those types of process is just a small feature of an BPMS, a small extension that BPM suites should add. In my experience that isn’t the case - building an system to manage these types of human processes is  no trivial task, and don’t expect BPM vendors to be able to do it - it just requires a different type of thinking -especially since to get people to adopt it you need to unseat an entrenched “competitor” - email.  Here are some of the quotes that relate directly to human process management:

• “in real life, processes change all the time; in fact, our interviews consistently show that processes never stop changing“

• “The outcome of a discounting decision may be captured in the BPMS by integrating or embedding a business rules engine, but the way the decision was made — the reason for the discount — is often recorded in an obscure email thread, if at all.”

• “But many real-world, people intensive processes are so rife with exceptions that it’s impossible to model all the permutations in a traditional process modeling tool. These ad hoc, chaotic processes are difficult to support even using today’s BPMS tools”

OK -so even for the most structured processes in an organization - the ones that have actually been implemented via a BPMS - even those processes are constantly in flux - which means that almost always the users are going to need to morph and change the process before the IT department can reprogram system - no matter how good the tools are. So how is this actually handled in the real world? - no surprise here, it is done via email. These above quotes from the paper make it clear that no matter how well designed the process implementation is - it can’t anticipate every nuance of the process, or every new context - there will always be the need for a tool that allows end users the flexibility to handle the ever changing requirements and demands of real life business processes without IT involvement - while still allowing for management, monitoring and optimizing. Email provides the flexibility, and HPMS built on top of email - provides the rest. If not - BPM initiatives will bring only limited business value.

So in short - even for companies embarking on enterprise BPMS - remember H comes shortly after B, and you’ll need a good HPMS to round out your BPMS.

I was reading various posts about modeling (in the BPMN sense) and it seems to me that many of them are confusing the ability to use modeling tools to codify requirements, and the use of modeling tools to actually generate the code needed to execute the process. 

I remember that we started thinking about model driven development over 10 years ago at IBM Research. If you could only let the business analyst model the process - and then generate the actual executable from that model - what a jump in productivity and agility. It was even worked with various toy examples. The difficulty is that designing a detailed enough model of a process to generate an actual executable program required the same effort (if not more) and the same set of skills that you needed to develop the program to implement the process.  So in reality the model became a spec or requirements doc for the developers. You could imagine these models being a good way to implement agile programming for BPM - where the analysts and developers use those tools as part of an iterative development process - but alas most of their usage is closer to waterfall development than iterative development.

Take a look at the  real life BPEL diagram shown in the drool’s blog - it made the process description the worst kind of unmanagable spaghetti. Even with all the complexity shown, it isn’t even close to the most complex business process you can find out there - whether it is a human process or a business process.

So are BPMN and BPEL a step forward or backward in enterprise application development? I think they are a good way to collect initial, high-level requirements (BPMN) and as a machine readable, system independent language for business models (BPEL). Lets not fool ourselves though, as with any tool for collecting requirements - it ends being a recommendation to the developers - not a production code generator. That is unless you limit the process domain being modeled to processes that consist of a small set of simple, well-defined, recurring, easily tailorable task templates - with relatively simple control flow logic between the tasks.

I have been looking at the mainframe market (yes, those IT dinosaurs that were supposed to be finished in the 1990s). It turns out that plenty of the beasts are still around. Mainframes still host around 70% of the world’s business critical data. That means that even if you are using your bank’s web front-end, there is a good chance that one of the tiers in the application still resides on a mainframe.

Not only are mainframes alive and well, so is mainframe software. CICS, Cobol and so on - they are still in use at many, if not most, enterprise data centers - and they won’t being going away anytime soon. Q32008  IBM System z hardware revenues increased 25% year/ year, with double digit revenue growth in all geographies. MIPS (capacity shipped) grew by 49 percent. And thats just hardware. I couldn’t find any recent data on the mainframe eco-system-but here is a chart I found for the 2004 server market eco-system:

$50B - the market may have changed in the last four years - but I am guessing it is still an impressive number. Given the way this technology is quietly hanging around even with some many trying to kill the market - I think we should call mainframes cockroaches rather than dinosaurs.An added benefit - mainframes are actually a “greener” alternativs to using a plethora of open systems…

The new uses of these existing, legacy systems (e,g, Web Interfaces, SOA, RSS and ATOM feeds) is putting demands on the systems that they weren’t originally designed for. That along will the dwindling number of mainframe skills available - leads me to the key question of how is all this legacy infrastructure and applications going to be managed and maintained…

I was reading an article on Gartner’s “four disruptions that will transform the software industry“. While I was reading it occured to me that three of the four disruptors have the same core - there is a new type of user out there, and they are becoming more vocal about having more control over the tools and applications they use. As John and Claire-Marie Karat wrote in our article ”Affordances, Motivation and the Design of User Interfaces” - “There is a paradox in human behavior that is valuable for designers of applications to keep in mind: Everyone wants to be in control, but nobody wants to be controlled.” This basic truth is driving the “Rise in New Technologies and Convergence of Existing Technologies” disruptor especially around SOA, device portability and mashups. It is also driving the other two disruptors “Change in Software User and Support Demographics” and “Revolutionary Changes in Software and How it is Consumed”.

I think that everyting Gartner says is true - but it isn’t that futuristic - just extrapolating from the trends we are seeing now in early adopters. As William Gibson wrote “The future is already here, it’s just not evenly distributed”.  What I think they are missing is that software is going to have to evolve to support a new type of work, not just a new type of worker. Most of todays packaged apps are around to support the highly strutured processes of the “old enterprise”- and I put BPM tools in that bucket. The next generation of enterprise software is going to have to provide much better support for knowledge work processes. Lotus Notes, MS SharePoint and Wikis are a start in providing support for collaboration - but not for the tacit interaction (or human processes) - which include individualized behaviour and social dynamics. Enterprises are going to need tools for the 80% of human centric business processes that are currently handled through ad-hoc use of email and documents - a Human Process Management System. As you know from my previous posts HPMS’ will be extensions of the way people use email and documents today as their basic framework for tacit interactions (or human processes) with a focus on traceability and flexibility rather than control.
 

I read an interesting blog post by Ross Mayfield today on email overload. It mentioned something that  I have been thinking about for a while - handling exceptions in structured processes (especially B2B). It had an interesting pointer to John Seely Brown and John Hagel’s book The Only Sustainable Edge that most employee time is not spent executing process, but handling exceptions to process. That meshes well with other things I have read and seen from experience.

The key point for me is that as process automation (through BPM and other applications) becomes more pervasive for standard processes, mechanisms to support human handling of exceptions will become more and more important (since that is where employees spend their time) - which means Human Process Management will come to the forefront.

I read some some articles on “Dynamic BPM”, one by Scott Byrnes of HandySoft (http://www.dmreview.com/dmdirect/20070323/1079086-1.html) and a newer take on the same theme by Ian Louw from PNMSoft (http://bpmfundamentals.wordpress.com/2008/09/16/information-overload-making-a-case-for-dynamic-bpm/) and thought that perhaps Dynamic BPM could be a possible candidate for a name instead of HPM. So I did a quick search of Dynamic BPM and came back with a blog post by Sandy Kemsley (http://www.column2.com/2008/09/gartner-bpm-dynamic-bpm/) which points to Gartner’s use of Dynamic BPM as a way to descibe the addition of external rules to a BPMS - a very different take on things than the other usage (which focused on email overload).  That brought home something that I suspected - any name that uses a prefix to “BPM” will be considered just an additional feature to BPM - and I believe that HPM is a fundamentally different (though complementary) category.  It isn’t that I don’t think external rules will be a valuable addition to BPM toolkits, it is just that they won’t solve the problem of ad-hoc, evolving, human intensive business processes now being managed through meetings, documents and emails. I think managing those types of processes will require something very different than what is available in today’s mainstream BPMS.

I was reading a Forrester report “The Emerging Technology Trends that CIOs Should Care About“  and was struck by what they call “Dynamic Business Applications”  where they claim that “Most business applications are too inflexible to keep pace with the businesses they support, as they force people to figure out how to map isolated pools of information and functions to their tasks and processes, and they force IT pros to spend too much budget to keep up with evolving markets, policies, regulations, and business models.” I couldn’t agree more, especially since they break this out separately from BPM.

This is especially relevant for the fluid environment needed for the kind of work done by knowledge workers, especially for ad-hoc processes or exception handling. This type of work needs a new paradigm where the ad-hoc, human way people actually handle most process needs to be the guiding mechanism. Knowledge workers use an implicit process framework that they use for their tasks, but almost every instance is some sort of exception. I think this one reason the use of email is so pervasively used by knowledge workers for their processes. This very different from the structured processes that are handled by BPM - and we’ll need a new paradigm for it. I have started to use the term Human Process Management - but I am not sure that is the best name for those ”process frameworks (usually handled via email and meetings) for ad-hoc “unstructured” human centric processes.

I was at the Gartner BPM conference last week. Walking around the vendor showcase, one thing that struck me was how similar all of the vendors offerings seemed to me (with a few exceptions). Sure some were traditional enterprise software, some were SaaS, some vendors stressed one set of features, while another stressed a different set - but to me all the vendors in certain BPM space (document centric vs. integration centric) all looked pretty similar. I am guessing most people interested in a BPMS feel the same way.

What interested me was how they proscribed the process for creating a BPMS based application to implement an existing  business process. For most, the first step is to create a model describing the process - using a BPMN modeling tool. The model is usually created by a business analyst (usually someone in the IT department) that understands the process.  This model is a high level description of the business process which is used to bridge the gap between the business (they understand the process) and IT (they understand implementation and data). What struck me was how much the methodology reminded me a of the “traditional” top down ways of creating software .  Since it very difficult to automatically create the actual complete, executable production system from the BPMN model - the model serves as a requirement definition for the development phase - which is handled by IT. Any end-user iteration and understanding is around the BPMN - which is a very abstract description of the process to be implemented.  This is then handed to the IT folks for implementation - with the standard lag of months between requirements and actual system. This will work fine for processes that are rigorously defined, unchanging and complete, it may work for processes that are rigorously defined with a small number of exceptions - and will completely breakdown for ad-hoc, unstructured Human Processes. The reason is that these ad-hoc human processes are not well defined, and exceptions are the rule. The only way to approach this is same way you approach building human intensive software  - iteratively, working intensely with customers on working prototypes - either low-fidelity or high-fidelity. John Gould and Stephen Boies taught me long ago that iterating on the spec (i.e. requirements or model) just doesn’t work.  I also learned that if you are implementing an existing process - you want to keep it as familiar as possible to the users, which means let users continue to use whatever they are used to (or feels natural to them) whenever possible.

This is why I think that existing BPMS vendors won’t do well in the ad-hoc, unstructured Human Process space. It will require a much lighter weight, flexible (or bottoms up) environment where processes can be easily created, modified and tested in the field - with the turnaround between versions (including the initial version) is measured in days (or hours) instead of weeks or months. I personally believe that the more Human Process Management Systems let users remain within familiar user environments (currently eMail and MS Office tools, Wikis and other tools in the future), the easier it will be to get these systems accepted by the organization and end users.

I have been thinking some more about Human Process Management, especially how it differs from Business Process Management. Certainly one key difference is whether the process is structured or not - i.e. whether you can prescribe the execution of the process based on some model of the business.

It is clear that there are a number of mainstream business processes that lend themselves to such a model (e.g. ERP,CRM), but I claim that most processes in an organization are human2human (or people2people) processes and the tend to be ad-hoc and dynamic. It turns out that even structured processes have a large number of exceptions - that tend to be handled in relatively ad-hoc, case-by-case manner.

I was reading an old blog post by Pravin Indurkar where he looked at B2B EDI purchase order transactions at a small business and found that though there was one standard process -there were 65(!) different variations depending on the nature of the order. While it would be possible to model all the possible process paths - it certainly would be time consuming and expensive. My guess is as soon as you model the 65 different possibility there will be a 66th and 67th that are used to take into account variation and business conditions - so the way these are handled is usually through an unmanaged human exception process. These human processes are either exclusively human to human process (collaboration) or human processes that invoke various systems as part of the process (or what Barry Briggs calls human-down processes).

These types of Human Process are far too fluid and dynamic to be made part of an Enterprise BPM system - and tend to handled through email - yet another cause of email Information Overload…

I have been thinking about documents and their usage context in organizations. Knowing how a document is used is just as important as knowing the content, though today’s document repositories don’t really know the usage context for the documents they store. At best they and let user try make up the gap with tagging and descriptions. Most human centric organizational processes entail the use of various documents as a natural part of the process - either as input to the process (e.g. research or background) or output (e.g. a findings report).  So the link between documents and their process context is a natural one, and critical if you really want to understand the document.

So it is surprising to me that this hasn’t come up more as an issue in document management systems - the need to really connect documents and the flow of the human centric process that uses them - even if the process is an ad-hoc one executed (as most are) over email.

You could decide to implement all processes as a workflow in a document mangement system - but for many processes that would be overkill (especially the ad-hoc kind), would just take too long and require to many IT resources - not to mention that it would require the users learning a new way to do things. If you decide to keep the documents in a standard repository - then you lose the connection between the process that used or generated the documents - which means that you really can’t understand how the document is actually used in an organizational context…