Data Science, Machine Learning & API / SOA: Insights and Best Practices
Ali_Arsanjani 120000D8QB Tags:  artificialintelligence ai_ethics ml machinelearning ai machine_learning 2,851 Views
Ali_Arsanjani 120000D8QB Tags:  virtual_agent intent-based_architecture watson_conversation_servi... chatbot 3,007 Views
Chatbots or Virtual Agents are rapidly ramping up to augment the human computer interaction starting from Self-help and gradaully moving up the Knowledge management chain to how an Agent (such as a technical support call center agent) uses a specialized chatbot inhouse for Agent Assist.
Some of these technologies are nascent, others more mature, such as Watson Virtual Agent.
The first thing that a virtual assistant does is to detect your intent: this is accomplished using Natural Language Understanding and Natural Language Classification. The process starts with a recognition of the goals of the personas interacting as well as being training on the most commonly recurring calls, requests, chat transcripts, etc.
What we are talking about here is more than Search or Text Analytics it is about Intent based understanding of content.
Search is more about Federation of search indices, Content level security, Content display facets, filtering, Community & Social extensions, Connectors ERP, DBs, etc.
Content Analytics on the other hand is about finding relevant words in text counting occurrences, Analytics for entity & associations extraction, Integration with analytics & prediction systems.
Intent-based Architectures allow you to understand the query down to the users intent, Execute interactive query refinement to be actionable, Generate a recommendation, Interactively access data with implied meaning & relationships Establish word / phrase proximity, document relationships.
All these capabilities are predicated upon the discovery of intent: intents that can be reused across personas in the interaction dialog .
An intent represents the purpose of a user's input. Each capability uses a natural language classifier that can evaluate a user utterance and find a predefined intent if it is present.
For example, a recommended practice is to review the intents that are identified most often just before users request to speak to a human agent. Investigating the causes of escalations can help you prioritize where to focus future training efforts. You can determine whether user inquiries are being misinterpreted, whether your service is missing a common intent altogether, whether the responses that are associated with an intent need improving, and so on.
Ali_Arsanjani 120000D8QB Tags:  cognitive_systems traceable_ml traceability governance cognitive_system_governan... 3,897 Views
In the age of voluminous unstructured data that can be curated manually, semi-automatically or eventually, with a fuller degree of automation with added approval processes and governance in place, it is more than ever necessary to secure not only the data, but secure the process around which data is curated, prepared, an ML algorithm(s) selected with resulting data being used trained for ingestion into ML algorithms, the curation of the data : data collection and aggregation sources (lineage), data preparation, including redundancy removal, null value replacement, range consolidation, homogeneity of content, etc.
Thus the age of Traceable Machine Learning Governance is born. Standards bodies should embark on this consolidation of views, across vendors, consumers, services organizations, data providers, curators and other stakeholders in the ML Training Life-cycle.
The problem of governance of ML is one of not only data governance as it passes through traceable, steps until ingested into an ML, but also of the process around the collection, curation, selection of ML and training set selections and partitioning.
So, here is our call to action:
1. Standards Bodies need to consolidate a standard around Traceable ML Governance to reduce the risk of fake training, bogus data used to train a "paper tiger ML", which has no substance.
2. Corporations should give serious consideration during the ML and Cognitive Computing Training Life-cycle to secure a Traceable ML and Cognitive Governance process. This process can begin in a lightweight fashion, but secure legal implications arising from the use and administration of Cognitive Systems using Machine Learning (ML) to make recommendations, provide insights, generate summaries, reports, news, reviews, etc.
3. Furthermore, as global impact of well-trained Cognitive Systems (ML systems, AIs) becomes more and more tangible, we will want to have demonstrable traceability on how these systems were trained, retrained and where the human in the loop influenced the AI and how it was brought to bear (SME, curation, etc) on the source data, where data was sourced, how it was curated (whether human, or other ML or Cognitive System). To do so, traceability in the Cognitive System life-cycle or the ML-training life-cycle will play a cardinal role in its adoption, trust and veracity of recommendations.
Many claims around Deep Learning being beyond governance or governability is a bogus claim founded only in the illusion that 'since the hidden layers of a Neural Network and their interactions are "dark matter" so we cannot govern their inputs and outputs.' In fact we can and should secure the training process, as well as the inputs and output configurations of ML systems in Logs that are themselves inspected by ML systems with a human in the loop approval process in place.
4. Organizations and Standard Bodies should consider the use of Blockchain technologies to secure and govern the process chain within the ML-training or Cognitive system life-cycle in a demonstrably traceable manner, which with little doubt will gradually find its way in local and global legislation.
Ali_Arsanjani 120000D8QB Tags:  longitudinal time-sequenced machinelearning rnn lstm ml 3,542 Views
Traditional artificial neural networks (ANN) have a memory problem : they cannot recall their previous reasoning about events to inform new ones.
If you have time-sequenced data, i.e., longitudinal data, you might want to consider using a Recurrent Neural Network (RNN) that allows information to flow from the past to the present using a feedback loop. Just as we experience the world and our sensors (senses) provide feedback to us constantly to reevaluate an unfolding situation.
Events are a connected series of vectors (or tensors) that have a way of bringing past experience to bear to the next step of the deliberation.
In recent years, RNN’s have enjoyed an “unreasonable success”, to quote Andrej Karpathy of Stanford University. These include speech recognition and categorization, language modeling and translation, image captioning and recognition at multiple points in a sequence of images, etc.
So RNNs add a feedback loop to the Neurons and allow information to flow without having a reset or restart everything they are executed.
Long Short-Term Memory (LSTM) is a variation of RNNs that have been most effective in processing these kinds of longitudinal data. We will further discuss them in a next entry.
Ali_Arsanjani 120000D8QB Tags:  gradientboosting machinelearning ai training ml cognitive 3,392 Views
Ali_Arsanjani 120000D8QB Tags:  bigdata mlinfused combine_structured_unstru... machinelearning datascience 3,816 Views
Context and problem. You have terabytes of structured data, petaBytes of unstructured data, that are not quite visible, and many areas are dark, they provide less valuable information. You have lots of dark data, but not a whole lot of insight. How can you not only see summaries and means and graphs of trends, but to gain insight, that you can count on to associate specific value-adding business activities and based off of them, make strategic business decisions. How can you shine light on all the dark data that is sitting in those ObjectStores, relational databases, datawarehouses, content stores, voice, images, text,....
Considerations. Leveraging data, through analytical processing is not just about processing structured data often residing in the organization's many systems of record or transaction-processing or even online analytics processing databases. It is rather about the ability to combine unstructured data coming in either from IoT devices, which are semi-structured data, and from content-based (think Enterprise Content Management (ECM)) systems that contain images, attachments, documents and text in free format.
Solution Path. Extracting data from unstructured content or text from images, transforming semi- and un-structured content into structured data, storing it in a DataLake and possibly into other Case Management systems, will allow you to start gathering the raw data you need to start your curation and data wrangling.
Then you can apply Data Science and Machine Learning to the curated Datalake to gain insights, and incorporate the conditional actions you wish to take as part of a BPM or Case Management solution. Alternatively you can wire a set of micro-services via APIs exposed from Software as a service vendors, to evaluate the insights and based on certain thresholds, invoke a workflow or display a result for human knowledge workers to take action.
Ali_Arsanjani 120000D8QB 3,625 Views
Ali_Arsanjani 120000D8QB Tags:  cognitive_systems ai ethics ai_ethics machine_learning data_curation augmented_intelligence 5,306 Views
Most major businesses are embarking on augmenting their analytics capabilities with Machine Learning. They are either demonstrating or are planning projects that showcase how machine learning can be applied to their applications, with Machine Learning. (This is called #MLInfused.) Enterprise software development firms are attempting to bring impactful predictive capabilities into their suites of products. IBM Watson through either Bluemix APIs or IBM ML On prem offers such capabilities.
When a mortgage application is submitted, ultimately human underwriters make the decision based on a set of rules. Although we cannot claim that the process is completely understandable, we can probably hold a human plus the regulations accountable in an audit, court case or compliance situation to demonstrate lack of unjust discrimination or bias. Enter Machine Learning. The data we use to train such a system, the humans who will curate and annotate the data, and the process they went through (biased or unbiased) will have a significant if not cardinal impact on the trained ML algorithm and thus it's recommendations and output.
To prepare ourselves as a society of scientists, engineers, businesspersons and regulators, etc., for a world where such processes and data will have major impact on individuals and society, we need a set of rules and regulations. They will come in due time forced by errors and omissions and uproar. But to prempt that we suggest for consideration some "laws" or best practices around machine learning of cognitive systems.
1. A cognitive system will not be trained on dark curated data . Yes maybe the hidden layers of neural networks are a black box, but the data itself should not be "dark": it's sourcing, inputs, annotations, training workflow and outcomes should be transparent. Training, testing and validation data sets should be traceable, white boxed and accessible for enterprise governance and compliance .
2. The data curation process will be transparent. Ends do not justify means : transparency of data curation process for machine learning is paramount . This means traceability and governance around where data is sourced, who curated and annotated it, who verified the annotations .
3. Cognitive system recommendations must provide traceable justification . Outcomes must be coupled with references to why they made the decision or recommendation.
4. Where human health is at stake cognitive systems with relevant but different training backgrounds will cross check each other before making a recommendation to a human expert .
These practices or initial variations of them should be considered as part of an overall governance process for the training, curating of datasets for machine learning of cognitive systems.
Ali_Arsanjani 120000D8QB Tags:  agile governance api microservices soa 1 Comment 4,436 Views
Fred Brooks has taught us in his Mythical man Month that if a project is going off track and running behind schedule, then adding people to the project will only make things worse: the ramp-up time to orient and on-board developers will have reached diminishing returns. We have have also seen this countless times on projects we have been involved.
But where is the root cause? Not what, but where. Say you have a multi-tier architecture, and the business logic tier or the service component layer, the layer or tier that implements the server side functionality. The backend logic is/can often be represented as some form of an object model, or class diagram. This object model is where the problem is.... Class diagrams or object models, represent the design that implement services in the services layer, which indeed do decrease complexity and risk and decouple systems and make life easier for everyone.
When a developer is onboarded they have to learn the complex convoluted backend dependencies of the business logic tier, essentially they need to digest the object model , the domain object model, the class diagram (or whichever other representation or name you choose to give it). Most of the time these dependencies are unnecessary and the rampup time increases because they feel they have to digest so much of what everyone else is doing.
So, partition the object model, the domain model, into a set of smaller (perhaps as close as you can get to 5-9 main classes) . These partitions of the domain model allow a natural separation of concerns and allow developers to ramp up much faster.
This has nothing to do with building microservices, or small grained services. In fact if you focus on smaller grained services only you will fall into the Service Proliferation trap.
It's about building tiny application not tiny services. Tiny applications can have medium or even one or more larger grained services.
The success of microapplications lies not in the granularity of the services they are composed of but on the size & complexity of their underlying domain model, which focus only on one functional area.
Remember, tiny domain models result in small code bases. Large code bases overwhelm developers, slow their development environment and increase dependencies and complexity, and make integration even harder.
Ali_Arsanjani 120000D8QB 4,328 Views
SOA (service -oriented architecture) evolved to reflect our deepening of understanding in separating interface from implementation from deployment, not just pro grammatically, but architecturally and in a business impactful fashion. It introduced a new layer in the software application architecture focused on well, just services or really, service description or service contracts. Interfaces were merrily separating interface from implementation in the object-oriented world which we take for granted today. But this was limited to mostly a programming paradigm.
SOA elevated this separation of concerns between interfaces/contracts, implementations and deployments higher in the foodchain: to the level of an architectural construct.
As we were compelled to move implementations/deployment locations off premise, in view of the savings, flexibility (elasticity) and consolidating power of the cloud, we moved to created or using software as a service. Software as a service, or , SaaS is primarily a software licensing and delivery model that has grown out of the SOA worldor consolidate them in a private cloud. Software is licensed on a subscription or "pay as you go" basis and is hosted often as a managed service, but sometimes in a public , private or most often hosted scenario. Platform, Infrastructure and other XaaS kinds of software creatures started evolving out of the cosmic goo of the ubiquitous and elastic cloud model.
SOA has morphed into RESTful APIs as the implementation or realization mechanism of choice. When people decide to use whatever underlying technology they wish and choose whatever programming language or framework they want and deploy little applications that are pretty much standalone, they tend to use the term microservices. This is often a misnomer, since the granularity may have historically started from fine grained services, but has gained stability and a more balanced medium grain service stature.
So how do we apply SOA principles to the build Cloudready applications ? We build APIs. Here are some tried and tested recommended practices for successful API development.
API best-practice 1: Use Tiny object models for design. SOA, implemented, not, with one huge underlying object model, but rather with a partitioned set of smaller object models in its design phases, smaller models that you can divvy up and feed to smaller independently functioning teams has been particularly useful and an adopted best practice. What is tiny? 7 +-2.
API Best Practice 2 : Each Team manages their own Service Portfolio. Plan the services that you need in your functional area. Maintain a categorized list of services that may breakdown into smaller grained services. Try to keep them as stateless as possible.
API Best Practice 3: Each team owns a Functional Area. When you start breaking up the design object model into tiny object model with minimal dependencies, each of those parts better align to an area of the business : a functional area. These can be different departments or more finer grained divisions such as a shopping cart, a product catalog, an order, etc.
API Best Practice 5: Each team Deploys Independently . This is so they are not on the critical path of another team, can be testing and running relatively independently. There will be semantic integration, dependencies and connections that are necessary, for example a single sign on security or session token that may need to be passed.
API Best Practice 6: Each Team Builds the finest grained APIs possible. Build RESTful APIs for the leaf nodes and medium grained services in your Service Portfolio.
API Best Practice 7: Have an Integration team that governs the overall Service Portfolio, and all things integration. Allow API access to the database that each team requested (fields, tables and all, or go NoSQL if you wish). The Integration team monitors and manages the dependencies between the teams, the functional areas.
Hot off the press. Hot from the BPM Oven. http://www.redbooks.ibm.com/redbooks/pdfs/sg248282.pdf
As a Business Process Management (BPM) architect, you always seek to design business process management solutions that deliver on the promise of business agility and business-centric visibility, among other things, while also ensuring that your design meets the preferred practices for IT architecture, security, and solution design. Often, this task can be daunting because your BPM solution must bridge the gap between business teams and information technology teams. In this post, we introduce five basic concepts for designing an effective business process management solution using IBM Business Process Manager (IBM BPM).
1. Architecture considerations
It is important to understand the practical considerations involved in high-level solution analysis and architecture of an IBM BPM solution. Considerations such as these:
2. Security considerations
BPM processes contain the very essence of a business, even including things such as competitive advantages and trade secrets. Despite this importance, and despite the fact that security breaches make embarrassing headlines, IBM repeatedly finds customers who consider their BPM projects secured simply because the BPM servers are hidden behind firewalls. So there are a number of specific items an architect must consider when planning how BPM will be integrated into an enterprise’s IT environment, including:
3. Solution design considerations
Implementation of business processes using a business process management software can be challenging and, at times, overwhelming. As a BPM architect, you should seek an understanding of the following items as you embark on the solution design phase, prior to implementing your business processes in IBM BPM:
4. Business-centric visibility
Business-centric visibility is one of the most important considerations for organizations looking to adopt business process management. Business-centric visibility, in the context of a business process, includes these factors:
In order to achieve any meaningful business-centric visibility, it is important to have access to both the business data and the process data. Business-centric visibility is achieved in IBM BPM using these capabilities:
5. Performance tuning and IT centric visibility
Performance tuning and IT-centric visibility are also key aspects of the IBM BPM design. In the context of these aspects, you need to keep in mind the following important tips:
Ali_Arsanjani 120000D8QB Tags:  bpm rest soa analytics microservices mobile cloud big_data api patterns 12,448 Views
Microservices reminds us of very fine grained, tiny, micro-scopic services. Perhaps services that grow on a Petri dish.
But actually, they refer to a deployment pattern for services developed in a Service-oriented Architecture along the constraints of a Functional Area that owns its own Data, and does not allow access to its Information willy nilly. You start with a domain break it down into a Functional Area and get down to the subsystems and components living there, deep in the bowels of the legacy systems, the packages inhabiting the digital ooze in the sewers of legacy code , interspersed with some fresh sprinklings of new additions of components implementing newer services required by the business.
Stratified along a Functional Area, immune to the curious and invasive glances of strange components and outside systems, the service rests peacefully in isolated, independent deployment. Choosing to deploy to brew this SOA flavor with a twist of independent deployment, along the lines of a functional area, owning its information, is detailed out in the implementation of finer grained services in SOMA (Service-oriented modeling and architecture) ,many companys' service-oriented method.
Don't mistake this as the only way to choose to constrain SOA by design along functional area boundaries and deploy along those lines. This is only one brew of SOA.
Some considerations are the need to communicate between deployed services that are other wise feeling a little isolated and engaged heads down in their servicing of requested for their functional area of the business. But business have cross cutting concerns and dimensions, often exemplified by BPM or business process management. Cutting across silos or functional areas, the process needs access to various data sources and systems of record, albeit some are prehistoric gauging by enterprise standards.
Sp cross cross cuts of SOA flavor are best served by BPM. Smaller granularity focused functional area services by Microservices. What about when mobile needs to access something? Well for that we have the API brew. RESTful (Representational State Transfer) APIs provide an HTTP/S level of simplicity of access (yes, both get and set) that allow access to underlying functionality via mobile devices and yes Software-as-a-services, or Cloud as is known in slang.
So choose your brew of SOA according to your mood, according to what you would, contingent on what you should
be doing with your processes, data or business functions. Choose wisely for each have pluses and minuses; opportunities and consequences, like any pattern. Or brew.
Ali_Arsanjani 120000D8QB 4,967 Views
What are API’s critical to enterprise business agility?
Today we see the importance of a confluence of mobile, cloud-based, service-oriented applications that were hitherto locked within the enterprise break free and start interacting with an ecosystem of partners opportunistically in order to create value on emerging opportunities especially those provided by mobile devices, social business, big data and analytics and cloud-based services. It is important to understand the evolution of APIs in order to better capitalize on opportunities presented by the economical dynamics of an ecosystem connecting and communicating through APIs.
The evolution of APIs
API’s or Application programming Interfaces were initially thought to be libraries of reusable code. They were focused very specifically on providing functionality that was written one often to support a complicated set of granular operations such as a graphics library, telecommunications library, security etc.
In the first decade of the 21st century, application programming interfaces morphed because of service-oriented architecture. The protocols used to implement Web services in a service-oriented architecture were numerous. The simple object access protocol or soap was one means of access. This sort of replaced RMI over IIOP in a distributed computing world.
Service-oriented architecture taught us that not only does interface have to be separated from implementation but also that implementation can be separated out into several layers. One layer is a more abstract specification of where the endpoint for the service implementation may reside. Often an enterprise service bus was used to virtualized the endpoint so that the optimal endpoint could be selected based upon configurations or input parameters or more pragmatic considerations pertaining to security, scalability or performance.
Implementation was separated into a realization decision, And a deployment set of options. The realization decision was primarily governed by the question: “how am I going to implement the service? Which component is going to be used to implement the service functionality?” The component could be anywhere from a.net component, and enterprise JavaBean or a legacy application interface or even a package application. The deployment option included not only the protocol by which the implementation would be realized but also the configuration options pertaining to the infrastructure or platform used to ultimately operationalize the solution.
Representational State transfer or REST was a protocol created to support a very lightweight mechanism that would replace the more complicated SOAP protocol and therefore could use HTTP or HTTPS. Therefore the verbs that could be used where the familiar get and put actions familiar to web programmers.
Restful APIs were APIs that extended enterprise capabilities hitherto reserved for webpages into the ecosystem. This ecosystem of partners who were now able to interact using restful APIs, created the opportunity for an API economy.
The characteristics of API-based services
At the core of the enterprise the concept of service-oriented architecture suggested the creation of the service model. Within it is a service portfolio that is created, governed and managed as an enterprise asset. This asset , the Service Portfolio represents the implementation of business capabilities that the enterprise would like to expose experience brings of partnership and security either within the enterprise but also extending towards the edge of the enterprise and more selectively, beyond it, to business partners.
In summary characteristics of the API-based services provide the enterprise the ability to :
· Externalize Business Capabilities for ecosystem consumption
· Capitalization and monetization of Enterprise Assets outside the organization
· Extend value of Enterprise capabilities beyond the enterprise
· Secure access to internal enterprise resources and capabilities
· Enable channel-agnostic connectivity with business partners and consumers
Internally, the APIs will be part of the enterprise portfolio that can provide elimination of redundancy through reuse of capabilities as well as providing governance through consistency of created assets.
Externally, exposure of internal capabilities to the partner ecosystem will allow value-added services to enhance existing services provided, bringing in a new set of clients.
Pitfalls and Best-practices
Lack of alignment of an API to a clearly defined set of business goals can be detrimental to the adoption of the technology and capabilities behind the API.
APIs should not focus on time sensitive technologies or merely on fulfilling user interface capabilities that access backend systems. An example of this is to limit the capabilities of an API to CR UD-based operations such as create, read update and delete. This severely limits the capabilities of expansion of the API into higher levels of maturity of an application lifecycle.
Exposure of architectural decisions that stem from interorganizational considerations that in turn have sprung up over time as a result of the attempt to maintain the big ball of mod that constitutes the legacy systems that have been developed over generations within the organization. One best practice is to mask the convoluted interdependencies on technologies, architectural decisions that were made due to expediency and legacy platforms and the scars of partially successful mergers. The API should not expose such backend architectural decisions and implementation considerations just as any good service in a service-oriented architecture would mask these backend assumptions that have no rationale beyond the walls of the enterprise.
Secure a path of growth so your applications can start evolving from merely accessing backend legacy systems to performing transactions, initiating business processes and engaging in decision management.
Designing for the glass. Designing mobile applications is a current fad which is driven by the strong adoption of mobile devices. This can be a severely limiting factor is the application is designed from the glass in words in other words consideration of the backend business processes must be taken into consideration along with considerations of usability at the screen level for the mobile device of choice.
The ability to orchestrate and compose APIs in order to provide a mediated or orchestrated experience of the business process is a critical success factor with the maturation and increased adoption of APIs within your ecosystem.
Ali_Arsanjani 120000D8QB 4,783 Views
Context: Industry adoption of development, life-cycle and architectural styles such as SOA tend to be imported into organizations and then move from an evangelist, grass roots kind of effort into a next phase of organizational maturity which tends to impose rigid constraints on the life-cycle, standards, governance and development of the architectural style. SOA is an example.
Problem(s): High level , holistic design and architectural work is needed to provide contractual visibility to clients. Architectural styles (OO, CBD, SOA etc) are co-mingled with the rigor of the top-down governance that is seen to be required, or they are sacrificed for rapid, quick and dirty, "let's not look around the bend until we get to it" approach.
Forces: The developers tend to push back to the imposition of increasing accountability and governance in general, including standardized development practices. The organization , in an effort to be accountable to the business, requires increased tracking, metrics and accountability. Developers will continue to resist increased accountability due to the perceived imposition on their time that is deemed less valuable than the actual development of the product. Against this, accountability and project visibility, allowing projections in order to fulfill "promises" (e.g., contracts, SLAs, etc) with clients (whether internal i.em business as a client, or external, as in other organizations that services/products are provided to).
(Re)solution: Combine a light top down governance and standardized software development process/method that looks holistically but does not require detail to the n-th degree prior to engaging in a project or program, coupled with agile iterative sprints.
Consequences: High level visibility and accountability are attained with some compromise with the "let's just do it" approach. Development is not impeded with heavyweight constraints. Promises to the business or external clients have a reduced risk of non-completion associated with them.
Ali_Arsanjani 120000D8QB 5,061 Views
SOA , as an architectural style and a discipline of software engineering brings business and IT to the same table by providing a portfolio of business intended services/capabilities as the common terminology that he business requires and IT will design, implement and maintain. SOA is focused on establishing a center of operations for design around a portfolio of services, which equate to a set of business capabilities, that the business looks to deliver. Integration has become a strong focus for SOA as well. There are various types of services, integration and business services to name just two categories.
In the traditional Application Programming Interfaces or APIs were merely a library of interfaces that components can access and communicate with each other. In an evolved definition that is becoming more and more prevalent, the notion of a RESTful API that is exposed outside an organization, that may expose a set of services in the backend is being promoted more widely, especially with the proliferation of interest in HTTP-based simple interfaces that can mobile enable an enterprise's subset of back end functionality that it chooses to expose.
Cloud services are services that are exposed externally, and can be consumed by all, often implicitly assumed to be outside an organization's boundaries just as an API or more precisely, a RESTful API. Cloud services may span the gamut of Infrastructure as a service (IaaS) all the way to Software as a service (SaaS) which may expose very large grain applications or smaller levels of granularity akin to the RESTful APIs we discussed above.
More detail on this later.
Ali_Arsanjani 120000D8QB 5,081 Views
Business process Management or BPM is about streamlining workflow and introducing efficiency into a process of a goal oriented business process.
A Smarter Process is a process that is augmented by decisions and business rules that are part of the activities within the process flow. An event will have triggered the process flow due to the monitoring of a situation or of bubbling up the analytics we are collecting to generate an event. Backend systems need to be accessed, invoked for a transaction or connected to for some outlandish reason such as updating a system of record. These summation of integrations, process flow, rules, events and analytics are collectively called a Smarter Process.
Architecture, software architecture specifically, used to be only defined in terms of components, connectors and constraints per Shaw and Garlan.
However, it is increasingly evident to those of us who implement architectures on a daily basis for clients and internally within our own companies
that architecture requires the addition of composition, context and containers to make it operational.
Components -- what are the structural building blocks of a solution, or a style of architecture, of the elements that can be combined to produce a larger structure
§Composition -- How to compose or what are the valid compositions of components
§Connectors -- The components need to connect with one another, whether statically (as in an Entity-relationship kind of relation) or dynamically, as in a composition in an SOA, where you may have orchestration or choreography.
§Constraints -- The constraints on the connectors and/or components that provide rules of engagement of what is permissible and what is not.
§Context -- the context of invocation of a component is critical to the designation of how that component will behave
§Containers -- the components must live in some runtime container that will provide uniform life-cycle management capabilities for them
The short answer is:
Cloud is an initial application of SOA to infrastructure
--its setup, configuration, monitoring , management , with the essential elements of metering and billing added to satisfy the goal of a charge model of e.g., pay-as-you-go
-- the specific requirements for XaaS: resource pooling and virtualization , elasticity and multi-tenancy, dynamic configuration and provisioning
Cloud paradigm leverages SOA to deliver a charge-based model for non-business specific services;
Several people have written articles on this relationship:
Towards a more precise definition of Business Agility: The Building Blocks of Business Agility (B3A)
Ali_Arsanjani 120000D8QB Tags:  agility building_blocks_of_busine... business_value business b3a 7,884 Views
Business Agility is an often discussed as a key desirable attribute. One of the ways of achieving Business Agility is through the portfolio management of a set of business capabilities and services , rather than application portfolio management which tends to intertwine the services and higher order business capabilities into an application -- not a robust and flexible way of enabling business agility and optimization.
The main building blocks that provide a platform for business agility are outlined below. I have elaborated the last one more than the others in this post, and will elaborate further in future posts.
Laws governing eco-system as a whole (e.g., to disallow financial market meltdown in the wake of blind rule based automated short selling in e.g., financial markets)
The ecosystem includes policies, rules and laws governing Business providers and Business Consumers, and Business Brokers.
environment, market, legislation and ecosystem within which the business is
operating and evolving, forcing to change and vary based on forces within the
Business Context. The Business Context is a
There are many changes that are constantly occurring within the business context. Only some are business significant or should be “bubbled up” or surfaced at an executive and business level, especially when Business Sensors detect a certain threshold above which apparently ordinary events become important enough and achieve a tipping point beyond which the business sensors should indicate that a potentially impactful variation has occurred.
Business Entities model and reflect the key business domain elements of the Business Eco-system that under Business State changes.
Many EA (Enterprise Architecture) practitioners are …exploring the links between desired business outcomes and architectural decisions.
A tipping point is reached when IT begins to understand that business executives are not primarily looking for products and services but rather looking for business outcomes including increased output, higher quality, lower costs, increased revenue and increased market share.
To effect the transition to become more business-driven and engaged with business leaders, we should focus EA efforts on business outcomes.
Instrumentation and mechanisms to monitor, track, help modulate and govern business change as key performance indicators track the events and occurrences within the Business ecosystem Context.
Typically, the board and top executives of a company state the vision and strategy for the company. It is critical for implementation of business performance solution in the company that the senior business leadership translates and decomposes the high level vision and strategy statements of the executive leadership to operational and actionable goals associating KPIs to the actionable sub-goals on the way.
To achieve such end-to-end monitoring, metrics and sensors must be injected in both planning processes and delivery processes .
Essential to this process is the Business Sensors that detect Business significant Events and pass them on to the Business Policies and Rules that would respond to them.
From a planning perspective, enterprise governance needs to ensure that the right changes are initiated for the right reasons at the right time. The underlying premise driving towards business agility is that such agility delivers superior business value. But what if haste to achieve agility results in low quality? Or what if speed of change is unsustainable from a business operational perspective, thereby leading to deteriorating efficiency? These are just two examples of the fundamental challenge that doing the wrong things in the wrong way very fast simply means ruining your own business very fast. There are two fundamental premises for agile change to be valuable over time:
For agile change to be sustainable the enterprise needs to carefully plan and maintain an appropriate balance between effectiveness and efficiency. Change in the large is based on continuous business re-engineering towards strategic objectives (effectiveness). Yet while on that strategic journey an enterprise needs to apply change in the small to continuously adjust and optimize the current state and ultimately maintain business integrity and performance (efficiency).
There are two levels of monitoring – there is the monitoring of the acting on plans and there is the monitoring of operations.
From a delivery perspective, the set of solutions designed and implemented to achieve the business goals should be designed for monitoring. Monitoring of business results is a key to business performance. Monitoring of business results enables sensing of business significant events. Ability to act upon to respond to the business significant events resulting from the business activity monitors is essential to improve the business performance in order to meet the business goals.
Therefore, business goals, solutions adequately instrumented for monitoring and alerts, monitoring of business results, sensing of business significant events, responding to these business events to take corrective or predictive actions are key for a business to be in a path of success and growth that can be adjusted and tuned based on actual results of the business.
Ali_Arsanjani 120000D8QB 6,786 Views
Based on our project experiences over the past years, we feel there are 6 key ingredients of success with SOA:
1. SOA 101 -- basic understanding of key SOA concepts e.g.,
2. SOA Maturity Assessment and Roadmap (using OSIMM ; Open Group Service Integration Maturity Model) that implements an
3. SOA Strategy, outlining the path and projects and initiatives to embark upon for sustained success with SOA enablement for business agility
4. SOA Method (SOMA) that provides the prescriptive guidance as to what to do within the software and business architecture life-cycles
and yet work in tandem with
5. SOA Governance to institutionalize the key processes needed to sustain success
6. Consideration of an SOA Reference Architecture now a industry standard,
Ali_Arsanjani 120000D8QB 10,234 Views
I've participated in a lot of SOA projects around the world and very often I'm asked to answer questions about aspects of SOA. These are often recurrent questions that most of my clients and her colleagues are asking. So a colleague of mine, Kerrie Holley and I decided to consolidate these questions and answers into a book so that more people can access the questions, see if they are relevant to their particular circumstances and hopefully benefit from the responses.
The book is entitled 100 SOA Questions: Asked and Answered .
I have started a separate blog at soaquestions.blogspot.com to discuss further questions and comments specifically about the book .
Ali_Arsanjani 120000D8QB Tags:  sustainable adaptive performance agile business_performance business 10,088 Views
Following the thread on Agile and Adaptive Business Performance and Optimization
Business Performance (BPer) is a measure of key performance indicators over a period of time. Within that period, we indicate there is BPer if the anticipated metrics and KPIs are exceeded by the actual KPIs.
For this to happen, Business Processes need to have been identified and engineered appropriately to achieve business goals measured by the KPIs.
The underlying Services that are orchestrated to produce the Business Processes should be selected or identified using a rationalized Service Identification method
(e.g., SOMA -- Service Oriented Modeling and Architecture) that encompasses the needs of various aspects of granularity, performance, agility, flexibility and complexity.
The Business Architecture should follow an agile and adaptive model to incorporate changes needed to accommodate sustained business performance and allow for continuous optimization.
Having an underlying Service Portfolio that allows the selection and combination of those services in nex contexts into Business Processes is one of the key factors in achieving and maintaining sustainable Business Performance.
I will discuss other factors in my next entries.
Greetings Dear Readers.
I would like to pose a question, to which I would like to invite your responses or further elaborations. Then we will discuss this in greater depth, as I begin to discuss my own views on this.
In your opinion how does a company achieve and maintain sustainable business performance?
What are the factors impeding the achievement of desired business performance?
When Service identification Becomes Critical: Service Proliferation Syndrome and Service Interface Changes
Ali_Arsanjani 120000D8QB Tags:  service_portfolio governance service_proliferation service_model service_monitoring soma 16,267 Views
As our industry continues to mature in the adoption of SOA and related technologies, we are witnessing not only the subsiding of hype but also, the emergence of new issues that are surfacing due to growing pains. These issues include the need to change service interfaces as they evolve in conjunction with changing business requirements. The service portfolio which is the single point of reference for an enterprise's business capabilities undergoes evolution with external and internal changes that are ongoing within and without the enterprise.
As the Service Portfolio, part of the Service Model, matures, changes to the service interface, and the service contracts are inevitable. I was approached with one solution: to put them under change control. However expedient this measure, it is not a cure, but temporary stabilizers of organizational turbulence. Changes to the underlying IT capabilities that reflect the business needs that are dealt with in a proactive manner tend to stabilize over time and yield the agility that helps power sustained business performance.
Properly designing the services, i.e., proper Service identification techniques and practices are paramount for alleviating these types of issues. Second best is to use a registry and repository to store and manage them centrally. And of course you need a source code control system to manage the code that is going to be using the services.
The use of correct Service identification for alleviating the risks and impact of Service Interface and Service Contract changes coupled with management and governance in a Service Registry and Repository are thus paramount.
Further, as services abound and the organization feels more comfortable in the creation of new or modified (see above) services, the need to manage and govern this Service Proliferation Syndrome becomes increasingly important. Again there are two aspects : one of which is doable at design time and another at runtime. Service Identification coupled with Service Refactoring and Rationalization (SRR) help mitigate the first aspect -- design time best-practices. Secondly, the monitoring of Services at runtime come hand in hand with the Service identification and Monitoring approach to alleviate Service Proliferation.
it is important to source IT constructs that directly support the business goals and processes. In order to do so we will explore the six fundamental constructs of service-oriented architecture. Elevation of the notion of the service or service interface and later on the service contract from a programming construct to the level of an architectural abstraction was a key paradigm shift in software engineering. By the service does not stand alone. It needs to be implemented using some component. The data has to be passed to the service to the component back to the databases and data warehouses and information systems in the backend, processing needs to be completed information retrieved and returned to the requester of the service. Thus information is paramount for services. As a service proceeds with its processing there are certain rules that apply. The invocation of the service should be a more intelligent invocation than that of the method on an object. Generally speaking it is common practice to invoke a method on an object without the necessary cards and without the necessary application of rules. It is typically left up to the application logic to implement those rules. With service-oriented architecture is important to although externalize the rules pertaining to a service but make the application of those rules visible to the service consumer or service requester. The service is also governed by a set of policies that allow the service to be changed without having to fundamentally change the implemented capability behind the service. Services should not only respond to a direct invocation but should be able to respond automatically in a subscriber will fashion to the triggering of that even
Thus it is critical to understand the six fundamental constructs of service-oriented architecture: services, processes, components, information, rules and policies and events. Services can be a comic or composite. As they participate as the steps of the business process, services can be invoked at each one of these steps allowing a shared set of common services to be utilized in a BPM context. therefore, processes are also one of the fundamental constructs of SOA.
Your architectural description should capture these six constructs as you meander through the journey of the software development lifecycle.
also, note that all of the six constructs are motivated by business level constructs. The following diagram establishes a relationship tween the business constructs and the corresponding SOA paradigm constructs.
A lot of market buzz is going on around BPM (business process management). This is a resurrection of one of the buzzwords of the earlier decade. However this time, it is BPM powered by SOA. Service-orientation is the underlying factor that will make a big difference in BPM. Rather than Business Process Management being solely about the management and optimization or re-engineering of business processes in traditional workflow contexts, with SOA, the activities the underlying orchestrated process (composite services) can be flexibly changed within a shorter duration, provided governance is fine with the change, and thus enable a greater degree of business agility.
Over the years as I have taught, designed and implemented software architecture the question I have been asked over and over again has been:
What is Architecture?
According to the Arsanjani view of Architecture :-), the Architecture of a system is the holistic view of the relative configuration of a set of static and dynamic elements that include what I call the 6C's :
Components -- what are the structural building blocks of a solution, or a style of architecture, of the elements that can be combined to produce a larger structure
Connectors -- The components need to connect with one another, whether statically (as in an Entity-relationship kind of relation) or dynamically, as in a composition in an SOA, where you may have orchestration or choreography.
Constraints -- The constraints on the connectors and/or components that provide rules of engagement of what is permissible and what is not.
Composition -- How to compose or what are the valid compositions of components
Context -- the context of invocation of a component is critical to the designation of how that component will behave
Containers -- the components must live in some runtime container that will provide uniform life-cycle management capabilities for them
Traditionally, the first three have been the main focus in universities and textbooks. But the latter three that I have added seem to be essential to a more complete depiction of the operational or actionable perspective on software architecture.
Let me know what you think.
Ali_Arsanjani 120000D8QB 7,261 Views
To engage in an eco-system, you need to sense the environment, the eco-system. Business sensors gather business significant events and apply business policies to see what actions need to be taken as they sense a threat, an advantage, a change in the business climate, in the eco-system architecture.
Thus, to measure business performance and monitor business agility, you need what I am coining as "business sensors".
Ali_Arsanjani 120000D8QB 10,107 Views
Architectural Paradigms of the Future Part 2: Eco-system Architecture engulfs Enterprise and Business Architecture in the Cloud
Ali_Arsanjani 120000D8QB Tags:  service-oriented-architec... enterprise-architecture business-architecture eco-system-architecture 17,097 Views
In the context of a cloud formation, where is the "enterprise", per se? Of course it exists in the traditional sense. And so does enterprise architecture. But in the context of a cloud, which may span private, public and hybrid instances, where is the "enterprise" now?
Rather than a physical location, the enterprise now seems to be more nebulous and "logical" or "conceptual". It now can be seen to expand possibly, with the cloud, as the amorphous mass expands so does the virtual enterprise. This is where the enterprise merges with the eco-system and you see the emergence and need for addressing aspects of the eco-system that span physical and logical enterprises. As we needed enterprise architecture, for the cloud we need an eco-system architecture.
Eco-systems are more dynamic, opportunistic in some instances; some shorter and some longer enduring alliances between interested business parties. This not only affects the IT side, as it were, in a "bottom up" fashion, but also the business architecture; in a more "top down" manner.
The structure and function of a business is defined by its model of what goods and services it provides and consumes; who it needs to partner with at what point in its dynamic existence. This relates to policies that cut across and senses the eco-system for more appropriate alliances and takes competitive advantage of this new configuration. A business architecture in an eco-system architecture is then configured by rules and policies driven by competitive business goals, not statically defined.
The cloud provides a virtual rockbed for the reconfigurable enterprise that supports its dynamic business architecture to sense and respond, anticipate and adapt to the ebb and flow of trends in the eco-system. The eco-system architecture defines and configures these elements that include the traditional business architecture, multiple enterprise architecture plus a set of rules, policies, that define desirable (to be sought) and undesirable (to be avoided) configurations of the business and IT as they ride on top of the virtual infrastructure provided by the cloud.
This constitutes the eco-system architecture of the future.
As the eco-system of business evolves into a more opportunistic convergence of business partners into an eco-system of value, similar to the notion of a value chain, but a more loosely coupled exchange of goods and services, the need is increasingly being felt for an upfront design of an architecture structure to enable smooth and measurable business operations.
Eco-system architecture defines the foundation of interaction between a set of semi-autonomous business entities that interact with each other using a set of configurable policies that define the valid business interactions among potential and prospective participants, such as business partners: providers, consumers and brokers.
Eco-system architecture extends Enterprise Architecture in a similar way that EA extends the notion of solution architecture. EA is analogous to "city planning" whereas ECSA is analogous to planning how the economy of the whole nation is structured through a set of rules, policies, standards and structures .
Even though SOA has been around for years, the community at large did not have a clear declaration of intent and "values", although individually, there appeared to be a large degree of consensus. Formalizing this consensus, a group of seasoned SOA practitioners got together recently and agreed to a high level declaration of intent.
Service orientation is a paradigm that frames what you do. Service-oriented architecture (SOA) is a type of architecture that results from applying service orientation.
We have been applying service orientation to help organizations consistently deliver sustainable business value, with increased agility and cost effectiveness, in line with changing business needs.
Through our work we have come to prioritize:
That is, while we value the items on the right, we value the items on the left more.
We follow these principles:
© 2009, the above authors, this declaration may be freely copied in any form, but only in its entirety through this notice.
We have just published the SOA Manifesto . This document helps guide the values and principles of service-orientation and service-oriented architecture.
"Service orientation is a paradigm that frames what you do.A set of 6 values are then outlined, followed by a set of 14 principles that help guide the implementation of the values.
When faced with the forces outlines in the value statements, we provide a preference for one rather than the other, even
though both may be viable under certain circumstances.
I will post links to other authors of the manifesto soon.
The IBM Service-oriented Modeling and Architecture(TM) (SOMA) is IBM’s end to end software development and integration method (aka, process life-cycle) based on service-oriented software engineering principles to produce enterprise and individual project-level SOA or non-SOA solutions. This method includes prescriptive guidance applicable during the phases and iterations required in a software development process life-cycle. This guidance includes the activities and tasks to be performed by various designated roles within the life-cycle requiring input work products and producing or updating output work products and deliverables at specified junctures within the life-cycle. The tasks leverage a set of techniques (“capability patterns”) that prescribe and promote the use of best-practices included in the method, in conducting and accomplishing the task and to instantiate the related work products and deliverables.
SOA enables business agility. It enables flexibility of IT systems that support the business architecture. As the business changes, the IT systems required to support the business in a volatile environment of competitive engagement, are less prone to change.
The history of software engineering started with the separation of functionality or processing and data. The data of the processes that manipulate the data were traditionally separate. There came a time when as the pendulum swung emphasis was placed more on the data in one era ended and the pendulum would swing and emphasis would be placed more on process. Eventually object oriented programming, which evolved into object-oriented design and object oriented analysis, played a key role in the unification of process and data. David Parnas was the first to suggest the notion of information hiding. The notion of information hiding was that access to information or access to data was strictly done through the functionality provided by an object. Those that object encapsulated its data and protected it from direct manipulation. It offered a set of operations that you could perform on the data, but you could only invoke those operations do not access the data directly.
Objects often reflected real-world objects. The identity of the real world object was used as an abstraction and implemented in a software system as a software object. Identity often corresponded to a real-world entity although helper objects evolve from the IT constructs necessary to implement the real-world construct in a computer system.
One of the most important principles of object oriented programming was based on a separation of concerns not only in the domain of abstraction but also in terms of separation of interface from implementation. Not only did we separate data from the operations that manipulated the data all in one object, but we did so one step further by separating the interface of the behavior from the actual implementation of that behavior. The behavior or rather its implementation, actually change the state of the object or as it were manipulated the data directly. However, there must be several ways in which we can implement this data manipulation. Even though there may be several ways in which we can implement this manipulation, there is typically one way to represent the interface or the externally visible signature that would be used by a consumer to request a change to be made to the data that the object owned.
This principle is called programming to interfaces rather than implementation.
These interfaces coupled with the notion of composing a set of similar objects that often had interdependencies among them into the next level of encapsulation which was called the component lead to a whole new era of -based development. It exported interface and could contain multiple objects all of which would typically be expected to be related to one another in some logically cohesive fashion. These objects were expected to be highly collaborative with one another and thus made sense to be called located within the packaging of a single component. This allowed not only manageable functionality but also decrease the risk that nonfunctional requirements would be violated.
The promise of object orientation
What is Cloud Computing? Similar to the Initial era in which SOA was trying to be defined, we are now in the era of more definitions. That is not in itself a negative thing, but a necessary part of the formation of new technology as they become mainstream.
Cloud computing is essentially an extension of SOA and service oriented computing. It provides concentrated infrastructure, platform and data center like capabilities offered at a set of services over the Internet. The cloud has several unique characteristics.
The cloud itself needs to be viewed from both a consumer and a provider perspective.
from the perspective of the consumer they are looking for a service-based delivery model over the web in which a highly reliable efficient self optimizing and feed your head up infrastructure capability are made available to them.
From a Provider perspective, the cloud involves two key aspects of self optimization and virtualization using a service-based delivery model.
The service-based delivery model can operate under one of the following kinds of services: platform as a service, software as a service an infrastructure as a service.
in order to connect the consumer with the provider, the provider must publish in some metadata standard information interchange format the quality of service that it can provide and the services that it can provide, so the service consumer can select among various provider of various services they provide and the various qualities of service that are offered for each type of service.
Could computing is fundamentally a service delivery model. It is instantiated by an architecture that provides the services and is consumed by a Service Consumer's own internal systems.
cloud computing is a
service delivery model
in which a virtualized set of
are offered by Cloud Service Providers with the choice of quality of service.
Service Consumers then have a choice
as to what provider to select
Further more, the model should support a seamless portable view of service such that services
from multiple providers can be choreographed by a consumer or broker without having to
know about the inner workings of the underlying cloud infrastructures.
What do you think?
Ali_Arsanjani 120000D8QB 7,505 Views
Amidst the current economic downturn, and in the mist of the survival instinct that is perculating around the world, it is tough to sit back and observe trends.But one unmistakable trend is that countries are shifting from an industry-focused economy (e.g., manufacturing, see the collapse of manufacturing toward a service economy. The impetus for this gradual change include globalization, technological change,and an overwhelming demand for (cheaper and cheaper) services. In observing this shift, it becomes increasingly clear that services and the service economy play an important role in today's and tomorrow's businesses. As a reflection of this trend, a realization or crystalization of the trend, service ecosystems have emerged: including GrandCentral, eBay, Google Base, Amazon.com, SalesForce.com, and SAP Business by Design to name a few. Such marketplaces enable trading products and also services between various legal entities and with consumers (sometimes referred to as the anonymous legal entity). One major challenge for service ecosystems is the fact that services are not products or goods. This has the potential to change the game.
Ali_Arsanjani 120000D8QB Tags:  service-orientedcomputing maturity pervasive services 8,464 Views
SOA continues to evolve through economic downturns and bends in the road. SOA has evolved into a set of underlying principles, concepts, technologies, paradigm and patterns of designing and building software; all predicated on a set of loosely coupled services that collectively support an organization's business processes and goals.These services provides the enterprise with an adaptive or agile capability to morph and transform itself in the face of environmental change.
As we look at the SOA Maturity Model we have called SIMM (Service Integration Maturity Model), we see a clear evolution path:
1. Simple Services (level 4.0 to 4.9) from 2005 -20072. Composite Services (level 5.0 to 5.9) from 2007 - 2009
now we see a lot of interest in paradigms and technologies that are BASED on Service principles, namely
3. SIMM level 6, Virtualized Services, in the form of Software-as-a-service (Saas) and Cloud Computing.
Services have become Pervasive, they can be found in many of the recent initiatives including Mashups, BPM, Infrastructure, Enterprise Architecture, Governance, etc.
The notion of an SOA is finding new applications in new areas and continues to morph and integrate into new things, ingesting new use-cases but retaining the base principles of service-oriented computing.
Ali_Arsanjani 120000D8QB Tags:  soa manners externalization voad variations voa 2 Comments 9,509 Views
In an effort to be responsive to a number of requests for an update to my old manuscript for "Principles of Advanced Software Engineering: Variation-oriented Analysis, Design and Implementation", let me give you folks some background on the subject. [Background]Originally, I wrote a manuscript back in 1999-2000 and used it to teach my classes of Graduate students in computer science for my advanced software engineering class. Since then I and a number of my colleagues have written several articles on this subject and have extended it into various domains: Variation-oriented engineering, variation-oriented requirements analysis, VOAD for SOA Solutions in addition to the base variation-oriented analysis and design. This topic also relates to my work on patterns for stability and symmetry in software architecture which I was writing for the pattern languages of programming conferences. I also did a blog entry on VOA that describes the high level basics of this paradigm.[More Current] VOAD is an integral part of the SOMA method for building end to end enterprise and applications solutions based on SOA principles (whether or not you have a full SOA implementation in mind. VOAD intersects with another very important notion that I have been researching and applying on projects, namely, context-aware services (CAS). Manners externalize semantics for on-demand composition of context-aware services.Manners define the context-aware behavior that is needed in order to externalize component manners to achieve greater maintainability and reuse. This article shows how you apply VOAD in order externalize the changing aspects of the software, so you can decrease maintenance costs, enhance reuse and actually increase flexibility in design and implementation of your software solution. This is a rather popular article and has been cited in a number of other publications . I think this is because we show the practical application of VOA to create a dynamically reconfigurable architectural style. Also, I believe this element is one of the characteristics that will be needed in more mature SOA Solutions. So as organization's adoption increases, they will need more mature ways of doing SOA, and VOAD is a key to achieving that.Empowering the business analyst for on demand computing explores Grammar-oriented object design (GOOD) which is the way in which we apply VOAD for externalizing manners. This article describes a project we did in the public sector for harmonization of multiple redundant business processes across the world. This is immediately reminiscent of the value that VOA brings: one of which is to consolidate business processes and optimize them by eliminating redundancy.I suspect that is why I am getting these emails on putting more out there on VOAD and to update my older work to reflect the impact and opportunities for SOA. I get the message :-) and will do so.
A new article in the IBM systems journal is out on SOMA that describes the method in a bit more detail than I did in 2004. This article describes a method that has now exercised hundreds of projects successfully and thousands trained worldwide in its delivery. This edition also features a number of articles on SOA.
Here is the abstract:
Service-oriented modeling and architecture (SOMA) has been used to conduct projects of varying scope in multiple industries worldwide for the past five years. We report on the usage and structure of the method used to effectively analyze, design, implement, and deploy service-oriented architecture (SOA) projects as part of a fractal model of software development. We also assert that the construct of a service and service modeling, although introduced by SOA, is a software engineering best practice for which an SOA method aids both SOA usage and adoption. In this paper we present the latest updates to this method and share some of the lessons learned. The SOMA method incorporates the key aspects of overall SOA solution design and delivery and is integrated with existing software development methods through a set of placeholders for key activity areas, forming what we call solution templates. We also present a fractal model of software development that can enable the SOMA method to evolve in an approach that goes beyond the iterative and incremental and instead leverages method components and patterns in a recursive, self-similar manner opportunistically at points of variability in the life cycle.
SOA is the next step in software engineering. The service-oriented paradigm comes to an increasing degree of maturity roughly 5 years after what I consider to be the start of serious corporate adoption.
We migrated from procedural computing to object-based and object-oriented computing. Then we moved to improve the state of objects with component-based software engineering. Now, we have evolved our best-practices as an industry and in academia and have moved into the era of service-oriented computing.
The gradual adoption and service maturity can be seen to move along a spectrum we have depicted in the Service Integration Maturity Model.Here we have seven levels of maturity across 7 dimensions from infrastructure to the business architecture.
The implication of this paradigm shift is that the best-practices of service orientation need to be integrated into the fabric of software engineering to be utilized on projects that (this is important, so you may want to mute the TV for a second) SOA may either not be part of it or a very small part of the project. Still, you would use SOA PRINCIPLES such as using the service model for a specification of the required functionality within a project or an enterprise.
What are your views on the use of SOA principles on non-SOA projects?[Read More]
In the world of SOA, the economics of funding determines what services actually get funded to be developed. This is why we need a set of criteria by which we can assess which candidate services are a priority for inclusion in your next budget cycle. We call these criteria or gating factors, the Service Litmus Tests (SLTs). SLTs are included in SOMA , the de facto end to end SOA Development Method .
Not all candidate services should be implemented as services; we may not have enough budget for that. But here this: we STILL need the functionality of the services that fail to pass the SLTs. Therefore those services still need to be implemented as part of some service component or realized by an application.
Economics is hand in glove with governance and of course SOA Gov includes the issues of economics, funding and budgets and ownership.
Ali_Arsanjani 120000D8QB 6,219 Views
Lots of SOA projects later, I have finally gotten back to this blog.
I am seeing a large surge in SOA projects, each with greater maturity and more complex needs; but some of the basic remain the same. Often, projectsface unpredictable and complex human situations which may defy rigorous algorithms and require the soft art of consulting. Some are attempting to bridle this erratic and unpredictable aspect of human and group behavior. More of the SOA Projects later.
A recent post by the Univ of Arizona talks about the development of a software that sifts through tons of data and "will use sophisticated computational methods based on game theory, co-evolution and genetic development models to find solutions that make sense in illogical times. Genetic algorithms analyze situations in an evolutionary context, where actions with the highest “fitness factor” (chance of achieving the greatest success) gravitate toward one another, produce offspring and eventually rise to the top."
This form of evolutionary, convergent behavioral computing promises to be used in more and more simulation situations.Typically, rule engines are cluncky (technical term) and finding relations between multiple rules is a tricky proposition.Patterns can help. For example, the Business Rule Pattern Language I have documented starts with a lighteweight way of handling rules and moves into increasingly complex ways of handling rules within object oriented applications.
Follwing its use on three past projects, I have recently revised this pattern language to support SOA. I have gotten quite a bit of email requesting this and I am responding by saying that I will be publishing a draft soon.
Ali_Arsanjani 120000D8QB 6,158 Views
As people stood in line sometimes a day before the big day, June 29, I was thinking of the partnership between Apple and AT&T. What about other Service providers who provide wireless services; could they not interoperate and have a device, which theoretically might be provider agnostic work with their phone was well? Of course economics, partnerships and politics are the key driving forces here...
So in the world of SOA, economics, partnerships and politics play a significant role as well. In the world of SOA, the economics of funding determines what services actually get funded to be developed. This is why we need a set of criteria by which we can assess which candidate services are a priority for inclusion in your next budget cycle. We call these criteria or gating factors, the Service Litmus Tests (SLTs). They are embedded in SOMA , the de facto end to end SOA Development Method.[Read More]
Ali_Arsanjani 120000D8QB 6,093 Views
SOA Solutions are most often hybrid solutions. Yes, they focus on a set of services; but they often do not soley rely on services for the realization of the functionality that needs to be in place for the business.
SOA solutions tend to rely on a combination of architectural styles and implementation and realization constructs to craft the underpinnings of an SOA solution. The SOMA method utilizes a combination of approaches to Service Identification. This includes 6 perspectives: top-down business process driven, business policy and rule-driven approaches, bottom up legacy integration, bottom-up legacy transformation (intrusive changes to rip out legacy modules and expose them via access points), information as a service typically used to consolidate multiple backend data stores and resolve inconsistencies in the access, rules and synchronization of the data stores and lastly the message-driven approach which seeks to integrate systems using a service interface.
Among the approaches above, although some are more established than others, information as a service affords a unique perspective in solving challenges relating to information. For example, data access interfaces and their underlying data access logic might need to be externalized judiciously if multiple channels are seeking to access and manipulate data from potentially multiple access points. The need for consolidation, synchronization and management of the data along with the need to have a coherent set of policies be applied to the data calls for the information service “entry point” to SOA.
Service-oriented modeling and architecture (SOMA) has become the industry de facto standard for SOA Methods. Introduced by IBM in Jan 2005, released recently as RUP/SOMA 2.4, it covers the identification, specification, realization and implementation of services, components, flows (processes), information and composition.
SOMA uses Information Analysis, Modeling and Planning during identification, an Information Specification during design and a number of artifacts during Realization and Implementation including considerations for Enterprise Information, Master Data Management, Conceptual, Logical and Physical Data Models.[Read More]
[b]S[/b]ervice. Refers to how we use a service interface as a contract to decouple or loosely-couple a service provider from their prospective service consumers. Providers publish services, prefereably and increasingly in registries or service repositories (where they can be better governed). Consumers are looking for capabilities / functionality along with a set of non-functional requirements or service level agreements (SLA) that they want to be able to declaratively specify.
Services allow a business to expose its main operations to a well defined set of partners, clients and world at large, without giving direct access to its underlying IT systems, butthrough the surrogate of a web service.
The structure of an SOA, introduces services at an architectural level (a layer dedicated to services in the architecture).In object-oriented programming we tried to separate interfaces from implementation. IN SOA, we have carried that programming practice up to the architectural level and now have a layer dedicated to enforce that programming best-practice at a design level.
[b]O[/b]riented. The orientation is not object-orientation or component-orientation, but rather, service-orientation. This orientation is a tendency towards using services above the other options; not to exclude those options. Thus, when we apply the service litmus tests to the portfolio of candidate services we will end up with a smaller set of services and a number of other capabilities that still need to be implemented using some technology: legacy, package or custom, even if it is not a web services implementation.
[b]A[/b]rchitecture. SOA is a style of architecture and relies on the sound principles of software architecture, including the fact that it is merely one style, to be combined with other styles to form hybrid solutions to handle complex projects and real-world situations. Booch talks about bringing the 'A' back into SOA. I would say that as we overcome the hype of the acronym, we turn our attention to each letter and take action on what it signifies; including the A part for Architecture.[Read More]
[First a bit of history]. SOMA, IBM's SOA Method is now (officially) 2 years old. On Nov 9, 2004, I published a short paper describing IBM's SOA method: Service-Oriented Modeling and Architecture (SOMA) . We had been working on extending current methods for SOA before that, and I have documented our efforts in a SOA redbook (chapter four) which describes a primitive version of SOMA. Since then, a lot of work went into the method and SOMA took a quantum leap in 2004-2006 timeframe, as we formed teams around it and have successfully completed a very large number of projects in various industries and geographies and have taught about a thousand consultants and many clients to effectively use and deploy this method when designing SOA.Chances are that an SOA project will need to do a bit more than ad hoc web services implementation and need a bit of design. Some may even need some analysis of services components and flows, which are the fundamental elements of an SOA. Current methods do not have support for SOA and its fundamental constructs. This is where you can use SOMA, to identify, specify, design and realize the services, components and flows(processes) of your SOA. Based on a large number of project experiences over 2002-2004, we extended existing analysis and design methods, including global services method, which is an internal IBM proprietary method, as well as the rational unified process (RUP) and added the tasks and work products and roles necessary for the analysis and design and implementation of a service-oriented architecture (SOMA 2.x).
[Cut to present]. Yesterday, IBM announced a number of important tool updates. One of the accompanying plug-ins for the Rational Method Composer includes SOMA on top of the familiar industry method, RUP. Previously, SOMA was only available with IBM's internal Global Services (GS) Method. We have now extended this reach due to popular demand. This was a collaborative effort within IBM between Global Business Services (SOA and Web Services Center of Excellence's SOMA team) and Software Group's Rational division. Simon Johnston and I have been working to deliver SOMA on top of RUP. Not all tasks in a method have to do with services and SOA; so the underlying method, like RUP, can take care of more mundane tasks like use-case modeling, etc.The end result of this work is the release of IBM RUP for Service-Oriented Modeling and Architecture V2.4 , which represents the combination of the Rational Unified Process (including RUP for SOA) and IBM's proprietary method, SOMA. This means that IBM has a single commercial method for the development of SOA solutions, whether you buy that method for your own use or you contract with IBM services; the customer gets the value of the combined experience of IBM's product and services communities.
SOMA is maturing based on new trends and new requirements; so you can expect to see further releases of RUP/SOMA in the future.
It is convenient to enforce the notion of scope in an SOA: "exposed services" only make sense when you define the scope and context in which they will be exposed. We refer to SOA as being fractal. This means that you can apply SOA and expose services in a fractal manner: you can define services for a project, a LOB, a few LOB's , an enterprise, an eco-system. For example, a Service Portfolio (part of the Service Model) will have an attribute of scope that helps define, for example, how each business unit has it's own set of services they use "internally" and also a set of services they expose "externally" to other LOBs and the rest of the enterprise. Each scope can be a Service Provider and a Service Consumer.
This addition of scope and role to the service model alleviates many issues in governance, boundaries, funding and indeed in the identification and specification of services in your SOA.[Read More]
As the need for IT to absorb variations increases, with the demand for greater business flexibility,we are confronted with some basic questions: how do we design simple for today, to get the current job done, but not "box " ourselves in a corner so we can support the required flexibility ?I think one major answer to this problem is Variation-oriented Analysis and Design (VOAD). VOAD consists of three main types or axes of variation: structural (type|data), process and policy/rule variations. Structural variations are often based on the identification of Types: Customer Type has variations like Gold Customer, PLatinum Customer and Normal Customer. Often this relates to variations in the structure of the class, or of attributes and data associated with the entity (looking from both OO and Data views).Process variations are when you recognize that a Gold Customer may start from a common base, but branch out into a different set of activities for registration or loan processing, for example. Policy or Rule Variations relate to the Rules and Policies (Rules about rules) that apply for each Type of Customer, for example.Clearly these three aspects of VOAD are related and complementary. It is often useful, in practice to distinguish and treat these three axes of variation.Once you analyze the variations along each axis of variation, you come up with a set of variation points: things that will tend to change or remain less stable. Deal with each variation point by applying a pattern. For example if a variation point for calculation of interest is required for various types of Customers, then a Strategy Pattern would be used to handle / instantiate that variation point.Note that variations tend to occur across all layers of an SOA..Not all that changes is a variation that is warranted to capture and model: only those that are architecturally significant will be worth your while to consider. How do you tell? An architecturally significant variation is one in which impacts the architectural /design decisions you will make and have a trickle down effect that will influence subsequent decisions in how you will build your architecture (e.g., SOA). The "domino" that will alter the course of other decisions is a significant or relevant variation.ANother FAQ is why focus on variations? Variations are more difficult to handle than commonalities. Previous literature focused more on identification of commonality which IMO has less of an architectural ripple effect than understanding, isolating and externalizing variations.
What do you think?[Read More]