Content, or unstructured Data, has a lifecycle. These are the typical states that content goes through. I will elaborate each one in another entry.
Data Science, Machine Learning & API / SOA: Insights and Best Practices
Ali_Arsanjani 120000D8QB 458 Views
Ali_Arsanjani 120000D8QB Tags:  cognitive_systems ai ethics ai_ethics machine_learning data_curation augmented_intelligence 1,222 Views
Most major businesses are embarking on augmenting their analytics capabilities with Machine Learning. They are either demonstrating or are planning projects that showcase how machine learning can be applied to their applications, with Machine Learning. (This is called #MLInfused.) Enterprise software development firms are attempting to bring impactful predictive capabilities into their suites of products. IBM Watson through either Bluemix APIs or IBM ML On prem offers such capabilities.
When a mortgage application is submitted, ultimately human underwriters make the decision based on a set of rules. Although we cannot claim that the process is completely understandable, we can probably hold a human plus the regulations accountable in an audit, court case or compliance situation to demonstrate lack of unjust discrimination or bias. Enter Machine Learning. The data we use to train such a system, the humans who will curate and annotate the data, and the process they went through (biased or unbiased) will have a significant if not cardinal impact on the trained ML algorithm and thus it's recommendations and output.
To prepare ourselves as a society of scientists, engineers, businesspersons and regulators, etc., for a world where such processes and data will have major impact on individuals and society, we need a set of rules and regulations. They will come in due time forced by errors and omissions and uproar. But to prempt that we suggest for consideration some "laws" or best practices around machine learning of cognitive systems.
1. A cognitive system will not be trained on dark curated data . Yes maybe the hidden layers of neural networks are a black box, but the data itself should not be "dark": it's sourcing, inputs, annotations, training workflow and outcomes should be transparent. Training, testing and validation data sets should be traceable, white boxed and accessible for enterprise governance and compliance .
2. The data curation process will be transparent. Ends do not justify means : transparency of data curation process for machine learning is paramount . This means traceability and governance around where data is sourced, who curated and annotated it, who verified the annotations .
3. Cognitive system recommendations must provide traceable justification . Outcomes must be coupled with references to why they made the decision or recommendation.
4. Where human health is at stake cognitive systems with relevant but different training backgrounds will cross check each other before making a recommendation to a human expert .
These practices or initial variations of them should be considered as part of an overall governance process for the training, curating of datasets for machine learning of cognitive systems.
Ali_Arsanjani 120000D8QB Tags:  governance agile api microservices soa 1 Comment 1,475 Views
Fred Brooks has taught us in his Mythical man Month that if a project is going off track and running behind schedule, then adding people to the project will only make things worse: the ramp-up time to orient and on-board developers will have reached diminishing returns. We have have also seen this countless times on projects we have been involved.
But where is the root cause? Not what, but where. Say you have a multi-tier architecture, and the business logic tier or the service component layer, the layer or tier that implements the server side functionality. The backend logic is/can often be represented as some form of an object model, or class diagram. This object model is where the problem is.... Class diagrams or object models, represent the design that implement services in the services layer, which indeed do decrease complexity and risk and decouple systems and make life easier for everyone.
When a developer is onboarded they have to learn the complex convoluted backend dependencies of the business logic tier, essentially they need to digest the object model , the domain object model, the class diagram (or whichever other representation or name you choose to give it). Most of the time these dependencies are unnecessary and the rampup time increases because they feel they have to digest so much of what everyone else is doing.
So, partition the object model, the domain model, into a set of smaller (perhaps as close as you can get to 5-9 main classes) . These partitions of the domain model allow a natural separation of concerns and allow developers to ramp up much faster.
This has nothing to do with building microservices, or small grained services. In fact if you focus on smaller grained services only you will fall into the Service Proliferation trap.
It's about building tiny application not tiny services. Tiny applications can have medium or even one or more larger grained services.
The success of microapplications lies not in the granularity of the services they are composed of but on the size & complexity of their underlying domain model, which focus only on one functional area.
Remember, tiny domain models result in small code bases. Large code bases overwhelm developers, slow their development environment and increase dependencies and complexity, and make integration even harder.
Ali_Arsanjani 120000D8QB 1,537 Views
SOA (service -oriented architecture) evolved to reflect our deepening of understanding in separating interface from implementation from deployment, not just pro grammatically, but architecturally and in a business impactful fashion. It introduced a new layer in the software application architecture focused on well, just services or really, service description or service contracts. Interfaces were merrily separating interface from implementation in the object-oriented world which we take for granted today. But this was limited to mostly a programming paradigm.
SOA elevated this separation of concerns between interfaces/contracts, implementations and deployments higher in the foodchain: to the level of an architectural construct.
As we were compelled to move implementations/deployment locations off premise, in view of the savings, flexibility (elasticity) and consolidating power of the cloud, we moved to created or using software as a service. Software as a service, or , SaaS is primarily a software licensing and delivery model that has grown out of the SOA worldor consolidate them in a private cloud. Software is licensed on a subscription or "pay as you go" basis and is hosted often as a managed service, but sometimes in a public , private or most often hosted scenario. Platform, Infrastructure and other XaaS kinds of software creatures started evolving out of the cosmic goo of the ubiquitous and elastic cloud model.
SOA has morphed into RESTful APIs as the implementation or realization mechanism of choice. When people decide to use whatever underlying technology they wish and choose whatever programming language or framework they want and deploy little applications that are pretty much standalone, they tend to use the term microservices. This is often a misnomer, since the granularity may have historically started from fine grained services, but has gained stability and a more balanced medium grain service stature.
So how do we apply SOA principles to the build Cloudready applications ? We build APIs. Here are some tried and tested recommended practices for successful API development.
API best-practice 1: Use Tiny object models for design. SOA, implemented, not, with one huge underlying object model, but rather with a partitioned set of smaller object models in its design phases, smaller models that you can divvy up and feed to smaller independently functioning teams has been particularly useful and an adopted best practice. What is tiny? 7 +-2.
API Best Practice 2 : Each Team manages their own Service Portfolio. Plan the services that you need in your functional area. Maintain a categorized list of services that may breakdown into smaller grained services. Try to keep them as stateless as possible.
API Best Practice 3: Each team owns a Functional Area. When you start breaking up the design object model into tiny object model with minimal dependencies, each of those parts better align to an area of the business : a functional area. These can be different departments or more finer grained divisions such as a shopping cart, a product catalog, an order, etc.
API Best Practice 5: Each team Deploys Independently . This is so they are not on the critical path of another team, can be testing and running relatively independently. There will be semantic integration, dependencies and connections that are necessary, for example a single sign on security or session token that may need to be passed.
API Best Practice 6: Each Team Builds the finest grained APIs possible. Build RESTful APIs for the leaf nodes and medium grained services in your Service Portfolio.
API Best Practice 7: Have an Integration team that governs the overall Service Portfolio, and all things integration. Allow API access to the database that each team requested (fields, tables and all, or go NoSQL if you wish). The Integration team monitors and manages the dependencies between the teams, the functional areas.
Hot off the press. Hot from the BPM Oven. http://www.redbooks.ibm.com/redbooks/pdfs/sg248282.pdf
As a Business Process Management (BPM) architect, you always seek to design business process management solutions that deliver on the promise of business agility and business-centric visibility, among other things, while also ensuring that your design meets the preferred practices for IT architecture, security, and solution design. Often, this task can be daunting because your BPM solution must bridge the gap between business teams and information technology teams. In this post, we introduce five basic concepts for designing an effective business process management solution using IBM Business Process Manager (IBM BPM).
1. Architecture considerations
It is important to understand the practical considerations involved in high-level solution analysis and architecture of an IBM BPM solution. Considerations such as these:
2. Security considerations
BPM processes contain the very essence of a business, even including things such as competitive advantages and trade secrets. Despite this importance, and despite the fact that security breaches make embarrassing headlines, IBM repeatedly finds customers who consider their BPM projects secured simply because the BPM servers are hidden behind firewalls. So there are a number of specific items an architect must consider when planning how BPM will be integrated into an enterprise’s IT environment, including:
3. Solution design considerations
Implementation of business processes using a business process management software can be challenging and, at times, overwhelming. As a BPM architect, you should seek an understanding of the following items as you embark on the solution design phase, prior to implementing your business processes in IBM BPM:
4. Business-centric visibility
Business-centric visibility is one of the most important considerations for organizations looking to adopt business process management. Business-centric visibility, in the context of a business process, includes these factors:
In order to achieve any meaningful business-centric visibility, it is important to have access to both the business data and the process data. Business-centric visibility is achieved in IBM BPM using these capabilities:
5. Performance tuning and IT centric visibility
Performance tuning and IT-centric visibility are also key aspects of the IBM BPM design. In the context of these aspects, you need to keep in mind the following important tips:
Ali_Arsanjani 120000D8QB Tags:  bpm rest soa analytics microservices mobile cloud big_data api patterns 6,703 Views
Microservices reminds us of very fine grained, tiny, micro-scopic services. Perhaps services that grow on a Petri dish.
But actually, they refer to a deployment pattern for services developed in a Service-oriented Architecture along the constraints of a Functional Area that owns its own Data, and does not allow access to its Information willy nilly. You start with a domain break it down into a Functional Area and get down to the subsystems and components living there, deep in the bowels of the legacy systems, the packages inhabiting the digital ooze in the sewers of legacy code , interspersed with some fresh sprinklings of new additions of components implementing newer services required by the business.
Stratified along a Functional Area, immune to the curious and invasive glances of strange components and outside systems, the service rests peacefully in isolated, independent deployment. Choosing to deploy to brew this SOA flavor with a twist of independent deployment, along the lines of a functional area, owning its information, is detailed out in the implementation of finer grained services in SOMA (Service-oriented modeling and architecture) ,many companys' service-oriented method.
Don't mistake this as the only way to choose to constrain SOA by design along functional area boundaries and deploy along those lines. This is only one brew of SOA.
Some considerations are the need to communicate between deployed services that are other wise feeling a little isolated and engaged heads down in their servicing of requested for their functional area of the business. But business have cross cutting concerns and dimensions, often exemplified by BPM or business process management. Cutting across silos or functional areas, the process needs access to various data sources and systems of record, albeit some are prehistoric gauging by enterprise standards.
Sp cross cross cuts of SOA flavor are best served by BPM. Smaller granularity focused functional area services by Microservices. What about when mobile needs to access something? Well for that we have the API brew. RESTful (Representational State Transfer) APIs provide an HTTP/S level of simplicity of access (yes, both get and set) that allow access to underlying functionality via mobile devices and yes Software-as-a-services, or Cloud as is known in slang.
So choose your brew of SOA according to your mood, according to what you would, contingent on what you should
be doing with your processes, data or business functions. Choose wisely for each have pluses and minuses; opportunities and consequences, like any pattern. Or brew.
Ali_Arsanjani 120000D8QB 2,012 Views
What are API’s critical to enterprise business agility?
Today we see the importance of a confluence of mobile, cloud-based, service-oriented applications that were hitherto locked within the enterprise break free and start interacting with an ecosystem of partners opportunistically in order to create value on emerging opportunities especially those provided by mobile devices, social business, big data and analytics and cloud-based services. It is important to understand the evolution of APIs in order to better capitalize on opportunities presented by the economical dynamics of an ecosystem connecting and communicating through APIs.
The evolution of APIs
API’s or Application programming Interfaces were initially thought to be libraries of reusable code. They were focused very specifically on providing functionality that was written one often to support a complicated set of granular operations such as a graphics library, telecommunications library, security etc.
In the first decade of the 21st century, application programming interfaces morphed because of service-oriented architecture. The protocols used to implement Web services in a service-oriented architecture were numerous. The simple object access protocol or soap was one means of access. This sort of replaced RMI over IIOP in a distributed computing world.
Service-oriented architecture taught us that not only does interface have to be separated from implementation but also that implementation can be separated out into several layers. One layer is a more abstract specification of where the endpoint for the service implementation may reside. Often an enterprise service bus was used to virtualized the endpoint so that the optimal endpoint could be selected based upon configurations or input parameters or more pragmatic considerations pertaining to security, scalability or performance.
Implementation was separated into a realization decision, And a deployment set of options. The realization decision was primarily governed by the question: “how am I going to implement the service? Which component is going to be used to implement the service functionality?” The component could be anywhere from a.net component, and enterprise JavaBean or a legacy application interface or even a package application. The deployment option included not only the protocol by which the implementation would be realized but also the configuration options pertaining to the infrastructure or platform used to ultimately operationalize the solution.
Representational State transfer or REST was a protocol created to support a very lightweight mechanism that would replace the more complicated SOAP protocol and therefore could use HTTP or HTTPS. Therefore the verbs that could be used where the familiar get and put actions familiar to web programmers.
Restful APIs were APIs that extended enterprise capabilities hitherto reserved for webpages into the ecosystem. This ecosystem of partners who were now able to interact using restful APIs, created the opportunity for an API economy.
The characteristics of API-based services
At the core of the enterprise the concept of service-oriented architecture suggested the creation of the service model. Within it is a service portfolio that is created, governed and managed as an enterprise asset. This asset , the Service Portfolio represents the implementation of business capabilities that the enterprise would like to expose experience brings of partnership and security either within the enterprise but also extending towards the edge of the enterprise and more selectively, beyond it, to business partners.
In summary characteristics of the API-based services provide the enterprise the ability to :
· Externalize Business Capabilities for ecosystem consumption
· Capitalization and monetization of Enterprise Assets outside the organization
· Extend value of Enterprise capabilities beyond the enterprise
· Secure access to internal enterprise resources and capabilities
· Enable channel-agnostic connectivity with business partners and consumers
Internally, the APIs will be part of the enterprise portfolio that can provide elimination of redundancy through reuse of capabilities as well as providing governance through consistency of created assets.
Externally, exposure of internal capabilities to the partner ecosystem will allow value-added services to enhance existing services provided, bringing in a new set of clients.
Pitfalls and Best-practices
Lack of alignment of an API to a clearly defined set of business goals can be detrimental to the adoption of the technology and capabilities behind the API.
APIs should not focus on time sensitive technologies or merely on fulfilling user interface capabilities that access backend systems. An example of this is to limit the capabilities of an API to CR UD-based operations such as create, read update and delete. This severely limits the capabilities of expansion of the API into higher levels of maturity of an application lifecycle.
Exposure of architectural decisions that stem from interorganizational considerations that in turn have sprung up over time as a result of the attempt to maintain the big ball of mod that constitutes the legacy systems that have been developed over generations within the organization. One best practice is to mask the convoluted interdependencies on technologies, architectural decisions that were made due to expediency and legacy platforms and the scars of partially successful mergers. The API should not expose such backend architectural decisions and implementation considerations just as any good service in a service-oriented architecture would mask these backend assumptions that have no rationale beyond the walls of the enterprise.
Secure a path of growth so your applications can start evolving from merely accessing backend legacy systems to performing transactions, initiating business processes and engaging in decision management.
Designing for the glass. Designing mobile applications is a current fad which is driven by the strong adoption of mobile devices. This can be a severely limiting factor is the application is designed from the glass in words in other words consideration of the backend business processes must be taken into consideration along with considerations of usability at the screen level for the mobile device of choice.
The ability to orchestrate and compose APIs in order to provide a mediated or orchestrated experience of the business process is a critical success factor with the maturation and increased adoption of APIs within your ecosystem.
Ali_Arsanjani 120000D8QB 2,050 Views
Context: Industry adoption of development, life-cycle and architectural styles such as SOA tend to be imported into organizations and then move from an evangelist, grass roots kind of effort into a next phase of organizational maturity which tends to impose rigid constraints on the life-cycle, standards, governance and development of the architectural style. SOA is an example.
Problem(s): High level , holistic design and architectural work is needed to provide contractual visibility to clients. Architectural styles (OO, CBD, SOA etc) are co-mingled with the rigor of the top-down governance that is seen to be required, or they are sacrificed for rapid, quick and dirty, "let's not look around the bend until we get to it" approach.
Forces: The developers tend to push back to the imposition of increasing accountability and governance in general, including standardized development practices. The organization , in an effort to be accountable to the business, requires increased tracking, metrics and accountability. Developers will continue to resist increased accountability due to the perceived imposition on their time that is deemed less valuable than the actual development of the product. Against this, accountability and project visibility, allowing projections in order to fulfill "promises" (e.g., contracts, SLAs, etc) with clients (whether internal i.em business as a client, or external, as in other organizations that services/products are provided to).
(Re)solution: Combine a light top down governance and standardized software development process/method that looks holistically but does not require detail to the n-th degree prior to engaging in a project or program, coupled with agile iterative sprints.
Consequences: High level visibility and accountability are attained with some compromise with the "let's just do it" approach. Development is not impeded with heavyweight constraints. Promises to the business or external clients have a reduced risk of non-completion associated with them.
Ali_Arsanjani 120000D8QB 2,362 Views
SOA , as an architectural style and a discipline of software engineering brings business and IT to the same table by providing a portfolio of business intended services/capabilities as the common terminology that he business requires and IT will design, implement and maintain. SOA is focused on establishing a center of operations for design around a portfolio of services, which equate to a set of business capabilities, that the business looks to deliver. Integration has become a strong focus for SOA as well. There are various types of services, integration and business services to name just two categories.
In the traditional Application Programming Interfaces or APIs were merely a library of interfaces that components can access and communicate with each other. In an evolved definition that is becoming more and more prevalent, the notion of a RESTful API that is exposed outside an organization, that may expose a set of services in the backend is being promoted more widely, especially with the proliferation of interest in HTTP-based simple interfaces that can mobile enable an enterprise's subset of back end functionality that it chooses to expose.
Cloud services are services that are exposed externally, and can be consumed by all, often implicitly assumed to be outside an organization's boundaries just as an API or more precisely, a RESTful API. Cloud services may span the gamut of Infrastructure as a service (IaaS) all the way to Software as a service (SaaS) which may expose very large grain applications or smaller levels of granularity akin to the RESTful APIs we discussed above.
More detail on this later.
Ali_Arsanjani 120000D8QB 2,230 Views
Business process Management or BPM is about streamlining workflow and introducing efficiency into a process of a goal oriented business process.
A Smarter Process is a process that is augmented by decisions and business rules that are part of the activities within the process flow. An event will have triggered the process flow due to the monitoring of a situation or of bubbling up the analytics we are collecting to generate an event. Backend systems need to be accessed, invoked for a transaction or connected to for some outlandish reason such as updating a system of record. These summation of integrations, process flow, rules, events and analytics are collectively called a Smarter Process.
Architecture, software architecture specifically, used to be only defined in terms of components, connectors and constraints per Shaw and Garlan.
However, it is increasingly evident to those of us who implement architectures on a daily basis for clients and internally within our own companies
that architecture requires the addition of composition, context and containers to make it operational.
Components -- what are the structural building blocks of a solution, or a style of architecture, of the elements that can be combined to produce a larger structure
§Composition -- How to compose or what are the valid compositions of components
§Connectors -- The components need to connect with one another, whether statically (as in an Entity-relationship kind of relation) or dynamically, as in a composition in an SOA, where you may have orchestration or choreography.
§Constraints -- The constraints on the connectors and/or components that provide rules of engagement of what is permissible and what is not.
§Context -- the context of invocation of a component is critical to the designation of how that component will behave
§Containers -- the components must live in some runtime container that will provide uniform life-cycle management capabilities for them
The short answer is:
Cloud is an initial application of SOA to infrastructure
--its setup, configuration, monitoring , management , with the essential elements of metering and billing added to satisfy the goal of a charge model of e.g., pay-as-you-go
-- the specific requirements for XaaS: resource pooling and virtualization , elasticity and multi-tenancy, dynamic configuration and provisioning
Cloud paradigm leverages SOA to deliver a charge-based model for non-business specific services;
Several people have written articles on this relationship:
Towards a more precise definition of Business Agility: The Building Blocks of Business Agility (B3A)
Ali_Arsanjani 120000D8QB Tags:  agility building_blocks_of_busine... business_value business b3a 5,092 Views
Business Agility is an often discussed as a key desirable attribute. One of the ways of achieving Business Agility is through the portfolio management of a set of business capabilities and services , rather than application portfolio management which tends to intertwine the services and higher order business capabilities into an application -- not a robust and flexible way of enabling business agility and optimization.
The main building blocks that provide a platform for business agility are outlined below. I have elaborated the last one more than the others in this post, and will elaborate further in future posts.
Laws governing eco-system as a whole (e.g., to disallow financial market meltdown in the wake of blind rule based automated short selling in e.g., financial markets)
The ecosystem includes policies, rules and laws governing Business providers and Business Consumers, and Business Brokers.
environment, market, legislation and ecosystem within which the business is
operating and evolving, forcing to change and vary based on forces within the
Business Context. The Business Context is a
There are many changes that are constantly occurring within the business context. Only some are business significant or should be “bubbled up” or surfaced at an executive and business level, especially when Business Sensors detect a certain threshold above which apparently ordinary events become important enough and achieve a tipping point beyond which the business sensors should indicate that a potentially impactful variation has occurred.
Business Entities model and reflect the key business domain elements of the Business Eco-system that under Business State changes.
Many EA (Enterprise Architecture) practitioners are …exploring the links between desired business outcomes and architectural decisions.
A tipping point is reached when IT begins to understand that business executives are not primarily looking for products and services but rather looking for business outcomes including increased output, higher quality, lower costs, increased revenue and increased market share.
To effect the transition to become more business-driven and engaged with business leaders, we should focus EA efforts on business outcomes.
Instrumentation and mechanisms to monitor, track, help modulate and govern business change as key performance indicators track the events and occurrences within the Business ecosystem Context.
Typically, the board and top executives of a company state the vision and strategy for the company. It is critical for implementation of business performance solution in the company that the senior business leadership translates and decomposes the high level vision and strategy statements of the executive leadership to operational and actionable goals associating KPIs to the actionable sub-goals on the way.
To achieve such end-to-end monitoring, metrics and sensors must be injected in both planning processes and delivery processes .
Essential to this process is the Business Sensors that detect Business significant Events and pass them on to the Business Policies and Rules that would respond to them.
From a planning perspective, enterprise governance needs to ensure that the right changes are initiated for the right reasons at the right time. The underlying premise driving towards business agility is that such agility delivers superior business value. But what if haste to achieve agility results in low quality? Or what if speed of change is unsustainable from a business operational perspective, thereby leading to deteriorating efficiency? These are just two examples of the fundamental challenge that doing the wrong things in the wrong way very fast simply means ruining your own business very fast. There are two fundamental premises for agile change to be valuable over time:
For agile change to be sustainable the enterprise needs to carefully plan and maintain an appropriate balance between effectiveness and efficiency. Change in the large is based on continuous business re-engineering towards strategic objectives (effectiveness). Yet while on that strategic journey an enterprise needs to apply change in the small to continuously adjust and optimize the current state and ultimately maintain business integrity and performance (efficiency).
There are two levels of monitoring – there is the monitoring of the acting on plans and there is the monitoring of operations.
From a delivery perspective, the set of solutions designed and implemented to achieve the business goals should be designed for monitoring. Monitoring of business results is a key to business performance. Monitoring of business results enables sensing of business significant events. Ability to act upon to respond to the business significant events resulting from the business activity monitors is essential to improve the business performance in order to meet the business goals.
Therefore, business goals, solutions adequately instrumented for monitoring and alerts, monitoring of business results, sensing of business significant events, responding to these business events to take corrective or predictive actions are key for a business to be in a path of success and growth that can be adjusted and tuned based on actual results of the business.
Ali_Arsanjani 120000D8QB 4,142 Views
Based on our project experiences over the past years, we feel there are 6 key ingredients of success with SOA:
1. SOA 101 -- basic understanding of key SOA concepts e.g.,
2. SOA Maturity Assessment and Roadmap (using OSIMM ; Open Group Service Integration Maturity Model) that implements an
3. SOA Strategy, outlining the path and projects and initiatives to embark upon for sustained success with SOA enablement for business agility
4. SOA Method (SOMA) that provides the prescriptive guidance as to what to do within the software and business architecture life-cycles
and yet work in tandem with
5. SOA Governance to institutionalize the key processes needed to sustain success
6. Consideration of an SOA Reference Architecture now a industry standard,
Ali_Arsanjani 120000D8QB 7,430 Views
I've participated in a lot of SOA projects around the world and very often I'm asked to answer questions about aspects of SOA. These are often recurrent questions that most of my clients and her colleagues are asking. So a colleague of mine, Kerrie Holley and I decided to consolidate these questions and answers into a book so that more people can access the questions, see if they are relevant to their particular circumstances and hopefully benefit from the responses.
The book is entitled 100 SOA Questions: Asked and Answered .
I have started a separate blog at soaquestions.blogspot.com to discuss further questions and comments specifically about the book .