Sample chapter from: Web Services Implementation Guide
Authors:
Brian Travis and Mae Ozkan
About the book:
Getting value from stuff you don't own is a compelling idea. Just think, someone else has created something that has value, and has made it available to you. You can now leverage its value in ways that makes all parties richer. That's the promise of web services.
You need to know about web services. The advent of standards-based web services marks a new era of system development. First mainframe, then client-server, the Web, now web services makes it possible to leverage every legacy system of the past and do so much more with your information assets.
This book will show you what a web service is, and how you can align your systems to take advantage of this new concept in system integration. This book covers the concepts and technologies of web services.
Web Services Implementation Guide is a multiple-volume set. Volume 1 shows you how to align your thinking and systems to get the most of web services. Volume 2 is a technical treatise for developers. Volume 3 discusses real-life implementations of web services solutions.
The integration models we have today are either loosely coupled or
tightly coupled. They are message-based or method-based. They are synchronous
or asynchronous. Putting these things together in an intelligent way is
essential if you are going to archive the promise of web services.
In this chapter, we will discuss all of these models, and show which ones are
used by web services and why.
Invoking Services
Objects can be invoked using methods or messages.
Method-based
A method-based invocation is one where the name of some method is known, and the
method is invoked by its name, passing whatever parameters are needed by the
method. Most object-oriented systems are method-based, including COM and
CORBA. Systems that want to access these objects need to know what the methods
are, and how they are exposed.
One way objects expose their methods is through an interface definition language
(IDL). By interrogating an object's IDL, a system will know its method names,
parameter names, and datatypes of the parameters, and the return values.
Message-based
A message-based system is one that relies on messages to invoke an object. These messages
are self-describing, and can be routed to the appropriate object via some intelligent
message-transport system that can read the message and determine its end point.
In a message-based system, a client that wants to access an object must know
only what data that object needs. As a result, implementation details of the object are
not necessarily known to the client. The data in the message interface is the only common
knowledge. An example of a message-based system is a document queuing system.
Accessing Services
Objects can be accessed synchronously or asynchronously.
Synchronous
A synchronous system requires that a response come back right away when a request
is made. Synchronous calls are usually made in high-performance systems that cannot
wait for an answer.
An example of a synchronous request is a method call of a CORBA object. When
a method is invoked, the calling program usually waits for some response. The program
will lock up until the response comes back. For synchronous calls, it is good programming
practice to set some kind of timeout, so that if the call does not come back
in a certain time, a supervisor program can unlock the calling routine and provide a
graceful exit.
Organizations need to communicate synchronously in order to find answers to
important questions that require decisions in real time. These real-time synchronous
communications usually work using a request/response model. For example, when a
travel agent is looking for a plane seat, she needs an answer right away. When you are at the supermarket checking out using your credit card, the credit card machine calls
the bank and gets approval in real time.
Asynchronous
An asynchronous system is one that will issue some kind of a request and wait patiently
for a response. A system that uses asynchronous calls will usually set some kind
of event handler to listen for a response, then it will make the request and continue on.
When the listener receives the event indicating that the response has been received, it
will inform the application that the data is ready.
Asynchronous systems are more efficient, because they do not spend time waiting
for a response. This multi-threaded approach is popular because of its versatility,
but there are some problems, also.
Programming for asynchronous systems is more difficult than for synchronous
systems because of the management of the listeners. Also, it is possible to have more
than one asynchronous thread happening simultaneously. If this happens, it is possible
that responses will come back out of order and need to be sorted by the application.
Synchronous vs. Asynchronous
To get an idea of asynchronous versus synchronous communication, think of the difference
between ordering a custom part for your monster truck through the mail versus
talking to someone on the phone.
To order by mail, you need to specify every detail of the part, including methods
for payment and shipment. When the receiving organization gets your
letter, they will process it and return a response to you.
To order by phone, you might say "I want to buy a part". The operator says,
"What kind of part?" You say, "A five ton gas filled shock absorber." Clerk:
"We have most colors in stock. What color would you like?" You: "It needs to
be blood red." Clerk: "That's funny, but we had a run on those parts just yesterday,
and they are on back order. Would you like a different color?" You:
"Yes, how about the screaming yellow one. Is that in stock?", ...and so on.
The first is an example of an asynchronous architecture using a self-describing
message. It was necessary that the message sent to the parts vendor contained all of
the information required to make the transaction work.
The second example is a synchronous communication system where a dialog is
made in real time. When you are on the phone with someone from the call center trying
to get some help, the questions you ask get answered in real time. This is more efficient
than getting support by mail, but it requires more resources from the call center
operator.
Synchronous communication is better at dealing with changes in real time, but
it is expensive. In the telephone example, the organization needs to keep telephone
operators ready to take calls, whether they come in or not. During peak times, callers
might receive a busy signal or need to listen to elevator music until a clerk is ready.
On the other hand, asynchronous messaging can be done in leisure time, creating
a system where peak loads do not have much of an effect on processing time.
The trick with asynchronous messaging is seen here. If there is a problem with
the order, the asynchronous message should have information about alternatives if the
desired item is not in stock. This is where most problems arise in asynchronous
messaging systems, and it is an area where you should take the time to anticipate all
conceivable problems before they arise.
A synchronous communications architecture usually requires that two systems
be somewhat compatible in order to communicate. In system terms, a synchronous
call might be a method invocation of a CORBA object, which returns useful data or a
result code. This tight integration is fine as long as all applications can speak the same
interface protocols. However, if there are two systems that do not speak the same interface
protocols, there is a requirement for bridges or other code that provides interpretation
between the incompatible systems. These bridges are usually pretty
unreliable and are a common point of failure.
Coupling
A system can be tightly coupled or loosely coupled.
Tightly Coupled
A tightly coupled system is one where the relationship between two or more providers
requires them to communicate with each other directly. A tightly coupled system
requires all parties to understand many implementation details about each other.
Tightly coupled client components can create dependencies on a specific hardware
platform, a specific operating system, a specific interface model, a specific programming
language, and a specific match between versions of client and server
components.
An example of a tightly coupled system is Microsoft's COM Transaction Integrator
(COMTI) component. COMTI can integrate with legacy mainframe applications by
pretending to be a mainframe terminal. The COMTI object can be used to fill in a
screen full of data, send it to a mainframe application, then scrape the response screen
coming from the mainframe. A COMTI application is designed for a particular mainframe
program user interface. Any changes in the program or the user interface requires
changes to the COMTI application.
In the tightly coupled model, then, the client needs to have intimate knowledge
of the implementation details of the server program.
Tightly coupled systems, therefore, can make some assumptions about trust.
That is, they can assume that the intimate knowledge the client and server have of
each other requires and demands that they trust each other.
Loosely Coupled
Two systems are considered loosely coupled if the only mandate imposed on both systems
is to understand the operations published by a service and the parameters required
to invoke that operation.
An example of a loosely coupled system is the topic-based publish-and-subscribe
model used by knowledge management systems.
This loosely coupled architecture must take into account the fact that the server
might not trust the client. This complicates the system, but is probably a good design
goal, anyway.
Tightly Coupled vs. Loosely Coupled
A tightly coupled system is more efficient than a loosely coupled system because the
calls invoked can be made directly rather than indirectly or through some intermediary.
For this reason, tightly coupled systems are used when execution speed is an important
factor.
However, a tightly coupled system is difficult to maintain, because changes in
either side usually require the other side to adapt immediately. For example, if the order
of the parameters is changed in an object call, or the datatypes are modified, the
calling system needs to be updated to reflect those changes.
On the other hand, implementation details in a loosely coupled system can be
changed, as long as those changes do not affect the functionality of the subsystems
that are called.
Consider the real life example described above in the section on asynchronous
messaging of the telephone call to the parts operator. This scenario requires that both
sides first of all are using the same type of telephone equipment. This is not really a
problem these days, because just about any telephone in the world can make a physical
call to any other. Once connected, however, both parties must speak the same human
language. If one party is speaking French and the other understands only German, a
compatibility problem arises.We need a bridge in the form of a human interpreter to
make the connection. This could result in inaccurate translations and
miscommunication between the parties.
Using asynchronous message-based architectures eliminates the need for systems
to be tightly linked to each other. Loosely coupled architectures have the advantage
that messages are passed, instead of method invocations. These messages provide
a degree of independence between the sending system and the receiving system. Systems
that use messaging are called loosely coupled architectures.
The Java Remote Method Invocation (RMI) architecture is natively a tightly
coupled procedure. Likewise, when a Visual Basic client calls COM object, it expects
the COM object to adhere to a tightly defined specification.
Routes to Access: The Integration Matrix
In real systems, you will have a combination of message-based/method-based, synchronous/
asynchronous, loosely coupled/tightly coupled interactions. In fact, these
concepts are usually tied together to create reliable or efficient systems.
For example, most method-based systems are tightly coupled and probably synchronous.
A message-based system is usually, by definition, a loosely coupled system.
Asynchronous calls can be made with messages or methods, depending on what you
need to do.
An asynchronous messaging architecture allows systems to communicate by
sending self describing messages back and forth. These messages contain enough information
to make some processing happen. The transmitting system is obliged to
create a message that contains enough information to convey its desire, and a receiving
system is obliged to read the message and do something useful.
The receiving system is also able to reject the message if there is a problem. It is
good manners, however, to tell the transmitting system what the problem is.
What if we want a synchronous loosely coupled system? We have technologies
to accomplish loosely coupled asynchronous systems. However, some scenarios require
synchronous transactions. Business is usually asynchronous because of reliability
and scalability reasons. However, some business functions need to be
synchronous, such as some queries for inventory, schedule changes, credit card transaction
approvals or disapprovals, etc.
The table in Table 4, The Integration Matrix shows a matrix of coupling versus
invocation.
Tightly Coupled, Method-based Systems
A method call requires the knowledge of the object names, support of object types,
and a request made in the object's language. As a result, the method calls, such as Remote
Method Invocation (RMI) and Remote Procedure Call (RPC), are tightly coupled,
method-based integration methods.
COMTI, COM/DCOM, CORBA are also examples of tightly coupled,
method-based systems. The integration is designed and architected for a certain interface,
form, and language. We cannot take already existing tightly coupled interfaces
and use them for other pieces of applications. The client and the server both need to
speak the same language and usually need to be members of the same technology.
Tightly Coupled, Message-based Systems
A tightly coupled message-based system is MAPI. The Mail API is a message-based architecture
because members communicate with other members by passing well-defined
messages to each other. However, the messages they send are tightly coupled to
the API, such that they are bound to that application and cannot be easily used by any
other.
However, in most cases a tightly coupled, message-based integration model is
usually an indication of bad architecture. Such a system has the disadvantage of being
message-based (complexity in creating and parsing messages), but does not have the
advantages of tight coupling (fast, synchronous access).
Loosely Coupled, Method-based Systems
SOAP supports synchronous calls by definition. It is natively designed for remote procedure
calls outside of the organization. Since method calls instantiate the objects outside
the organization, it is not built on a trusted architecture. SOAP provides a
structured message definition, which enables loosely coupled architecture even for
method invocations which are synchronous.
SOAP has two personalities, which are reflected here. SOAP is inherently a message-
based architecture. However, when SOAP is invoking methods on remove procedures,
it feels more like a method-based architecture. In that case, we will put SOAP for
RPC in this category. See The Two Personalities of SOAP on page 150 for more on the
two personalities of SOAP.
Loosely Coupled, Message-based Systems
SOAP fits here, too. SOAP can be used to send documents that describe themselves.
These self-describing messages can be processed by any system that understands
them. One category of such a system is a queuing mechanism.
Message queues such as Microsoft's MSMQ and IBM‘s MQ Series offer asynchronous
integration in a reliable manner, meaning they ensure one-time guaranteed
delivery with error-handling and error-recovery mechanisms.
In addition to message queue applications, SOAP is message-based and loosely
coupled by its definition. Consider two programmers, one that is writing client applications
in C on AIX, and the other who writes COBOL programs on a mainframe computer
server. They use IBM MQ Series queuing system to communicate.
These applications exchange business documents or record layouts. Since the
client is unaware of the server's applications, languages, and error handling mechanisms
it is loosely coupled system.
If a mainframe application and an AIX application are integrated using MQ Series,
than the programmers on the mainframe side only need to know how to program
in mainframe applications like COBOL and how to use MQ Series to exchange
messages. Only the structure of these messages and schemas need to be known to
both sides. Likewise, on the AIX side, programmers need to know only how to program
their native applications and how to use MQ Series to exchange messages.
Therefore, programmers on both sides know a standard application and their native
platform and language. The integration architecture is a loosely coupled and asynchronous.
Consider a scenario where there is a COM object that we need to access. The first
thought is to write a Visual Basic client, since it speaks the same language as the COM
object and has the same type definitions. The client and server live in the same world.
It is the same scenario for the Java programs calling CORBA objects.
However, imagine a loosely coupled system where a program, whether it is a Visual
Basic client or a Java application, sends a message to the server object. In response
to that message, the server object's behavior will be exactly the same regardless of who
the requestor is and what native language it is written in.
In our case the server, our COM object, is unaware whether a Java client or a Visual
Basic client is the requestor calling to invoke a method. There is no difference between
a Visual Basic and a Java program calling the COM object, because the data that
COM object needs for a method invocation has been sent using a standard language
and format that COM object understands.
As a result, this remote procedure call, which was natively a tightly coupled system,
becomes a message-based, loosely coupled system. This scenario is described in
Chapter 7, Simple Object Access Protocol (SOAP) on page 139.
The client requesting an answer is unaware of the server's implementation, language,
alphabet, and native format. When the answer comes back to the Java or Visual
Basic client, the answer is not in a COM definition. It is in a standard message that
will work for Java, COM, COBOL, and whatever else can create and use these messages.
Mutual Ignorance
As a result, there is mutual ignorance between the client and the server. Loosely coupled
systems ease integration problems. Yesterday, a COM object was accessed by a Visual
Basic application. Today, a Java application can call the object. Tomorrow, a Java
server application can expose its data to a Visual Basic application.
As a result, any object can feed its data to any other object in a message-based,
loosely coupled way synchronously and asynchronously, whatever the business and
applications require, as long as the applications support the same standard message-
based interface, speaking the standard language.
This standard language is SOAP. SOAP is embraced by the industry and solution
providers that support Java, COBOL, COM, and many others. As a result, these stated
heterogeneous technologies now support SOAP as a protocol in addition to their native
supports.
This utopia is the basis of web services.
Loosely Coupled Systems and XML
In loosely coupled systems, the communication is usually accomplished by messages.
Let's think about a scenario where a brand company is outsourcing some of its
production to a manufacturing company that has spare capacity. The brand owner
company's production system is updated in real time when the formula of their product
changes, or when their inventories change. These changes need to be transmitted
to the manufacturer.
Production needs to be updated synchronously. However, because the manufacturer
is outside of the brand company's firewalls, how will the production get updated
in real time as requirements change?
If the applications are outside of the company, tightly coupled systems do not
work. No company will give the details of its implementation so that the client will access
the server that resides behind the company firewalls with all of its valuable information.
So how do we create a system that can get information, in real time, between
enterprises?
Answer: SOAP
As a result, we need the advantages of message-based systems in addition to the advantages
of synchronous systems. SOAP defines a standard structured message that is
exchanged between partners. The good news is that SOAP enables synchronous as well
as asynchronous communication opportunities in loosely coupled way, without having
all parties concerned with each others' implementation details.
If the tools and knowledge for building SOAP messages and structures was expensive,
such as they are in the EDI world, SOAP would be adopted only by the largest
players. However, XML syntax, and the tools for building vocabularies using schemas,
are cheap and easy to use. Translation from one XML vocabulary to another is also
cheap using XSLT.
Therefore, XML syntax, along with other standards that are built on top of it, are
important in loosely coupled environments. The standards and tools provide easy
plumbing and integration in a loosely coupled manner. That is why XML standardization
and loosely coupled applications go hand-in-hand. The structured and standardized
message formats enable the communication and integration between different
applications and objects inside and across the firewalls.
SOAP in EAI and Middleware Applications
As the SOAP standard became understood, providers of Enterprise Application Integration
(EAI) solutions announced that they planned to support SOAP in their application-
to-application integration solutions.
Why? Because SOAP is becoming a standard protocol just as FTP, queuing, SMTP,
and many others that we now take for granted.
The reason why we call SOAP a protocol is because it is just a foundation for
other solutions. These solutions are called extensions, and are built on top of SOAP.
These SOAP extensions will solve communication and integration problems in a standard
way. SOAP extensions are being planned to solve problems such as reliable
messaging, security, transaction handling, error handling, message routing, and orchestration.
These extensions are covered in Chapter 11, Turning a Protocol Into a
Framework on page 251.
Natively supporting XML and SOAP standards will enable middleware technologies
to take advantage of many new opportunities. An e-commerce infrastructure is
being built by consortiums with substantial industry support from not only application
providers but also manufacturers, distributors, and many vertical enterprises.
SOAP is not only for cross-enterprise application and document integration.
SOAP standards can be used internally as well. An organization will describe an internal
services that will be delivered to other internal service partners, such as departments
or other application groups. In SOAP in Application-to-Application Integration
on page 204, we discuss how you can use SOAP to integrate internal applications.
Business Decision Making in Real Time
Today's early SOAP implementations have been mainly document exchange, such as
BizTalk or ebXML. However, SOAP was originally a solution to solve application access
problems, which are structured, loosely coupled, synchronous procedure calls.
When an organization moves to synchronous, real-time transactions, it has new opportunities
in messaging and partner relationships.
As a result, an organization can thoroughly integrate and collaborate with customers,
trading partners, and applications that do not necessarily reside in their organization.
For example, for credit card processing, an organization needs the data which is
stored in a bank's system. When a purchase happens, the organization needs to reach
this valuable information and give a decision in real time.
If an organization can reach a bank's credit information, transactions happen
easier, faster, and cheaper, and they get their money sooner. Money today is always
better than money tomorrow.
Another example is found it search engines. Google now offers a SOAP API for
searching the Internet. An organization can use this API and include search capabilities
in their applications. An organization that does not have and will never want to
build a search engine themselves now have a service that they can use simply using the
SOAP API that Google offers.
Reaching the utopia of "getting value from data that we do not own" is a compelling
concept. SOAP provides new business opportunities in this respect.
As the software industry adopts a standard protocol and syntax, such as SOAP
and XML, it becomes easier to integrate everything. The integration problems that are
caused heterogeneous applications are gone.
As SOAP becomes a standard protocol that applications support, integration will
not be a problem. As a result, the data will not be held hostage in a proprietary format
any more. Data will flow through the applications and services, and applications will
have the ability to use any data no matter what format data natively resides in.
Integration problems will be solved not only in internal application integration
and behind the firewall but also across the firewalls with partners.
Integration Problem Statement
In Chapter 3, WhereWe've Been, WhereWe're Headed on page 39 we told the story of
Jean and David. They are clerks that work in two organizations that are doing business.
David is a clerk in the supplier organization. He is the one who carries data from
one party to another in the organization, whether that party be a machine or a person.
David is the one who is responsible for the business flow and he is the one who integrates
computer applications, people, and other resources inside and outside the organization.
Human Integration
In our scenario, David talked to his counterpart, Jean, on the telephone. During this
conversation, data flowed through speech. The two people understand each other because
they are speaking the same language, and using the same business terminologies
and nomenclature.
When David talks to his director, he knows what the director is talking about by
understanding her speech. When they are communicating with e-mail, he understands
what is written in e-mail. Again, because they speak the same language, use the
same terminologies, and have the same human speech and recognition capabilities.
When David enters data into a computer that has a user interface, he understands
it because it is designed with humans and their capabilities in mind. David
knows nothing about the internal processes of the computer application. However, he
faces an interface that he understands and he is trained in using it because it is familiar
with what he has seen all of his life.
Until now, we have not had a problem integrating humans with humans or applications
with humans. We have a standard in communication, which is language.
We know about humans and we design our systems for them. As a result, data flows
around, and humans and applications integrate just fine. Human capabilities are keys to
system integration.
However, when organizations deploy different applications in-house and they
want fast and automated business processes that do not involve human integration,
how will they do it?
Application-to-Application Integration
Organizations have heterogeneous systems. They have applications and data that belong
to different computational generation eras, written using different programming
languages, which employ different vocabularies and syntax rules, data types, and
many other incompatibilities.
Because these systems cannot integrate with each other, they need humans to
free the data.Without human interaction the data will be held hostage. Organizations
have very important and valuable data. It takes significant resources to maintain that
data. How can they leverage their data in an automated manner? How can they get rid
of the need for human interactions that carry data from one system to another? We
have discussed "JESSIE" and what troubles she causes when she is the central integration
solution in Chapter 2, Integration Today on page 29.
We need to integrate our businesses, our applications, and our business processes
in order to pass valuable data and as a result do commerce with one another.
Only then will we be able to achieve the promise of web services.
The Integration Problem
The problem is to integrate applications or, by extension, trading partners.We define
integration as applications or trading partners passing data from one to another and
consuming it without human interaction.
The application integration problem has been with us since the day organizations
started using two computers. Traditionally, getting systems to talk to each other
required building some kind of connector that could translate between two different
systems.
An organization would need to build some kind of integration code, and expose
a connection point in order to pass and use each other's data.
This is called "point-to-point" integration and is illustrated in Figure 101,
Point-to-Point Integration.
In a point-to-point architecture, integrating systems requires custom code for
each interface.
When any changes happen in either application A or B, The interface programs
must be updated and changed. Therefore, the integration solution using
point-to-point architecture is hard to manage in the long term.
Application integration becomes increasingly difficult as new applications are
added to the environment. For every new system added, it is necessary to create integration
code exposed by connection point interfaces for each existing system.
For example, in a point-to-point integration architecture, consider five different
applications. Each application speaks and understands its proprietary language, has
proprietary data types, and uses different protocols to communicate. If we integrate
Application A and Application B, we need to add a connection point and codes on
both sides so they can communicate. The code on application A will be speaking the
language of A and B. When application A wants data from B, this code translates the
request to B's format and data type, with B's object or method names, and transfers it
to B in a protocol that B supports.
When application B needs data from application A, then the code in B will be
converting B's request to A's language, with A's specific implementation details.
As a result, for a two-way communication between application A and B, programmers
need to write applications that enable integration that sits on both sides.
This integration code is usually called an "adapter". Just like an adapter for an electrical
plug, they convert the existing standard into another standard that a specific application
can consume.
The number of adapters needed grows exponentially. There is a formula that describes
this, it is N2-N. So if there are five applications, and all of them require interactions
with each other, the calculation is 52 - 5 = 20.
This is commonly called the "N-squared problem", and is common where there
is a requirement for a many-to-many interaction. An example with five applications is
shown in Figure 102, The N-squared Problem.
Problems With Point-to-Point Integration
Applications can be integrated now. However, any changes in applications
break the integrity of the connections. They need to be changed and updated.
Changes in one application interface means changing N adapter applications.
Business Partner Integration
This same scenario is true when we are trying to integrate N external business partners,
such as customers or suppliers, in a point-to-point manner. Trading partners all
have different frameworks and applications that handle e-commerce in the best way
for each one. The problem becomes integrating not just applications but entire organizations
that sit outside of your system. It is an even a harder problem, because their
business practices require us to consider another set of integration points.
In order to integrate N trading partners, programmers need to write N2 - N different
applications that support N frameworks at each interface. These applications
will be updated every time some trading partner changes the framework that they use.
If we add another trading partner into the picture, we need to integrate all existing
trading partners with the new one.
Hub Model With Adapters
Now that we've seen the "N-squared" problem, what is the answer? What kind of architecture
can solve this exponential integration problem?
One of the models organizations use in addition to point-to-point integration is
called the "hub" model. An organization will use a system that sits in the middle of the
systems that need to be integrated to decrease the number of interface points. This
central system is called a "hub", because it sits in the center, and looks like a wheel with
spokes when the connections are made. See Figure 103, The Hub Model.
When the number of interface points decreases, the integration problem becomes
a linear problem. The adapters that organizations need to add to their applications
for interfacing drops to N. And when a new application is added to the system,
then only one adapter is needed for its integration to all the existing applications.
Now, instead of requestor application communicating with the respondent, the
requestor communicates with hub application, which then communicates with the respondent
application.
When application A wants data from application B, A sends the request to hub
application using the hub application's language and protocol. The hub translates the
request and sends it to application B, which takes the request and converts to its own
format.
The adapter on A understands only its language and the hub application's language.
In the same manner, the adapter on B understands only its language and the
hub's language.
The Broker
Using a hub is the system equivalent of humans using a broker or some other kind of
middleman. Let's say we are trying to buy a house.We could drive around town looking
for signs posted on people's front lawns indicating their desire to sell, and then
contact each owner and ask about prices and terms. When we find a house we like, we
could do negotiating directly with the seller and take care of everything ourselves.
What a hassle. And what a danger. We do not know all of the laws having to do with
the transfer of real property. We do not know everything about title searches and inspection
reports and mortgages and deeds and and and...
Instead, we use a broker to do much of the work for us. Let's call her Meg.
Meg first meets with us to understand what we want from the house. She wants
to know the price range we are interested in, the locations, and other factors like the
size, number of bedrooms and bathrooms, and whether it has room for our Elvis-
on-velvet collection.
All of the brokers have gotten together and created a network of information
where they share all of the properties that they have. Meg search searches this set of
listings, looking for properties that match our specifications.
Then she runs us around in her car so we can pick out the perfect house. But
that's just the start. Now, we negotiate with the seller, not directly, but through Meg.
Meg does not want the buyer and seller ever talking to each other because problems
might arise. Apparently buying a house is quite an emotional time, and sometimes
people say the wrong things. But all is well, because Meg is taking care of the interface.
She is the hub.
Meg deals with the inspector, the various government agencies, the banks, the
insurance people, and everything else. If we have any question whatsoever, we can just
call Meg, and she will get the answer.
Adding a middleman into any business process has its advantages and disadvantages.
The advantages of middlemen are pretty nice:
She takes care of details in communicating with the house owners.
She is a one-stop information center for us.
She is an expert in house nomenclature and the local market.
She knows details of the transaction, or has experts available that can cover
every part of it.
However, Meg has some disadvantages, as well:
She is expensive.
She can lose details when communicating with the seller or the other parties.
She has her own agendas, which might conflict with ours. For example, our
agenda is to get the lowest cost, while hers is to make most profit out of the
deal.
The hub model of integration also has its advantages and disadvantages, which
are much like the ones in the human world:
The main advantage of the hub model is that it decreases the number of interface
points. As a result, the adapters we need decreases to a linear proportion.
Like its human counterpart, the hub can be a specialist in all things having to
do with a particular type of transaction.
The main disadvantage of using a hub model is the cost. By using an intermediary,
the transaction executes slower than in a point-to-point integration
model.
Also, by putting a hub application in the middle, all applications in the environment
need to learn how to speak the hub application's language and protocols.
The integration standard becomes some other application's proprietary
language and protocol. Application A, B, C, D and E all need hub-standard
adapters.
Using a hub-based integration architecture does not solve all of an organization's
problems. An organization will usually deploy more than one hub application
because of the limited environments and functions hub applications support. For example,
some hub applications are good for integrating applications using message
queues as a protocol. Other hub applications offer protocols and specialize in adapters
for a certain number of platforms.
By deploying a hub-based integration architecture, organizations accept the
need to modify their applications to be understandable to the hub application's proprietary
language and protocol. Other applications do not know anything about the
hub application except at its interface points. Programmers need to write adapters—
or organizations need to buy adapters—that enable existing applications to communicate
with the hub application and through the hub application with other
applications that require their data to operate.
We will see where the next generation of interoperability is, but first, let's take a
look at integrating internal applications with SOAP.
SOAP in Application-to-Application Integration
We have discussed how SOAP works in cross-enterprise integration in Chapter 7, Simple
Object Access Protocol (SOAP) on page 139.We used HTTP and port 80 in order to
exchange SOAP request and response messages as shown in Figure 104, SOAP Over
HTTP.
The SOAP client's job is to form a request document using an XML vocabulary
and to send it to SOAP server using HTTP port 80. The SOAP server's job is to read the
SOAP message and translate the body of SOAP message into a language that the remote
object speaks and pass it that information. Once the remote object gets the document
and reads the content, it does what it has been doing all its life and returns an answer
to the SOAP server. The SOAP server takes the answer, converts it to a standard response
language and sends it back as a SOAP document using HTTP port 80.
Now we would like to change the scenario. Instead of integrating objects across
the firewall, let's integrate objects inside the same organization.
SOAP over HTTP is not necessarily scalable for application-to-application communication.
HTTP requires a certain amount of overhead that is required for the
untrusted Web, but is overkill for internal use, and not fast enough.
The organization probably has an internal network that will connect applications
such as SOAP client and SOAP server messaging. It would make sense, then, to
use the internal network connection, using a high-performance protocol such as TCP,
instead of HTTP.
In this scenario, The SOAP client is sending the SOAP request document over
TCP, or any internal network. This is shown in Figure 105, SOAP Over Internal Network.
However, in real life business environments, client applications do not invoke
objects by themselves, send requests every once in a while to the local server, get answers
and throw them into the trash. Usually a requestor object that is in need of data that the responder object can provide will be sending a message to the responder local
object.
Therefore, let's attach an object to the SOAP client. This is shown in Figure 106,
Object is The Client. Now we have a local object instantiating the SOAP client. The
SOAP client understands the local object's language and translates it into the standard
language and packages the request as a SOAP document; and sends it over TCP.
Now, the local object can get information from the other local object using
SOAP over an internal network.
Instead of referring to the objects "local objects", we would like to give them
names. They can be a Java program and a COBOL program that have been encapsulated
as objects. This is shown in Figure 107, Objects Now Have Names.
The Java object sends a SOAP request document to the COBOL object. The
COBOL object runs its application and returns an answer. The answer is sent to the
Java object as a SOAP response document.
Java does not speak COBOL‘s language and COBOL does not speak Java's language.
They do not have the same type definitions, the same method call syntaxes, or
the same data structures. However, the SOAP server that sits on COBOL program can
speak COBOL and SOAP, The SOAP client that sits on the Java object can speak Java and
SOAP. Therefore, the integration between two different worlds is enabled by a SOAP
server and client that act as translators.
Now, what if the COBOL program needs to get data from the Java program? The
scenario we have discussed above can be reversed by reversing SOAP client and server
programs, as shown in Figure 108, Client and Server Reversed.
Now, the SOAP client program is on COBOL‘s side, and SOAP Server is on Java's
side. As a result of this change, COBOL program can access the Java object.
The COBOL program is sending a SOAP request document with its SOAP client
program. The Java object is getting the request and invoking the methods, and returning
an answer back to the COBOL program as a SOAP response document. The
SOAP client document gets the SOAP response and feeds it into COBOL program, as a
screen input or a COBOL record layout.
What if we needed two-directional information flows? What if some COBOL
programs and functions needed data that the Java object has, and some Java functions
needed data that COBOL has? Then we need SOAP client and SOAP server on both
sides.
As a result, in Figure 109, Everyone is a Server and a Client, SOAP clients and SOAP
servers reside on both sides.
This enables two-way messaging. Java objects and COBOL objects can communicate
with each other and access each other's data with SOAP clients and SOAP servers
that act as translators.
The SOAP client and SOAP server programs form a "SOAP implementation"
when they are combined together. From now on, we will refer to the SOAP client and SOAP server programs together as a SOAP implementation. This is shown in Figure
110, SOAP Implementation.
SOAP implementations work as plumbing for the objects. This evolution is the
same as the evolution from separate programs to COM objects. Before COM, the objects
and applications in the Microsoft world were separate, they could not access each
other's information. Data was imprisoned.
However, with COM, objects that are developed on the Microsoft platform have
plumbing that helps the objects integrate with each other using that plumbing. This
plumbing was meant to act as we have described here. That is, allowing system architects
and programmers to integrate different applications without the need for adapters.
However, they enable data flowing from one object to another only as long as
everyone they need to access is COM-compliant.
In contrast to that, SOAP implementations act as standardized plumbing, which
enables heterogeneous applications to talk each other. In addition to that, SOAP implementations
enable objects of different types to integrate with each other using
open standards that are built on top of XML.
These objects can be from completely different eras, architectures, and locations.
Their data can be useful to each other's application and process now because
SOAP implementations have integrated the applications and data, as shown in Figure
111, Objects Communicating Directly.
In our scenario, we are referring to the internal network as TCP. However it can
be any protocol that SOAP supports or will support, such as User Datagram Protocol
(UDP), or IBM MQ Series. For example, IBM‘s SOAP implementation in theirWeb Services
Toolkit has native IBM MQ-Series support. The communication cloud can be
named with whatever communication protocol Java and COBOL use in their organization,
as shown in Figure 112, Any Network or Transport Technology.
What will happen when a different object that speaks a completely different language
is added to the business? in Integration Problem Statement on page 197, we
mentioned the "N-squared problem". What kind of changes do we need to make in
our web service-enabled integration model in order to embrace COM object and its
valuable data?
All we need to do to communicate with each other is to use SOAP implementations
as our plumbing. The COM object needs to speak the standard language in the
organization with the help of a SOAP implementation around the COM object.
As we see in Figure 113, New Object, when we add a SOAP implementation that
can speak COM‘s language as well as the standard language, the integration problem is
solved. A SOAP implementation that speaks COM and SOAP is Microsoft's SOAP
Toolkit. As are most SOAP implementations, Microsoft's SOAP Toolkit is free. As a result,
the objects of different types, and programs built during different eras, can communicate
with the help of SOAP implementations that act as plumbing.
This is a loosely coupled system. The COM object does not know what kind of
object is accessing it. It behaves exactly the same whether it is a Visual Basic client or a
CORBA or a COBOL object accessing the COM object. CORBA and COBOL with the
SOAP implementation behave exactly the same.
Services Point-of-View
Web services are not just for external communications. By using web services, organizations
need to focus on how they can make their organization a service provider.
They need to adopt a services point-of-view.
Any application or group of applications, departments, or organizations that
deliver a service can be exposed as a web service. Considering this point in the application-
to-application integration scenario, the heterogeneous objects or programs or
groups of these objects or programs integrates using SOAP implementation. They
communicate with each other. For example, in the scenario we just discussed, a COM
object can be an ordering service, the Java program can be a shipping service and
COBOL object can be an inventory service.
As a matter of fact, SOAP in application-to-application integration is not scalable
enough to be deployed for now. Open text messages are bigger than binary files to
transfer. SOAP messages have overhead due to envelope definition. We do not have
fast enough networks to integrate applications using SOAP.
We do not think SOAP will replace the integration middleware solutions we have
today in the near future. However, support for SOAP in the middleware is a fact. Many
middleware technologies support SOAP in internal integration.
In addition to that, is be conceivable and possible to integrate web services in an
organization using SOAP implementations. You could call this "services-to-services"
integration. Integration of services will happen using SOAP implementations whether
it is integrating services inside or outside the organization.
Each of these objects defines a service in the organization. Therefore, integration
of these services is enabled by SOAP implementation as shown in Figure 114,
SOAP Implementations as Web Services.
Beyond the Hub: A Standard Infrastructure
In Hub ModelWith Adapters on page 201, we covered the Hub model for integration.
This model has some benefits over point-to-point integration because it reduces
the number of adapters required. But what if we were to get rid of adapters
altogether?
As we stated in the beginning of this chapter, we have an integration problem
because applications do not use the same protocols, speak the same syntax, and adhere
to the same frameworks to pass parameters and exchange data.
If applications were all written using a single format, syntax, and protocol, we
would not have a problem.
Would not it be nice if there was a standard syntax and protocol that all application
vendors support? If this was the case, then all applications can communicate with
each other without the need for adapters. Applications will support the one agreed
syntax and agreed protocols out of the box, just like they support FTP, HTTP, without
adding extra code.
Now let's consider integrating web services.We have already discussed that web
services are business processes exposed by an interface that supports XML and standards
that are built on top of it, such as SOAP and WSDL.
Each web service supports XML syntax and the SOAP envelope protocol. A description
of the package is expressed in a standard language called WSDL.
Due to standards, web services will not have integration problems. Every web
service can communicate because its syntax, definitions, and protocols are all standardized
by definition.
Standardization in interfacing with the outside world eliminates the integration
problem. Integrating N different web services requires no adapters at all. This is because
they already speak using the standard alphabet (XML), using a standard protocol
(SOAP), and a standard description language (WSDL).
Web services integration is shown in Figure 115, Integrating Systems with SOAP.
Notice that there is no hub and no adapters. When web service A needs data that
web service B has:
Web service A reads the contract of web service B, which is B's WSDL file.
Web service A composes a request in XML syntax, using B's language that it
learned from the WSDL file.
Web service A sends the request to B using SOAP as a protocol.
Web service B gets the request from A, processes it according to its business
process definition, that is described by a workflow, and then sends an answer
back to web service A. The language of the answer is also described in B's WSDL file.
Response from web service B arrives to A using SOAP protocol. Web service
A reads the response and uses the data to service its business processes.
Web services can reside inside or outside of an organization's firewalls. Whether
they are inside or outside the organization, they are defined and expressed using the
same standards. They all support the standardized alphabet, vocabulary definition,
protocols, and frameworks by definition.
In conclusion, a web service supports loosely coupled, message-based integration
using XML, SOAP and WSDL. And because of this fact, there are no integration
problems for integrating web services.
We will discuss the integration of web services inside the organization in the following
section.
Web Services Inside of an Organization
We discuss the use of web services outside of company's firewalls in Chapter 3, Where
We've Been, WhereWe're Headed on page 39. Now, let's take another scenario. In that
scenario, the company that provided spare parts, ToyCarParts, has a web service for
ordering. The ordering process has a business flow that is made up of many applications.
The business process was initiated by a purchase order that arrives at the order
processing department.
We discuss the importance of integration, workflow and automation in web services
among the entities that make up a business process in Table 2, I.T. Architecture
Grid.
Inside the organization there are business processes that can be described as services.
These services are not necessarily directly consumed by outsiders, but they are
consumed by applications, or business processes inside the organization.
As we see the workflow for ordering in Figure 116, Process Workflow, checking
the customer account is a web service exposed for internal use.
When a business process is analyzed and formed as a web service, it can be
modularized and reused in the organization. Once the process of checking a customer
account is a service, then the details of this business process do not need to be defined
every time it is needed. It will be defined in one place and reused by many.
After a production decision is given in the workflow by a human decision-
maker, the production system needs to be notified. For an ordering service
workflow, the details of the production business processes are out of its scope. However, if the production system is not a self-contained service, the ordering process
needs to define and handle errors, problems, or other exceptions coming from the
production business process.
In the same manner, if we define our production system, integration, workflow,
and automation in a production web service definition, then we can reuse it for many
other services. Production will be a web service that is for internal use only because
the data needs to be private. The outcomes of the production web service can be a
purchase order acknowledgment or other notifications if things go wrong or the delivery
date changes.
Integration
The ordering service is made up of many applications that integrate with each other
following a certain workflow. The production and credit approval web services work
the same way.
The ordering web service calls the credit approval and production web services.
As a result, the integration problem within the organization between the ordering
web service, the production web service, and the credit approval web service becomes
a problem of simply integrating several web services.We can also add shipment
and accounting web services, which will be needed for the goods to be delivered and
invoice to be sent. This is illustrated in Figure 117, Object as Web Service.
As a result, once organizations define web services to be delivered inside and
outside of their organizations, they integrate these services with the same standard
implementations, XML and SOAP. The integration is not a problem as we have discussed
in integration problem Integration Problem Statement on page 197.
Therefore, vendor organizations leverage already existing integration architecture,
while taking this further and integrating business processes within the organization
by standards, such as XML and SOAP.
In conclusion, data is no longer imprisoned in applications or objects, but flowing
through services inside and outside of organizations. The architecture and technologies
used for internal or external web services are the same. This is another
advantage to leverage companies' know-how. Once organizations build the web services
integration architecture, the same technologies are used inside and outside the
organization.
So, What's It All Mean?
The integration architectures we have today are very limited.
Integration can be synchronous. If it is synchronous, then probably we are using
a proprietary direct object call. This option is tightly coupled and scalable. However,
there are management issues as we have discussed in Integration Problem Statement on
page 197.
If we want to deploy an asynchronous integration option, then we can do it
with a message-based loosely coupled system such as a queuing application.
The integration model using SOAP can be synchronous or asynchronous, but
because its message-based architecture, it enables loosely coupled integration in all
cases. Therefore, SOAP gives us a solution to integrate heterogeneous objects, and objects
in different physical boxes in a loosely coupled manner both in synchronous or
asynchronous models.
SOAP is an accepted standard by the leaders of application and solution developers
industry such as Microsoft, IBM, Sun Microsystems, BEA and many other organizations
including open source initiatives.
Organizations have been successfully using methods to send messages to trading
partners without knowing anything about their business implementations for
years using paper documents. By deploying SOAP over HTTP, an organization can
make DCOM or CORBA calls over the Internet in the same way, without knowing
about the implementation details of the target organization. We can also use these
technologies to do business document exchange.
SOAP is another alternative to existing protocols for sending business documents
in a loosely coupled manner. SOAP also allows organizations to treat trading
partners' services and offerings as a part of their in house applications by its message-
based method invocation.
Even if web services become common in communicating with trading partners
and leveraging their information, it is important to understand that a web service is
not about only trading partner relationships. It can be an internal department that is
delivering a service, or an application or even an object.
The new technologies built on top of XML enable services, whether in-house or
external, to communicate easily. These technologies ensure that the building blocks
for application communication are easy to implement and extensible.
The new technologies give opportunities for integration in document and
method-based models. As a result, the web and any integration protocols will change
how data lives its life. Data can flow among heterogeneous systems no matter where
the application sits, behind or across the firewall, or on what platform the application
lives, and what language the application speaks. This is enabled by high adoption of
XML technologies and the existence of implementations of these technologies in every
platform, for every application, from Microsoft to Java, from COBOL to enterprise integration
applications.
