Integration of information systems using the corporate service bus (ESB)
Among the best practices for integrating complex information systems is the construction of logical data marts, as well as the creation of centralized data exchange infrastructures using MDM systems and corporate service buses (ESB, Enterprise Service Bus). Our solutions, including the ArchiGraph.MDM system, are suitable for use as part of the operating system special purpose Astra Linux Special Edition, as well as Alt Linux.
Why is an integration bus needed?
Any company that uses more than two software products that work with overlapping sets of information knows what the price of not communicating between them is. Unsynchronized customer lists or product ranges and other information between ERP, CRM and other corporate applications incur constant losses of time, resources, company reputation. The only correct solution to this problem is the implementation of an enterprise service bus (ESB), in conjunction with a master data management (MDM) system.
Solutions based on regular uploads and transformations of information (ETL), service-oriented architecture (SOA), provide only a temporary solution to such problems, have many disadvantages, and limit the growth of IT infrastructure.
Implementation of the integration bus
When introducing an integration bus, the task of mapping (comparing) the structure of information about the same objects present in different information systems arises. It is necessary to solve this problem by creating a common, neutral in relation to each application, information model. Such a model can be most effectively implemented using semantic technologies. For optimal implementation results, a professional data architect is required to develop the model.
An important part of the implementation project is the implementation of connectors (programming interfaces) for all applications involved in data exchange. Our specialists have experience in developing and connecting connectors for a wide range of software on various platforms.
We carry out integration projects together with partners based on IBM WebSphere MQ, Integration Service Bus, WSO2 Message Broker, Apache Synapse, and Business Semantics bus solutions.
Often a project to implement an integration bus is carried out as part of a general reengineering of the enterprise information architecture.
2005
Enterprise Service Bus - a "budget" approach to solving integration problems
Prepared by: based on materials from foreign sites
Translation: Intersoft Lab
Continuing to acquaint the reader with various approaches to integration, we decided to focus on a relatively new and very attractive technology - the corporate service bus (Enterprise Service Bus, abbr. ESB).
What is an Enterprise Service Bus and how does it compare to the Enterprise Application Integration (EAI) discussed in previous issues of the DWH, OLAP and XML Connoisseurs Club Magazine? According to the established tradition, we will first give the floor to experts in this field.
Gartner analysts define ESB as a new type of middleware that integrates functionality others already existing types intermediate support. The Enterprise Service Bus supports Web services by implementing the SOAP (Simple Object Access Protocol) protocol and using the Web Services Description Language (WSDL) and the Universal Description, Discovery and Integration (UDDI) specification. Universal description, detection and integration). Many enterprise service buses also support other communication styles, including guaranteed delivery and publish and subscribe. The services provided by these buses have an "added value" that lightweight messaging middleware does not have - they provide message inspection, transformation, content-based routing, security, service discovery for service-oriented architecture, load balancing, and registration. Some services are built into the bus base, others run in plug-ins. In addition, buses support XML and other message formats.
Why is the corporate service bus so attractive? First of all, its relative cheapness. ESB products are generally positioned as affordable, or as they say, "budget" solutions.
Indeed, today there is an increase in demand for integration technologies. And while earlier the deployment of EAI products was associated with the achievement of strategic goals and, therefore, paid off in the long term, the tasks that in this moment companies have to face are tactical in nature and require new approaches. "Modern business realities" have drawn attention to areas where EAI vendors have traditionally been weak - transformation, developer-centric programming (such as Java), and integration with external technologies. All this "prepared fertile ground" for the emergence of a new category of products - ESB.
Speaking about the virtues of an enterprise service bus, it is worth quoting the words of Roy Schulte, vice president and member of the research department at Gartner: “The usual software the middle layer can no longer support new applications that use service-oriented (Service Oriented Architecture, abbr. SOA) and event-driven architecture (Event Driven Architecture, abbr. EDA), Web services and business process management. This is the main reason why information systems architects and managers need to leverage their corporate information infrastructures with ESBs.”
Gartner Lead Analyst highlights groups of ESB vendors. He refers to the first group of ESB products, which are positioned as "budget" integration solutions that are best suited to support composite applications and SOA. The second group are products intended for the Web services market, and finally, the last ones are software, providing EDA support. According to Roy Schulte, the ESB market will densify over the next years, driven by growing demand for Web services and multi-protocol and event-driven solutions.
Interestingly, in a number of companies ESB is treated not as a category of products, but as an architecture. For example, at IBM, an enterprise service bus is considered an "architectural pattern".
Thus, it can be stated that there is still no clear definition of what ESB is. In fact, the discussion revolves around two questions:
- Is the ESB an architecture (and one that doesn't even need to be standardized), a "one-sided approach" or a standalone product.
While it is beneficial for a number of vendors that do not currently have off-the-shelf solutions to talk about Enterprise Service Bus as an architecture, the current situation is such that customers need their products to offer ESB functionality. Therefore, we should expect an increase in the number of vendors offering ESB capabilities over the next two years.
- What is the place and future of ESB products, namely, is the enterprise service bus a more advanced message queuing system that provides simple XML transformation, as well as routing and messaging, or will the use of application adapters, automation and business process modeling allow ESB to be successfully replaced EAI.
At the moment, there are no definitive answers to these questions.
However, it is worth emphasizing that while there is no clarity about the Enterprise Service Bus, it is clear that the ESB's open standards approach can significantly reduce acquisition and implementation costs.
Note that the word "service" in the term "corporate service bus" is central. Thus, Forrester Research analysts consider ESB as "a layer of middleware with which you can access a set of basic (reusable) business services." SOA allows most functionality to be exposed as a "service" in an enterprise service bus that forwards, transforms, and validates input and output data into XML format received from these services.
ESB and XML
It would be unfair if we did not emphasize the special role of XML - XML is the basis for integration. If we accept that XML is more of an "alphabet" than just a language, it becomes clear that fully implementing integration requires orchestrating business processes, managing XML transformation, and validating and forwarding XML messages throughout the organization. All these functionalities form the basis of the Enterprise Service Bus.
XML can provide a comprehensive representation of a set of data. The popularity of this language can be attributed to its high flexibility and extensibility. Indeed, the XML syntax allows you to modernize and develop specialized XML schemas for processing various data.
At the same time, the rapid growth in the volume of data to be transferred creates serious difficulties for the functioning of the corporate information structure. So, the first integration projects that used the capabilities of XML were "living" proof of the promise of this language, but now there are certain problems with the increase in the number, size and complexity of XML documents. The following are the main reasons for the existing difficulties (insufficient scalability):
- Whole document parsing: Typically, you want to parse entire documents, even if only a portion of it needs to be extracted for routing and filtering. If the documents become large, the waiting time increases.
- Rescan. Documents are often re-parsed - at each stage of the business process, in other words, the same documents can be scanned several times. Since this practice is extremely resource intensive, performance and throughput are degraded.
- Single strand execution. In this case, the next processing step cannot be started until the current one is completed; as a result, latency increases because the entire process depends on the slowest step.
To overcome the lack of scalability, it is recommended to use a processing architecture that supports the following features:
- Document Streaming - Ensures that XML documents are processed as each element arrives, i.e. ensures low latency. This approach makes it possible to process large messages as productively as small ones.
- Selective processing, which achieves a very significant performance improvement by processing only relevant fragments rather than the entire XML document.
- Multi-threaded processing, in which the processor manages the alignment of sequential steps on a channel, the parallel execution of individual steps, and the load balancing of identical steps when processing multiple XML fragments.
- The only scan in which, instead of several repeated readings of the structure of the same document from the very beginning, all the necessary fragments are extracted in one transfer.
The above functions can be implemented using the Enterprise Service Bus - and without special coding and configuration.
What is the difference between an enterprise service bus (ESB) and message brokers (like RabbitMQ)?
As a result, the problem of unsatisfactory performance can be solved.
Conclusion
Judging by publications in foreign Internet publications and estimates by analysts from leading research companies, the corporate service bus is no longer just new technology with great potential. Indeed, Gartner predicts that in 2005 most large companies will use ESB. IDC believes that the corporate service bus should "revolutionize" Information Technology and enable flexible and scalable distributed processing.
Indeed, support for open standards (and especially XML) provides an inexpensive but effective solution and guarantees a quick return on investment, i.e. high rate ROI. As Steve Craggs, vice president of the Integration Consortium, notes, "ESB is the basis for integration, provides a flexible and customizable environment that allows fruitful, successful and systematic implementation of integration projects."
And yet, the uncertainty with the ambiguity of the term "corporate service bus" still remains. Today, ESB stands for any technology needed to implement SOA. This is exactly the point of view held by ZapThink, a company specializing in the development and application of service-oriented architecture. In this regard, ZapThink analysts warn that unless a real and concrete definition of an enterprise service bus is developed in 2005, the term ESB "will forever leave the SOA lexicon." As for SOA itself, it will be discussed in the next article.
Publications
- Beth Gold-Bernstein Is an ESB Critical To Your Future?
- Nigel Thomas and Warren Buckley Rise of the ESB.
- Materials published on the website of the Integration Consortium.
What is ESB and SOA¶
An excellent description of system-of-systems thinking
Nick Coghlan, Core Python Developer
also available in Catala, Deutsch, English, Francais, Italiano, Nederlands, Portugues, Turkish and 中文 .
The abbreviation ESB and its associated SOA can be a source of confusion. ESB stands for Enterprise Service Bus. SOA - Service Oriented Architecture.
These names mean little, so the following is more full information in plain language, without excessive corporate vocabulary.
All the truth¶
Let's imagine what happens when you log into your bank's front-end application:
- Your name is shown
- Your account balance is displayed
- Showing your credit and debit cards
- There may be a list of your mutual funds
- You also receive a list of pre-calculated loans you may be interested in
With a high degree of probability, we can say that all these pieces of information belong to different systems and applications, each of which provides data through some interface (HTTP, JSON, AMQP, XML, SOAP, FTP, CSV, or any other):
- from CRM running Linux and Oracle
- from a COBOL system running on a z/OS mainframe
- they say this information came from a mainframe system, but these guys are too silent to say anything other than that they prefer CSV to everything else
- from a mix of PHP and Ruby running on Windows
- from PostgreSQL, Python and Java running on Linux and Solaris
The question is, how can you get the frontend application to talk to systems 1-5? Well, no way.
This is fundamental to making sure such environments can scale beyond multiple systems. You don't force them to communicate directly with each other.
In the diagram below, each call to a service that is provided by another system is represented by a line with different thickness and style:
Notice we didn't even show any high-level processes (App1 calls App2 and either App3 or App5 depending on whether the previous response from App6 was successful, so that App4 can later take the data that was generated by App2, but only if App1 doesn't forbid it, etc.).
Also, note that we are not talking about servers - each of the systems can run on 10 physical servers, so at least 60 physical components will communicate with each other.
However, some issues become apparent.
How to separate interfaces? How to schedule a download? How do you coordinate updates or scheduled downtime when each application is managed by different development teams, vendors, or departments, and half of the original developers have already left the company?
If you think you can manage 6 apps, how about 30?
Can you handle 400? And since 2000? Each application can be its own unique ecosystem, requiring a dozen servers or other devices to run, so that's 20,000 moving parts spread across continents with all sorts of technical and cultural boundaries. And all these parts constantly and non-stop want to exchange messages all the time without any downtime whatsoever. (We'll spare you the schema.)
There is a great name for this situation - a mess.
How can you fix the situation?¶
The first step is to recognize that the situation is out of control. This allows you to look for a solution without feeling a strong sense of guilt. Well, it happened, you didn't know how to do it better, but there is a chance that all this can be fixed.
This may bring organizational changes in the way we approach IT, but another step is to remember that systems and applications are not just about sharing data. They are designed to support business processes, regardless of the business itself, be it banking, audio recordings, radar devices, whatever.
When you clearly define these two points for yourself, you can start designing or redesigning your systems around services.
A service is something interesting, reusable, and atomic that is provided by one system to other applications that want to use it, but the service is never directly exposed one-to-one. This is the shortest and most meaningful description of all possible.
If a given system functionality satisfies these three requirements:
- I interesting (interesting)
- R eusable (reusable)
- A tomic (atomic)
then there is a good chance that the system can and should be exposed to other systems as a service, but never connected directly.
Let's discuss the IRA approach with a few examples.
| Variable | Notes |
|---|---|
| Environment | Electric company CRM system |
| Functionality | Return a list of customers who were active on the Self-Service Portal in Q3 2012 |
| It is interesting? | Yes, interesting enough. This can be used to generate all sorts of useful reports and statistics. |
| It's possible | Not, no so much. Although this allows you to create |
| repeatedly | high-level constructs such as statistics for the entire year, |
| use? | it is clear that we will not need this in 2018. |
| Is it atomic? | Probably yes. If there are already similar services for other quarters, then you can view the entire year. |
| How to make an IRA out of this? |
|
| Variable | Notes |
|---|---|
| Environment | e-commerce site |
| Functionality | Return every piece of information that has ever been collected for the specified user |
| It is interesting? | In general, yes. If you have access to the whole, you can always choose what you need. |
| It's possible | Oddly enough, not really. There will only be a few |
| repeatedly | applications, if any, who will be interested |
| use? | use absolutely every piece of information. |
| Is it atomic? | Definitely not. This monster of functionality is destined to be logically composed of dozens of smaller parts. |
| How to make an IRA out of this? |
|
| Variable | Notes |
|---|---|
| Environment | Any CRM system, anywhere |
| Functionality | Update CUST_AR_ZN column in C_NAZ_AJ table after someone has created an account |
| It is interesting? | Absolutely uninteresting. This is an internal function of the CRM system. No one in their right mind wants to deal with such low-level functionality. |
| It's possible | Yes, probably. Account can be created via |
| repeatedly | multiple channels so it seems to be something multiple |
| use? | used. |
| Is it atomic? | Seems to be yes. This is a simple update of one column in a table. |
| How to make out | |
| this IRA? | Don't even try to make a service out of it. It is not interesting. Nobody wants to think about specific columns and tables in one system. This is a tricky part of a CRM system, and even though it's reusable and atomic, it's not worth building a service out of. It's yours and CRM, it's your responsibility to think about it, don't force others to bear it too |
| Variable | Notes |
|---|---|
| Environment | Mobile operator |
| Functionality | Replenishment of a prepaid communication card in billing |
| It is interesting? | Extremely. Everyone wants to use it, through text messages, IVR, IM, portals, gift cards etc. |
| It's possible | Yes. It can take part in many high-level |
| repeatedly | processes. |
| use? | |
| Is it atomic? | Yes, from the point of view of the calling application, the card can be topped up or not. The fact that billing implements this through a series of steps is not important. From a business point of view, this is atomic, it is an indivisible service provided by billing. |
| How to make out | It's already an IRA. |
| this IRA? |
If you've done even a little bit of programming in the last 50 years, it will become obvious to you that providing a service is like providing an API in one piece of code for another. The only difference is that you are not dealing with submodules of one system, you are working at the level of a whole environment of individual systems.
Availability of services in ESB and SOA¶
Now that you know that systems don't communicate directly and you understand what a service is, you can start using the ESB effectively.
It is now the ESB's job to provide and invoke integrated systems services. Thus, in most cases only one access method, one interface must be defined between each system and the ESB.
So if, as in the diagram above, you have 8 systems - then you have 16 interfaces (one in each direction) to create, maintain, manage, and provision.
Without the ESB, you would have 56 interfaces to create and manage (assuming each system talks to every other).
No additional 40 interfaces means less wasted time and more money saved. This is one of the reasons why your Fridays will be less stressful.
This fact alone should prompt you to consider implementing an ESB.
If one of the systems is rewritten, transferred to another owner, divided between departments or vendors, then it will be the task of the ESB guys to make the appropriate changes. None of the other systems will even notice this, since their interface with the ESB will be untouched.
Once you start breathing IRA services on a regular basis, you can start thinking about composite services.
Remember the “give-me-whatever-you-can-do-about-this-client” service above?
It wasn't a good idea to create one, but you will sometimes run into client applications that need to aggregate information. The ESB guys will be in charge of this and it will be their job to select the best atomic services to build a composite service for that particular client system requiring that particular composite data set.
Over time, the entire organization will begin to realize that it is no longer about database tables, files, packages, functions, subroutines, or records. Now we're talking about an architecture centered around interesting, reusable, and atomic services provided by ESB applications.
People will no longer think that applications and systems are sending messages to each other. They will see the ESB as a one-stop gateway to interesting services that their own systems can use. And they won't even check who provides what, their systems will only deal with ESB.
It will take time, patience and coordinated efforts, but it is doable.
But beware…¶
Most The best way destroy the whole concept of SOA - deploy ESB and expect problems to fix themselves. While a great idea, just deploying an ESB won't be enough, unfortunately.
At best, trying to hide something under the carpet, as in the diagram below, will lead to nothing:
Your IT guys will hate the system, and while managers will initially tolerate ESB as a new solution, it will eventually become a laughing stock. “What, is this the same new silver bullet? Hahaha.”
Such consequences are inevitable if the ESB is not part of a larger development plan.
So, is ESB only for banks and the like?¶
Not at all. it good decision in any situation that requires the coordinated work of several data sources and several access methods to achieve an interesting result.
For example, the task of collecting the latest temperature readings and publishing them to multiple channels such as email alerts and iPhone apps is a good fit for an integration platform.
Periodically checking and monitoring the operation of all instances of a critical application, and if not all of them are running, running a configured script and sending a text message to administrators is also great.
Anything that needs to be integrated into a clear, well-defined environment is likely a good fit for an ESB service. But, as always, the decision of whether something is really suitable for integration comes with experience.
corporate service bus
Of course, the Zato team can help.
But I heard that SOA is XML, SOAP and web services¶
Yes, some people would like you to believe that.
If the people or vendors you worked with sent a BASE64-encoded CSV file in a SAML2-protected SOAP message, then it's understandable why you got that impression.
XML, SOAP, and web services have their use cases, but like everything else, they can be misused.
SOA is about understandable and manageable architecture. Whether a service may or may not use SOAP is practically irrelevant. SOA as an architecture approach will still be valid even if no SOAP service is used at all.
If an architect designs a beautiful building, he cannot influence the color of the interior too much.
So no, SOA is not XML, SOAP and web services. They can be used, but they are only a part, not the basis.
You can refer lost colleagues to this article so that they understand what SOA is.
Further more¶
This chapter only covers the very basics, but should nevertheless provide a solid understanding of what ESB and SOA should look like and what it takes to be successful.
Other topics not covered here include (but are not limited to):
- How to get support from managers to introduce ESB
- How to assemble SOA architects and analytics teams
- Representing the Canonical Data Model (CDM) in an organization
- Key Performance Indicators (KPIs) - now that you have a common and unified method of providing services between systems, you should start to observe and analyze what is actually provided to you
- Business Process Management (BPM) - how and when to choose a BPM platform for service management (the answer is not too soon, learn how to build nice and useful services first)
- What to do with systems without API? For example, should the ESB get direct access to their databases (the answer is different, there is no golden rule)
So what is Zato?¶
Zato is an ESB and application server written in Python and can be used to build middleware and backend systems. This software is open source code with commercial and community support. And Python is a programming language known for its simplicity and efficiency.
Using Python and Zato allows you to increase productivity and waste less time.
Zato was written pragmatists for pragmatists. This isn't another hastily built system by a vendor in the wake of the ESB/SOA hype.
In fact, Zato was built from practical experience in "putting out fires" caused by such systems. Indeed, the authors of Zato have spent so much time fighting such environments that they have become virtually immune to any fires.
This is the forge from which Zato came into the world and therefore it can offer performance and ease of use not found in other similar solutions.
See you at here!
(Enterprise Service Bus) is designed to build a distributed information landscape of an enterprise. Software ensures the interaction of all integrated applications in one center, combining existing sources of information and providing a centralized exchange of data between different information systems.
Enterprise Service Data Bus DATAREON ESB is an effective means of ensuring the stability and completeness of information exchange, increasing the overall performance of the information system and reducing labor costs for its administration.
Enterprise Service Bus
Software DATAREON ESB officially included in the unified register of Russian programs for electronic computers and databases that can be purchased by state and municipal institutions (https://reestr.minsvyaz.ru/).
To integrate 2-3 information systems in small companies, DATAREON offers a software product based on DATAREON ESB - DATAREON MQ.
Functionality of DATAREON ESB
Tasks solved using the corporate service data bus
- Data transfer between different information systems (routed or point-to-point)
- Formation of a single information space in heterogeneous environments
- Building a distributed system based on an event model in the following options:
- building applications with end-to-end business processes based on the event model;
- creation of a system with synchronization of business applications in various information systems
- Receipt scalable control architecture enterprise/holding level
- Deployment data exchange systems at the transport layer and at the business logic level
- Delegating the task of building information flows analytical departments
- Reducing the overall complexity of the integration scheme and reducing the requirement for bandwidth channels
- Increase in overall stability data transport layer
- Reduced transaction costs when exchanging data between different departments
- Reduced overall costs maintenance and support of the information system.
Benefits of the DATAREON ESB Enterprise Service Bus
- Fast integration
- High reliability
- Ability to reuse resources
), formerly known as Axelot Datareon ESB, is designed to build a distributed enterprise information landscape. The software product ensures the interaction of all integrated applications in one center, combining existing sources of information and providing a centralized data exchange between different information systems.
The Datareon ESB enterprise data bus is a means of ensuring the stability and completeness of information exchange, increasing the overall performance of the information system and reducing labor costs for its administration.
The Datareon ESB software product is officially included in the unified register of Russian programs for electronic computers and databases, which can be purchased by state and municipal institutions.
Functionality
- Support for various standards and integration scenarios
- Centralized management of the integration landscape with the Eclipse ecosystem
- Data transformation (multi-step data transformation algorithms with control of various conditions)
- Data transfer of any size (vertical and horizontal scaling)
- Easy integration with products based on the 1C:Enterprise 8 platform
- Ensuring secure data transfer
- Diagnostics and monitoring of the state of the entire data network
Tasks to be solved
- Data transfer between different information systems (with routing or point-to-point)
- Formation of a single information space in heterogeneous environments
- Building a distributed system based on the event model in the following options:
- building applications with end-to-end business processes based on the event model;
- creation of a system with synchronization of business applications in various information systems
- Obtaining a scalable enterprise / holding level management architecture
- Deployment of a data exchange system at the transport level and at the business logic level
- Delegating the task of building information flows to analytical departments
- Reducing the overall complexity of the integration scheme and reducing the requirement for bandwidth of the channels
- Increasing the overall stability of the data transport layer
- Reducing transaction costs when exchanging data between different departments
2017
Axelot Datareon ESB 2.1.0.0
The AXELOT Datareon ESB solution was included in the Gold Application Development competency list - a fact confirming the high quality of the product and its compatibility with Microsoft products.
AXELOT Datareon ESB provides a number of key benefits for businesses:
- Possibility of integration;
- Reliability and reusability of resources;
- Obtaining a scalable enterprise / holding level management architecture;
- Delegating the task of building information flows to analytical departments;
- Reducing the overall complexity of the integration scheme and reducing the requirement for channel bandwidth;
- Increasing the overall stability of the transport layer of data transfer;
- Reducing transaction costs when exchanging data between different departments;
- Reducing the overall cost of maintaining and maintaining the information system.
The main features of the system:
- A large number of connectors various systems: 1С:Enterprise 8, SOAP services, REST services, MS SQL, IBM DB2, Oracle DB, PostgreSQL, SharePoint, OData, TCP, Siemens TeamCenter and others;
- Plugin mechanism for independent development of connectors;
- Support for various programming languages and technologies when developing interaction scenarios: 1C:Enterprise 8, JavaScript, T-SQL;
- Setting up multi-step data transformation scenarios using visual mapping mechanisms and arbitrary XSLT transformations;
- Work with various formats data (XML, JSON, XLS, DBF, CSV, Base64 and others);
- Static and dynamic routing of information packets;
- High speed of interaction and fault tolerance: reduced requirements for network bandwidth, load balancing, isolation of information domains, the ability to monitor the status of integration nodes;
- Event model support, synchronous and asynchronous calls, guaranteed delivery;
- Changing the integration scenarios of subscriber systems (mechanisms for unloading/loading, transformation and routing) in "hot" mode without the need to stop them (including configurations on the 1C:Enterprise 8 platform);
- Diagnostics and monitoring of all integration processes, debugging and tracing information packages.
Particular attention is paid to the integration of applications on the 1C:Enterprise 8 platform. The delivery includes a special subsystem that is built into any typical configuration on the 1C:Enterprise 8 platform and provides all the necessary mechanisms for quick and convenient integration setup and administration. AXELOT: ESB Service Data Bus interacts with the configuration on the 1C:Enterprise 8 platform via SOAP and REST services.
Server components "AXELOT: ESB Service Data Bus" are developed in C++. Administration and configuration of "AXELOT: ESB Service Data Bus" is carried out in the Eclipse development environment and can be performed in conjunction with the development of systems on the "1C: Enterprise 8" platform in "1C: Enterprise Development Tools". "AXELOT: ESB Service Data Bus" is multi-platform and supports Operating Systems MS Windows and Linux.
AXELOT Datareon ESB is a completely Russian development and is in the process of being included in the unified register of Russian programs for electronic computers and databases, which can be purchased by state and municipal institutions to solve certain problems.
In my opinion, there are two approaches to building an enterprise integration bus:
- "from integrable systems";
- "from implemented processes".
Approach "from integrable systems"
In this case, the integration bus is considered as a kind of transport that performs routing and negotiation of messaging protocols. All messages go through the chain: input channel of the adapter of the source system -> router -> output channel of the receiver system. The type of communication between these components and specific technologies depend on whether messages coming from one source system can have multiple destination systems, on the expected load and approach to ensuring data integrity (using a common transaction for all source systems, or data is transferred to each source system in its transaction).- Dependence on systems, not message types. Usually, the number of integrated systems is several times less than the number of message types transmitted.
- Ease of connection of new receiver systems: to connect a new receiver system, it is enough to enter the data in the routing table.
- Ease of implementation of the monitoring system of the integration solution: data for the monitoring system can be generated in one place - in the router (however, this point can only be accepted with reservations, since there is data that is generated only in the adapters of integrated systems).
- Ease of support solution. Since all messages pass through a single router, all the logic of message passing and tracking dependencies between messages can be implemented in one place - in this router.
- Sharing the system between developers. Since the system core and all adapters are independent of each other (communication is provided only through dedicated and described interfaces), the tasks for their development can be divided between programmers, which allows parallelizing the process of creating and implementing an integration solution.
- The solution is applicable only for the implementation of unified message passing logic, i.e. if there are rules for tracking dependencies and transformation that are common to all or most messages. If different types messages have completely different dependency tracking and exchange management logic, it will either have to be moved to adapters, which eliminates advantage 4, or it will be impossible to implement at all.
- This scheme is suitable for implementing an asynchronous exchange. In the case of synchronous or mixed exchange, the complexity in the implementation this approach increases significantly.
- Solution performance degradation may occur. For example, if a message is to be transmitted to each of the receiver systems in a separate transaction, the separation of the source system, the kernel, and the receiver systems by queues is required. These queues can become the bottleneck of the system.
Approach "from implemented processes"
In this case, each business process is considered separately, in which data exchange between several systems is required. Bus implements this exchange. The event that starts the exchange process is the receipt of a message from the source system. The message received from the source system is transmitted to one or several recipient systems, while not only transport functions are implemented, but the result of message processing is also monitored and the transmitted message is correlated with others.This approach has the following advantages:
- Flexibility. This approach allows you to implement your own, separate exchange logic for each type of message. This logic can be quite non-trivial.
- The complexity of the implementation of both asynchronous and synchronous exchange is approximately the same.
- Independence of threads, more precisely in this case it is more correct to talk about processes. Technical decisions made during the implementation of one exchange process do not affect the complexity of the implementation of another.
This approach has the following disadvantages:
- Dependence on message types. Usually the number of message types is many times greater than the number of integrated systems. When connecting a new source system to the bus, it is necessary to route messages by types and implement a separate exchange process for each type of message.
- If the same exchange logic must be implemented for several types of messages, then duplication of code and / or bus settings is possible.
- Message passing processes depend on system adapters and can depend on each other, as well as on service processes. The presence of such dependencies reduces the degree of parallelization of the process of developing and implementing an integration solution. The developers of some components depend on the results of the work of the developers of other components of the integration solution.
The choice of approach is carried out according to the following algorithm:
- Obtain a list and description of integrated systems and message types from analysts.
- Obtain from analysts a list and description of business processes that involve systems requiring integration.
- If the processes are trivial and there are much fewer systems than message types, the exchange is predominantly asynchronous, and it is also required to transfer one message to several systems, we choose the first approach. Decide on the transaction management policy.
- If the processes imply a predominantly synchronous exchange, while the processes are complex, i.e. the passage of a message depends on the results of its processing in receiver systems, then we choose the second approach. Another argument in favor of this approach is the fact that the number of message types is comparable to the number of integrated systems.
It must be clearly understood that these implementation methods are not a dogma, it is not necessary to choose only the first approach or only the second. They can always be combined, modern enterprise service tires ( ESB) allow you to do so.
Liked the post -
With this article, I would like to open a cycle dedicated to IBM WebSphere ESB (hereinafter referred to as ESB) in the context of development for this product. And, first of all, you will have to get acquainted with technologies of this kind.
Enterprise service bus (enterprise service bus) is a middleware that provides a centralized and unified event-oriented messaging between various information systems based on the principles of a service-oriented architecture.
Of course, it is possible without special software (maybe something general will still have to be developed) to build a corporate system based on this approach, and what happens as a result can be called a service bus. But in the product from IBM there is not only a ready-made apparatus for centralized messaging and control of this process, but also full set opportunities for developing flexible service-oriented applications specifically for ESB. In summary, the following features and benefits of IBM WebSphere ESB can be highlighted:
- Order and uniformity of architectural relationships
- Centralized Management
- Server side application configuration
- Implementation of Service Component Architecture (SCA) technology in the spirit of the principles of service-oriented architecture
- Protocol-independence of the developed program code
- Extensive bus and application configuration options
The classic ESB assembly topology is a cluster that provides horizontal scalability and fault tolerance. According to official recommendations, increasing the number of cluster members increases performance more efficiently than increasing server capacity in a stand-alone topology. In addition, the cluster can be rebooted (or part of it may fail) without stopping service.
ESB is usually used as a service layer in IBM BPM, but may well play a leading role in building an interaction model. corporate systems as a powerful integration device (meaning ESB as an add-on over IBM WebSphere Application Server).
This, in fact, is required from the ESB, since this is a “service collection point” - if you need a service that will work with other services (possibly external), then the integration between these services is most logically done on the ESB. For external or heterogeneous services, you can make the "wrapper" an ESB service. Let's illustrate a little the convenience of using "single housing" for services:
Order
The larger the system, the more important order and uniformity are in it. If we are talking about a complex of systems of a large enterprise, then it certainly can be called a system of large size. Of course, you can always find an administrator with a diagram of the interaction of hundreds of servers in his head, or a bunch of volumes of unrelated documentation for each software module, which describes what and how it interacts with. 
But it is much easier to have a service (ESB) that allows you to conduct all interaction through itself. With this approach, part of the interaction architecture in any subsystem is already clear - there is no mess in the communications between systems, servers and applications: everything is connected to the ESB and the ESB is connected to everything. 
Centralized Management
It is always more convenient to configure systems centrally - whether it be configuration, adaptation to server relocation, fault tolerance, load balancing, error handling, or monitoring and analytics. 
For example, when moving a database server, you don’t need to get into the configuration of all existing application servers, and into the settings of specific applications in particular - it’s enough to have one environment variable in the ESB, which specifies the database address, and then the changes will need to be made in just one point.
Or if one of the external systems was unavailable for a long time, and not a single request to it should be lost, you can use the service for processing failed events to “throw in” undelivered messages when it is convenient.
If you need to regulate the number of simultaneous requests to any system, or monitor these requests, analyze the load, look for bottlenecks - you need to go to the messaging control center - to the ESB server console.
Server side configuration
"Single housing" for services, in terms of configuration, allows you to achieve several useful goals. The first is configuration reuse (similar to code and module reuse, which is so useful in SOA), because different modules and applications can use the same database connection parameters, resources, authentication parameters, environment variables, and so on. 
Secondly, when configuring on the server side, it is the application environment that can largely influence it, which allows you to transfer applications between different circuits (test and production), tune and even fix bugs without making changes to the application.
If you use all these advantages, applications get the ability of a true chameleon - they are so flexible that they become part of the environment in which they work, and at the same time bring their important functionality.
But the flexibility of applications under IBM WebSphere ESB is not limited to the environment in which they run. Development capabilities make a huge contribution to this. Since systems not only need to have a place to run, but also need to be developed and finalized, these interesting points should not be missed:
SCA
This architecture is based on the principle that a component provides its functionality as a service available to other components. Within one module, the components are program blocks (java code) that fully implement some functionality described by the corresponding interface. The execution logic of the components is implemented by linking them into a structure by interfaces and references (Partner Reference).
It is very convenient to develop, check, develop, change and maintain such a module structure. The atomicity of the functionality implemented in the components allows you to operate on the components as a whole without going down to the code level. On the other hand, it is logically necessary due to the execution of component implementations in a transactional context.
Each component has an interface(s) that it provides an implementation of. Thus, by linking the components together, there is no need to know them internal features– it is enough that they implement the necessary interfaces.
Through this architecture, you can also solve all tasks that require parallel work, without "manual" flow control (for example, you can make asynchronous calls to several components with a delayed response).
Non-java components, such as the Export and Import types, allow you to provide services for external use or use external services accordingly; The Mediation Flow component provides low-level access to messages exchanged between other components and allows for various transformations when working with heterogeneous interfaces.
In addition to interfaces, the IBM business object framework provides very useful features. Business objects (BO), represented by xsd schemas, are used as objects for data transfer in interfaces, both between components and for communication between modules. They are directly integrated, for example, into the wsdl schema for describing web services. That is, for example, if module "A" provides its functionality as a web service, to use it, it is enough for module "B" to connect the interface and ready-made BOs, and it will be able to fully work with such a service without creating any additional java -objects for data transfer. BO is also convenient to use when exchanging data with the database, if this data is used by other components (this, of course, goes against the DAO pattern, but it eliminates unnecessary java objects and data rewriting operations back and forth).
Protocol-independence of the program code
As you can see, protocol-independence of the code is achieved by using the Export and Import components. Since communication with these components goes through interfaces and references, programming code completely independent of the protocol used for interaction. The same functionality can be easily made available on any number of supported protocols and on any the right interfaces. The following figure shows how to add an export with an SCA binding to a bean that already exposes its interface as HTTP, JMS, and a Web service. 
The benefits are obvious - flexibility, versatility, code reuse, speed of development and modification.
By the way, SCA binding uses a special protocol and is intended for communication between modules within the same server/cluster. Communication through this binding is less resource intensive and faster than other protocols.
Configuration
Server and application configuration is done through the server's IBM console.
ESB, like IBM WebSphere in general, has quite a few specific features and artifacts. For example, when using the same imports and exports, you can configure the end-points of the corresponding services on the fly. For service calls, you can configure policy sets with various rules (for example, you can install support for the WS-AT mechanism, which allows you to call a web service in the same transaction in which the client works; but transactionality is already a topic for a full article), set authentication parameters, connect certificates, and more.
Through configuration, you can set up some mechanisms for automatic response to exceptional situations (for example, automatic repetition of component execution in case of errors). You can set up component tracing on the fly or change logging levels. A failure event management service is also available, which can be deliberately used for bulk error handling.
And, of course, you can configure many other things according to the Java2EE specification, which is implemented, sometimes quite strictly, in the IBM Application Server.
All of the above confirms the ESB as a convenient, powerful and flexible integration apparatus, albeit not always easy to learn. In the future, you just need to learn how to use it.
The following images were used in the article:
I have been personally familiar with SOA for 10 years, during this time I have developed a clear associative chain: the basis of SOA is the data bus, well, since the data bus, then we are talking about integration projects.
I will not stir up history, we all remember how integration developed in its time: a long and painful path from the point-to-point concept to a bus with a dedicated data model in the integration layer. Today we are already talking about the Neoflex Integra integration product, which has an analogue on Russian market there is no banking automation yet, that the initial stage of analytics can be greatly reduced, and at the same time immediately evaluate the result of a future integration project.
Neoflex Integra appeared after the 450th or even 455th integration project carried out at the bank. It's hard to pinpoint exactly. But we tried to calculate the man-days devoted to integration practice by our employees, and it turned out to be approximately 137 years. Integration has always been one of the main areas of Neoflex's work, so you should not be surprised at such a figure. During this time we managed to understand two things. First, landscapes in lending institutions are classifiable. Second, there is a lot in common between integration projects, at least at the analytics stage. We have singled out the basis in the form of a general canonical model containing application objects and relationships between them, combining atomic services into business processes in the areas of activity of a financial organization. It seems fantastic, but now we can come to the bank with a set of ready-made “details” and use them to form a future integration model. Neoflex Integra is a new product, we announced it in the March issue of The Retail Finance. But today we can already say that it will allow to reduce the term of the integration project by about a quarter, in addition, the bank is no longer buying a "pig in a poke", as it was before, it pays for a very definite result. However, let's get back to SOA, and for those who are interested in the topic of an integration product, I invite you to visit the webinar "New Neoflex Integra product: the quintessence of project experience" on November 20th.
So, what will happen next, how will SOA develop, will the data bus remain, and will it take the same place of honor as it does now in the landscape of the credit, and, if you look more broadly, then not only the credit organization? I'll start from afar, about SOA in general, about what is the fate of this concept in the banking industry. In fact, the right IT architecture for a bank is the one that can meet its business needs. The larger the bank, the higher the requests. In Russia, with its multi-branch, geographically distributed structures, there are enough large banks, but even they cannot always afford a high-quality SOA landscape, and not only because it is very expensive, but also because it is a huge risk. SOA projects don't always end well. With our product, we plan to reduce the risks, and at the same time the threshold for entering SOA.
Do not think that the SOA landscape is a pipe business. Neoflex projects show that there are no non-integrable systems. Banks usually start to create a new landscape little by little, starting with the areas that are most important to them, but recently we have come across a scale that we have not seen before. It happened on a project at VTB24 bank. The company worked there to create a Universal Banking Services Layer based on the Oracle SOA Suite. The solution included more than 600 services that ensure the bank's work with all clients through all communication channels. Such a large and complex decision, according to colleagues from Oracle, they will not remember in Russia. And if it succeeded, then other projects are purely technically feasible.
The need to form a single logical layer and optimize the IT space is most experienced by banks that run a large number of retail projects. The speed of customer service is crucial here, and the techniques used in SOA practice can significantly increase it due to faster data flow. Therefore, leading banks such as VTB24, Probusinessbank ( Life group), which we also work with, adopted long-term SOA transition programs some time ago. Whether smaller organizations will follow their example is hard to say: the crisis is after all. But purely practical considerations are as follows. If we try to draw a conditional boundary between those who need SOA and those who can get by with simpler solutions, then this boundary, before the stagnation in the market began, passed approximately through the top 400 banks.
In other areas of banking automation, the concept of maximizing the use of ready-made functionality, or the product concept, successfully took root after the systems became componentized. I think SOA should not be an exception. But this is a matter for the future. In the meantime, many banks live in completely different realities, and the question on their agenda is not which integration product to use, but which bus to implement. Let's go down from heaven to earth.
What to do with IBM WebSphere ESB?
Undoubtedly, IBM WebSphere ESB is the market leader among integration buses both in terms of the number of projects and the number of implementations. Out of 10 integration projects, 8 are done on IBM WebSphere ESB. And only in the last 2 years the situation began to change a little, and the movement of other major global vendors began to appear on the market. I would not like to compare them within the framework of this article, since without it there is a sufficient number of comparisons and analytical studies on this topic. My article is addressed to those who at one time took a step towards SOA, and did it together with IBM WebSphere ESB. And they had to worry in earnest: all of a sudden, like snow on their heads, last year's news that IBM has changed the product line of integration software since 2014, and IBM Web Sphere ESB will no longer be developed, and support for this complex has only recently has been extended through April 2018. Previously, it was planned to be completed in 2014.
In this regard, more and more of our customers who used this integration platform began to ask questions about what awaits them.
Miracles do not happen, and I am not a magician, but I want to offer two ways:
1. Start thinking about using another platform that IBM is betting on in the future - Integration Bus (former Message Broker).
It has obvious advantages over IBM WebSphere ESB: performance and reliability. There are also enough projects and implementations with this platform, and someone from the very beginning chose it as an integration bus.
Its main technological feature is the internal use of Message Queuing (MQ) as a transport, which gives us guaranteed message delivery and, as a result, reliability.
2. Already now, when developing an integration solution on IBM WebSphere ESB, try to separate implementation levels in order to create new code not on IBM WebSphere ESB, but on other components, the use of which in the integration solution corresponds to the further evolution strategy. There are not many suitable products from IBM, only 2-3. One of them, in my opinion, the most promising one is IBM WebSphere Data Power, a hardware and software system for solving integration tasks that provides high performance and certified security. The advantages of this product are the highest performance in its class and the ability to solve a wide range of security tasks, both external and internal.
To date, we have more than impressive results from one of the pilot projects with IBM Data Power, completed at Renaissance Credit Bank: 40 requests per second are processed with 5,000 simultaneous connections, and the processing time for 1 request is less than 1 ms. It is worth noting that the CPU load during the tests was at the level of 5%!
Using IBM Data Power will help separate the layers of the integration solution between IBM WebSphere ESB and IBM DataPower, so when replacing IBM WebSphere ESB with a new integration bus, for example, with IBM Integration Bus, you do not have to redo the part for IBM DataPower.
This hybrid bus approach can also be implemented based on products from other vendors, i.e. not only IBM. Because the development of products of many vendors often goes in the same direction, but this is a topic for a separate discussion.
What should an integration data bus look like in the near future?
Here I just want to bring you to the fact that the proposals I have described correspond very well with the idea of \u200b\u200bthe tire, which, I think, will take place in the future. By introducing the concept of a "hybrid" bus and combining technologies, we are achieving a solution to various integration problems. Moreover, we are talking about both combining software and diluting “pure” software with the help of software and hardware systems.
Nowadays it is very difficult to cover all the requirements of a business with any one product. The old, tried-and-true "best of breed" principle is therefore perfectly applicable to integration tires.
Take several products, combine them with one common task, separate the components between them, “plant” them on a common integration data model, and you will not only get a multifunctional solution, but also protect yourself from “product addiction”.
So we found two simple solutions, each of which has its own advantages:
· IBM WebSphere ESB in combination with IBM Integration Bus is used to build SOA platforms, integrate complex applications and IT environments;
High performance IBM product DataPower, characterized by ease of configuration, is used to process simple requests and to provide multi-functional security.
Undoubtedly, hybrid buses are the future, unless we have a data bus in the cloud, even if it looks unlikely now. We will return to this issue in a year or two. Do you agree?
The block of BPM-systems was intentionally not considered here, because. this is a separate extensive topic.