What is the recommended way to integrate to other services from Magento 2?

Question

As part of building out an E-Commerence website for Company X, integrations into other systems within the company are required. These integrations include Ordering and Payment services. The APIs for these services are exposed via Rest and SOAP web services.

There are different approaches suggested by developers/engineers as to how we should write these integrations (as listed below). What is the 'industry standard' when integrating into other services from magento 2?

The following suggestions were provided by different developers/engineers working on this E-Commerence website:

1. Create a separate PHP client library for Ordering and Payment services, and import these libraries within Magento. As the customers place orders and make payments in Magento, use these client libraries to communicate with relevant services. The business logic will reside within Magento.
2. Create a standalone service independent of Magento, which implements the Ordering and Payment services API and exposes its own combined Rest API. As the customers place orders and make payments in Magento, take the in coming request objects, convert them to json, and forward json payload to this standalone service. This standalone service will extract the data from json payload and make calls to Ordering and Payment services. It will also have business logic and make API calls back into Magento for anything that needs to be updated.

---

Reasons for providing #2 solution are listed below and reasons against or reasons that nullify #2 are added as bullet points:

1. Portability - Install NodeJS or Java and run standalone servers anywhere
   • Magento 2 uses PHP 7 and is independent of operating system. The PHP client libraries will also be portable.
2. Enterprise Solution - any group within the company can makes calls to the standalone service and use it for their purpose
   • Considering the standalone services make API calls back to Magento API, other groups within the company cannot use the standalone service since it will effect this E-Commerence website.
3. Scalability
   • It does not make sense to scale the standalone services without scaling Magento since Magento will be forwarding calling to these services (source of the load). In order to handle increased load, both Magento and standalone services will need to be scaled together.
4. Separation of Concern
   • Using PHP client library, we can keep communication with Ordering and Payment services in the 'services' layer.
   • The business logic should remain within Magento.
5. Backward Compatibility - Upgrading to newer version of Magento will not affect communication with Ordering and Payment service.
   • Creating PHP client libraries for Ordering and Payment services will be independent from Magento and relies on PHP 7, not Magento.
   • If newer version of Magento changes the object structure for ordering or payments, the standalone services will not be able to parse new structure without an update.
6. Testing - Standalone services can be tested without Magento
   • True, but is it necessary to have the standalone services at all?

Answer 1

Here's how we went about implementing the Magento-To-Other-Systems integration on a recent client engagement.

Different Systems Involved:

• Order Capture System (Magento 2x)
• Integrator/Workflow Orchestration (Dell Boomi/Jitterbit)
• ERP/Order Processing System (NetSuite/Oracle EBS/SalesForce ECC)
• Marketing/Emails (SalesForce/ExactTarget)
• Data Warehousing (RedShift)
• Taxes (Avalara)
• Payment (Stripe/PayPal)
• CDN (Cloudinary/Cloudflare)

Some things that need to be hashed out before project/development kick-off:

• Identify plugins that can to be purchased v/s the ones that need to be built.

  • Example: Classy Llama - AvaTax / OOTB M2 - PayPal / MageDelight - Stripe / etc

• Identify the objects that need to be synchronized between the systems.

  • Identify the data needed by the different system need to perform CRUD operations on the different objects
  • Perform Data mapping between the different objects across the different systems.

• Define what system(s) will be the source-of-truth and which system(s) will be the system-of-reflection for the different objects.

  • Example - Magento is source of truth for Orders ; ERP system is the system of truth for RMAs, etc

• Identify the (ideal) sync schedule and frequency for the different objects.

  • Example: Sync Orders every 10 mins, Sync Shipments every hour, Sync RMAs/Credit Memos every 24 hours, etc.
  • Decide if object-sync workflows should be staged (i.e. perform step1-step2-step3) or transactional (perform all-steps-or-none-at-all).

• Define Happy-Path and Exception-Path flows for different object syncs.

  • Outline notification mechanisms - for errors/exceptions/success - per sync object.

• Identify the workflows that will need to be built on Dell Boomi (or Jitterbit) to perform the synchronization activities between the different systems.

For the integration between the systems:

• On a pre-defined schedule, your integration platform (Jitterbit/Dell Boomi) kicks off Workflows that:
  • Perform Read/Write from/to Magento (via the OOTB Magento 2x REST API)
  • Perform Read/Write from/to your ERP system (via APIs available on your ERP system)
  • Perform Read/Write from/to your Marketing Systems
  • Perform Read/Write from/to your Data Warehousing systems.
• For operations that need to be handled by system(s) that is inaccessible via (REST/SOAP) APIs you could also leverage the ETL methodology:
  • Create XML/CSV data file -> FTP file to remote server -> Kick off cron job to process file dropped -> Notify user(s) of processing success/failure)

A few salient thoughts to consider:

• To address client-specific requirements, you may need to build custom API's and/or tweak existing OOTB Magento REST APIs.
• You're going to be better of identifying an MVP solution and then iterating on it v/s building an exhaustive-and-complex-heavy-on-features integration between the different systems.
• Try to use M2 OOTB APIs whenever/wherever possible.
• M2 supports OAuth and Bearer tokens for Authorization/Authentication API calls.

In case of any additional questions/concerns - don't hesitate to send me a DM! :)

Answer 2

In different scenarios both options can be used, but it is still recommended to use option 1. Using option 1, you can create a Magento 2 module and create libraries which can handle communication with the external services and use them in that module.

Mage provides different events for ordering and payments, which you can use in your module and create Observers for those events to utilize your libs for external services.

An outside implementation for the service will be hard to manage, and still it will be fully dependent on Mage. You have mentioned in your points, scalability will be also an issue, but I dont think so it will be a problem, because, number of orders are not always as high as traffic to your Mage.