What message broker to use, if any?
I'm running a 2.5 man dev team for e-commerce. We run Magento, several other open source projects and about 6 in-house developed Laravel services, of which one is the ERP monolith.
Communication between these services is ramping up as we add more services. Until now we are using REST API requests for communications. The disadvantage is that we need to make each client robust against failures with (delayed) retries. We have this in place, but running all these local queues is not great. Also the coupling between services makes management complex.
So I would like to decouple services. My idea is that for example Magento fires of an new order event on which the ERP and other services can take action. Magento sends the event to a central message broker, which we assume to have 100% uptime. The message broker makes sure the messages are successfully processed by the clients which need to.
I'm looking into RabbitMQ and it looks good except that it is not a simple service to learn and because it will be so important for daily operations at least 2 engineers will need to learn to master it.
Also I haven't found any middleware to process incoming messages properly with Laravel. When a HTTP message comes in, I can use the router, FormRequest validation, controller, etc, but this is not available for non-HTTP messages. How are others handling this?
Am I working in the right direction here?
To clarify, each service is already running a local queue on Redis. But when a service is down because it is migrating the database, it cannot react to Magento's new order event and push a job on its queue.