The heart of Duetto’s application is our server logic authored in Java. In this post we’ll describe some of the philosophies, tools, techniques and patterns that have proven effective in delivering high-quality innovation.
[Editor’s note: This is the third entry in a three-part “Building a Better RMS” series, in which Duetto CTO Craig Weissman explains the finer points of the cloud-based architecture that powers Duetto and its GameChanger application. Read Craig’s first post here, about why Duetto runs GameChanger exclusively on Amazon Web Services, and his second post here, about how MongoDB databases help the company scale rapidly.]
Duetto’s multi-tenancy is based on both a shared-data model as well as a shared set of stateless Java application servers that operate in parallel for all of our customers. The primary workloads for these server processes include:
- Ingesting all our customer data on the “back end” as well as sending outbound messages. Duetto’s application lives within an ecosystem of many other hospitality services and products and must interact in both directions with many of these systems. This includes receiving large ETL text files and inbound API calls, as well as pushing our rates and restrictions in a reliable manner to hundreds of external web services running on-premise at our customers or elsewhere in the cloud.
- Background processing in support of the above. We extensively use a scheduler and queueing system to perform periodic and reliable work for both inbound and outbound data. Our pricing algorithm itself runs in the background, as do many other housekeeping tasks.
Each of our deployed servers run all of our code and each server is able to handle all of these workloads. This allows for easy scale up and scale down along with our live zero-downtime deployment process. In practice, we slowly add new app servers to our pools because our code efficiency allows us to run the universe of Duetto customers on a small hardware footprint.
Foreground vs. Background work
In terms of load balancing our public application address, we use AWS ELB to route traffic from our end user application as well as inbound API requests to our back-end servers. During our deployment process we remove and add servers to these pools to orchestrate a release and code rollout.
We could decide to separate the front-end application (human user interface) from our inbound API endpoints, but so far we are using the same load balancer and server pools.
One thing to keep in mind with ELB is that requests must complete in less than 60 seconds (this is now a default value, but still a reasonable one). Therefore any slow work that will require anywhere near this amount of time we tend to push into the background so that the foreground experience will be snappy. HTTP requests shouldn’t live a long time so this encourages proper discipline for synchronous processes versus background ones, which can take longer.
As for our integrations and background processing, we use Quartz within our Java code and all app servers run tasks that poll for work. We increasingly rely on message queues for reliable processing of tasks with retry and locking semantics. The queues themselves are currently implemented as Mongo collections, and we use distributed locks (also implemented with Mongo as the backing store) to coordinate critical tasks to avoid race conditions.
If a third-party system does not respond successfully to an outbound request from us, we will try again for a reasonable period of time. We also encourage our partner vendors to move to an asynchronous message flow whenever possible for receiving messages such as rate changes. These best practices increase scale and reliability in our complex ecosystem.
Leveraging the best Java Technologies
In terms of basic infrastructure, Duetto uses modern open-source Java technologies including Oracle JDK8 and Google’s guava library (more on this soon and functional programming), the Jetty application server container, Spring and Jackson for our MVC layer and dependency injection, and Quartz for scheduling as mentioned above.
The Java language itself has evolved and stayed relevant for more than 20 years by incorporating new best practices, and JDK8 is no exception. The introduction of Lambda expressions similar to Scala and other functional languages is a major syntactic and semantic improvement. Google’s Guava library has led the way on such changes and continues to extend the Java language in ways that are later incorporated into the JDK.
Duetto’s query engine in particular has become a heavy user of Java8’s Lambda syntax. Duetto maintains a robust set of internal coding guidelines, and we spend considerable time training new developers on these practices as well as our development processes.
Jackson is used in support of this for JSON serialization and deserialization.
However, over time we have more deeply embraced the power of Dependency Injection (DI) and component management in Spring to write more modular and testable code.
Our engineers have broken out many internal “services” for separate tasks such as reporting, pricing, and room type optimization. These services “inject” only the low-level components they need, and in turn these services are wired into only those high-level controllers that need access to those services. Spring manages the wiring of these components at runtime using reflection — it often appears “magical” but leads to a lot less explicit wiring of code and greater flexibility.
The smaller resulting services are more easily mocked and tested by our developers, leading to robust and scalable development.
Open-Source Infrastructure, Custom Code
As mentioned above, our application makes extensive use of background processing including queue management for reliable execution of tasks such as outbound rate pushes and periodic auto-optimization of our hotels.
Quartz is a robust open-source library for scheduling and repeated job execution. Quartz works well as a Spring managed component. Some of our tasks run as singletons (only one at a time system wide) whereas others are running in parallel across our cluster of application servers.
The overall theme here is that Duetto combines open-source machinery with custom code to arrive at the correct infrastructure for our use cases.
Duetto’s end user application is a modern HTML5 browser-based single-page app which interacts with our server Spring MVC endpoints. In a future post we’ll describe our core UI rendering technologies, including a move towards Facebook’s React framework.
Duetto recently delivered its first iOS application as part of our Strategy Edition for loyalty program management. This application is rendered largely with the same HTML5 compatible MVC code on the back end.
It is quite possible that Duetto’s application servers will be used to render other runtime components in the future — for instance, a Duetto-powered Open Pricing booking engine.
Duetto’s technology organization and tech stack have generally evolved and grown up incrementally from our starting choices more than four years ago. But as this “Building a Better RMS” series has shown, our foundation of AWS for hosting, MongoDB for Big Data management, and robust server-side Java coding for continually improving the application gives us an ideal base upon which Duetto can scale over the long run.