I have built a decent number of API’s and integrations over the course of my career. Before joining Duetto to help develop its hotel technology, I was a lead engineer at Salesforce, where I built everything from dynamic DNS updates to RESTful implementations. The development always followed a similar pattern:
- Get the message exchange working: Can I send a request and get a response? Great.
- Make it functional: Work through test cases and make sure we’re handling the flow of data correctly.
- Get it to scale: Does it perform? How many concurrent requests can it handle? Is the system protected from a spike of requests?
- Error handling: Can we gracefully handle bad data or intermittent network failures? Do we have good error messages?
At this point, I’m fairly confident the integration works. Sure, we’ll discover some edge cases in production, but we’re not going to be that far off the mark, right? After all, an integration is a partnership between two technology providers making sure they can correctly exchange data.
Evolving Hotel Technology Integrations
Today I’m at Duetto, where we provide Revenue Strategy technology to hotels and casinos. Our core philosophy is Open Pricing, which allows us to independently yield hotel room rates based on a variety of dimensions, including segments, room types and distribution channels. Our segmentation engine provides a flexible way to use all of the different attributes on bookings to group guests into buckets which can then be forecasted, priced and reported on independently. Pricing rules can be created and modified on the fly, and hotels can see the changes instantaneously.
None of this would be possible with summary-level data (think room counts by day instead of individual bookings), which is why our application relies so heavily on having an accurate and detailed view of all of a hotel’s bookings. Naturally then, the most important API message to us is the Booking message — specifically, any time a booking is made or modified at the hotel, we need to know about it.
About three years ago, we started building our first integrations for receiving bookings. For the most part, they followed a standard SOAP specification called HTNG, which defines the shape of the SOAP message body. Within a few weeks, I had built the appropriate code to handle this message with a little async notification handler for performance. We run through a set of validation tests with the property management system (PMS), a process to make sure it was working correctly. Create a booking, modify a booking, cancel a booking, send some bogus data and check we’re handling errors — OK great, it all works, I thought. Although this was our first integration, I felt somewhat confident that things would work just fine. The PMS provider we were working with had been in the industry for decades — if we’ve covered all of their test cases as well as our own, how far off could we be?
However, soon after we went live with our first customer, we got reports of data inconsistencies — specifically the number of bookings didn’t match the number in the PMS. I scoured our logs for XMLs, expecting to find evidence of exceptions being thrown or network blips. I didn’t find any. Nor would random network blips really contribute to being off by double-digit percentages. Something else was really wrong.
At the same time as the rollout and investigation of this first integration was going out, we were expanding the number of partners we connect with. Since many of them used the same HTNG specification, I was able to reuse pretty much all of the code, expecting the API to behave the same. Our basic connectivity and test cases passed, so it was a green light to start rolling this out. We hadn’t yet fully understood why the first integration was not going well.
Of course, we also had problems with our second integration deployment. And again, there didn’t seem to be any technical reason for why the numbers didn’t reconcile.
So what was the problem in both cases? There were many, but they all followed a similar theme. We didn’t really think about all of the different actions, operations and features available to the PMS. I never expected that I could receive bookings for specific room types that should be multiplied because they’re for suites that are made up of smaller rooms. Or that two bookings with different IDs should count as one because the guests decided to share the room but pay separately. As an engineer, you make some basic assumptions — such as a unique ID identifies a unique entity. But that was not really the case here! My whole world was being shattered.
A tad dramatic probably, but you get the point. I didn’t even know these were questions to ask, as they seemed so fundamental, nor did my more experienced partners. Once we learned about them, I had to go back and figure out how to identify these bookings. Once identified, we needed to actually handle them correctly. And after handling them, we had to fix all of the existing data that was already loaded.
And although the various systems we connect to adhere to the standard HTNG specification, they all handle these scenarios differently. Because although the specification defines the attributes of the message, it doesn’t really go into the semantics of how to communicate different business scenarios. As a result, it’s up to each technology provider to decide which attributes it will use to communicate each operation, and it turns out, each technology provider did have its own implementation flavor. Given that, it was somewhat surprising that it wasn’t part of their standard test cases. As we soon learned, this was a pervasive problem in the industry.
Where We Are Today
It has been three years since we’ve built our first integrations in the hospitality industry. We now have a world-class process for developing, validating and piloting new integrations, thanks to our Head of Product for Integrations, and a new Partner Manager role that combines expertise in the industry with strong analytical, technical and relationship management skills. The number of test cases we execute while validating a new integration has ballooned, and even before any development happens, we have conversations with our partners to learn how their system works and share how ours does.
This has resulted in a solid, yet extremely flexible architecture where we can add new integrations with minimal amount of code while still being able to handle the various business rules that each technology provider seems to handle slightly differently. Although this ends up taking a little longer during our discovery and testing phase, we end up with a higher-quality integration that reflects the true actuals of a hotel, so that customers using our product can trust both our pricing recommendations and reporting.
We have also built extensive tooling, such as indexing every message that comes in by booking ID and an automated validation system that reports on the days that are different from the PMS given a source of truth file. This has allowed us to move to a model where we don’t need to wait for customers to report discrepancies to us — we know about them before they do and can tackle them head on. As a result, our deployments now are smoother and faster and we encounter vastly fewer issues.
Building integrations for the hospitality industry challenged many of the core fundamentals I had learned in my previous experiences. The real journey for me has been thinking beyond the technical aspects of the integration. Sending and receiving a SOAP message is relatively easy. Even scaling and making it perform is a mostly solved problem. But interpreting the message and understanding the semantics of what’s being communicated and what operation triggered this message in the source system, that’s a different beast altogether.
The systems that we’re integrating with and the message standards we’re using to communicate with were not built with data warehousing in mind. And that’s not surprising, as there wasn’t much of a standard in analytics and reporting in this industry. Until now. We’re not just innovating from a technical and Revenue Strategy standpoint, but we’re blazing new ground in the way technology providers build, test and deploy new integrations in this industry — an industry with thousands of disparate systems running all sorts of different software, using different protocols varying from decades-old to just a couple years old, all trying to exchange the same data.
It takes experience to learn how these things work, but more importantly, also a top-notch team to make a process out of it to constantly validate, improve and make these integrations robust. Luckily, we have that here at Duetto. We are now an integration machine.