It's well known that only two man-made objects are visible from space: the Great Wall of China and the Amadeus Revenue Accounting Database. It's an urban legend: you cannot see the Great Wall of China from space.
One thing you can be sure: keeping track of all Financial Transactions of Airline Passenger Operations will give you a huge database. And with large database comes large headaches.
Bringing the size of our database back to sanity meant: moving older documents from the hot (and expensive) Oracle to the cold (and cheap) S3 Blob Storage and completely changing how we compress our data (goodbye GZIP, hello VCDIFF!). With a little help from the GDPR. All with the help of C++, Oracle, S3 and a dust of Python.
All this while keeping data available (with a caveat), while our clients were rightfully asking "What have you done with my data".
A journey where we learnt some hard lessons, like having layers in your software libraries is great, but having those layers in the wrong order not so much. And that you should not worry about wasting storage space when it's cheap. A journey with a happy end. A journey that will be for you will be much shorter if you come to learn from our successes and our mistakes.
Because it's never too early, and never too late to streamline your database. For safety and cost.