Tuesday, 4th Feb 2025
A new way to manage structured data on AWS: S3 Tables
A couple of announcements from AWS Re:Invent, which might on the face of it seem fairly small, are *S3 Tables* and *S3 Metadata*.
S3 Tables are a new type of object storage specifically for tabular data. They use Apache Iceberg underneath, and can give much faster query times when using services such as Athena to query data using SQL. And, despite appearances, this is actually a fairly exciting update. For a long time now, it's been possible to build data lakes using S3 as the storage layer. But with Apache Iceberg support, it's now possible to create something more akin to a lakehouse architecture.
This means that you can build fully ACID-compliant transactional databases, similar to what you might get with a traditional data warehouse such as Redshift. This lakehouse architecture gives you the flexibility of a data lake, whilst being able to store and query more structured data. It will be perfect for more performant analytics and business intelligence applications. Especially when paired with something like QuickSight. Glue Data Catalogues can be used to catalogue and publish S3 tables.
The S3 metadata service uses S3 Tables functionality to enable the management of fine-grained metadata for traditional S3 objects. This allows you to query object metadata at scale, making discovery and advanced searches much, much easier for objects stored in S3. I'll be writing more about S3 metadata in a future post.
I'm sure we'll see a lot more services start to make use of this metadata and ability to store structured data. This is particularly exciting in terms of data governance.
It’s worth noting that right now, S3 Tables are still in preview. It's possible for anybody to try this by heading to the S3 console and enabling the service. You'll have to use the CLI or SDK to interact with some features of the service, though.
Look out for more exciting updates in our next AWS Future-Ready Features blog.
