Enhanced Management Features in Amazon's AWS Lake Formation: A Dive Into Advanced Configurability

Enhanced Management Features in Amazon’s AWS Lake Formation: A Dive Into Advanced Configurability

Amazon Web Services on Tuesday announced the general availability of tools that bring more fine-grained control to data lake management .

First, Row and Cell-Level Security for Lake Formation “puts the right data in the hands of the right people,” AWS CEO Adam Selipsky said during his AWS re:Invent keynote address.

Lake Formation already enables customers to move data into S3 data lakes, clean, and classify it using machine learning and secure access to sensitive data. The new tool now lets customers enforce access controls for individual rows and cells. Instead of creating multiple tables for each user and managing data pipelines, a customer can define a set of policies for specific rows for specific users. Customers control access to specific rows and columns in query results and within AWS Glue ETL jobs based on the identity of who is performing the action.

Meanwhile, Transactions for Governed Tables in Lake Formation eliminates the need for batching updates. “Data isn’t static,” Selipsky said. “More and more data is being added and moved rapidly.”

Now customers can create a new type of table – a governed table – and Lake Formation automatically manages conflicts and errors for consistent view of data. Users will be able to keep up with the data in real-time.

Governed tables support ACID transactions that let multiple users concurrently and reliably insert and delete data across multiple governed tables. ACID transactions also let customers run queries that return consistent and up-to-date data. In case of errors in ETL processes, or during an update, changes are not committed and will not be visible.

Customers using governed tables can use automatic compaction for storage optimization. When this option is enabled, Lake Formation automatically compacts small S3 objects in governed tables into larger objects to optimize access via analytics engines, such asAmazon Athena andAmazon Redshift Spectrum .