|Athena Query Alterer Open Sourced|
|Written by Alex Denham|
|Tuesday, 12 February 2019|
A tool that alerts you if users are running expensive queries on Amazon Athena query engine has been open sourced by the developers.
The Athena Alerter was developed by a team at Fandom Engineering, and is designed to overcome the problem that the AWS Athena big data query engine is easy to use, but can also easily run up large bills as a consequence of its simplicity.
It is possible to write less expensive queries by using partitions to limit the amount of data being accessed, but that requires the query writer to remember to use a partition and to understand how expensive a question could be. This is particularly the case when working in an external tool.
Because of the potential to spend money on Athena so quickly, the development team at Fandom Engineering developed Athena Alerter. This is an open source set of lambda functions designed to work together to track which queries are run, how much data do they scan (which directly maps to costs) and notify users when they run costly queries.
Because Fandom Engineering uses Slack for internal communication, Athena Alerter notifies users by sending Slack messages. However the developers point out that given the very modular nature of the tool, it’s easy to adjust the notification function to use a different mechanism.
Internally, the alerting tool uses Cloudtrail, Lambda, DynamoDB, SQS, and S3, and is set up using a CloudFormation script which will create all the AWS components that are needed for you. The user needs to provide their specific configuration and then use the provided makefile.
To process the information about Athena queries, the tool first processes Cloudtrail logs to learn who started which query, then uses the Athena API to track the query and get information about the amount of data scanned. This informatin is then pushed to DynamoDB and SQS, and users are notified.
At its heart the tool consist of three lambda functions:
Athena Alterer is available on Github.
or email your comment to: firstname.lastname@example.org
|Last Updated ( Tuesday, 12 February 2019 )|