DuckDB to the Rescue
I have been working with databases generally for over 2 decades. First Oracle (that was my main bread-n-butter), but also MySQL, Teradata, Redshift, Netezza and others.
In my most recent role, I actively support Postgres, SQLServer and Snowflake (I am Snowflake certified). We work exclusively in an AWS environment.
While I *like* Snowflake, upon using it the first time over a year ago I began to perceive it could become expensive. Since then, I've been seeking a way to essentially replace something like Snowflake for e.g. a large database or so-called data pipeline.
Lately, I'm convinced that DuckDB will be part of that equation. For inline data processing, it performs well in tests, is written in C, has a small memory and CPU footprint and has excellent SQL support including reading external data and support for parsing JSON.
Given JSON is so heavily used for messaging and data, having an inline database that can load and make available with a SQL API to database developer and data engineers seems pretty powerful.
DuckDB also serves as an amazing desktop tool to investigate data including CSV.
If you are data professional of any sort I would recommend giving DuckDB a try and making it part of your toolbelt as common as MS Excel.
Author: Marcus
Post Date: 2023-06-19
By Marcus