I am using the "data lake" concept without realizing it. It just seemed like the right thing to do.
I am working on a "Google for SEC filings". There are about 15 million company filings and other data spanning 20 years available on sec.gov. The data is about 700 GB compressed, but unfortunately you have to download each filing individually from their FTP server. When I first started, I wrote a script that would download the filing and then process it into the format I wanted. However their FTP server is very slow and there are 15 million individual downloads, so it was taking forever. Rather I wrote a script that mirrored the FTP server to S3 as fast as possible while still being respectful to their bandwidth and server capacity. And this still took almost 3 weeks.
Now I have a "data lake" of raw SEC filings and other data which I can pull from at any time on S3. And the important part is the performance is significantly better so the processing time is relatively small.
OT but funny enough I worked with an accounting PhD student to extract SEC filings and mine for some keywords and associated numbers and tables, from this same FTP and I remember it being so dog-slow. This was like 8 years ago too, sad its in the same shape.
I am working on a "Google for SEC filings". There are about 15 million company filings and other data spanning 20 years available on sec.gov. The data is about 700 GB compressed, but unfortunately you have to download each filing individually from their FTP server. When I first started, I wrote a script that would download the filing and then process it into the format I wanted. However their FTP server is very slow and there are 15 million individual downloads, so it was taking forever. Rather I wrote a script that mirrored the FTP server to S3 as fast as possible while still being respectful to their bandwidth and server capacity. And this still took almost 3 weeks.
Now I have a "data lake" of raw SEC filings and other data which I can pull from at any time on S3. And the important part is the performance is significantly better so the processing time is relatively small.