Sign in Subscribe

Apache Spark

Building ScriptScope Part 1: Extracting Top Used JS Libraries from Common Crawl using Apache Spark

Building ScriptScope Part 1: Extracting Top Used JS Libraries from Common Crawl using Apache Spark

Learn how you can build website technology analysis tools like builtWith and Wappalyzer! In this blog, we identify the top-used JS libraries for 2024!

BigBanyanTree: Parsing HTML source code with Apache Spark & Selectolax

BigBanyanTree: Parsing HTML source code with Apache Spark & Selectolax

Dive into the world of data extraction! Learn how to parse HTML source code from Common Crawl WARC files with Apache Spark and Selectolax for insightful analysis and unlock the potential of HTML source code.

Zero to Spark - BigBanyanTree Cluster Setup

Zero to Spark - BigBanyanTree Cluster Setup

This blog details the setup of an Apache Spark cluster in standalone mode for data engineering using Docker compose on a dedicated Hetzner server. It also covers the setup of other utilities such as Jupyterlab and Llama-3.1 8B LLM service.