Working with Parquet: From Browser Inspection to Production Pipelines

Apache Parquet has become the de facto standard for analytical data storage. It's used across data lakes, warehouses, and streaming pipelines — from Spark and Flink to DuckDB and Polars. But most engineers treat it as a black box: write data in, read data out. Understanding the internals unlocks better schema design, faster queries, and more efficient storage. This guide goes from format internals to browser-based inspection to production-grade partitioning.

Want to explore a Parquet file right now? Open the Parquet Inspector — it reads files entirely in your browser using DuckDB WASM.

1. Parquet Format Internals

Parquet is a columnar storage format. Unlike row-oriented formats (CSV, JSON, Avro), Parquet stores all values for a single column together. This layout enables three key advantages: better compression (similar values cluster), predicate pushdown (skip entire column chunks), and projection pushdown (read only the columns you need).

File structure: the four levels

A Parquet file is organized in a hierarchy:

┌─────────────────────────────────────┐
│           Parquet File              │
├─────────────────────────────────────┤
│  Row Group 0                        │
│  ├── Column Chunk: id              │
│  │   ├── Page 0 (data page)        │
│  │   └── Page 1 (data page)        │
│  ├── Column Chunk: name            │
│  │   └── Page 0 (data page)        │
│  └── Column Chunk: amount          │
│      └── Page 0 (data page)        │
├─────────────────────────────────────┤
│  Row Group 1                        │
│  ├── Column Chunk: id              │
│  ├── Column Chunk: name            │
│  └── Column Chunk: amount          │
├─────────────────────────────────────┤
│  Footer (schema + metadata)         │
│  Footer Length (4 bytes)            │
│  Magic: PAR1 (4 bytes)             │
└─────────────────────────────────────┘

• Row Group — a horizontal partition of the data, typically 128 MB (Spark default) or 512 MB–1 GB in optimized configs. Each row group contains all columns for a subset of rows.
• Column Chunk — all data for one column within one row group. Column chunks can be independently compressed and encoded.
• Page — the smallest unit of I/O. Each column chunk is divided into pages (default 1 MB). Pages carry the actual encoded values, repetition levels, and definition levels.
• Footer — stored at the end of the file. Contains the schema (column names, types, nesting), row group offsets, column statistics (min/max, null counts), and key-value metadata. Readers parse the footer first, then seek directly to needed column chunks.

Encoding schemes

Parquet doesn't just store raw values — it encodes them for compression efficiency:

Modern Parquet writers (Spark 3.x, Arrow, DuckDB) automatically select the best encoding per column chunk based on data characteristics. Dictionary encoding is tried first; if the dictionary exceeds a threshold (typically 1 MB), the writer falls back to plain encoding for that column chunk.

Compression codecs

Applied on top of encoding at the page level. The most common options:

• Snappy — fast compression/decompression, moderate ratio. Default in Spark.
• ZSTD — better compression ratio than Snappy with comparable decompression speed. Recommended for storage-bound workloads.
• Gzip — highest compression ratio, but slowest. Good for cold archives.
• LZ4 — fastest decompression, lowest ratio. Best for latency-sensitive reads.

2. Inspecting Parquet in the Browser with DuckDB WASM

Traditionally, inspecting a Parquet file required Python (pyarrow) or a Spark shell. DuckDB's WebAssembly build changed that — you can now query Parquet files entirely in the browser with zero server interaction.

How the DataDevKIT Parquet Inspector works

The Parquet Inspector loads DuckDB WASM, registers your dropped file as a virtual table, and runs SQL against it. Here's the core pattern:

import * as duckdb from '@duckdb/duckdb-wasm';

// Initialize DuckDB in the browser
const DUCKDB_BUNDLES = duckdb.getJsDelivrBundles();
const bundle = await duckdb.selectBundle(DUCKDB_BUNDLES);
const worker = new Worker(bundle.mainWorker);
const logger = new duckdb.ConsoleLogger();
const db = new duckdb.AsyncDuckDB(logger, worker);
await db.instantiate(bundle.mainModule, bundle.pthreadWorker);

const conn = await db.connect();

// Register a local file (from drag-and-drop or file input)
await db.registerFileBuffer('data.parquet', new Uint8Array(fileBuffer));

// Query schema
const schema = await conn.query(`
  SELECT column_name, column_type, null_percentage
  FROM parquet_schema('data.parquet')
`);

// Query metadata (row groups, sizes)
const meta = await conn.query(`
  SELECT * FROM parquet_metadata('data.parquet')
`);

// Preview data
const preview = await conn.query(`
  SELECT * FROM 'data.parquet' LIMIT 100
`);

Key DuckDB Parquet functions

DuckDB exposes several functions specifically for Parquet introspection:

-- Schema: column names, types, and structure
SELECT * FROM parquet_schema('data.parquet');

-- Metadata: row groups, column chunk offsets, sizes, encodings
SELECT * FROM parquet_metadata('data.parquet');

-- File-level stats: row count, row group count, created_by
SELECT * FROM parquet_file_metadata('data.parquet');

-- Column statistics: min/max values, null counts per row group
SELECT row_group_id, column_id, stats_min, stats_max, stats_null_count
FROM parquet_metadata('data.parquet');

What to look for during inspection

1. Row group sizes — are they balanced? Highly unbalanced row groups (e.g., one with 10K rows and another with 10M) indicate a compaction problem.
2. Column encodings — check if high-cardinality columns fell back from dictionary to plain encoding. If a low-cardinality column is using plain encoding, the writer configuration may need tuning.
3. Null percentages — columns with 90%+ nulls may be candidates for restructuring or removal.
4. Statistics availability — if min/max stats are missing, predicate pushdown won't work. This happens with some older writers or when column types don't support statistics.

3. Schema Design for Parquet

Good schema design has a massive impact on query performance and storage efficiency.

Flat vs. nested schemas

Parquet supports nested types via its Dremel encoding (repetition and definition levels). While you can store deeply nested JSON-like structures, flat schemas are strongly preferred:

• Query engines optimize flat schemas more aggressively
• Predicate pushdown works on top-level columns only
• Statistics (min/max) are collected per column chunk — nested fields don't get their own statistics
• Most BI tools expect flat tabular data

Type selection matters

Choosing the right Parquet logical type affects both storage size and query semantics:

# PyArrow schema definition — be explicit about types
import pyarrow as pa

schema = pa.schema([
    pa.field('id', pa.int64()),           # Not string — enables delta encoding
    pa.field('amount', pa.decimal128(18, 2)),  # Not float64 — exact decimals
    pa.field('event_date', pa.date32()),   # Not string — 4 bytes vs ~10 bytes
    pa.field('created_at', pa.timestamp('us')),  # Microsecond precision
    pa.field('country', pa.dictionary(pa.int16(), pa.string())),  # Force dictionary encoding
    pa.field('is_active', pa.bool_()),     # 1 bit per value with bit-packing
])

4. Apache Spark Integration

Spark is the most common engine for reading and writing Parquet at scale.

Reading Parquet

from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .config("spark.sql.parquet.enableVectorizedReader", "true") \
    .config("spark.sql.parquet.filterPushdown", "true") \
    .getOrCreate()

# Read with predicate pushdown — Spark skips row groups
# where stats_max < 1000 or stats_min > 5000
df = spark.read.parquet("s3://bucket/events/") \
    .filter("amount BETWEEN 1000 AND 5000") \
    .select("id", "amount", "event_date")

# Only the id, amount, and event_date column chunks are read (projection pushdown)
# Only row groups where amount range overlaps [1000, 5000] are read (predicate pushdown)

Writing optimized Parquet

# Key Spark configurations for Parquet output quality
spark.conf.set("spark.sql.parquet.compression.codec", "zstd")
spark.conf.set("spark.sql.parquet.writeLegacyFormat", "false")  # Use modern INT96-free format
spark.conf.set("spark.sql.files.maxRecordsPerFile", 1000000)   # Cap file size

df.repartition(20) \
  .sortWithinPartitions("event_date") \
  .write \
  .mode("overwrite") \
  .parquet("s3://bucket/events-optimized/")

Why sortWithinPartitions? Sorting data within each output file by a frequently-filtered column (like date) dramatically improves row group statistics. When values are sorted, each row group's min/max range is narrow, so predicate pushdown can skip more row groups.

Adaptive query execution

Spark 3.x AQE dynamically coalesces small partitions and optimizes joins at runtime. Combined with Parquet column statistics, this means:

• Small files are merged during reads (reduces task overhead)
• Skew joins are automatically rebalanced
• Partition filters are pushed down to the Parquet scan

# Enable AQE (default in Spark 3.2+)
spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "true")
spark.conf.set("spark.sql.adaptive.skewJoin.enabled", "true")

5. Partitioning Strategies

Partitioning is how you organize Parquet files on disk to enable partition pruning — the ability to skip entire directories of data during queries.

Hive-style partitioning

The most common pattern. Data is stored in directories named column=value:

s3://bucket/events/
  year=2025/month=01/part-00000.parquet
  year=2025/month=01/part-00001.parquet
  year=2025/month=02/part-00000.parquet
  year=2026/month=01/part-00000.parquet

# Write with Hive partitioning in Spark
df.write \
  .partitionBy("year", "month") \
  .mode("overwrite") \
  .parquet("s3://bucket/events/")

# Queries with partition filters skip unneeded directories
spark.read.parquet("s3://bucket/events/") \
  .filter("year = 2026 AND month = 1")
# Only reads s3://bucket/events/year=2026/month=01/ — everything else is pruned

Choosing partition columns

The right partition key has these properties:

1. Low to moderate cardinality — 10–10,000 distinct values. Too many partitions (e.g., partitioning by user_id with 50M users) creates millions of tiny files (the "small file problem").
2. Frequently filtered — choose columns that appear in WHERE clauses of most queries.
3. Time-based columns are usually best — date, year/month, year/month/day.

File compaction

Over time, append-mode writes create many small files within partitions. This degrades read performance because each file adds task scheduling overhead. Compaction merges small files into larger ones:

# Compaction: read small files, write fewer large files
partition_path = "s3://bucket/events/year=2026/month=01/"
df_partition = spark.read.parquet(partition_path)

df_partition.coalesce(4) \
  .write \
  .mode("overwrite") \
  .parquet(partition_path)

# Target: 128 MB – 1 GB per file for optimal read performance

Delta Lake, Apache Iceberg, and Apache Hudi all include automatic compaction as a table maintenance operation — highly recommended over manual compaction scripts.

Z-Order / Hilbert clustering

For queries that filter on multiple columns simultaneously, Hive partitioning on one column isn't enough. Z-ordering (available in Delta Lake and Iceberg) interleaves the sort order of multiple columns to improve data skipping:

# Delta Lake Z-Order (Databricks or open-source Delta)
spark.sql("""
  OPTIMIZE events
  ZORDER BY (customer_id, event_date)
""")
-- Queries filtering on customer_id AND/OR event_date
-- will skip more row groups than single-column sorting

6. Browser to Production: The Workflow

Here's a practical workflow that combines browser-based inspection with production pipeline design:

1. Inspect — Drop the Parquet file into the Parquet Inspector. Check schema types, row group sizes, null percentages, and statistics.
2. Validate — Verify column types match expectations. Flag high-null columns and unexpected encodings.
3. Design schema — Based on the inspection, define an explicit schema in PyArrow or Spark StructType. Choose partition columns.
4. Write pipeline — Implement the Spark/DuckDB pipeline with proper compression, sort order, and partitioning.
5. Monitor — Periodically re-inspect output files. Check for row group balance, file sizes, and statistics health.

Key Takeaways

• Parquet organizes data as row groups → column chunks → pages. Understanding this hierarchy is key to performance tuning.
• DuckDB WASM makes Parquet inspection possible in the browser — no Python or Spark install needed. Try the Parquet Inspector.
• Sort data within files by your most-filtered column to maximize predicate pushdown effectiveness.
• Partition by low-to-moderate cardinality columns (usually date-based). Avoid high-cardinality partition keys.
• Use ZSTD compression over Snappy when storage cost matters more than write speed.
• Compact small files regularly — or use a table format (Delta Lake, Iceberg) that does it automatically.