← Back to projects

json2csv

a JSON file walks in, a relational schema walks out.

Built for a compiler-construction course, json2csv reads a JSON file up to 30 MiB and turns it into a set of relational CSV tables. It's the kind of normalization an ETL pipeline usually does by hand, done here by an actual lexer and parser instead of a library call.

source

the pipeline

  1. lexing

    A Flex-generated scanner tokenizes the input and tracks line and column, so a lexical error points at the exact character.

  2. parsing

    A Bison grammar consumes the token stream and builds an abstract syntax tree, the same grammar-driven approach a real compiler front end uses.

  3. AST construction

    Plain C structs, plus GLib's GHashTable and GList, hold the tree in memory once parsing succeeds.

  4. schema inference

    Every object gets a signature from its sorted scalar keys. Two objects with the same signature land in the same table; a new signature spawns a new one.

  5. relational unflattening

    Nested objects and object arrays become child tables carrying a parent_id foreign key. Arrays of scalars become junction tables of id, parent id, index, and value.

  6. CSV emission

    One CSV per inferred table, headers included, values quoted and escaped, nulls rendered as an empty field.

grouping objects by shape

This is the part I'm most pleased with. Every object's scalar keys get sorted into a signature string; a hash table maps that signature straight to its table. Two JSON objects that were never declared to share a type end up in the same CSV file simply because they happen to have the same keys.

char* current_object_key_signature = g_list_to_sorted_csv_string(scalar_keys_list);

TableInfo* ti = (TableInfo*)g_hash_table_lookup(
    ctx->tables_by_signature, current_object_key_signature);

if (!ti) {
    ti = (TableInfo*)safe_malloc(sizeof(TableInfo));
    ti->actual_table_name = generate_unique_table_name(ctx, name_hint_for_object);
    ti->key_signature = safe_strdup(current_object_key_signature);
    /* ...register the new table under this signature... */
}
csv_processor.c, process_object_node

C, Flex, Bison, GLib, Make