Understanding the Git Workflow: Working Directory, Staging, Commit, and Push.

Introduction

Data science projects evolve through iterative experimentation—adjusting features, tuning models, and validating results. Without a disciplined way to track changes, reproducing a specific experiment becomes guesswork. Git provides the foundation for versioning not just source code, but also the artifacts that define a data science workflow: notebooks, scripts, configuration files, and even lightweight model metadata. This article walks through the four core states of Git—working directory, staging area, commit, and push—tailored to the realities of a data science team.

Why This Matters

In production machine learning pipelines, a single untracked change can invalidate a model’s performance claim. When multiple analysts share a repository, overlapping edits cause merge conflicts that waste time and erode trust in results. By mastering Git’s workflow, data scientists gain:

• Reproducibility: Every commit captures a snapshot of the exact code and data versions used for a given result.
• Collaboration: Branches and pull requests let teammates review feature engineering or model architecture changes before they land in main.
• Auditability: Regulatory or internal reviews can trace a model’s lineage back to the originating commit.

Skipping any stage—say, committing directly from the working directory without reviewing what’s staged—introduces risk. The following sections break down each stage, show how to automate safety checks, and illustrate patterns that keep a data science repository clean and reliable.

How It Works

Git’s workflow moves changes through four distinct areas. The diagram below captures the flow as it appears in a typical data science repository, highlighting where validation hooks and CI pipelines intervene.

flowchart TD
    %% Nodes
    WS[Working Directory] -->|Edit notebooks, scripts, configs| A[Staging Index]
    A -->|git add -p or add specific files| B[Local Repository]
    B -->|git commit -m "msg"| C[Commit History]
    C -->|git push| D{Remote Validation}
    D -->|Passes schema & metric checks| E[Remote Origin]
    D -->|Fails validation| WS
    E -->|Pull/merge| F[Collaborative Lab]
    F -->|Fetch updates| WS
    %% Side notes
    WS -->|Large data, logs, env| G[.gitignore / LFS]
    G -.->|Exclude from staging| A
    style D fill:#f9f,stroke:#333,stroke-width:2px

Step‑by‑step explanation

1. Working Directory – This is where you edit Jupyter notebooks (*.ipynb), Python modules, YAML configs, and shell scripts. Untracked files appear here after git status.
2. Staging Index – Acting as a proposal buffer, the index lets you select exactly which changes belong in the next commit. Using git add -p (patch mode) or git add <specific‑file> prevents accidental inclusion of large data dumps or temporary logs.
3. Commit – Once the index reflects the desired snapshot, git commit creates an immutable object in the local object database. Each commit points to a tree of blobs (file contents) and stores author, timestamp, and a commit message.
4. Push – git push transmits new commits to a remote (e.g., GitHub, GitLab). Before the push is accepted, many teams run a pre‑receive or continuous integration gate that validates schema compatibility, checks that required metrics are present, and ensures no prohibited files (like raw CSVs) have slipped through.

The diagram emphasizes two validation points: a local pre‑commit hook (shown implicitly by the return path from D to WS) and a remote CI check (the diamond D). Both are essential for maintaining integrity in a data science context.

Core Concepts

Working Directory as a Computational Sandbox

The working directory mirrors the state of your local environment. In data science, this often includes:

• Notebooks (analysis.ipynb) – interactive exploration.
• Source code (src/feature_build.py, src/train.py) – reusable functions.
• Configuration (params.yaml, docker-compose.yml) – experiment hyperparameters and service definitions.
• Ephemeral artifacts (*.log, *.tmp, data/raw/) – files that should never be versioned.

Git only tracks files that have been added to the index at least once. Untracked files are ignored until you explicitly git add them.

Staging Area – The Curated Snapshot

The index is not a simple copy; it stores metadata (mode, SHA‑1) for each path. This design enables powerful workflows:

• Partial staging (git add -p) lets you commit a bug fix in a script while leaving unrelated experiments unstaged.
• File renames are detected automatically when the content similarity exceeds a threshold, useful when you rename a notebook for clarity.
• Ignoring patterns (.gitignore) prevent the index from ever seeing files that match the rules (e.g., data/*, __pycache__/).

Commit – Immutable Record of Experimentation

A commit’s SHA‑1 hash cryptographically binds the snapshot’s content. In data science, treat each commit as a lab notebook entry:

• Atomicity – Commit only changes that together represent a single logical step (e.g., “add feature X and update model Y”).
• Message convention – Use a structured format like feat: add rolling window feature or fix: correct data leakage in train/test split. This enables automated changelog generation and helps reviewers understand intent.
• Signed commits (git commit -S) provide non‑repudiation, important when results influence business decisions.

Push – Synchronizing the Lab Network

Pushing publishes your commits to a shared remote. In a data science team, consider:

• Branch strategy – main holds the vetted, reproducible baseline. Feature branches (feature/timeseries-cv) isolate work. Release branches (release/v1.2) tag a set of experiments destined for production.
• Protected branches – Prevent direct pushes to main; require pull request approvals and successful CI runs.
• Git LFS – For larger artifacts (model weights >100MB, medium‑size Parquet files), Git Large File Storage stores pointers in the repo while keeping the actual binaries in a separate store, avoiding repository bloat.

Examples & Code Walkthrough

Example 1: Selective Staging of Notebooks and Scripts

This Python wrapper scans the workspace, stages only relevant files, and updates .gitignore if it spots large CSV files that should be tracked via LFS.

#!/usr/bin/env python3
"""
stage_workspace.py
Curate what gets added to the Git index in a data science repo.
- Stages .ipynb, .py, .yaml, .yml files.
- Warns about large CSV/Parquet files and suggests git lfs track.
- Leaves logs, tmp, and env files untouched.
"""

import subprocess
import pathlib
import sys

def run(cmd):
    result = subprocess.run(cmd, shell=True, capture_output=True, text=True)
    if result.returncode != 0:
        print(f"[ERROR] {cmd}\n{result.stderr}", file=sys.stderr)
        sys.exit(1)
    return result.stdout.strip()

def main():
    repo_root = pathlib.Path.cwd()
    # Patterns we *always* want to track for reproducibility
    track_globs = ["*.ipynb", "*.py", "*.yaml", "*.yml", "*.json"]
    # Patterns that suggest LFS or ignore
    large_globs = ["*.csv", "*.parquet", "*.pkl", "*.h5"]

    to_add = []
    lfs_candidates = []

    for pattern in track_globs:
        to_add.extend(repo_root.rglob(pattern))

    for pattern in large_globs:
        for path in repo_root.rglob(pattern):
            # Skip if already ignored or tracked via LFS
            if path.is_file():
                lfs_candidates.append(path)

    if to_add:
        run(f"git add {' '.join(str(p.relative_to(repo_root)) for p in to_add)}")
        print(f"Staged {len(to_add)} notebook/script/config files.")
    else:
        print("No notebooks/scripts/configs to stage.")

    if lfs_candidates:
        print("\n[WARNING] Large data files detected:")
        for p in lfs_candidates[:5]:
            print(f"  - {p}")
        print("Consider running:")
        print("  git lfs track \"*.csv\"")
        print("  git add .gitattributes")
    else:
        print("\nNo large data files needing LFS attention.")

if __name__ == "__main__":
    main()

How it fits the workflow

• Working Directory – The script walks the repo, identifying what you’ve edited.
• Staging Index – It calls git add only on the curated set, keeping the index clean.
• Commit – After running this, you can git commit -m "feat: add customer churn feature" with confidence that only relevant changes are included.
• Push – CI will later verify that no large files slipped through (see Example 2).

Example 2: Pre‑Commit Hook That Validates Model Metrics

This hook prevents committing to main unless a JSON file with evaluation metrics exists and meets a minimum threshold.

#!/usr/bin/env bash
# .git/hooks/pre-commit
# Enforce that any commit touching main includes a valid metrics.json

BRANCH=$(git rev-parse --abbrev-ref HEAD)
PROTECTED="main"
METRICS_FILE="metrics.json"
MIN_ACC=0.80   # Example threshold

if [[ "$BRANCH" == "$PROTECTED" ]]; then
    if [[ ! -f "$METRICS_FILE" ]]; then
        echo "[ERROR] $METRICS_FILE missing on $BRANCH commit."
        echo "Please run evaluation and commit the metrics file first."
        exit 1
    fi

    # Extract accuracy (assumes JSON with top-level "accuracy" key)
    ACC=$(jq -r '.accuracy // empty' "$METRICS_FILE")
    if [[ -z "$ACC" ]] || (( $(echo "$ACC < $MIN_ACC" | bc -l) )); then
        echo "[ERROR] Model accuracy $ACC is below threshold $MIN_ACC."
        echo "Commit blocked. Improve model or adjust threshold."
        exit 1
    fi

    echo "[INFO] Metrics check passed: accuracy=$ACC"
fi

# Allow the commit to proceed
exit 0

Make it executable: chmod +x .git/hooks/pre-commit.

Why this matters

• Working Directory – You may have modified train.py and produced a new metrics.json.
• Staging – git add train.py metrics.json places both in the index.
• Commit – The hook runs before the commit object is created; if metrics fail, the commit is aborted, forcing you to fix the model or adjust the threshold.
• Push – Only commits that have passed this local gate reach the remote, reducing the chance of broken models entering main.

Example 3: Automated Push with Schema Validation (CI)

A simplified GitHub Actions workflow that checks Parquet schema before allowing a push to main.

name: Validate & Push

on:
  push:
    branches: [ main ]

jobs:
  schema-check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0   # Need full history for LFS objects
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: "3.11"
      - name: Install dependencies
        run: |
          pip install pandas pyarrow
      - name: Verify Parquet schema
        run: |
          python - <<'PY'
          import pandas as pd
          import sys
          path = "data/processed/features.parquet"
          df = pd.read_parquet(path)
          expected_cols = {"customer_id", "feature_1", "feature_2", "label"}
          actual = set(df.columns)
          if expected_cols != actual:
              sys.exit(f"Schema mismatch: expected {expected_cols}, got {actual}")
          print("Schema OK")
          PY
      - name: Push LFS objects (if any)
        run: |
          git lfs push origin $(git rev-parse --abbrev-ref HEAD)

This CI job runs on every push to main. If the Parquet file’s columns drift, the job fails and the push is rejected—protecting downstream training pipelines that depend on a stable contract.

Best Practices

1. Stage with intention – Use git add -p or explicit pathspecs; avoid git add . unless you’ve verified the workspace with git status --ignored.
2. Write commit messages that tell a story – Follow the Conventional Commits standard (<type>: <subject>). Include a ticket ID or experiment number when relevant.
3. Leverage hooks for early feedback – Pre‑commit hooks catch formatting issues, missing metrics, or forbidden file types before they become part of history.
4. Tag releases, not just commits – After a set of experiments passes validation, create an annotated tag (git tag -a v1.2 -m "Release for Q3 forecast"). Tags survive rebases and make rollback trivial.
5. Manage large data with Git LFS or external storage – Keep the repo lightweight; store raw datasets in an object store (S3, GCS) and version only pointers or metadata.
6. Rebase locally, merge on remote – Keep a linear feature‑branch history with git rebase origin/main before opening a pull request; let the merge commit on main preserve the branch topology for auditability.
7. Automate version bumping – Use tools like setuptools-scm or versioneer to derive package versions from Git tags, ensuring your Python packages reflect the exact code shipped.

Common Mistakes & Anti-Patterns

Mistake	Why it’s harmful	Fix
Committing raw CSV/JSON files directly	Bloats repo, slows clones, risks leaking PII.	Add patterns to .gitignore; use Git LFS or store files in a data lake and version only a manifest file (e.g., data_manifest.yaml).
Skipping the staging step (git commit -a)	Commits unintended changes like debug prints or half‑finished experiments.	Always run git status first; use git add -p to review each hunk.
Using vague commit messages (“fix stuff”, “update notebook”)	Makes code review and future bisecting nearly impossible.	Adopt a message template: <type>(<scope>): <short summary> + optional body with rationale and metric numbers.
Pushing directly to main without review	Increases chance of breaking the shared baseline; no opportunity for peer feedback.	Protect main; require pull requests with at least one approval and passing CI.
Ignoring merge conflicts in notebooks	Results in corrupted JSON or lost cells when resolved incorrectly.	Use nbdime or jupyter nbmerge to resolve conflicts at the notebook level, or convert notebooks to scripts for merging and re‑export after.

Performance Considerations

Aspect	Impact	Mitigation
Object database size	Each commit stores full snapshots of tracked files (as blobs). Frequent large file commits inflate .git/objects.	Use Git LFS for binaries; compress intermediate artifacts (e.g., joblib.dump(..., compress=3)).
Index scanning	git status walks the working directory to detect changes; slows with hundreds of thousands of files.	Keep the repo focused on code and metadata; exclude data directories via .gitignore.
Hook execution latency	Complex pre‑commit/push hooks add seconds to each developer action.	Offload heavy validation to CI; keep hooks lightweight (linting, file size checks).
Network push overhead	Pushing many large LFS objects can saturate bandwidth.	Use git lfs fetch --recent to retrieve only needed LFS files; schedule large pushes during off‑peak hours.
CPU for diff generation	Git’s diff algorithm runs on every git status and git log -p.	For notebooks, store outputs separately or clear them before committing (jupyter nbconvert --ClearOutputPreprocessor.enabled=True).

Real-World Usage

• Netflix – Their machine learning platform (Metaflow) integrates Git for versioning workflow definitions. Each step’s code lives in a repo; data artifacts are versioned via external storage, while Git captures the exact pipeline specification.
• Uber – Michelangelo, Uber’s ML platform, stores feature definitions and model code in Git. A pre‑merge validation