I had a custom metrics table. It was working. Batch durations, row counts, streaming heartbeats, all landing in Delta. One problem: the `job_id` and `run_id` columns were null in every row.

These two columns exist to enable you to join custom metrics to Databricks system tables. Without them, my per-batch timing data lives in isolation. With them, one SQL join gives you batch internals correlated with job cost, cluster utilization, and run outcomes. The whole point of the table.

So I asked my LLM coding agent to fix it. What followed was an afternoon of increasingly creative hallucinations, each delivered with full confidence, each completely wrong.

Hallucination 1: Spark conf

The agent’s first suggestion:

job_id = spark.conf.get(”spark.databricks.job.id”)

run_id = spark.conf.get(”spark.databricks.job.runId”)

Sensible-looking. There are plenty of Stack Overflow answers and blog posts mentioning these keys. The agent had probably trained on hundreds of them.

The result on our serverless compute:

ERROR: [CONFIG_NOT_AVAILABLE] Configuration spark.databricks.job.id is not available.

Not “key not found.” Not “returns null.” A hard error with a JVM stack trace 80 lines long. This config key doesn’t exist in the Spark Connect protocol that serverless uses. The agent had no way to know that because it trained on content from the classic compute era.

Hallucination 2: environment variables

After the Spark conf failure, the agent pivoted to environment variables:

import os

job_id = int(os.environ[”DATABRICKS_JOB_ID”])

run_id = int(os.environ[”DATABRICKS_RUN_ID”])

This one was interesting because the agent didn’t just suggest reading env vars. It invented the variable names. `DATABRICKS_JOB_ID` is not a real environment variable that Databricks sets. The agent generated a plausible-sounding name, wrote the code with confidence, and I deployed it.

The metrics kept showing null.

I dumped every environment variable matching “JOB”, “RUN”, or “DATABRICKS” from a running job. Here’s what Databricks actually sets:

DATABRICKS_RUNTIME_VERSION=client.5.1

DATABRICKS_CLUSTER_LIBS_PYTHON_ROOT_DIR=python

DATABRICKS_GANGLIA_ENABLED=FALSE

Runtime metadata. Library paths. Nothing about job or run identity. `DATABRICKS_JOB_ID` doesn’t exist. The agent made it up.

Hallucination 3: dbutils notebook context

Third attempt. The agent went deeper into the Databricks internals:

ctx = dbutils.notebook.entry_point.getDbutils().notebook().getContext()

job_id = int(ctx.tags().get(”jobId”).get())

run_id = int(ctx.tags().get(”idInJob”).get())

This is a real API. It actually works, in notebooks. But we run Python wheel tasks, not notebooks. The error:

module ‘pyspark.dbutils’ has no attribute ‘notebook’

The `pyspark.dbutils` module exists in wheel task context but the `notebook` sub-module doesn’t load. There’s no notebook. The module doesn’t load because there’s nothing to load it for. But the agent found an API that looks right, generated the code, and moved on.

Hallucination 4: dynamic references in spark_env_vars

At this point I pulled up the Databricks docs myself. I found the page on dynamic value references. The agent read it too and proposed putting `{{job_id}}` in the DAB `spark_env_vars`:

spark_env_vars:

  DATABRICKS_JOB_ID: “{{job_id}}”

  DATABRICKS_RUN_ID: “{{run_id}}”

Two problems. First, the syntax was wrong. The correct dynamic reference is `{{job.id}}`, not `{{job_id}}`. Second, and more fundamentally, `spark_env_vars` doesn’t resolve dynamic value references at all. The values pass through as literal strings. The cluster environment showed:

DATABRICKS_RUN_ID={{run_id}}

Not the run ID. The literal text `{{run_id}}`.

The Databricks docs don’t say “dynamic value references don’t work in spark_env_vars.” They just don’t list spark_env_vars as a supported location. The docs describe where they do work (task parameters, job parameters), but they never explicitly say where they don’t. That silence is a trap for both humans and language models.

The documentation problem

The Databricks documentation for dynamic value references says you can use `{{job.id}}` in “parameters or fields that pass context into tasks.” It gives examples for notebook `base_parameters` and job-level `parameters`. For Python wheel tasks, it says “parameters defined in the task definition are passed as keyword arguments to your code.”

What it doesn’t say:

• Which specific YAML fields support resolution and which don’t

• That `spark_env_vars` passes values through without resolving them

• That the old `spark.databricks.job.id` conf key doesn’t work on serverless

• That `dbutils.notebook` doesn’t load in non-notebook task types

Each hallucination mapped to a gap in the documentation. The agent wasn’t generating random nonsense. It was generating reasonable-sounding answers to questions the docs leave unanswered. Incomplete docs don’t just confuse humans. They give LLMs just enough information to construct confident wrong answers.

The human fix: stop guessing, start testing

After four failed attempts, I did what I should have done first. I wrote a test script:

import sys

import os

print(”=== sys.argv ===”)

print(sys.argv)

print(”\n=== Job-related env vars ===”)

for key in sorted(os.environ):

    if “JOB” in key or “RUN” in key or “DATABRICKS” in key:

        print(f”  {key}={os.environ[key]}”)

print(”\n=== dbutils context ===”)

try:

    from dbruntime.databricks_repl_context import get_context

    ctx = get_context()

    print(f”  jobId={ctx.jobId}”)

    print(f”  idInJob={ctx.idInJob}”)

except Exception as e:

    print(f”  repl_context failed: {e}”)

Just a few lines. Created a Databricks job, added job parameters with `{{job.id}}` and `{{job.run_id}}`, set the task parameters to pass them as CLI args, ran it.

The output told me everything in one shot:

• `sys.argv` had the resolved job and run IDs from the task parameters

• Every env var approach was dead

• Spark conf threw hard errors

• `dbruntime.databricks_repl_context` actually worked too (undocumented but functional)

Ten minutes from “let me just test this” to knowing exactly which approaches work and which don’t. Compare that to four rounds of agent suggestions, deployments, and failures.

The working solution

Job-level parameters with dynamic value references, referenced from task `named_parameters`. The values arrive as `sys.argv` and get parsed with argparse:

# DAB job definition

parameters:

  - name: job_id

    default: “{{job.id}}”

  - name: run_id

    default: “{{job.run_id}}”

tasks:

  - python_wheel_task:

      entry_point: “my-workflow”

      named_parameters:

        job_id: “{{job.parameters.job_id}}”

        run_id: “{{job.parameters.run_id}}”

@staticmethod

def _parse_job_context():

    import argparse, sys

    parser = argparse.ArgumentParser()

    parser.add_argument(”--job_id”, type=int, default=None)

    parser.add_argument(”--run_id”, type=int, default=None)

    args, _ = parser.parse_known_args(sys.argv[1:])

    return args.job_id, args.run_id

We put the parsing in the base `Workflow` class. Any workflow that enables metrics gets job context automatically. The only per-workflow work is adding the `parameters` and `named_parameters` blocks to the DAB YAML.

Not an anti-AI post

I’m not writing this to dunk on LLMs. I use one every day. It wrote most of the boilerplate in our metrics writer. It’s genuinely good at generating code when the problem is well-understood and the patterns are common.

But there’s a specific failure mode that showed up four times in one afternoon: the agent treats documentation gaps as opportunities to interpolate. When the docs don’t say how to do something, it constructs an answer from adjacent knowledge. `spark.databricks.job.id` exists in older Databricks content, so it suggests that. `DATABRICKS_` is a common prefix for their env vars, so it invents one. The `dbutils.notebook.entry_point` chain works in notebooks, so it assumes it works everywhere.

Each interpolation sounds plausible. Each fails for a reason the agent can’t know without testing.

The fix wasn’t more prompting or a better model. It was stepping back and writing a test script. Isolating the problem. Running it. Reading the output. Deciding based on evidence instead of confidence.

That’s not a prompting skill. That’s an engineering skill. The specific one where you stop asking “what should work?” and start asking “what actually works right now, on this compute, in this runtime?”

LLMs write code. Engineers figure out which code to write. Those are different skills, and this afternoon was a good reminder that the second one isn’t going anywhere.

Not because it wouldn’t work. We tested it. We documented it. The code was ready. The answer was one cluster per task — true driver isolation, one JVM per pipeline, failures that cannot, by construction, spread sideways.

The reason we hadn’t switched: budget and scale. That’s a completely legitimate engineering decision.

What “real” isolation actually means

Everything in Parts 1 and 2 lived in this shape:

Shared Job Cluster
Python Proc A Python Proc B
\ /
┌─────────────┐
│ Spark JVM │
│ (shared) │
└─────────────┘
│
Executors


One driver. Everything that fails, fails together — or worse, fails silently while the rest keeps running.

One cluster per task looks like this:

Dedicated Clusters
Task A Task B
Cluster A Cluster B
Python Proc Python Proc
│ │
┌─────────┐ ┌─────────┐
│ JVM A │ │ JVM B │
└─────────┘ └─────────┘
│ │
Executors Executors


Task B fails → Cluster B restarts → Task A: unaffected


That last line is the one that matters. Task B crashing cannot reach Task A. No shared driver, no shared memory, no shared scheduler. The failure boundary is the cluster, not the job.

What the experiment showed

We ran it. Moved our streaming queries onto separate clusters — some as separate tasks in the same job, some as fully independent jobs. Both approaches worked.

Failures stopped spreading. When we deliberately triggered one, the other pipeline kept running correctly — not “mostly healthy.” The UI was accurate for the first time, which sounds like a minor thing until you remember how long we’d been staring at misleading dashboards.

Debugging got faster in a way that surprised me. Each cluster has its own driver logs, its own metrics, its own lifecycle. “Which pipeline failed?” stopped requiring timestamp correlation across a shared log stream.

Sizing became honest too. A heavy pipeline gets the compute it needs. A lightweight one runs on something smaller. You’re not picking a number that has to cover the worst case across everything.

The code was already ready

Something we hadn’t fully appreciated: we didn’t need to rewrite anything.

Our pipelines were structured so the same code could run in any deployment shape:

• All queries in one task, one cluster — the Part 1 setup

• Multiple tasks, shared cluster — the Part 2 setup

• Multiple tasks, dedicated cluster per task — the target architecture

Deployment and configuration change, not a code change. The streaming logic, the Delta writes, the checkpointing — all identical. Only the compute allocation changed.

That was intentional. It also made the experiment cheap to run, which helped.

Why we’re waiting

Subscribe now

Cost and scale, honestly.

Right now, our workload fits on a shared cluster without catastrophic failure — especially with awaitAnyTermination providing fail-fast semantics on the streaming side and task retries on the batch side. It’s not elegant. But it’s stable enough for where we are.

Once device count crosses the threshold, the calculus changes. When the operational cost of a missed or false alert exceeds the infrastructure cost of a dedicated cluster, you stop debating. We know roughly where that line is. We’re watching the numbers.

When to make the move

If any of these apply to your setup, you’re probably ready:

• A failure in one pipeline has caused a correctness problem in another

• You’re spending more time debugging shared-cluster incidents than the extra cluster would cost per month

• Customer impact from partial failures is measurable — missed alerts, false positives, SLA breaches

• Individual pipelines have meaningfully different compute profiles and you’re sizing the shared cluster to the worst case

None of these require a crisis. The best time to make the architecture change is before the next incident, not after.

The progression in one diagram

Part 1: Multi-query, shared cluster
→ Problem: silent query failures
→ Fix: awaitAnyTermination (band-aid)


Part 2: Multi-task, shared cluster
→ Problem: still one driver, same failure modes
→ Fix: task-level retry (better, still shared)


Part 3: One cluster per task
→ True isolation
→ Failures contained by construction
→ Ready when the scale justifies it


The band-aids in Parts 1 and 2 weren’t mistakes. We took a known shortcut because the right answer cost more than the problem did at the time. Knowing that, and knowing what would change it — that’s what matters. We know both.

The code is sitting there, ready to go. We flip the switch when the numbers say it’s time.

The incident that kicked off this series also involved a ConcurrentAppendException that went deeper than a simple retry could fix — Delta isolation levels, isBlindAppend, and why you can’t always do what you think you can inside foreachBatch. That’s the next post, and probably the most technically dense thing I’ll write this year.

If running eight queries in one task means one failure can hide while others keep running — split them into multiple tasks. Separate concerns. Give each component its own retry boundary.

Right instinct. Wrong infrastructure assumption.

We tried it

While the multi-query incident from Part 1 was still fresh, we were already experimenting with a multi-task approach on a separate workflow. Two tasks, same shared job cluster:

• Task 1: feature extraction — processing sensor data into feature tables

• Task 2: inference — ML model outputs written to downstream Delta tables

Sequential dependency. Task 2 reads what Task 1 writes. Clean separation on paper.

Then Task 2 hit a wall.

The incident — external location mismatch

Task 2 was writing to a Delta table registered in Unity Catalog. The catalog entry pointed to external location A. The actual data sat at location B.

A misconfiguration. Easy to make during migration, hard to spot before it fails in production.

Task 2 failed. Task 1 kept running.

And here’s where it felt familiar: the job didn’t fail. No restart triggered. One task retrying. The other healthy. The UI said RUNNING.

Same story as Part 1. Different packaging.

The detail that changes everything: there’s still one driver

Here’s what multi-task on a shared cluster actually looks like at runtime:

Multi-Task on a Shared Job Cluster

Task 1 (Python Process A)     Task 2 (Python Process B)
          \                           /
           \                         /
            ┌────────────────────────┐
            │      Spark Driver      │
            │         JVM            │
            │    (shared by all)     │
            └────────────────────────┘
                       │
                  Executors

Multiple Python processes. One Spark driver JVM.

Compare that to multi-query single task from Part 1:

Multi-Query Single Task

     Python Process (single)
              │
     ┌────────────────────┐
     │    Spark Driver    │
     │        JVM         │
     │  Q1  Q2  Q3 ... Q8 │
     └────────────────────┘
              │
         Executors

The difference between these two diagrams is smaller than it looks. Both share the same driver. Both share the same executors. Multi-task adds Python process separation — but that’s not where streaming failures originate. Streaming failures live in the JVM, in the query scheduler, in the Delta transaction layer. All of which is still shared.

What multi-task actually adds on a shared cluster

Splitting into tasks on a shared cluster gives you:

• Multiple Python processes on the same driver node

• Multiple SparkSession lifecycles, each with its own initialisation overhead

• More listeners, more logging, more scheduler registration

• Concurrent memory pressure when tasks run in parallel

• A failing task retrying repeatedly can destabilise the cluster for every other task

You get the operational complexity of multiple processes without the isolation you were looking for.

The fix — and why it works here

For the external location incident, we added task-level retry configuration: three retries per task on the continuous job. Once exhausted, Databricks restarts the entire job.

It works. And it’s a better failure story than Part 1 — Task 2 eventually fails loudly and triggers a restart rather than running silently while Task 1 keeps writing data nobody will ever resolve.

But here’s the key distinction: it works because Task 1 and Task 2 are sequential. Task 2 depends on Task 1. They don’t run simultaneously. No concurrent driver contention. Failure propagates cleanly up the chain.

Multi-task on a shared cluster is a reasonable pattern for sequential batch ETL. Feature extraction feeds inference. Inference feeds output. Tasks chain, failures surface, retries make sense.

The problem is assuming the same pattern works for parallel long-running streaming. That’s where the shared driver becomes a liability instead of a trade-off.

The rule we wrote down

After both incidents, this became our working principle:

Multi-task on a shared cluster: right for sequential batch ETL, wrong for parallel streaming.

The difference is contention. Sequential tasks don’t compete for the driver simultaneously. Parallel streaming queries do — continuously, for the lifetime of the job.

If you’re running parallel streaming on a shared cluster, a multi-query single task with awaitAnyTermination (Part 1) gives you a cleaner failure boundary than splitting into tasks.

If you’re running sequential batch ETL, multi-task with task-level retry is a legitimate approach within the budget constraints of a shared cluster.

But this still isn’t the real answer

Both fixes share the same problem.

awaitAnyTermination in Part 1 makes query failures loud. Task retry in Part 2 makes task failures recoverable. Neither prevents a failure in one component from affecting the shared driver — and everything attached to it.

The real answer is what we’d resisted for months: one cluster per task. A failure in the inference pipeline that cannot, by construction, affect the ingestion pipeline.

That’s Part 3 — when we made the architectural change, what it cost, and what got better overnight.

→ Part 3: One Cluster per Task — What Real Isolation Actually Looks Like

Then the message landed: “Customer is complaining. Alerts are wrong.”

I opened the Databricks UI. Seven of eight queries: green. One: failed silently.

The job said RUNNING.

The customer said something different.

That gap — between what the system reports and what is actually happening — is what this series is about. Not the catastrophic failures. The quiet ones.

The incident

We were running a multi-query continuous job on a single shared cluster. Eight streaming queries, one Python process, one Spark driver.

Yes, you could call this a bad design. And you wouldn’t be wrong. But it was a conscious decision — budget-constrained infrastructure, deliberately consolidated. We knew the trade-offs. Or thought we did.

• Query 1: bronze ingestion — reading raw telemetry from EventHub into Delta tables

• Queries 2–7: structuring raw data into individual domain tables

• Query 8: alert resolution — resolving alerts in near real-time that were created by a separate ML job

All of them running together. All of them sharing the same driver.

Bronze ingestion was completely fine. But Query 8 — alert resolution — hit a ConcurrentAppendException. A Delta Lake conflict that fires when two writers try to commit to the same table at the same time.

What followed was worse than a crash:

• Queries 1–7 kept running

• Query 8 silently died

• For 12 minutes, offsets stopped committing for that stream while the UI still said RUNNING

• No monitoring fired

Then came the customer messages.

Why 12 minutes matters more than it sounds

Twelve minutes of stopped offset commits is not just a monitoring gap.

It means the system was consuming from EventHub without acknowledging it. If the job had restarted in that window: reprocessing, potential duplicates, depending on your downstream semantics.

But the job didn’t restart. Nothing triggered a restart. The driver was alive. The other seven queries were healthy. Spark had no reason to terminate the run.

The system was wrong for 12 minutes and had no idea.

The monitoring irony

We had monitoring. Job failure alerts, cluster termination alerts, retry notifications.

None of it fired.

Our monitoring was designed to catch job failures. What happened here wasn’t a job failure — it was a streaming query failure inside a job that kept running. The job never restarted. Our alerts never triggered.

The part that still stings: the query that died silently was the one responsible for resolving alerts.

Here’s what happened end to end: an ML signal created a CREATE alert. A downstream event should have triggered Query 8 to write a RESOLVE and close it. Query 8 was dead. The RESOLVE never came. The alert stayed open. The customer saw a false positive that should have been resolved hours earlier.

The system designed to catch problems couldn’t catch its own.

Two classes of failure — the mental model that finally clicked

After this incident, a pattern became clear. There are two fundamentally different ways a streaming job can fail.

Query-scoped failures (driver stays alive)

ConcurrentAppendException. Schema mismatch. Logic errors in foreachBatch. Wrong Delta path.

In these cases: one stream fails, the Spark driver JVM stays up, the other queries keep running, and the UI looks mostly healthy. This is the dangerous state — half a pipeline, half a truth.

Driver-scoped failures (everything stops)

Driver OOM. JVM crash. Deep scheduler corruption.

Loud. Painful. But honest. Everything stops, Databricks restarts, you get a clean slate.

The uncomfortable reality: driver-scoped failures are easier to operate.

A full crash is an operational problem. A partial failure is a product problem. The customer doesn’t care that Queries 1–7 were fine if Query 8 was quietly dead.

Why we didn’t catch it sooner — the awaitTermination trap

Here’s something Databricks doesn’t shout from the rooftops: when a streaming query fails inside a multi-query job, the Spark JVM doesn’t automatically know — or care — that the query is gone. The other queries keep running. The driver stays up. The job keeps running.

Our job had eight calls to query.awaitTermination(), one per stream, in sequence. The Python process blocks on the first one. When Query 8 failed, Python was still waiting on an earlier stream’s termination. It never reached the point of checking Query 8.

From Python’s perspective: blocked, waiting. From the JVM’s perspective: everything is fine. From the UI’s perspective: RUNNING. From the customer’s perspective: something is very wrong.

Worth noting: Databricks documentation doesn’t recommend either awaitTermination or awaitAnyTermination for production — they want you to rely on job-level management instead. But in our case, we needed the Python process to exit to trigger a JVM failure, which Databricks would then restart.

”But continuous jobs restart automatically”

They do — eventually. Continuous jobs retry with backoff and trigger a new run after exhausting retries.

The catch: for that to happen, the Spark JVM has to actually fail first. With query.awaitTermination() on each stream individually, the Python process never exited. Spark never failed. Databricks never restarted.

The job just sat there. RUNNING. Wrong.

The fix — and why it’s a band-aid

Once we understood the failure chain, the fix was one line.

Replace individual query.awaitTermination() calls with:

spark.streams.awaitAnyTermination()

Now, if any stream fails, the Python process exits. Python exits → Spark JVM dies → Databricks detects the failure → 3 retries → new run. Proper failure semantics, clean restart, no more silent death.

Did it work? Yes. Is it a proper solution? No.

But who has time for a proper solution when production is on fire?

awaitAnyTermination gave us fail-fast behaviour while we worked on the actual fix. It stopped the lying. It didn’t fix the underlying architecture.

The underlying architecture — why running eight streaming queries in one job on a shared cluster creates these failure modes in the first place, and what we did about it — is what Part 2 is about.

→ Part 2: Multi-Task on a Shared Cluster — Why That’s Also Not Enough

When working with externally managed Delta tables and traditional partitioning strategies (for example by day, week, or month), one common challenge is:

How large are my partitions actually?

Before deciding whether to partition by day vs. week vs. month, it’s important to understand how data is physically distributed across partitions.

This article shows how to extract partition-level size statistics directly from the Delta transaction log.

⚠️ This approach is useful when using traditional partitioning and external table strategies.
If you’re using Unity Catalog managed tables or Liquid Clustering, partition sizing decisions are handled differently.

Why Look at the Delta Log?

Every Delta table maintains a _delta_log directory containing JSON transaction files.
Each add action inside the log includes:

• File path

• File size (in bytes)

• Partition values

By reading these logs, we can compute:

• Number of files per partition

• Total bytes per partition

• Size distribution across partitions

This gives direct visibility into the physical layout of your table.

Example: Compute Partition Sizes by startDate

from pyspark.sql import functions as F

delta_log = spark.read.json(
    "abfss://container@storage-account.dfs.core.windows.net/table_location/_delta_log/*.json"
)

files = (
    delta_log
    .filter("add is not null")
    .select(
        F.col("add.path").alias("path"),
        F.col("add.size").alias("size"),
        F.col("add.partitionValues.startDate").alias("startDate")
    )
)

(
    files.groupBy("startDate")
    .agg(
        F.count("*").alias("numFiles"),
        F.sum("size").alias("totalBytes")
    )
    .withColumn("sizeGB", F.col("totalBytes") / (1024**3))
    .orderBy("startDate", ascending=False)
    .show(20, False)
)

Sample Output

+----------+--------+-----------+--------+
|startDate |numFiles|totalBytes |sizeGB  |
+----------+--------+-----------+--------+
|2026-02-16|  xxx   |   xxx     |   xx   |
|2026-02-15|  xxx   |   xxx     |   xx   |
+----------+--------+-----------+--------+

How This Helps With Partition Strategy

Once you have partition size metrics, you can evaluate:

Are partitions too small?

If daily partitions are only a few MB:

• You may be over-partitioning.

• Consider partitioning by week or month instead.

Are partitions too large?

If partitions exceed hundreds of GB:

• Queries may scan too much data.

• Consider a finer-grained partitioning strategy.

Are files unevenly distributed?

High numFiles with small average size indicates small file issues.

Decision Guidelines

Partition Size (per partition) Recommendation

< 1 GB Likely over-partitioned

1–20 GB Usually healthy

50+ GBConsider finer partitioning

100+ GB May impact performance

(Adjust based on workload and query patterns.)

Bonus: Average File Size Per Partition

You can extend the analysis:

.withColumn("avgFileSizeMB", (F.col("totalBytes") / F.col("numFiles")) / (1024**2))

This helps detect small file problems inside partitions.

When Should You Use This?

This approach is particularly useful when:

• Using external Delta tables

• Managing your own storage layout

• Designing a new partition strategy

• Migrating legacy Hive-style tables

• Troubleshooting performance issues

It gives a low-level, transparent view of how data is physically stored.

Final Thoughts

Partitioning decisions should be based on:

• Query access patterns

• Partition cardinality

• Physical partition size

• File size distribution

Reading the Delta transaction log provides a simple yet powerful way to understand your table layout — before committing to a partitioning strategy.

Welcome, Aspiring Data Engineers, to the second installment of our Data Engineering Essentials series!

Our second post, Unlocking Data Engineering: Simplifying the Journey for Beginners, guided many through the process of reading from APIs and writing into Snowflake. It was like a walk in the park for many beginners, and writing into Snowflake felt like a breeze. If you haven't read it yet, I highly recommend doing so for some quick gratification!

Now, in our third newsletter post, "Simple Steps Towards Data Processing," I've ensured that the code remains straightforward, making it easy for you to understand and execute successfully.

If you ever find yourself stuck at any point, don't hesitate to reach out to me on LinkedIn at kirankbs!

Let's embark on this beginner-friendly journey together, understand the process, and achieve another swift victory!

Agenda: Basic Data Processing

• Data Cleaning: Convert Dates & Non numerical values

• Feature Engineering: Create new features by deriving insights from existing columns

• Data Exploration: Show Summary Statistics & Outliers

For Prerequisites & Setting Up Python Project, please refer to the previous article

Please note that this example utilizes MacOS and Python version 3.8.8. However, users on Linux and Windows systems should encounter no issues!

Data Cleaning

When extracting data from source systems, never assume that the quality of the data is excellent and in the expected formats. As a Data Engineer, you must ensure that it's converted into the correct type for further processing and to clean any unwanted or corrupted data. Make sure the data is read with the proper schema.

# Define the Schema
schema = {
    "vendorid": "str",
    "lpep_pickup_datetime": "str",
    "lpep_dropoff_datetime": "str",
    "store_and_fwd_flag": "str",
    "ratecodeid": "str",
    "pickup_longitude": "float",
    "pickup_latitude": "float",
    "dropoff_longitude": "float",
    "dropoff_latitude": "float",
    "passenger_count": "int",
    "trip_distance": "float",
    "fare_amount": "float",
    "extra": "float",
    "mta_tax": "float",
    "tip_amount": "float",
    "tolls_amount": "float",
    "imp_surcharge": "float",
    "total_amount": "float",
    "payment_type": "str",
    "trip_type": "str"
}

# Fetch data from API
api_url = "https://data.cityofnewyork.us/resource/hvrh-b6nb.json"
response = requests.get(api_url)
data = response.json()

# Convert data to Pandas DataFrame
df = pd.DataFrame(data, columns=schema.keys())

Convert the date from a string type to a DateTime type for further processing. By converting the date from a string to a DateTime type, you gain access to a variety of helper methods in Pandas that can be utilized.

# Data cleaning
# Example: Convert data types
df['lpep_pickup_datetime'] = pd.to_datetime(df['lpep_pickup_datetime'])
df['lpep_dropoff_datetime'] = pd.to_datetime(df['lpep_dropoff_datetime'])

# Remove non-numeric values from 'trip_distance' to get insights on numbers
df['trip_distance'] = pd.to_numeric(df['trip_distance'], errors='coerce')  # Convert non-numeric values to NaN

Well done! The data is now clean and prepared for further processing.

Feature Engineering

Generating new features from existing columns offers users deeper insights and makes visualizing aggregated information much clearer.

Let's derive additional insights from our existing columns. For instance, we can calculate metrics such as trip duration, pickup hour, pickup day of the week, and speed.

df['trip_duration'] = (df['lpep_dropoff_datetime'] - df['lpep_pickup_datetime']).dt.total_seconds() / 60
df['pickup_hour'] = df['lpep_pickup_datetime'].dt.hour
df['pickup_dayofweek'] = df['lpep_pickup_datetime'].dt.dayofweek
df['speed'] = df['trip_distance'] / df['trip_duration']

Data Exploration

We've finished cleaning the data and adding new features. Now, let's take the final step and explore the data! The top 100 records from the dataset are displayed in the console.

View Data

Color.printGreen("Check the first few rows of the DataFrame:")
print(df.head(100))

View Missing Values

Color.printGreen("Check for missing values:")
print(df.isnull().sum())

View Summary for Trip Duration

trip_duration_stats = df['trip_duration'].describe()
Color.printGreen("Summary Statistics for Trip Duration:")
print(trip_duration_stats)

View Outliers

df['fare_amount'] = pd.to_numeric(df['fare_amount'], errors='coerce')
outliers = df[df['fare_amount'] < 0]
Color.printGreen("Outliers in fare_amount column:")
print(outliers)

Write To File

Define the features you want to include in the new DataFrame:

selected_features = ['vendorid', 'lpep_pickup_datetime', 'lpep_dropoff_datetime', 'passenger_count', 'trip_distance',
                     'fare_amount', 'total_amount', 'trip_duration', 'pickup_hour', 'pickup_dayofweek', 'speed']

Select the specified features from the original DataFrame

new_df = df[selected_features].copy()

Write the new DataFrame to a CSV file

new_df.to_csv('trips_insights.csv', index=False)

Congratulations, If you have come across this far!

This is it! It is this simple to fetch data from API and process it. You don’t need Spark, Flink, or Notebook .. to write simple code.

Note: This application is far fetch from completion but let’s take basic steps and improve this application further!

What Is Next

In the upcoming newsletter, I'll delve into captivating visualizations and dashboards without using PowerBI or Tableau.

Please subscribe for more such content and share for the far reach!

Navigating the vast landscape of Data Engineering tools and concepts can be overwhelming for newcomers, with a multitude of elements to consider: data sources, storage solutions, schemas, formats, pipelines, visualization tools, and processing engines etc

Aspiring Data Engineers, Getting quick wins is crucial for maintaining momentum and motivation!

Inspiration for this newsletter post is from Data Expert Linkedin Post

For this newsletter post, I deliberately began with a small dataset and relied solely on Python Pandas to experience the quick win and gain a nutshell understanding of end-to-end data processing.

Subscribe now

At any point in time you think you got stuck, feel free to ping me on Linkedin kirankbs!

Let's stroll through this beginner-friendly process to grasp what's going on, and I'll help you navigate through it to achieve a swift victory!

Prerequisites

• Python

• pip

Please note that this example utilizes MacOS and Python version 3.8.8. However, users on Linux and Windows systems should encounter no issues!

Set Up Python Project

Download or Clone source code from GitHub Repository. You can open following project any IDE such as IntelliJ/PyCharm, VSCode or simply Text Editor.

Let’s create a Python virtual environment to install project dependencies locally. Run the below commands inside the project folder

python -m venv venv

Activate Python Virtual environment

source venv/bin/activate

Install Dependencies

pip install -r requirements.txt

Now, the project is downloaded and dependencies are installed!

However, you'll still need to set up a Snowflake account and configure the project before running the application.

Subscribe now

Snowflake Set Up

Snowflake Cloud offers a 30-day free trial, providing ample time to explore and gain practical experience. Once you've signed up and created an account, you should configure the code block mentioned below inside the file WriteToSnowflake.py

conn_params = {
    'account': 'account name', # Replace with your account name
    'user': 'user name', # Replace with your username
    'password': 'password',   # Replace with your password
    'warehouse': 'NYCTAXI',
    'database': 'NYCTAXIDATABASE',
    'schema': 'public'
}

with the following details:

Account: This serves as the unique identifier for your Snowflake account. You typically receive it upon signing up for Snowflake. You can also find it in the URL you use to access the Snowflake web interface. For instance, if the URL is https://app.snowflake.com/lzkqavc/dq12000, the account name would be lzkqavc-dq12000. Alternatively, you can locate it under Admin → Accounts, resembling https://<account>.snowflakecomputing.com.

User: This is the username you use to log in to Snowflake.

Password: This is the password associated with your Snowflake user account.

Warehouse: In Snowflake, a warehouse is a computing resource that executes SQL queries. Create a new warehouse NYCTAXI by navigating to the Admin -> Warehouses tab.

Database: A database in Snowflake is a container for your data and database objects. Create a new database NYCTAXIDATABASE by logging in to Snowflake and navigating to the Data → Databases tab.

Schema: A schema in Snowflake is a logical container for database objects, such as tables, views, and functions. You can find the available schemas by logging in to Snowflake and selecting the desired database. public schema is the default schema in Snowflake.

Create table Table inside the public schema with the following definition

create or replace TABLE NYCTAXIDATABASE.PUBLIC.TRIPS (
	VENDORID VARCHAR(16777216),
	LPEP_PICKUP_DATETIME VARCHAR(16777216),
	LPEP_DROPOFF_DATETIME VARCHAR(16777216),
	STORE_AND_FWD_FLAG VARCHAR(16777216),
	RATECODEID VARCHAR(16777216),
	PICKUP_LONGITUDE NUMBER(38,0),
	PICKUP_LATITUDE NUMBER(38,0),
	DROPOFF_LONGITUDE NUMBER(38,0),
	DROPOFF_LATITUDE NUMBER(38,0),
	PASSENGER_COUNT NUMBER(38,0),
	TRIP_DISTANCE NUMBER(38,0),
	FARE_AMOUNT NUMBER(38,0),
	EXTRA NUMBER(38,0),
	MTA_TAX NUMBER(38,0),
	TIP_AMOUNT NUMBER(38,0),
	TOLLS_AMOUNT NUMBER(38,0),
	IMP_SURCHARGE NUMBER(38,0),
	TOTAL_AMOUNT NUMBER(38,0),
	PAYMENT_TYPE VARCHAR(16777216),
	TRIP_TYPE VARCHAR(16777216)
);

Run the Application

As the code is pretty self-explanatory, let’s run the below commands inside the project folder

python src/nyc-taxi-data-e2e/WriteToSnowflake.py

You will find the following result

in Console

Click Refresh button at Data → Databases → NYCTAXIDATABASE → public → Tables → TRIPS → Data Preview. Tada! There you go with the Data!

Congratulations, If you have come across this far!

This is it! It is this simple to fetch data from API and insert it into the Snowflake Database. You don’t need Spark, Flink, or Notebook .. to write simple code.

Note: This application is far fetch from completion but let’s take basic steps and improve this application further!

Please subscribe for more such content and share for the far reach!

Subscribe now

The first time I encountered the OAuth mechanism, I was puzzled by a particular question: How does the OAuth provider reach the web app endpoint, such as a service running at localhost?

Here's the thing: When it comes to OAuth provider, it doesn't directly contact your local service. Instead, it leads the user's browser back to your service with an authorization code.

But hold on a second... Doesn't redirection imply calling my service's endpoint?

Here, the crucial point to remember is the fourth one listed below: You'll receive an HTTP 302 response, with a `Location` header pointing to your callback URL.

If you're still feeling unsure, or if you're in the same boat as I was, let's dive deeper:

I'll explain using the GitHub OAuth Provider and animated flow as an example

.

Callback URL: This is a URL in your application where GitHub's OAuth service will redirect the user after they have authorized your application. This URL is provided by you when you redirect the user to GitHub's OAuth service. In your case, it's localhost:8080/github/callback.

Redirecting to GitHub's OAuth service: When your application wants to authenticate a user, it redirects them to GitHub's OAuth service. This is done by creating a URL to GitHub's OAuth service that includes your application's client ID, requested scopes, and the callback URL. This URL is generated by `oAuthConfig.AuthCodeURL(stateToken)` in your code.

User authorizes your application: The user logs in to GitHub and is asked if they want to give your application the permissions it's requesting (the scopes). If they agree, GitHub's OAuth service will redirect them back to your application using the callback URL you provided.

Redirect back to your application: GitHub's OAuth service redirects the user's browser back to your application by sending an HTTP 302 response with a `Location` header set to your callback URL. It appends an authorization code as a query parameter to this URL. The user's browser follows this redirect, making a request to your callback URL with the authorization code in the query string.

Your application exchanges the authorization code for an access token: Your application extracts the authorization code from the query string and makes a server-to-server request to GitHub's OAuth service to exchange the authorization code for an access token.

Summary

So, GitHub's OAuth service doesn't directly call your callback URL. Instead, it relies on the user's browser to make a request to your callback URL with the authorization code. This is why the callback URL must be a URL that the user's browser can reach.

Welcome to our Substack newsletter! We explore complex concepts around data and Software Engineering, sharing insights from my daily experiences and personal projects.

What makes us unique? We break down complex topics Byte by Byte, empowering you to become a better data engineer. Our goal? Simplify the tech world for you!

Ready to dive in and level up your skills? Subscribe now and let's unleash your inner engineer!

Subscribe now