Udemy - Databricks Data Engineering - Build Pipelines With Lakeflow

Databricks Data Engineering: Build Pipelines With Lakeflow
https://WebToolTip.com
MP4 | Video: h264, 1920x1080 | Audio: AAC, 44.1 KHz

Language: English | Size: 2.99 GB | Duration: 4h 11m
Build production-grade, dynamic data pipelines with Lakeflow Connect, Lakeflow Jobs, Auto Loader, and Delta Lake.
What you'll learn

Course Overview & Learning Path

What Databricks Lakeflow Is and Why It Matters

Understanding Lakeflow Connect and Lakeflow Jobs

Building Production-Grade Data Ingestion Pipelines

From Raw Data to Delta Tables

Working with Local Files in Databricks

Creating Tables from Local Files

Understanding Data Ingestion in Databricks

Building and Managing Data Connections

Standard Connectors vs Managed Connectors

Understanding Modern Data Pipeline Orchestration

Introduction to Lakeflow Jobs ·

How Jobs, Tasks, and Dependencies Work

Building Your First Lakeflow Job

Preparing Notebook and SQL File Tasks

Creating a Silver Transformation Notebook

Building Data Quality Check SQL Tasks

Creating Gold Aggregation Outputs

Connecting Tasks with Dependencies

Running an End-to-End Lakeflow Job

Validating Silver, Quality Check, and Gold Outputs

Understanding Workflow Automation in Databricks

Conditional Workflows in Lakeflow Jobs

Building If/Else Logic in Pipelines

Creating Alternative Execution Paths

Using Job Metadata in Conditions

Running and Validating Conditional Pipelines

Building Dynamic Pipelines with For Each Loops

Creating Parameterized Notebooks

Passing Loop Inputs into Notebook Tasks

Running Multiple Iterations Automatically

Using Concurrency in Loop-Based Pipelines

Monitoring and Debugging Databricks Jobs

Understanding Successful, Skipped, and Failed Tasks

Debugging Task Outputs and Execution Details

Managing Quota Limit Issues in Databricks Free Edition

Using SQL Rows Output in Dynamic Pipelines

Generating Loop Inputs from SQL Query Results

Connecting SQL Output Rows to For Each Loops

Building Data-Aware Dynamic Pipelines

Testing Pipelines with New Incoming Data

Driving Pipelines with First Row Logic

Selecting the Most Frequent Order Status

Passing First Row Values Between Tasks

Creating Focused Gold Analysis Tables

Data-Driven Pipelines with Dynamic Parameter Passing

Generating Dynamic Workloads from Silver Tables

Using Task Values to Share Data Between Tasks

Building Warehouse and Shipping-Based Pipeline Logic

Creating If/Else Controls for Empty Workloads

Passing Multiple Parameters into Notebook Tasks

Building Delivery Summary Outputs with Dynamic Parameters

Using SQL Tables as Dynamic Loop Inputs

Creating SQL Lookup Tables for Pipeline Control

Reading Lookup Tables as SQL Row Outputs

Creating Target Gold Tables Before Loop Execution

Running SQL Table-Driven For Each Pipelines

Avoiding Duplicate Results in Repeated Pipeline Runs

Building Production-Ready Data-Driven Workflow Patterns

Designing Scalable Lakeflow Pipeline Architectures

End-to-End Pipeline Automation with Databricks Lakeflow
Requirements

A basic understanding of Databricks fundamentals, including notebooks, clusters, and Delta tables, is recommended

A working computer (Windows, Mac, or Linux)

A stable internet connection to access Databricks

Access to Databricks Free Edition or any Databricks workspace

Basic understanding of SQL

Basic understanding of Python is helpful but not mandatory

Basic familiarity with data tables, columns, and simple queries

Interest in data engineering and real-world data pipelines

Curiosity about workflow orchestration and pipeline automation

Motivation to build dynamic, scalable, and production-ready data workflows

No prior Lakeflow experience required

No advanced Spark knowledge required

Just you, your keyboard, and your passion for becoming a modern data engineer!