DataFrame Expectations

DataFrameExpectations is a Python library designed to validate Pandas and PySpark DataFrames using customizable, reusable expectations. It simplifies testing in data pipelines and end-to-end workflows by providing a standardized framework for DataFrame validation.

Instead of using different validation approaches for DataFrames, this library provides a standardized solution for this use case. As a result, any contributions made here, such as adding new expectations, can be leveraged by all users of the library.

See the starter guide here. See the complete list of expectations here.

Contents:

• Getting Started
  • Installation
  • Requirements
  • Quick Start
  • Validation Patterns
  • Advanced Features
  • Development Setup
  • Contributing
  • Security
• Adding Your Expectations
  • Understanding Expectation Categories
  • Defining Your Expectations
  • Adding Aggregation-Based Expectations
  • Custom Expectations with Full Control
  • Automatic Integration with DataFrameExpectationsSuite
  • Adding Unit Tests
  • Updating the Documentation and Stubs
• Expectation Gallery
  • Expectations Summary
  • Column Aggregation Expectations
  • Column Expectations
  • DataFrame Aggregation Expectations
• API Reference
  • Core Infrastructure
  • Expectation Registry
  • Result Messages
  • Utilities
  • Exception Classes