Patch Changelog

Data Packages are code libraries with a live connection to an underlying data source. They provide a powerful interface for querying, access control, versioning, performance optimization and more; over all your data, no matter where it lives. This could be any database or file system.

The Data Package can be installed using a package manager like pip . Then, it's imported like a library dependency into your code, whether that’s a backend service performing machine learning or enrichment tasks, a customer-facing application, or even an external consumer buying access from you directly to build using the package.

A dpm-agent intelligently routes queries submitted by consumers of a Data Package to the appropriate backend source, enforces access policies and applies performance optimizations. 

Today, we're excited to announce support for generated Python packages!

Sign up for early access at www.dpm.sh!

Data Packages are code libraries with a live connection to an underlying data source. They provide a powerful interface for querying, access control, versioning, performance optimization and more; over all your data, no matter where it lives. This could be any database or file system.

The Data Package is imported like a library dependency into your code, whether that’s a backend service performing machine learning or enrichment tasks, a customer-facing application, or even an external consumer buying access from you directly to build using the package.

A dpm-agent intelligently routes queries submitted by consumers of a Data Package to the appropriate backend source, enforces access policies and applies performance optimizations. 

Today, we're excited to announce support for generated Node.js & TypeScript packages!

Today, upstream schema changes break data pipelines and subsequently break application code. Data Packages prevent this by decoupling the client from the infrastructure to enable stable versions of a dataset to continue being served even if upstream systems have broken.

Next, Data Packages decouple application code from specific data infrastructure so that a dataset can be moved to different storage systems by data teams without impacting application teams.

Finally, Data Package versioning enables rollbacks & proper testing against newer versions  - achieving a true CI/CD flow.

Data packages make engineers immediately productive with data by decoupling applications from data infrastructure.

You can read more on our blog, our docs, or get in touch to request access. The alpha release supports Typescript and Python, and the project will be open sourced soon.

Patch now supports a versioning workflow to safely update source tables & schemas, while continuing to serve production traffic. Simply run the following command, update your dataset, and Patch will increment the version. Then, you can write queries against the new version in your dev environment, while the previous version continues to serve traffic with fresh data. Once you're done testing, simply promote the new code and version to production. Learn more in our docs!

$ pat dataset update <my_dataset_name> --edit-tables
...
Dataset update submitted! Your dataset will now be updated to version 2.

Patch customers connect offline sources like data warehouses, so they can run complex queries with ultra low latency from their application. Today, we're excited to announce our Databricks integration. In just a few minutes, you can connect your cluster, define a dataset over your data lake, and query data from a customer-facing environment. Check out the docs to get started!

Datasets can include multiple tables. Now, you can run queries that join data from one or more tables, including aggregations and filters. You simply define an edge relationship and start querying. Check out our docs for examples and more details!

query {
  salesTransactionsQuery(
    orderBy: [{ dateOccurred: desc }]
    ) {
    dateOccurred
    transactionItem {
      idCount
      type
      }
    }
}

We are excited to announce some improvements to the onboarding experience with Patch. When users first log in, there is now a demo Snowflake source already connected with sample data. Check out our new Quick Start guide to build your first dataset and query live data in just a few minutes!

Historically, Patch has supported creating datasets from tables in the data warehouse. Now, users can create datasets from materialized views and views in BigQuery. Check out our docs to connect BigQuery or create your first dataset!

We're excited to announce a major update to our platform! We've added support for GraphQL queries with Boolean filters, which will give users the ability to apply AND and OR operators across filters, including nesting. Check out our docs for more info and examples!

Ex: "items that have 'shirt' and 'cotton' in the name, OR items that are of type 'top' and season 'summer'"

itemsQuery(filter: {
    or: [
        { and: [{ name: { like: "%shirt%" } }, { name: { like: "%cotton%" } }] },
        { and: [{ item_type: { eq: "top" } }, { season: { eq: "summer" } }] }
    ]
} ) { ... }

Patch customers can now use Key Pair Authentication as an alternative to password to connect Snowflake as a data source. Run pipx upgrade patch-cli to see the command tree below and check out our docs for more info.

The Patch CLI is now available on PyPI. Users can install and upgrade the package with pip or pipx. A login is still required, so head to www.patch.tech to sign up for our waitlist and we'll get back to you asap.

~ pipx install patch-cli
  installed package patch-cli 0.2.0, installed using Python 3.9.13
  These apps are now globally available
    - pat
  done! ✨ 🌟 ✨

Many teams have multiple customers' data in the same table. In a customer-facing feature, you only want to display that particular customer's data.  To ensure you can safely limit customer access to the right data, you can now rely on row-level access controls encoded in JWTs.  Patch can issue JWTs or respect ones that you supply with the filters encoded.

$ pat access signing-template create <name> <JWK file> --source snowflake_prod --dataset exec_dashboard --filters user_events:account_id:384752

We've simplified the BigQuery connection flow, fixed a bug related to empty tables & null values, and updated docs. Check them out here!

When querying datasets, you've always been able to use common aggregations like avg, sum, max, and min. Now, you can also aggregate by any numeric column's median. Check out our docs for more info.

query {
  engagementTableQuery(
    filter: {
      videoViewStartTime: {
        after: "2022-05-31T00:00:00"
        before: "2022-11-01T00:00:00"
      }
    }
    groupBy: [videoViewStartTimeYear, videoViewStartTimeMonth]
    orderBy: [
      { videoViewStartTimeYear: asc }
      { videoViewStartTimeMonth: asc }
    ]
    limit: 1000
    offset: 0
  ) {
    videoViewStartTimeYear
    videoViewStartTimeMonth
    videoViewDurationMedian
  }
}

You can now connect to BigQuery using the CLI or programmatically via the Warehouse API. Check out our docs here.

You can now authenticate with the Patch Data APIs using Data Access Keys without an expiry. This enables you to call the APIs from a production service without having to refresh standard bearer tokens. Check out our docs for creating, listing and revoking Data Access Keys.

Today, we introduced support for ARRAY columns. This means you can run queries that return ARRAY column values, but also write filters that check if a column hasAny or hasAll values in an array you specify. This is helpful when supporting features like multi-select filters, including the case when you are aggregating the result set.

query {
  eventsTableQuery(
    filter: {tags:{hasAny: ["prod", "bot_user"]}}
    groupBy: [eventType, eventCreatedTimeDate]
  ) {
    eventType
    eventCount
    eventCreatedTimeDate
  }
}

When querying datasets, you can now look up data for the "last 30 days" or "between 7 and 14 days ago". Simply specify a filter on a timestamp or datetime column with the newerThan and/or olderThan operators. Check out our docs here!

query {
  eventsTableQuery(
    filter: {
      eventCreatedTime: {
        newerThan: {days: 7},
          olderThan: {days: 3}
      }
    }
    orderBy: {eventCreatedTimeDate: desc}
    groupBy: [eventType, eventCreatedTimeDate]
  ) {
    eventType
    eventTypeCount
    eventCreatedTimeDate
  }
}

When querying time series datasets, you can now easily group data by week. A new derived field <columnName>WeekDate will return the first day of the week the time value falls within. With an ISO-standard default of MONDAY, the firstDayOfWeek is configurable with an options configuration parameter.

query {
  <tableName>Query(
    groupBy: [eventCreatedTimeWeekDate]
    orderBy: {eventCreatedTimeWeekDate:desc}
    options: {firstDayOfWeek: SUNDAY}
  ) {
    eventCount
    eventCreatedTimeWeekDate
  }
}

When you create a dataset, you can now immediately inspect the generated GraphQL schema and run example queries using the dataset playground. In the CLI, simply run $ pat dataset playground <dataset_name> and follow the prompts. Learn more in our docs!

When querying datasets with the GraphQL Data APIs, you can now bucket time series data by common grains (day, week, etc) and aggregate data within time buckets. You can also groupBy other dimensions to segment the data. Check out our docs for more info!

query {
    purchasesQuery(
      filter: {
        item_type: {
          in: ["pants", "skirts", "shorts"]
        }
        payment_status: {
          eq: "successful"
        }
      }
      groupBy: [purchase_timeDay,item_type]
    ) {
      purchase_amountSum 
      purchase_timeDay 
      item_type 
    }
}

For users that manage connections to their customers' data warehouses, such as Warehouse-Native SaaS companies, it is now possible to create sub-Tenants for each new customer, without having to create Users as well. The multi-tenant account (MTA) can also use the Warehouse API on behalf of their sub-tenants in order to create and manage Sources or discovering their table catalog. Check out our docs for more info.

Patch now generates GraphQL APIs to query datasets. Today, users can look up records by primary key value or list records with standard filters, limits, and offsets. Learn more about the generated schemas and common queries in our documentation!

query {
  sales_transactionsQuery(filter: {
    item_type: {
      eq: ‘shirts’
    }
    transaction_amount: {
      ne: null
    }
  }) {
    dateOccurred
    item_name
  }
}

You can now edit a subset of Source configuration fields using the sourceUpdateSnowflake endpoint. Editable fields include user, password, and warehouse. This is essential for credential rotations.

When creating datasets, it's helpful to verify that individual rows of data match between the source table in your data warehouse and the data in Patch. You can now easily look up records by primary key.

$ pat dataset inspect test_columns uk_real_estate_records_fl pk_6H2rAn0vuIGeNyeBSJsB
Active source: snowflake-source
                     Table uk_real_estate_records_fl                     
                    ╷                         ╷                          
  Column            │ Cache                   │ Source (count: 1)        
╶───────────────────┼─────────────────────────┼─────────────────────────╴
  city              │ WASHINGTON              │ WASHINGTON               
  county            │ TYNE AND WEAR           │ TYNE AND WEAR            
  date_of_transfer  │ 2004-05-14              │ 2004-05-14               
  district          │ SUNDERLAND              │ SUNDERLAND               
  duration          │ F                       │ F                        
  old_new           │ N                       │ N                        
  ppd_category_type │ A                       │ A                        
  price             │ 167000                  │ 167000                   
  property_type     │ D                       │ D                        
  record_status     │ A                       │ A                        
  transaction_id    │ pk_6H2rAn0vuIGeNyeBSJsB │ pk_6H2rAn0vuIGeNyeBSJsB  

When querying data with the Patch Query API, you can now use the _agg parameter without _group_by. For example, you can now COUNT or SUM all rows, instead of just the rows within a segment. You can find our documentation here.

When querying datasets, you can now filter for time ranges, such as "Last 7 Days", "Between Jan 1 and Mar 30" or "Before Dec 31". You can find the documentation here.

We launched a new documentation site at https://docs.patch.tech.