Column groups

Introduction

BMS allows you to group columns together, and set policies for updates affecting these groups, in order to better reflect their logical hierarchy and to enforce consistency when updating. This is especially useful when a model has many columns that might be queried or updated at different times or cadences.

Groups, like columns, are stored per version and can be changed by performing updates.

Group hierarchy

Groups can be nested inside each other, up to a maximum depth of 4.

When referring to columns inside groups, or groups inside groups, it is generally necessary to use their qualified titles. The qualified title of a resource is formed from the titles of all groups up to that resource, followed by the resource's bare title. These are separated by the qualified title separator, which can be set per-request (for both update and query) but defaults to ▸ (U+25B8 BLACK RIGHT-POINTING SMALL TRIANGLE).

An example hierarchy, with bare and qualified titles for each resource, is as follows:

• Group Au_grade: qualified title Au_grade
  • Column mean: qualified title Au_grade▸mean
  • Group ensemble: qualified title Au_grade▸ensemble
    • Column realisation_1: qualified title Au_grade▸ensemble▸realisation_1

As such, a CSV update file affecting both of these columns could look like:

i, j, k, Au_grade▸mean, Au_grade▸ensemble▸realisation_1
0, 0, 0, 1.0,           1.0

Note that the qualified title separator is always a single character, with no spaces as padding. It is not possible to choose a separator that appears in a column or group title, or to create a column or group whose title includes the separator.

Group title and column title uniqueness are validated in separate namespaces, so a column and group can share the same qualified title, but two groups cannot. This can be useful to represent additional data attached to a column.

Bare vs qualified titles

Resources have two naming forms depending on the operation:

• Bare title: The immediate name of a column or group on its own (for example, mean or Au_grade).
  • Namespaces: Bare titles are scoped to their parent. Grouped columns must be unique within their group; ungrouped columns and root groups must be unique across the block model. Columns in different groups can share the same bare title (for example, Au_grade▸mean and Cu_grade▸mean).
  • Defining and renaming: Bare titles are used when creating a resource (title in columns.new or groups.new) or renaming it (new_title in columns.rename, or title in update_metadata.values). Renaming changes only the bare title—it never moves a column or group to another parent.
  • Ungrouped columns always use bare titles.
• Qualified title: The complete path from the root group down to the resource, with segments joined by the separator (for example, Au_grade▸ensemble▸realisation_1).
  • Referencing existing items: Qualified titles are used when referencing existing grouped columns in updates (columns.update, columns.delete, columns.rename[].title, columns.update_metadata[].title), in data upload headers (CSV or Parquet), and in query column lists (columns).
  • Referencing groups: Qualified group paths are used when specifying a group for a column (group), a parent for a child group (parent_group), targeting a group for deletion (groups.delete) or metadata updates (groups.update_metadata[].title), and setting policy overrides (group_missing_column_override).

Hidden groups

If a group contains many columns, it may be helpful to hide these by default when performing queries. By setting is_hidden when creating or updating a group, its columns are excluded from wildcard (*) queries. For example, if you have the following hierarchy:

• Column Density
• Group Au_grade, is_hidden left as false
  • Column mean
  • Group ensemble, is_hidden set to true
    • Columns realisation_1 up to realisation_10

Then a query requesting the column set ["*"] will return Density and Au_grade▸mean, but not any of the Au_grade▸ensemble▸realisation columns. However, a query requesting specifically ["Au_grade▸ensemble▸realisation_1"] will still return that column.

Hidden groups will still be present in the responses for other requests, along with their columns, so it is still possible for clients to determine that they exist and request them explicitly. Hiding a group also hides any columns in nested groups, regardless of their depth or is_hidden status. This means that child groups cannot opt back into being visible.

When performing a query with include_hidden set, the is_hidden status of groups has no effect, and their columns are included by wildcard expansion.

Missing column policies

If columns in a group ought to be updated together, you can enforce this coupling by setting the missing_column_policy field on the group when creating or updating it. This policy affects what happens when an update touches some columns in a group but not others. The available values are:

• USE_PREVIOUS: Columns not included in the update retain the values they had beforehand.
• SET_NULL: Acts as if all rows in the update have NULL for the missing columns.
• REJECT: Prevents the update from occurring if some columns in the group are updated but others are missing.
• INHERIT: Use the same policy as the parent group, or USE_PREVIOUS if there isn't a parent.

INHERIT is the schema default; however if there are no explicitly set policies anywhere in a group's parent chain, then the effective behaviour is USE_PREVIOUS.

Note that SET_NULL does not set all blocks' values to NULL, only the ones actually touched by the update. For example, for a block model like:

i, j, k, Au_grade▸mean, Au_grade▸variance
0, 0, 0, 1.0,           0.5
1, 0, 0, 2.0,           0.75

Supposing the group Au_grade has policy SET_NULL, then this update:

i, j, k, Au_grade▸mean
0, 0, 0, 1.2

Produces:

i, j, k, Au_grade▸mean, Au_grade▸variance
0, 0, 0, 1.2,           NULL
1, 0, 0, 2.0,           0.75

Inheritance and policy zones

When nesting groups inside other groups, the INHERIT policy becomes important. This is because any groups inheriting from a parent group are merged into a single zone for the purpose of policy evaluation.

For example, with this hierarchy:

• Group Au_grade: SET_NULL
  • Column mean
  • Group percentiles: INHERIT (inheriting from Au_grade, so joining its zone)
    • Column 10
    • Column 50
    • Column 90
  • Group ensemble: REJECT (not inheriting, so this is a new zone)
    • Column realisation_1
    • Column realisation_2
    • Column realisation_3

An update changing only mean would succeed, but would set values in 10, 50, and 90 to NULL, since these are in the same zone (since percentiles inherits the policy of Au_grade). However, the values in the realisation columns would be unaffected. Similarly, an update changing values in 10, 50, and 90 would set mean to NULL, since zone membership is symmetric.

Conversely, an update changing only realisation_1 would be rejected, since that column is in the ensemble zone (which is separate from Au_grade, and doesn't interact with it).

Policy override

If you need to override the policy of a group for a single update request, you can use the group_missing_column_override field on that update, and specify the qualified group titles to override, along with the policies to apply. Currently it is only possible to apply USE_PREVIOUS, since the main use case of overrides is to weaken validation when intentionally changing only part of a group. For instance, this could be used to fix an incorrect value in one member of an ensemble, without affecting the rest. Note that overrides apply to only a single group; child groups, and other groups in the same zone, are unaffected and retain the policies they previously inherited.

Group updates

This section covers the mechanics of updating groups and referencing columns within updates. For general information on block model updates, see the update guide.

There are two primary places in the update endpoint where groups are specified:

• The groups object, which allows you to create (new), delete (delete), and modify (update_metadata) groups;
• The group field on individual columns when creating a column or updating a column's metadata, which assigns or moves a column to a group.

Pre-state vs post-state resolution

When referencing resources in API operations, the resolution rules depend on the operation and the role of the reference:

• Queries resolve against the pre-state (the queried version): When querying an existing version, all column titles and group paths resolve against the group hierarchy as it already exists in that version.
• Group targets resolve against the pre-state: groups.update_metadata[].title and groups.delete[] identify groups by their paths in the existing hierarchy.
• Update destinations and column references resolve against the post-state: Paths used to assign or move a group or column, and references to existing columns, are resolved against the hierarchy resulting from the update.

These rules mean:

1. New groups can be referenced immediately: A group defined in groups.new can be referenced by name in the same update—for instance, as a child group's parent_group or as a new column's group.
2. Renamed or moved groups: Use the new (post-state) path when assigning or moving a group or column, or when referencing an existing member column. Use the existing (pre-state) path to target a group in groups.update_metadata[].title or groups.delete[].
3. Existing columns in modified groups: When targeting an existing grouped column (in columns.update, columns.delete, columns.rename, or columns.update_metadata), the target is identified by its new post-state group path followed by its current bare title.
   • Example: If group Assays is renamed to Au_grade in groups.update_metadata, an existing member column mean must be referenced as Au_grade▸mean in columns.update or columns.delete, as well as in the header of the uploaded data file.
4. Order independence: Update payloads are declarative. Operations do not depend on the order in which they appear in the JSON request; use the pre-state or post-state path according to the reference rules above.

Group lifecycle constraints

• Deleting a group: A group cannot be deleted if it would contain any columns or child groups in the post-state. You must delete, move, or ungroup all contained columns and child groups—either in previous updates or within the same update.
• Ungrouping and moving to root: The empty string "" is a special sentinel:
  • Setting a column's group to "" in update_metadata ungroups the column.
  • Setting a group's parent_group to "" in groups.update_metadata moves the group to the top level (root).
• Moving vs renaming columns: Renaming a column (columns.rename or values.title in columns.update_metadata) modifies only its bare title and leaves its group unchanged. Moving a column between groups is done by setting values.group in columns.update_metadata; this metadata-only change does not require uploading column data.