Metadata Writeback
Write quality-controlled Hoppa metadata back to the originating document system (SharePoint, Autodesk ACC / BIM 360) as custom columns/attributes.
The subpackage exposes a uniform, source-agnostic contract that a caller (the Hoppa
API) drives over any supported source. Each source has one WritebackConnector,
resolved from a registry by storage_provider:
from workbench.writeback import get_writeback_connector, FieldMapping, MappedField, WritebackReport
connector = get_writeback_connector(document_version.storage_provider)
# 1. Discover deduped write targets (pure — no source I/O)
targets = connector.discover_targets(session)
# 2-3. Read the target's columns/attributes, then auto-match to the standard
schema = connector.list_columns(target, client) # client injected by the caller
suggestion = connector.suggest_matches(schema, session.metadata_specification)
# 4. Reconcile source column definitions to the standard (report-only on Autodesk)
mapping = FieldMapping(target.target_key, [MappedField("document_summary", "Description")])
sync = connector.validate_columns(target, mapping, session.metadata_specification, client)
# Each result carries an explicit action (created/updated/unchanged/validated/missing)
# and the resolved column id + types; proceed when none carries an `error`.
blocked = [r for r in sync if r.error is not None]
# 5-6. Preview current-vs-new, then write per document (idempotent)
previews = connector.read_current_values(target, mapping, docs, session.metadata_specification, client)
results = [connector.write_values(target, mapping, d, session.metadata_specification, client) for d in docs]
report = WritebackReport.from_results(results, target_keys=[target.target_key])
Key design points:
- The caller injects the authenticated source
client(SharePointClient/AutodeskClient); the connector holds no auth state. - Values are QC-merged — writes use the user/AI-edited value (via
merged_value), never raw workflow output — and value coercion is per-source. - Capabilities differ per source. SharePoint can create and update column
definitions; Autodesk cannot (its
validate_columnsis report-only) and also reconstructs the file naming standard, which can rename the file. Drive UI from the wholeWritebackCapabilitiesobject.
Contract & types
The WritebackConnector abstract base class.
One concrete subclass per source system (SharePoint, Autodesk), registered in
workbench.writeback.registry.WRITEBACK_REGISTRY keyed by storage_provider
— mirroring the read-side workbench.bindings.connectors registry, but kept
separate because writeback is a distinct concern from read/download.
The connector — never the API — resolves Hoppa values to source field values, so
read_current_values and write_values take mappings, not pre-resolved
field values (see spec-metadata-writeback.md §6). This guarantees preview and
execute share one value path and cannot diverge.
Bases: ABC
Uniform writeback contract over one source system.
Concrete subclasses are stateless singletons registered by key (aside from
an internal per-target read cache). Class attributes advertise the connector's
capabilities so the API/frontend degrade gracefully without branching on the
source key.
Auth / clients. The API owns source-client construction (it already builds
SharePointClient(user_id=...) for reads) and injects the authenticated
client into each source-hitting verb via client. The connector holds no
auth state — this keeps it a pure singleton and makes it trivial to test with a
fake client. client is the connector's own source client type
(SharePointClient / AutodeskClient); each concrete connector documents
what it expects.
default_picklist_format_for(property_type)
The pre-selected format, or None when there is no choice to make.
describe_property(key, prop)
Describe one Hoppa property's writeback options for the mapping UI.
abstractmethod
discover_targets(session)
Group the session's documents into deduped write targets.
Pure (no source I/O): derives target_key from each document's
connection_details via
:func:workbench.writeback.targets.target_key_for. SharePoint targets are
site+drive; Autodesk targets are project+folder. display_name is
best-effort from local data — a richer label may need a source call the
caller makes separately. → UX Step 7.
abstractmethod
list_columns(target, client)
Read the target's writable columns/attributes and type vocabulary.
Returns the full mapping-step payload: normalized + native column types,
allowed values, supported_column_types, the Hoppa→source
type_compatibility map, and capabilities. → UX Steps 5-6.
picklist_formats_for(property_type)
The formats offered for one Hoppa property type, possibly none.
abstractmethod
read_current_values(target, mapping, documents, metadata_spec, client, custom_properties=None)
Current source value AND resolved new value, per doc per mapped field.
Both values come from the connector (one value path) so preview and execute never diverge. → UX Step 9.
abstractmethod
suggest_matches(schema, metadata_spec, custom_properties=None)
Auto-match Hoppa properties to existing source columns. → UX Step 5.
Pure: operates on the schema from :meth:list_columns so no second
source round-trip is needed. custom_properties (from
Session.custom_properties) are merged with the governed spec and matched
identically — pass them so ungoverned "custom columns" appear in the mapping.
abstractmethod
validate_columns(target, mapping, metadata_spec, client, custom_properties=None)
Reconcile source column definitions to the Hoppa spec (UX Step 8).
Returns one :class:ColumnSyncResult per mapped field, carrying an
explicit action and the resolved/created column id + types. SharePoint
creates/updates definitions to match (CREATED/UPDATED/
UNCHANGED); Autodesk is report-only — it reports VALIDATED /
MISSING and cannot alter ACC definitions. Idempotent; the user may
proceed when no result is blocking (none carries an error).
abstractmethod
write_values(target, mapping, document, metadata_spec, client, custom_properties=None)
Write one document's mapped values to the source. → UX Steps 11-12.
Per-document by design — the API owns the fan-out loop and progress rows.
Must be idempotent (safe under queue retries). May return a PARTIAL
status (Autodesk: attributes vs filename rename).
Source-agnostic data types for metadata writeback.
These are the serializable DTOs exchanged across the SDK↔API boundary. They are
plain dataclasses with a to_dict() that yields JSON-ready snake_case dicts
(enums collapse to their .value); the API layer re-maps to its own camelCase
Pydantic models. No HTTP or framework types leak in here — the SDK stays
framework-agnostic.
See spec-metadata-writeback.md §4 for the contract these implement.
Bases: StrEnum
What validate_columns did to (or found for) one mapped column.
SharePoint (can mutate definitions): CREATED / UPDATED / UNCHANGED.
Autodesk: VALIDATED (exists and compatible) / CREATED (new definition
added at the project root) / MISSING (mapped to an id that isn't there).
ERROR marks any blocking failure — the reason is
on ColumnSyncResult.error. Non-blocking actions are
CREATED/UPDATED/UNCHANGED/VALIDATED; the caller may proceed
when no result is ERROR/MISSING (equivalently, when no result carries
an error).
dataclass
Bases: _DTO
One writable source column/attribute, normalized for the mapping step.
Bases: StrEnum
What a writable field is in the source system.
SharePoint fields are all COLUMN. Autodesk splits into ATTRIBUTE
(custom attribute, set by id) and NAMING_PART (a component of the file
naming standard — writing it reconstructs and renames the file).
dataclass
Bases: _DTO
Per-mapped-field outcome of validate_columns (UX Step 8).
Explicit action (never inferred from create_new) plus the resolved or
newly-created source_column_id and its normalized/native types, so the API
can pass column ids + types straight through and the frontend can render an
authoritative "prepared N columns" readout. error is set (and action is
ERROR or MISSING) when the field is blocking.
notice is advisory and never blocking: something the user should know that
does not stop the run (e.g. a picklist format switch leaving the old
vocabulary in place). error blocks; notice informs.
dataclass
Bases: _DTO
Current-vs-new values for one document (UX Step 9).
error is set (and the current values left blank) whenever the document's source item
could not be read; the SDK surfaces the source's own message rather than classifying
per-source error codes. The one deliberate exception is not-found (404): the item no
longer exists (deleted/moved) — a terminal state — so available is set False and the
app can flag/exclude it. Every other read error keeps available True (the item may
still exist; the app shows the message and lets the user decide).
dataclass
Bases: _DTO
Outcome of writing one document (UX Steps 11-12).
Returned per-document — the API owns the fan-out loop and persists these as
polling rows. status may be PARTIAL (Autodesk: attributes vs rename).
dataclass
Bases: _DTO
The confirmed mapping for one target: a list of :class:MappedField.
overwrite_existing=False is "fill in the blanks only": a field that
already holds any value in the source is left alone, even if Hoppa would
write something different. Only genuinely empty fields are filled.
It is enforced at write time rather than by filtering upstream, because that is the only place the guarantee holds — a caller filtering on a preview is working from a snapshot, and a value written by someone else in between would be clobbered anyway. It is per-target-and-run, not per-field: the same answer applies to every mapped field on every document in the run.
by_key()
Index the mapped fields by Hoppa property key.
dataclass
Bases: _DTO
Current-vs-new value for one mapped field on one document.
changed always describes the value diff — whether new differs from
current — and nothing else. skipped says the write will be withheld even
so. Keeping them separate is what lets the UI tell "keeping the existing
value" apart from "no change needed": both write nothing, but only the first
is discarding a value Hoppa would otherwise have written.
changed=True, skipped=True → preserving current, withholding new
changed=False, skipped=False → already correct, nothing to do
changed=True, skipped=False → will be written
current_value and new_value stay populated when skipped, so the user
can see both the value being kept and the one being withheld.
error marks a field that cannot be written — today, a classification whose
code is no longer in the standard. new_value is then None and changed
False: the preview shows the field as unwritable rather than inventing a value
for it, and write_values reports the same field as FAILED.
dataclass
Bases: _DTO
Outcome of writing one mapped field on one document.
dataclass
Bases: _DTO
One Hoppa-property → source-column mapping decision.
source_column_id names an existing column; create_new requests a new
one (rejected by connectors whose can_create_definitions is False).
Properties absent from a :class:FieldMapping are skipped at writeback.
picklist_format selects what a classification is written as, and must be
drawn from the connector's supported_picklist_formats. It governs the
column's choice list and the value together — they share one derivation — so
changing it changes both. No effect on non-classification properties; tags
are label-only.
dataclass
Bases: _DTO
Auto-match result for one target (UX Step 5).
matched and needs_update both point at existing columns;
needs_update additionally requires a definition sync (e.g. choice list
out of date). to_create / unmatched hold Hoppa property keys.
properties describes every property considered, across all four buckets,
so the mapping UI can render each row's options without re-deriving them.
dataclass
Bases: _DTO
A Hoppa property as the mapping step sees it, with its writeback options.
picklist_formats is what the user may choose from for this property on
this connector — the support is a function of both, which is why it is not a
connector-level flag. Empty means no format choice, for either reason:
- the property is not a picklist at all (freetext, date); or
- it is a picklist whose label is fixed —
tagsentries are{id, title}and the id is an internal handle, so a tag is always written as its title.
So the frontend never has to know what a tag is: an empty list means show
no picker. A single entry means the format is fixed (an ACC attribute's
allowed values belong to ACC, so ["code"]), and several means offer a
choice. default_picklist_format is None exactly when the list is empty.
dataclass
Bases: _DTO
A blocking issue from validate_columns (UX Step 8).
message carries the source's own error verbatim where relevant (e.g.
"Choice column has conflicting values", or an Autodesk missing-attribute).
Surfaced inline on the failing :class:ColumnSyncResult.
dataclass
Bases: _DTO
An auto-suggested mapping of a Hoppa property to an existing column.
dataclass
Bases: _DTO
Everything the mapping step (UX 5-7) needs for one target.
type_compatibility maps a Hoppa property type (e.g. "classification")
to the source WritebackDataTypes it may legally map to, so the frontend
can flag conflicts and constrain the create-new type picker.
Bases: StrEnum
Outcome of writing one document.
PARTIAL is required for Autodesk, where custom attributes and the
filename rename are two independent writes that can each fail.
dataclass
Bases: _DTO
Per-connector capability flags driving graceful UI degradation.
The whole object drives behaviour — never branch on a single flag. Autodesk
can create attribute definitions via API but cannot edit existing ones, so
can_create_definitions is True and can_update_definitions is False.
Picklist-format support is deliberately not here: it varies by property
type as well as by connector (tags are label-only whatever the connector), so
it is declared per property on :class:PropertyDescriptor.
Bases: StrEnum
Normalized column/attribute data type, source-agnostic.
Each connector maps its native types onto this vocabulary and preserves the
source's own label in ColumnDescriptor.native_type for display.
dataclass
Bases: _DTO
Aggregate of a writeback run — the API logs this as audit "Raw details".
Assembled from the stream of :class:DocumentWriteResults. counts holds
at least total/succeeded/failed/partial/skipped.
classmethod
from_results(results, *, target_keys=None, operation='WritebackResults')
Tally a list of per-document results into a report.
dataclass
Bases: _DTO
A unique write destination within a session.
scope holds the structured source locator — SharePoint
{site_id, drive_id}; Autodesk {hub_id, project_id, folder_id}.
target_key is derived by :func:workbench.writeback.targets.target_key_for
and is the join key across every writeback call.
status_from_fields(field_results)
Roll one document's per-field outcomes up to its document status.
Shared by every connector so the meaning of a status cannot drift between sources. Precedence, most severe first:
- nothing to report →
SKIPPED - every field failed →
FAILED - some field failed →
PARTIAL(a bad field never sinks the document: Autodesk's attributes-vs-rename split, or one unresolvable code) - anything was written →
WRITTEN; filling one blank while preserving another still counts as a write - anything was withheld →
SKIPPED; reserved for documents nothing was written to because existing values were preserved - otherwise →
UNCHANGED; nothing needed doing and nothing was withheld
Target identity
Write-target identity — the single source of truth for target_key.
The SDK owns target-key derivation because the source-scope ids are not part of
the canonical connection_details projection (workbench/clients/metadata.py)
— they are tree-client-injected and provider-shaped. SharePoint carries
site_id / drive_id at the top level; Autodesk's write-scope lives in the
provider-specific raw.relationships sub-tree, and its hub_id / project_id
may be absent. No layer above the SDK should parse connection_details / raw
— they call :func:target_key_for instead so both sides derive the identical key.
Formats (see spec-metadata-writeback.md §4a):
- SharePoint:
sharepoint:{site_id}:{drive_id} - Autodesk:
autodesk:{project_id}:{folder_id}— folder-scoped, because ACC custom attributes and naming standards vary per folder.
autodesk_scope(connection_details)
Extract the Autodesk write-scope from a connection-details blob.
Returns {hub_id, project_id, folder_id, version_id} (any may be None).
Centralises all knowledge of the provider-specific raw.relationships shape
so no layer above the SDK parses it — the connector and :func:target_key_for
both read scope through here.
target_key_for(connection_details)
Derive the stable target_key for a document's source location.
Returns None when the provider is unsupported or the required scope ids
are not resolvable from the blob (the caller treats that document as having
no write target).
Value access
The single seam for reading quality-controlled metadata values.
Writeback must write the QC-merged value, not raw workflow output. That merge
already happens inside the binding: DocumentVersion.results.body overlays
workflow results (method="workflow") with AI edits (method="cleanup") and
then user edits (method="user", applied last and therefore highest priority)
— see AzureBlobResults.body / _get_results in workbench/bindings/azure.py.
So this module does not re-implement the merge; it is the one place the
connectors read merged values through, keeping value access uniform and giving a
single point to adapt if the overlay model ever changes. Connectors call
:func:merged_value / :func:merged_values rather than touching results.body
directly.
Note: the returned PropertyValue.value shape varies by provenance — a workflow
classification is {"code", "description"}, whereas a user edit carries whatever
the edit JSON stored. Turning that into a source-native field value is the
connector's job (per-source coercion), not this module's.
document_label(document)
Human-facing document name for display (previews, results).
Preference order: name → file_name → id. On external sources
DocumentVersion.name is the source file name (e.g. SharePoint's
A-101.pdf) while file_name is a storage-facing name derived from the
id (the config-blob / zip-member name — see DocumentVersion.file_name), so
name must win. For direct uploads the two coincide. file_name and
id are last-resort fallbacks for the rare doc with no source name.
is_blank(value)
Is a source field effectively empty, i.e. free to fill?
The judgement behind FieldMapping.overwrite_existing=False, and the reason
that flag is enforced here rather than upstream: "empty" depends on the
source's value shape, which is the connector's knowledge. A field is blank
when it is absent, null, an empty/whitespace-only string, or an empty
collection (a multi-select with nothing selected).
Deliberately not not value: 0 and False are values a user
chose, and treating them as blank would overwrite real data — the precise
failure this flag exists to prevent.
A column's default value is likewise not blank. Once SharePoint has applied a default the field holds it, and nothing distinguishes it from a deliberate pick of the same option — so it counts as present and is preserved. That follows the rule as agreed: any value present is left alone.
merged_properties(metadata_spec, custom_properties=None)
Governed spec properties overlaid with ungoverned custom columns.
Custom columns (from Session.custom_properties) share the governed-property shape, so
the mapping / validate / write path treats them identically — every connector method
merges them here rather than special-casing. None/empty leaves the governed set
unchanged, so a backend without custom columns behaves exactly as before.
merged_value(document, property_key)
Return the QC-merged :class:PropertyValue for one property, or None.
None means the document has no value for property_key (the property is
skipped at writeback). Reading results.body may trigger a lazy load from
storage; callers that iterate many properties should hold the result of
:func:merged_values rather than calling this repeatedly.
merged_values(document, property_keys=None)
Return the QC-merged values for a document as {property_key: value}.
Loads results.body once. With property_keys given, returns only those
keys that have a value; with None, returns every property present on the
document. Properties absent from the results are omitted (not None-valued),
so key in result is a clean "has a value to write" test.
Connectors
SharePoint metadata-writeback connector.
Productionizes notebooks/templates/04_sharepoint_metadata_writeback.ipynb into
the uniform :class:WritebackConnector contract, reusing the pure mapping helpers
in workbench/clients/sharepoint_helpers.py and the Graph calls on
SharePointClient.
SharePoint models metadata as list columns: a document library drive is backed by a list, whose columns are field definitions the connector can create and update, and whose list items hold the per-document field values. The connector caches the (drive→list, columns) context per target so a writeback loop over many documents pays those reads once.
Bases: WritebackConnector
Writeback connector for SharePoint document libraries (Microsoft Graph).
client on every source-hitting verb is a SharePointClient the API
constructs and injects.
clear_cache(target_key=None)
Drop cached target context (all, or one). The API may call between runs.
Autodesk ACC / BIM 360 metadata-writeback connector.
Autodesk models document metadata two ways, and this connector writes both:
- Custom attributes — folder-scoped definitions (
get_folder_named_attributes) set on a document version by numeric id (post_version_custom_attributes). New definitions can be created (create_folder_custom_attribute, at the project root so subfolders inherit them), but existing ones cannot be altered — so a mapping to an existing attribute is validated report-only: a picklist missing a value it needs is an error to fix in Forma, not something to sync. - Naming standard — a folder-scoped, ordered set of filename parts
(
get_folder_naming_standard). Writing these reconstructs the filename and renames the version (patch_version_attributeswith{"name": ...}).
Both are folder-scoped, so a write target is a project + folder (see
targets.autodesk_scope / target_key_for); the per-document version (tip)
rides on each document. Attributes are written before the rename — doing it the
other way round can lock the file. A document can therefore end PARTIAL when one
of the two writes fails.
Current values come from two places, because Forma keeps them in two: naming parts
are parsed back out of the filename, and attribute values are batch-read
(get_versions_custom_attributes). That read is best-effort — on failure preview
degrades to blank current values rather than raising. Only value-bearing attributes
come back, so an unset one is simply absent and previews as blank.
Forma has no multi-value attribute type and its array type is a
single-select drop-list, so tags are written — and created — as text holding a
comma-separated join. Picklist vocabulary and written values share one derivation
(clients.picklist); see :meth:AutodeskWritebackConnector._acc_create_spec.
Bases: WritebackConnector
Writeback connector for Autodesk ACC / BIM 360.
client on every source-hitting verb is an AutodeskClient the API
constructs and injects.
build_filename(naming_standard, values_by_field, file_type)
Reconstruct a filename from the naming standard and per-field values.
Returns (filename, {field_name: status}). values_by_field should hold
the intended value for every field (mapped Hoppa value, else current segment,
else None to fall back to defaults).
coerce_acc_value(prop_type, value)
Coerce a Hoppa PropertyValue.value to an ACC field string.
Classifications resolve to their code (ACC picklists/naming parts use the
short code, e.g. "A", not the description). Tags join to a comma list.
parse_current_segments(file_name, naming_standard)
Split a current filename into {field_name: segment} by the standard.
Best-effort positional split on the delimiter (extension stripped). Lets unmapped naming fields keep their current value during a rename instead of being reset to a default.
resolve_autodesk_labels(targets, client)
Upgrade Autodesk target labels from opaque ids to human project names, in place.
discover_targets runs without a client, so Autodesk targets start with a provisional
"Autodesk · <project-guid>" label. Given a client, this resolves the real project
name via the Forge Data Management API and rewrites display_name. Non-Autodesk
targets are left untouched, and it is best-effort: any Forge failure leaves the
provisional label in place rather than raising.
Cost is minimised: one get_projects(hub_id) call per DISTINCT hub across the targets
(usually one for a whole session) — independent of the number of targets/projects. The
call for a hub is reused across every target in that hub. (The hub/account name would be
a fixed +1 get_hubs() call if richer labels are ever wanted.)