hsfs.core.sharepoint_sync #

SharePointSync #

A sync that copies a folder of a SharePoint document library into a dataset of the project.

The copy runs as an ordinary Python job, either once when started or on a cron schedule, with rsync semantics from SharePoint to Hopsworks. Create one with SharePointConnector.create_sync or SharePointConnector.copy_folder.

Every method that changes the sync needs the Data Owner role; reading its status and audit log needs project membership.

active_execution_id `property` #

active_execution_id: int | None

Execution that holds the sync's run claim, if a run is in progress.

alert_receiver `property` #

alert_receiver: str | None

Alert receiver notified when a run of the sync job fails or is killed.

audit_path `property` #

audit_path: str | None

Directory of the sync's audit log, with one date=YYYY-MM-DD partition per day of CSV files.

bytes_total `property` #

bytes_total: int | None

Size of the files in the dataset after the most recent run.

checksum_every `property` #

checksum_every: int | None

Number of runs between checksum runs when verify_checksums is off.

connector_name `property` #

connector_name: str | None

Name of the SharePoint data source the sync reads from.

created_by `property` #

created_by: Any

User who created the sync; scheduled runs run as this user.

created_on `property` #

created_on: Any

When the sync was created.

cron_expression `property` #

cron_expression: str | None

Cron expression of the schedule, None for a copy-once sync.

dataset_id `property` #

dataset_id: int | None

Id of the synced dataset, None once it was deleted outside the sync.

dataset_name `property` #

dataset_name: str | None

Top-level dataset of the project the files are copied into, fixed at creation.

drive_id `property` #

drive_id: str | None

Graph id of the document library, fixed at creation.

drive_name `property` #

drive_name: str | None

Display name of the document library.

exclude_patterns `property` #

exclude_patterns: list[str]

Filter patterns, in rclone syntax, of the files to skip.

files_total `property` #

files_total: int | None

Number of files in the dataset after the most recent run.

folder_item_id `property` #

folder_item_id: str | None

Graph id of the synced folder, empty for the library root.

folder_path `property` #

folder_path: str | None

Path of the synced folder inside the library, empty for the library root.

id `property` #

id: int | None

Id of the sync, unique in the project.

include_patterns `property` #

include_patterns: list[str]

Filter patterns, in rclone syntax, of the files to copy; empty copies everything not excluded.

index_feature_group `property` #

index_feature_group: str | None

Name of the index feature group (version 1), or None when the sync does not index.

index_feature_group_id `property` #

index_feature_group_id: int | None

Id of the index feature group, or None until a run has created it.

index_files `property` #

index_files: bool | None

Whether each run indexes the dataset's files in a feature group.

job_id `property` #

job_id: int | None

Id of the Python job that runs the sync.

job_name `property` #

job_name: str | None

Name of the Python job that runs the sync, sharepoint_<connector>_<dataset>.

last_audit_file `property` #

last_audit_file: str | None

Audit file written by the most recent run that changed files, relative to the project.

last_error `property` #

last_error: str | None

Error of the most recent failed or partial run.

last_run_execution_id `property` #

last_run_execution_id: int | None

Execution of the most recent run that reported a result.

last_run_finished `property` #

last_run_finished: Any

End of the most recent run.

last_run_started `property` #

last_run_started: Any

Start of the most recent run.

last_success `property` #

last_success: Any

End of the most recent run that copied everything without errors.

max_file_size_mb `property` #

max_file_size_mb: int | None

Files larger than this many MB are skipped; 0 copies files of any size.

mirror_deletions `property` #

mirror_deletions: bool | None

Whether files gone from SharePoint, or added in Hopsworks, are deleted from the dataset.

paused `property` #

paused: bool | None

Whether the schedule is disabled.

resync_requested_at `property` #

resync_requested_at: Any

When a full resync was requested that no run has done yet.

scheduled `property` #

scheduled: bool | None

Whether the sync has a schedule; a sync without one copies only when started.

state `property` #

state: str | None

CREATING, IDLE, RUNNING, FAILED or DELETING.

transfers `property` #

transfers: int | None

Number of files downloaded in parallel; the cluster default when None.

verify_checksums `property` #

verify_checksums: bool | None

Whether every run compares file contents by checksum.

delete #

delete(
    keep_files: bool = False, force: bool = False
) -> bool

Delete the sync and its job.

With force, the backend stops a run in progress first; when that run does not stop in time the delete is left pending. A pending delete keeps the sync in state DELETING, and retry or another delete finishes it.

Deletes the synced dataset unless keep_files is set

Without keep_files, the dataset and the sync's audit log are deleted with every file in them, and so is the index feature group of a sync that indexes its files. Files are not recoverable from Hopsworks afterwards; only SharePoint still has them.

PARAMETER	DESCRIPTION
`keep_files`	Keep the dataset, the audit log and the index feature group; the audit log is then no longer rotated. TYPE: `bool` DEFAULT: `False`
`force`	Stop a run in progress instead of refusing the delete. TYPE: `bool` DEFAULT: `False`

RETURNS	DESCRIPTION
`bool`	`True` when the sync is deleted, `False` when the delete is still pending.

RAISES	DESCRIPTION
`hopsworks.client.exceptions.RestAPIError`	If a run is in progress and `force` is not set.

force_full_resync #

force_full_resync(run: bool = True) -> Execution | None

Make the next run compare file contents by checksum, not only size and modification time.

A run normally detects a changed file by its size or modification time, so a file rewritten with the same size and time is only caught by a checksum run. The sync is reloaded afterwards, so resync_requested_at shows the pending request.

PARAMETER	DESCRIPTION
`run`	Start the run now; otherwise the next scheduled or manual run does the full comparison. TYPE: `bool` DEFAULT: `True`

RETURNS	DESCRIPTION
`Execution \| None`	The started execution, or `None` when `run` is not set.

get_audit #

get_audit(
    day: str | datetime.date | None = None,
) -> list[dict[str, Any]]

Return the paths changed by the runs of one day, as recorded in the audit log.

Runs that changed nothing write no audit rows. The backend keeps a cluster-wide number of days, 7 by default.

PARAMETER	DESCRIPTION
`day`	Day in UTC, as `YYYY-MM-DD` or a date; today when not given. TYPE: `str \| datetime.date \| None` DEFAULT: `None`

RETURNS	DESCRIPTION
`list[dict[str, Any]]`	One dict per changed path with `date`, `execution_id`, `op` (`added`, `updated`, `deleted` or `renamed`), `kind` (`file` or `dir`), `path` and `src_path` (the old path of a rename).

get_index_feature_group #

get_index_feature_group() -> FeatureGroup | None

Get the feature group that indexes the sync's files.

It holds one row per file in the dataset: SharePoint id, path, link, type, size, times, authors, quickXorHash and path in HopsFS, partitioned by ingestion_date, the day the file was first indexed. The first run after indexing is turned on creates it.

RETURNS	DESCRIPTION
`FeatureGroup \| None`	The index feature group, or `None` until a run has created it.

RAISES	DESCRIPTION
`ValueError`	If the sync does not index its files.

pause #

pause() -> SharePointSync

Disable the schedule, keeping it so that resume restores it.

RETURNS	DESCRIPTION
`SharePointSync`	This sync, updated.

RAISES	DESCRIPTION
`hopsworks.client.exceptions.RestAPIError`	If the sync has no schedule.

refresh #

refresh() -> SharePointSync

Reload the sync from Hopsworks.

RETURNS	DESCRIPTION
`SharePointSync`	This sync, with its fields as the backend reports them now.

resume #

resume() -> SharePointSync

Re-enable a paused schedule.

RETURNS	DESCRIPTION
`SharePointSync`	This sync, updated.

RAISES	DESCRIPTION
`hopsworks.client.exceptions.RestAPIError`	If the sync has no schedule.

retry #

retry() -> SharePointSync

Resume a sync left in CREATING or DELETING by a failed create or delete.

RETURNS	DESCRIPTION
`SharePointSync`	This sync, updated.

run #

run() -> Execution | Any

Start an execution of the sync job now, also while the sync is paused or has no schedule.

The execution runs as the caller, who must be a Data Owner. When a run of this sync is already in progress, the new execution finishes immediately without copying anything.

RETURNS	DESCRIPTION
`Execution \| Any`	The started execution, or the backend's response when it does not describe one.

set_folder #

set_folder(folder_path: str) -> SharePointSync

Point the sync at another folder of the same document library.

Under mirror semantics the next run deletes the files of the old folder from the dataset and copies the new one.

PARAMETER	DESCRIPTION
`folder_path`	Folder path inside the library, `""` for the library root. TYPE: `str`

RETURNS	DESCRIPTION
`SharePointSync`	This sync, updated.

set_schedule #

set_schedule(cron_expression: str | None) -> SharePointSync

Add, change or remove the schedule.

PARAMETER	DESCRIPTION
`cron_expression`	Quartz cron expression in UTC, with seconds first, e.g. `"0 0 * ? * "` for hourly or `"0 0 2 ? "` for daily at 02:00. `None` removes the schedule, so the sync runs only when started. Runs closer together than the cluster's minimum interval (5 minutes by default) are refused. TYPE:* `str \| None`

RETURNS	DESCRIPTION
`SharePointSync`	This sync, updated.

status #

status() -> dict[str, Any]

Reload the sync and return its run state and counters.

A run that ended without reporting a result, because its pod was killed or crashed, is reported as FAILED with last_error saying so.

RETURNS	DESCRIPTION
`dict[str, Any]`	`state`, `paused`, `scheduled`, `cron_expression`, `active_execution_id`, `resync_requested_at`, the `last_run_*` fields, `last_success`, `last_error`, `last_audit_file`, `files_total` and `bytes_total`.

update #

update(**fields: Any) -> SharePointSync

Change settings of the sync; they apply from the next run.

The keyword arguments are any of include_patterns, exclude_patterns, max_file_size_mb, mirror_deletions, verify_checksums, checksum_every, transfers, index_files, index_feature_group, cron_expression (None removes the schedule), folder_path and take_ownership. Fields not given keep their value. Under mirror semantics, files that a new filter or size cap excludes are deleted from the dataset on the next run. index_files=False stops indexing and keeps the feature group; a new index_feature_group starts a new one, which the next run fills with every copied file still in the folder. take_ownership=True re-creates the schedule as the caller, so scheduled runs run as the caller, which adopts a sync whose creator left the project.

RETURNS	DESCRIPTION
`SharePointSync`	This sync, updated.

RAISES	DESCRIPTION
`ValueError`	If a field cannot be changed, such as `drive_id` or `dataset_name`.

hsfs.core.sharepoint_sync #

SharePointSync #

active_execution_id property #

alert_receiver property #

audit_path property #

bytes_total property #

checksum_every property #

connector_name property #

created_by property #

created_on property #

cron_expression property #

dataset_id property #

dataset_name property #

drive_id property #

drive_name property #

exclude_patterns property #

files_total property #

folder_item_id property #

folder_path property #

id property #

include_patterns property #

index_feature_group property #

index_feature_group_id property #

index_files property #

job_id property #

job_name property #

last_audit_file property #

last_error property #

last_run_execution_id property #

last_run_finished property #

last_run_started property #

last_success property #

max_file_size_mb property #

mirror_deletions property #

paused property #

resync_requested_at property #

scheduled property #

state property #

transfers property #

verify_checksums property #