Skip to content

Commit 24ecd5a

Browse files
authored
[FSTORE-1780] onlinefs observability docs (#491)
1 parent fc89745 commit 24ecd5a

File tree

3 files changed

+102
-0
lines changed

3 files changed

+102
-0
lines changed
Loading
Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
---
2+
description: Documentation on Online ingestion observability in Hopsworks.
3+
---
4+
5+
# Online ingestion observability
6+
7+
## Introduction
8+
9+
Knowing when ingested data becomes available for online serving—and understanding the cause of any ingestion failures—is crucial for users. To address this, the Hopsworks API provides observability features for online ingestion, allowing you to monitor ingestion status and troubleshoot issues.
10+
11+
This guide explains how to use these observability features for online feature groups in Hopsworks, with examples using both the HSFS APIs and the user interface.
12+
13+
## Prerequisites
14+
15+
Before you begin this guide we suggest you read the [Feature Group](../../../concepts/fs/feature_group/fg_overview.md) concept page to understand what a feature group is and how it fits in the ML pipeline.
16+
17+
## Using the Hopsworks API
18+
19+
### Create a Feature Group and Ingest Data
20+
21+
First, create an online-enabled feature group and insert data into it:
22+
23+
=== "Python"
24+
25+
```python
26+
fg = fs.create_feature_group(
27+
name="feature_group_name",
28+
version=feature_group_version,
29+
primary_key=feature_group_primary_keys,
30+
online_enabled=True
31+
)
32+
33+
fg.insert(fg_df)
34+
```
35+
36+
### Retrieve Online Ingestion Status
37+
38+
After inserting data, you can monitor the ingestion progress:
39+
40+
**Get the latest ingestion instance**
41+
42+
=== "Python"
43+
44+
```python
45+
oi = fg.get_latest_online_ingestion()
46+
```
47+
48+
**Get a specific ingestion by its ID**
49+
50+
=== "Python"
51+
52+
```python
53+
oi = fg.get_online_ingestion(ingestion_id)
54+
```
55+
56+
### Use the Online Ingestion Object
57+
58+
The online ingestion object provides methods to track and debug the ingestion process:
59+
60+
**Wait for completion**
61+
62+
Wait for the online ingestion to finish (equivalent to `fg.insert(fg_df, wait=True)`):
63+
64+
=== "Python"
65+
66+
```python
67+
oi.wait_for_completion()
68+
```
69+
70+
**Print mini-batch results**
71+
72+
Check the results of the ingestion. If the status is `UPSERTED` and the number of rows matches your data, the ingestion was successful:
73+
74+
=== "Python"
75+
76+
```python
77+
print([result.to_dict() for result in oi.results])
78+
# Example output: [{'onlineIngestionId': 1, 'status': 'UPSERTED', 'rows': 10}]
79+
```
80+
81+
**Print ingestion service logs**
82+
83+
Retrieve logs from the online ingestion service to diagnose any issues:
84+
85+
=== "Python"
86+
87+
```python
88+
oi.print_logs(priority="error", size=5)
89+
```
90+
91+
## Using the UI
92+
93+
### Viewing Online Ingestion Status
94+
95+
After inserting data into an online-enabled feature group, you can track the ingestion progress in the `Recent activities` section of the feature group in the Hopsworks UI.
96+
97+
<p align="center">
98+
<figure>
99+
<img src="../../../../assets/images/guides/feature_group/online_ingestion.png" alt="See online ingestion status">
100+
</figure>
101+
</p>

mkdocs.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -85,6 +85,7 @@ nav:
8585
- Advanced guide: user_guides/fs/feature_monitoring/feature_monitoring_advanced.md
8686
- Notification: user_guides/fs/feature_group/notification.md
8787
- On-Demand Transformations: user_guides/fs/feature_group/on_demand_transformations.md
88+
- Online Ingestion Observability: user_guides/fs/feature_group/online_ingestion_observability.md
8889
- Feature View:
8990
- user_guides/fs/feature_view/index.md
9091
- Overview: user_guides/fs/feature_view/overview.md

0 commit comments

Comments
 (0)