feat(browser_extension) add browser_extension and openadapt.browser (#…

…744)

* add old chrome extension files

* add old changes back, related to chrome extension, in openadapt dir

* try web socket solution, add asyncio and websockets libraries to poetry

* now we can log the dom changes ina a desctop running python process

* remove the old nativeMessaging extension code

* rename dir

* save the dom chages into dummy db: chrome.db

* fix actions tests

* ran `poetry run black --preview . --exclude 'alembic'`

* remove sockets.py

* fix falke8 formatting issues

* ran `poetry run black --preview . --exclude 'alembic'`

* add browser event code

* chrome extension work wip

* fix reset_db python file.

* now, extension messages can be seen in logs when the recording is started.

* till trying to fix the issue: record functione execution gets stuck at read_browser_events due to asyncio.runForever

* browser_events can be seen in db

* ran black and flake8

* add documentation in extension side js files and remove unused files

* #744 (review)

Update readme.md

* #744 (comment)
remove mlds logo

* #744 (comment)

add todo

* #744 (comment)

add NAMED_CONSTANT

* #744 (comment)

remove unused lines in content.js

* #744 (comment)

uncomment for `alembic upgrade`

* #744 (comment)

resolve this

* #744 (comment)

remove unwanted obfuscate keys

* #744 (comment)

used joinedload

* #744 (comment)

remove unwanted return

* #744 (comment)

Ran the following:
- `alembic revision --autogenerate -m "regenrate with browser_event table"`
- `alembic upgrade head`

* #744 (comment)

remove unwanted commetns lines

* #744 (comment)

remove commented lines

* https://github.com/OpenAdaptAI/OpenAdapt/pull/744/files#r1668802201

added this todo

* #744 (comment)

optimize browser events record  by adding an option to include to record or not like audio

* #744 (comment)

move constants to config

* #744 (comment)

update readme

* update readme again

* #744 (comment)

update config default for RECORD_BROWSER_EVENTS

* #744 (comment)

removed unessary comment in events.py

* Update openadapt/models.py

#744 (comment)

Co-authored-by: Richard Abrich <richard.abrich@mldsai.com>

* #744 (comment)

resolve d

* https://github.com/OpenAdaptAI/OpenAdapt/pull/744/files/3836c3d165bb11c5615c83a0df65f6cdc155f93f#r1676068390

resolved

* ran black on openadapt dir
`poetry run black --preview openadapt/`

* #744 (comment)

fixed faiing tests

* update the form type script

* add the record browser flag in classification group

* #744 (comment)

Co-authored-by: Richard Abrich <richard.abrich@mldsai.com>

* #744 (comment)

Co-authored-by: Richard Abrich <richard.abrich@mldsai.com>

* #744 (comment)

Co-authored-by: Richard Abrich <richard.abrich@mldsai.com>

* Update openadapt/config.py: #744 (comment)

#744 (comment)

Co-authored-by: Richard Abrich <richard.abrich@mldsai.com>

* Update openadapt/record.py

#744 (comment)

Co-authored-by: Richard Abrich <richard.abrich@mldsai.com>

* Update openadapt/config.py: https://github.com/OpenAdaptAI/OpenAdapt/pull/744/files/aa7b1ae7df8fb7045abb5d71d1d41a09f2d8bea2#r1691430453

https://github.com/OpenAdaptAI/OpenAdapt/pull/744/files/aa7b1ae7df8fb7045abb5d71d1d41a09f2d8bea2#r1691430453

Co-authored-by: Richard Abrich <richard.abrich@mldsai.com>

* Remove 3 unused functions owing to:

#744 (comment)

#744 (comment)

#744 (comment)

* Update openadapt/record.py: #744 (comment)

#744 (comment)

Co-authored-by: Richard Abrich <richard.abrich@mldsai.com>

* Rename owing to: #744 (comment)

* Rename const: #744 (comment)

* Remove unused: #744 (comment)

* Use ClassVar again: #744 (comment)

* Update openadapt/models.py: #744 (comment)

#744 (comment)

Co-authored-by: Richard Abrich <richard.abrich@mldsai.com>

* Update openadapt/record.py: #744 (comment)

#744 (comment)

Co-authored-by: Richard Abrich <richard.abrich@mldsai.com>

* Update openadapt/record.py: #744 (comment)

#744 (comment)

Co-authored-by: Richard Abrich <richard.abrich@mldsai.com>

* Move string to const: #744 (comment)

* Correct Grammatical errors: #744 (comment)

* fix visualize and reformat

* fix alembic migrations

* Intersection/MutationObserver; _repr_ignore_attrs; fix process_events

* synchronize timestamps

* attachWindowEventListeners

* recreate alembic migration

* sync; buffer

* add dtw.py (wip)

* remove sorting and syncing; working experiments/dtw.py

* fix content.js

* report screenX/screenY; fix getScreenCoordinates; compute coordinate differences

* wip

* remove observers; composite distance; task_by_name

* replace eventBuffer with coordMappings; compute tlbr-screen in browser.py

* assign_browser_events in events.py

* black; docstrings

* flake8

* update test

* black

* add test_browser.py

---------

Co-authored-by: pate1595 <kmp.patel@utoronto.ca>
Co-authored-by: Krish Patel <65433817+KrishPatel13@users.noreply.github.com>

README.md

@@ -164,6 +164,38 @@ possible memory leak
 pointing the cursor and left or right clicking, as described in this
 [open issue](https://github.com/OpenAdaptAI/OpenAdapt/issues/145)
 
+
+### Capturing Browser Events
+
+To capture (record) browser events in Chrome, follow these steps:
+
+1. Go to: [Chrome Extension Page](chrome://extensions/)
+
+2. Enable `Developer mode` (located at the top right):
+
+![image](https://github.com/OpenAdaptAI/OpenAdapt/assets/65433817/c97eb9fb-05d6-465d-85b3-332694556272)
+
+3. Click `Load unpacked` (located at the top left).
+
+![image](https://github.com/OpenAdaptAI/OpenAdapt/assets/65433817/00c8adf5-074a-4655-b132-fd87644007fc)
+
+4. Select the `chrome_extension` directory:
+
+![image](https://github.com/OpenAdaptAI/OpenAdapt/assets/65433817/71610ed3-f8d4-431a-9a22-d901127b7b0c)
+
+5. You should see the following confirmation, indicating that the extension is loaded:
+
+![image](https://github.com/OpenAdaptAI/OpenAdapt/assets/65433817/7ee19da9-37e0-448f-b9ab-08ef99110e85)
+
+6. Set the flag to `true` if it is currently `false`:
+
+![image](https://github.com/user-attachments/assets/8eba24a3-7c68-4deb-8fbe-9d03cece1482)
+
+7. Start recording. Once recording begins, navigate to the Chrome browser, browse some pages, and perform a few clicks. Then, stop the recording and let it complete successfully.
+
+8. After recording, check the `openadapt.db` table `browser_event`. It should contain all your browser activity logs. You can verify the data's correctness using the `sqlite3` CLI or an extension like `SQLite Viewer` in VS Code to open `data/openadapt.db`.
+
+
 ### Visualize
 
 Quickly visualize the latest recording you created by running the following command:

chrome_extension/background.js

@@ -0,0 +1,68 @@
+/**
+ * @file background.js
+ * @description Creates a new background script that listens for messages from the content script
+ * and sends them to a WebSocket server.
+*/
+
+let socket;
+let timeOffset = 0; // Global variable to store the time offset
+
+/* 
+ * TODO: 
+  * Ideally we read `WS_SERVER_PORT`, `WS_SERVER_ADDRESS` and 
+  * `RECONNECT_TIMEOUT_INTERVAL` from config.py, 
+  * or it gets passed in somehow. 
+*/
+let RECONNECT_TIMEOUT_INTERVAL = 1000; // ms
+let WS_SERVER_PORT = 8765;
+let WS_SERVER_ADDRESS = "localhost";
+let WS_SERVER_URL = "ws://" + WS_SERVER_ADDRESS + ":" + WS_SERVER_PORT;
+
+
+function socketSend(socket, message) {
+  console.log({ message });
+  socket.send(JSON.stringify(message));
+}
+
+
+/*
+ * Function to connect to the WebSocket server.
+*/
+function connectWebSocket() {
+  socket = new WebSocket(WS_SERVER_URL);
+
+  socket.onopen = function() {
+    console.log("WebSocket connection established");
+  };
+
+  socket.onmessage = function(event) {
+    console.log("Message from server:", event.data);
+    const message = JSON.parse(event.data);
+  };
+
+  socket.onclose = function(event) {
+    console.log("WebSocket connection closed", event);
+    // Reconnect after 5 seconds if the connection is lost
+    setTimeout(connectWebSocket, RECONNECT_TIMEOUT_INTERVAL);
+  };
+
+  socket.onerror = function(error) {
+    console.error("WebSocket error:", error);
+    socket.close();
+  };
+}
+
+// Create a connection to the WebSocket server
+connectWebSocket();
+
+/* Listen for messages from the content script */
+chrome.runtime.onMessage.addListener((message, sender, sendResponse) => {
+  const tabId = sender.tab.id;
+  message.tabId = tabId;
+  if (socket && socket.readyState === WebSocket.OPEN) {
+    socketSend(socket, message);
+    sendResponse({ status: "Message sent to WebSocket" });
+  } else {
+    sendResponse({ status: "WebSocket connection not open" });
+  }
+});