Mouse simulation issue on macOS

[onion108]

I helped my blind friend to modify this mod so that he can read and operate with the history messages using LALT+UP/DOWN/RIHGT/ENTER. While I'm working with the code, I find a problem that the mouse simulation doesn't work on macOS, and that's because it only detects Windows and Linux, but macOS won't be recognized as any of them, so mouse simulation simply doesn't work without any error messages. That may confuse those blind users who uses macOS.

[khanshoaib3]

History messages?? Are you talking about the chat messages then I've already patched that, you can use alt + number key to speak the last 9 messages.
I actually don't own a mac so I can't really work on that bit. Sorry.

[khanshoaib3]

@onion108 Any update??

[boholder]

that's because it only detects Windows and Linux, but macOS won't be recognized as any of them, so mouse simulation simply doesn't work without any error messages

Can you specify it cleanly which part of code you've found that has problem? Currently only InventoryControls has mouse key simulation and perhaps you're referring this part of code?

minecraft-access/common/src/main/java/com/github/khanshoaib3/minecraft_access/utils/KeyBindingsHandler.java

Lines 328 to 342 in ee82d1c

	/**
	* Checks whether the given keybinding is currently pressed or not. This works even if the keybinding is duplicate i.e. another keybinding has the sane key bound to it.
	*
	* @param keyBindingToCheck The keybinding we want to check.
	* @return Returns true if the keybinding is currently pressed else false.
	*/
	public boolean isPressed(KeyBinding keyBindingToCheck) {
	MinecraftClient minecraftClient = MinecraftClient.getInstance();
	if (minecraftClient == null) return false;
	return InputUtil.isKeyPressed(
	minecraftClient.getWindow().getHandle(),
	InputUtil.fromTranslationKey(keyBindingToCheck.getBoundKeyTranslationKey()).getCode()
	);
	}

[onion108]

I'm referring to this:

minecraft-access/common/src/main/java/com/github/khanshoaib3/minecraft_access/utils/MouseUtils.java

Lines 42 to 60 in ee82d1c

	public static void move(int x, int y) {
	try {
	MainClass.infoLog("Moving mouse to x:%d y:%d".formatted(x, y));
	if (OsUtils.isLinux()) {
	Runtime.getRuntime().exec("xdotool mousemove %d %d".formatted(x, y));
	}
	if (OsUtils.isWindows()) {
	if (mainInterface == null) initializeUser32dll();
	if (!mainInterface.SetCursorPos(x, y))
	MainClass.errorLog("\nError encountered on moving mouse.");
	}
	} catch (Exception e) {
	MainClass.errorLog("\nError encountered on moving mouse.");
	e.printStackTrace();
	}
	}

and this part as well:

minecraft-access/common/src/main/java/com/github/khanshoaib3/minecraft_access/utils/MouseUtils.java

Lines 84 to 191 in ee82d1c

	/**
	* Perform left click at the current pixel location.
	*/
	public static void leftClick() {
	MinecraftClient minecraftClient = MinecraftClient.getInstance();
	if (minecraftClient == null)
	return;
	try {
	int x = (int) minecraftClient.mouse.getX(), y = (int) minecraftClient.mouse.getY();
	MainClass.infoLog("Performing left click at x:%d y:%d".formatted(x, y));
	if (OsUtils.isLinux()) {
	Runtime.getRuntime().exec("xdotool click 1");
	}
	if (OsUtils.isWindows()) {
	if (mainInterface == null) initializeUser32dll();
	mainInterface.mouse_event(MouseEventFlags.LEFTDOWN.getValue(), 0, 0, 0, 0);
	mainInterface.mouse_event(MouseEventFlags.LEFTUP.getValue(), 0, 0, 0, 0);
	}
	} catch (Exception e) {
	MainClass.errorLog("\nError encountered on performing left mouse click.");
	e.printStackTrace();
	}
	}
	/**
	* Perform right click at the current pixel location.
	*/
	public static void rightClick() {
	MinecraftClient minecraftClient = MinecraftClient.getInstance();
	if (minecraftClient == null)
	return;
	try {
	int x = (int) minecraftClient.mouse.getX(), y = (int) minecraftClient.mouse.getY();
	MainClass.infoLog("Performing right click at x:%d y:%d".formatted(x, y));
	if (OsUtils.isLinux()) {
	Runtime.getRuntime().exec("xdotool click 3");
	}
	if (OsUtils.isWindows()) {
	if (mainInterface == null) initializeUser32dll();
	mainInterface.mouse_event(MouseEventFlags.RIGHTDOWN.getValue(), 0, 0, 0, 0);
	mainInterface.mouse_event(MouseEventFlags.RIGHTUP.getValue(), 0, 0, 0, 0);
	}
	} catch (Exception e) {
	MainClass.errorLog("\nError encountered on performing right mouse click.");
	e.printStackTrace();
	}
	}
	/**
	* Performs mouse scroll up
	*/
	public static void scrollUp(){
	MinecraftClient minecraftClient = MinecraftClient.getInstance();
	if (minecraftClient == null)
	return;
	try {
	MainClass.infoLog("Performing scroll up");
	if (OsUtils.isLinux()) {
	Runtime.getRuntime().exec("xdotool click 4");
	}
	if (OsUtils.isWindows()) {
	if (mainInterface == null) initializeUser32dll();
	mainInterface.mouse_event(MouseEventFlags.WHEEL.getValue(), 0, 0, 120, 0);
	}
	} catch (Exception e) {
	MainClass.errorLog("\nError encountered on performing scroll up.");
	e.printStackTrace();
	}
	}
	/**
	* Performs mouse scroll down
	*/
	public static void scrollDown(){
	MinecraftClient minecraftClient = MinecraftClient.getInstance();
	if (minecraftClient == null)
	return;
	try {
	MainClass.infoLog("Performing scroll down");
	if (OsUtils.isLinux()) {
	Runtime.getRuntime().exec("xdotool click 5");
	}
	if (OsUtils.isWindows()) {
	if (mainInterface == null) initializeUser32dll();
	mainInterface.mouse_event(MouseEventFlags.WHEEL.getValue(), 0, 0, -120, 0);
	}
	} catch (Exception e) {
	MainClass.errorLog("\nError encountered on performing scroll up.");
	e.printStackTrace();
	}
	}

They only handle the operation for Linux and Windows, but on macOS neither OsUtils.isLinux() nor OsUtils.isWindows() returns true, resulting in doing nothing.

Another caution is that xdotools isn't available for macOS because Minecraft doesn't run under X11 on macOS, so it may be necessary to find another way to simulate mouse operations on macOS.

[khanshoaib3]

Yeah, we will have to look into this. Thanks for bringing this up.

[boholder]

There might be more things to do for compatibility with macOS.
For example, I found out that the Narrator does not have a macOS version...... the Narrator calls native API so I guess currently the "interrupting narration" feature won't work on macOS.

@onion108 Would you please help us on this? I'm using Windows while Shoabi uses Linux... We don't have a device for testing...
BTW, join the Discord server if you like, maybe we can find more macOS users in there...

[khanshoaib3]

We can implement the libspeak for macos narrator.
We probably should.

[boholder]

#39 (comment)

Still searching for a built-in method to move the cursor (so we can get rid of OS-specific implementation).

[onion108]

There might be more things to do for compatibility with macOS. For example, I found out that the Narrator does not have a macOS version...... the Narrator calls native API so I guess currently the "interrupting narration" feature won't work on macOS.

@onion108 Would you please help us on this? I'm using Windows while Shoabi uses Linux... We don't have a device for testing... BTW, join the Discord server if you like, maybe we can find more macOS users in there...

I'd like to help~! I've joined the server just now (with the name 27Onion Nebell or Discord Username 27onion)

[Isaac-Leonard]

Is there any progress on this?

[BrailleBennett]

This issue should be closed now, as full Mac support was added.

[boholder]

Resolved by #340