Home/ Questions/Q 9228567
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-18T05:21:24+00:00

I am using Selenium WebDriver in conjunction with java.awt.Robot to better simulate user interaction

  • 0

I am using Selenium WebDriver in conjunction with java.awt.Robot to better simulate user interaction with our web application. Yes, I know it’s probably unnecessary, but the customers I serve demand it.

Currently things are working pretty well, however I have a minor snag in that I can’t seem to find a good way to get the web elements’ on screen position. Things like the title bar, menu bar, navigation bar, etc all push the content down on the physical screen (which Robot gets its coordinates from), but has no impact on where Selenium reports the element is.

When I call: element.getLocation(); on a Selenium WebElement, it always gives me its location relative to the HTML content render pane, not the browser window itself.

A better illustration is: driver.findElement(By.tagName("body")).getLocation(); always returns 0,0, regardless of the window’s actual on-screen location.

Right now I am hacking it by adding a vertical and horizontal offset after maximizing the window, but these aren’t the same between different browsers (IE’s top decorations take up more room then Firefox’s, for example), and may be different for each user if they have bookmark tool bars, search bars, etc added in.

Yes, I know I could run in full screen mode, but I’d rather not, if at all possible.

Is there a way to use WebDriver to get the physical on-screen location of elements in a reliable manner?

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    I believe there’s no way to get the real on-screen location of the elements on the page.

    I also think the fullscreen mode is your best bet.

    That said, I wrote a RobotCalibration class that can detect the real offsets of your current browser. It opens a specially crafted page and uses the Robot class to click on it. The algorithm starts in the center of the browser and then uses bisecting to find the top left corner of the browser viewport.

    Tested on IE8 and FF18. Works for both maximized and windowed browsers. Known issue: If you have a top Bookmarks Toolbar enabled, it may click on some of the bookmarks and therefore redirect. It can be handled pretty easily, but I left it up to you if you needed that :).

    The testing page:

    <!DOCTYPE html>
<html lang="en" onclick="document.getElementById('counter').value++">
<head>
    <meta charset="utf-8" />
    <title>Calibration Test</title>
</head>
<body>
    <img height="1" width="1" style="position: absolute; left: 0; top: 0;"
        onclick="document.getElementById('done').value = 'yep'" />
    <input type="text" id="counter" value="0" />
    <input type="text" id="done" value="nope" />
</body>
</html>

    The RobotCalibration class. It’s a little bit long, so I suggest you to copypaste it into your favourite IDE and explore it there:

    import java.awt.AWTException;
import java.awt.Dimension;
import java.awt.Point;
import java.awt.Robot;
import java.awt.Toolkit;
import java.awt.event.InputEvent;
import java.nio.file.Paths;

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.ie.InternetExplorerDriver;

public class RobotCalibration {

    public static Point calibrate(WebDriver driver) {
        return new RobotCalibration(driver).calibrate();
    }

    /** Time for which to wait for the page response. */
    private static final long TIMEOUT = 1000;

    private final WebDriver driver;
    private final Robot r;

    private final Point browserCenter;
    private int leftX;
    private int rightX;
    private int midX;
    private int topY;
    private int bottomY;
    private int midY;

    private RobotCalibration(WebDriver driver) {
        this.driver = driver;
        try {
            driver.manage().window().getSize();
        } catch (UnsupportedOperationException headlessBrowserException) {
            throw new IllegalArgumentException("Calibrating a headless browser makes no sense.", headlessBrowserException);
        }

        try {
            this.r = new Robot();
        } catch (AWTException headlessEnvironmentException) {
            throw new IllegalStateException("Robot won't work on headless environments.", headlessEnvironmentException);
        }

        Dimension screenSize = Toolkit.getDefaultToolkit().getScreenSize();
        org.openqa.selenium.Dimension browserSize = driver.manage().window().getSize();
        org.openqa.selenium.Point browserPos = driver.manage().window().getPosition();

        // a maximized browser returns negative position
        // a maximized browser returns size larger than actual screen size
        // you can't click outside the screen
        leftX = Math.max(0, browserPos.x);
        rightX = Math.min(leftX + browserSize.width, screenSize.width - 1);
        midX = (leftX + rightX) /2;

        topY = Math.max(0, browserPos.y);
        bottomY = Math.min(topY + browserSize.height, screenSize.height - 1);
        midY = (topY + bottomY) /2;

        browserCenter = new Point(midX, midY);
    }

    private Point calibrate() {
        driver.get(Paths.get("files/RobotCalibration.html").toUri().toString());

        // find left border
        while (leftX < rightX) {
            click(midX, midY);
            if (clickWasSuccessful()) {
                rightX = midX;
            } else {
                leftX = midX + 1;
                // close any menu we could have opened
                click(browserCenter.x, browserCenter.y);
            }
            midX = (leftX + rightX) /2;
        }

        // find top border
        while (topY < bottomY) {
            click(midX, midY);
            if (clickWasSuccessful()) {
                bottomY = midY;
            } else {
                topY = midY + 1;
                // close any menu we could have opened
                click(browserCenter.x, browserCenter.y);
            }
            midY = (topY + bottomY) /2;
        }

        if (!isCalibrated()) {
            throw new IllegalStateException("Couldn't calibrate the Robot.");
        }
        return new Point(midX, midY);
    }

    /** clicks on the specified location */
    private void click(int x, int y) {
        r.mouseMove(x, y);
        r.mousePress(InputEvent.BUTTON1_DOWN_MASK);
        r.mouseRelease(InputEvent.BUTTON1_DOWN_MASK);

        // for some reason, my IE8 can't properly register clicks that are close
        // to each other faster than click every half a second
        if (driver instanceof InternetExplorerDriver) {
            sleep(500);
        }
    }

    private static void sleep(int millis) {
        try {
            Thread.sleep(millis);
        } catch (InterruptedException ignored) {
            // nothing to do
        }
    }

    private int counter = 0;
    /** @return whether the click on a page was successful */
    private boolean clickWasSuccessful() {
        counter++;

        long targetTime = System.currentTimeMillis() + TIMEOUT;
        while (System.currentTimeMillis() < targetTime) {
            int pageCounter = Integer.parseInt(driver.findElement(By.id("counter")).getAttribute("value"));
            if (counter == pageCounter) {
                return true;
            }
        }
        return false;
    }

    /** @return whether the top left corner has already been clicked at */
    private boolean isCalibrated() {
        long targetTime = System.currentTimeMillis() + TIMEOUT;
        while (System.currentTimeMillis() < targetTime) {
            if (driver.findElement(By.id("done")).getAttribute("value").equals("yep")) {
                return true;
            }
        }
        return false;
    }

}

    Sample usage:

    WebDriver driver = new InternetExplorerDriver();
Point p = RobotCalibration.calibrate(driver);
System.out.println("Left offset: " + p.x + ", top offset: " + p.y);
driver.quit();

    Feel free to ask any questions if something is unclear.

    • 0