Question

The acos function works in the interval -1 <= x <= 1.

Sometimes, because of rounding, I’m getting 1.00001 to calculate the acos over, then the following where clause is giving me a floating number error:

    where
        (6371 * acos(cos(radians(@ga_where)) * cos(radians(thing_geolocation.ga_thing)) * cos(radians(thing_geolocation.go_thing) - radians(@go_where)) + sin(radians(@ga_where)) * sin(radians(thing_geolocation.ga_thing)))) < 1.5

So, I’ll have to refactor this where clause into something like this:

    where
        (
            6371 *
            acos
            (
                case
                    when cos(radians(@latitude)) * cos(radians(thing_geolocation.ga_thing)) * cos(radians(thing_geolocation.go_thing) - radians(@longitude)) + sin(radians(@latitude)) * sin(radians(thing_geolocation.ga_thing)) > 1 then 1
                    when cos(radians(@latitude)) * cos(radians(thing_geolocation.ga_thing)) * cos(radians(thing_geolocation.go_thing) - radians(@longitude)) + sin(radians(@latitude)) * sin(radians(thing_geolocation.ga_thing)) < -1 then -1
                    else cos(radians(@latitude)) * cos(radians(thing_geolocation.ga_thing)) * cos(radians(thing_geolocation.go_thing) - radians(@longitude)) + sin(radians(@latitude)) * sin(radians(thing_geolocation.ga_thing))
                end
            )
        ) < 1.5 

But, is not sounding good (it will cause to calculate the distance 3x, right?).

I have other ideas, like a subquery or a outerapply, but because the entiry query is very complex and the same fields are used to other purposes, I’m trying to solve it without change the whole query.

ps: I tried to use the point.distance function, but it doesn’t work well with null values, so I choose the keep this way.

Any ideas?

Answer 1

I would suggest changing everything that you’re passing into acos into a user-defined scalar-valued function. Within that function, you can account for values greater than 1.0 or less than -1.0.

By doing this, you would not have to perform any of the mathematical calculations more than what is necessary.

I think your function would look something like this:

CREATE FUNCTION [dbo].[udf_GetACOSInputValueByLatLong]
(
    @LatStart DECIMAL(10,6),
    @LongStart DECIMAL(10,6),
    @LatEnd DECIMAL(10,6),
    @LongEnd DECIMAL(10,6)
)
RETURNS DECIMAL(10,6) AS 
BEGIN

DECLARE @Result DECIMAL(10, 6)
SET @Result = ( 
    cos( radians( @LatStart ) ) * 
    cos( radians( @LatEnd ) ) * 
    cos( radians( @LongEnd ) - radians(@LongStart) ) + sin( radians(@LatStart) ) * 
    sin( radians( @LatEnd ) ) 
)

IF @Result > 1.0
    SET @Result = 1.0
ELSE IF @Result < -1.0
    SET @Result = -1.0

RETURN (@Result)

END

Which could then be used like the following:

where
    (
        6371 * acos (YourDatabase.dbo.udf_GetACOSInputValueByLatLong(
            @latitude, 
            @longitude, 
            thing_geolocation.ga_thing, 
            thing_geolocation.go_thing))
    ) < 1.5