Question

The code below is used to calculate the miles between two cities. In this case, it’s for the distance from Yarmouth, ME to Yarmouth, ME – obviously zero – meaning that results for cities within X miles of Yarmouth should include Yarmouth itself.

The problem is that the latitude and longitude for Yarmouth seem to be causing some kind of floating point problem (I haven’t seen this with other cities):

DECLARE @fromlong FLOAT, @fromlat FLOAT, @tolong FLOAT, @tolat FLOAT, @test FLOAT

SET @fromlong = 43.8219
SET @fromlat = -70.1758
SET @tolong = 43.8219
SET @tolat = -70.1758
SET @test = SIN(@fromlong / ( 180 / PI() )) * SIN(@tolong / ( 180 / PI() )) + COS(@fromlong / ( 180 / PI() )) * COS(@tolong / ( 180 / PI() )) * COS(@fromlat / ( 180 / PI() ) - @tolat / ( 180 / PI() ))

PRINT @test /*** Displays "1" ***/

SELECT 3963.0 * ACOS(@test) /*** Displays "a domain error has occurred" ***/

First, is this a SQL Server bug?

Second, what can I do to get around it? I know in the example above I could have some logic for IF @test > 1, but this example is distilled from a query embedded in a web app (not my choice), so I need to fix the query, i.e. fix the calculation, without resorting to TSQL if possible, and without distorting any other return values. Any ideas?

Answer 1

The consensus from the comments seems to be the use of FLOAT. I only used FLOAT in the example because that was the datatype of the columns from which the latitude/longitude was being read, but that seems to be the heart of the problem. Since I can’t change the datatypes of the columns themselves, the simplest solution was to change the datatype of the resulting calculation, which seems to work fine in all cases:

SELECT 3963.0 * ACOS(CONVERT(DECIMAL(10, 6), @test))

I realize leaving the calculation as a float could result in small rounding errors, but they are acceptably small for this application. Thanks all who commented.