Question

I want to use the ad automatic differentiation package for learning neural network weights in Haskell. I have found some functions that might just have what I need, however I can’t figure out what they expect as the first parameter. It must be the function to optimize, but I don’t know what form exactly.
They have signatures like this:

gradientDescent :: (Traversable f, Fractional a, Ord a) => (forall s. Mode s => f (AD s a) -> AD s a) -> f a -> [f a]

I have found out forall s. means something named an existential quantifier but nothing more.
My question is, that how could I pass my cost function with a signature like cost :: [Double] -> Double (it takes the list of weights) to this library?

Answer 1

So the first argument is a function on any traversable of AD to a single AD. For the traversable, we can substitute in something like a list to start with. That function must be polymorphic in mode. So let’s ignore that and just not do something that specifies a mode! This function is obviously the thing we’re optimizing. The next argument is the initial value we pass in. We’ll also call that a list for now. And the result is a list of steadily more optimized choices for improved guesses at our target.

Note that AD s a is an instance of Num and Fractional for all modes s, as long as a is Num and Fractional. So just write a polymorphic function from a list of integers to a single integer, pass in an initial state, and the function you provided will optimize it for you.

I.e. don’t specify your cost function as over doubles, but specify it as polymorphic over any Num and Fractional, and let the library take care of the rest!

You may prefer to get used to this style by trying other, more basic functions such as diff first.