Question

I have a question regarding polling or detecting changes in XML files on websites. I have a pretty hard-coded system below that reads an XML file from the NY Fed about their daily securities lending. It’s working well, and I have no issues with that. My main question is how I poll the website for changes every weekday. I understand that I can check the “Last Modified” field on the XML file (or in this case, the “Prepared” field), but I’m curious about how I go about doing this so that when a change is actually made to the XML site, my computer automatically loads and parses the new XML file. Getting this information as soon as it comes out is extremely important to me, so polling it every X minutes doesn’t really work as well as I would like. Any ideas?

using System;
using System.Collections.Generic;
using System.Linq;
using System.Xml;
using System.IO;
using System.Xml.Linq;
using System.Diagnostics;

namespace ConsoleApplication2 {
    class Program {
        static void Main(string[] args) {
            // XML text reader stuff
            //XmlReader textReader = XmlReader.Create("http://www.newyorkfed.org/markets/seclend/xml/v3_0/secLendingXML.cfm");
            XmlTextReader textReader = new XmlTextReader("http://www.newyorkfed.org/markets/seclend/xml/v3_0/secLendingXML.cfm");
            //SyndicationFeed feed = SindicationFeed.Load(textReader);
            //XmlTextReader textReader = new XmlTextReader("LOCATION");
            //string[] Fed_Array = new string[] {"Actual Available to Borrow", "Outstanding Loans",
            //"Par Submitted", "Par Accepted", "WTD Average Rate"};
            int j = 0;
            int icount = 0;
            using(StreamWriter writer = new StreamWriter("LOCATION"))
            //using (StreamWriter writer = new StreamWriter("LOCATION"))    
            while (textReader.Read()) {
                switch (textReader.NodeType) {
                case XmlNodeType.Element:
                    for (int i = 0; i < textReader.AttributeCount; i++) {
                        textReader.MoveToAttribute(i);
                        switch (textReader.Name) {
                        case "securityMaturityDate":
                            string textvalmatd = textReader.Value;
                            if (i == 0 && icount == 0) {
                                writer.Write("N/A,");
                                Console.Write("N/A,");
                                icount = 1;
                                i = -1;
                            } else {
                                writer.Write(textvalmatd + ",");
                                Console.Write(textvalmatd + ",");
                                icount = 0;
                            }
                            break;
                        case "couponRate":
                            string textvalcoup = textReader.Value;
                            writer.Write(textvalcoup + ",");
                            Console.Write(textvalcoup + ",");
                            break;
                        case "securityType":
                            string textvalsec = textReader.Value;
                            writer.Write(textvalsec + ",");
                            Console.Write(textvalsec + ",");
                            j = 0;
                            break;
                        case "value":
                            string textval = textReader.Value;
                            writer.Write(textval + ",");
                            Console.Write(textval + ",");
                            j = j + 1;
                            if (j > 4) {
                                writer.Write(Environment.NewLine);
                                Console.Write(Environment.NewLine);
                            }
                            break;

                        }
                    }
                    break;
                }
            }
        }

        private static IDisposable StreamWriter(string p) {
            throw new NotImplementedException();
        }
    }
}

Answer 1

Unless the Fed provides some sort of event indicating that a new file is available (which I doubt is the case), the theoretical best that you can do is to download the latest XML file and check if it’s newer than the previous one.

If the file is really released once per day, you could generate your own statistics on when, exactly, the file is released each day. Intensify your polling in a window surrounding that typical release time (be sure you don’t violate any terms of service in your polling frequency, however).