Ok, here’s the situation… I have an application that generates about 8 files per second. Each file is 19-24kb. This generates about 10 to 11 MB per minute. This question is not about how to ftp, because I have that solution already… The question is more about how to keep up with the flow of data (only a 2mb upload bandwidth in most cases, unless I am travelling to a client site that has a large pipe). I dont care if ftp takes longer to transfer then the rate of flow, but I want to know if anyone has an idea on how to batch the files to move them so that when the ftp process is finished it will delete just those files it transfered and then move on to the next batch. Here is what I was thinking:
Multi thread the app, first thread runs the app, second thread is a timer that creates a text file every ‘N’ minutes with all the files created in that time span. StreamRead the file and move the files that are in text to another location (maybe create a temp folder) and then ftp those files, then delete files, folder and textfile… in the mean time, more text files are being written and temp folders being created. Does this sound feasible? I will take any suggestions that anyone has under advisement, just looking for the fastest and most reliable path.
Please dont ask to see the code, there is no reason to see it considering we are working with hypotheticals.
Wihtout realy knowing any more details on why you need to keep all the work in a single application and deal with threading complexity, one could argue to keep the part that generates the files and the part that FTPs the files in separate applications.
Separation of Responsibility. Ensure each application does only one job and does it right and fast.
One Serivce or app(desktop/web which ever) generating the files.
Another Service which watches a folder and moves any incoming files into a temp filder, does what it needs to do, FTPs and deletes.
Seeing I don’t know your setup and where you get the content from for your files, writing it in a single app might be the best choice exactly how you suggested.
Basically to anwser your question. Yes, it does sound feasable what you want to do.
How you implement it and what you are happy with implementing is up to you.
If you get stuck somewhere during implementation, feel free to post any issues in a new threat with some code samples on how you have a specific feature implemented and what the issue is you are experiencing.
Until then, hypothetically, any approach you feel is able to manage what you need to achieve is perfectly valid.
EDIT
Seeing you stated you already got the application which generates the files done and you already have a solution which FTPs means using 2 separate applications sounds more plausible.
All you need then is wrap a service around the FTP solution and happy days.
No need to interfeere with the original application which generates the files if it is already working.
Why risk breaking it, unless you must add the fTP feature into it and you have no choice.