Tuesday, October 6, 2009

Running Scheduled Jobs

Working in my company's integration department, it's often necessary for us to set up scheduled data import/export jobs to and from services of various origins. Our current solution to this, which has been adequate thus far, is a simple single threaded Windows service with a tad quirky timing mechanism. Integrations are scheduled and can be turned on and off my editing a table in our database. Errors are logged into a database when jobs fail. Job processing logs, however, leave a lot to be desired. I hacked in processing logs having my hands pretty much tied. With the permission of my boss, now, I am beginning work on a new version of our job scheduling service. Features which I would like to include are:
  • Integration processing for multiple clients should be able to happen synchronously.
  • Jobs need to run every x minutes/hours/days without any variation.
  • Error and processing logs should be able to be kept.
  • Integrations should be able to be stopped and started on a per Integration/per client basis.
  • Service should be able to allow all jobs to complete processing before shutting down.
  • The service should be able to be monitored by an external application and controlled almost entirely from that application. Upcoming jobs, running jobs and completed jobs as well as error and processing logs should be viewable.
This is going to be a fun project I think. I'll talk more about design and implementation ideas at a later time.