Batch applications

Sometimes there is a need to process data at regular scheduled intervals. For instance, if your business depends on data delivered by an external partner every night by means of a batch file.

Suppose the data that is delivered to you is formatted in a way that is not readily usable by your infrastructure. A typical solution would be to "ETL" the data file: extract, transform and load every line from the file into your infrastructure.

Depending on the size of the dataset to load and the complexity of the operatoins to perform (validation, transformation, combination with your data), this can easily become a multi-hour process during which your infrastructure undergoes a heavy load that has a negative impact on response times for end users.

Therefore batch applications, like all other applications, must be designed with care and different measures must be taken to minimize the impact for online users.

Batch technology may well be the most mature of software technologies. With decades of batch systems performing various tasks, tons of experiences have been documentend and turned into best practices.