This is just a story of when things work, when things go wrong, and finding a way back to a working state.
For nearly ten years now, I have and still have a very good relationship with our address verification/correction vendor. They sell quality products, and they have very good support. Moreover, their products work.
Despite my selling the idea of address verification and correction all over our municipality, only one department is interested in and has to have it, the Treasurer/Bill collector. The Treasurer/Collector sends out tax, utility (water), and parking ticket bills, lots of them. Every US domestic address that can be address-verified and corrected, can be barcoded. Every bill run that contains 500 pieces and greater of barcoded mail qualifies for the barcode discount. That discount cuts costs in mailing out bills.
Our address verification/correction software comes in two separate pieces, the main application and address verification/correction engine, whose annual renewal covers the support costs of the second piece, a COM toolkit. The COM toolkit offers a way for several Windows interfaces, ASP (VBScript), MS Access, and Visual Basic among several others, to talk to the main verification/correction engine. In other words, the COM toolkit allows ancillary applications to be written that can make use of the verification/correction application engine. Unless you deploy Apache and its ASP page module, using the ASP page usually implies deploying Microsoft’s Internet Information Service (IIS).
This application is used in three ways.
1) A C module is embedded into an Informix customized runner. The C module accepts parameters from Informix, makes the http socket call, parses the return parameters, and returns results of a single address verification. This is how individual addresses are corrected, like mailing address changes.
2) For batch verification, a Clojure application reads in a list of addresses to be verified in the form of a .csv file that was programmatically extracted from our database; submits each address to the application using a web socket connection; and then writes out a .csv file of all addresses that were correct.
3) An MS Access application reads in a file of all corrected and uncorrectable addresses (outside the US for example), and produces what the US Post Office calls a CASS report. Bulk mail that is subject to barcode discount must be accompanied by a CASS report.
This simple ASP application running on the Windows 2000 Server’s version of IIS was a real workhorse. IIS processed up to 29,000 addresses fairly quickly. I determined that the sending client, needed to wait 1/3 second between requests, and the IIS application could handle the load.
Recently we had to
1) Upgrade to a new Windows Server, because
2) We needed a newer (2012) version of the COM toolkit (instead of 2004’s version), and that toolkit would not install on Windows 2000 Server. (The newer COM toolkit contained all the logic to cover the new US Post Office’s verification and correction rules.)
Guess what? IIS crashed using the old application on a Windows 2003 Server, and when we tried the application on a Windows 7 workstation, IIS crashed there too. IIS crashed even if the request spacing was 1 second all the way up to 5 seconds, but IIS does not crash if there are very, very slow, one at a time requests.
Think about 29,000 addresses. 1/3 second delay for an http request is acceptable, but increasing the delay much past 1/2 second is not tolerable. No one, and we had a very capably, technically staffed vendor support could come up with a reason for this, could come up with a reason, and we configured and tweaked IIS to its limits.
So what did we do to get out of trouble? We wrote an ActiveState Perl script to act as a specialized web server for our application, and so far, testing is positive.
Some things cannot be solved at all; other things can’t be solved without spending a lot of money; and some things are not worth lots of time trying to diagnose. I do not like the workaround, but sometimes that is what you have to do.