white paper

On Screen Scrapers and GUI Terminal Emulators

Screen scraping refers to the technology of utilizing APIs provided by the terminal emulation connectivity software to gather ("scrape off") certain information pieces from predefined screens in a manner invisible to the user. This information is then included in a graphical application developed with traditional tools like Visual Basic, Power Builder, etc. Most of the time a very limited subset of all available APIs is utilized - Connect, Wait for String, Send Key, just to name a few. In most cases the status of the terminal session is not even displayed to the user.

The major problem in utilizing the "screen scraping" approach is caused by its inherent inability to handle unexpected events related to the host connection behavior, like keyboard lockups, session disconnections, and broadcast messages from hosts, in addition to unexpected error messages coming from the "scraped" application.

A typical screen scraping implementation includes automating the log-on process by waiting for certain keywords like "User ID", "Password" etc. and replying (filling in the screen) with the appropriate text. It is not unusual to actually find User ID and Password strings hardcoded in such GUI programs.

After sending the required information to the host session, the GUI programs typically wait for the next screen, "hoping for the best" that the information it sent to the host is safely digested there. Most frequently the identity of the next screen is determined by the occurrence of a certain text string in a predetermined (by row and column position) screen location.

Since there is usually an unpredictable delay between sending the information to the host and receiving a reply, unreliable timing loops are frequently implemented to manage it. Obviously, any deviation from the expected sequence of events, like expiration of passwords, for example, will throw a monkey wrench into the scheme of things. All possible deviations from the expected replies must be handled by the GUI program. Unfortunately, in most cases these programs are developed by programmers with a limited knowledge of the host system’s behavior.

In a way, the screen scraping method of managing host data can be compared to crossing a busy intersection while blindfolded, relying on the traffic noise for guidance. Technically it can be done, but survival rate cannot be guaranteed. All this leads to to the current state of affairs in which the screen scraping approach achieves a bad reputation associated with its instability - too many blind pedestrians get hit, so to speak.

In order to overcome these obstacles and work reliably, graphical presentation programs must behave like true terminals. This means that the status of the connected session has to be displayed to keep users informed when the host is busy and thus the keyboard is locked, as well as alert them of disconnect situations. Additionally, the keyboard handling should provide full terminal functionality. This includes support for "exotic" terminal keys like "Roll Up", "Home", "SF16", "Page Erase Line" and others.

This need is more obvious when the connected host happens to be an AS/400 computer, which is capable of notifying users of problems by sending them messages along with "Message Waiting" indications usually displayed on the last line of the screen. There is a special combination of keystrokes that allow the users to interrupt their current work to view and reply to those messages.

While OutsideViewWEB technology utilizes the APIs of the terminal emulators included in OutsideViewWEB for host session information, this is where the similarities end. AppView provides a fully functional graphical terminal display that includes a status bar with the Operator Information Area (OIA) showing the state of the host session, its short name, message waiting indicators, etc. It even displays the row and column of the cursor’s position as seen by the host computer.

Additionally, all terminal keyboard functionality is fully implemented. The function keys work as intended, e.g. the "Enter" key behaves as expected instead of simulating the clicking on the "OK" button as is the case in many screen scraping implementations.

Host application users who are comfortable working on so-called "dumb" terminals are not hampered by the AppView GUI implementation. They can still perform their work without being needlessly forced to use the mouse.

But what if the user has an application which is working mostly with data from sources other then mainframe or midrange hosts? Can AppView products be used in situations where host data constitute such a small portion of the application that the screen scraping approach seems a perfetct fit? The answer is yes.

The advanced features of AppView such as the Get-To-The-Point facility, Web Links, Global Variables and others are designed specifically to be used in situations where developers would normally implement screen scraping techniques.