Return to the top of
this page.
Since the Fall of 1993, we have been supplying access to MVS ADABAS data over our TCP/IP network using Gopher and World Wide Web (WWW) clients like Mosaic and Netscape. The WWW clients can easily merge several kinds of data from several sources with little or no client side coding required.
Instead of generating extracts or documents that need to be rebuilt and transferred to some server every time the data changes, we use Natural subprograms to generate the requested information directly from our MVS production ADABAS files. By dynamically generating HTML forms and sending them to WWW clients, mainframe Natural routines can provide screens with embedded links, pull down windows, radio buttons, checkboxes, images, and a growing list of other GUI features.
Developed in parallel on MVS and a Sun Unix system using ADABAS C, the TCP/IP interface consists of two modules written in C. NATSVAL is used to create servers and NATCLI is used to create clients capable of requesting services from other TCP/IP based servers.
The initial driving force behind this effort was the need to provide easy student access to personal data and to deliver this by the Fall of 1993. This involved:
Return to the top of
this page.
Basically a client specifies a file type, file name combination and sends this to a Gopher server running at some Internet address and listening to a TCP/IP port (default port number 80). The server reads the file and sends the data back to the client.
The World Wide Web (WWW) was begun by the European Council for Nuclear Research (CERN) and is based on the HyperText Transfer Protocol (HTTP). The clients address resources through a naming scheme of Uniform Resource Locators (URLs). For example "http://www.sagus.com" refers to Software AG's home page.
HTTP has the capability to transfer complex data types using Multipurpose Internet Mail Extensions (MIME) headers and this allows us to do hypermedia. Communication between HTTP servers and other software is managed by the Common Gateway Interface (CGI) mechanism. CGI uses the Unix concept of Standard In (STDIN) and Standard Out (STDOUT) to pass the request data to a Practical Extraction and Report Language (PERL) or C program and receive the response.
MOSAIC, originated with the National Center for Supercomputer Applications (NCSA) located at the University of Illinois at Urbana-Champaign. This software provided an easy GUI interface to the HTTP world. Of late commercial versions of WWW viewers have also become available. For the U of D, NETSCAPE with the Secure Socket Layer (SSL) handshaking and encryption has made client server applications via the internet possible.
Return to the top of
this page.
The U of D uses mostly IBM PC's and compatibles at the desktop. Our emphasis was on the content and not so much on the look. We needed something simple but effective.
With the creation of our UDiscover Gopher based campus wide user information system, Gopher servers and clients were a hot topic on campus in 1993. We decided the idea of a server written in Natural, calling a subroutine to do all the TCP/IP interfacing and imitating Gopher access would meet our requirements. Existing user familiarity with the clients would also save both time and money.
Return to the top of
this page.
Sending multiple read commands across our network was not the best of answers. Running the same Natural program against local Unix files and files on MVS revealed that when reading logically by a key the local read time was averaging between 3 and 4 milliseconds but the same read across the network took about 50 milliseconds. If the network was "busy" this number could be higher. This would apply to any distributed system that requires many calls across the network to build a response, especially if the files were spread across some unknown number of machines.
For example: to read 500 records during a "normal" daytime load. (MVS ADABAS v5.1.9) (Unix ADABAS v1.2.1, Natural v1.1.4.1)
MVS Unix
MVS local From Unix Unix local From MVS via TCP/IP
--------- --------- ---------- -------- in seconds
1.6 23.1 3.5 25.0
1.8 20.8 3.4 22.7
1.5 27.6 3.2 22.9
1.7 28.5 3.2 21.9
-------- --------- ---------- --------
1.65 25.0 3.33 23.13 average
Return to the top of
this page.
Being able to route database calls around to multiple machines is very handy, but must be used with discretion. In an integrated environment, splitting the enterprise's data across multiple machines will insure that the traffic on the network will rise. For example, if the student information and payroll/personnel databases were on separate machines, what happens with an application that involves students who are also employees?
Why not just copy the data from MVS to Unix? Timing is everything. Is yesterday's data good enough for today's questions?
Return to the top of
this page.
The routines NATSVAL (server) and NATCLI (client) were first developed on a Sun Unix box to take advantage of the FORK system call. This call clones the parent process and allows the child process to handle the request while the parent process goes back to listening to the master port. When the child is done servicing the request it terminates. The code was migrated to MVS providing the same functionality on both platforms. The FORK function was not available on MVS so the code had to be modified slightly. Both NATSVAL and NATCLI have been kept very simple to allow as much of the application logic as possible to be placed in the Natural code.
Return to the top of
this page.
|User's PC| --- TCP/IP -------- THE NETWORK ------ TCP/IP
|client | routines routines
|session | on client on Host
|
v
Natural ---------- Natural main ----- NATSVAL interface
service subprogram program in C
| | | | |
| v | | -------------------------------- |
| local |
| ADABAS |
V |
local |
Net-Work |
| V
| NATCLI ----- TCP/IP ------- "other"
| interface routines servers
v
remote Net-Work ---- remote ADABAS
"Normal" processing would be for the subprogram to return to the
main program and then main would send the reply. If the reply is
more than 4080 characters, the subprogram would make multiple calls
to NATSVAL until the last send was reached, then it returns to the
main program which does the last send. The service subprogram can also communicate with other servers via NATCLI. For example, receive a form from our client and then act as a client to SMTP and mail it.
Return to the top of
this page.
In its most simple form, the process works like this.
o The MAIN Natural program calls the C routine NATSVAL
o NATSVAL server routine - asks the TCP/IP network
interface for a port number x1 to listen to
---o Listens to x1 for a message - accepts it on port x2
| o Use a FORK system call to create a child (an exact
| clone of the C routine and MAIN at that instant)
|--- o The parent closes port x2 and returns to listening
o The child closes port x1 and reads in the request
o Return the incoming request to the clone of MAIN
o The requested subprogram is called
o The subprogram fills in the buffer and returns
o The clone of MAIN calls the child copy of NATSVAL with
the buffer to be sent
o The child sends the buffer to the client using x2
o The child closes port x2 and terminates itself
o The clone of MAIN terminates
Return to the top of
this page.
In its most simple form, the process works like this.
o The MAIN Natural program calls the C routine NATCLI
o NATCLI client routine - (initial call) asks the TCP/IP
interface to connect to the IP address / port number
combination, and transmits
o Listens to the socket number returned by the TCP/IP
connection for a reply
o Continue to read from the socket until End of Data (EOD)
indicator or the receive buffer is full, set any required
flags (TCP/IP treats data as a stream of data that may be
broken into several transmissions by the sender)
o If EOD has been received close the connection
o Return to the caller
o If the client is smart enough and the server sends more
data than the buffer can hold, you should process what
has already been received and then return to NATCLI to
read some more
o If the client is smart enough and the server requires it,
the application may be written to be conversational (as
with SMTP) with as many sends and receives as the
protocol demands
Return to the top of
this page.
The use of the MVS COBOL routines triggered the porting of the NATSVAL routine from Unix to MVS. This would allow the all Natural part to run on either MVS or Unix while any COBOL parts ran only on MVS.
By adding a locally defined Gopher type F, we allowed users to look at a form , update it, and send it back to the server. It could then be mailed by writing to SMTP's port using the NATCLI (client) routine. This fit in well with our existing Email EZFORMS processing allowing user comments on the system to be mailed to developers from within the Gopher client.
Return to the top of
this page.
The Gopher clients were modified to look for certain queries to trigger a request for SSN and PIN. Once entered and encrypted, they are sent to the Natural server to be decrypted and checked against the SSN/PIN stored in the production database. The server routine then creates the requested data and passes it back to the client.
As new versions are released, the user's current client can be used to download the replacement code from a server. To run secured requests the client machine must have an IP address in the U of D subnet. The Windows client is a modified version of HGOPHER.
New in 1995 is YOU-VIEW. Instead of modifying the client we use the setup features to invoke our YOU-VIEW module as a helper application for files with "ud1" as the last qualifier.
Some sample screens from the GOPHER SIS system:
Return to the top of
this page.
The use of Hyper Text Markup Language (HTML) to create documents with embedded images and links is the key attraction. With HTML pieces of a document can reside on several different servers with only the Uniform Resource Locators (URLs) references actually being in the main text. As long as the name and location remain unchanged, the pieces can be revised at will without any kind of total reconstruction or relinking required.
The price was also right. Free copies of server and client code are available via File Transfer Protocol (FTP) as well as reasonably priced commercially supported versions.
To demonstrate the possible uses of HTML documents and forms, several examples were created, tested and listed on a master demo directory page. This has been frequently updated as new and improved examples have evolved.
Return to the top of
this page.
We don't even have to take a course or buy a book to learn about HTML and how CGI works. There are several online tutorials available at various WWW sites which will supply a good working knowledge. To name a few:
Prior to resolving the security issue only "unsecured" functions were in production at our site. We have replaced paper forms for the nomination of individuals for teaching excellence. Since anyone could submit a nomination, we placed a form on WWW.UDEL.EDU that would transmit the data to a Natural server capable of dealing with GET and POST commands from a Mosaic client. After the Natural routine extracts the data, it is Emailed to the appropriate person.
Also available is a form describing our Podium software and allowing you to place an order. This also uses Email to send the order after extracting the information and echoing it back to the client.
Return to the top of
this page.
Encryption of data but not the MIME header is one approach. Because the MIME header contains fairly standard stuff, encrypting it could actually help someone to break our coded messages. How many resources to commit to the task? Do we need public keys and private keys? Do we port the encryption and decryption to MVS or offload some of the work?
If we have messages going between machines on the same TCP/IP subnet, then the messages should never leave that subnet. If a Unix box and MVS were on the same subnet in a secure location, they could talk in clear text. The less expensive Unix system could do handshakes, authentication, and decryption before forwarding a request to MVS or another Unix server. On the return trip the data would be encrypted and sent along.
Return to the top of
this page.
Initially we were modifying the Unix NATSVAL to use Netscape's SSL protocol with the client and NATCLI to forward the request to MVS. But this plan changed when we purchased a server from Netscape.
The server receives the request and uses a CGI program containing a modified version of NATCLI to forward the request to one of the existing MVS Natural servers. The mainframe Natural server invokes the subprogram and sends the response back to the CGI program on our Unix system. The Netscape server accepts the data from the CGI program, applies the encryption and forwards the result to the Netscape client.
This offloads the CPU cycles for handshaking, encryption, and encryption to the less expensive machine without using extracts or replication of production data. As of July 1995, this implementation is proceeding with several demo screens now available. To See these new examples just click here.
Return to the top of
this page.
When invoked from a Netscape client via a request like, "https://www.mvs.udel.edu/cgis/sslmvs.cgi?udmt1/mfu9001/x.html", the part after the "?" is of greatest interest. This "query string" is passed to SSLMVS via Unix standard in (STDIN) read statements. The "udt1" tells us which Natural server is to receive the request for subprogram "mfu9001" to be executed.
A modified version of NATCLI is called to do the communication with the Natural server and a transmission a header is inserted to make it appear as a direct request from a WWW client. This includes a fake MIME header and information on the original source of the request. As the data is received, it is passed back to the Netscape server via Unix standard out (STDOUT) using printf statements.
This technique uses no intermediate files to pass data. This is much more efficient than having the CGI program invoke some other service that would create a file, and write to it. Since it would then be required to open the file, read, close, and finally delete it.
The use of STDOUT is a slight problem during the debugging of the routines because it forces us to direct our debug write statements to a disk file to avoid having the Netscape server intercept them as returned information.
Return to the top of
this page.
This allows us to treat the Natural routines like SQL stored procedures. No need to extract, transmit, repackage, and store into another database on a server unless you really want to. If the access is not to intense, the files in question are large and the primary use is still on MVS, then leave it there until something better is available. Do not just move things in an effort to be cool.
Return to the top of
this page.
Return to the top of
this page.
Do we always need a thousand dollars worth of software plus our application code on a $1500 PC to provide good service?
Do we need to use up resources running and rerunning extracts and download in an effort to keep them current?
Do we need to spend large amounts of time and money converting Cobol and Natural programmers to C and Unix?
Do we need to totally re-engineer applications (the good as well as the bad), if all we really want to do is provide additional access routes?
Do we need to degrade our network's performance with uncountable numbers of packets to read records from dozens of servers just to build one screen?
More questions than answers? Maybe not. I think what we have presented is a way to get some more good mileage out of our existing resources. We do not have to choose between very large expenditures and winding up as road kill on the information superhighway.
In a networked world, some things must be scrapped and recreated but not everything. A handful of "nasty" requests can use up everyone's bandwidth and so every new solution must be evaluated for cost as well as benefit.
Just as an example think of placing files containing full motion, true color video on a server and running viewers from your PC's. It sounds reasonable, but a couple of these going at once could bring any network to a crawl. Any finite resource can and will be consumed be it clean water, forests, CPU cycles, or network bandwidth.
Departmental data and standalone systems seldom stay that way in a large enterprise. It is very easy to say yes to many small solutions that may require major effort to integrate later. BE FORWARD THINKING, but as history repeats itself, do not repeat the mistakes of the past.
Return to the top of
this page.