MTool - Performance monitoring tool for multiplatform distributed systems

Introduction

Quite a long title for such a small software package. But it tells you exactly what this article is about. In our networked world the amount of information is growing and becoming less and less manageable. The first idea for a such tool came from the CERT engineer, who works for one of the Europe's academic network - SWITCH, Swiss Academic and Research Network. SWITCH head office is located in Zurich. Less than two dozen of highly qualified engineers operate some 50 sites throughout Switzerland as well as the central SWITCH system at ETH Zurich computing center. While they are already using some performance monitoring tools the procedures are somehow not appropriate. Each time they want to monitor a remote system, they have to log in and use common tools such as top, ps, df, etc. This kind of procedure also brings some new security holes and this is the least something they want.

Performance monitoring

What they want is a simple, reliable and secure tool which could be easily used from any authorized place in the network (or Internet). The tool should be compact. That means that it could not take to much of the system's resources. Next, it should be multiplatform oriented. Eventhough they maintain mostly SUN workstations, the amount of Linux boxes and other UNIX systems they monitor is not negligible. Primarily we were interested in monitoring just basic system resources: CPU/memory/swap usage and some specific processes (daemons like inetd). They provide a first and quite faithful picture of a monitored system. In the second stage we decided to observe some additional parameters such as network traffic, disk usage, user/ftp connections and some other processes.

 

 Figure 1: Three-tier MTool's architecture
 

Three-tier architecture and Java

The demands for multiplatform, secure tool have brought us to a simple decision. As we have a strong students' Java team at our lab, Java was chosen as the basic development tool. It is a network oriented, robust tool and provides enough security to fulfill our expectations. We won't go deeply into details of the language itself but it is clear that Java is not appropriate for system programming which is needed for accessing system resources. That was the reason that we'd chosen a three-tier architecture (Fig. 1). This architecture represents a response to some of the drawbacks of the standard two-tier client/server network architecture. Client/server applications are quite easy to program but the problem is how to manage client side software. The problem is even bigger when we are surrounded with the multiplatform environment. Three-tier architecture introduces so called middleware (Java server on Fig. 1). Generally, this is a server software representing a bridge between a client and a server. The middle tier is not just capable of handling more client connections but it provides much more secure way of communication. And since the middleware and client part can be written in Java we come to the very important issue - the platform independence. Clients are no more urged to use platform dependent code. Even more, a copy of Java enabled network browser such a Netscape would suffice. The developers can now take a breath since they don't have to worry about many different platform dependent versions of their software. This is also a basic idea of the network computers (NC). Whenever the developers change the clients' code, a new version is always available on middle-tier server and next time they run the program, a new version would be automatically downloaded. In our case both client's software and middleware are written in Java. The platform dependent software which provides information to the clients over the middleware is written in ANSI C. The communication between middleware and monitored computers is realized through the sockets. More about that later in the technical part.

Linux

As we have already mentioned, first we had a daemon (mtoold) written for SUN workstations since they represent a majority in Switzerland's academic network. But the environment we come from is mainly equipped with HP workstations and, if we don't count Windows 95/NT computers, we have more than 70 Linux boxes around the classrooms and labs. I can say that we've learned a lot from Linux and the first beginnings reach in the year 1993 when we built our first Linux box using the 0.99pl15 kernel. These times are not so far away and we are very glad that Linux has made such a tremendous progress since that. So Linux as a development "playground" was then really a logical choice. Linux has a very well documented /proc file system which was also exhaustively explained in the April issue of Linux Journal. Talking about documentation there is really no such a documented operating system like Linux. What we really miss is a good Java development tool. If Linux would have a GUI like Symantec's Visual Cafe we would be very happy, By then, we are still forced to use command line Java compilers and interpreters that come with standard Java development kit (JDK 1.0 or JDK 1.1). In the mean time we use Visual Cafe under Windows for creation of the user interface and after we add some specific code manually.

Architecture of MTool

Three-tier architecture turned out as to be a good solution. We cope with three different levels:
 Figure 2: Three-tier architecture of MTool in details
 
To monitor specific systems you have to be connected to the network. Actually you have to connect to the middle tier Java server. Client's side presents just a GUI (Graphical User Interface). Since it is written in Java a Java capable browser is all that you need. After you make a connection to the Java server a GUI code is transferred to your local machine. Code is then interpreted with Java built-in interpreter (if you use i.e. Netscape). The central point of our system is of course that middle tier Java server. It does not only hold Java classes which are transferred to the local clients but it also takes care about the monitoring itself. Java server is a program that runs all the time but it demands only a few system resources. It addresses monitored computers and asks them for new data. Communication is realized via sockets and is very simple. We introduced also a very simple protocol for data transfer. Let's take a closer look to each part of our performance tool.

Client software

At the client's side one has to run a starting applet which brings up a basic window where all the computers that could be monitored are listed (Fig. 3). Once we choose a computer's name from the list a second window appears containing all the information about the observed system. We use a tabbed panel to display different sort of data (Fig. 4). We've chosen a tabbed panel due to ease of upgrade. If we would add some new system information there are no problems with the graphical interface - we simply add a new panel.

 

 Figure 3: Main MTool window
 
For the time being we can provide the following system resources:
 Figure 4: CPU and network usage panel
 
 Figure 5: Disk info and memory usage panel
 
 Figure 6: Processes (preconfigured selection) panel
 
 Figure 7: Users connections panel
 

Middleware - Java server

Java server is supposed to run on an independent computer but it can run also on one of the clients of one of the monitored computers. The communication with the monitored computers via sockets is very short and it lasts just for the time during the data transfer. We can choose any time how often Java server requests system resources data. Then a ``ping'' to all monitored computers is sent. To make a long story short at the end mtoold is woke up by the inetd. MToold reads system resources information and forwards it to the middle tier Java server. After completion a mtoold daemon ``dies'' till the next server's ping. The platform dependent daemon mtoold therefore ``lives'' and takes system resources just for a short time. For its operation Java server is supported by one configuration file, named MTool.conf. Its first part contains the list of computers to be monitored. A time between monitoring interval follows next, and the third part contains the list of clients' computers that are authorized to communicate with Java server and thus monitor selected computer systems. We can use both IP addresses or IP host names. As you can see from the sample configuration file below one can use also asterisk in defining domain name or subnetwork address. 
# First come hosts, that are to be monitored

cebelica.uni-mb.si
mravljica.uni-mb.si
cmrlj.uni-mb.si
strela.fcs.uni-mb.si
grom.uni-mb.si
164.8.253.16

# Monitoring time interval in sec
4

# Clients with access to monitored hosts
*.hermes.si
130.216.54.51
164.8.253.99
130.15.40.*
193.246.15.*
164.8.253.*

Platform dependent daemons

Just a few more words about the daemon mtoold. It is written in ANSI C and it is platform dependent. As a matter of fact it depends quite strong upon the platform. With Linux we use /proc file system for CPU, memory and network info, while SUN and HP do not provide such an elegant approach. There we have to use /dev/kmem files. In April's LJ there was a very explanatory article about Linux's /proc file system so there is no need to go into details. We should also mention a protocol defined for socket communication. Information is transferred as ASCII stream concatenated from reserved words such as GRAPH, VALUE, disk, user, process etc. and specific values which are separated with delimiter ``|''. A sample stream looks like: 
GRAPH|LOAD_avg1|0.06|GRAPH|LOAD_avg2|0.12|GRAPH|LOAD_avg3|0.22|
GRAPH|CPU_user|1.00|GRAPH|CPU_nice|0.00|GRAPH|CPU_system|0.00|
GRAPH|CPU_idle|99.00|VALUE|MEM_real|14652K|VALUE|MEM_free|252K|
VALUE|MEM_swap|33260K|VALUE|MEM_swap_free|31620K|
Without any difficulties we can add new observed parameter. For now one could intercept data transferred over the network. Therefore we also prepare Java encryption classes which would enable secure data transfer. We are in the middle of realization of DES and RSA algorithms. RSA should serve for key exchange (public and private) while DES which is faster would serve for the data transfer itself. A mtoold uses one simple configuration file which holds the names of the processes to be monitored. If the file is empty an information about all processes that are running on a monitored computer are transferred over the socket communication line.

At the end

Actually, we are not at the end. Current version of MTool is just an intermediate step towards more sophisticated and usable tool. But even now it provides a comfortable way of system monitoring. The main issue is that if we are authorized, we can monitor selected systems from any place in the world. A Java capable browser is enough. We also hope that we pointed out the main benefits of such an approach and that MTool it is a small but a powerful tool. Linux as a development environment was a really good and logical choice but still we would like to appeal to the (non)commercial software companies to provide more Java development tools under Linux. Linux and Java together represent a competitive, reliable and cheap development system.