C++ compilers for Win32

David Adams - 1996-09-19 16:21 (updated 2024-05-23 08:49)

A review of the following C++ compilers for Win32 software development: Borland C++ 5.01, IBM VisualAge for C++ for Windows 3.5, Microsoft Visual C++ 4.1 and 4.2, Symantec C++ 7.21, Watcom C++ 10.6.

• Introduction
• Borland C++ 5.01
• IBM VisualAge for C++ for Windows 3.5
• Microsoft Visual C++ 4.1
• Symantec C++ 7.21
• Watcom C++ 10.6
• Product summaries
• The tests
• Test 1: Hello world
• Test 2: BYTEMark version 2
• Test 3: Memory allocation
• Test 4: TLX class library
• Test 5: Constrained optimization
• Ad hoc tests
• Conclusions from the tests

Introduction

Amidst all the Internet and Java turmoil, one would almost forget that C++ still plays an important role in application development for Windows platforms. In this article, I’ll review 5 recently released C++ compilers for Win32 executables.

At the time of testing (June - September 1996) the most recent versions of the various C++ compilers were:

• Borland C++ 5.0/5.01, for DOS 16/32-bits, and Windows 16/32-bits applications;
• IBM VisualAge for C++ for Windows 3.5, for Windows 32-bits applications;
• Microsoft Visual C++ 4.1 (superseded by 4.2), for Windows 32-bits applications;
• Symantec C++ 7.21, for DOS 16/32-bits, and Windows 16/32-bits applications;
• Watcom C++ 10.6, for DOS 16/32-bits, Windows 16/32-bits, OS/2 1.x to 3.x, and Novell NetWare applications and modules.

All packages include, in addition to the compiler proper, a development environment with editors for text and resources, build tools such as MAKE and LINK utilities, appropriate debuggers, runtime libraries, SDK tools, etc. Furthermore, all compilers but the one from IBM include the Microsoft Foundation Classes (different versions; Borland includes source code for MFC 4.1 as of their 5.01 release). The Borland compiler also comes with the ObjectWindows Library, and the IBM compiler, finally, includes their Open Class library. MFC and OWL are probably familiar to you; Open Class is IBM’s class library (it isn’t a true application framework) with user interface and general data structure classes that is portable across Win32, OS/2, AIX, and MVS - in short, IBM’s C++ target platforms.

Throughout the review, I focus on the C++ compilers and directly related tools, and leave the application frameworks and additional tools (such as database designers) out of consideration. Also, the target platform is Win32 throughout (Intel only), even though some compilers do address other platforms as well. For testing, I used a Windows NT 3.51 computer (Servicepack 4, Pentium 120 CPU, 64 Mbytes RAM, 256 kbytes level-2 cache) and a Windows 95 computer (Servicepack 1, Pentium 120 CPU, 40 Mbytes RAM, 256 kbytes level-2 cache). Times reported are for the Windows NT environment.

Back to the top

Borland C++ 5.01

The Borland C++ compiler comes in three editions: Standard, Development Suite, and Development Suite with Design Tools. All versions include the same compilers and C++ development tools, plus Visual Database Tools for database-oriented applications. The Development Suite adds CodeGuard 32/16 (Borland’s runtime error detection tool, see my previous review "Testing Testers"), InstallShield Express, a light edition of the PVCS version manager, and some Java tools. On top of that, the Design Tools version bundles the Together/C++ CASE tool from Object International. The extra tools were not considered in the review.

The Borland C++ 5.01 environment must be hosted on Win32 platforms (although some tools, such as TLINK for 16-bits applications, are still 16-bits executables). It can target DOS 16-bits, DPMI-286 and DPMI-386 executables (be warned though, that Borland’s DOS PowerPack with its DOS extenders is not supported by the 5.0 compiler), Windows 16-bits, and Windows 32-bits executables. As an extra, you get an Intel-developed backend for the 32-bits compiler, which offers some advanced code optimizations. The ObjectWindows Library is suitable for both 16- and 32-bits Windows targets, with minor differences between the targets. As of version 5.01, source code for MFC 4.1 is included; all 5.0 versions already came with tools to patch MFC 3.2 and 4.0 in order to compile it with the Borland C++ compiler. Compilation can be performed both from the Integrated Development Environment and from the command line; in all cases, you’ll be working mostly with 32-bit tools, regardless of your target.

Borland includes stand-alone debuggers for DOS, Win16 and Win32, and an integrated debugger for Win32. Profilers are available for DOS and Win16 only. The IDE itself now includes the Resource Workshop, which used to be a separate program in previous releases.

Figure 1: Borland C++ 5.0 IDE (click here to see full size, 37 KB)

Experience report

In practice, I found the Borland IDE a reasonably good working environment, if fairly unstable (it used to crash a few times a day, although more so on Windows 95 computers than on Windows NT ones). It provides a lot of flexibility, and tools such as TargetExpert and the use of hierarchical displays for options settings make it fairly easy to change compilation options without losing sight of the big picture. The text editor is alright, and so are the resource editors. If you work with ObjectWindows, you’ll find AppExpert and ClassExpert useful to get you started, although for serious work, you’ll probably need to resort to manual editing after all. Compilation is fairly quick and the resulting executables have fairly good performance, if a bit overweight (see the test results at the end of the article for details).

New with the 5.0 release of the Borland C++ compiler is the cScript scripting language to control the IDE (or any OLE Automation server, for that matter). Through its C++-like syntax, you can access all of the many objects that make up the IDE and its constituting parts, and control them in whatever way you like. The IDE itself and several add-on tools (such as the Java tools and PVCS) make use of this capability, but you can also add your own scripts if you like or modify the standard ones to suit your taste. Apart from making some modifications to the layout of the various IDE windows, I did not use cScript. You may find it useful to automate build procedures etc., though.

What I find wanting overall in the IDE, is perhaps best described by a lack of "pointedness". For all its flexibility through Source Pools, Style Sheets, and, as of release 5.0, an extensive scripting language for the IDE, I still find myself configuring each and every project. The default settings are neither good for debugging nor for release, and their is no easy way to switch from one configuration to the other. Sure enough, Style Sheets and Source Pools let you create any configuration you like and switch from one to the other, but what I like is a good and easy setup for the two most commonly used configurations: Debug and Release. To add to the problem, it is tedious, to say the least, to share or migrate Style Sheets and such from one project to the next. Other areas that troubled me were the fact that a change in compilation options does not cause the IDE to mark affected files for recompilation (you’ll have to do a Build All manually) and on top of that, that a Build All does not clean up output files first, which means that if your rebuild stops after 98 of 100 files due to an error, you’ll have to re-rebuild the first 97 files in order to ensure that files 98-100 get rebuilt as well. In the end, I resorted to hand-crafted Makefiles for most projects worth the name. Some final quirks: the Borland C++ compiler looks for #include "header.h" files in the current directory rather than in the directory containing the including source or header file. While this is perfectly legal, the behavior differs from most other C++ compilers and also leads to problematic situations when referring to another project’s nested header files from a new project. Finally, the code generated by the Intel compiler backend is not always reliable; in fact quite a few test programs crashed when their code was generated by the Intel rather than the Borland backend. Since the gains in performance are normally not so dramatic and the compilation time increases noticeable, using the Intel backend did not make sense for me.

Conclusion

Overall, I find the Borland C++ environment satisfactory to good, with a lot of useful tools if you buy one of the Development Suite versions. Notwithstanding my criticisms, the IDE is very useful for development work. The compilation output is also acceptable to good, but if Borland added function-level linking, that would probably reduce the size of the final executables. Another reason to choose the Borland tools could be the ObjectWindows Library, which is probably the best way to go if you look for 16/32-bits Windows portability.

Paradoxical as it may seem, I think that the IDE would benefit from less flexibility and more to-the-point support for common configurations and situations. After all, development work is hard enough as it is, and the last thing I need is having to develop my development environment before I start development – if you get my meaning.

Back to the top

IBM VisualAge for C++ for Windows 3.5

With the VisualAge for C++ for Windows, Version 3.5 (sic) product, IBM is entering the market for Win32 C++ development tools. According to IBM, one goal of this product is to offer its customers portability to all platforms IBM thinks important (Win32, OS/2, AIX, MVS, not necessarily in that order). VisualAge for C++ comes with the compiler for Win32 targets, editors for text and resources (incidentally, the resource editors are a Win32 incarnation of Borland’s old Resource Workshop), a debugger, a profiler, Visual Builder, and a WorkFrame to integrate the lot. To ease the transition from OS/2, the Workframe and several other GUI tools have a distinct OS/2 look and feel.

The product must be hosted on a Win32 platform and targets only Win32 (although source code compatibility is maintained with other IBM platforms as long as you don’t use platform-specific APIs). The Open Class library is a large collection of C++ classes addressing areas such as data structures and user interface elements; it is not a true application framework, nor are there any "AppExperts" or similar wizard-like tools available for it. (IBM does provide "Project Smarts", though.)

Figure 2: IBM VisualAge for C++ for Windows 3.5 WorkFrame plus tools (click here to see full size, 41 KB)

Experience report

Let’s not beat around the bush: VisualAge for C++ for Windows is a disaster as far as I am concerned. It is by far the most anti-productive environment I came across, C++ compilers or otherwise (and I did use IBM CSet and CSet++ on OS/2). It all started during installation on my Windows 95 system; since then, that system has a tendency to open all its folders in Small Icon view with a toolbar, regardless of what it used to be. I don’t mind IBM having its own preferences, but I don’t want them to change my system without, at the very least, asking. Anyway, on a Windows 95 system it then requires no less than 20000 bytes of environment space. On both Windows 95 and Windows NT systems, it adds 22 different environment variables, with room for extensions, to communicate settings between the various tools. Apparently, they never heard of the Registry.

Starting the WorkFrame, which is IBM’s idea of an integrated development environment, takes quite some time. To improve load times for the second and subsequent runs, they simply leave 6 processes running when the WorkFrame session ends. According to IBM this is accepted practice on OS/2; still I was not pleased with it on my Win32 systems. Having started the WorkFrame, you can then choose from various application models through the use of Project Smarts. A static library is not among the options as I found during one of my tests, but for the ones that are, you sometimes wish you’d never started in the first place. The wizard-like approach for project configuration takes you through a number of tabbed dialog pages à la OS/2 (with spiral binders and all that), and they simply look gross on the Windows platforms. What’s worse, elementary options such as a Browse button to navigate to a project directory are missing. Furthermore, the overall project model (see below) is quite different from all other Win32 compilers, which makes for some very confusing initial steps.

Once you arrive at the end of the wizard sequence, the regular WorkFrame window appears. Where other C++ environments consider a "project" as a repository containing references to the actual source files (wherever those may be located), a WorkFrame project is simply the combined contents of all directories that you tell it to consider as project directories. As a result, your WorkFrame window will show you all files indiscriminately, be it source files, object files, makefiles, backup files, you name it. For any sizable project, this will run to hundreds and hundreds of files. Filters allow you to clean up the visible mess a bit, but a mess it is.

Before you can actually compile anything, you must use tools such as MakeMake to designate which files are source files and which aren’t, tell the darn thing that it should use the C++ compiler for .cpp files and the resource compiler for .rc files (and don’t even try to build a static library), and specify compilation options for all these types. Then, MakeMake will generate a Makefile for you which supposedly compiles your sources. Well, sometimes it does, and sometimes it doesn’t. MakeMake is particularly strong at adding superfluous items (such as "echo Compile" to every compilation target’s build rules – as if you wouldn’t know when you see the compiler’s banner), but it is distinctly weak in transferring the right compilation options to the Makefile (as a rule, they were not transferred correctly) and in generating the header dependencies for each source file. This last point was illustrated when I tried to compile MFC 4.1 sources with this compiler: it would take almost 18 minutes to generate all the dependencies. Contrast this with less than 5 minutes for a straightforward Makedep tool, a minute and a half for the Symantec C++ compiler, and 35 seconds for an optimized MkDep tool that I subsequently developed. Worse, after each change to compilation options, MakeMake insists on a (needless) rescan of the dependencies which takes, in the MFC case, another 15 to 18 minutes. In the end, I just gave up on WorkFrame and its tools and used my hand-crafted Makefiles if I had to test IBM’s compiler.

As to the other tools, I frankly admit that I never came to use most of them. The whole product just exasparated me. One that I did use, just for laughs, was the text editor called "Live Parsing Editor". In normal speak, this means that it has syntax coloring. What sets this editor off from other vendors’ editors, is its lack of speed (I could literally see syntax coloring and toolbar updates in progress, even on my Pentium 120 systems), and its ignorance about tab characters, of all things. Enough said.

I also used the online help. Rather than using the WinHelp engine, IBM’s engineers went one better and ported OS/2’s Information Presentation Facility to Win32. As a result, I have to push my chair back when reading their information (since ancient times, IPF comes configured with ISO and DIN compliant fonts which are way too large for my 20" monitor and fill up 80% of its not inconsiderable screen area), and have to live without the search and indexing capabilities of WinHelp 4.0. Just another nail on VisualAge’s coffin.

Conclusion

Don’t, unless your boss forces you and you can’t go anywhere else. As a Win32 development system it’s just laughing stock; as a portability tool from OS/2, AIX, or MVS it may be somewhat more acceptable, but if you are serious about developing Win32 applications, look elsewhere. Every other C++ development system in this review is better, and often by a very, very long way.

Back to the top

Microsoft Visual C++ 4.1

Microsoft Visual C++ 4.1 is the subscription upgrade to the 4.0 version (I also tested the 4.2 successor to 4.1). This range of compilers only targets Win32 platforms; Win16 is out, as far as Microsoft is concerned (although the most recent 16-bits version of the compiler, 1.52, is still bundled with 4.x). The version I tested is for Intel host and target platforms; there is also a Macintosh cross compiler (hosted on Win32 for Intel), and there are Win32 compilers for MIPS, AXP, and PowerPC. Both the compiler and the linker are of the incremental type, which means that they will only recompile or relink those parts of a source file or executable that have actually changed since the last build.

The Microsoft development environment is called the Developer Studio; it is shared by the C++ tools, Fortran PowerStation, Visual Test, the Developer Network, and recently, J++. Of course, you’ll have to obtain the non-C++ tools separately if you want them. Visual C++ 4.1 comes with the Microsoft Foundation Classes 4.1 (4.2 comes with MFC 4.2), which is all but the de facto standard for Win32 C++ programming. In fact, several development tools that used to be separate products, such as the OLE Control Development Kit, are now integrated into MFC. Recent upgrades have concentrated on adding, in particular, Internet support classes to MFC, but also C++ language enhancements and, in version 4.2, a draft-standard compliant C++ library (to replace STL and the older IOStreams).

The Developer Studio integrates the text and resource editors, the debugger, and project management tools; from it, the actual compilation and build process is performed through command line versions of the compiler, linker, etc. To support programming with MFC, AppWizard and ClassWizard are included. The online help is particularly good and deserves special mention; it now uses the same InfoViewer tools as MSDN, and also allows you to access the MSDN CD’s from within the Developer Studio. A profiler and several other tools are present as command line programs, but accessible through the Tools menu.

Figure 3: Microsoft Visual C++ 4.1 Developer Studio (click here to see full size, 33 KB)

Experience report

This is one development environment I really like. It is not quite so configurable as Borland’s or Symantec’s, but almost everything in it is spot on. In terms of usability, I consider this environment nearly on a par with Microsoft’s Office applications, and I wouldn’t say that of any other development environment. Just about everything feels right. Initial project configurations are alright, switching between them or adding new ones is a breeze, commands are available just where you expect them to be, it looks good, and it feels very responsive (even though some timings indicate that this is not the fastest compiler around). The online help is just superb and I have never felt more comfortable with a debugger than with the one integrated in the Developer Studio. Also, support for external tools is good to very good, and in my experience, the whole thing is as stable as a rock. In fact, this is the only development environment that I routinely use and don’t feel tempted to leave for real work.

Still, there are a few things wanting here. First of all, some macro programming capability for the editor would be nice, if only to insert commonly used text such as file and function headers. Next, a multi-file search and replace facility would be helpful. Further, and this relates to the compiler itself, I got quite some internal compiler error messages. Thanks to the online help, I found out that most were caused by the compiler running out of steam during optimizations, and switching off one of the optimizations (as indicated by the online help) solved the problem in most cases. Still, version 4.1 generated just plain bad code in one of my test programs (this problem went away with version 4.2). Finally, the build manager (or the incremental compiler or ditto linker) is sometimes overoptimistic and fails to recompile some of the dependents of a piece of source code that has changed. In these cases, you’ll have to perform a Build All to get everything right.

Conclusion

If you’re targeting Win32 applications only, Microsoft Visual C++ is probably the best choice you can make. Its Developer Studio is exemplary, the compiler consistently generates compact and fast code, and MFC is a widely accepted C++ framework for Win32 programming. For some programming tasks, such as OLE controls, using this product is all but mandatory because it is the only product to support them. Finally, many additional tools (such as error detection and source control) come out with Visual C++ support first, and support for other compilers, if at all, next.

On the downside, portability to 16-bits platforms (be it DOS or Windows) is more difficult to achieve. The 16-bits C++ compilers from Microsoft do not support C++ features such as templates and exception handling, and also the 16-bits MFC version differs from its 32-bits counterpart. Still, an exceptionally strong combination of power and usability.

Back to the top

Symantec C++ 7.21

Figure 4: Symantec C++ 7.21 IDDE (click here to see full size, 39 KB)

Symantec C++ (née Zortech C++) is positioned as the alternative to Borland and Microsoft C++ compilers. As such, it offers compatibility in both directions, and adds speed, portability, and extensive compilation options. In recent years, usability has improved quite somewhat, and the current Symantec C++ IDDE is comparable to or better than most. As a special feature, Symantec C++ sports NetBuild, which is the ability to distribute the build process across a number of networked computers.

Symantec C++ 7.21 comes with Windows 16- and 32-bits versions of its environment (including all tools), and targets DOS 16-bits and 286 and 386 extender models, and Windows 16- and 32-bits. As a result, you can use every combination of (Windows) host and target platforms for program development. The product includes debuggers for all platforms (developments of the famous MultiScope debuggers), text and resource editors, and of course things like Make and link tools. The latter deserves special mention: it is the latest version of OPTLINK (another acquisition of Symantec) and links executables in unbelievably short times.

As far as frameworks are concerned, Symantec C++ 7.21 includes MFC 2.52 (16-bits) and 3.2 (32-bits), and comes with AppExpress and ClassExpress to help you on the way.

Experience report

Symantec C++ is fast, if anything. Speed permeates from the startup of the IDDE, through all compilation and build tools, to the final executables, which are among the smallest and fastest of the lot. In addition, the wide range of target platforms (and host platforms, for that matter), make cross-platform development almost fun. The IDDE user interface model is somewhat different from the usual MDI (multi-document interface) type of environments, in that the basic IDDE consists of nothing more than a title bar, menus, a tool bar, and a set of tabs to select a workspace (Edit, Debug, Output, etc.). The actual content windows just float as ever so many windows on the normal Windows desktop, which remains visible all the time. Basically, this is a nice idea as it allows more visual integration with and easier access to other applications on your desktop, but in practice I found that it tended to increase visual clutter and made it more difficult to tell at a glance which windows belonged to Symantec C++, and which to other applications.

In daily work, the general pattern of operations follows the pre-4.x Microsoft Visual C++ Workbench model, including accelerator key assignments. There are standard provisions for Debug and Release versions of each project, for subprojects, and for migration of options between projects. On the whole, however, I found it not quite as easy to use as the new (4.x) Visual C++ environments, but still more than adequate.

The C++ compiler is very quick, but somewhat too permissive in my view. It lets many questionable constructs pass without warning, and the diagnostics that it does generate are not always as self-descriptive or accurate as they should be. The latter problem becomes worse with errors in templates; in that case, the diagnostic often refers to the last line of the file containing the template definition, rather than to the offending line within that file. Templates also seem to be a problem as far as compilation speed is concerned: while Symantec C++ is normally very quick to compile C++ source code, the presence of template expansions tends to increase overall compilation time quite dramatically.

The Symantec C++ IDDE includes a Symantec Basic scripting language which can be used to automate tasks from within the IDDE. Although the feature looks enticing and its language is familiar, I must confess that I never used it during all the time I worked with the IDDE. This may be different, however, if you rely mainly on the Symantec C++ IDDE and find it worthwhile to invest your time in developing scripts for it.

Conclusion

Symantec C++ was, and still is, a viable alternative to both Borland C++ and Microsoft C++. In comparison to the former, it has the advantage of being quicker and generating smaller and faster code on the whole; compared to the latter it offers better cross-platform portability and quicker operation. The drawback, compared to both, is that it will normally lag somewhat as far as framework support is concerned (MFC is one version behind, OWL is not even included), and that it normally only ranks third or fourth when it comes to support from external tool vendors. Still, with a number of "best of breed" tools such as the compiler, linker and debuggers, it is a strong choice.

Back to the top

Watcom C++ 10.6

Watcom (acquired by PowerSoft) has a reputation for cross-platform development tools. With Watcom C++ 10.6, you get compilers that target DOS 16-bits, 286 and 386 extended, Windows 16- and 32-bits, OS/2 1.x through 3.x, and Novell NetWare. As far as hosting is concerned, you can run all command line tools on all platforms (excluding NetWare) and all GUI tools on Windows and OS/2.

The product further includes the necessary supporting tools (such as debuggers and profilers), and lots of libraries and toolkits, among which MFC 2.52 and 3.2, and a SOM toolkit for OS/2. Blue Sky’s Visual Programmer assists in developing MFC applications. The compilers and linkers are all command line tools, but they are normally run from within the IDE.

Figure 5: Watcom C++ 10.6 IDE (click here to see full size, 35 KB)

Experience report

"Looks don’t count for machos" seems to apply to the Watcom IDE. The IDE acts as the central switchboard and project manager, and while it is very usable, it does not look quite so slick as some of the others. Of course, this is at least partly caused by the fact that Watcom uses the same IDE across all GUI-supporting platforms, and that necessarily precludes using some of the more spiffy features of each individual platform.

Having said that, I find the Watcom designers masters at achieving maximum effect with minimal means. For example, when I first tried establishing build dependencies between subprojects, I could not find the right options to do so; just moments later I discovered that the project manager will automatically establish the correct relationships, just based on knowledge of target types (.lib, .dll, etc.) The same applies to issues such as a Build All, which is no big deal to the IDE: just set the date/time stamps of all output files to 1/1/80 and perform a regular Make action. Changes to compilation options, either C or C++, are likewise recognized and dealt with. While this is not as sophisticated as, say, Borland’s project manager, I find it a good deal more effective and much more reliable as well.

All tools, including the editor, are external to the IDE and are simply run as separate processes. This makes for a very fast loading IDE, and also makes it easy to substitute different tools for the default ones. It does take a bit more time to load a text file, but it is quite acceptable in practice. What I found more bothersome, however, was the overall lack of speed in the compiler. For small programs, this was not such a problem, but it became quite noticeable for larger projects. For example, compiling the MFC 3.2 sources from scratch takes 10:14 (minutes:seconds) for the Watcom C++ compiler, but only 2:49 for its Symantec counterpart. The same pattern recurred quite consistently for all but the smallest programs. In addition to that, the compiler output is quite verbose, even if you tell it not to be. Four lines produced by a single diagnostic is not uncommon, and the problem is exacerbated by the fact that selectively suppressing warnings by pragmas in the source code is not very intuitive.

Source code compatibility with Borland and Microsoft C++ compilers is not as good as I had hoped for; furthermore, the compiler does not yet support C++ runtime type information and the new-style typecasts. Finally, I had some problems with the profiler (which is implemented as separate sampling and presentation tools): somehow or other I managed to crash one or the other every time I used them.

Conclusion

If you’re in to multi-platform C++ development, Watcom C++ is probably one of your primary candidates. It is one of the few compilers that supports such a wide range of target platforms (in fact, not of the others in this review matches it, and you’d have to go to MetaWare to find comparable support) and offers a load of supporting tools and libraries to back that up.

You may find the fairly low compilation speed or lack of support for the latest C++ features a bit of a problem, but on the other hand the resulting executables are usually among the smaller and the fastest of the round-up.

Back to the top

Product summaries

The following table summarizes the most important features of the C++ development systems reviewed here. Please note that every compiler is assumed to address both Windows 95 and Windows NT as host and target platforms. Only features that come "in the box" are taken into account.

Note: Borland 32-bits inline assembly requires separate TASM32 product; inline assembly is supported for 16-bits targets.

Back to the top

The tests

In addition to the general experience reports, I have conducted a number of tests to find out how different compilers behave on different tasks. The requirement to run the tests across the board necessitated some limitations (compatibility is never so good as to allow you to compile larger projects on all compilers without substantial extra effort, which I couldn’t afford). Having said that, I tried to select tests that although small, will highlight important differences between compilers. Here’s the overview:

• Hello world, the classical starters program. Included in both C and C++ versions to find out how small an executable the compiler would create.
• BYTEMark version 2. BYTE’s C-based benchmark program, intended for evaluation of system performance but quite usable for compiler comparisons as well, it turns out.
• Memory allocation, to measure the performance of the runtime malloc() and free() routines. Based on Arthur Applegate’s article in DDJ of June 1994.
• TLX class library, a C++ class library with data structures and algorithms that includes things such as class and function templates, yet is portable across a wide range of C++ compilers.
• Constrained optimization, an application of the TLX library which contains a mixture of data structure manipulations, calculations, and memory management operations.
• A number of ad hoc tests, in particular some experiences with MFC compilations on several compilers.

Test results as reported apply to the Windows NT system mentioned in the introduction of this article. Compiler settings were the following, unless otherwise noted:

• Pentium code generation, with 8-byte alignment for data structures.
• Optimize for speed, fast function call model (arguments passed in registers), fast floating point.
• No standard stack frame, no stack overflow checking.
• Function-level linking, dead code elimination, unreferenced function removal (if available).
• C++ exception handling and RTTI enabled, if available, with destructor cleanup.
• No debug or browser information; file section alignments at their smallest allowable value.
• Runtime libraries were statically linked, using the non-debug, single-threaded versions.
• Precompiled headers were neither generated nor used.
• All build times refer to a full rebuild, either from the compiler’s IDE (if possible) or from a separate Makefile.

For the Borland compiler, I ran each test for both the Borland and the Intel compiler backend; the results are reported separately. All tests, including the build processes themselves, were run several times to obtain the final results, interspersed with unrelated applications to flush NT’s disk caches. Background activity on the test system was reduced to a minimum; the test results turned out to be very reproducible.

Test 1: Hello world

This test is provided in both C (using <stdio.h> and printf()) and C++ (using <iostream.h> and cout) versions. It is intended to check the minimal executable size. Optimizations were set for size rather than speed.

*: Borland results are reported as Borland/Intel backend.

**: 33280 bytes without exception handling; Watcom C++ 10.6 does not support RTTI.

Comments. Even small programs, in particular small C++ programs, end up with quite some code in their executable. This is due to features such as exception handling (although support for C++ exception handling shares code with support for Win32 Structured Exception Handling that is also present in C programs) and the additional overhead for C++ class libraries. For example, the IOStreams library is often implemented in terms of the C Stdio facilities. Compilers with function-level linking have an advantage in this test.

Test 2: BYTEMark version 2

This test is in essence the BYTE benchmark of the same name. For the test, I cleaned up the code somewhat and added support for the compilers that were not supported in the original BYTE source code. The program is fully C based.

*: Borland results are reported as Borland/Intel backend.

Comments. The scores as mentioned in this table refer to separate subtests of the benchmark program and are normalized relative to BYTE’s standard system for these tests, a Dell 90 MHz Pentium, and a benchmark program compiled with Watcom C++ 10.0. Given that I ran my tests on a 120 MHz Pentium system, one would expect, everything else being equal, benchmark scores of about 1.33. Any other results should be attributed to a non-linear scale up of the computer hardware and, more to the point for the present purpose, differences in code generated by the compilers.

The Watcom C++ 10.6 benchmark results, which are supposedly comparable to BYTE’s Watcom C++ 10.0 results, indicate that as far as the hardware is concerned, the scaling effect holds to within a few percent. If we take this as the reference point for the other compilers, we find that only Microsoft’s compiler performs consistently better. As far as integer scores are concerned, IBM is close to Watcom and Borland is noticeably better (in particular with the Intel backend), but Microsoft heads the list. Floating point is where the most dramatic differences are to be found, however. Borland and Symantec are performing quite poorly on this aspect of the test, and IBM’s is below par as well. Again, only Watcom and Microsoft perform as expected.

Test 3: Memory allocation

This test intends to measure the performance of the memory allocation routines in the compiler’s C runtime library (which are also used to implement C++ memory allocators). In a program based on Arthur Applegate’s June 1994 article in DDJ, I let each program allocate and free (in some random order) a number of memory blocks of varying sizes. The net result in a somewhat fragmented heap. As an extension to the original program, I varied the allocation pattern both with respect to block size and to overall number of bytes allocated.

Comments. The reason I chose the two dimensions, block size and total allocation size, is that I noticed quite different behavior along these dimensions. For the best performer in the test, Symantec C++, the only noticeable effect was from increased block size: with larger blocks, it spends more time per block regardless of the overall allocation size. This also holds true for Borland C++, the second best performer. I attribute the block size effect to the fact that these compilers implement their own suballocators and thereby amortize (expensive) system allocation calls over a number of subblocks. With larger subblocks, each system allocation call serves fewer subblocks and hence incurs a greater cost per block.

However, with the other compilers time per block also varied within a given block range size. In those cases, I presume their suballocators deteriorate as the heaps become more fragmented during the test and free pool search times become more and more important. A point in this case is Microsoft’s compiler, which is notorious for its poor memory allocator. Starting with version 4.2, Microsoft included an improved suballocator for small blocks and its effect can be seen in the table. Apart from lower times per block for small blocks, they are also virtually constant for a given block size range.

Test 4: TLX class library

This class library contains about 80 C++ classes and class templates and was developed to provide commonly used data structures and algorithms for use in operations research. The source code is portable across some 15 C++ compilers for PC and Unix systems, which makes it suitable for compiler comparisons.

*: Borland results are reported as Borland/Intel backend.

Comments. The main objective of the test is to compare build times for a non-trivial amount of C++ source code (about 25000 lines) and the resulting library size. The latter is only an indirect indication of the size of the object code, because .LIB files also contain extra information such as dictionaries.

Having said that, the Symantec compiler is both the fastest compiler and the one that generates the most compact library file. Microsoft’s compiler generated the largest .LIB file. The .EXE file mentioned in the table is a very small program that does nothing but print out the library’s version and build information. It is used as a rudimentary test of the proper compilation of the library. In this case, it indicates that as far as overall executable size is concerned, .LIB file sizes are no indication at all.

Test 5: Constrained optimization

This test builds on the previous one and applies the TLX library classes to a real optimization problem. It tests the compiler’s ability to generate code from class templates and, at runtime, the performance of the resulting code in situations that do not have the locality of reference of for example the BYTE benchmarks. As such, this test is probably more representative of "real" programs.

*: Borland results are reported as Borland/Intel backend.

**: The code generated by the Intel backend crashed shortly after program start.

***: The code generated by the Microsoft compiler (version 4.1) contained errors in a switch statement embedded in a while loop. As a result, the program entered an endless loop after finding the first solution. Version 4.2 did not have this problem; it completed the test in 1:13. Its executable size was 175616 bytes.

Comments. The .EXE sizes are fairly similar across the board, with the exception of a large Intel-generated executable for Borland and a small executable for Microsoft. Compilation times were more different, though, and again the worst performer was the Intel backend. Coupled with errors in the generated code, my advice would be not to use the Intel backend at all. Of the "normal" compilers, Watcom was once more the slowest of the lot, although the resulting code was reasonably compact and among the faster ones.

Run times to the first solution were pretty evenly matched, as were in fact the run times until 100000 problem nodes had been explored. Version 4.1 of the Visual C++ compiler had a code generation problem that created an endless loop in the program; version 4.2 did generate correct code and completed the test in the fastest overall time. Still the results seem to indicate that for a program that spends it time more evenly spread over various activities (control structures, memory allocations, calculations), different compilers behave more alike than focused tests such as the BYTE benchmark would indicate.

Ad hoc tests

I did some experiments with other source code as well, in particular the MFC source code. However, these were not performed consistently across all compilers in the review, so you’ll have to take the results with a grain of salt.

• Compiling MFC 4.1 source code with the Borland C++ compiler to a non-debug DLL generated a BFC40.DLL file of 1343488 bytes. Microsoft’s MFC40.DLL for MFC 4.1 is 921872 bytes.
• Compiling MFC 3.2 source code to a non-debug DLL with Symantec C++ resulted in a DLL file of 375808 bytes; Watcom’s was 526848 bytes. However, the latter contained more function entry points in its import library, some I am not quite sure if the DLLs contained the same classes. Microsoft’s DLL for this version measures 322822 bytes. Build time for Symantec was 2:49, while Watcom took 10:14.

Conclusions from the tests

Overall, Microsoft’s compiler created the fastest and smallest executables, although in one case, the version 4.1 compiler generated faulty code (4.2 was alright here). Build times were midrange for this compiler, but bear in mind that all builds were "clean"; there was no use made of either incremental compiling or incremental linking, which could improve times for this compiler in the case of partial builds. The weak point of version 4.1 is its memory allocator; the version 4.2 allocator is better for small block sizes, but still not as good as Symantec’s.

Symantec’s compiler output was often comparable to Microsoft’s, but build times were noticeably shorter. However, floating point performance in the BYTE benchmark was disappointing and does need further investigation.

Borland and Watcom performed comparably, but behind the Microsoft and Symantec compilers, with Watcom being rather slow in terms of build times. However, Watcom’s executables were generally smaller than Borland’s. The Intel compiler backend that comes with the Borland compiler is no great success as far as I am concerned; although it improves performance in some respects (see the BYTE benchmark), it also leads to increased build times and executable sizes, plus sometimes faulty code on top of that.

IBM’s compiler, finally, is a mixed case. Its executables sport midrange behavior, but are among the largest of the lot. Build times aren’t very good either. Most important, however, is its lack of good non-command line support. As it is, the Makefiles generated by Workframe were worse than useless and the required effort to get something compiled was disproportionate. I am sure that this also affected the outcome of some of the benchmarks, because between the disfunctioning Workframe and the less-than-accessible online docs, I had a hard time just finding command line compiler options, let alone tuning them to their best.