Skip navigation

Standard Performance Evaluation Corporation

 
 

JVM98 Bugs and Requests for Enhancements

08-Mar-99

Current Problems

These are open bugs that are either flaws in the benchmark suite, or are still under investigation and may be flaws in the benchmark suite.

8 Not 100% Pure Java. ASCII character set dependencies
12 InstallShield seg faults when you "Cancel" before you install.
13 Cannot install using JDK1.2
17 Running SpecApplication doesn't work in JDK1.2
20 Microsoft JVM "hangs"
22 Security failure with JDK1.2beta4 on Windows 98
23 InstallShield occasionally "hangs" on NT
26 Not 100% Pure Java. Line termination character dependencies.
29 Console button incorrect if window closed manually
30 Memory allocations listed are misleading
31 Incorrect comment in spec.harness.ProgrammerRunner
32 IllegalThreadStateException: cannot find file
33 Array bounds test does not work as intended
34 Class files missing from 1.03 update installation

Other

These are requests for enhancements or bugs that have been closed. They are listed here to document solutions for other users.

1 Out of memory
2 Applet won't run. Security exception.
3 Netscape 4.04 won't run
4 Execution times are not reported correctly
5 No network activity during benchmark run
6 Corrupted installation
7 Uses deprecated methods
9 Result pages don't print correctly
10 Problems moving result pages
11 Autorun per thread
14 Cannot browse source code from a web server
15 _213_javac fails due to hardcoded path names
16 Superfluous subdirectory
18 Wrong path names are constructed on NT
19 Wrong release number in documentation
21 Why did I get "invalid category" for memory?
24 Will SPECjvm98 support PersonalJava?
25 Cannot use MacOS web server. File name length limitation.
27 Text files have UNIX line termination
28 UNIX scripts are installed without execute permission

Keywords

  • allocation: 30
  • API1.0.2: 7
  • API1.1: 7
  • applet: 2
  • ASCII: 8
  • autorun: 31
  • BasePeakGraph: 10
  • CLASSPATH: 5
  • codebase: 10
  • compile: 6
  • console: 29
  • deprecated: 7
  • directory: 16
  • EBCDIC: 8
  • embedded: 24
  • file: 2
  • File.separatorChar: 15 , 17 , 18 , 26 , 32
  • filename: 25
  • GIF: 10
  • hang: 20
  • IDE: 6
  • install: 6 , 12 , 13 , 23 , 27
  • JAR: 20
  • JDK1.1.2: 3
  • JDK1.2: 13 , 17 , 18 , 22 , 32
  • jview: 18
  • LINE_SEPARATOR: 26 , 27
  • MacOS: 25 , 26 , 27
  • mainframe: 8
  • memory: 1 , 21 , 24 , 30
  • MSIE: 20
  • Netscape: 3 , 9
  • network: 5
  • NoClassDefFoundError: 1
  • NT: 12 , 13 , 18 , 23 , 32
  • OutOfMemoryError: 1
  • permission: 28
  • PersonalJava: 24
  • print: 9
  • reporting: 34
  • run: 28
  • script: 28
  • SDK: 18 , 20
  • security: 2
  • size: 31
  • source: 14
  • thread: 3 , 11
  • timing: 4
  • UNIX: 28
  • validation: 33
  • version: 19
  • web: 5 , 14
  • Windows: 27
  • Windows-98: 22

Types

bug
JVM98 doesn't work as it should.
rfe
Request For Enhancement - a desirable addition for a future release.
usage
Problem with how the benchmark is used. WorkAround gives the solution.

Note

Versions "v99" refer to pre-release development versions
"9.99" refer to CDROM releases
"9.99_99" refer to web downloaded updaters.

Bugid: 1
Type: usage
Status: closed
Release: 1.0
Synopsis: Out of memory
Keywords: memory OutOfMemoryError NoClassDefFoundError
Opened: 19980819
Fixed:  
Description:
The JVM Client98 program comes up OK, but then when I try to run
benchmarks it fails with the error java.lang.OutOfMemoryError

Appletviewer fails with a java.lang.NoClassDefFoundError complaining
that it cannot find some file. What is the problem?
WorkAround:
The benchmarks require at least 20MB of heap memory. On different systems
they may require more heap, and will usually run faster with more heap. Many
java interpreters have a property or command line flag to control the heap
size. E.g.

appletviewer -J-mx32m http://server/jvm98/SpecApplet.html
Comments: Closed: user configuration error

Bugid: 2
Type: usage
Status: closed
Release: 1.0
Synopsis: Applet won't run. Security exception.
Keywords: applet security file
Opened: 19980819
Fixed:  
Description:
I'm running Netscape Communicator, loading SpecApplet.html from the
directory where I installed the suite, and all I see is a grey box. The
applet doesn't run. I opened Netscape's Java console and saw the error
message:

Security Exception: class from local disk trying to access url:file:/home/jvm98/props/user
WorkAround:
You're trying to run the applet from a local disk (file:/) and Netscape's
security manager refuses to allow the applet to read files from your disk.
First you should note that the run rules require for reportable results that
you run from a web server instead of from a local disk. If you do this, then
Netscape's security manager will not prevent the applet from reading the
necessary configuration files from the applet host.

However, you may want to run from the local disk for test purposes, not for
public reporting of results. In this case you may use another web browser
that allows you finer control of your security preferences, e.g.
appletviewer from the Java Development Kit, Microsoft Internet Explorer, or
HotJava. In this case you may need to set properties or answer dialog boxes
giving authorization for the applet to read from your disk. See Advanced
Topics for more information.
Comments: Closed: user configuration error

Bugid: 3
Type: usage
Status: closed
Release: 1.0
Synopsis: Netscape 4.04 won't run
Keywords: Netscape JDK1.1.2 thread
Opened: 19980819
Fixed:  
Description:
I'm getting errors with Netscape Communicator. E.g., all I see is a
grey box where the applet should be, or I get an "Applet exception", or
a benchmark "hangs" and never completes.
WorkAround:
You need Netscape Communicator 4.05 or later. Also note that the JVM's in
Communicator 4.05 for different platforms may be different. Open the Java
Console and read the version of Java. Communicator versions based on 1.1.2
instead of 1.1.5 may have incomplete JVM implementations. Communicator
versions 4.02 and 4.04 with JVM's based on 1.1.2 will run most of the
benchmarks but _227_mtrt, the multi-threaded ray tracer, may not complete.
Comments: Closed: user configuration error

Bugid: 4
Type: usage
Status: closed
Release: 1.0
Synopsis: Execution times are not reported correctly
Keywords: timing
Opened: 19980819
Fixed:  
Description:
I'm trying to run SPECjvm98 on an x86/Win-95 system using JDK1.1.6 The
time reported by the run is bogus - it always reports near zero
execution time (no matter how long it took):

======= _201_compress Finished in 0.010 secs
WorkAround:
You need a patch:

The JIT Update for JDKTM 1.1.6 Software, Early Access 1 (or later) whose URL
as of this writing is:
http://developer.javasoft.com/developer/earlyAccess/jit/
Comments: Closed: user configuration error

Bugid: 5
Type: usage
Status: closed
Release: 1.0
Synopsis: No network activity during benchmark run
Keywords: network web CLASSPATH
Opened: 19980819
Fixed:  
Description:
I'm running the benchmarks from a web server, but my network monitor
shows there is little or no HTTP activity going on. How can this be?
WorkAround:
Were you running in application mode earlier? If so, you may have set your
CLASSPATH to load from local disk when possible instead of from the web
server. Check and reset your CLASSPATH, or log out and log back in.
Comments: Closed: user configuration error

Bugid: 6
Type: usage
Status: closed
Release: 1.0
Synopsis: Corrupted installation
Keywords: install IDE compile
Opened: 19980819
Fixed:  
Description:
Earlier I was running the benchmarks and everything was OK, but now
some of them are consistently running faster or slower than before, and
some of them are getting incorrect results errors. I'm certain that
nothing has changed in my configuration. What could have gone wrong?
WorkAround:
Have you been studying the benchmarks using an Integrated Development
Environment? Such a tool is very useful in navigating the source code and
examining relationships among classes. However if you're not careful it may
recompile the source code which could make a benchmark run faster or slower.
The run rules prohibit modifying the class files.

Also, if there is a bug in the IDE's compiler, it could produce incorrect
code causing the JVM class loader to reject the invalid class file, or
producing incorrect results.

The _999_checkit program can be used to detect recompiled class files or
installations that have been otherwise corrupted. If that is the case you
should remove the benchmark suite and reinstall.
Comments: Closed: user configuration error

Bugid: 7
Type: rfe
Status: open
Release: 1.0
Synopsis: Uses deprecated methods
Keywords: deprecated API1.0.2 API1.1
Opened: 19980819
Fixed:  
Description:
The benchmark uses deprecated methods. E.g., DataInputStream.readLine().
Development of the suite began under the 1.0.2 API and concluded under
the 1.1 API. We expect that later releases will be based entirely on 1.1.
WorkAround:  
Comments:  

Bugid: 8
Type: bug
Status: open
Release: 1.0
Synopsis: Not 100% Pure Java. ASCII character set dependencies
Keywords: mainframe ASCII EBCDIC
Opened: 19980819
Fixed:  
Description:
I'm running on a mainframe computer that uses the EBCDIC character set,
and the benchmarks fail. There are dependencies on the ASCII character
set. Unicode should be used throughout.
WorkAround:  
Comments:
Unfortunately there are areas in the benchmarks where the portability
features available in the Java platform have not been used as they
should be. These limitations may be addressed in a future revision to
the suite that would also remove deprecated API 1.0.2 methods.

Note: Bug 8 was split into two separate bugs, 8 and 26.

Bugid: 9
Type: usage
Status: closed
Release: 1.0
Synopsis: Result pages don't print correctly
Keywords: print Netscape
Opened: 19980819
Fixed:  
Description:
The results page shows up fine in my web browser, Netscape Communicator
4.04, but when I print it the bar charts disappear.
WorkAround:
You need a web browser that can print applets, e.g. Netscape Communicator
4.05, Microsoft Internet Explorer 4.01, or Sun HotJava 1.02. If you're using
an older browser, turn off Java and the SPEC ratio bar chart will appear as
GIF graphs. It's not quite as nice as the Java applet graph, and the memory
utilization chart does not appear, but you can view and print it in almost
any browser.
Comments: Closed: user configuration error

Bugid: 10
Type: usage
Status: closed
Release: 1.0
Synopsis: Problems moving result pages
Keywords: GIF codebase BasePeakGraph
Opened: 19980819
Fixed:  
Description:
When I first generated results pages they looked OK. But then I moved
them to another directory and the bars in the chart are all just white
outlines.

When I first generated results pages they looked OK. But then I moved
them to another directory and the bars charts are just gray rectangles.
WorkAround:
Wherever you move results pages you need to also have copies of the GIF
images and Java class files used by the pages. Normally (if you have not
overridden the default location when you generated the reporting page) the
required files are:

* class/BasePeakGraph.class
* class/Graph.class
* images/spec-sm.gif
* images/basebar.gif
* images/invalid.gif
* images/peakbar.gif
Comments: Closed: user configuration error

Bugid: 11
Type: rfe
Status: open
Release: v20c
Synopsis: Autorun per thread
Keywords: thread
Opened: 19980629
Fixed:  
Description: Provide means to run each benchmark in its own thread.
WorkAround:  
Comments:
Here are the changes I hacked in to my version of
v20c/spec/benchmarks/ProgramRunner.java to get each
autorun run to run in its own thread. This is useful
if you have tools that let you profile individual
threads (which we do). Here I've just hacked it
in unconditionally. Obviously "the right thing"
would be to have a command line flag (reuse -t?)
that turns it on and off. I also don't really deal
with the exceptions from t.start() or t.join().
On the other hand, I can watch HotSpot spending
less time compiling for subsequent runs, and more
time running compiled code.

$ diff v20/spec/harness/ProgramRunner.java v20c-thread/spec/harness/ProgramRunner.java
391,392c391,393
< spec.io.FileOutputStream.clearCount();
< ((SpecBenchmark)prog).harnessMain( args );
---
> spec.io.FileOutputStream.clearCount();
> //((SpecBenchmark)prog).harnessMain( args );
> runInThread(prog, args);
499a501,526
>
> void runInThread(Object prog, String[] args) {
> Thread t = new java.lang.Thread(new ThreadRunner(prog, args));
> try {
> t.start();
> t.join();
> } catch (IllegalThreadStateException itse) {
> Context.out.println("runInThread: IllegalThreadStateException");
> } catch (InterruptedException ie) {
> Context.out.println("runInThread: InterruptedException");
> }
> }
> }
>
> class ThreadRunner implements java.lang.Runnable {
> ThreadRunner(Object p, String[] a) {
> prog = p;
> args = a;
> }
>
> public void run() {
> ((SpecBenchmark)prog).harnessMain( args );
> }
>
> private Object prog = null;
> private String[] args = null;

Bugid: 12
Type: bug
Status: open
Release: 1.0
Synopsis: InstallShield seg faults when you "Cancel" before you install.
Keywords: install NT
Opened: 19980626
Fixed:  
Description:
Observed on NT
I get a pop up that says:

InstallShield Java Edition
One Java Virtual Machine found.
Click OK to install the application.
C:\WINDOWS\jview.exe

If I type in the path to my java.exe (or jre.exe), I get:

This program has performed an illegal operation and will be shut down.
Observed on NT
WorkAround: Use install.class instead of install.exe
Comments:  

Bugid: 13
Type: bug
Status: open
Release: v20c
Synopsis: Cannot install using JDK1.2
Keywords: install JDK1.2 NT
Opened: 19980626
Fixed:  
Description:
Observed on NT.
InstallSheild doesn't work so well with the new JDK1.2beta4 class loader code

$ $Users/jdk1.2beta4I/bin/java v20c
InstallShield Java (TM) Edition
Extracting installation code...............................done
Sorry, could not extract this archive
java.lang.NoClassDefFoundError: v20c
at v20c.initiateGUI(install.java:281)
at v20c.<init>(Compiled Code)
at v20c.main(install.java:438)
**ERROR failed to install
WorkAround: Install with 1.1 and then run with 1.2
Comments:  

Bugid: 14
Type: bug
Status: closed
Release: 1.0
Synopsis: Cannot browse source code from a web server
Keywords: source web
Opened: 19980804
Fixed: 1.03_05, 1.03
Description:
From the HTML users guide, I can browse the benchmark source code from
the CD-ROM or local disk, but not from a web server.
WorkAround:
You need to add world read permission to the benchmark directories:
chmod a+r spec/benchmarks/*
Comments:  

Bugid: 15
Type: bug
Status: closed
Release: 1.0
Synopsis: _213_javac fails due to hardcoded path names
Keywords: File.separatorChar
Opened: 19980604
Fixed: 1.03_01, 1.03
Description:
Observed on Win32/x86 and Solaris/SPARC.
Observed on JDK1.1.6 and JDK1.2beta4.

Using "filename = this.getPath()" in spec.io.File constructors broke _213 on JDK1.1.6 Win32.
And I tracked that problem down to spec.benchmarks._213_javac.Main where the argument to
-classpath is "lib/" which is the root cause of the problems we were seeing. "/" will
not work on Win32 as file separator and using "/" is not 100% pure Java.

_213_javac was failing with the following
> errors on 1.2beta4-H -
>
> Javac benchmark starting...
> JavaLex.java:43: Class java.io.InputStream not found in import.
WorkAround:  
Comments:
A solution would be to use just "lib" or "lib" + java.io.File.separatorChar. Using
"lib", along with the File constructor fix, works fine on Win32 and Solaris, 1.1.6 and
1.2beta4.

> Instead filename = this.getPath() should be used so that no assumption
> is made about how file names are constructed.
>
> Also Mark Reinhold pointed out that it is not 100% pure to construct
> filenames by concatenating Strings. It is always best to use the
> appropriate methods in File class.
>
> The changes are simple - in all the three constructors of spec.io.File filename
> should be set to this.getPath().
>
> I made the changes and tested it on JDK1.2beta4, JDK1.2beta3 and JDK1.1.6

Bugid: 16
Type: bug
Status: closed
Release: 1.0
Synopsis: Superfluous subdirectory
Keywords: directory
Opened: 19980810
Fixed: 1.03_05, 1.03
Description:
In jvm98/spec/benchmarks/_228_jack, the directory spec/benchmarks/_228_jack
should not exist.
WorkAround: Ignore it. Cosmetic.
Comments:  

Bugid: 17
Type: bug
Status: open
Release: 1.0
Synopsis: Running SpecApplication doesn't work in JDK1.2
Keywords: File.separatorChar JDK1.2
Opened: 19980824
Fixed:  
Description:
The bug is in spec/harness/SpecProps.java. The constructor should set
userFile to (new File(baseDir, "props" + File.separatorChar +
"user")).getPath(), instead of userFile = baseDir + "props/user". This
applies to all the properties files.

An even better fix is to set userFile to
new File(baseDir, new File("props", "user"))
thereby avoiding the explicit use of File.separatorChar. Using the File
constructors instead of concatenating strings is generally the best way to
write truly platform-independent filename-manipulation code. I think the "100%
Pure" guide contains similar advice.
WorkAround: Run as applet from web server
Comments:
Running SpecJava as SpecApplication doesn't work in JDK1.2fcs builds as
it is unable to the "props/user" file and hangs after throwing a
FileNotFoundException. This is because it is trying to load
/home/dviswana/specjavaprops/user instead of
/home/dviswana/specjava/props/user. This bug shows up because
File.getAbsolutePath does not return a trailing slash in JDK1.2fcs.

Bugid: 18
Type: bug
Status: closed
Release: 1.0
Synopsis: Wrong path names are constructed on NT
Keywords: SDK jview JDK1.2 File.separatorChar NT
Opened: 19980918
Fixed: 1.03_05, 1.03
Description:
Microsoft jview and SDK looks for .props and .spec instead of props and spec

Run as an application you get the error java.lang.NullPointerException
The error message in Java console: Error loading property file
file: /C:/InetPub/wwwroot/specjvm98/.props/user: com.ms.security
Run as an applet you get the error

jview /a http://server/specjvm98/SpecApplet.html
The benchmark came up with exception: java.lang.NullPointerException
The error message in Java console: Error loading property file
http://server/specjvm98/.props/user: java.io.FileNotFound

Run from Internet Explorer there is no problem.

Reported on NT Server 4.0 + Service pack 3, IE 4.0, IIS 4.0,
and jview.exe version 4.79.2252.
Also MS Java SDK 3.0 and MS Java SDK3.1.

--
JDK-1.2fcs-L doesn't run the SPEC GUI. It seems to be more of the same
with directory separator problems. On my WindowsNT box, with $ULJ
pointed at /usr/local/java, and jvm98-1.03.01 being a SPEC JVM98 Client
installation, patched with the 1.03.01 updater, I do:

$ cd jvm98-1.03.01/
$ $ULJ/jdk1.2/win32/bin/java -version
java version "1.2fcs"
Classic VM (build JDK-1.2fcs-L, native threads)
$ $ULJ/jdk1.2/win32/bin/java SpecApplication
Exception occurred during event dispatching:
java.lang.NullPointerException
...
Error loading property file file:///E:/SPEC/jvm98-1.03.01props/user : java.io.FileNotFoundException: E:\SPEC\jvm98-1.03.01props\user (The system cannot find the path specified)
WorkAround:
Use Internet Explorer to access the Microsoft JVM.

Or rename or copy all directories: .class, .doc, .props, .images, .spec,
.SpecDoc, and .License.
Comments:
Maybe the JVM is interpreting ".\props" as ".props" because "\p" isn't
a standard backslash-escape sequence. But since the problem occurs on
both Microsoft and Sun JVM's it doesn't seem likely to be JVM bugs.

This appears related to the directory separator bug 17, and it is
known there are portability problems in the JVM98 code in how it
treats directory separators.

Bugid: 19
Type: bug
Status: closed
Release: 1.0
Synopsis: Wrong release number in documentation
Keywords: version
Opened: 19980929
Fixed: 1.03_05, 1.03
Description: In doc/support.html ",JVM Client98 release 1.01," should be removed.
WorkAround: Ignore it. Cosmetic.
Comments:  

Bugid: 20
Type: bug
Status: open
Release: 1.0
Synopsis: Microsoft JVM "hangs"
Keywords: hang MSIE SDK JAR
Opened: 19981002
Fixed:  
Description:
Microsoft Internet Explorer almost stops before the test finishes and I
can't get any satisfactory results.

With Microsoft's SDK-Java.31, the harness hangs during some benchmark
run when a web server on the local machine is used. It doesn't happen
with web servers on other machines. We tried both Microsoft's web
server and the Java web server.
WorkAround:  
Comments:
From: pbk Tue Oct 13 18:11:25 1998
%Bin\SDK-Java.31\bin\jview /a http://localhost/jvm98/SpecApplet.html

? I find that it hangs at the beginning (!) of many of the tests.
If I open the Java Console (not the SPEC console), sometimes typing
at that will kick it into proceeding. If I open the SPEC console,
I usually see it something like

======= _202_jess Starting =======
Run 0 start. Total memory=3735552 free memory=2475800

So it's into the next test. Is that the point at which it tries
to load the classes for that test? Sometimes it just takes a while
to get started, sometimes I have to tickle it (e.g. with a "t" on
the Java Console), sometimes nothing works. Sometimes it hangs
between the runs of the autorun, which shouldn't be loading classes
(but it might be trying to open files). If I leave the SPEC
console open it seems to do better for a while. Could it be stream
flushing to a closed/unopened console stream? Of course, performance
numbers with the SPEC console open will not be optimal.

I don't think it's the "localhost" in the URL. On my PCs I'm
running JavaWebServer (1.1.1). I've also pointed it at my Sun
where I'm running a venerable version of Apache. It's more
reliable that way, but not perfect.

It has been reported that there is a Windows registry setting for
TCP/IP parameters on the client that avoids this problem, but we
do not yet know what that setting is and cannot confirm the fix.

Bugid: 21
Type: rfe
Status: open
Release: 1.0
Synopsis: Why did I get "invalid category" for memory?
Keywords: memory
Opened: 19981002
Fixed:  
Description:
This is because you entered "96 MB" in the client memory field, which
it failed to convert to an integer in order to classify the memory
size in the "48 to 256 MB" range. Please change this to "96" as the
MB is implicitly understood.

The software should be more robust and accept either form, and in
case of errors should give better messages.
WorkAround:  
Comments: User configuration error, but left open as a request for enhancement.

Bugid: 22
Type: bug
Status: open
Release: 1.0
Synopsis: Security failure with JDK1.2beta4 on Windows 98
Keywords: Windows-98 JDK1.2
Opened: 19981002
Fixed:  
Description:
I tested JDK1.2 beta4 with Windows 98 about ten times on my two
machines, but got satisfactory results only once on each machine. In
many case, "***NOT VALID***" messages appeared with _213_ javac test.
Besides I couldn't send reports with the following error:

>Message Send error: java.security.AccessControlException: access denied (java.net.SocketPermission mail.yk.rim.or.jp resolve)
WorkAround:  
Comments:
Not enough information to diagnose or reproduce the problem.

This looks like the security manager refusing to allow the program to
open a socket to talk to the SMTP server, possibly because it could not
resolve the host name. You might be able to change your security
preferences to allow it. I'm not sure how 1.2 works. Previously I
have run appletviewer so I can change its security settings through
the menu.

Another thing you might try is to enter the numeric IP address of the
SMTP server, or of it and the web server, instead of using the
alphabetic name. (E.g., 123.45.67.89).

Bugid: 23
Type: bug
Status: open
Release: 1.0
Synopsis: InstallShield occasionally "hangs" on NT
Keywords: install NT
Opened: 19980918
Fixed:  
Description:
I get a pop up that says:

InstallShield Java Edition
One Java Virtual Machine found.
Click OK to install the application.
C:\WINDOWS\jview.exe

If I click OK, the pop up goes away and nothing happens.
Observed on NT
WorkAround:
Since it sometimes works and sometimes fails on different machines,
it's possible that re-installing your JVM or re-installing NT would
fix the problem.
Comments:
Attempts to reproduce the problem were inconclusive.
This appears to be a generic problem with the operation of InstallShield
on NT rather than a bug in the benchmark suite:

Tried to run on a NT machine. Javainstallshield package hangs with JDK
1.1.6. The installation failed.

We tried a second machine in the Lab and this time it ran with two
different versions of JVM (1.1.6 and MS).

Bugid: 24
Type: rfe
Status: open
Release: 1.0
Synopsis: Will SPECjvm98 support PersonalJava?
Keywords: PersonalJava embedded memory
Opened: 19980827
Fixed:  
Description:
| 1. Will SPECjvm98 support PersonalJava (and maybe even embedded Java) ?

No. The benchmark driver uses multiple frames and as I understand it,
PersonalJava supports only a single frame. You may be able to work around
this for test purposes by running the benchmarks from the "command line"
as an application instead of from a web server as an applet.

| 2. Is it suitable for small memory devices (<8MB) ?

Probably not. Two of the benchmarks, _202_jess and _228_jack, run with
as little as 2MB of JVM heap. So they might run in 8MB total memory.
You could test this by downloading the original programs from their
web sites:

http://herzberg.ca.sandia.gov/jess/
http://www.suntest.com/JavaCC/

The other benchmarks require more than 8MB of heap. Beyond what you
can freely download from the above sites, SPECjvm98 would only give you
workloads and some components you may be able to adapt to your use.
But then it's not particularly expensive either.
WorkAround:  
Comments:  

Bugid: 25
Type: rfe
Status: open
Release: 1.0
Synopsis: Cannot use MacOS web server. File name length limitation.
Keywords: filename MacOS
Opened: 19980927
Fixed:  
Description:
On the MacOS file names are limited to 31 characters. The test "_213_javac"
requires loading classes which are longer than 31 characters (e.g.
ArrayIndexOutOfBoundsException.class, CloneNotSupportedException.class,
IllegalMonitorStateException.class, IllegalThreadStateException.class,
IncompatibleClassChangeError.class, NegativeArraySizeException.class,
StringIndexOutOfBoundsException.class). We usually solve this problem by
one of three methods: (1) packaging the classes in a JAR or ZIP file, (2)
creating a "mapping" file which maps shorter (unique) names onto longer
names, (3) accepting (hopefully unique) truncated file names. Unfortunately
(1) violates the "Test Environment" assertions, (2) requires building the
mapping into the VM, and (3) requires a standard "truncation" scheme (a la
Win3.1).
WorkAround:
The easiest way around this problem for testing MacOS
clients would be to use a web server on some non-MacOS system that
handles long file names. However see also bug #26.
Comments:  

Bugid: 26
Type: bug
Status: open
Release: 1.0
Synopsis: Not 100% Pure Java. Line termination character dependencies.
Keywords: MacOS File.separatorChar LINE_SEPARATOR
Opened: 19980819
Fixed:  
Description:
I'm running on MacOS Runtime for Java 2.0. The benchmarks seem to be
running, but the display reports an error, invalid results. I opened
the benchmark console and it looks like all the output for each
benchmark is printed on a single line.

In some places a particular line termination character (e.g. '\n' for
UNIX) is hard coded. Some systems handle multiple line termination
sequences, but this will cause spurious errors on other systems.
Character.LINE_SEPARATOR should be used instead.
WorkAround:  
Comments:
Unfortunately there are areas in the benchmarks where the portability
features available in the Java platform have not been used as they
should be. These limitations may be addressed in a future revision to
the suite that would also remove deprecated API 1.0.2 methods.

Note: Bug 8 was split into two separate bugs, 8 and 26.

Bugid: 27
Type: rfe
Status: closed
Release: 1.0
Synopsis: Text files have UNIX line termination
Keywords: LINE_SEPARATOR install Windows MacOS
Opened: 19980818
Fixed: 1.03_05, 1.03
Description:
Documentation text files (*.txt) have UNIX line terminators (\n) instead
of DOS line terminators (\r\n). The default program association for
Windows is textedit which does not understand and translate different
line endings appropriately.
WorkAround:
For MacOS use BBedit, or a similar editor which automatically translates
line endings from different systems.
Comments:
InstallShield can be set to perform line ending translation at time of
installation based on file extension. We had not used this feature in
the 1.0 release due to a risk of translating a file used by the
benchmark which could cause the test to fail validation spuriously.

However, the only txt files in the benchmarks are in _228_jack and
are not used for validation, so InstallShield translation is safe
and should be performed.

Bugid: 28
Type: bug
Status: closed
Release: 1.0
Synopsis: UNIX scripts are installed without execute permission
Keywords: permission UNIX script run
Opened: 19981105
Fixed: 1.03_05, 1.03
Description:
The scripts run.sh and run_commandline.ksh have execute permission
on the CD-ROM, but do not when the suite is installed to disk.
WorkAround: chmod +x run.sh run_commandline.ksh
Comments:  

Bugid: 29
Type: bug
Status: open
Release: 1.03_05
Synopsis: Console button incorrect if window closed manually
Keywords: console
Opened: 19981208
Fixed:  
Description:
Bug in gui logic -> manually closing console window throws off main
app's view button ( console off/on ) state logic.
I.e., if you close the console window using the window's own
(platform dependent) close mechanism, the main window does not
"notice" that it was closed. The label on the button incorrectly
remains "Console Off" instead of "Console On".
WorkAround:
1. Always close the console window using the button on the main window, or
2. If you close the console window manually and the button label becomes
incorrect, press that button once and it will become correct.
Comments:  

Bugid: 30
Type: bug
Status: open
Release: 1.0
Synopsis: Memory allocations listed are misleading
Keywords: memory allocation
Opened: 19991215
Fixed:  
Description:
D:\jvm98>java -mx32m spec.benchmarks._201_compress.Main

The benchmark program description said that this program allocates 334
MB of memory. But my jdk indicates that only about 119MB memory is
allocated. I tried the other programs and found out that only 1/3 to 1/2
the total memory allocation indicated in the benchmark program
description "Gc" field is reported by my JDK. I don't know which one is
correct. Or maybe the total memory allocated indicated in the "Gc" field
of program description is the total memory allocated in 2 to 3
iterations of the same program.
WorkAround:  
Comments:
Yes, the memory requirements listed are for a typical run of 3 iterations.
For a thorough treatment of SPECjvm98 memory allocation, see

A Study of the Allocation Behavior of the SPECjvm98 Java Benchmarks
by Dieckmann and Hölzle, University of California Santa Barbara (TRCS98-33)
http://www.cs.ucsb.edu/TRs/

Bugid: 31
Type: bug
Status: open
Release: 1.0
Synopsis: Incorrect comment in spec.harness.ProgrammerRunner
Keywords: autorun size
Opened: 19990104
Fixed:  
Description:
The doc comment says that the percentTimes100 field is:
/**
* The percentage run of the benchmark. The benchmark can be run at 1%,
* 10% and 100% of its full length. percentTimes100 is used to represent
* this value.
*/
However, the comment in props/user is correct:
# Select tolerance for auto-run specified in percent times 100
# I.e., 200 means 2.0%. If successive runs do not improve by at least
# this amount, then terminate an autorun sequence
WorkAround:  
Comments:  

Bugid: 32
Type: bug
Status: open
Release: v20
Synopsis: IllegalThreadStateException: cannot find file
Keywords: File.separatorChar JDK1.2 NT
Opened: 19990119
Fixed:  
Description:
I get the following error while trying to run _213_javac of SpecJava v20
with the JDK1.2 (with and without JIT) on NT -

spec.io.FileInputStream: Caught exception in constructor:
java.io.FileNotFoundEx
ception:
/H:/benchmarks/SpecJava/spec/benchmarks/_213_javac/input/lib/java/lang/
IllegalThreadStateException.class (The system cannot find the file
specified)
trying to open: lib\java\lang\IllegalThreadStateException.class
spec.io.FileInputStream: Attempting recovery with
System.runFinalization()
----------------------------------
spec.io.FileInputStream: Caught exception in constructor at second
level: java.i
o.FileNotFoundException:
/H:/benchmarks/SpecJava/spec/benchmarks/_213_javac/inpu
t/lib/java/lang/IllegalThreadStateException.class (The system cannot
find the fi
le specified)
----------------------------------
......

Looks like it is a slash problem again - notice the /H: Also, there is
no IllegalThreadStateException.class in spec/benchamrks/_213_javac/input.
WorkAround:  
Comments:  

Bugid: 33
Type: bug
Status: open
Release: 1.0
Synopsis: Array bounds test does not work as intended
Keywords: validation
Opened: 19990210
Fixed:  
Description:
Class File :spec.benchmarks._200_check.PepTest
Method :testArray

Enclosed is the relevant source code from "testArray" in "PepTest.java".

###################################################################
String testArray() {
int x[];
x = new int[6];
// ******** Source code omitted ********
// David Detlefs suggested the following test. 9/97
boolean hitit = true;
try {
for (int i = 0; i < 5; i++) {
x[i + 3] = i;
}
} catch (java.lang.ArrayIndexOutOfBoundsException e) {
hitit = true;
}
if (!hitit) return "missing exception";
// ******** Source code omitted ********
}
###################################################################

No matter what happens, "hitit" will be true and hence this test will pass.
The problem is in the initialization of "hitit". It should be initialized
to false --> "boolean hitit = false;"
WorkAround:  
Comments:  

Bugid: 34
Type: bug
Status: open
Release: 1.03_05
Synopsis: Class files missing from 1.03 update installation
Keywords: reporting
Opened: 19990219
Fixed:  
Description:
When I used the reporting page generating program
as "java spec.reporter.Reporter -a -r myresult -o myresult.txt", the
following exception occurred:

>Exception in thread "main" java.lang.NoClassDefFoundError:
> spec/reporter/AsciiReport
> at spec.reporter.Reporter.main(Reporter.java:56)

I found that spec/reporter/AsciiReport.class didn't exist and managed
to fix the problem by generating the class file with javac. Then other
class files of TextBlock.class and TextColumn.class were generated.
Perhaps the installation program didn't installed the class files.
WorkAround: Compile AsciiReport.java
Comments: