fail to setup appserver with v1.04

[johnnyli]

Hi sir,
I follow  with  v1.04 guide, setting up appserver get failed with the first error "Waiting for spec2004-1 to start .Error starting domain spec2004-1"
Any suggestion is welcome, thanks.

[root@appserver appvm_scripts]# ./setup_files.sh
/usr/bin/java
openjdk version "1.8.0_171"
OpenJDK Runtime Environment (build 1.8.0_171-b10)
OpenJDK 64-Bit Server VM (build 25.171-b10, mixed mode)
SPECvirt_sc2013 v1.1, build: 80
mkdir: cannot create directory `/opt/SPECjAppServer2004/jdbc': File exists
CLI306 Warning - The server located at /opt/glassfish3/glassfish/domains/spec2004-1 is not running.
Command stop-domain executed successfully.
+ . ./env.sh
++ . ../config/glassfish.env
+++ JAS_HOME=/opt/SPECjAppServer2004
+++ JAVA_HOME=/opt/jdk
+++ FS=:
+++ BINDWAIT=45
+++ J2EE_HOME=/opt/glassfish3/glassfish
+++ J2EE_JARS=/opt/glassfish3/glassfish/lib
+++ JAS_ADMIN_PORT=4848
+++ J2EE_INSTANCE=server
+++ DB_HOST=specdb
+++ JDBC_CLASSPATH=/opt/SPECjAppServer2004/jdbc/postgresql.jar
+++ CLASSPATH=/opt/SPECjAppServer2004/jars/corp.jar:/opt/SPECjAppServer2004/jars/mfg.jar:/opt/SPECjAppServer2004/jars/orders.jar:/opt/SPECjAppServer2004/jars/supplier.jar:/opt/SPECjAppServer2004/jars/util.jar:/opt/glassfish3/glassfish/lib/appserv-rt.jar:/opt/glassfish3/glassfish/lib/javaee.jar
+++ JAVAX_JAR=/opt/glassfish3/glassfish/lib/appserv-rt.jar:/opt/glassfish3/glassfish/lib/javaee.jar
+++ JAS_HOST=specdelivery
+++ JAS_PORT=8000
+++ JAS_PREFIX=/
+++ EMULATOR_HOST=specemulator
+++ EMULATOR_PORT=8080
+++ EMULATOR_PREFIX=/
+++ IIOP_PORT=3700
+++ NAMING_PROVIDER=iiop://specdelivery:3700
+++ IIOP_ENDPOINTS=specdelivery:3700
+++ JAVA='/opt/jdk/bin/java -server -XX:+DisableExplicitGC -Xmx256m -Xss256k -Djava.naming.provider.url=iiop://specdelivery:3700 -Djava.naming.provider.url.pkgs=com.sun.enterprise.naming -Dcom.sun.appserv.iiop.orbconnections=5 -Dcom.sun.appserv.iiop.endpoints=specdelivery:3700 -Dorg.omg.CORBA.ORBInitialHost=specdelivery -Dorg.omg.CORBA.ORBInitialPort=3700 -Dnetworkaddress.cache.ttl=0 -Dsun.net.inetaddr.ttl=0 '
+++ export JAS_HOME JAVA_HOME JAVAX_JAR JAS_HOST JAS_PORT EMULATOR_HOST EMULATOR_PORT LD_LIBRARY_PATH CLASSPATH J2EE_HOME
++ ADMIN_USER=admin
++ PASSFILE=./passwordfile
++ ADMIN_PORT=4848
++ DOMAIN=spec2004-1
++ CONNECTION_POOL=SpecJPool
++ DBHOST=specdb
++ DBNAME=specdb
++ DBPORT=5432
++ DBUSER=spec
++ DBPASS=spec
++ JARSDIR=/opt/SPECjAppServer2004/jars
++ EMU_HOME=/opt/tomcat
++ PRIOCNTL=
+ [[ -d /opt/glassfish3/glassfish/domains/spec2004-1 ]]
+ echo 'Deleting existing domain'
Deleting existing domain
+ /opt/glassfish3/glassfish/bin/asadmin --user admin --passwordfile ./passwordfile delete-domain spec2004-1
Domain spec2004-1 deleted.
Command delete-domain executed successfully.
+ /opt/glassfish3/glassfish/bin/asadmin --user admin --passwordfile ./passwordfile create-domain --adminport 4848 --instanceport 8000 spec2004-1
Using port 4848 for Admin.
Using port 8000 for HTTP Instance.
Using default port 7676 for JMS.
Using default port 3700 for IIOP.
Using default port 8181 for HTTP_SSL.
Using default port 3820 for IIOP_SSL.
Using default port 3920 for IIOP_MUTUALAUTH.
Using default port 8686 for JMX_ADMIN.
Using default port 6666 for OSGI_SHELL.
Using default port 9009 for JAVA_DEBUGGER.
Distinguished Name of the self-signed X.509 Server Certificate is:
[CN=appserver,OU=GlassFish,O=Oracle Corporation,L=Santa Clara,ST=California,C=US]
Distinguished Name of the self-signed X.509 Server Certificate is:
[CN=appserver-instance,OU=GlassFish,O=Oracle Corporation,L=Santa Clara,ST=California,C=US]
No domain initializers found, bypassing customization step
Domain spec2004-1 created.
Domain spec2004-1 admin port is 4848.
Domain spec2004-1 admin user is "admin".
Command create-domain executed successfully.
+ echo

+ echo 'The original master password is changeit, please change to javaperf here:'
The original master password is changeit, please change to javaperf here:
+ echo

+ /opt/glassfish3/glassfish/bin/asadmin --user admin change-master-password spec2004-1
Enter the current master password>
Enter the new master password>
Enter the new master password again>
Command change-master-password executed successfully.
+ /opt/glassfish3/glassfish/bin/asadmin --user admin --passwordfile ./passwordfile start-domain spec2004-1
Enter master password (3) attempt(s) remain)>
Waiting for spec2004-1 to start .Error starting domain spec2004-1.
The server exited prematurely with exit code 1.
Before it died, it produced the following output:

Launching GlassFish on Felix platform
ERROR: Error parsing system bundle export statement: org.osgi.framework; version=1.6.0, org.osgi.framework.launch; version=1.0.0, org.osgi.framework.wiring; version=1.0.0, org.osgi.framework.startlevel; version=1.0.0, org.osgi.framework.hooks.bundle; version=1.0.0, org.osgi.framework.hooks.resolver; version=1.0.0, org.osgi.framework.hooks.service; version=1.1.0, org.osgi.framework.hooks.weaving; version=1.0.0, org.osgi.service.packageadmin; version=1.2.0, org.osgi.service.startlevel; version=1.1.0, org.osgi.service.url; version=1.0.0, org.osgi.util.tracker; version=1.5.0, , org.glassfish.embeddable;org.glassfish.embeddable.spi;version=3.1.1 (org.osgi.framework.BundleException: Exported package names cannot be zero length.)
ERROR: Bundle jaxb-api [2] Error starting file:/opt/glassfish3/glassfish/modules/endorsed/jaxb-api-osgi.jar (org.osgi.framework.BundleException: Unresolved constraint in bundle jaxb-api [2]: Unable to resolve 2.0: missing requirement [2.0] osgi.wiring.package; (osgi.wiring.package=javax.activation))
ERROR: Bundle org.glassfish.metro.webservices-api-osgi [3] Error starting file:/opt/glassfish3/glassfish/modules/endorsed/webservices-api-osgi.jar (org.osgi.framework.BundleException: Unresolved constraint in bundle org.glassfish.metro.webservices-api-osgi [3]: Unable to resolve 3.0: missing requirement [3.0] osgi.wiring.package; (&(osgi.wiring.package=javax.xml.bind)(version>=2.2.0)) [caused by: Unable to resolve 2.0: missing requirement [2.0] osgi.wiring.package; (osgi.wiring.package=javax.activation)])
ERROR: Bundle org.glassfish.hk2.osgi-adapter [46] Error starting file:/opt/glassfish3/glassfish/modules/osgi-adapter.jar (org.osgi.framework.BundleException: Unresolved constraint in bundle org.glassfish.hk2.osgi-adapter [46]: Unable to resolve 46.0: missing requirement [46.0] osgi.wiring.package; (&(osgi.wiring.package=com.sun.enterprise.module)(version>=1.1.0)) [caused by: Unable to resolve 58.0: missing requirement [58.0] osgi.wiring.package; (&(osgi.wiring.package=org.jvnet.hk2.config)(version>=1.1.0)) [caused by: Unable to resolve 214.0: missing requirement [214.0] osgi.wiring.package; (osgi.wiring.package=javax.management)]])
ERROR: Bundle org.glassfish.main.core.glassfish [50] Error starting file:/opt/glassfish3/glassfish/modules/glassfish.jar (org.osgi.framework.BundleException: Activator start error in bundle org.glassfish.main.core.glassfish [50].)
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=192m; support was removed in 8.0
OpenJDK 64-Bit Server VM warning: ignoring option PermSize=64m; support was removed in 8.0
org.osgi.framework.BundleException: Exported package names cannot be zero length.
   at org.apache.felix.framework.util.manifestparser.ManifestParser.normalizeExportClauses(ManifestParser.java:729)
   at org.apache.felix.framework.util.manifestparser.ManifestParser.<init>(ManifestParser.java:191)
   at org.apache.felix.framework.ExtensionManager.<init>(ExtensionManager.java:220)
   at org.apache.felix.framework.Felix.<init>(Felix.java:374)
   at org.apache.felix.framework.FrameworkFactory.newFramework(FrameworkFactory.java:28)
   at com.sun.enterprise.glassfish.bootstrap.osgi.OSGiFrameworkLauncher.launchOSGiFrameWork(OSGiFrameworkLauncher.java:77)
   at com.sun.enterprise.glassfish.bootstrap.osgi.OSGiGlassFishRuntimeBuilder.build(OSGiGlassFishRuntimeBuilder.java:128)
   at org.glassfish.embeddable.GlassFishRuntime._bootstrap(GlassFishRuntime.java:157)
   at org.glassfish.embeddable.GlassFishRuntime.bootstrap(GlassFishRuntime.java:110)
   at com.sun.enterprise.glassfish.bootstrap.GlassFishMain$Launcher.launch(GlassFishMain.java:112)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:498)
   at com.sun.enterprise.glassfish.bootstrap.GlassFishMain.main(GlassFishMain.java:97)
   at com.sun.enterprise.glassfish.bootstrap.ASMain.main(ASMain.java:55)
Jun 01, 2018 1:43:59 AM BundleProvisioner install
WARNING: Failed to install file:/opt/glassfish3/glassfish/modules/flashlight-extra-jdk-packages.jar
org.osgi.framework.BundleException: Could not create bundle object.
   at org.apache.felix.framework.Felix.installBundle(Felix.java:2785)
   at org.apache.felix.framework.BundleContextImpl.installBundle(BundleContextImpl.java:165)
   at com.sun.enterprise.glassfish.bootstrap.osgi.BundleProvisioner.install(BundleProvisioner.java:446)
   at com.sun.enterprise.glassfish.bootstrap.osgi.BundleProvisioner.installBundles(BundleProvisioner.java:208)
   at com.sun.enterprise.glassfish.bootstrap.osgi.OSGiGlassFishRuntimeBuilder.build(OSGiGlassFishRuntimeBuilder.java:139)
   at org.glassfish.embeddable.GlassFishRuntime._bootstrap(GlassFishRuntime.java:157)
   at org.glassfish.embeddable.GlassFishRuntime.bootstrap(GlassFishRuntime.java:110)
   at com.sun.enterprise.glassfish.bootstrap.GlassFishMain$Launcher.launch(GlassFishMain.java:112)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:498)
   at com.sun.enterprise.glassfish.bootstrap.GlassFishMain.main(GlassFishMain.java:97)
   at com.sun.enterprise.glassfish.bootstrap.ASMain.main(ASMain.java:55)
Caused by: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
   at java.util.ArrayList.rangeCheck(ArrayList.java:657)
   at java.util.ArrayList.get(ArrayList.java:433)
   at org.apache.felix.framework.ExtensionManager.addExtensionBundle(ExtensionManager.java:401)
   at org.apache.felix.framework.Felix.installBundle(Felix.java:2751)
   ... 13 more

[johnnyli]

Follow 6.1 Application server workload, try 3 web link in clientserver, the first two are unable to connect, the third one seems ok.
server.log and domain.xml in appserver are attached.
What else I can check?

http://specdelivery:8000/SPECjAppServer/app?action=atomicityTests
Unable to connect
Firefox can't establish a connection to the server at specdelivery:8000.

http://specdelivery:8000/Supplier/DeliveryServlet
Unable to connect
Firefox can't establish a connection to the server at specdelivery:8000.

http://specemulator:8080/Emulator/EmulatorServlet
Emulator Servlet seems to work OK
JAS_HOST : specdelivery
JAS_PORT : 8000
Servlet URL : Supplier/DeliveryServlet

Number of Transactions : 0
Servlet invoked without command specified

[root@client1 Desktop]# ping specdelivery
PING appserver (192.168.1.147) 56(84) bytes of data.
64 bytes from appserver (192.168.1.147): icmp_seq=1 ttl=64 time=0.258 ms
64 bytes from appserver (192.168.1.147): icmp_seq=2 ttl=64 time=0.354 ms
64 bytes from appserver (192.168.1.147): icmp_seq=3 ttl=64 time=0.361 ms
64 bytes from appserver (192.168.1.147): icmp_seq=4 ttl=64 time=0.315 ms
64 bytes from appserver (192.168.1.147): icmp_seq=5 ttl=64 time=0.368 ms

[RamyaMeruva]

Hi Johnnyli,

Can u check if your SElinux & firewall are disabled?

Also, can u check if u have the following entry in /etc/hosts on appserver VM:

 <appserver1 IP> appserver1 appserver specdelivery specemulator

[johnnyli]

>>>>Can u check if your SElinux & firewall are disabled?
1)firmware is disabled in /opt/other_scripts/services_off.sh, which is called by all VMs
Also, can u check if u have the following entry in /etc/hosts on appserver VM:
# Firewall
service iptables stop; chkconfig iptables off
service ip6tables stop; chkconfig ip6tables off

2)selinux is disalbed in grub
[root@appserver1-int Desktop]# cat /proc/cmdline
ro root=/dev/mapper/VolGroup-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_NO_MD rd_LVM_LV=VolGroup/lv_swap SYSFONT=latarcyrheb-sun16 crashkernel=129M@0M rd_LVM_LV=VolGroup/lv_root  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet selinux=0 audit=0

>>>>Also, can u check if u have the following entry in /etc/hosts on appserver VM:
all VMs have below hosts file, and i confirm that each can ping successfully. The ip address is assigned through dhcp, it is not assigned statically.
[root@appserver1-int Desktop]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

##
# Defaults used for the SPECvirt_sc2013 Example VM Setup Guide.

# External VM-to-client communications
192.168.1.144     infraserver infraserver1
192.168.1.128     webserver webserver1
192.168.1.193     mailserver mailserver1
192.168.1.147     appserver appserver1 specdelivery specemulator
192.168.1.194     dbserver dbserver1
192.168.1.123     batchserver batchserver1

192.168.1.105   client1 specdriver

# Internal VM-to-VM only communications
192.168.1.176       infraserver1-int
192.168.1.129       webserver1-int
192.168.1.139       appserver1-int
192.168.1.162       dbserver1-int specdb

ping spec specemulator and specdelivery from client VM sucessully
[root@client1 Desktop]# ping specemulator
PING appserver (192.168.1.147) 56(84) bytes of data.
64 bytes from appserver (192.168.1.147): icmp_seq=1 ttl=64 time=1.31 ms
64 bytes from appserver (192.168.1.147): icmp_seq=2 ttl=64 time=0.396 ms
64 bytes from appserver (192.168.1.147): icmp_seq=3 ttl=64 time=0.468 ms
64 bytes from appserver (192.168.1.147): icmp_seq=4 ttl=64 time=0.433 ms
--- appserver ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3596ms
rtt min/avg/max/mdev = 0.396/0.652/1.311/0.381 ms
[root@client1 Desktop]# ping specdelivery
PING appserver (192.168.1.147) 56(84) bytes of data.
64 bytes from appserver (192.168.1.147): icmp_seq=1 ttl=64 time=0.388 ms
64 bytes from appserver (192.168.1.147): icmp_seq=2 ttl=64 time=0.497 ms
64 bytes from appserver (192.168.1.147): icmp_seq=3 ttl=64 time=0.361 ms

[RamyaMeruva]

Can u verify if glassfish is actually installed and running by going to http://specdelivery:4848 ?
if yes, then the problem is the SPECjApp .EAR has not deployed correctly. Try re-installing GF and run setup_files.sh

Also, what version of Glassfish are u trying to use?
Is it GF312? if its a different version, it might not work with existing scripts.

Also, did u manage to resolve the error u posted on June 1st?
if not, can u use Java 7 instead of Java 8 because GF3.12 seems have to compatibility issues with Java 8.

[johnnyli]

>>>>Can u verify if glassfish is actually installed and running by going to http://specdelivery:4848 ?
>>>>if yes, then the problem is the SPECjApp .EAR has not deployed correctly. Try re-installing GF and run setup_files.sh
Unable to connect, Firefox can't establish a connection to the server at specdelivery:4848.

>>>>Also, what version of Glassfish are u trying to use?
>>>>Is it GF312? if its a different version, it might not work with existing scripts.
GF is downloaded from below link, http://download.java.net/glassfish/3.1.2.2/release/glassfish-3.1.2.2.zip, follow the example guide.

>>>>Also, did u manage to resolve the error u posted on June 1st?
No, still pending on that

>>>>if not, can u use Java 7 instead of Java 8 because GF3.12 seems have to compatibility issues with Java 8.
only install java jdk according to &2.1 of example guide: yum install -y java-1.8.0-openjdk java-1.8.0-openjdk-devel
shall i need to roll back to java-1.7.0-openjdk?

[root@appserver1-int Desktop]# java -showversion
openjdk version "1.8.0_171"
OpenJDK Runtime Environment (build 1.8.0_171-b10)
OpenJDK 64-Bit Server VM (build 25.171-b10, mixed mode)

[RamyaMeruva]

If you're not able to connect to http://specdelivery:4848, then GF did not start successfully.

There is a way to get GF3.12 working with Java 8. Add the following line at the end of the file /opt/glassfish3/glassfish/config/osgi.properties:

jre-1.8=${jre-1.7}

If the above workaround does not work, can u rollback to Java 7 & re-install GF3.12 ?

[johnnyli]

setup still failed after roll back java to 1.7.0. 
how can I reinstall GF? in the guide, there is only unzip command "unzip glassfish-3.1.2.2.zip", no installation command

-----------------------------------------------------------------------------------------------------------------------
[root@localhost appvm_scripts]# ./setup_files.sh
/usr/bin/java
java version "1.7.0_181"
OpenJDK Runtime Environment (rhel-2.6.14.8.el6_9-x86_64 u181-b00)
OpenJDK 64-Bit Server VM (build 24.181-b00, mixed mode)
SPECvirt_sc2013 v1.1, build: 80
mkdir: cannot create directory `/opt/SPECjAppServer2004/jdbc': File exists
CLI306 Warning - The server located at /opt/glassfish3/glassfish/domains/spec2004-1 is not running.
Command stop-domain executed successfully.
+ . ./env.sh
++ . ../config/glassfish.env
+++ JAS_HOME=/opt/SPECjAppServer2004
+++ JAVA_HOME=/opt/jdk
+++ FS=:
+++ BINDWAIT=45
+++ J2EE_HOME=/opt/glassfish3/glassfish
+++ J2EE_JARS=/opt/glassfish3/glassfish/lib
+++ JAS_ADMIN_PORT=4848
+++ J2EE_INSTANCE=server
+++ DB_HOST=specdb
+++ JDBC_CLASSPATH=/opt/SPECjAppServer2004/jdbc/postgresql.jar
+++ CLASSPATH=/opt/SPECjAppServer2004/jars/corp.jar:/opt/SPECjAppServer2004/jars/mfg.jar:/opt/SPECjAppServer2004/jars/orders.jar:/opt/SPECjAppServer2004/jars/supplier.jar:/opt/SPECjAppServer2004/jars/util.jar:/opt/glassfish3/glassfish/lib/appserv-rt.jar:/opt/glassfish3/glassfish/lib/javaee.jar
+++ JAVAX_JAR=/opt/glassfish3/glassfish/lib/appserv-rt.jar:/opt/glassfish3/glassfish/lib/javaee.jar
+++ JAS_HOST=specdelivery
+++ JAS_PORT=8000
+++ JAS_PREFIX=/
+++ EMULATOR_HOST=specemulator
+++ EMULATOR_PORT=8080
+++ EMULATOR_PREFIX=/
+++ IIOP_PORT=3700
+++ NAMING_PROVIDER=iiop://specdelivery:3700
+++ IIOP_ENDPOINTS=specdelivery:3700
+++ JAVA='/opt/jdk/bin/java -server -XX:+DisableExplicitGC -Xmx256m -Xss256k -Djava.naming.provider.url=iiop://specdelivery:3700 -Djava.naming.provider.url.pkgs=com.sun.enterprise.naming -Dcom.sun.appserv.iiop.orbconnections=5 -Dcom.sun.appserv.iiop.endpoints=specdelivery:3700 -Dorg.omg.CORBA.ORBInitialHost=specdelivery -Dorg.omg.CORBA.ORBInitialPort=3700 -Dnetworkaddress.cache.ttl=0 -Dsun.net.inetaddr.ttl=0 '
+++ export JAS_HOME JAVA_HOME JAVAX_JAR JAS_HOST JAS_PORT EMULATOR_HOST EMULATOR_PORT LD_LIBRARY_PATH CLASSPATH J2EE_HOME
++ ADMIN_USER=admin
++ PASSFILE=./passwordfile
++ ADMIN_PORT=4848
++ DOMAIN=spec2004-1
++ CONNECTION_POOL=SpecJPool
++ DBHOST=specdb
++ DBNAME=specdb
++ DBPORT=5432
++ DBUSER=spec
++ DBPASS=spec
++ JARSDIR=/opt/SPECjAppServer2004/jars
++ EMU_HOME=/opt/tomcat
++ PRIOCNTL=
+ [[ -d /opt/glassfish3/glassfish/domains/spec2004-1 ]]
+ echo 'Deleting existing domain'
Deleting existing domain
+ /opt/glassfish3/glassfish/bin/asadmin --user admin --passwordfile ./passwordfile delete-domain spec2004-1
Domain spec2004-1 deleted.
Command delete-domain executed successfully.
+ /opt/glassfish3/glassfish/bin/asadmin --user admin --passwordfile ./passwordfile create-domain --adminport 4848 --instanceport 8000 spec2004-1
Using port 4848 for Admin.
Using port 8000 for HTTP Instance.
Using default port 7676 for JMS.
Using default port 3700 for IIOP.
Using default port 8181 for HTTP_SSL.
Using default port 3820 for IIOP_SSL.
Using default port 3920 for IIOP_MUTUALAUTH.
Using default port 8686 for JMX_ADMIN.
Using default port 6666 for OSGI_SHELL.
Using default port 9009 for JAVA_DEBUGGER.
Distinguished Name of the self-signed X.509 Server Certificate is:
[CN=localhost.localdomain,OU=GlassFish,O=Oracle Corporation,L=Santa Clara,ST=California,C=US]
Distinguished Name of the self-signed X.509 Server Certificate is:
[CN=localhost.localdomain-instance,OU=GlassFish,O=Oracle Corporation,L=Santa Clara,ST=California,C=US]
No domain initializers found, bypassing customization step
Domain spec2004-1 created.
Domain spec2004-1 admin port is 4848.
Domain spec2004-1 admin user is "admin".
Command create-domain executed successfully.
+ echo

+ echo 'The original master password is changeit, please change to javaperf here:'
The original master password is changeit, please change to javaperf here:
+ echo

+ /opt/glassfish3/glassfish/bin/asadmin --user admin change-master-password spec2004-1
Enter the current master password>
Enter the new master password>
Enter the new master password again>
Command change-master-password executed successfully.
+ /opt/glassfish3/glassfish/bin/asadmin --user admin --passwordfile ./passwordfile start-domain spec2004-1
Waiting for spec2004-1 to start ...
Successfully started the domain : spec2004-1
domain  Location: /opt/glassfish3/glassfish/domains/spec2004-1
Log File: /opt/glassfish3/glassfish/domains/spec2004-1/logs/server.log
Admin Port: 4848
Command start-domain executed successfully.
+ . ./env.sh
++ . ../config/glassfish.env
+++ JAS_HOME=/opt/SPECjAppServer2004
+++ JAVA_HOME=/opt/jdk
+++ FS=:
+++ BINDWAIT=45
+++ J2EE_HOME=/opt/glassfish3/glassfish
+++ J2EE_JARS=/opt/glassfish3/glassfish/lib
+++ JAS_ADMIN_PORT=4848
+++ J2EE_INSTANCE=server
+++ DB_HOST=specdb
+++ JDBC_CLASSPATH=/opt/SPECjAppServer2004/jdbc/postgresql.jar
+++ CLASSPATH=/opt/SPECjAppServer2004/jars/corp.jar:/opt/SPECjAppServer2004/jars/mfg.jar:/opt/SPECjAppServer2004/jars/orders.jar:/opt/SPECjAppServer2004/jars/supplier.jar:/opt/SPECjAppServer2004/jars/util.jar:/opt/glassfish3/glassfish/lib/appserv-rt.jar:/opt/glassfish3/glassfish/lib/javaee.jar
+++ JAVAX_JAR=/opt/glassfish3/glassfish/lib/appserv-rt.jar:/opt/glassfish3/glassfish/lib/javaee.jar
+++ JAS_HOST=specdelivery
+++ JAS_PORT=8000
+++ JAS_PREFIX=/
+++ EMULATOR_HOST=specemulator
+++ EMULATOR_PORT=8080
+++ EMULATOR_PREFIX=/
+++ IIOP_PORT=3700
+++ NAMING_PROVIDER=iiop://specdelivery:3700
+++ IIOP_ENDPOINTS=specdelivery:3700
+++ JAVA='/opt/jdk/bin/java -server -XX:+DisableExplicitGC -Xmx256m -Xss256k -Djava.naming.provider.url=iiop://specdelivery:3700 -Djava.naming.provider.url.pkgs=com.sun.enterprise.naming -Dcom.sun.appserv.iiop.orbconnections=5 -Dcom.sun.appserv.iiop.endpoints=specdelivery:3700 -Dorg.omg.CORBA.ORBInitialHost=specdelivery -Dorg.omg.CORBA.ORBInitialPort=3700 -Dnetworkaddress.cache.ttl=0 -Dsun.net.inetaddr.ttl=0 '
+++ export JAS_HOME JAVA_HOME JAVAX_JAR JAS_HOST JAS_PORT EMULATOR_HOST EMULATOR_PORT LD_LIBRARY_PATH CLASSPATH J2EE_HOME
++ ADMIN_USER=admin
++ PASSFILE=./passwordfile
++ ADMIN_PORT=4848
++ DOMAIN=spec2004-1
++ CONNECTION_POOL=SpecJPool
++ DBHOST=specdb
++ DBNAME=specdb
++ DBPORT=5432
++ DBUSER=spec
++ DBPASS=spec
++ JARSDIR=/opt/SPECjAppServer2004/jars
++ EMU_HOME=/opt/tomcat
++ PRIOCNTL=
+ /opt/glassfish3/glassfish/bin/asadmin -u admin --passwordfile ./passwordfile -H specdelivery -p 4848 create-jms-resource --restype javax.jms.QueueConnectionFactory --property max-pool-size=2000:steady-pool-size=16:pool-resize-quantity=16:idle-timeout-in-seconds=600 jms/QueueConnectionFactory
java.net.NoRouteToHostException: No route to host (Host unreachable)
Command create-jms-resource failed.

[RamyaMeruva]

Hi Johnnyli,

Can u attach server.log & domain.xml as well?

Thanks

[johnnyli]

Recreate appserver and rollback to java7, http://specdelivery:4848 is able to access now, see Screenshot.png
but ./setup_files.sh report "SEVERE: Could not contact localhost:8005. Tomcat may not be running." at the end.  Is appserver ok now?

server.log and domain.xml are attached.

Below 3 tests seem ok
http://specdelivery:8000/SPECjAppServer/app?action=atomicityTests
see AtomicityTest.txt

http://specdelivery:8000/Supplier/DeliveryServlet
Servlet seems to work OK
Number of Transactions : 0
Servlet invoked without command specified

http://specemulator:8080/Emulator/EmulatorServlet
Emulator Servlet seems to work OK
JAS_HOST : specdelivery
JAS_PORT : 8000
Servlet URL : Supplier/DeliveryServlet
Number of Transactions : 0
Servlet invoked without command specified

----------------------------------------------------------------------------------------------------------------------------------------------------
-Djava.naming.provider.url=iiop://specdelivery:3700 -Djava.naming.provider.url.pkgs=com.sun.enterprise.naming -Dcom.sun.appserv.iiop.orbconnections=5 -Dcom.sun.appserv.iiop.endpoints=specdelivery:3700 -Dorg.omg.CORBA.ORBInitialHost=specdelivery -Dorg.omg.CORBA.ORBInitialPort=3700 -Dnetworkaddress.cache.ttl=0 -Dsun.net.inetaddr.ttl=0 '
+++ export JAS_HOME JAVA_HOME JAVAX_JAR JAS_HOST JAS_PORT EMULATOR_HOST EMULATOR_PORT LD_LIBRARY_PATH CLASSPATH J2EE_HOME
++ ADMIN_USER=admin
++ PASSFILE=./passwordfile
++ ADMIN_PORT=4848
++ DOMAIN=spec2004-1
++ CONNECTION_POOL=SpecJPool
++ DBHOST=specdb
++ DBNAME=specdb
++ DBPORT=5432
++ DBUSER=spec
++ DBPASS=spec
++ JARSDIR=/opt/SPECjAppServer2004/jars
++ EMU_HOME=/opt/tomcat
++ PRIOCNTL=
+ rm /opt/tomcat/webapps/Emulator.war
rm: cannot remove `/opt/tomcat/webapps/Emulator.war': No such file or directory
+ rm -rf /opt/tomcat/webapps/Emulator
+ cp /opt/SPECjAppServer2004/jars/emulator.war /opt/tomcat/webapps/Emulator.war
Using CATALINA_BASE:   /opt/tomcat
Using CATALINA_HOME:   /opt/tomcat
Using CATALINA_TMPDIR: /opt/tomcat/temp
Using JRE_HOME:        /opt/jdk
Using CLASSPATH:       /opt/tomcat/bin/bootstrap.jar:/opt/tomcat/bin/tomcat-juli.jar
Jun 25, 2018 8:48:21 PM org.apache.catalina.startup.Catalina stopServer
SEVERE: Could not contact localhost:8005. Tomcat may not be running.
Jun 25, 2018 8:48:22 PM org.apache.catalina.startup.Catalina stopServer
SEVERE: Catalina.stop:
java.net.ConnectException: Connection refused (Connection refused)
   at java.net.PlainSocketImpl.socketConnect(Native Method)
   at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
   at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
   at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
   at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
   at java.net.Socket.connect(Socket.java:580)
   at java.net.Socket.connect(Socket.java:529)
   at java.net.Socket.<init>(Socket.java:429)
   at java.net.Socket.<init>(Socket.java:209)
   at org.apache.catalina.startup.Catalina.stopServer(Catalina.java:498)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:606)
   at org.apache.catalina.startup.Bootstrap.stopServer(Bootstrap.java:370)
   at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:457)

Using CATALINA_BASE:   /opt/tomcat
Using CATALINA_HOME:   /opt/tomcat
Using CATALINA_TMPDIR: /opt/tomcat/temp
Using JRE_HOME:        /opt/jdk
Using CLASSPATH:       /opt/tomcat/bin/bootstrap.jar:/opt/tomcat/bin/tomcat-juli.jar
Tomcat started.
[root@appserver appvm_scripts]#

[RamyaMeruva]

Hi Johnnyli

Did u try restarting tomcat ?

-Ramya

[johnnyli]

tomcat is started by command "./catalinas.sh start" in /opt/tomcat/bin. Thanks
Finally all the servers are setup done. Thanks

BUT, runspecvirt reports error!
[root@client1 other_scripts]# ./test_hostentries.sh
Testing the external IP addresses of the Virtual Machines and client

Pinging Infraserver1 VM (infraserver1)...Passed!
Pinging Webserver1 VM (webserver1)...Passed!
Pinging Mailserver1 VM (mailserver1)...Passed!
Pinging Appserver1 VM (specdelivery)...Passed!
Pinging DBserver1 VM (dbserver1)...Passed!
Pinging Batchserver1 VM (batchserver1)...Passed!
Pinging Client1 VM (client1)...Passed!

Testing the internal IP addresses of the Virtual Machines
This does not work on the client since it has no internal
network adapter. Run this test on web/infraserver and app/dbserver.

Pinging Infraserver1 VM (infraserver1-int)...Passed!
Pinging Webserver1 VM (webserver1-int)...Passed!
Pinging Appserver1 VM (appserver1-int)...Passed!
Pinging DBserver VM (specdb)...Passed!
[root@client1 SPECvirt]# ./runspecvirt.sh
runspecvirt.sh started at: Wed Jun 27 22:28:59 EDT 2018
Tile count = 1
Warning: Permanently added 'client1,192.168.1.107' (RSA) to the list of known hosts.
java: no process killed
.
SPECvirt Host = client1
Tile count = 1
client1:
Wed Jun 27 22:29:14 EDT 2018
dbserver1
Warning: Permanently added 'dbserver1,192.168.1.196' (RSA) to the list of known hosts.
Wed Jun 27 22:29:14 EDT 2018
appserver1
Wed Jun 27 22:29:14 EDT 2018
batchserver1
Warning: Permanently added 'batchserver1,192.168.1.125' (RSA) to the list of known hosts.
Wed Jun 27 22:29:14 EDT 2018
mailserver1
Warning: Permanently added 'mailserver1,192.168.1.195' (RSA) to the list of known hosts.
Wed Jun 27 22:29:14 EDT 2018
infraserver1
Warning: Permanently added 'infraserver1,192.168.1.146' (RSA) to the list of known hosts.
Wed Jun 27 22:29:15 EDT 2018
webserver1
Warning: Permanently added 'webserver1,192.168.1.130' (RSA) to the list of known hosts.
Wed Jun 27 22:29:15 EDT 2018
Preparing to start clientmgr processes for 1 tile(s) -
you have 20 seconds to Control-C to cancel
Starting 1 tile test
2018-06-27 22:30:00:639 Redirecting STDERR to primectrl.out
2018-06-27 22:30:00:640 Redirecting STDOUT to primectrl.out
To monitor test, type command: tail -f primectrl.out
[root@client1 SPECvirt]# tail -f primectrl.out
2018-06-27 22:30:00:761 Wed Jun 27 22:30:00 EDT 2018
2018-06-27 22:30:00:795 RMI server started: client1:9990
2018-06-27 22:30:00:799 [INFO] This is a perf-only benchmark run. Skipping active idle polling interval.
2018-06-27 22:30:00:800 PrimeControl: preparing client drivers.
2018-06-27 22:30:00:886 PrimeControl: starting clients...
2018-06-27 22:30:00:919 PrimeControl: PTDs not used for this benchmark run!
2018-06-27 22:30:00:919 PrimeControl: starting 4 masters.
2018-06-27 22:33:40:942 PrimeControl: waiting on 4 prime client(s).
2018-06-27 22:33:41:318 [ERROR] Received abort signal from client1:1092. Terminating.
2018-06-27 22:33:41:319 PrimeControl: sending abortTest() to prime clients.
2018-06-27 22:33:41:322 PrimeControl: stopping remote client processes
.
2018-06-27 22:33:43:328 PrimeControl: stopping local client threads
2018-06-27 22:33:43:328 Terminating client processes
2018-06-27 22:33:43:329 Sending termination signal to clientmgr processes
2018-06-27 22:33:43:329 Terminating prime client processes
2018-06-27 22:33:43:329 Sending termination signal to prime clientmgr processes
2018-06-27 22:33:43:329 Exiting...
2018-06-27 22:33:43:329 PrimeControl: terminating run. Please wait...
2018-06-27 22:33:43:332 specvirt: benchmark run failed!
2018-06-27 22:33:43:332 specvirt: Done!
[root@client1 SPECvirt]# cat Clientmgr1_1092.out
2018-06-27 22:29:41:960 Creating clientmgr using RMI Registry port 1092
2018-06-27 22:29:42:070 client1:1092 ready...
2018-06-27 22:33:40:929 Starting client1:1092
2018-06-27 22:33:40:931 Command line: [java, -jar, specbatch.jar, -sv, client1,
-svp, 9990, -id, 3, -tile, 0, -wkld, 3]
2018-06-27 22:33:40:941 Started client1:1092

Messages from: client1:1092
-> 2018-06-27 22:33:41:062 Looking up SPECvirt controller: client1
-> 2018-06-27 22:33:41:188 masterID: 3, tile: 0, workload: 3
-> 2018-06-27 22:33:41:188 hostname: client1
-> 2018-06-27 22:33:41:221 loadscale:1.0 copycount:10
-> 2018-06-27 22:33:41:238 RMI server started: client1:9903
-> 2018-06-27 22:33:41:316 Remote Exception calling setConfig() on client1:1902.
 Aborting...
-> RemoteException was: java.rmi.ServerException: RemoteException occurred in se
rver thread; nested exception is:
->    java.rmi.ConnectException: Connection refused to host: 127.0.0.1; nested
 exception is:
->    java.net.ConnectException: Connection refused (Connection refused)

[RamyaMeruva]

Hi Johnnyli,

What are the values you are using for RAMP_SECONDS &  WARMUP_SECONDS in Control.config ? Are u using the minimum values specified in the config file?

Also, after u start the run, after u see the msg "starting clients", can u check if specbatchclient.jar is running on client1 or not?

Can u attach Control.config & /etc/hosts here?

Thanks,
Ramya

[johnnyli]

RAMP_SECONDS &  WARMUP_SECONDS are default values (300 and 900) in yesterday's test. I changed to the minimum values (180, 300) today, the result is the same. Control.config is attached
RAMP_SECONDS = 180
WARMUP_SECONDS= 300

specbatchclient.jar is running after "starting clients"
[root@client1 Desktop]# ps aux | grep -i spec
root     21387  0.7  0.8 5651544 70388 pts/0   Sl   21:15   0:00 java -jar specvirt.jar -l
root     21419  0.2  0.6 5049416 49960 ?       Sl   21:15   0:00 java -jar specjappclient.jar -p 1091
root     21422  0.2  0.5 5049380 47972 ?       Sl   21:15   0:00 java -jar specwebclient.jar -p 1010
root     21428  0.3  0.6 5115952 54540 ?       Sl   21:15   0:00 java -jar specimapclient.jar -p 1200
root     21437  0.2  0.6 5049384 49748 ?       Sl   21:15   0:00 java -jar specbatchclient.jar -p 1902
root     21646  0.0  0.0 103320   928 pts/1    S+   21:17   0:00 grep -i spec


[root@client1 Desktop]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

##
# Defaults used for the SPECvirt_sc2013 Example VM Setup Guide.

# External VM-to-client communications
192.168.1.146     infraserver infraserver1
192.168.1.130     webserver webserver1
192.168.1.195     mailserver mailserver1
192.168.1.141     appserver appserver1 specdelivery specemulator
192.168.1.196     dbserver dbserver1
192.168.1.125     batchserver batchserver1

192.168.1.107   client1 specdriver

# Internal VM-to-VM only communications
192.168.1.177       infraserver1-int
192.168.1.131       webserver1-int
192.168.1.150       appserver1-int
192.168.1.164       dbserver1-int specdb

[RamyaMeruva]

Hi Johnnyli,

Can u use pollme.jar which binds to the network interface that corresponds with the name by which the VM knows itself. In this case, its going to be client1.

$java -jar /opt/SPECpoll/pollme.jar -n client1 -p 1902

Restart the workload and let me know what happens?

Thanks,
Ramya