SPEC Community
Product Support => SPECvirt_sc2013 => Topic started by: johnnyli on June 01, 2018, 01:55:24 AM
-
Hi sir,
I follow with v1.04 guide, setting up appserver get failed with the first error "Waiting for spec2004-1 to start .Error starting domain spec2004-1"
Any suggestion is welcome, thanks.
[root@appserver appvm_scripts]# ./setup_files.sh
/usr/bin/java
openjdk version "1.8.0_171"
OpenJDK Runtime Environment (build 1.8.0_171-b10)
OpenJDK 64-Bit Server VM (build 25.171-b10, mixed mode)
SPECvirt_sc2013 v1.1, build: 80
mkdir: cannot create directory `/opt/SPECjAppServer2004/jdbc': File exists
CLI306 Warning - The server located at /opt/glassfish3/glassfish/domains/spec2004-1 is not running.
Command stop-domain executed successfully.
+ . ./env.sh
++ . ../config/glassfish.env
+++ JAS_HOME=/opt/SPECjAppServer2004
+++ JAVA_HOME=/opt/jdk
+++ FS=:
+++ BINDWAIT=45
+++ J2EE_HOME=/opt/glassfish3/glassfish
+++ J2EE_JARS=/opt/glassfish3/glassfish/lib
+++ JAS_ADMIN_PORT=4848
+++ J2EE_INSTANCE=server
+++ DB_HOST=specdb
+++ JDBC_CLASSPATH=/opt/SPECjAppServer2004/jdbc/postgresql.jar
+++ CLASSPATH=/opt/SPECjAppServer2004/jars/corp.jar:/opt/SPECjAppServer2004/jars/mfg.jar:/opt/SPECjAppServer2004/jars/orders.jar:/opt/SPECjAppServer2004/jars/supplier.jar:/opt/SPECjAppServer2004/jars/util.jar:/opt/glassfish3/glassfish/lib/appserv-rt.jar:/opt/glassfish3/glassfish/lib/javaee.jar
+++ JAVAX_JAR=/opt/glassfish3/glassfish/lib/appserv-rt.jar:/opt/glassfish3/glassfish/lib/javaee.jar
+++ JAS_HOST=specdelivery
+++ JAS_PORT=8000
+++ JAS_PREFIX=/
+++ EMULATOR_HOST=specemulator
+++ EMULATOR_PORT=8080
+++ EMULATOR_PREFIX=/
+++ IIOP_PORT=3700
+++ NAMING_PROVIDER=iiop://specdelivery:3700
+++ IIOP_ENDPOINTS=specdelivery:3700
+++ JAVA='/opt/jdk/bin/java -server -XX:+DisableExplicitGC -Xmx256m -Xss256k -Djava.naming.provider.url=iiop://specdelivery:3700 -Djava.naming.provider.url.pkgs=com.sun.enterprise.naming -Dcom.sun.appserv.iiop.orbconnections=5 -Dcom.sun.appserv.iiop.endpoints=specdelivery:3700 -Dorg.omg.CORBA.ORBInitialHost=specdelivery -Dorg.omg.CORBA.ORBInitialPort=3700 -Dnetworkaddress.cache.ttl=0 -Dsun.net.inetaddr.ttl=0 '
+++ export JAS_HOME JAVA_HOME JAVAX_JAR JAS_HOST JAS_PORT EMULATOR_HOST EMULATOR_PORT LD_LIBRARY_PATH CLASSPATH J2EE_HOME
++ ADMIN_USER=admin
++ PASSFILE=./passwordfile
++ ADMIN_PORT=4848
++ DOMAIN=spec2004-1
++ CONNECTION_POOL=SpecJPool
++ DBHOST=specdb
++ DBNAME=specdb
++ DBPORT=5432
++ DBUSER=spec
++ DBPASS=spec
++ JARSDIR=/opt/SPECjAppServer2004/jars
++ EMU_HOME=/opt/tomcat
++ PRIOCNTL=
+ [[ -d /opt/glassfish3/glassfish/domains/spec2004-1 ]]
+ echo 'Deleting existing domain'
Deleting existing domain
+ /opt/glassfish3/glassfish/bin/asadmin --user admin --passwordfile ./passwordfile delete-domain spec2004-1
Domain spec2004-1 deleted.
Command delete-domain executed successfully.
+ /opt/glassfish3/glassfish/bin/asadmin --user admin --passwordfile ./passwordfile create-domain --adminport 4848 --instanceport 8000 spec2004-1
Using port 4848 for Admin.
Using port 8000 for HTTP Instance.
Using default port 7676 for JMS.
Using default port 3700 for IIOP.
Using default port 8181 for HTTP_SSL.
Using default port 3820 for IIOP_SSL.
Using default port 3920 for IIOP_MUTUALAUTH.
Using default port 8686 for JMX_ADMIN.
Using default port 6666 for OSGI_SHELL.
Using default port 9009 for JAVA_DEBUGGER.
Distinguished Name of the self-signed X.509 Server Certificate is:
[CN=appserver,OU=GlassFish,O=Oracle Corporation,L=Santa Clara,ST=California,C=US]
Distinguished Name of the self-signed X.509 Server Certificate is:
[CN=appserver-instance,OU=GlassFish,O=Oracle Corporation,L=Santa Clara,ST=California,C=US]
No domain initializers found, bypassing customization step
Domain spec2004-1 created.
Domain spec2004-1 admin port is 4848.
Domain spec2004-1 admin user is "admin".
Command create-domain executed successfully.
+ echo
+ echo 'The original master password is changeit, please change to javaperf here:'
The original master password is changeit, please change to javaperf here:
+ echo
+ /opt/glassfish3/glassfish/bin/asadmin --user admin change-master-password spec2004-1
Enter the current master password>
Enter the new master password>
Enter the new master password again>
Command change-master-password executed successfully.
+ /opt/glassfish3/glassfish/bin/asadmin --user admin --passwordfile ./passwordfile start-domain spec2004-1
Enter master password (3) attempt(s) remain)>
Waiting for spec2004-1 to start .Error starting domain spec2004-1.
The server exited prematurely with exit code 1.
Before it died, it produced the following output:
Launching GlassFish on Felix platform
ERROR: Error parsing system bundle export statement: org.osgi.framework; version=1.6.0, org.osgi.framework.launch; version=1.0.0, org.osgi.framework.wiring; version=1.0.0, org.osgi.framework.startlevel; version=1.0.0, org.osgi.framework.hooks.bundle; version=1.0.0, org.osgi.framework.hooks.resolver; version=1.0.0, org.osgi.framework.hooks.service; version=1.1.0, org.osgi.framework.hooks.weaving; version=1.0.0, org.osgi.service.packageadmin; version=1.2.0, org.osgi.service.startlevel; version=1.1.0, org.osgi.service.url; version=1.0.0, org.osgi.util.tracker; version=1.5.0, , org.glassfish.embeddable;org.glassfish.embeddable.spi;version=3.1.1 (org.osgi.framework.BundleException: Exported package names cannot be zero length.)
ERROR: Bundle jaxb-api [2] Error starting file:/opt/glassfish3/glassfish/modules/endorsed/jaxb-api-osgi.jar (org.osgi.framework.BundleException: Unresolved constraint in bundle jaxb-api [2]: Unable to resolve 2.0: missing requirement [2.0] osgi.wiring.package; (osgi.wiring.package=javax.activation))
ERROR: Bundle org.glassfish.metro.webservices-api-osgi [3] Error starting file:/opt/glassfish3/glassfish/modules/endorsed/webservices-api-osgi.jar (org.osgi.framework.BundleException: Unresolved constraint in bundle org.glassfish.metro.webservices-api-osgi [3]: Unable to resolve 3.0: missing requirement [3.0] osgi.wiring.package; (&(osgi.wiring.package=javax.xml.bind)(version>=2.2.0)) [caused by: Unable to resolve 2.0: missing requirement [2.0] osgi.wiring.package; (osgi.wiring.package=javax.activation)])
ERROR: Bundle org.glassfish.hk2.osgi-adapter [46] Error starting file:/opt/glassfish3/glassfish/modules/osgi-adapter.jar (org.osgi.framework.BundleException: Unresolved constraint in bundle org.glassfish.hk2.osgi-adapter [46]: Unable to resolve 46.0: missing requirement [46.0] osgi.wiring.package; (&(osgi.wiring.package=com.sun.enterprise.module)(version>=1.1.0)) [caused by: Unable to resolve 58.0: missing requirement [58.0] osgi.wiring.package; (&(osgi.wiring.package=org.jvnet.hk2.config)(version>=1.1.0)) [caused by: Unable to resolve 214.0: missing requirement [214.0] osgi.wiring.package; (osgi.wiring.package=javax.management)]])
ERROR: Bundle org.glassfish.main.core.glassfish [50] Error starting file:/opt/glassfish3/glassfish/modules/glassfish.jar (org.osgi.framework.BundleException: Activator start error in bundle org.glassfish.main.core.glassfish [50].)
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=192m; support was removed in 8.0
OpenJDK 64-Bit Server VM warning: ignoring option PermSize=64m; support was removed in 8.0
org.osgi.framework.BundleException: Exported package names cannot be zero length.
at org.apache.felix.framework.util.manifestparser.ManifestParser.normalizeExportClauses(ManifestParser.java:729)
at org.apache.felix.framework.util.manifestparser.ManifestParser.<init>(ManifestParser.java:191)
at org.apache.felix.framework.ExtensionManager.<init>(ExtensionManager.java:220)
at org.apache.felix.framework.Felix.<init>(Felix.java:374)
at org.apache.felix.framework.FrameworkFactory.newFramework(FrameworkFactory.java:28)
at com.sun.enterprise.glassfish.bootstrap.osgi.OSGiFrameworkLauncher.launchOSGiFrameWork(OSGiFrameworkLauncher.java:77)
at com.sun.enterprise.glassfish.bootstrap.osgi.OSGiGlassFishRuntimeBuilder.build(OSGiGlassFishRuntimeBuilder.java:128)
at org.glassfish.embeddable.GlassFishRuntime._bootstrap(GlassFishRuntime.java:157)
at org.glassfish.embeddable.GlassFishRuntime.bootstrap(GlassFishRuntime.java:110)
at com.sun.enterprise.glassfish.bootstrap.GlassFishMain$Launcher.launch(GlassFishMain.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.sun.enterprise.glassfish.bootstrap.GlassFishMain.main(GlassFishMain.java:97)
at com.sun.enterprise.glassfish.bootstrap.ASMain.main(ASMain.java:55)
Jun 01, 2018 1:43:59 AM BundleProvisioner install
WARNING: Failed to install file:/opt/glassfish3/glassfish/modules/flashlight-extra-jdk-packages.jar
org.osgi.framework.BundleException: Could not create bundle object.
at org.apache.felix.framework.Felix.installBundle(Felix.java:2785)
at org.apache.felix.framework.BundleContextImpl.installBundle(BundleContextImpl.java:165)
at com.sun.enterprise.glassfish.bootstrap.osgi.BundleProvisioner.install(BundleProvisioner.java:446)
at com.sun.enterprise.glassfish.bootstrap.osgi.BundleProvisioner.installBundles(BundleProvisioner.java:208)
at com.sun.enterprise.glassfish.bootstrap.osgi.OSGiGlassFishRuntimeBuilder.build(OSGiGlassFishRuntimeBuilder.java:139)
at org.glassfish.embeddable.GlassFishRuntime._bootstrap(GlassFishRuntime.java:157)
at org.glassfish.embeddable.GlassFishRuntime.bootstrap(GlassFishRuntime.java:110)
at com.sun.enterprise.glassfish.bootstrap.GlassFishMain$Launcher.launch(GlassFishMain.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.sun.enterprise.glassfish.bootstrap.GlassFishMain.main(GlassFishMain.java:97)
at com.sun.enterprise.glassfish.bootstrap.ASMain.main(ASMain.java:55)
Caused by: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
at java.util.ArrayList.rangeCheck(ArrayList.java:657)
at java.util.ArrayList.get(ArrayList.java:433)
at org.apache.felix.framework.ExtensionManager.addExtensionBundle(ExtensionManager.java:401)
at org.apache.felix.framework.Felix.installBundle(Felix.java:2751)
... 13 more
-
Follow 6.1 Application server workload, try 3 web link in clientserver, the first two are unable to connect, the third one seems ok.
server.log and domain.xml in appserver are attached.
What else I can check?
http://specdelivery:8000/SPECjAppServer/app?action=atomicityTests
Unable to connect
Firefox can't establish a connection to the server at specdelivery:8000.
http://specdelivery:8000/Supplier/DeliveryServlet
Unable to connect
Firefox can't establish a connection to the server at specdelivery:8000.
http://specemulator:8080/Emulator/EmulatorServlet
Emulator Servlet seems to work OK
JAS_HOST : specdelivery
JAS_PORT : 8000
Servlet URL : Supplier/DeliveryServlet
Number of Transactions : 0
Servlet invoked without command specified
[root@client1 Desktop]# ping specdelivery
PING appserver (192.168.1.147) 56(84) bytes of data.
64 bytes from appserver (192.168.1.147): icmp_seq=1 ttl=64 time=0.258 ms
64 bytes from appserver (192.168.1.147): icmp_seq=2 ttl=64 time=0.354 ms
64 bytes from appserver (192.168.1.147): icmp_seq=3 ttl=64 time=0.361 ms
64 bytes from appserver (192.168.1.147): icmp_seq=4 ttl=64 time=0.315 ms
64 bytes from appserver (192.168.1.147): icmp_seq=5 ttl=64 time=0.368 ms
-
Hi Johnnyli,
Can u check if your SElinux & firewall are disabled?
Also, can u check if u have the following entry in /etc/hosts on appserver VM:
<appserver1 IP> appserver1 appserver specdelivery specemulator
-
>>>>Can u check if your SElinux & firewall are disabled?
1)firmware is disabled in /opt/other_scripts/services_off.sh, which is called by all VMs
Also, can u check if u have the following entry in /etc/hosts on appserver VM:
# Firewall
service iptables stop; chkconfig iptables off
service ip6tables stop; chkconfig ip6tables off
2)selinux is disalbed in grub
[root@appserver1-int Desktop]# cat /proc/cmdline
ro root=/dev/mapper/VolGroup-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_NO_MD rd_LVM_LV=VolGroup/lv_swap SYSFONT=latarcyrheb-sun16 crashkernel=129M@0M rd_LVM_LV=VolGroup/lv_root KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet selinux=0 audit=0
>>>>Also, can u check if u have the following entry in /etc/hosts on appserver VM:
all VMs have below hosts file, and i confirm that each can ping successfully. The ip address is assigned through dhcp, it is not assigned statically.
[root@appserver1-int Desktop]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
##
# Defaults used for the SPECvirt_sc2013 Example VM Setup Guide.
# External VM-to-client communications
192.168.1.144 infraserver infraserver1
192.168.1.128 webserver webserver1
192.168.1.193 mailserver mailserver1
192.168.1.147 appserver appserver1 specdelivery specemulator
192.168.1.194 dbserver dbserver1
192.168.1.123 batchserver batchserver1
192.168.1.105 client1 specdriver
# Internal VM-to-VM only communications
192.168.1.176 infraserver1-int
192.168.1.129 webserver1-int
192.168.1.139 appserver1-int
192.168.1.162 dbserver1-int specdb
ping spec specemulator and specdelivery from client VM sucessully
[root@client1 Desktop]# ping specemulator
PING appserver (192.168.1.147) 56(84) bytes of data.
64 bytes from appserver (192.168.1.147): icmp_seq=1 ttl=64 time=1.31 ms
64 bytes from appserver (192.168.1.147): icmp_seq=2 ttl=64 time=0.396 ms
64 bytes from appserver (192.168.1.147): icmp_seq=3 ttl=64 time=0.468 ms
64 bytes from appserver (192.168.1.147): icmp_seq=4 ttl=64 time=0.433 ms
--- appserver ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3596ms
rtt min/avg/max/mdev = 0.396/0.652/1.311/0.381 ms
[root@client1 Desktop]# ping specdelivery
PING appserver (192.168.1.147) 56(84) bytes of data.
64 bytes from appserver (192.168.1.147): icmp_seq=1 ttl=64 time=0.388 ms
64 bytes from appserver (192.168.1.147): icmp_seq=2 ttl=64 time=0.497 ms
64 bytes from appserver (192.168.1.147): icmp_seq=3 ttl=64 time=0.361 ms
-
Can u verify if glassfish is actually installed and running by going to http://specdelivery:4848 ?
if yes, then the problem is the SPECjApp .EAR has not deployed correctly. Try re-installing GF and run setup_files.sh
Also, what version of Glassfish are u trying to use?
Is it GF312? if its a different version, it might not work with existing scripts.
Also, did u manage to resolve the error u posted on June 1st?
if not, can u use Java 7 instead of Java 8 because GF3.12 seems have to compatibility issues with Java 8.
-
>>>>Can u verify if glassfish is actually installed and running by going to http://specdelivery:4848 ?
>>>>if yes, then the problem is the SPECjApp .EAR has not deployed correctly. Try re-installing GF and run setup_files.sh
Unable to connect, Firefox can't establish a connection to the server at specdelivery:4848.
>>>>Also, what version of Glassfish are u trying to use?
>>>>Is it GF312? if its a different version, it might not work with existing scripts.
GF is downloaded from below link, http://download.java.net/glassfish/3.1.2.2/release/glassfish-3.1.2.2.zip, follow the example guide.
>>>>Also, did u manage to resolve the error u posted on June 1st?
No, still pending on that
>>>>if not, can u use Java 7 instead of Java 8 because GF3.12 seems have to compatibility issues with Java 8.
only install java jdk according to &2.1 of example guide: yum install -y java-1.8.0-openjdk java-1.8.0-openjdk-devel
shall i need to roll back to java-1.7.0-openjdk?
[root@appserver1-int Desktop]# java -showversion
openjdk version "1.8.0_171"
OpenJDK Runtime Environment (build 1.8.0_171-b10)
OpenJDK 64-Bit Server VM (build 25.171-b10, mixed mode)
-
If you're not able to connect to http://specdelivery:4848, then GF did not start successfully.
There is a way to get GF3.12 working with Java 8. Add the following line at the end of the file /opt/glassfish3/glassfish/config/osgi.properties:
jre-1.8=${jre-1.7}
If the above workaround does not work, can u rollback to Java 7 & re-install GF3.12 ?
-
setup still failed after roll back java to 1.7.0.
how can I reinstall GF? in the guide, there is only unzip command "unzip glassfish-3.1.2.2.zip", no installation command
-----------------------------------------------------------------------------------------------------------------------
[root@localhost appvm_scripts]# ./setup_files.sh
/usr/bin/java
java version "1.7.0_181"
OpenJDK Runtime Environment (rhel-2.6.14.8.el6_9-x86_64 u181-b00)
OpenJDK 64-Bit Server VM (build 24.181-b00, mixed mode)
SPECvirt_sc2013 v1.1, build: 80
mkdir: cannot create directory `/opt/SPECjAppServer2004/jdbc': File exists
CLI306 Warning - The server located at /opt/glassfish3/glassfish/domains/spec2004-1 is not running.
Command stop-domain executed successfully.
+ . ./env.sh
++ . ../config/glassfish.env
+++ JAS_HOME=/opt/SPECjAppServer2004
+++ JAVA_HOME=/opt/jdk
+++ FS=:
+++ BINDWAIT=45
+++ J2EE_HOME=/opt/glassfish3/glassfish
+++ J2EE_JARS=/opt/glassfish3/glassfish/lib
+++ JAS_ADMIN_PORT=4848
+++ J2EE_INSTANCE=server
+++ DB_HOST=specdb
+++ JDBC_CLASSPATH=/opt/SPECjAppServer2004/jdbc/postgresql.jar
+++ CLASSPATH=/opt/SPECjAppServer2004/jars/corp.jar:/opt/SPECjAppServer2004/jars/mfg.jar:/opt/SPECjAppServer2004/jars/orders.jar:/opt/SPECjAppServer2004/jars/supplier.jar:/opt/SPECjAppServer2004/jars/util.jar:/opt/glassfish3/glassfish/lib/appserv-rt.jar:/opt/glassfish3/glassfish/lib/javaee.jar
+++ JAVAX_JAR=/opt/glassfish3/glassfish/lib/appserv-rt.jar:/opt/glassfish3/glassfish/lib/javaee.jar
+++ JAS_HOST=specdelivery
+++ JAS_PORT=8000
+++ JAS_PREFIX=/
+++ EMULATOR_HOST=specemulator
+++ EMULATOR_PORT=8080
+++ EMULATOR_PREFIX=/
+++ IIOP_PORT=3700
+++ NAMING_PROVIDER=iiop://specdelivery:3700
+++ IIOP_ENDPOINTS=specdelivery:3700
+++ JAVA='/opt/jdk/bin/java -server -XX:+DisableExplicitGC -Xmx256m -Xss256k -Djava.naming.provider.url=iiop://specdelivery:3700 -Djava.naming.provider.url.pkgs=com.sun.enterprise.naming -Dcom.sun.appserv.iiop.orbconnections=5 -Dcom.sun.appserv.iiop.endpoints=specdelivery:3700 -Dorg.omg.CORBA.ORBInitialHost=specdelivery -Dorg.omg.CORBA.ORBInitialPort=3700 -Dnetworkaddress.cache.ttl=0 -Dsun.net.inetaddr.ttl=0 '
+++ export JAS_HOME JAVA_HOME JAVAX_JAR JAS_HOST JAS_PORT EMULATOR_HOST EMULATOR_PORT LD_LIBRARY_PATH CLASSPATH J2EE_HOME
++ ADMIN_USER=admin
++ PASSFILE=./passwordfile
++ ADMIN_PORT=4848
++ DOMAIN=spec2004-1
++ CONNECTION_POOL=SpecJPool
++ DBHOST=specdb
++ DBNAME=specdb
++ DBPORT=5432
++ DBUSER=spec
++ DBPASS=spec
++ JARSDIR=/opt/SPECjAppServer2004/jars
++ EMU_HOME=/opt/tomcat
++ PRIOCNTL=
+ [[ -d /opt/glassfish3/glassfish/domains/spec2004-1 ]]
+ echo 'Deleting existing domain'
Deleting existing domain
+ /opt/glassfish3/glassfish/bin/asadmin --user admin --passwordfile ./passwordfile delete-domain spec2004-1
Domain spec2004-1 deleted.
Command delete-domain executed successfully.
+ /opt/glassfish3/glassfish/bin/asadmin --user admin --passwordfile ./passwordfile create-domain --adminport 4848 --instanceport 8000 spec2004-1
Using port 4848 for Admin.
Using port 8000 for HTTP Instance.
Using default port 7676 for JMS.
Using default port 3700 for IIOP.
Using default port 8181 for HTTP_SSL.
Using default port 3820 for IIOP_SSL.
Using default port 3920 for IIOP_MUTUALAUTH.
Using default port 8686 for JMX_ADMIN.
Using default port 6666 for OSGI_SHELL.
Using default port 9009 for JAVA_DEBUGGER.
Distinguished Name of the self-signed X.509 Server Certificate is:
[CN=localhost.localdomain,OU=GlassFish,O=Oracle Corporation,L=Santa Clara,ST=California,C=US]
Distinguished Name of the self-signed X.509 Server Certificate is:
[CN=localhost.localdomain-instance,OU=GlassFish,O=Oracle Corporation,L=Santa Clara,ST=California,C=US]
No domain initializers found, bypassing customization step
Domain spec2004-1 created.
Domain spec2004-1 admin port is 4848.
Domain spec2004-1 admin user is "admin".
Command create-domain executed successfully.
+ echo
+ echo 'The original master password is changeit, please change to javaperf here:'
The original master password is changeit, please change to javaperf here:
+ echo
+ /opt/glassfish3/glassfish/bin/asadmin --user admin change-master-password spec2004-1
Enter the current master password>
Enter the new master password>
Enter the new master password again>
Command change-master-password executed successfully.
+ /opt/glassfish3/glassfish/bin/asadmin --user admin --passwordfile ./passwordfile start-domain spec2004-1
Waiting for spec2004-1 to start ...
Successfully started the domain : spec2004-1
domain Location: /opt/glassfish3/glassfish/domains/spec2004-1
Log File: /opt/glassfish3/glassfish/domains/spec2004-1/logs/server.log
Admin Port: 4848
Command start-domain executed successfully.
+ . ./env.sh
++ . ../config/glassfish.env
+++ JAS_HOME=/opt/SPECjAppServer2004
+++ JAVA_HOME=/opt/jdk
+++ FS=:
+++ BINDWAIT=45
+++ J2EE_HOME=/opt/glassfish3/glassfish
+++ J2EE_JARS=/opt/glassfish3/glassfish/lib
+++ JAS_ADMIN_PORT=4848
+++ J2EE_INSTANCE=server
+++ DB_HOST=specdb
+++ JDBC_CLASSPATH=/opt/SPECjAppServer2004/jdbc/postgresql.jar
+++ CLASSPATH=/opt/SPECjAppServer2004/jars/corp.jar:/opt/SPECjAppServer2004/jars/mfg.jar:/opt/SPECjAppServer2004/jars/orders.jar:/opt/SPECjAppServer2004/jars/supplier.jar:/opt/SPECjAppServer2004/jars/util.jar:/opt/glassfish3/glassfish/lib/appserv-rt.jar:/opt/glassfish3/glassfish/lib/javaee.jar
+++ JAVAX_JAR=/opt/glassfish3/glassfish/lib/appserv-rt.jar:/opt/glassfish3/glassfish/lib/javaee.jar
+++ JAS_HOST=specdelivery
+++ JAS_PORT=8000
+++ JAS_PREFIX=/
+++ EMULATOR_HOST=specemulator
+++ EMULATOR_PORT=8080
+++ EMULATOR_PREFIX=/
+++ IIOP_PORT=3700
+++ NAMING_PROVIDER=iiop://specdelivery:3700
+++ IIOP_ENDPOINTS=specdelivery:3700
+++ JAVA='/opt/jdk/bin/java -server -XX:+DisableExplicitGC -Xmx256m -Xss256k -Djava.naming.provider.url=iiop://specdelivery:3700 -Djava.naming.provider.url.pkgs=com.sun.enterprise.naming -Dcom.sun.appserv.iiop.orbconnections=5 -Dcom.sun.appserv.iiop.endpoints=specdelivery:3700 -Dorg.omg.CORBA.ORBInitialHost=specdelivery -Dorg.omg.CORBA.ORBInitialPort=3700 -Dnetworkaddress.cache.ttl=0 -Dsun.net.inetaddr.ttl=0 '
+++ export JAS_HOME JAVA_HOME JAVAX_JAR JAS_HOST JAS_PORT EMULATOR_HOST EMULATOR_PORT LD_LIBRARY_PATH CLASSPATH J2EE_HOME
++ ADMIN_USER=admin
++ PASSFILE=./passwordfile
++ ADMIN_PORT=4848
++ DOMAIN=spec2004-1
++ CONNECTION_POOL=SpecJPool
++ DBHOST=specdb
++ DBNAME=specdb
++ DBPORT=5432
++ DBUSER=spec
++ DBPASS=spec
++ JARSDIR=/opt/SPECjAppServer2004/jars
++ EMU_HOME=/opt/tomcat
++ PRIOCNTL=
+ /opt/glassfish3/glassfish/bin/asadmin -u admin --passwordfile ./passwordfile -H specdelivery -p 4848 create-jms-resource --restype javax.jms.QueueConnectionFactory --property max-pool-size=2000:steady-pool-size=16:pool-resize-quantity=16:idle-timeout-in-seconds=600 jms/QueueConnectionFactory
java.net.NoRouteToHostException: No route to host (Host unreachable)
Command create-jms-resource failed.
-
Hi Johnnyli,
Can u attach server.log & domain.xml as well?
Thanks
-
Recreate appserver and rollback to java7, http://specdelivery:4848 is able to access now, see Screenshot.png
but ./setup_files.sh report "SEVERE: Could not contact localhost:8005. Tomcat may not be running." at the end. Is appserver ok now?
server.log and domain.xml are attached.
Below 3 tests seem ok
http://specdelivery:8000/SPECjAppServer/app?action=atomicityTests
see AtomicityTest.txt
http://specdelivery:8000/Supplier/DeliveryServlet
Servlet seems to work OK
Number of Transactions : 0
Servlet invoked without command specified
http://specemulator:8080/Emulator/EmulatorServlet
Emulator Servlet seems to work OK
JAS_HOST : specdelivery
JAS_PORT : 8000
Servlet URL : Supplier/DeliveryServlet
Number of Transactions : 0
Servlet invoked without command specified
----------------------------------------------------------------------------------------------------------------------------------------------------
-Djava.naming.provider.url=iiop://specdelivery:3700 -Djava.naming.provider.url.pkgs=com.sun.enterprise.naming -Dcom.sun.appserv.iiop.orbconnections=5 -Dcom.sun.appserv.iiop.endpoints=specdelivery:3700 -Dorg.omg.CORBA.ORBInitialHost=specdelivery -Dorg.omg.CORBA.ORBInitialPort=3700 -Dnetworkaddress.cache.ttl=0 -Dsun.net.inetaddr.ttl=0 '
+++ export JAS_HOME JAVA_HOME JAVAX_JAR JAS_HOST JAS_PORT EMULATOR_HOST EMULATOR_PORT LD_LIBRARY_PATH CLASSPATH J2EE_HOME
++ ADMIN_USER=admin
++ PASSFILE=./passwordfile
++ ADMIN_PORT=4848
++ DOMAIN=spec2004-1
++ CONNECTION_POOL=SpecJPool
++ DBHOST=specdb
++ DBNAME=specdb
++ DBPORT=5432
++ DBUSER=spec
++ DBPASS=spec
++ JARSDIR=/opt/SPECjAppServer2004/jars
++ EMU_HOME=/opt/tomcat
++ PRIOCNTL=
+ rm /opt/tomcat/webapps/Emulator.war
rm: cannot remove `/opt/tomcat/webapps/Emulator.war': No such file or directory
+ rm -rf /opt/tomcat/webapps/Emulator
+ cp /opt/SPECjAppServer2004/jars/emulator.war /opt/tomcat/webapps/Emulator.war
Using CATALINA_BASE: /opt/tomcat
Using CATALINA_HOME: /opt/tomcat
Using CATALINA_TMPDIR: /opt/tomcat/temp
Using JRE_HOME: /opt/jdk
Using CLASSPATH: /opt/tomcat/bin/bootstrap.jar:/opt/tomcat/bin/tomcat-juli.jar
Jun 25, 2018 8:48:21 PM org.apache.catalina.startup.Catalina stopServer
SEVERE: Could not contact localhost:8005. Tomcat may not be running.
Jun 25, 2018 8:48:22 PM org.apache.catalina.startup.Catalina stopServer
SEVERE: Catalina.stop:
java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:580)
at java.net.Socket.connect(Socket.java:529)
at java.net.Socket.<init>(Socket.java:429)
at java.net.Socket.<init>(Socket.java:209)
at org.apache.catalina.startup.Catalina.stopServer(Catalina.java:498)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.catalina.startup.Bootstrap.stopServer(Bootstrap.java:370)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:457)
Using CATALINA_BASE: /opt/tomcat
Using CATALINA_HOME: /opt/tomcat
Using CATALINA_TMPDIR: /opt/tomcat/temp
Using JRE_HOME: /opt/jdk
Using CLASSPATH: /opt/tomcat/bin/bootstrap.jar:/opt/tomcat/bin/tomcat-juli.jar
Tomcat started.
[root@appserver appvm_scripts]#
-
Hi Johnnyli
Did u try restarting tomcat ?
-Ramya
-
tomcat is started by command "./catalinas.sh start" in /opt/tomcat/bin. Thanks
Finally all the servers are setup done. Thanks
BUT, runspecvirt reports error!
[root@client1 other_scripts]# ./test_hostentries.sh
Testing the external IP addresses of the Virtual Machines and client
Pinging Infraserver1 VM (infraserver1)...Passed!
Pinging Webserver1 VM (webserver1)...Passed!
Pinging Mailserver1 VM (mailserver1)...Passed!
Pinging Appserver1 VM (specdelivery)...Passed!
Pinging DBserver1 VM (dbserver1)...Passed!
Pinging Batchserver1 VM (batchserver1)...Passed!
Pinging Client1 VM (client1)...Passed!
Testing the internal IP addresses of the Virtual Machines
This does not work on the client since it has no internal
network adapter. Run this test on web/infraserver and app/dbserver.
Pinging Infraserver1 VM (infraserver1-int)...Passed!
Pinging Webserver1 VM (webserver1-int)...Passed!
Pinging Appserver1 VM (appserver1-int)...Passed!
Pinging DBserver VM (specdb)...Passed!
[root@client1 SPECvirt]# ./runspecvirt.sh
runspecvirt.sh started at: Wed Jun 27 22:28:59 EDT 2018
Tile count = 1
Warning: Permanently added 'client1,192.168.1.107' (RSA) to the list of known hosts.
java: no process killed
.
SPECvirt Host = client1
Tile count = 1
client1:
Wed Jun 27 22:29:14 EDT 2018
dbserver1
Warning: Permanently added 'dbserver1,192.168.1.196' (RSA) to the list of known hosts.
Wed Jun 27 22:29:14 EDT 2018
appserver1
Wed Jun 27 22:29:14 EDT 2018
batchserver1
Warning: Permanently added 'batchserver1,192.168.1.125' (RSA) to the list of known hosts.
Wed Jun 27 22:29:14 EDT 2018
mailserver1
Warning: Permanently added 'mailserver1,192.168.1.195' (RSA) to the list of known hosts.
Wed Jun 27 22:29:14 EDT 2018
infraserver1
Warning: Permanently added 'infraserver1,192.168.1.146' (RSA) to the list of known hosts.
Wed Jun 27 22:29:15 EDT 2018
webserver1
Warning: Permanently added 'webserver1,192.168.1.130' (RSA) to the list of known hosts.
Wed Jun 27 22:29:15 EDT 2018
Preparing to start clientmgr processes for 1 tile(s) -
you have 20 seconds to Control-C to cancel
Starting 1 tile test
2018-06-27 22:30:00:639 Redirecting STDERR to primectrl.out
2018-06-27 22:30:00:640 Redirecting STDOUT to primectrl.out
To monitor test, type command: tail -f primectrl.out
[root@client1 SPECvirt]# tail -f primectrl.out
2018-06-27 22:30:00:761 Wed Jun 27 22:30:00 EDT 2018
2018-06-27 22:30:00:795 RMI server started: client1:9990
2018-06-27 22:30:00:799 [INFO] This is a perf-only benchmark run. Skipping active idle polling interval.
2018-06-27 22:30:00:800 PrimeControl: preparing client drivers.
2018-06-27 22:30:00:886 PrimeControl: starting clients...
2018-06-27 22:30:00:919 PrimeControl: PTDs not used for this benchmark run!
2018-06-27 22:30:00:919 PrimeControl: starting 4 masters.
2018-06-27 22:33:40:942 PrimeControl: waiting on 4 prime client(s).
2018-06-27 22:33:41:318 [ERROR] Received abort signal from client1:1092. Terminating.
2018-06-27 22:33:41:319 PrimeControl: sending abortTest() to prime clients.
2018-06-27 22:33:41:322 PrimeControl: stopping remote client processes
.
2018-06-27 22:33:43:328 PrimeControl: stopping local client threads
2018-06-27 22:33:43:328 Terminating client processes
2018-06-27 22:33:43:329 Sending termination signal to clientmgr processes
2018-06-27 22:33:43:329 Terminating prime client processes
2018-06-27 22:33:43:329 Sending termination signal to prime clientmgr processes
2018-06-27 22:33:43:329 Exiting...
2018-06-27 22:33:43:329 PrimeControl: terminating run. Please wait...
2018-06-27 22:33:43:332 specvirt: benchmark run failed!
2018-06-27 22:33:43:332 specvirt: Done!
[root@client1 SPECvirt]# cat Clientmgr1_1092.out
2018-06-27 22:29:41:960 Creating clientmgr using RMI Registry port 1092
2018-06-27 22:29:42:070 client1:1092 ready...
2018-06-27 22:33:40:929 Starting client1:1092
2018-06-27 22:33:40:931 Command line: [java, -jar, specbatch.jar, -sv, client1,
-svp, 9990, -id, 3, -tile, 0, -wkld, 3]
2018-06-27 22:33:40:941 Started client1:1092
Messages from: client1:1092
-> 2018-06-27 22:33:41:062 Looking up SPECvirt controller: client1
-> 2018-06-27 22:33:41:188 masterID: 3, tile: 0, workload: 3
-> 2018-06-27 22:33:41:188 hostname: client1
-> 2018-06-27 22:33:41:221 loadscale:1.0 copycount:10
-> 2018-06-27 22:33:41:238 RMI server started: client1:9903
-> 2018-06-27 22:33:41:316 Remote Exception calling setConfig() on client1:1902.
Aborting...
-> RemoteException was: java.rmi.ServerException: RemoteException occurred in se
rver thread; nested exception is:
-> java.rmi.ConnectException: Connection refused to host: 127.0.0.1; nested
exception is:
-> java.net.ConnectException: Connection refused (Connection refused)
-
Hi Johnnyli,
What are the values you are using for RAMP_SECONDS & WARMUP_SECONDS in Control.config ? Are u using the minimum values specified in the config file?
Also, after u start the run, after u see the msg "starting clients", can u check if specbatchclient.jar is running on client1 or not?
Can u attach Control.config & /etc/hosts here?
Thanks,
Ramya
-
RAMP_SECONDS & WARMUP_SECONDS are default values (300 and 900) in yesterday's test. I changed to the minimum values (180, 300) today, the result is the same. Control.config is attached
RAMP_SECONDS = 180
WARMUP_SECONDS= 300
specbatchclient.jar is running after "starting clients"
[root@client1 Desktop]# ps aux | grep -i spec
root 21387 0.7 0.8 5651544 70388 pts/0 Sl 21:15 0:00 java -jar specvirt.jar -l
root 21419 0.2 0.6 5049416 49960 ? Sl 21:15 0:00 java -jar specjappclient.jar -p 1091
root 21422 0.2 0.5 5049380 47972 ? Sl 21:15 0:00 java -jar specwebclient.jar -p 1010
root 21428 0.3 0.6 5115952 54540 ? Sl 21:15 0:00 java -jar specimapclient.jar -p 1200
root 21437 0.2 0.6 5049384 49748 ? Sl 21:15 0:00 java -jar specbatchclient.jar -p 1902
root 21646 0.0 0.0 103320 928 pts/1 S+ 21:17 0:00 grep -i spec
[root@client1 Desktop]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
##
# Defaults used for the SPECvirt_sc2013 Example VM Setup Guide.
# External VM-to-client communications
192.168.1.146 infraserver infraserver1
192.168.1.130 webserver webserver1
192.168.1.195 mailserver mailserver1
192.168.1.141 appserver appserver1 specdelivery specemulator
192.168.1.196 dbserver dbserver1
192.168.1.125 batchserver batchserver1
192.168.1.107 client1 specdriver
# Internal VM-to-VM only communications
192.168.1.177 infraserver1-int
192.168.1.131 webserver1-int
192.168.1.150 appserver1-int
192.168.1.164 dbserver1-int specdb
-
Hi Johnnyli,
Can u use pollme.jar which binds to the network interface that corresponds with the name by which the VM knows itself. In this case, its going to be client1.
$java -jar /opt/SPECpoll/pollme.jar -n client1 -p 1902
Restart the workload and let me know what happens?
Thanks,
Ramya
-
pollme.jar is killed after launch ./runspecvirt.sh.
[root@client1 SPECvirt]# java -jar /opt/SPECpoll/pollme.jar -n client1 -p 1902
Creating RMI listener using RMI Registry port 1902
client1/192.168.1.107:1902 ready...
Killed
set DEBUG_LEVEL to 5 in Control.config get some more logs
Do I need to install JAVA 1.7.0 for all VMs? Currently only appserver1 install JAVA 1.7.0.
[root@client1 SPECvirt]# ./runspecvirt.sh
runspecvirt.sh started at: Sun Jul 1 22:59:56 EDT 2018
Tile count = 1
SPECvirt Host = client1
Tile count = 1
client1:
Sun Jul 1 23:00:11 EDT 2018
dbserver1
Sun Jul 1 23:00:11 EDT 2018
appserver1
Sun Jul 1 23:00:11 EDT 2018
batchserver1
Sun Jul 1 23:00:11 EDT 2018
mailserver1
Sun Jul 1 23:00:11 EDT 2018
infraserver1
Sun Jul 1 23:00:12 EDT 2018
webserver1
Sun Jul 1 23:00:12 EDT 2018
Preparing to start clientmgr processes for 1 tile(s) -
you have 20 seconds to Control-C to cancel
Starting 1 tile test
2018-07-01 23:00:57:521 Redirecting STDERR to primectrl.out
2018-07-01 23:00:57:521 Redirecting STDOUT to primectrl.out
To monitor test, type command: tail -f primectrl.out
[root@client1 SPECvirt]# tail -f primectrl.out
2018-07-01 23:00:57:731 PrimeControl: switch: jobNumber=10; id=2
2018-07-01 23:00:57:731 PrimeControl: got job: 10; id=3
2018-07-01 23:00:57:731 PrimeControl: switch: jobNumber=10; id=3
2018-07-01 23:00:57:731 PrimeControl: getting job... id=3
2018-07-01 23:00:57:731 PrimeControl: got job: 10; id=1
2018-07-01 23:00:57:731 PrimeControl: switch: jobNumber=10; id=1
2018-07-01 23:00:57:731 PrimeControl: got job: 10; id=0
2018-07-01 23:00:57:731 PrimeControl: switch: jobNumber=10; id=0
2018-07-01 23:01:11:488 PrimeControl: getting job... id=1
2018-07-01 23:01:26:924 PrimeControl: getting job... id=2
2018-07-01 23:04:36:608 PrimeControl: getting job... id=0
2018-07-01 23:04:36:608 PrimeControl: calling primeJob.setJob(0).
2018-07-01 23:04:36:608 PrimeControl: setting job: 0
2018-07-01 23:04:36:609 PrimeControl: got job: 0; id=0
2018-07-01 23:04:36:609 PrimeControl: switch: jobNumber=0; id=0
2018-07-01 23:04:36:610 PrimeControl: master[0][0] sleeping 20 sec.
2018-07-01 23:04:36:610 PrimeControl: got job: 0; id=2
2018-07-01 23:04:36:610 PrimeControl: switch: jobNumber=0; id=2
2018-07-01 23:04:36:610 PrimeControl: got job: 0; id=1
2018-07-01 23:04:36:610 PrimeControl: master[0][2] sleeping 20 sec.
2018-07-01 23:04:36:610 PrimeControl: switch: jobNumber=0; id=1
2018-07-01 23:04:36:610 PrimeControl: master[0][1] sleeping 20 sec.
2018-07-01 23:04:36:611 PrimeControl: got job: 0; id=3
2018-07-01 23:04:36:611 PrimeControl: switch: jobNumber=0; id=3
2018-07-01 23:04:36:611 PrimeControl: master[0][3] sleeping 20 sec.
2018-07-01 23:04:56:610 PrimeControl: calling startMaster: client1:1098 for tile 0, benchmark 0
2018-07-01 23:04:56:610 PrimeControl: calling startMaster: client1:1094 for tile 0, benchmark 2
2018-07-01 23:04:56:611 PrimeControl: calling startMaster: client1:1096 for tile 0, benchmark 1
2018-07-01 23:04:56:611 PrimeControl: calling startMaster: client1:1092 for tile 0, benchmark 3
2018-07-01 23:04:56:628 PrimeControl: getting job... id=0
2018-07-01 23:04:56:628 PrimeControl: getting job... id=1
2018-07-01 23:04:56:633 PrimeControl: getting job... id=2
2018-07-01 23:04:56:636 PrimeControl: getting job... id=3
2018-07-01 23:04:56:641 PrimeControl: waiting on 4 prime client(s).
2018-07-01 23:04:56:851 Sending config to client1:1098
2018-07-01 23:04:56:860 Sending config to client1:1096
2018-07-01 23:04:56:863 Sending config to client1:1092
2018-07-01 23:04:56:874 Sending config to client1:1094
2018-07-01 23:04:56:899 specvirt: primeRmiStarted = 1; id=0
2018-07-01 23:04:56:903 specvirt: primeRmiStarted = 2; id=3
2018-07-01 23:04:56:944 [ERROR] Received abort signal from client1:1092. Terminating.
2018-07-01 23:04:56:944 PrimeControl: sending abortTest() to prime clients.
2018-07-01 23:04:56:944 PrimeControl: setting job: 6
2018-07-01 23:04:56:944 PrimeControl: got job: 6; id=3
2018-07-01 23:04:56:944 PrimeControl: switch: jobNumber=6; id=3
2018-07-01 23:04:56:944 PrimeControl: got job: 6; id=2
2018-07-01 23:04:56:944 specvirt: primeRmiStarted = 3; id=2
2018-07-01 23:04:56:944 PrimeControl: id=3, abortID=3
2018-07-01 23:04:56:944 PrimeControl: getting job... id=3
2018-07-01 23:04:56:944 PrimeControl: got job: 6; id=1
2018-07-01 23:04:56:946 PrimeControl: switch: jobNumber=6; id=1
2018-07-01 23:04:56:946 PrimeControl: id=1, abortID=3
2018-07-01 23:04:56:946 PrimeControl: getting job... id=1
2018-07-01 23:04:56:944 PrimeControl: switch: jobNumber=6; id=2
2018-07-01 23:04:56:946 PrimeControl: id=2, abortID=3
2018-07-01 23:04:56:946 PrimeControl: getting job... id=2
2018-07-01 23:04:56:946 PrimeControl: got job: 6; id=0
2018-07-01 23:04:56:947 PrimeControl: switch: jobNumber=6; id=0
2018-07-01 23:04:56:947 PrimeControl: id=0, abortID=3
2018-07-01 23:04:56:947 PrimeControl: getting job... id=0
2018-07-01 23:04:56:951 PrimeControl: stopping remote client processes
2018-07-01 23:04:56:951 PrimeControl: setting job: 1
2018-07-01 23:04:56:951 PrimeControl: client: waiting for all done...done=0
2018-07-01 23:04:56:951 PrimeControl: got job: 1
2018-07-01 23:04:56:951 PrimeControl: calling client.stopClients() ...
2018-07-01 23:04:56:976 specvirt: primeRmiStarted = 4; id=1
2018-07-01 23:04:57:041 PrimeControl: client1:1094 (PRIME_HOST[0][2]) run complete; numStarted = -1
.
2018-07-01 23:04:57:642 PrimeControl: calling primeJob.setJob(1)
2018-07-01 23:04:57:642 PrimeControl: setting job: 1
2018-07-01 23:04:57:642 PrimeControl: got job: 1; id=0
2018-07-01 23:04:57:642 PrimeControl: switch: jobNumber=1; id=0
2018-07-01 23:04:57:642 PrimeControl: got job: 1; id=3
2018-07-01 23:04:57:643 PrimeControl: Naming lookup for 192.168.1.107:9900
2018-07-01 23:04:57:643 PrimeControl: switch: jobNumber=1; id=3
2018-07-01 23:04:57:643 PrimeControl: got job: 1; id=1
2018-07-01 23:04:57:643 PrimeControl: Naming lookup for 192.168.1.107:9903
2018-07-01 23:04:57:643 PrimeControl: switch: jobNumber=1; id=1
2018-07-01 23:04:57:643 PrimeControl: got job: 1; id=2
2018-07-01 23:04:57:643 PrimeControl: switch: jobNumber=1; id=2
2018-07-01 23:04:57:643 PrimeControl: Naming lookup for 192.168.1.107:9902
2018-07-01 23:04:57:643 PrimeControl: Naming lookup for 192.168.1.107:9901
2018-07-01 23:04:57:653 PrimeControl: done in Naming lookup for 192.168.1.107:9902
2018-07-01 23:04:57:653 PrimeControl: calling getHostVMs()...
2018-07-01 23:04:57:653 PrimeControl: done in Naming lookup for 192.168.1.107:9900
2018-07-01 23:04:57:654 PrimeControl: calling getHostVMs()...
2018-07-01 23:04:57:654 PrimeControl: hostVMs[2] = mailserver:8001
2018-07-01 23:04:57:654 PrimeControl: calling getBuildNumber()...
2018-07-01 23:04:57:655 PrimeControl: buildNum[2] = 80
2018-07-01 23:04:57:655 PrimeControl: getting job... id=2
2018-07-01 23:04:57:658 PrimeControl: hostVMs[0] = specdelivery:8001,specdb:8001
2018-07-01 23:04:57:658 PrimeControl: calling getBuildNumber()...
2018-07-01 23:04:58:955 PrimeControl: client: getting job...done=1
2018-07-01 23:04:58:956 PrimeControl: stopping local client threads
2018-07-01 23:04:58:956 PrimeControl: setting job: 2
2018-07-01 23:04:58:956 PrimeControl: client: waiting for all done...done=0
2018-07-01 23:04:58:956 PrimeControl: got job: 2
2018-07-01 23:04:58:956 PrimeControl: exiting clientThread.
2018-07-01 23:04:58:959 Terminating client processes
2018-07-01 23:04:58:959 PrimeControl: setting job: 1
2018-07-01 23:04:58:959 Sending termination signal to clientmgr processes
2018-07-01 23:04:58:959 PrimeControl: setting job: 2
2018-07-01 23:04:58:959 Terminating prime client processes
2018-07-01 23:04:58:959 PrimeControl: setting job: 4
2018-07-01 23:04:58:959 Sending termination signal to prime clientmgr processes
2018-07-01 23:04:58:959 PrimeControl: setting job: 5
2018-07-01 23:04:58:959 Exiting...
2018-07-01 23:04:58:959 calling control.stop()...
2018-07-01 23:04:58:959 PrimeControl: terminating run. Please wait...
2018-07-01 23:04:58:959 PrimeControl: got job: 4; id=2
2018-07-01 23:04:58:960 PrimeControl: switch: jobNumber=4; id=2
2018-07-01 23:04:58:960 PrimeControl: stopping prime client client1:1094
2018-07-01 23:04:58:963 specvirt: benchmark run failed!
2018-07-01 23:04:58:963 specvirt: Done!
^C
[root@client1 SPECvirt]# cat Clientmgr1_1092.out
2018-07-01 23:00:37:649 Creating clientmgr using RMI Registry port 1092
2018-07-01 23:00:37:692 client1:1092 ready...
2018-07-01 23:04:56:617 Starting client1:1092
2018-07-01 23:04:56:619 Command line: [java, -jar, specbatch.jar, -sv, client1, -svp, 9990, -id, 3, -tile, 0, -wkld, 3]
2018-07-01 23:04:56:634 Started client1:1092
Messages from: client1:1092
-> 2018-07-01 23:04:56:736 Looking up SPECvirt controller: client1
-> 2018-07-01 23:04:56:863 masterID: 3, tile: 0, workload: 3
-> 2018-07-01 23:04:56:863 hostname: client1
-> 2018-07-01 23:04:56:888 SPECbatch prime result directory:20180701-230057
-> 2018-07-01 23:04:56:888 loadscale:1.0 copycount:10
-> 2018-07-01 23:04:56:903 RMI server started: client1:9903
-> 2018-07-01 23:04:56:904 Getting host VM name for masterID=3
-> 2018-07-01 23:04:56:905 Host VM names for masterID=3: batchserver:1901
-> 2018-07-01 23:04:56:943 Remote Exception calling setConfig() on client1:1902. Aborting...
-> RemoteException was: java.rmi.ServerException: RemoteException occurred in server thread; nested exception is:
-> java.rmi.ConnectException: Connection refused to host: 127.0.0.1; nested exception is:
-> java.net.ConnectException: Connection refused (Connection refused)
[root@client1 SPECvirt]# cat Clientmgr1_1094.out
2018-07-01 23:00:37:666 Creating clientmgr using RMI Registry port 1094
2018-07-01 23:00:37:704 client1:1094 ready...
Sun Jul 1 23:00:56 EDT 2018: Starting mailInit.sh
Stopping Dovecot Imap: [ OK ]
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/VolGroup-lv_root
17938864 5443812 11577140 32% /
tmpfs 1962248 80 1962168 1% /dev/shm
/dev/vda1 487652 40116 421936 9% /boot
/dev/vdb 30832636 12009036 17250736 42% /mailstore
Sun Jul 1 23:01:15 EDT 2018: Starting dovecot
Starting Dovecot Imap: [ OK ]
11 93 759
Sun Jul 1 23:01:26 EDT 2018: Ending mailInit.sh
2018-07-01 23:04:56:618 Starting client1:1094
2018-07-01 23:04:56:619 Command line: [java, -jar, specimap.jar, -calibrate, -sv, client1, -svp, 9990, -id, 2, -tile, 0, -wkld, 2]
2018-07-01 23:04:56:631 Started client1:1094
Messages from: client1:1094
-> 2018-07-01 23:04:56:740 Looking up SPECvirt controller: client1
-> 2018-07-01 23:04:56:873 masterID: 2, tile: 0, workload: 2
-> 2018-07-01 23:04:56:873 RMI hostname: client1
-> fixedFile: IMAP_fixed.rc; configFile: IMAP_config.rc; workloadFile: IMAP_sysinfo.rc
-> 2018-07-01 23:04:56:943 RMI server started: client1:9902
-> 2018-07-01 23:04:56:948 SpecimapControl: Message initialization=false
-> 2018-07-01 23:04:56:948 SpecimapControl: Folder initialization=false
-> 2018-07-01 23:04:56:948 SpecimapControl: Message verification=false
-> 2018-07-01 23:04:56:948 SpecimapControl: Folder verification=false
-> 2018-07-01 23:04:56:948 SpecimapControl: Message clean=false
-> 2018-07-01 23:04:56:948 SpecimapControl: Folder clean=false
-> 2018-07-01 23:04:56:948 SpecimapControl: rampUp=true
-> 2018-07-01 23:04:56:948 SpecimapControl: warmUp=true
-> 2018-07-01 23:04:56:948 SpecimapControl: benchmark=true
-> 2018-07-01 23:04:56:948 SpecimapControl: overwritten=false
-> 2018-07-01 23:04:56:948 SpecimapControl: calibrate=true
-> 2018-07-01 23:04:56:953 ******************** Specmail benchmark started
-> 2018-07-01 23:04:56:981 Adding host client1:1200
-> 2018-07-01 23:04:56:987 Checking for all required parameters in config.rc
-> 2018-07-01 23:04:56:987 All required parameters in config.rc file are present
-> 2018-07-01 23:04:56:987 TEST LOAD = 100%
-> 2018-07-01 23:04:56:987 Setting workload...
-> 2018-07-01 23:04:56:988 Current load factor is: 100%
-> 2018-07-01 23:04:56:988 Provisioned users involved in this test:500
-> 2018-07-01 23:04:56:988 Workload: client1:1200,CT,test1-test500,mailserver,6
-> 2018-07-01 23:04:56:991 Adding clients...
-> 2018-07-01 23:04:56:991 Adding host[0]: client1:1200
-> 2018-07-01 23:04:56:992 Initializing clients (this may take a few minutes)...
-> [ERROR] specimapclient specimapclient_Stub[UnicastRef [liveRef: [endpoint:[192.168.1.107:10421](remote),objID:[2ede5b1:16458efa554:-7fff, -6394004371953872297]]]] returned a faulty response to getVmSysTimes() call. Check this client's console output for more information about the cause.
-> 2018-07-01 23:04:57:041 Abortiing.
-> Error: Problem encoding the raw results file.
2018-07-01 23:05:00:961 Terminating processes. Please wait...
2018-07-01 23:05:00:962 Killing master procs ...
2018-07-01 23:05:00:962 Done killing procs ...
[root@client1 SPECvirt]# cat Clientmgr1_1096.out
2018-07-01 23:00:37:636 Creating clientmgr using RMI Registry port 1096
2018-07-01 23:00:37:682 client1:1096 ready...
Sun Jul 1 23:00:57 EDT 2018: Starting webInit.sh
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/VolGroup-lv_root
17938864 5472620 11548332 33% /
Stopping httpd: [ OK ]
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/VolGroup-lv_root
17938864 5472632 11548320 33% /
tmpfs 510048 228 509820 1% /dev/shm
/dev/vda1 487652 40116 421936 9% /boot
/dev/vdb 61796348 33286072 25364548 57% /home/webfiles/downloads
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/VolGroup-lv_root
17938864 5426744 11594208 32% /
Stopping httpd: [ OK ]
Starting httpd: [ OK ]
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/VolGroup-lv_root
17938864 5426768 11594184 32% /
tmpfs 5064832 80 5064752 1% /dev/shm
/dev/vda1 487652 40116 421936 9% /boot
infraserver1-int:/home/webfiles/downloads
61796352 33286016 25364608 57% /var/www/html/support/downloads
Sun Jul 1 23:01:11 EDT 2018: Ending webInit.sh
2018-07-01 23:04:56:615 Starting client1:1096
2018-07-01 23:04:56:617 Command line: [java, -jar, specweb.jar, -sv, client1, -svp, 9990, -id, 1, -tile, 0, -wkld, 1]
2018-07-01 23:04:56:626 Started client1:1096
Messages from: client1:1096
-> 2018-07-01 23:04:56:733 Looking up SPECvirt controller: client1
-> 2018-07-01 23:04:56:859 masterID: 1, tile: 0, workload: 1
-> 2018-07-01 23:04:56:859 hostname: client1
-> 2018-07-01 23:04:56:929 Sun Jul 01 23:04:56 EDT 2018
-> 2018-07-01 23:04:56:976 RMI server started: client1:9901
-> 2018-07-01 23:04:56:979 SpecwebControl: **** SPECweb2005 benchmark started
-> 2018-07-01 23:04:56:981 SpecwebControl: * Running SPECweb_Support workload
-> 2018-07-01 23:04:56:981 Configuration: Clearing workload.
-> 2018-07-01 23:04:56:984 RemoteLoadGen: Total clients: 1
-> 2018-07-01 23:04:57:006 Connection: [ERROR] IOException creating socket. Exception is:
-> java.net.ConnectException: Connection refused (Connection refused)
-> server name: infraserver; port: 81
-> 2018-07-01 23:04:57:007 Connection: [ERROR] IOException creating socket. Exception is:
-> java.net.ConnectException: Connection refused (Connection refused)
-> server name: infraserver; port: 81
-> 2018-07-01 23:04:57:007 RemoteLoadGen: [ERROR] Unable to successfully initialize workload variables. Terminating.
-> 2018-07-01 23:04:57:007 SpecwebControl: [ERROR] Could not create all client threads.
-> 2018-07-01 23:04:57:007 SpecwebControl: [ERROR] setupWorkload() failed!
-> 2018-07-01 23:04:57:007 SpecwebControl: [ERROR] runTests() failed!
-> 2018-07-01 23:04:57:007 SpecwebControl: [ERROR] Benchmark run failed!
-> 2018-07-01 23:04:57:007 SpecwebControl: Terminating run. Please wait...
[root@client1 SPECvirt]# cat Clientmgr1_1098.out
2018-07-01 23:00:37:698 Creating clientmgr using RMI Registry port 1098
2018-07-01 23:00:37:723 client1:1098 ready...
Sun Jul 1 23:00:57 EDT 2018: Starting jAppInitRstr.sh (restore database from backup)
Stopping Appserver (will fail if already stopped)
Waiting for the domain to stop ....
Command stop-domain executed successfully.
Stopping Emulator (will fail if already stopped)
Clearing Appserver message queue and log files
Remove postgres transaction logs
Stopping postgresql service: [ OK ]
/var/lib/pgsql/data is missing. Use "service postgresql initdb" to initialize the cluster first.
[FAILED]
Restore the Database
Restore Started...
Sun Jul 1 23:01:27 EDT 2018
Sun Jul 1 23:04:12 EDT 2018
Restore Completed
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/VolGroup-lv_root
17938864 5614536 11406416 33% /
tmpfs 4030348 228 4030120 1% /dev/shm
/dev/vda1 487652 40116 421936 9% /boot
/dev/vdb 30832636 6813064 22446708 24% /dbstore
Truncate largeorder table to clear pending requests from previous runs
TRUNCATE TABLE
DROP TABLE
CREATE TABLE
Starting Appserver
Waiting for spec2004-1 to start .............
Successfully started the domain : spec2004-1
domain Location: /opt/glassfish3/glassfish/domains/spec2004-1
Log File: /opt/glassfish3/glassfish/domains/spec2004-1/logs/server.log
Admin Port: 4848
Command start-domain executed successfully.
Starting Emulator
Tomcat started.
Sun Jul 1 23:04:36 EDT 2018: Ending jAppInitRstr.sh
2018-07-01 23:04:56:615 Starting client1:1098
2018-07-01 23:04:56:617 Command line: [java, -jar, specjapp.jar, -sv, client1, -svp, 9990, -id, 0, -tile, 0, -wkld, 0]
2018-07-01 23:04:56:624 Started client1:1098
Messages from: client1:1098
-> 2018-07-01 23:04:56:725 Looking up SPECvirt controller: client1
-> 2018-07-01 23:04:56:850 masterID: 0, tile: 0, workload: 0
-> 2018-07-01 23:04:56:851 hostname: client1
-> Hostname of prime client: client1
-> 2018-07-01 23:04:56:875 Sun Jul 01 23:00:57 EDT 2018
-> 2018-07-01 23:04:56:899 RMI server started: client1:9900
-> 2018-07-01 23:04:56:899 Total clients: 1
-> 2018-07-01 23:04:56:899 Adding host client1:1091
-> 2018-07-01 23:04:56:913 Setting up clients...
-> 2018-07-01 23:04:57:654 calling getHostVM() on jappclient...
-> 2018-07-01 23:04:59:275 Remote exception in setup() from client1:1091
-> java.rmi.UnmarshalException: Error unmarshaling return header; nested exception is:
-> java.io.EOFException
-> 2018-07-01 23:04:59:276 Aborting...
-> 2018-07-01 23:04:59:283 Exception occurred calling specvirt controller's abortTest() method. Exception was java.rmi.ConnectException: Connection refused to host: 192.168.1.107; nested exception is:
-> java.net.ConnectException: Connection refused (Connection refused)
-> 2018-07-01 23:04:59:284 clientsRunning = true
-> 2018-07-01 23:04:59:284 calling client.exit()...
-
I see there are "Unknown host:" error in infraserver, webserver and dbserver.
but /etc/hosts seems ok in each VM
[root@infraserver downloads]# cat /tmp/pollme.out
Creating RMI listener using RMI Registry port 8001
[ERROR] RMI server bind fails: java.rmi.UnknownHostException: Unknown host: infraserver1-int-int; nested exception is:
java.net.UnknownHostException: infraserver1-int-int
java.rmi.UnknownHostException: Unknown host: infraserver1-int-int; nested exception is:
java.net.UnknownHostException: infraserver1-int-int
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:616)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:338)
at sun.rmi.registry.RegistryImpl_Stub.rebind(RegistryImpl_Stub.java:147)
at java.rmi.Naming.rebind(Naming.java:177)
at org.spec.specpoll.pollme.main(pollme.java:195)
Caused by: java.net.UnknownHostException: infraserver1-int-int
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at java.net.Socket.<init>(Socket.java:434)
at java.net.Socket.<init>(Socket.java:211)
at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:148)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
... 6 more
[root@webserver Desktop]# cat /tmp/pollme.out
Creating RMI listener using RMI Registry port 8001
[ERROR] RMI server bind fails: java.rmi.UnknownHostException: Unknown host: webserver1-int-int; nested exception is:
java.net.UnknownHostException: webserver1-int-int
java.rmi.UnknownHostException: Unknown host: webserver1-int-int; nested exception is:
java.net.UnknownHostException: webserver1-int-int
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:616)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:338)
at sun.rmi.registry.RegistryImpl_Stub.rebind(RegistryImpl_Stub.java:147)
at java.rmi.Naming.rebind(Naming.java:177)
at org.spec.specpoll.pollme.main(pollme.java:195)
Caused by: java.net.UnknownHostException: webserver1-int-int
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at java.net.Socket.<init>(Socket.java:434)
at java.net.Socket.<init>(Socket.java:211)
at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:148)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
... 6 more
[root@dbserver Desktop]# cat /tmp/pollme.out
[ERROR] RMI server failure:
Exception: java.rmi.ser[ERROR] RMI server bind fails: java.rmi.UnknownHostException: Unknown host: dbserver-int; nested exception is:
java.net.UnknownHostException: dbserver-int
java.rmi.UnknownHostException: Unknown host: dbserver-int; nested exception is:
java.net.UnknownHostException: dbserver-int
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:616)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:338)
at sun.rmi.registry.RegistryImpl_Stub.rebind(RegistryImpl_Stub.java:147)
at java.rmi.Naming.rebind(Naming.java:177)
at org.spec.specpoll.pollme.main(pollme.java:195)
Caused by: java.net.UnknownHostException: dbserver-int
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at java.net.Socket.<init>(Socket.java:434)
at java.net.Socket.<init>(Socket.java:211)
at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:148)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
... 6 more
[root@infraserver downloads]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
##
# Defaults used for the SPECvirt_sc2013 Example VM Setup Guide.
# External VM-to-client communications
192.168.1.146 infraserver infraserver1
192.168.1.130 webserver webserver1
192.168.1.195 mailserver mailserver1
192.168.1.141 appserver appserver1 specdelivery specemulator
192.168.1.196 dbserver dbserver1
192.168.1.125 batchserver batchserver1
192.168.1.107 client1 specdriver
# Internal VM-to-VM only communications
192.168.1.177 infraserver1-int
192.168.1.131 webserver1-int
192.168.1.150 appserver1-int
192.168.1.164 dbserver1-int specdb
-
Hi Johnnyli,
1. Can u please install JAVA 7 on all VMs.
2. I dont understand why the hostname is " infraserver1-int-int" when its supposed to be "infraserver1-int" and same is the case on webserver. Please ensure that u have correct entries of all VMs (IPs & hostnames) in /etc/hosts on all VMs.
[ERROR] RMI server bind fails: java.rmi.UnknownHostException: Unknown host: webserver1-int-int;
[ERROR] RMI server bind fails: java.rmi.UnknownHostException: Unknown host: infraserver1-int-int; nested exception is:
3. Reboot all the vms & use the helper script timesynctiles.sh to make sure that all Vms are time syced.
4. Also you might want to try just running the following command by itself especially on infraserver1-int & webserver1-int and see if it works fine
java -jar /opt/SPECpoll/pollme.jar -n `hostname`-int -p 8001
5. let DEBUG_LEVEL be 5 and Re-run
Thanks,
Ramya
-
I'm rebuilding all VMs, will back to you once done. Thanks
Some quick questions
1, In infraserver, as i test, I must run ./setup_files.sh before ./make_infraserver.sh vdb, other wise there are many errors. It's different order in the guide. Am i right?
2, How to do mailserver warmup, should i do the warmup after all VMs setup done?
Mailserver warmup
Use the harness to run a test using mailserver and populate the indexes, creating a "warmed up" mailstore. See Section 5 for running a test.
Dovecot requires an initial "mailserver warmup run" with a duration of one to two hours depending on your storage subsystem latencies, CPU performance, and so on. You may see significant IMAP timeout errors in the Clientmgr_1088.out log during the first 20 or 30 minutes of the initial warmup run, and you can ignore these. Wait at least 30 minutes after all IMAP timeouts disappear in the IMAP server log, then stop the test, back up the existing mailstore, and use this for all future runs.
3, Do I need to do mailserver backup for 1 tile test?
Mailserver backup
After warming up the mailstore, issue the following to back it up:
cd /opt/other_scripts
./backupmailstore.sh
You can copy and use this "warmed up" mailstore backup on the other tiles; that is, you do not need to create a backup mailstore on the other mailservers.
-
rollback to java1.7 and rebuild all VMs, still failed to run runspecvirt.sh
No transaction is detected by pollme in infraserver1-int and webserver1-int
All log and config files are attached.
2018-07-04 07:01:35:282 PrimeControl: stopping prime client client1:1092
2018-07-04 07:01:35:282 calling control.stop()...
2018-07-04 07:01:35:282 PrimeControl: terminating run. Please wait...
2018-07-04 07:01:35:282 PrimeControl: got job: 4; id=2
2018-07-04 07:01:35:283 PrimeControl: switch: jobNumber=4; id=2
2018-07-04 07:01:35:283 PrimeControl: stopping prime client client1:1094
2018-07-04 07:01:35:286 specvirt: benchmark run failed!
2018-07-04 07:01:35:286 specvirt: Done!
[root@infraserver infravm_scripts]# hostname infraserver1
[root@infraserver infravm_scripts]# java -jar /opt/SPECpoll/pollme.jar -n `hostname`-int -p 8001
Creating RMI listener using RMI Registry port 8001
infraserver1-int/192.168.1.148:8001 ready...
[root@webserver webvm_scripts]# hostname webserver1
[root@webserver webvm_scripts]# java -jar /opt/SPECpoll/pollme.jar -n `hostname`-int -web -p 8001
Creating RMI listener using RMI Registry port 8001
webserver1-int/192.168.1.149:8001 ready...
-
Reboot all VM, set NUM_WORKLOADS = 1 in Control.config, everthing seems good. Please have a look.
[root@client1 SPECvirt]# tail -50 primectrl.out
0,0,2018-07-04 09:58:53:329,98754,7.1,98694,3.4,197226,8.6,293946,1.75
0,0,2018-07-04 09:59:03:330,98869,7.1,98807,3.4,197431,8.6,294264,1.75
0,0,2018-07-04 09:59:13:328,98976,7.1,98926,3.4,197645,8.6,294558,1.75
0,0,2018-07-04 09:59:23:329,99086,7.1,99043,3.4,197857,8.6,294868,1.75
0,0,2018-07-04 09:59:33:328,99203,7.1,99157,3.4,198083,8.6,295189,1.75
0,0,2018-07-04 09:59:43:329,99319,7.1,99289,3.4,198332,8.6,295509,1.75
0,0,2018-07-04 09:59:53:328,99449,7.1,99412,3.4,198557,8.6,295843,1.75
0,0,2018-07-04 10:00:03:328,99566,7.1,99527,3.4,198836,8.6,296299,1.75
0,0,2018-07-04 10:00:13:327,99717,7.1,99683,3.4,199150,8.6,296746,1.75
0,0,2018-07-04 10:00:23:327,99869,7.1,99849,3.4,199498,8.6,297209,1.75
0,0,2018-07-04 10:00:33:329,100037,7.1,100026,3.4,199800,8.6,297660,1.75
0,0,2018-07-04 10:00:43:328,100288,7.1,100255,3.4,200283,8.6,298545,1.75
0,0,2018-07-04 10:00:53:328,100555,7.1,100563,3.4,200918,8.6,299471,1.75
0,0,2018-07-04 10:01:03:328,100864,7.0,100916,3.4,201557,8.6,300378,1.75
0,0,2018-07-04 10:01:13:328,101210,7.0,101267,3.4,202254,8.6,301319,1.75
0,0,2018-07-04 10:01:23:328,101566,7.0,101645,3.4,202920,8.6,302171,1.75
0,0,2018-07-04 10:01:33:328,101863,7.0,101956,3.4,203552,8.6,303007,1.75
0,0,2018-07-04 10:01:43:328,102196,7.0,102294,3.4,204166,8.6,303828,1.75
0,0,2018-07-04 10:01:53:328,102533,7.0,102610,3.4,204813,8.6,304635,1.75
0,0,2018-07-04 10:02:03:328,102855,7.0,102910,3.4,205447,8.5,305431,1.75
0,0,2018-07-04 10:02:13:328,103136,7.0,103207,3.4,206023,8.5,306180,1.75
0,0,2018-07-04 10:02:23:329,103425,6.9,103483,3.3,206595,8.5,306944,1.75
0,0,2018-07-04 10:02:33:329,103727,6.9,103758,3.3,207213,8.5,307745,1.75
0,0,2018-07-04 10:02:43:328,103973,6.9,104017,3.3,207731,8.5,308340,1.75
0,0,2018-07-04 10:02:53:328,104217,6.9,104262,3.3,208206,8.5,308972,1.75
[root@dbserver ~]# cat /tmp/pollme.out
Creating RMI listener using RMI Registry port 8001
dbserver/192.168.1.178:8001 ready...
dbserver: 1530707072081
[root@appserver ~]# cat /tmp/pollme.out
Creating RMI listener using RMI Registry port 8001
appserver/192.168.1.151:8001 ready...
2018-07-04 08:24:31:825 contacting host 'specdb'...
appserver: 1530707071873
specdb: 1530707072081
-
set NUM_WORKLOADS = 1, all the test passed. Detail results and logs are attached
[root@client1 SPECvirt]# ./pollmecheck.sh
Tile 1 VMs:
dbserver1:
root 2317 1 0 00:08 ? 00:00:00 java -jar /opt/SPECpoll/pollme.jar -n dbserver1-int -p 8001
appserver1:
root 2069 1 0 00:08 ? 00:00:00 java -jar /opt/SPECpoll/pollme.jar -n appserver -app -p 8001
batchserver1:
root 1963 1 0 00:08 ? 00:00:00 java -jar /opt/SPECpoll/pollme.jar -n batchserver -p 8001
mailserver1:
root 1970 1 0 00:08 ? 00:00:00 java -jar /opt/SPECpoll/pollme.jar -n mailserver -p 8001
infraserver1:
root 2049 1 0 00:08 ? 00:00:00 java -jar /opt/SPECpoll/pollme.jar -n infraserver1-int -p 8001
webserver1:
root 2097 1 0 00:10 ? 00:00:00 java -jar /opt/SPECpoll/pollme.jar -n webserver1-int -web -p 8001
[root@client1 SPECvirt]# tail -f primectrl.out
2018-07-05 02:39:10:532 PrimeControl: clients in runtime decremented: 0
2018-07-05 02:39:10:536 PrimeControl: stopping result polling.
2018-07-05 02:39:10:536 PrimeControl: setting job: 2
2018-07-05 02:39:10:536 PrimeControl: waiting for all workloads to stop...
2018-07-05 02:39:10:539 PrimeControl: finished pollMaster().
2018-07-05 02:39:10:539 PrimeControl: getting job... id=0
2018-07-05 02:39:10:539 PrimeControl: got job: 2; id=0
2018-07-05 02:39:10:539 PrimeControl: switch: jobNumber=2; id=0
2018-07-05 02:39:10:539 Sending resetCounters(false) command.
2018-07-05 02:39:10:541 PrimeControl: getting job... id=0
2018-07-05 02:39:13:616 PrimeControl: client1:1098 (PRIME_HOST[0][0]) run complete; numStarted = 0
2018-07-05 02:39:13:617 PrimeControl: calling doneMonitor.notifyAll().
2018-07-05 02:39:20:537 PrimeControl: all workloads stopped
2018-07-05 02:39:20:537 PrimeControl: stopping data reporting.
2018-07-05 02:39:20:537 PrimeControl: setting job: 8
2018-07-05 02:39:20:537 PrimeControl: exiting dataReporter thread.
2018-07-05 02:39:20:537 PrimeControl: got job: 8; id=0
2018-07-05 02:39:20:537 PrimeControl: switch: jobNumber=8; id=0
2018-07-05 02:39:20:538 PrimeControl: calling getValidationRept() ...
2018-07-05 02:39:20:540 No validation errors reported by workloads.
2018-07-05 02:39:20:540 PrimeControl: getting job... id=0
2018-07-05 02:39:20:540 PrimeControl: setting job: 9
2018-07-05 02:39:20:540 PrimeControl: got job: 9; id=0
2018-07-05 02:39:20:540 PrimeControl: switch: jobNumber=9; id=0
2018-07-05 02:39:20:540 PrimeControl: calling getResFiles() ...
2018-07-05 02:39:20:551 PrimeControl: aggregate audit...
2018-07-05 02:39:20:555 PrimeControl: getting job... id=0
2018-07-05 02:39:20:555 PrimeControl: validating aggregate audit...
2018-07-05 02:39:20:556 PrimeControl: setting job: 12
2018-07-05 02:39:20:556 PrimeControl: got job: 12; id=0
2018-07-05 02:39:20:557 PrimeControl: switch: jobNumber=12; id=0
2018-07-05 02:39:20:566 PrimeControl: getting job... id=0
2018-07-05 02:39:20:566 PrimeControl: setting job: 13
2018-07-05 02:39:20:566 PrimeControl: got job: 13; id=0
2018-07-05 02:39:20:566 PrimeControl: switch: jobNumber=13; id=0
2018-07-05 02:39:20:573 PrimeControl: getting job... id=0
2018-07-05 02:39:20:573 PrimeControl: calling primeJob.setJob(4)
2018-07-05 02:39:20:573 PrimeControl: setting job: 4
2018-07-05 02:39:20:573 PrimeControl: got job: 4; id=0
2018-07-05 02:39:20:573 PrimeControl: switch: jobNumber=4; id=0
2018-07-05 02:39:20:573 PrimeControl: stopping prime client client1:1098
2018-07-05 02:39:22:576 PrimeControl: getting job... id=0
2018-07-05 02:39:22:610 PrimeControl: calling primeJob.setJob(11).
2018-07-05 02:39:22:610 PrimeControl: setting job: 11
2018-07-05 02:39:22:610 PrimeControl: got job: 11; id=0
2018-07-05 02:39:22:610 PrimeControl: switch: jobNumber=11; id=0
2018-07-05 02:39:22:610 PrimeControl: getting job... id=0
2018-07-05 02:39:22:610 PrimeControl: calling primeJob.setJob(5)
2018-07-05 02:39:22:610 PrimeControl: setting job: 5
2018-07-05 02:39:22:611 PrimeControl: got job: 5; id=0
2018-07-05 02:39:22:611 PrimeControl: switch: jobNumber=5; id=0
2018-07-05 02:39:22:611 PrimeControl: exiting primeClientThread.
2018-07-05 02:39:22:611 PrimeControl: stopping clients.
2018-07-05 02:39:22:611 PrimeControl: stopping remote client processes
2018-07-05 02:39:22:611 PrimeControl: setting job: 1
2018-07-05 02:39:22:611 PrimeControl: client: waiting for all done...done=0
2018-07-05 02:39:22:611 PrimeControl: got job: 1
2018-07-05 02:39:22:611 PrimeControl: calling client.stopClients() ...
2018-07-05 02:39:24:614 PrimeControl: client: getting job...done=1
2018-07-05 02:39:24:614 PrimeControl: stopping local client threads
2018-07-05 02:39:24:614 PrimeControl: setting job: 2
2018-07-05 02:39:24:614 PrimeControl: client: waiting for all done...done=0
2018-07-05 02:39:24:614 PrimeControl: got job: 2
2018-07-05 02:39:24:614 PrimeControl: exiting clientThread.
2018-07-05 02:39:24:614 PrimeControl: done stopping clients.
> Loading Raw Result File..
2018-07-05 02:39:24:654 calling control.stop()...
2018-07-05 02:39:24:654 PrimeControl: terminating run. Please wait...
2018-07-05 02:39:25:657 specvirt: Done!
[root@appserver ~]# tail -f /tmp/pollme.out
Creating RMI listener using RMI Registry port 8001
appserver/192.168.1.151:8001 ready...
2018-07-05 00:19:09:083 contacting host 'specdb'...
appserver: 1530764349092
specdb: 1530764349295
[root@dbserver ~]# tail -f /tmp/pollme.out
Creating RMI listener using RMI Registry port 8001
dbserver1-int/192.168.1.152:8001 ready...
dbserver1-int: 1530764349295
-
set NUM_WORKLOADS = 2, the test is failed. Detail results and logs are attached, any suggestion?
[root@client1 SPECvirt]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
##
# Defaults used for the SPECvirt_sc2013 Example VM Setup Guide.
# External VM-to-client communications
192.168.1.180 infraserver infraserver1
192.168.1.137 webserver webserver1
192.168.1.161 mailserver mailserver1
192.168.1.151 appserver appserver1 specdelivery specemulator
192.168.1.178 dbserver dbserver1
192.168.1.170 batchserver batchserver1
192.168.1.132 client1 specdriver
# Internal VM-to-VM only communications
192.168.1.145 infraserver1-int
192.168.1.149 webserver1-int
192.168.1.173 appserver1-int
192.168.1.152 dbserver1-int specdb
[root@client1 other_scripts]# ./test_hostentries.sh
Testing the external IP addresses of the Virtual Machines and client
Pinging Infraserver1 VM (infraserver1)...Passed!
Pinging Webserver1 VM (webserver1)...Passed!
Pinging Mailserver1 VM (mailserver1)...Passed!
Pinging Appserver1 VM (specdelivery)...Passed!
Pinging DBserver1 VM (dbserver1)...Passed!
Pinging Batchserver1 VM (batchserver1)...Passed!
Pinging Client1 VM (client1)...Passed!
Testing the internal IP addresses of the Virtual Machines
This does not work on the client since it has no internal
network adapter. Run this test on web/infraserver and app/dbserver.
Pinging Infraserver1 VM (infraserver1-int)...Passed!
Pinging Webserver1 VM (webserver1-int)...Passed!
Pinging Appserver1 VM (appserver1-int)...Passed!
Pinging DBserver VM (specdb)...Passed!
[root@client1 SPECvirt]# ./pollmecheck.sh
Tile 1 VMs:
dbserver1:
root 2392 1 0 03:19 ? 00:00:00 java -jar /opt/SPECpoll/pollme.jar -n dbserver1-int -p 8001
appserver1:
root 2117 1 0 03:19 ? 00:00:00 java -jar /opt/SPECpoll/pollme.jar -n appserver -app -p 8001
batchserver1:
root 1963 1 0 03:19 ? 00:00:00 java -jar /opt/SPECpoll/pollme.jar -n batchserver -p 8001
mailserver1:
root 1968 1 0 03:21 ? 00:00:00 java -jar /opt/SPECpoll/pollme.jar -n mailserver -p 8001
infraserver1:
root 2050 1 0 03:19 ? 00:00:00 java -jar /opt/SPECpoll/pollme.jar -n infraserver1-int -p 8001
webserver1:
root 2175 1 0 03:20 ? 00:00:00 java -jar /opt/SPECpoll/pollme.jar -n webserver1-int -web -p 8001
[root@client1 SPECvirt]# tail -f primectrl.out
2018-07-05 03:25:20:681 PrimeControl: entered startMasters(); remoteMasters.length = 2
2018-07-05 03:25:20:682 PrimeControl: getting job... id=0
2018-07-05 03:25:20:684 PrimeControl: calling primeJob.setJob(10).
2018-07-05 03:25:20:684 PrimeControl: setting job: 10
2018-07-05 03:25:20:684 PrimeControl: getting job... id=1
2018-07-05 03:25:20:684 PrimeControl: got job: 10; id=1
2018-07-05 03:25:20:684 PrimeControl: switch: jobNumber=10; id=1
2018-07-05 03:25:20:685 PrimeControl: got job: 10; id=0
2018-07-05 03:25:20:685 PrimeControl: switch: jobNumber=10; id=0
2018-07-05 03:25:34:379 PrimeControl: getting job... id=1
2018-07-05 03:28:17:260 PrimeControl: getting job... id=0
2018-07-05 03:28:17:260 PrimeControl: calling primeJob.setJob(0).
2018-07-05 03:28:17:260 PrimeControl: setting job: 0
2018-07-05 03:28:17:261 PrimeControl: got job: 0; id=0
2018-07-05 03:28:17:261 PrimeControl: switch: jobNumber=0; id=0
2018-07-05 03:28:17:261 PrimeControl: got job: 0; id=1
2018-07-05 03:28:17:261 PrimeControl: master[0][0] sleeping 20 sec.
2018-07-05 03:28:17:261 PrimeControl: switch: jobNumber=0; id=1
2018-07-05 03:28:17:261 PrimeControl: master[0][1] sleeping 20 sec.
2018-07-05 03:28:37:261 PrimeControl: calling startMaster: client1:1098 for tile 0, benchmark 0
2018-07-05 03:28:37:262 PrimeControl: calling startMaster: client1:1096 for tile 0, benchmark 1
2018-07-05 03:28:37:270 PrimeControl: getting job... id=0
2018-07-05 03:28:37:271 PrimeControl: getting job... id=1
2018-07-05 03:28:37:271 PrimeControl: waiting on 2 prime client(s).
2018-07-05 03:28:37:420 Sending config to client1:1098
2018-07-05 03:28:37:452 Sending config to client1:1096
2018-07-05 03:28:37:454 specvirt: primeRmiStarted = 1; id=0
2018-07-05 03:28:37:543 specvirt: primeRmiStarted = 2; id=1
.
2018-07-05 03:28:38:272 PrimeControl: calling primeJob.setJob(1)
2018-07-05 03:28:38:272 PrimeControl: setting job: 1
2018-07-05 03:28:38:272 PrimeControl: got job: 1; id=1
2018-07-05 03:28:38:272 PrimeControl: switch: jobNumber=1; id=1
2018-07-05 03:28:38:272 PrimeControl: got job: 1; id=0
2018-07-05 03:28:38:273 PrimeControl: switch: jobNumber=1; id=0
2018-07-05 03:28:38:273 PrimeControl: Naming lookup for 192.168.1.132:9901
2018-07-05 03:28:38:273 PrimeControl: Naming lookup for 192.168.1.132:9900
2018-07-05 03:28:38:285 PrimeControl: done in Naming lookup for 192.168.1.132:9900
2018-07-05 03:28:38:285 PrimeControl: calling getHostVMs()...
2018-07-05 03:28:38:288 PrimeControl: hostVMs[0] = specdelivery:8001,specdb:8001
2018-07-05 03:28:38:289 PrimeControl: calling getBuildNumber()...
2018-07-05 03:28:40:275 RMI name lookup error for prime client 192.168.1.132 on port 9901: exception java.rmi.ConnectException: Connection refused to host: 192.168.1.132; nested exception is:
java.net.ConnectException: Connection refused (Connection refused)
2018-07-05 03:28:40:275 PrimeControl: done in Naming lookup for 192.168.1.132:9901
2018-07-05 03:28:40:275 PrimeControl: calling getHostVMs()...
2018-07-05 03:28:40:275 PrimeControl: [ERROR] masters[0] build numbers (null) do not match the specvirt prime controller's (80). Please update complete harness and retry.
2018-07-05 03:28:40:276 PrimeControl: [ERROR] masters[1] build numbers (null) do not match the specvirt prime controller's (80). Please update complete harness and retry.
2018-07-05 03:28:40:276 PrimeControl: [ERROR] startMasters() failed!
2018-07-05 03:28:40:276 PrimeControl: sending abortTest() to prime clients.
2018-07-05 03:28:40:276 PrimeControl: setting job: 6
Exception in thread "Thread-6" java.lang.NullPointerException
at org.spec.virt.PrimeControl$PrimeClientThread.run(PrimeControl.java:1650)
2018-07-05 03:30:52:981 PrimeControl: buildNum[0] = 80
2018-07-05 03:30:52:981 PrimeControl: getting job... id=0
2018-07-05 03:30:52:981 PrimeControl: got job: 6; id=0
2018-07-05 03:30:52:981 PrimeControl: switch: jobNumber=6; id=0
2018-07-05 03:30:52:981 PrimeControl: id=0, abortID=-1
2018-07-05 03:30:52:981 PrimeControl: masters[0]=client1:1098
2018-07-05 03:31:01:651 specvirt: waiting on 1 prime clients.
2018-07-05 03:31:03:299 PrimeControl: getting job... id=0
[root@appserver ~]# tail -f /tmp/pollme.out
Creating RMI listener using RMI Registry port 8001
appserver/192.168.1.151:8001 ready...
2018-07-05 03:31:01:196 contacting host 'specdb'...
appserver: 1530775861209
specdb: 1530775861418
[root@dbserver ~]# tail -f /tmp/pollme.out
Creating RMI listener using RMI Registry port 8001
dbserver1-int/192.168.1.152:8001 ready...
dbserver1-int: 1530775861418
[root@webserver ~]# tail -f /tmp/pollme.out
Creating RMI listener using RMI Registry port 8001
webserver1-int/192.168.1.149:8001 ready...
-
Hi Johnnyli,
The Connection refused errors are usually due to firewall, SELinux or SPECpoll not running. I see that u have all of these in place.Can u confirm if webserver & infraserver can ping each other via internal network? are you using Java JDK 7 on all VMs?
From the prime client, can u please check the specvirt build version of all workload VMs, clients, and prime client as follows:
java -jar /opt/SPECvirt/specvirt.jar -v
U should see the following :
SPECvirt_sc2013 v1.1, build: 80
if u don't, you need to make sure you've installed SPECvirt correctly on all VMs.
Also, are you using 10GbE between the clients and SUT? One tile uses about 1.4GbE per tile. If you don't have 10GbE, you can split off webserver onto its own client.
Can u also attach httpd error & access logs from webserver & infraserver?
Thanks,
Ramya
-
Hi Ramya,
I confirmed again in all VMs, firewall and SELinux are disabled, JAVA is 1,7.0_181, specvirt version is SPECvirt_sc2013 v1.1, build: 80.
There are httpd error & access logs in webserver, but not in infraserver
Do I need to set $JAVA_HOME in all VMs? I see $JAVA_HOME is /usr/lib/jvm/java-openjdk in appserver and client1, is /usr/lib/jvm/jre in dbserver, and is empty in the rest.
I don't use 10GbE LAN card, I'll find one.
[root@webserver logs]# service iptables status
iptables: Firewall is not running.
[root@webserver logs]# service ip6tables status
ip6tables: Firewall is not running.
[root@webserver logs]# cat /proc/cmdline
ro root=/dev/mapper/VolGroup-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_NO_MD rd_LVM_LV=VolGroup/lv_swap SYSFONT=latarcyrheb-sun16 crashkernel=129M@0M rd_LVM_LV=VolGroup/lv_root KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet selinux=0 audit=0
[root@webserver logs]# java -version
java version "1.7.0_181"
OpenJDK Runtime Environment (rhel-2.6.14.8.el6_9-x86_64 u181-b00)
OpenJDK 64-Bit Server VM (build 24.181-b00, mixed mode)
[root@webserver logs]# java -jar /opt/SPECvirt/specvirt.jar -v
SPECvirt_sc2013 v1.1, build: 80
[root@webserver logs]# ping infraserver
PING infraserver (192.168.1.180) 56(84) bytes of data.
64 bytes from infraserver (192.168.1.180): icmp_seq=1 ttl=64 time=1.42 ms
^C
--- infraserver ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 959ms
rtt min/avg/max/mdev = 1.423/1.423/1.423/0.000 ms
[root@webserver logs]# ping infraserver1-int
PING infraserver1-int (192.168.1.145) 56(84) bytes of data.
64 bytes from infraserver1-int (192.168.1.145): icmp_seq=1 ttl=64 time=0.998 ms
64 bytes from infraserver1-int (192.168.1.145): icmp_seq=2 ttl=64 time=0.372 ms
64 bytes from infraserver1-int (192.168.1.145): icmp_seq=3 ttl=64 time=0.360 ms
^C
--- infraserver1-int ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2192ms
rtt min/avg/max/mdev = 0.360/0.576/0.998/0.299 ms
[root@webserver ~]# java -jar /opt/SPECvirt/specvirt.jar -v
SPECvirt_sc2013 v1.1, build: 80
[root@infraserver logs]# cd /etc/httpd/logs/
[root@infraserver logs]# ls
[root@webserver ~]# cd /etc/httpd/logs/
[root@webserver logs]# ls
access_log error_log ssl_access_log ssl_error_log ssl_request_log
[root@webserver logs]# cat access_log
192.168.1.132 - - [05/Jul/2018:03:28:37 -0400] "GET /support/init.php?BESIM_HOST=infraserver&BESIM_PORT=81&BESIM_URI=/fcgi-bin/besim_fcgi.fcgi&BESIM_PERSISTENT=false&PADDING_DIR=/var/www/html/support/dynamic_padding/&SMARTY_DIR=/var/www/html/Smarty-2.6.26/libs/&SMARTY_SUPPORT_DIR=/var/www/html/support/&SEND_CONTENT_LENGTH=true HTTP/1.1" 200 537 "-" "-"
[root@webserver logs]# cat error_log
[Thu Jul 05 03:25:33 2018] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu Jul 05 03:25:34 2018] [notice] Digest: generating secret for digest authentication ...
[Thu Jul 05 03:25:34 2018] [notice] Digest: done
[Thu Jul 05 03:25:34 2018] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.3.3 mod_ssl/2.2.15 OpenSSL/1.0.1e-fips configured -- resuming normal operations
[root@webserver logs]# cat ssl_access_log
[root@webserver logs]# cat ssl_error_log
[Thu Jul 05 03:25:33 2018] [warn] RSA server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[Thu Jul 05 03:25:34 2018] [warn] RSA server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[root@webserver logs]# cat ssl_request_log
-
Use a 10Gb card for internal network, issue is the same.
[root@client1 ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
##
# Defaults used for the SPECvirt_sc2013 Example VM Setup Guide.
# External VM-to-client communications
192.168.1.180 infraserver infraserver1
192.168.1.137 webserver webserver1
192.168.1.161 mailserver mailserver1
192.168.1.151 appserver appserver1 specdelivery specemulator
192.168.1.178 dbserver dbserver1
192.168.1.170 batchserver batchserver1
192.168.1.132 client1 specdriver
# Internal VM-to-VM only communications
10.10.10.36 infraserver1-int
10.10.10.37 webserver1-int
10.10.10.38 appserver1-int
10.10.10.39 dbserver1-int specdb
[root@client1 other_scripts]# ./test_hostentries.sh
Testing the external IP addresses of the Virtual Machines and client
Pinging Infraserver1 VM (infraserver1)...Passed!
Pinging Webserver1 VM (webserver1)...Passed!
Pinging Mailserver1 VM (mailserver1)...Passed!
Pinging Appserver1 VM (specdelivery)...Passed!
Pinging DBserver1 VM (dbserver1)...Passed!
Pinging Batchserver1 VM (batchserver1)...Passed!
Pinging Client1 VM (client1)...Passed!
Testing the internal IP addresses of the Virtual Machines
This does not work on the client since it has no internal
network adapter. Run this test on web/infraserver and app/dbserver.
Pinging Infraserver1 VM (infraserver1-int)...FAILED
Pinging Webserver1 VM (webserver1-int)...FAILED
Pinging Appserver1 VM (appserver1-int)...FAILED
Pinging DBserver VM (specdb)...FAILED
[root@webserver other_scripts]# ./test_hostentries.sh
Testing the external IP addresses of the Virtual Machines and client
Pinging Infraserver1 VM (infraserver1)...Passed!
Pinging Webserver1 VM (webserver1)...Passed!
Pinging Mailserver1 VM (mailserver1)...Passed!
Pinging Appserver1 VM (specdelivery)...Passed!
Pinging DBserver1 VM (dbserver1)...Passed!
Pinging Batchserver1 VM (batchserver1)...Passed!
Pinging Client1 VM (client1)...Passed!
Testing the internal IP addresses of the Virtual Machines
This does not work on the client since it has no internal
network adapter. Run this test on web/infraserver and app/dbserver.
Pinging Infraserver1 VM (infraserver1-int)...Passed!
Pinging Webserver1 VM (webserver1-int)...Passed!
Pinging Appserver1 VM (appserver1-int)...Passed!
Pinging DBserver VM (specdb)...Passed!
[root@client1 SPECvirt]# tail -f primectrl.out
2018-07-05 23:48:07:925 PrimeControl: entered startMasters(); remoteMasters.length = 2
2018-07-05 23:48:07:927 PrimeControl: getting job... id=0
2018-07-05 23:48:07:927 PrimeControl: getting job... id=1
2018-07-05 23:48:07:929 PrimeControl: calling primeJob.setJob(10).
2018-07-05 23:48:07:929 PrimeControl: setting job: 10
2018-07-05 23:48:07:929 PrimeControl: got job: 10; id=1
2018-07-05 23:48:07:929 PrimeControl: got job: 10; id=0
2018-07-05 23:48:07:930 PrimeControl: switch: jobNumber=10; id=1
2018-07-05 23:48:07:930 PrimeControl: switch: jobNumber=10; id=0
2018-07-05 23:48:21:689 PrimeControl: getting job... id=1
2018-07-05 23:51:18:642 PrimeControl: getting job... id=0
2018-07-05 23:51:18:643 PrimeControl: calling primeJob.setJob(0).
2018-07-05 23:51:18:643 PrimeControl: setting job: 0
2018-07-05 23:51:18:643 PrimeControl: got job: 0; id=0
2018-07-05 23:51:18:643 PrimeControl: switch: jobNumber=0; id=0
2018-07-05 23:51:18:643 PrimeControl: got job: 0; id=1
2018-07-05 23:51:18:643 PrimeControl: master[0][0] sleeping 20 sec.
2018-07-05 23:51:18:643 PrimeControl: switch: jobNumber=0; id=1
2018-07-05 23:51:18:644 PrimeControl: master[0][1] sleeping 20 sec.
2018-07-05 23:51:38:644 PrimeControl: calling startMaster: client1:1098 for tile 0, benchmark 0
2018-07-05 23:51:38:644 PrimeControl: calling startMaster: client1:1096 for tile 0, benchmark 1
2018-07-05 23:51:38:652 PrimeControl: getting job... id=0
2018-07-05 23:51:38:655 PrimeControl: getting job... id=1
2018-07-05 23:51:38:655 PrimeControl: waiting on 2 prime client(s).
2018-07-05 23:51:38:830 Sending config to client1:1098
2018-07-05 23:51:38:885 specvirt: primeRmiStarted = 1; id=0
2018-07-05 23:51:38:908 Sending config to client1:1096
2018-07-05 23:51:38:996 specvirt: primeRmiStarted = 2; id=1
.
2018-07-05 23:51:39:656 PrimeControl: calling primeJob.setJob(1)
2018-07-05 23:51:39:656 PrimeControl: setting job: 1
2018-07-05 23:51:39:656 PrimeControl: got job: 1; id=1
2018-07-05 23:51:39:656 PrimeControl: switch: jobNumber=1; id=1
2018-07-05 23:51:39:656 PrimeControl: got job: 1; id=0
2018-07-05 23:51:39:656 PrimeControl: Naming lookup for 192.168.1.132:9901
2018-07-05 23:51:39:657 PrimeControl: switch: jobNumber=1; id=0
2018-07-05 23:51:39:657 PrimeControl: Naming lookup for 192.168.1.132:9900
2018-07-05 23:51:39:669 PrimeControl: done in Naming lookup for 192.168.1.132:9900
2018-07-05 23:51:39:669 PrimeControl: calling getHostVMs()...
2018-07-05 23:51:39:672 PrimeControl: hostVMs[0] = specdelivery:8001,specdb:8001
2018-07-05 23:51:39:672 PrimeControl: calling getBuildNumber()...
2018-07-05 23:51:41:684 RMI name lookup error for prime client 192.168.1.132 on port 9901: exception java.rmi.ConnectException: Connection refused to host: 192.168.1.132; nested exception is:
java.net.ConnectException: Connection refused (Connection refused)
2018-07-05 23:51:41:684 PrimeControl: done in Naming lookup for 192.168.1.132:9901
2018-07-05 23:51:41:684 PrimeControl: calling getHostVMs()...
2018-07-05 23:51:41:684 PrimeControl: [ERROR] masters[0] build numbers (null) do not match the specvirt prime controller's (80). Please update complete harness and retry.
2018-07-05 23:51:41:685 PrimeControl: [ERROR] masters[1] build numbers (null) do not match the specvirt prime controller's (80). Please update complete harness and retry.
2018-07-05 23:51:41:685 PrimeControl: [ERROR] startMasters() failed!
2018-07-05 23:51:41:685 PrimeControl: sending abortTest() to prime clients.
2018-07-05 23:51:41:685 PrimeControl: setting job: 6
Exception in thread "Thread-6" java.lang.NullPointerException
at org.spec.virt.PrimeControl$PrimeClientThread.run(PrimeControl.java:1650)
2018-07-05 23:53:54:928 PrimeControl: buildNum[0] = 80
2018-07-05 23:53:54:928 PrimeControl: getting job... id=0
2018-07-05 23:53:54:928 PrimeControl: got job: 6; id=0
2018-07-05 23:53:54:929 PrimeControl: switch: jobNumber=6; id=0
2018-07-05 23:53:54:929 PrimeControl: id=0, abortID=-1
2018-07-05 23:53:54:929 PrimeControl: masters[0]=client1:1098
2018-07-05 23:54:05:683 specvirt: waiting on 1 prime clients.
2018-07-05 23:54:07:329 PrimeControl: getting job... id=0
-
Hi Johnnyli,
Can you check if your infraserver nfs share is mounted on webserver before you ran Wafgen?
On my specvirt_sc setup, I have JAVA_HOME set to /usr/lib/jvm/jre. Can u change this and retry?
The connection refused errors you're seeing are probably because of some kind of network misconfiguration. From the details u shared so far, everything looks good to me. Can u also post ifconfig output from all your VMs here?
Thanks,
Ramya
-
Rebuild infraserver and webserver, confirmed infraserver1-int:/home/webfiles/downloads is mounted before Wafgen.
All ifconfig commands are dumped in attached.
This time the error is different. Don't it's better or worse than before.
Which JAVA_HOME need to set to /usr/lib/jvm/jre, All? I don't do this now.
[root@client1 SPECvirt]# cat primectrl.out
2018-07-09 04:35:46:588 Mon Jul 09 04:35:46 EDT 2018
2018-07-09 04:35:46:588 specvirt: maxPreRunTime = 481
2018-07-09 04:35:46:589 specvirt: runTime = 7200
2018-07-09 04:35:46:589 specvirt: runTime = 7200
2018-07-09 04:35:46:589 specvirt: runTime = 600
2018-07-09 04:35:46:589 specvirt: runTime = 600
2018-07-09 04:35:46:590 Validator: [WARNING] NUM_WORKLOADS value is: 2; should be 4
2018-07-09 04:35:46:591 Validator: [WARNING] Non-compliant configuration.
2018-07-09 04:35:46:591 [WARNING] This will be a non-compliant benchmark result!
2018-07-09 04:35:46:634 RMI server started: client1:9990
2018-07-09 04:35:46:641 modVals: [Ljava.lang.String;@41cb0ed6
2018-07-09 04:35:46:642 [INFO] This is a perf-only benchmark run. Skipping active idle polling interval.
2018-07-09 04:35:46:643 PrimeControl: preparing client drivers.
2018-07-09 04:35:46:643 PrimeControl: PRIME_HOST 0 = client1:1098
2018-07-09 04:35:46:643 PrimeControl: PRIME_HOST 1 = client1:1096
2018-07-09 04:35:46:644 PrimeControl: Master 1: client1:1098
2018-07-09 04:35:46:644 PrimeControl: Master 2: client1:1096
2018-07-09 04:35:46:645 PrimeControl: adding host client1:1098
2018-07-09 04:35:46:654 PrimeControl: adding host client1:1096
2018-07-09 04:35:46:665 First client for 0: 192.168.1.132:1091
client[0][0]: 192.168.1.132:1091
remote client "192.168.1.132" added
2018-07-09 04:35:46:670 PrimeControl: done in client for loop
2018-07-09 04:35:46:677 First client for 1: 192.168.1.132:1010
client[1][0]: 192.168.1.132:1010
2018-07-09 04:35:46:679 PrimeControl: done in client for loop
creating ClientThread-0
2018-07-09 04:35:46:681 PrimeControl: setting job: 0
2018-07-09 04:35:46:681 PrimeControl: client: waiting for all done...done=0
2018-07-09 04:35:46:681 PrimeControl: client: getting job...done=0
2018-07-09 04:35:46:681 PrimeControl: got job: 0
2018-07-09 04:35:46:681 PrimeControl: starting clients...
2018-07-09 04:35:46:681 PrimeControl: clients.length = 2
2018-07-09 04:35:46:681 PrimeControl: clients[0].length = 1
2018-07-09 04:35:46:681 PrimeControl: starting clients[0][0]: 192.168.1.132:1091
2018-07-09 04:35:46:691 PrimeControl: started client: 192.168.1.132:1091
2018-07-09 04:35:46:691 PrimeControl: clients[1].length = 1
2018-07-09 04:35:46:691 PrimeControl: starting clients[1][0]: 192.168.1.132:1010
2018-07-09 04:35:46:696 PrimeControl: started client: 192.168.1.132:1010
2018-07-09 04:35:46:696 PrimeControl: client: getting job...done=1
2018-07-09 04:35:46:696 PrimeControl: done starting clients
2018-07-09 04:35:46:696 PrimeControl: PTDs not used for this benchmark run!
2018-07-09 04:35:46:696 PrimeControl: starting 2 masters.
2018-07-09 04:35:46:697 PrimeControl: entered startMasters(); remoteMasters.length = 2
2018-07-09 04:35:46:698 PrimeControl: getting job... id=0
2018-07-09 04:35:46:699 PrimeControl: calling primeJob.setJob(10).
2018-07-09 04:35:46:699 PrimeControl: setting job: 10
2018-07-09 04:35:46:700 PrimeControl: got job: 10; id=0
2018-07-09 04:35:46:700 PrimeControl: switch: jobNumber=10; id=0
2018-07-09 04:35:46:700 PrimeControl: getting job... id=1
2018-07-09 04:35:46:700 PrimeControl: got job: 10; id=1
2018-07-09 04:35:46:700 PrimeControl: switch: jobNumber=10; id=1
2018-07-09 04:36:00:360 PrimeControl: getting job... id=1
2018-07-09 04:38:45:206 PrimeControl: getting job... id=0
2018-07-09 04:38:45:206 PrimeControl: calling primeJob.setJob(0).
2018-07-09 04:38:45:206 PrimeControl: setting job: 0
2018-07-09 04:38:45:206 PrimeControl: got job: 0; id=0
2018-07-09 04:38:45:207 PrimeControl: switch: jobNumber=0; id=0
2018-07-09 04:38:45:207 PrimeControl: got job: 0; id=1
2018-07-09 04:38:45:207 PrimeControl: master[0][0] sleeping 20 sec.
2018-07-09 04:38:45:207 PrimeControl: switch: jobNumber=0; id=1
2018-07-09 04:38:45:207 PrimeControl: master[0][1] sleeping 20 sec.
2018-07-09 04:39:05:207 PrimeControl: calling startMaster: client1:1098 for tile 0, benchmark 0
2018-07-09 04:39:05:207 PrimeControl: calling startMaster: client1:1096 for tile 0, benchmark 1
2018-07-09 04:39:05:215 PrimeControl: getting job... id=0
2018-07-09 04:39:05:215 PrimeControl: getting job... id=1
2018-07-09 04:39:05:215 PrimeControl: waiting on 2 prime client(s).
2018-07-09 04:39:05:395 Sending config to client1:1096
2018-07-09 04:39:05:400 Sending config to client1:1098
2018-07-09 04:39:05:427 specvirt: primeRmiStarted = 1; id=0
2018-07-09 04:39:05:504 specvirt: primeRmiStarted = 2; id=1
2018-07-09 04:39:05:859 [ERROR] Received abort signal from client1:1096. Terminating.
2018-07-09 04:39:05:859 PrimeControl: sending abortTest() to prime clients.
2018-07-09 04:39:05:859 PrimeControl: setting job: 6
2018-07-09 04:39:05:859 PrimeControl: got job: 6; id=1
2018-07-09 04:39:05:859 PrimeControl: switch: jobNumber=6; id=1
2018-07-09 04:39:05:859 PrimeControl: got job: 6; id=0
2018-07-09 04:39:05:859 PrimeControl: id=1, abortID=1
2018-07-09 04:39:05:860 PrimeControl: switch: jobNumber=6; id=0
2018-07-09 04:39:05:860 PrimeControl: getting job... id=1
2018-07-09 04:39:05:860 PrimeControl: id=0, abortID=1
2018-07-09 04:39:05:860 PrimeControl: getting job... id=0
2018-07-09 04:39:05:860 PrimeControl: stopping remote client processes
2018-07-09 04:39:05:860 PrimeControl: setting job: 1
2018-07-09 04:39:05:860 PrimeControl: client: waiting for all done...done=0
2018-07-09 04:39:05:860 PrimeControl: got job: 1
2018-07-09 04:39:05:860 PrimeControl: calling client.stopClients() ...
.
2018-07-09 04:39:06:215 PrimeControl: calling primeJob.setJob(1)
2018-07-09 04:39:06:216 PrimeControl: setting job: 1
2018-07-09 04:39:06:216 PrimeControl: got job: 1; id=0
2018-07-09 04:39:06:216 PrimeControl: switch: jobNumber=1; id=0
2018-07-09 04:39:06:216 PrimeControl: got job: 1; id=1
2018-07-09 04:39:06:216 PrimeControl: Naming lookup for 192.168.1.132:9900
2018-07-09 04:39:06:216 PrimeControl: switch: jobNumber=1; id=1
2018-07-09 04:39:06:216 PrimeControl: Naming lookup for 192.168.1.132:9901
2018-07-09 04:39:06:228 PrimeControl: done in Naming lookup for 192.168.1.132:9900
2018-07-09 04:39:06:228 PrimeControl: calling getHostVMs()...
2018-07-09 04:39:06:231 PrimeControl: hostVMs[0] = specdelivery:8001,specdb:8001
2018-07-09 04:39:06:231 PrimeControl: calling getBuildNumber()...
2018-07-09 04:39:07:865 PrimeControl: client: getting job...done=1
2018-07-09 04:39:07:865 PrimeControl: stopping local client threads
2018-07-09 04:39:07:865 PrimeControl: setting job: 2
2018-07-09 04:39:07:865 PrimeControl: client: waiting for all done...done=0
2018-07-09 04:39:07:865 PrimeControl: got job: 2
2018-07-09 04:39:07:866 PrimeControl: exiting clientThread.
2018-07-09 04:39:07:866 Terminating client processes
2018-07-09 04:39:07:866 PrimeControl: setting job: 1
2018-07-09 04:39:07:866 Sending termination signal to clientmgr processes
2018-07-09 04:39:07:866 PrimeControl: setting job: 2
2018-07-09 04:39:07:866 Terminating prime client processes
2018-07-09 04:39:07:866 PrimeControl: setting job: 4
2018-07-09 04:39:07:866 Sending termination signal to prime clientmgr processes
2018-07-09 04:39:07:866 PrimeControl: setting job: 5
2018-07-09 04:39:07:866 Exiting...
2018-07-09 04:39:07:866 calling control.stop()...
2018-07-09 04:39:07:867 PrimeControl: terminating run. Please wait...
2018-07-09 04:39:07:869 specvirt: benchmark run failed!
2018-07-09 04:39:07:869 specvirt: Done!
[root@client1 SPECvirt]# cat Clientmgr1_1096.out
2018-07-09 04:35:26:623 Creating clientmgr using RMI Registry port 1096
2018-07-09 04:35:26:658 client1:1096 ready...
Mon Jul 9 04:35:45 EDT 2018: Starting webInit.sh
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/VolGroup-lv_root
17938864 5009552 12011400 30% /
Stopping httpd: [ OK ]
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/VolGroup-lv_root
17938864 5009556 12011396 30% /
tmpfs 510048 72 509976 1% /dev/shm
/dev/vda1 487652 40116 421936 9% /boot
/dev/vdb 51475068 33286072 15567556 69% /home/webfiles/downloads
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/VolGroup-lv_root
17938864 5044960 11975992 30% /
Stopping httpd: [ OK ]
Starting httpd: [ OK ]
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/VolGroup-lv_root
17938864 5044996 11975956 30% /
tmpfs 5064832 72 5064760 1% /dev/shm
/dev/vda1 487652 40116 421936 9% /boot
infraserver1-int:/home/webfiles/downloads
51475072 33286016 15567616 69% /var/www/html/support/downloads
Mon Jul 9 04:35:59 EDT 2018: Ending webInit.sh
2018-07-09 04:39:05:211 Starting client1:1096
2018-07-09 04:39:05:212 Command line: [java, -jar, specweb.jar, -sv, client1, -svp, 9990, -id, 1, -tile, 0, -wkld, 1]
2018-07-09 04:39:05:214 Started client1:1096
Messages from: client1:1096
-> 2018-07-09 04:39:05:314 Looking up SPECvirt controller: client1
-> 2018-07-09 04:39:05:394 masterID: 1, tile: 0, workload: 1
-> 2018-07-09 04:39:05:394 hostname: client1
-> 2018-07-09 04:39:05:490 Mon Jul 09 04:39:05 EDT 2018
-> 2018-07-09 04:39:05:503 RMI server started: client1:9901
-> 2018-07-09 04:39:05:509 SpecwebControl: **** SPECweb2005 benchmark started
-> 2018-07-09 04:39:05:510 SpecwebControl: * Running SPECweb_Support workload
-> 2018-07-09 04:39:05:510 Configuration: Clearing workload.
-> 2018-07-09 04:39:05:512 RemoteLoadGen: Total clients: 1
-> 2018-07-09 04:39:05:535 HttpRequestSched: [ERROR] Valid SERVER_TIME value not provided in header.
-> 2018-07-09 04:39:05:535 HttpRequestSched: [ERROR] Response was:
-> HTTP/1.1 200 OK
-> Date: Mon, 09 Jul 2018 08:39:05 GMT
-> Server: Apache/2.2.15 (CentOS)
-> X-Powered-By: PHP/5.3.3
-> Content-Length: 139
-> Content-Type: text/html; charset=UTF-8
->
-> <html>
-> <head>
-> <title>SPECweb2005 Support Workload Init</title>
-> </head>
-> <body>
-> Error: Could not write to init_vars.php
-> </body>
-> </html>
-> 2018-07-09 04:39:05:535 SPECweb_Support: [ERROR] Error! setServerDate() failed.
-> 2018-07-09 04:39:05:536 RemoteLoadGen: [ERROR] Unable to successfully initialize workload variables. Terminating.
-> 2018-07-09 04:39:05:536 SpecwebControl: [ERROR] Could not create all client threads.
-> 2018-07-09 04:39:05:536 SpecwebControl: [ERROR] setupWorkload() failed!
-> 2018-07-09 04:39:05:536 SpecwebControl: [ERROR] runTests() failed!
-> 2018-07-09 04:39:05:536 SpecwebControl: [ERROR] Benchmark run failed!
-> 2018-07-09 04:39:05:538 SpecwebControl: Terminating run. Please wait...
-
JAVA_HOME on client1, set it to set to /usr/lib/jvm/jre.
-Ramya