Mailserver load generation failed for ExampleVM

[tdeneau]

Trying to follow the load generation steps for a mailserver example VM.
    nohup ./run_load_gen.sh
    ./run_init.sh

What does the following error indicate?


2016-11-18 17:10:14:493 Waiting for Message CLEAN to terminate...
__ratelimit: 72 callbacks suppressed
type=1400 audit(1479510616.019:187): avc:  denied  { write } for  pid=2507 comm="imap" name="maildir" dev=vdb ino=131074 scontext=system_u:sy\
stem_r:dovecot_t:s0 tcontext=unconfined_u:object_r:file_t:s0 tclass=dir
type=1400 audit(1479510616.021:188): avc:  denied  { write } for  pid=2507 comm="imap" name="maildir" dev=vdb ino=131074 scontext=system_u:sy\
stem_r:dovecot_t:s0 tcontext=unconfined_u:object_r:file_t:s0 tclass=dir
2016-11-18 17:10:16:346 Error: ******************** Remote exception java.rmi.UnmarshalException: Error unmarshaling return header; nested ex\
ception is:
    java.io.EOFException
java.rmi.UnmarshalException: Error unmarshaling return header; nested exception is:
    java.io.EOFException
    at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:228)
    at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:161)
    at org.spec.specimap.specimapclient_Stub.waitComplete(Unknown Source)
    at org.spec.specimap.RemoteLoadGenerators.waitComplete(RemoteLoadGenerators.java:312)
    at org.spec.specimap.SpecimapControl.doMessageClean(SpecimapControl.java:1042)
    at org.spec.specimap.SpecimapControl.doBenchmark(SpecimapControl.java:135)
    at org.spec.specimap.SpecimapControl.access$000(SpecimapControl.java:37)
    at org.spec.specimap.SpecimapControl$1.run(SpecimapControl.java:1788)
Caused by: java.io.EOFException
    at java.io.DataInputStream.readByte(DataInputStream.java:267)
    at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:214)
    ... 7 more
2016-11-18 17:10:16:348 ******************** SPECimap benchmark completed
2016-11-18 17:10:16:349 Error: SpecmailControl.doBenchmark() throws java.lang.NullPointerException
java.lang.NullPointerException
    at org.spec.specimap.specimap.abortTest(specimap.java:268)
    at org.spec.specimap.SpecimapControl.doBenchmark(SpecimapControl.java:239)
    at org.spec.specimap.SpecimapControl.access$000(SpecimapControl.java:37)
    at org.spec.specimap.SpecimapControl$1.run(SpecimapControl.java:1788)
Exception in thread "Thread-2" java.lang.NullPointerException
    at org.spec.specimap.specimap.abortTest(specimap.java:268)
    at org.spec.specimap.SpecimapControl$1.run(SpecimapControl.java:1792)

[AnoopGupta]

Please check and confirm that your webserver (dovecot) is up and running. Also, disable selinux and iptables service.

Please attach the full output of the load generation if you still see the errors.

[Miles]

Hi
The same error occurs in my run.
Is this issue fixed?

client-127.0.0.1_1200.log
2017-06-27 17:11:20:363 SPECvirt-IMAP version SPECvirt-IMAP v1.1
2017-06-27 17:11:20:372 specimapclient: client1:1200 ready...
2017-06-27 17:12:09:084 QoSThreshold[SIMPLE] is: 5000 ms
2017-06-27 17:12:09:084 QoSPercent[SIMPLE] is: 95%
LoadGenerator: localHost is: client1; index = 0
With hostname (client1)
Retrieving workload key (client1:1200)
2017-06-27 17:12:09:084 Inspecting tuple: (client1:1200,CT,test1-test500,mailserver)
2017-06-27 17:12:09:084 There are 1 tuples to run
2017-06-27 17:12:09:084 userIdMin is:1
2017-06-27 17:12:09:084 userIdMax is:500
2017-06-27 17:12:09:084 This client will simulate 500 users.
2017-06-27 17:12:09:096 Loadgenerator tuple "client1:1200,CT,test1-test500,mailserver" users 1..500
2017-06-27 17:12:09:102 EVENT: ClientType_6_EventScheduler:This client will simulate 500 Client Type 6 users.
2017-06-27 17:12:09:102 specimapclient: Threads created.
2017-06-27 17:12:09:102 Looking up polling host: mailserver:8001
2017-06-27 17:12:10:845 specimapclient: Starting threads...
2017-06-27 17:12:10:845 LoadGenerator: Threads started, test in progress...
2017-06-27 17:12:13:205 ERROR: : Received NULL response from mailserver
2017-06-27 17:12:13:205 test1:Reconnecting user due to error condition during LOGIN
2017-06-27 17:12:13:500 ERROR: : Received NULL response from mailserver
2017-06-27 17:12:13:500 test2:Reconnecting user due to error condition during LOGIN
2017-06-27 17:12:13:803 ERROR: : Received NULL response from mailserver

Dovecot service is running and firewall is disabled
[root@mailserver Desktop]# rpm -qa | grep dovecot
dovecot-2.2.10-4.el7.x86_64
[root@mailserver Desktop]# systemctl status dovecot
dovecot.service - Dovecot IMAP/POP3 email server
   Loaded: loaded (/usr/lib/systemd/system/dovecot.service; enabled)
   Active: active (running) since Sat 2017-06-17 17:11:41 CST; 1 weeks 3 days ago
  Process: 1264 ExecStartPre=/usr/libexec/dovecot/prestartscript (code=exited, status=0/SUCCESS)
 Main PID: 1660 (dovecot)
   CGroup: /system.slice/dovecot.service
           ├─ 1660 /usr/sbin/dovecot -F
                   ...
           ├─23446 dovecot/imap
           ├─23451 dovecot/imap
           └─23452 dovecot/imap
                   ...


[root@mailserver Desktop]# systemctl status firewalld
firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled)
   Active: inactive (dead)

[lroderic]

Did you ever get the mailstore loaded? Did you find any errors in Clientmgr1_1094.out?

Lisa

[lroderic]

Hi Miles,

Still waiting to hear back from you if you have this working.

Lisa

[Miles]

Hi
Sorry for the late reply.
There is no error or fail message in Clientmgr1_1094.out.
How to check the mailstore loaded? any script or command?
Thanks.

[lroderic]

Miles, there's a problem with the run_load_gen.sh script. On the mailserver VM please edit /opt/SPECimap/run_load_gen.sh so that it contains this:

chown -R 777 /mailstore/dovecot
java -jar -mx1500m -ms1500m specimapclient.jar -p 1200

Then try again:

# nohup ./run_load_gen.sh &

Check that the specimapclient.jar is running in the background:

# ps -ef|grep specimap
root      7733  7731  0 13:03 pts/0    00:00:01 java -jar -mx1500m -ms1500m specimapclient.jar -p 1200

If it's not, the next commands don't work so let me know. If you see the process running, then issue:

# ./run_init.sh

To watch progress, ssh into the mailserver VM from another window and:

# tail -f /opt/SPECimap/nohup.out

This should take about an hour depending on your configuration.

Lisa

[Miles]

Hi Lisa
I have modified run_load_gen.sh but error still exists.

[root@mailserver SPECimap]# ./run_init.sh
fixedFile: IMAP_fixed.rc; configFile: IMAP_config.rc; workloadFile: IMAP_sysinfo.rc
2017-07-24 14:02:52:181 SpecimapControl: Message initialization=true
2017-07-24 14:02:52:181 SpecimapControl: Folder initialization=true
2017-07-24 14:02:52:181 SpecimapControl: Message verification=false
2017-07-24 14:02:52:181 SpecimapControl: Folder verification=false
2017-07-24 14:02:52:181 SpecimapControl: Message clean=true
2017-07-24 14:02:52:181 SpecimapControl: Folder clean=true
2017-07-24 14:02:52:181 SpecimapControl: rampUp=false
2017-07-24 14:02:52:181 SpecimapControl: warmUp=false
2017-07-24 14:02:52:181 SpecimapControl: benchmark=false
2017-07-24 14:02:52:182 SpecimapControl: overwritten=false
2017-07-24 14:02:52:182 SpecimapControl: calibrate=false
2017-07-24 14:02:52:182 ******************** Specmail benchmark started
2017-07-24 14:02:52:237 Adding host mailserver1:1200
2017-07-24 14:02:52:252 Checking for all required parameters in config.rc
2017-07-24 14:02:52:252 All required parameters in config.rc file are present
2017-07-24 14:02:52:252 Setting Message CLEAN workload...
2017-07-24 14:02:52:252 There are 1 clients involved in this test
2017-07-24 14:02:52:252 THREADS_PER_CLIENT is 8
2017-07-24 14:02:52:252 Client mailserver1:1200 will use 500 users
2017-07-24 14:02:52:252 Adding Workload: mailserver1:1200,IMAPMSGCLEAN,test1-test63,mailserver
2017-07-24 14:02:52:253 Adding Workload: mailserver1:1200,IMAPMSGCLEAN,test64-test126,mailserver
2017-07-24 14:02:52:253 Adding Workload: mailserver1:1200,IMAPMSGCLEAN,test127-test189,mailserver
2017-07-24 14:02:52:253 Adding Workload: mailserver1:1200,IMAPMSGCLEAN,test190-test252,mailserver
2017-07-24 14:02:52:253 Adding Workload: mailserver1:1200,IMAPMSGCLEAN,test253-test314,mailserver
2017-07-24 14:02:52:253 Adding Workload: mailserver1:1200,IMAPMSGCLEAN,test315-test376,mailserver
2017-07-24 14:02:52:253 Adding Workload: mailserver1:1200,IMAPMSGCLEAN,test377-test438,mailserver
2017-07-24 14:02:52:253 Adding Workload: mailserver1:1200,IMAPMSGCLEAN,test439-test500,mailserver
2017-07-24 14:02:52:263 Adding CLEAN clients...
2017-07-24 14:02:52:263 Adding host[0]: mailserver1:1200
2017-07-24 14:02:52:264 Initializing clients (this may take a few minutes)...
2017-07-24 14:02:52:345 Starting Message CLEAN...
2017-07-24 14:02:52:351 Waiting for Message CLEAN to terminate...
2017-07-24 14:03:20:601 Error: ******************** Remote exception java.rmi.UnmarshalException: Error unmarshaling return header; nested exception is:
   java.io.EOFException
java.rmi.UnmarshalException: Error unmarshaling return header; nested exception is:
   java.io.EOFException
   at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:228)
   at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:161)
   at org.spec.specimap.specimapclient_Stub.waitComplete(Unknown Source)
   at org.spec.specimap.RemoteLoadGenerators.waitComplete(RemoteLoadGenerators.java:312)
   at org.spec.specimap.SpecimapControl.doMessageClean(SpecimapControl.java:1042)
   at org.spec.specimap.SpecimapControl.doBenchmark(SpecimapControl.java:135)
   at org.spec.specimap.SpecimapControl.access$000(SpecimapControl.java:37)
   at org.spec.specimap.SpecimapControl$1.run(SpecimapControl.java:1788)
Caused by: java.io.EOFException
   at java.io.DataInputStream.readByte(DataInputStream.java:267)
   at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:214)
   ... 7 more
2017-07-24 14:03:20:603 ******************** SPECimap benchmark completed
2017-07-24 14:03:20:603 Error: SpecmailControl.doBenchmark() throws java.lang.NullPointerException
java.lang.NullPointerException
   at org.spec.specimap.specimap.abortTest(specimap.java:268)
   at org.spec.specimap.SpecimapControl.doBenchmark(SpecimapControl.java:239)
   at org.spec.specimap.SpecimapControl.access$000(SpecimapControl.java:37)
   at org.spec.specimap.SpecimapControl$1.run(SpecimapControl.java:1788)
Exception in thread "Thread-2" java.lang.NullPointerException
   at org.spec.specimap.specimap.abortTest(specimap.java:268)
   at org.spec.specimap.SpecimapControl$1.run(SpecimapControl.java:1792)


Thanks.

[lroderic]

What's the version of Java? What's the CentOS version?

[ChrisFloyd]

Miles,

Can you verify that your IMAP mailserver is actually up and accepting connections by using telnet from the client:

# telnet mailserver1 143

..This should connect to port 143 and display a message like:
* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE STARTTLS AUTH=PLAIN] Dovecot ready.

Now, at the dovecot "prompt", try entering:
a login "test1" "test1"

You should get an "a OK Logged in." reply

then try:
b select INBOX

Please let us know if those steps work, or if not what the errors are.

[Miles]

Hi
After umount mailstore and re-run makeme_mailserver.sh, my mailserver is working now.

# telnet mailserver1 143

..This should connect to port 143 and display a message like:
* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE STARTTLS AUTH=PLAIN] Dovecot ready.

  It works in this step.

 "at the dovecot "prompt" means prompt in mailserver?

[lroderic]

OK! If the mailstore load ran correctly, the /mailstore file system should be 50% full. Follow the guide to warm up the mailstore then back it up.

Lisa