.. _workload_images: ********************************** Preparing Workload Images ********************************** SPEC Cloud IaaS 2016 Benchmark has two workloads, namely, YCSB and K-Means. [For development only] If your company is a member of SPEC, you can download the workload images for Cassandra/YCSB and Hadoop/KMeans from SPEC Miami site. These images are not available to benchmark licensees or general public. CentOS:: Cassandra/YCSB:: image: https://spec.cs.miami.edu/private/osg/cloud/images/latest/cb_speccloud_cassandra_2111_centos.qcow2 md5sum: https://spec.cs.miami.edu/private/osg/cloud/images/latest/cb_speccloud_cassandra_2111_centos.qcow2.md5 K-Means:: image: https://spec.cs.miami.edu/private/osg/cloud/images/latest/cb_speccloud_kmeans_hadoop_271_centos.qcow2 md5sum: https://spec.cs.miami.edu/private/osg/cloud/images/latest/cb_speccloud_kmeans_hadoop_271_centos.qcow2.md5 Ubuntu:: Cassandra/YCSB:: image: https://spec.cs.miami.edu/private/osg/cloud/images/latest/cb_speccloud_cassandra_2111_ubuntu1404.qcow2 md5sum: https://spec.cs.miami.edu/private/osg/cloud/images/latest/cb_speccloud_cassandra_2111_ubuntu1404.qcow2.md5 K-Means:: image: https://spec.cs.miami.edu/private/osg/cloud/images/latest/cb_speccloud_hadoop_271_ubuntu1404.qcow2 md5sum: https://spec.cs.miami.edu/private/osg/cloud/images/latest/cb_speccloud_hadoop_271_ubuntu1404.qcow2.md5 Timezone ======== It is highly recommended to configure UTC timezone on all instance images and ensure that the timezone of instances and the benchmark harness machine matches. The configuration of a timezone can be specific to a \*Nix distribution. Tips for Preparing Workload Images ================================== It is recommended that Cassandra and Hadoop be setup in different images. The following process may be followed: (1) Create a common image. (2) Create an instance from common image and install Cassandra and YCSB. Take a snapshot of this instance. (3) Create another instance from common image and install Hadoop and KMeans. Take a snapshot of this instance. .. _prepare_workload: .. include:: ./prepare_vm_images/x86_64/prepare_common_ubuntu_qcow2_x86_64.rst .. include:: ./prepare_vm_images/x86_64/prepare_cassandra_ubuntu_qcow2_x86_64.rst .. include:: ./prepare_vm_images/x86_64/prepare_kmeans_ubuntu_qcow2_x86_64.rst .. include:: ./prepare_vm_images/x86_64/prepare_common_centos_qcow2_x86_64.rst .. include:: ./prepare_vm_images/x86_64/prepare_cassandra_centos_qcow2_x86_64.rst .. include:: ./prepare_vm_images/x86_64/prepare_kmeans_centos_qcow2_x86_64.rst .. include:: ./prepare_vm_images/ppc/prepare_common_qcow2_ppc.rst .. include:: ./prepare_vm_images/ppc/prepare_cassandra_qcow2_ppc.rst .. include:: ./prepare_vm_images/ppc/prepare_kmeans_qcow2_ppc.rst .. include:: ./prepare_vm_images/ec2/prepare_common_ami_ec2.rst .. include:: ./prepare_vm_images/ec2/prepare_cassandra_ami_ec2.rst .. include:: ./prepare_vm_images/ec2/prepare_kmeans_ami_ec2.rst .. include:: ./prepare_vm_images/gcloud/prepare_common_gcloud_ubuntu_x86_64.rst .. include:: ./prepare_vm_images/gcloud/prepare_cassandra_gcloud_ubuntu_x86_64.rst .. include:: ./prepare_vm_images/gcloud/prepare_kmeans_gcloud_ubuntu_x86_64.rst .. _upload_image_launch_vm_ai: ***************************************************** Upload Images in Your Cloud, Launch VM, and Launch AI ***************************************************** It is assumed that a tester knows how to upload images in the cloud under test. Sample instructions for an OpenStack cloud are provided below. [For development only] YCSB and K-Means images (centos & ubuntu) are available on SPEC Miami repository. The tester can download these images:: centos: https://spec.cs.miami.edu/private/osg/cloud/images/latest/cb_speccloud_cassandra_centos.qcow2 https://spec.cs.miami.edu/private/osg/cloud/images/latest/cb_speccloud_kmeans_hadoop_271_centos.qcow2 ubuntu: wget https://spec.cs.miami.edu/private/osg/cloud/images/latest/cb_speccloud_cassandra_2111.qcow2 wget https://spec.cs.miami.edu/private/osg/cloud/images/latest/cb_speccloud_hadoop_271.qcow2 Assume OpenStack credentials have been sourced, downloaded the cassandra image at the following path on cloud controller:: centos: /home/ubuntu/images/cb_speccloud_cassandra_centos.qcow2 ubuntu: /home/ubuntu/images/cb_speccloud_cassandra_2111.qcow2 Upload the image into OpenStack cloud:: centos: glance image-create --name cb_speccloud_cassandra_centos.qcow2 --disk-format qcow2 --container-format bare --is-public True --file /home/ubuntu/images/cb_speccloud_cassandra_centos.qcow2 ubuntu: glance image-create --name cb_speccloud_cassandra_2111.qcow2 --disk-format qcow2 --container-format bare --is-public True --file /home/ubuntu/images/cb_speccloud_cassandra_2111.qcow2 Similar steps can be followed to upload the KMeans image. Configuring CBTOOL with the Images Uploaded in OpenStack Cloud ============================================================== Edit the cloud definitions file and add the [VM_TEMPLATES : OSK_CLOUDCONFIG] after the "ADD BELOW" line:: vi ubuntu_cloud_definitions.txt [USER-DEFINED : CLOUDOPTION_MYOPENSTACK] OSK_ACCESS = http://PUBLICIP:5000/v2.0/ # Address of controlled node (where nova-api runs) OSK_CREDENTIALS = admin-admin-admin # user-password-tenant OSK_SECURITY_GROUPS = default # Make sure that this group exists first OSK_INITIAL_VMCS = RegionOne # Change "RegionOne" accordingly OSK_LOGIN = cbuser # The username that logins on the VMs OSK_KEY_NAME = spec_key # SSH key for logging into workload VMs OSK_SSH_KEY_NAME = spec_key # SSH key for logging into workload VMs OSK_NETNAME = public ADD BELOW [VM_TEMPLATES : OSK_CLOUDCONFIG] # setting various CBTOOL roles and images # the images have to exist in OpenStack glance. #choose required images (centos or ubuntu) and comment other images. # centos images CASSANDRA = size:m1.medium, imageid1:cb_speccloud_cassandra_centos YCSB = size:m1.medium, imageid1:cb_speccloud_cassandra_centos SEED = size:m1.medium, imageid1:cb_speccloud_cassandra_centos HADOOPMASTER = size:m1.medium, imageid1:cb_speccloud_kmeans_centos HADOOPSLAVE = size:m1.medium, imageid1:cb_speccloud_kmeans_centos #ubuntu images CASSANDRA = size:m1.medium, imageid1:cassandra_ubuntu YCSB = size:m1.medium, imageid1:cb_speccloud_cassandra SEED = size:m1.medium, imageid1:cb_speccloud_cassandra HADOOPMASTER = size:m1.medium, imageid1:cb_speccloud_kmeans HADOOPSLAVE = size:m1.medium, imageid1:cb_speccloud_kmeans .. _launch_a_vm_and_test_it Launch a VM and Test It ======================= In your CBTOOL CLI, type the following:: centos: (MYOPENSTACK) cldalter vm_defaults run_generic_scripts=False (MYOPENSTACK) cldalter MYOPENSTACK ai_defaults dont_start_load_manager True (MYOPENSTACK) typealter MYOPENSTACK cassandra_ycsb cassandra_conf_path /etc/cassandra/conf/cassandra.yaml (MYOPENSTACK) vmattach cassandra ubuntu: (MYOPENSTACK) cldalter vm_defaults run_generic_scripts=False vmattach cassandra This will create a VM from Cassandra image that uploaded earlier. Once the VM is created, test ping connectivity to the VM. CBTOOL will not run any generic scripts into this image. Here is a sample output:: (MYOPENSTACK) vmattach cassandra status: Starting an instance on OpenStack, using the imageid "cassandra_ubuntu_final" () and size "m1.medium" (), network identifier "[{'net-id': u'24bf3e75-c883-41e6-9ee7-4ca0f16e8fa9'}]", on VMC "RegionOne" status: Waiting for vm_3 (cloud-assigned uuid a5f1047d-8127-435b-b2f6-10fa96e19fdc) to start... status: Trying to establish network connectivity to vm_3 (cloud-assigned uuid a5f1047d-8127-435b-b2f6-10fa96e19fdc), on IP address 9.47.240.221... status: Checking ssh accessibility on vm_3 (cbuser@9.47.240.221)... status: Boostrapping vm_3 (creating file cb_os_paramaters.txt in "cbuser" user's home dir on 9.47.240.221)... status: Sending a copy of the code tree to vm_3 (9.47.240.221)... status: Bypassing generic VM post_boot configuration on vm_3 (9.47.240.221)... VM object 9440AAF5-B5D9-5FF6-8845-45DBCFBA0270 (named "vm_3") sucessfully attached to this experiment. It is ssh-accessible at the IP address 9.47.240.221 (cb-ubuntu-MYOPENSTACK-vm3-cassandra). (MYOPENSTACK) Configure /etc/ntp.conf with the NTP server in your environment and then run:: sudo ntpd -gq If NTP is not installed in your image, then the image must be recreated with NTP package installed. If the result of running the above command is zero (``echo $?``), then the instance can reach the intended NTP server. Next, restart CBTOOL:: cd ~/osgcloud/cbtool ./cb --hard_reset -c configs/ubuntu_cloud_definitions.txt Create a VM again:: vmattach cassandra If VM is successfully created, CBTOOL is able to copy scripts into the VM. Adjust the number of attemps CBTOOL makes to test if VM is running. This comes handy if the cloud under test takes a long time to provision. This value is automatically set during a compliant run based on maximum of average AI provisioning time measured during baseline phase:: cldalter MYOPENSTACK vm_defaults update_attempts 100 Repeat this process for all role names, that is:: vmattach ycsb vmattach hadoopmaster vmattach hadoopslave .. include:: ./launch_first_ai.rst