VMware Experts Program Big Data

This week VMware is holding the VMware Experts Program Big Data, Scientific & Engineering Workloads at VMware Corporate Headquarters, Palo Alto, Ca. This program is an extension of the very successful VMWare Expert Programs held all over the world on Oracle and Microsoft SQL Server.

VMware Experts Program Big Data Attendees

Kailash Marthi? EMC, Kris Applegate DELL Tom McCuch HortonWorks, Roman Zajic?Quanta Cloud Technology (QCT),??Tej Luthra Merlin, Jamal Robinson Intel, Dev Reddy, Chris Dagdigian @chris_dag BioTeam, Tony Foster?@wonder_nerd EMC, Motti Beck? @MottiBeck??Mellonox Technologies, Fredrick Van Haren Evaluator Group

To attend the VMware Experts Program Big Data one guest flew in all the way from Taiwan.

AdvanceResource Management -Deep Dive Workshop

& ESXi Scheduler Operation

Mark Achtemichuk Performance Staff Engineer, VMWare

We have optimized the stack for latency in both directions

There isn?t overhead in running virtual machines anymore

TPS is now secure by default

The Protocol war continues, NFS, iSCSI, and FC each are performant, bandwidth required

JUMBO frames Not faster but reduced overhead cost

Right Size, Right Size, Right Size Spend effort on rightsizing workloads for VCPU count and assigned memory

Enable Hyper-Threading – Vsphere understands and uses hyper-threading to its advantage

Don’t use vCPU Hot-Add – As it disables vNUMA and presents the virtual machine with UMA topology

Use vmxnet3 guest network driver ? very efficient and required for maximum performance

Evaluate disabling interrupt coalescing ? default may induce small amounts of latency in favor of throughput, evaluate disabling it as cost today is negligible

It?s a 10Gb+ World ? 1Gb saturation is real, more bandwidth required today. Especially in light of vSAN, Monster BM vMotion

VSphere is tuned to be conservative out of the gate (Default Behavior all flash world, not appropriate out of the gate).

Meeting at the Edge

Ray O’Farrell EVP & CTO VMware

The Internet of Things is an industrial revolution of sorts

How do you leverage the new technologies to change how your company works

Your ability to get at CPU Cycles has dropped in price enormously

New ways of analyzing all this data keep emerging

Challenge of revolutions, people die. So how do we help people make the transitions

Companies that failed to make the transition are likely to be eclipsed by companies that do

All the data is being generated at the edge and needs to be pushed back

Our focus is on Security of infrastructure, management of the infrastructure, performance of the infrastructure

Big Data Reference Architectures for vSphere ? Justin Murray

Customers want to get off costly data platforms

Older data warehouse technology not serving their needs

Provide data access to our customers directly

Exploit new application architecture for developer productivity

S3 is an object storage system rather than a file system, S3 is eventually consistent

The answer is yes we do containers

We suggest you turn hyper-threading on

Extreme?Performance: Fast virtualized Haddop and Spark on All-Flash Disks

Dave?Jaffe

Memory, CPU increasing critical for new technologies like Spark. CPU: Larger core count equally as important as faster clock speed

Use Flash Disks Appropriately

10 GbE crucial, starting to see 25 GbE

Number of Servers determined by size of workload, number of concurrent users

Use Eager Zero Thick format for virtual disks

Limit number of disks per data node to maximize utilization of each disk ? 4-6 good starting?point

Reserve 5-6% of total server memory for ESXi, use remainder for VMs

dfs.blocksize ? Tradeoff between size and number of tasks ? 256 MB good initial choice for most workloads

dfs.replication ? 3 typical for availability

Powerful new technologies like YARN, Spark and Machine Leasing yield excellent performance on vSphere when tuned properly

Performance Results show virtualized Spark and Hadoop is 10% better than native

Virtualizing HPC: Why and How

Josh Simons, Sr. Director & Chief Technologist High-Performance Computing VMware

Josh Simons VMware — Josh Simons Sr. Director & Chief Technologist, High Performance Computing, VMware

Increase IT Agility and Flexibility and decrease time to discovery for researchers, scientist, and engineers

Life science folks want to do their science, they don?t want to do the technology. It?s a means to the end

Benefits of Virtual Machines: Heterogeneity, Multitenant data security, Fault Isolation, Reproducibility, Fault resiliency, Dynamic Load Balance & Performance

Using Compute Accelerators with vSphere increasing common in traditional and emerging HPC (ML/DL, etc.) – GPGPU, FPGA, Intel, Xeon, Phi, Etc

Three approaches Available, – 1. Direct Passthrough (VM Direct Path I/0), 2. PCI SR-IOV for generic device sharing to multiple VMs, 3. NVIDIA GRID for GPGPU(And GPU) sharing and enterprise management

Dean Bolton LicenseFortress — Dean Bolton, LicenseFortress

History of VMware Experts Program

The VMware Experts Program began in 2013 as a collaborative effort between the Cloud Platform Business Unit, then referred to as vSphere Product Marketing team and the VMware Global Center of Excellence. The program brings together experts in a variety of applications, database and other technical fields at VMware in Palo Alto (Now globally) to meet with VMware and partner personnel from the highest-level executives to the engineers who architect the platform and write code. A variety of open discussions and exercises facilitate an intense learning environment leading to the expansion of the already broad communities of technical experts that have extensive knowledge of VMware virtualized infrastructure.