Thin and Thick Provisioning Storage

Thick Provisioning:

In this type of storage provisioning, an estimate is made about the storage requirements for a virtual machine for its entire life cycle. Then, a fixed amount of space is provisioned to its virtual disk in advance and have the entire space committed to the virtual disk. The virtual disk takes up the entire provisioned space.

Types of thick provisioning:

Thick Provision Lazy Zeroed:

A thick provisioned lazy zeroed VMDK is similar to the eager zeroed except that the zeroing operating is performed just before a write operation, not at creation. The space is still allocated to the VMDK so after creating a VMDK with this format the datastore will show that the space is no longer available, but there is the additional overhead of zeroing out at write time.

Thick Provision Eager Zeroed:

When a thick provisioned eager zeroed disk is created the maximum size of the disk is allocated to the VMDK and all of that space is zeroed out. The creation of this disk format takes a while to be created, this is because of the zeroing process.

Thin Provisioning:

Thin provisioned VMDK's do not allocated or zero out space when they are created but instead do it only at write time. When an 80GB VMDK is created that is thin provisioned, only a little bit of metadata is written to the datastore. The 80GB does not show up in the datastore as in use like it does with thick provisioned. Instead, only when data is actually written does it take up space for a thin provisioned VMDK. At write time space is allocated on the datastore, the metadata of the VMDK is updated, then the block or blocks are zeroed out, then the data is written. Because of all the overhead at write time thin provisioned VMDK's have the lowest performance of the three disk formats. This overhead though is very small and most environments will not notice it until they have very write intensive VMs. 

Types of thin provisioning:

The difference in the two types of thin provisioning is not due to the difference in feature as in thick provisioning but due to the level at which provisioning is done. Both types of thin provisioning work in the same way as described above.

Virtual Disk Thin Provisioning:

In this type the provisioning is done at the virtual disk level. For a thin virtual disk, ESXi provisions the entire space required for the disk’s current and future activities, for example 40GB. However, the thin disk uses only as much storage space as the disk needs for its initial operations. As the disk requires more space, it can grow into its entire 40GB provisioned space.

Array Thin Provisioning:

In this type provisioning is done at the storage array level ie with LUNs. Space allocated as devices (volumes or LUNs) is created on the storage device, but the consumption of this space is only as required. Storage array thin provisioning requires ESXi 5 and a storage device with a fi rmware version that supports T10-based Storage APIs: Array Integration (Thin Provisioning).

When Storage APIs - Array Integration is used, the host can integrate with physical storage and become aware of underlying thin-provisioned LUNs and their space usage.

Using thin provision integration, host can perform these tasks:

• Monitor the use of space on thin-provisioned LUNs to avoid running out of physical space. As your datastore grows or if you use Storage vMotion to migrate virtual machines to a thin-provisioned LUN, the host communicates with the LUN and warns you about breaches in physical space and about out-of-space conditions.

• Inform the array about the datastore space that is freed when files are deleted or removed from the datastore by Storage vMotion. The array can then reclaim the freed blocks of space. 

Both of these approaches allow for the overprovisioning of storage resources. This can be a powerful feature and can provide cost savings, but it must be used with caution. If a thin-provisioned storage device runs out of space, the results are never good. Because of this, monitoring is essential with both forms of thin provisioning.

Software Defined Storage

SDS is a class of storage solutions that can be used with commodity storage media and compute hardware; where storage media and compute hardware have no special intelligence embedded in them.  All the intelligence of data management and access is provided by a software layer. The solution may provide some or all the feature of modern enterprise storage systems like scale up and out architecture, reliability and fault tolerance, high availability, unified storage management and provisioning, geographically distributed data center awareness and handling, disaster recovery, QoS, resource pooling, integration with existing storage infrastructure, etc. It may provide some or all data access methods like file, block and object.

A generic data flow in a SDS solution is explained in the figure below:

Source : http://thenewstack.io/understanding-software-defined-storage/

VMware defines the Software-defined Storage Architecture as follows:

SDS is a new approach to storage that enables a fundamentally more efficient operational model. We can accomplish this by:

• Virtualizing the underlying hardware through the Virtual Data Plane
• Automating storage operations across heterogeneous tiers through the Policy-Driven Control Plane

Virtual Data Plane

In the VMware SDS model, the data plane, responsible for storing data and applying data services (snapshots, replication, caching, and more, is virtualized by abstracting physical hardware resources and aggregating them into logical pools of capacity (virtual datastores) that can be flexibly consumed and managed. By making the virtual disk the fundamental unit of management for all storage operations in the virtual datastores, exact combinations of resources and data services can be configured and controlled independently for each VM. 

The VMware implementation of the virtual data plane is delivered through:

• Virtual SAN – for x-86 hyperconverged storage
• vSphere Virtual Volumes – for external storage (SAN/NAS)

Policy-Driven Control Plane

In the VMware SDS model, the control plane acts as the bridge between applications and infrastructure, providing standardized management and automation across different tiers of storage. Through SDS, storage classes of service become logical entities controlled entirely by software and interpreted through policies. Policy-driven automation simplifies provisioning at scale, enables dynamic control over individual service levels for each VM and ensures compliance throughout the lifecycle of the application. 

The policy-driven control plane is programmable via public APIs used to control policies via scripting and cloud automation tools, which in turn enable self-service consumption of storage for application tenants. 

The VMware implementation of the policy-driven control plane is delivered through:

• Storage Policy-Based Management – provides management over external storage (SAN/NAS) through vSphere Virtual Volumes and over x86 storage through Virtual SAN.

Nutanix which is another player in the field of Software-defined Storage follows a similar approach but the controller here is a seperate VM on top of hypervisor and requires Nutanix hardware to implement the approach.

You can read more on software defined storage in this ebook written by Scott Lowe

Coho Data which is based out of Sunnyvale, California uses a SDN enabled data stream switch to connect the VMs to storage implemented as MicroArray Nodes containing PCIe flash and hard drives.

Data Hypervisor Software on the MicroArray virtualizes storage hardware to create a high performance, bare metal object store that scales to support different application needs without static storage tiers.

Coho Data Architecture: http://www.cohodata.com/coho-scale-out-storage-architecture

Storage Area Network

Challenges with Directly Attached Storage:

1.  Storage remains isolated and underutilized.
2.  Complexity in sharing storage resources across multiple servers.
3.  High cost of managing information.
4.  Challenges in scalability.

An effective information management system must provide:

1. Timely information to business users
2. Flexible and resilient storage infrastructure.

A storage area network(SAN) provides such a solution.

A storage area network is a high-speed, dedicated network designed to deliver block-level storage to computers that are not directly connected to the storage devices or drive arrays. The storage in a SAN is not owned by any server unlike DAS(Directly Attached Storage) but is accessible by all of the servers on the network.

Advantages of SAN:

Enables sharing of storage resources across multiple servers.
Centralizes storage and management
Meets increasing storage demands efficiently with better economics of scale.

SAN Classification:

• Fibre Channel (FC) SAN:                            uses Fiber Channel protocol for communitcation.
• IP SAN:                                                        uses IP-based protocolss for communication
• Fibre Channel over Ethernet (FCoE) SAN: uses FCoE protocol for communication.

Understanding Fibre Channel:

High-speed network technology: Supports upto 16 Gbps

Highly Scalable : accomodates approximately 15 million devices.

Components of FC SAN:

Node (server and storage) ports: Provide physical interface for communicating with other nodes.

Exist on 

    - HBA in server

    - Front-end adapters in storage

Each port has a transmit(Tx) link and a receive (Rx) link

Cables:  

SAN implementation uses

    - Optical fiber cables for long distances

    - Copper cables for short distance

Two types of optical cables: 

Single-mode: Carries single beam of light and carries signal upto 10 km

Multimode   : Can carry multiple beams of light simultaneously. Used for short distance

Connectors: 

Attached at the end of a cable

Enable swift connection and disconnection of the cable to and from a port

Commonly used connectors for fiber optic cables are:

      Standard Connector(SC): Duplex connectors

      Lucent Connector(LC) : Duplex connectors

      Straight Tip(ST) : Patch panel connectors and Simplex connectors.

Interconnecting Devices:

Commenly used interconnecting devices in FC SAN are:

     - Hubs, switches and directors

Hubs provide limited connectivity and scalability

Switches and directors are intelligent devices

    - Switches are available with fixed port count or modular design   

    - Directors are always modular, and its port count can be increased by inserting additional 'line             cards' or 'blades'.

    - High-end switches and directors contain redundant components.

    - Both switches and directors have management port to connect to SAN management servers.

SAN Management Software:

    - A suite of tools used in a SAN to manage interfaces between host and storage arrays

    - Provides intergrated management of SAN environment

    - Enables web-based management using GUI or CLI

FC Interconnectivity Options:

Point-to-Point Connectivity: 

• Simplest FC configuration which enables direct connection between nodes.
• Offers limited connectivity and scalability
• Used in DAS environment

FC-AL Connectivity:

• Provides shared loop to attached nodes: Nodes must arbitrate to gain control
• Implemented using ring or star topology. May also use hub which uses star topology.
• Limitations of FC-AL :

                - Only one device can perform I/O operation at a time

                - Uses 8 bit of the 24 bit fiber channel addressing.1 address is reserved to connect to FC                          switch port. Supports upto 126 nodes.

                - Addition or removal of a node causes momentary pause in loop traffic

FC- SW Connectivity :

• Creates a logical space(called fabric) in which all nodes communicate using switches. Interswitch links(ISL) enable switches to be connected.
• Provides dedicated path between nodes.
• Addition/removal of node does not affect traffic of other nodes.
• Each port has unique 24 bit FC address.

Port Types in Switch Fabric:

Port provides physical interface to a device to connect to other devices. The types are:

N_port: or Node port is typically a host port of storage array switch.

E_port: or Extension port which is connected to E-port of other switch

F_port : or Fabric port is a port in switch which connects to N_port

G_port: or Generic port can work as F_port or  E_port which is automatically done.

Fibre Channel Protocol (FCP) Overview:

• Traditional technologies such as SCSI have limited scalability and distance
• Network technologies provide greater scalability and distance but have high protocol overhead.
• FCP provides benefits of both channel and network technologies
•      - High performance with low protocol overheads
•      - High scalability with long distance capability
• Implements SCSI over FC network
• Storage devices attached to SAN, appear as local storage devices to host operating system

Addressing in switched Fabric:

The server or disk array which has a HBA reports itself to the network using Fabric Login(FLogi). It advertises its NWWN(Node World Wide Name). The FC switch replies with the FC ID for that device. This functionality is similar to that of a DHCP.

A FC switch has a block of addresses assigned to it represented by its Domain ID.

Domain ID is a unique number provided to each switch in the fabric. Domain IDs can be statistically or dynamically configured. Since permission is required to assign domain ID, it never overlaps. One switch is elected as the principal switch. This is elected based on priority value and system WWN. The lowest one wins. No backup principal switch is elected unlike DR/BDR selection in normal switches. If the principal switch dies, new now is elected. The failover is fast.

    - 239 addresses are available for domain ID.

Maximum possible number of node ports in a switched fabric:

    - 239 domains * 256 areas * 256 ports = 15,663,104

In case of multiple switches, FSPF(Fabric Shortest Path First) is used for routing which uses Fabric IDs. Fiber channel routing table is checked for routing.

Address Resolution using Fiber Channel Name Server(FCNS):

FCNS has a list of PWWN and FC ID. This server is run by principal switch. FCNS database is distributed across all switches so there is no need of backup. As soon as the device gets reply from switch with the FC ID, the host will send PLogi message with PWWN and FC ID thus registering itself with principal switch. For address resolution, host will send query to FCNS with PWWN and FCNS replise with FC ID. Thus routing is based on FC ID which is a logical address. Thus fiber channel is a layer-3 protocol.

Advantages of Distributed vSwitch

The dvSwitch feature which is available in Enterprise Plus Edition or above includes all the capabilities of vSwitch, plus the following additional capabilities:

1. Bidirectional Virtual Machine Rate Limiting(Traffic Shaping): vSwitch can perform traffic shaping on outbound traffic only. The dvSwitch can also perform traffic shaping on inbound traffic. Traffic shaping is used when traffic shaping need to be imposed on virutal machines.

2. Centralized vCenter Administration and Provisioning: dvSwitches are administered and provisioned from within vCenter, meaning that there is a single configuration to manage which is better than managing individual vSwitches.

3. Cisco Nexus 1000V Virtual Switch: Third party dvSwitches like Cisco 1000V can be used which introduces features like ACLs, port security and more. Moreover, it gives environment to which they already understand how to use.

4. Dynamic Adjustment of Load-Based NIC teaming: Regularly checks the load on each NIC. If one NIC is overloaded, a port-NIC assignment will occur to attempt to balance the load. Thus, this process keeps the load on teamed NIC balanced.

5. Enhanced Security and monitoring for vMotion traffic: Virtual machine networking state, including counters and port statistics, is tracked as virtual machines are migrated with vMotion from host to host in a dvSwitch. This provides a more consistent view of the virtual machine’s network interfaces, regardless of the VM’s location or migration history, and simplifies the troubleshooting and network monitoring for virtual machines.

6. IEEE 802.1p tagging: IEEE 802.1p tagging is a standard used to provide quality of service (QoS) at the media access control(MAC) level. This capability can be used to generate I/O resources and is applied to outbound network traffic.

7. LLDP:  LLDP is a standard based(802.1AB) vendor neutral discovery protocol. It is used to discover information about network devices.

8. NetFlow: Netflow available in vSphere version 5 and above allows monitoring of traffic flows. This Netflow data helps in capacity planning and ensure that I/O resources are properly used in the virtual infrastructure.

9. Network I/O Control: Network I/O Control Network I/O Control allows the creation of resource pools containing network bandwidth. Administrators can create new resource pools to associate with
port groups and specify 802.1p tags, allowing different virtual machines to be in different resource pools. This allows a subset of virtual machines to be given a higher or lower share of bandwidth than the others.

10. Port Mirroring: Port mirroring is when a network switch sends a copy of network packet from a port or an entire VLAN to a network monitoring device connected to another switch port. This is also known as switched port analyzer(SPAN) on Cisco Switches. Port mirroring is used for monitoring and troubleshooting.

11. Private VLAN Support: Private VLAN which is a nested VLAN is VLAN located within a VLAN. It is used to provide isolation between computers on the same subnet. The first VLAN is primary whereas the nested VLANs are secondary.

There are three types of PVLAN ports:

• Promiscuous: Can communicate with all ports including isolated and community ports.
• Isolated: Can communicate only with promiscuous ports
• Community: Can communicate with the same secondary PVLAN and promiscuous PVLAN

12. Management Network Rollback and Recovery: This feature is used to ease management network use in the dvSwitch. It works by detecting configuration changes to the management network.

13. Network Health Check: This feature is used to help vSphere administrator quickly identify configuration errors in the network. It monitors VLAN, MTU and network adapter teaming at regular intervals. If these checks fail, a warning will be displayed in the vSphere web client.

14. Link Aggregation Control Procotol (LACP) : LACP is a standards-based link aggregation protocol used to group physical network adapter into a single logical link. The dynamic implementation included in the dvSwitch allows verification of correct setup and features automatic configuration, negotiation and renegotation of detected link failures.

15. Traffic filtering and marking: This feature is used for fi ltering and priority tagging for network traffic to virtual machines, VMkernel adapters, or physical adapters. It can be used to protect these connections from security attacks, to fi lter out unwanted traffic, or toestablish QoS.

VMWare vMotion Migration Process

vMotion is used to move a powered on virtual machine from one host to another

VMWare vMotion can be be used to :

• Improve overall hardware utilization
• Allow continued virtual machine operation while accomodating scheduled hardware downtime.
• Allow vSphere Distributed Resource Scheduler(DRS) to balance virtual machines between hosts.

How vMotion Migration works:

vMotion Migration is achieved by moving the memory state from one host to another across the vMotion network which is private, non-routed, gigabit or faster network connection between two hosts involved in the vMotion migration.

However, vMotion can be done only if certain requirements are met:

1. The hosts between which the vMotion need to be performed must have a shared storage.

2. The participating hosts must have identical networks.

In addition to that, requirements for VM are:

1. A virtual machine should not have connection to a virtual device with a local image mounted.

2. A virtual machine must not have connection to an internal switch(vSwitch with zero uplinks)

3. A virtual machine must not have CPU affinity configured.

Moreover, source and destination hosts must have:

1. Visibility to all storage (Fiber Channel, iSCSI or NAS) used by the virtual machine.

2. At least a gigabit ethernet network

• Four concurrent vMotion migration on a 1 Gbps network
• Eight concurrent vMotion migration on a 10 Gbps network

3. Access to same physical network

4. Compatible CPUs