In the past few years, adoption of virtualization by enterprises has been increasing rapidly. The companies are virtualizing more number of server into fewer physical servers to reduce the server maintenance costs like rack space, power and support costs etc. . With virtualization, all the servers that are virtulized into a physical machine share hardware resources like CPU, memory, disk and network cards of physical machine.
As the important asset of any IT organsastion is data, all the servers have to be backed up based on the organization’s backup policies. Backing up virtual servers is tricky part as backing up one virtual machine may cause resource contention for all other virtual machines that are sharing the resources. So these virtual machines should be backed up by some other means other than conventional method (of installing netbackup client on the machine).

Beginning with netbackup 6.5 release, symantec has provided an advanced technology designed specifically for VMWare protection. This is VMWare VCB method. With this backup client is not required to be installed in virtual machines. This avoids purchasing of client licesnes for all virtual machines.

How VCB Proxy API works:

In this method, we need to desingnate a windows server as virtual consolidated backup (VCB) proxy server. Netbackup Media server software should be installed in this proxy server.

Recommendations for choosing the proxy server.

1) The proxy server should preferably be vCenter server so that the server can list all VMs that are listed under it.
2) The proxy server should have access to all the ESX datastore LUNs. This will help us in backing up data through SAN method. Otherwise, backup has to happen thorugh LAN network. This is explianed later in this blog.
3) The OS version should be compatible with netbackup 6.5. (Please check netbackup 6.5 compatibility matrix for supported OS versions).

A backup policy is created in netbackup master server for the virtual machine that needs to be backed up. Backup will be triggered as per the schedule in backup policy. Remember, I told that netbackup client is not required in the virtual machine. So backup policy is configured with the help of backup proxy server, by selecting virtual machine, from the virtual machines list of backup proxy server. (This is the reason why we need to install backup proxy software in vCenter server). So backup will be triggered as per the schedule in the policy and request will go backup proxy server. Backup proxy server quiesce the virtual machine with the help of Windows volume shadow copy service provider and ensures that virtual machine is in a consistent state before the start of backup. This step is important as the data that is backed up when server is in inconsistent state is not useful. Then it takes the snapshot of the virtual machine and transfers the snapshot to itself and mounts it at it’s own namespace. (This location is specified when creating the backup policy). If backup proxy server has access to all ESX datastore LUNs, this transfer can happen through SAN route, else it has to happen by LUN route which is slower. From this backup proxy server sends the data to netbackup media server through network, from which backup will be written to tape/disk based on the storage pool configured in backup policy.

This method eliminates the process of installing netbackup client on each virtual machine and also overloading of ESX server resources when virtual machine backups.

The backups can be either Full virtual machine or file level backups. In virtual machine backups, full virtual machine file (.vmdk files) are backed up. Backing up vmdk files will help in disaster recover scenarios as restoring vmdk file will restore the virtual machine to earlier state in the matter of minutes.

File level backups can be useful in backing up individual directories without the need of backing up the entire machine. Please note that when backing up virtual machine’s individual files/directories backup proxy server creates a lock file which prevents the starting of other virtual machine’s file level backup.

It’s better if we backup entire vmdk files atleast once in a month and individual direcories daily.

Troubleshooting:

In this method, for whatever reason backups fail with error code: 156 i.e., snapshot error. So we need to check the logs before finding the root cause. The logs are recorded by bpfis daemon and are stored in bpfis directory under NETBACKUP_INSTALL_DIR/logs directory. These logs can be captured only in backup proxy server not in master server. If any backup fails, check the logs in bpfis file and we will be able to find out the root cause.

Most common errors would be

1) Vmware lock file will be created in backup proxy server. This might be due to failure of earlier file level backup because of which netbackup might not have cleared the lock file.

2) Backup proxy server might not be having access to ESX datastore LUNs and backup would have configured in SAN transfer method which is not possible without LUN access to backup proxy server. In this case either change the backup methond to NBD or provide access to ESX datastore to backup proxy server.

With the latest advent in technology many disks manufacturers are coming up with 1TB enterprise disks and almost all enterprise array manufacturers are supporting them in their arrays. Now comes the question of should we really go for this large sized disks in enterprise storage arrays. We will see the answers for this question now.

As almost all the arrays now have implemented wide striping feature in which all I/O requests will be striped across all the available disks, using fewer number of large disks will reduce the number of IOPS required for different applications as the disks will be fewer in number with large sized disks than with small sized ones. This has an impact on performance of applications.

Also modern databases are now capable of spreading the workload across multiple database files and multiple disks. With large number of small disks, it’s possible to process the I/O requests faster as more disks are involved than with fewer large sized disks.

The only reason that I can see for people going to large sized disks is to reduce rack space that array occupies which in turn reduces the power and cooling costs, thereby reducing the carbon foot print. This is also not a problem now. As you know that most array vendors started using 2.5 inch SAS disks in their enterprise arrays which occupies lesser space than conventional 3.5 inch disks that have been in use till now. With this we can fit more number of 2.5 inch disks in the same rack space when compared to 3.5 inch disks.

So if I have the opportunity to buy a few huge disks or a large number of small, fast disks , I will go with small, fast disks.

1) quotas can be set on qtree so that size/number of files allocated to qtree can be limited.

2) Security settings can be set up on qtree to restrict it’s access to certain people.

There can be 4,995 qtrees configured in a volume.

Creating a qtree.

qtree can be created using

# qtree create <volumename/qtreename>

For example, # qtree create /vol/vol2/updates

Check the status of qtree created using

# qtree status

It displays the status of all qtrees in device. Limit to one qtree by specifying name of qtree as it’s argument.

Security Settings on qtree:

The security settings of a qtree determines whether files/directories in a qtree or volume use Windows NT ACLs or UNIX level permissions.

There are three different qtree settings that can be configured on qtree.

1. NTFS

2. NFS

3. Mixed

Security settings can be set using

# qtree security <qtreename> <ntfs|nfs|mized>

There is one more attribute that can be set on a qtree. This is opportunistic locks (oplocks).

Oplocks enable the CIFS redirector on a client to get a lock on file and cache the read/write data of the file at client side. This reduces the network traffic as there is no need for client to remind the CIFS server that it requires access to the file. By default oplocks is enabled for all qtrees. Modify this feature by

# qtree oplocks <qtreename> enable|disable.

Quotas:

quotas can be set on qtrees using /etc/quotas file.

Edit /etc/quotas file and add an entry for qtree to which quota has to be set. Then run

# quota resize command to re-read the quotas. This will check any differences to the quotas file instead of checking through all quotas which is time consuming.

The quota can be set at qtree or user or group level.

However with invention of zfs this is not a problem any more. ZFS calculates the checksum of a block when it is writing to it and compares the checksum whenever block is read back into memory. If there is any difference in the checksum, then the data on disk is changed by some other means. ZFS will automatically fix the block for us if there is any redundancy configured in zfs volume like RAID 1 or RAID 5 etc.

If you want to test the integrity of zfs data manually, there is an option provided by zfs. This is zfs scrub.

Initiate scrub on zfs file system and come to know for any data corruption in the underlying data. Please note that scrub consumes lot of I/O resources on the system. So run it when there is less load on the server.

It can be inititated with below command.

# zpool scrub <poolname>

Check the status of the scrub using

# zpool status <poolname>

pool: <poolname>

state: ONLINE

scrub: scrub in progress for 0h0m, 3.81% done, 0h18m to go

config:

NAME        STATE     READ WRITE CKSUM

<poolname>  ONLINE       0     0     0

c1t0d0s0  ONLINE       0     0     0

errors: No known data errors

This feature will give us number of read/write errors and checksum errors associated with disk.

If your production I/O operations are being affected, you can stop scrub using

# zpool scrub -s <poolname>

and verify if it is stopped using

# zpool status <poolname>

With this branded zones, now any server running either Solaris 8 or Solaris 9 can be configured as a branded zone on Solaris 10 and run applications in the same way as in physical server. No modifications are required from application point of view. All this procedure is seamless and transparent that users don’t even know that they are in virtual server. But there are some differences in the way Solaris 10 zones (non-branded zones) and branded zones developed from OS administration point.

As zones share same kernel space with physical server, all devices are owned by physical server. Thus zone administrator won’t have any ownership to devices and he/she can’t run any commands that change/analyze the physical device sructure like formant, snoop etc. This is achived by the concept of privileges in Solaris. By default the zone won’t have all privileges associated with phsical server. All other administrative commands and trouble shooting commands like ping, traceroute etc.  works fine.

There ia a privilege called net_rawaccess in Solaris which helps in accessing network raw device and run commands like packet capture (snoop),configuring the interface (ifconfig) etc. on network device.

With Solaris non-branded zones, you don’t require this privilege to run traceroute command in non-global zone.

But you need to give thir privilege to Solaris branded zones, to run traceroute command. You will get an error message like below when you attempt to run traceroute command from Solaris 10 Branded zone without net_rawaccess privilege.

bash-2.03$ /usr/sbin/traceroute x.x.x.x
traceroute: raw socket: Protocol not supported

This is due to difference in the way these two zones are developed and integrated into Solaris 10 kernel. The privileges can be given through zonecfg interface.

zonecfg -z <ZoneName>

set priv=default, net_rawaccess.

commit

exit

Reboot the zone

zoneadm -z <ZoneName> reboot

But even though you have given net_rawaccess privilege, commands like snoop, ifconfig won’t work as device in not physically owned by Zone.

How snapshot works.

Snapshot copies the file system/volume when requested to do so. If we have to copy all the data in file system/volume using traditional OS mechanisms, it takes a lot of time and consumes lot of space in system. Snapshots overcome this problem by copying only the blocks that have changed. This is explained below.

If we take a snapshot of a file system/volume, no new data is created or new space is consumed in the system. Instead of this system copies the inode information of the file system to snapshot volume. (inode information consists of file permissions, owner, group, access/modification times and also pointers to data blocks etc.). The inode pointers of snapshot volume point to same data blocks of the file system for which snapshot created. In this way snapshot consumes very minimal space (metadata of original file system).

What happens if block has been changed in original file system? Before changing the data block, system copies the data block to snapshot area and overwrites the original data block with new data. Inode pointer will be updated in snapshot to point to the data block that is written in snapshot area. In this manner, changing the data block involves reading the original data block (read operation), writing it to snapshot area and overwriting the original data block with new data (two write operations). This causes performance degradation to some extent. But you don’t need much disk space with this method, as we will record only the changes made to file system/volume. This is called Copy-On-Write (COW) snapshot

Now we will see how Netapp does it differently.

While changing the block in volume with snapshot created, instead of copying the original block to snapshot area, Netapp writes the new volume to snapreserve space. This involves only one write instead of two writes and one read with COW snapshot. However this also has some performance impact as this involves the changing he inode pointers of file system/volume, but this is minimal if compared to CO which is why Netapp snapshot method is superior when compared to other vendor snapshots.

Also during restores also, Netapp changes the pointers of filesystem to snapshot block. With COW snapshot we need to copy the file from snapshot volume to original volume. So restore with NetApp if faster when compared to COW snapshots.

Thus Netapp snapshot methodology is superior and faster compated to COW snapshots.

While bridging and routing can both be used to distribute information about the locations of resources on the network, they differ in several ways. Routing is implemented at the IP layer (L3) and uses routing protocols.No routing protocols are used on the datalink layer. Instead, the destinations of forwarded packets are determined by examining the network traffic that is received on the links that are attached to the bridge.

When a packet is received, its source address is examined. The packet’s source address associates the node from which the packet was sent to the link on which it is received. Thereafter, when a received packet uses that same address as the destination address, the bridge forwards the packet over the link to that address.

To forward packets to their destinations, bridges must listen in promiscuous mode on every link that is attached to the bridge. Listening in promiscuous mode causes bridges to become vulnerable to the occurrences of forwarding loops, in which packets circle forever at full line rate. Thus, bridging uses the Spanning Tree Protocol (STP) mechanism to prevent network loops that would render the subnetworks unusable.

In addition to using STP and the Rapid Spanning Tree Protocol (RSTP) for bridges, Oracle Solaris supports the TRILL protection enhancement. STP is used by default, but you can use TRILL by specifying the -P trill option for the bridging commands.

Using a bridge configuration simplifies the administration of the various nodes in the network by connecting them into a single network. By connecting these segments through a bridge, all the nodes share a single broadcast network. Thus, each node can reach the others by using network protocols such as IP rather than by using routers to forward traffic across network segments. If you do not use a bridge, you must configure IP routing to permit the forwarding of IP traffic between nodes.

Bridge networks can be formed into rings that physically connect several bridges together. Such configurations are common in networks. This type of configuration could cause problems with old packets saturating the network links by endlessly looping around the ring. To protect against such looping conditions, Oracle Solaris bridges implement both the STP and TRILL protocols. Note that most hardware bridges also implement STP loop protection.

All the interfaces that are part of IPMP group should belong to the same LAN i.e., the links should be part of same L2 broadcast domain. IPMP and aggregation are different technologies to achieve network performance and high availability. In general, aggregation is deployed to achieve network performance while aggregation is to achieve high availability. In link aggregations, incoming traffic is spread across multiple network interfaces due to which network performance is improved. Whereas in IPMP data traffic is flowing between two IP addresses irrespective of the number of interfaces configured. Aggregation doesn’t span multiple

switches whereas IPMP can span switches. Thus switch becomes single point of failure with aggregation.

How IPMP Works:

IPMP maintains the network availability by maintaining the active and standby connections originally configured when group was created. IPMP failure detection can be link-based or probe-based. If it finds the interface is failed, then it flags the interface as failed and is no longer usable. All the data addresses associated with the interface is migrated to another usable interface in the group. If configured, it uses standby interface configured in the group to maintain the original number of active connections.

Consider a three-interface IPMP group itops0 with an active-standby configuration, as shown in following figure.

The group itops0 is configured as follows:

■ Two data addresses are assigned to the group: 192.168.10.10 and 192.168.10.15.

■ Two underlying interfaces are configured as active interfaces and are assigned flexible link

names: subitops0 and subitops1.

■ The group has one standby interface, also with a flexible link name: subitops2.

■ Probe–based failure detection is used, and thus the active and standby interfaces are

configured with test addresses, as follows:

■ subitops0: 192.168.10.30

■ subitops1: 192.168.10.32

■ subitops2: 192.168.10.34

IPMP maintains network availability by managing the underlying interfaces to preserve the

original number of active interfaces. Thus, if subitops0 fails, then subitops2 is deployed to

ensure that the group continues to have two active interfaces. The activation of the subitops2 is

shown in Figure.

# ipmpstat -i

INTERFACE ACTIVE GROUP FLAGS LINK PROBE STATE

subitops0 no itops0 ——- up failed failed

subitops1 yes itops0 –mb— up ok ok

subitops2 yes itops0 -s—– up ok ok

After subitops0 is repaired, then it reverts to its status as an active interface. In turn, subitops2 is returned to its original standby status.

Failure and Repair detection:

To ensure continuous availability of network to send or receive traffic, IPMP performs failure detection on underlying interfaces. The failed interfaces remain unusable and standby interfaces are deployed to maintain the active connections in the group. If all the interfaces are failed, then the IPMP group is termed as failed and network connection is unavailable to the server.

In.mpathd daemon is the one which uses to monitor the state of the interfaces configured in IPMP group. It uses link-based or probe-based failure detection to do this.

Link-based: It is always enabled provide the link supports this feature. In this method IPMP checks the status of the link state continuously and migrate the IP address to other active interface in the group if it has failed and sets the interface state to be FAILED. The support of link for this type of failure detection can be found by “ipmpstat –i” command in which LINK shouldn’t be unknown.

Probe-based: in.mpathd daemon performs probe based failure detection on each interface that has test address configured on it. It involves sending and receiving test messages from the test addresses to one or more configured addresses called “test addresses” on the same LAN. The test addresses can be configured. If no test addresses are configured, then in.mpathd searches for the entries in routing

table for the target systems that are on the same subnet and takes these as target addresses. If no target systems are found on network, then in.mpathd sends the multicast messages to determine which hosts can be used as target addresses. The first five packets are chosen as targets for probing. It is doesn’t find suitable targets, then probe based failure detection is disabled and probe status is unknown in “ipmpstat –i” command.

If no response has come for the five test probes then it flags the interface as failed. The probing rate is based on Failure Detection Time which is 10 seconds by default. It can be tuned in /etc/default/mpathd file. Repaire detection time is twice failure detection time.

IPMP implementation in Solaris 11 Express is different from Solaris 10.

1.     IPMP group is treated like another IP interface. This is treated like another network interface on the system. All IP administrative work, ARP tables, routing tables work in the same manner like network interface.

2.     Distributing traffic across data addresses is responsible of the system.  In previous IPMP implementation, system administrator has to configure IP address on the network interface and this IP address is bound to that interface. But in current IPMP implementation, the data IP addresses belongs to IPMP interface as an address pool and kernel automatically and randomly binds the addresses to the underlying interfaces in the IPMP interface.

3.     “Ipmpstat” tool is introduced to obtain information about IPMP group. It provides all the information about all network interfaces in group like the status of the data and test addresses, types of failure being detected etc.

4.     The IPMP IP interface can be assigned a custom name just like an network interface and this can be plumbed and unplumbed in the similar manner as of IP interface.

scale up: dynamically to meet the consolidation demands of increasingly powerful virtual server clusters

scale out: dynamically across a pool of shared virtual storage resources instead of a cluster of standalone storage silos

and

scale deep: to extend scale up and scale out capabilities to externally attached storage.

In the next section we will go through the hardware architecture of VSP.

Hardware Architecture

 VSP is very intelligent and powerful enterprise storage system in the industry. It is based on Hi-star-E network architecture: a fifth generation of switch architecture. This is first implemented in Hitachi Lightning 9900 series enterprise storage systems.

The virtual storage system has next generation, fault tolerant, high performance and non-blocking Hi-Start-E switched network that used to transfer data and communicate between various components. The actual performance of the system depends upon the actual components installed on the system.

Hi-Star-E network is made up of PCI-Express Grid Switches (GSW). Each switch has 24 ports which are capable of 1GB/Sec send and 1GB/Sec receive for a total of 2GB/Sec bandwidth per port.

Out of these 24 ports, 8 are used to connect to Front End Director and Back End directors through which both metadata and user traffic flows. Four of the ports connect to Virtual Switch Directors through which metadata and storage array configuration information flows. 8 ports are connected to Data Cache Adapters, through which copies of reads and writes and system metadata from VSDs. The final four ports are used to cross connect the grid switches to the corresponding grid switches on second chassis. If no second chassis is installed, these are left empty.

Coming to redundancy, although each switch has only one connection per component, grid switches are employed in pairs, so all the components have two connections from two redundant grid switches which provide performance and protection in case of a link failure.

Each chassis can have two or four grid switches. The DCA Boards, VSD Boards, FED Boards and BED boards are connected to Grid Switch using PCI-Express 4-lane paths. Each switch has 24 duplex ports resulting in aggregate band width of 24GB/sec send and 24GB/sec receive.

Data cache and control memory:

The complete set of eight DCAs in single chassis system has up to 32 links connecting the data cache to Hi-Star-E network resulting in full duplex 64GB/sec aggregate total bandwidth. Each controller must have atleast two data cache adapters. The connection between GSW and DCA only see user data and metadata. It is not required that all the DCAs must be populated with their full memory limit for maximum performance, however, all DCAs must be populated with same amount of memory.

Data cache in divided into two logic boundaries and all control and user data is replicated across power boundaries to provide power redundancy.

Data Cache

Cache memory is used to cache the user data blocks that either come from BED during read operation and from FED during write operation. The write data is mirrored while read data is not mirrored. Each DCA contains 32GB(64GB) thus total of 256GB(512GB) in VSP.

In case of power failure, it is equipped with recovery mechanism to backup cache user data and also control memory. The DCA has onboard 32GB SSD to backup this information. Also it has battery onboard to destage this information to SSD in case of shutdown/unscheduled outage.

Control Memory

The control memory is place in Virtual Switch Director board which allows micro processor to access memory without any latency. The VSDs with their control memory and main processors

store and manage all of the internal operational metadata and state information about the system. The metadata related to array groups, internal and external LDEVs, runtime tables, mappings, data for various software products and the overall state of the system are stored, referenced and executed from this memory.

There can be a total of 4GB control memory in VSD board, hence total of 16GB per controller chassis. The control memory data is not mirrored. However it is backed up to first data cache adapter to prevent in case of any outage. Also since cache data is mirrored, there will be three copies of control data in the VSP array.

Performance Enhancements in VSP when reading/writing from/to cache.

The VSP has several performance enhancement cache algorithms that delivers high performance. Read operatins to VSP uses  built-in algorithms to determine whether data is being accessed sequentially. If sequential reads are detected, VSP allocates 256KB of cache slots for the sequential data to be loaded to cache. Read operations will be faster since data is directly from cache.

Hitachi Cache Residency Manager: This allows data to be permanently reside in cache in real time. Administrators can add, delete or change which datasets cache residency manager feature manages at any time.

Front End Ports

Virtual Storage Platform supports Fibre Channel for open systems and SAN connectivity. Virtual Storage Platform with two chassis supports up to 128 Fibre Channel Additional ports, up to 64, can be added by replacing one to four BEDs per controller chassis with FEDs. With the additional FEDs, Virtual Storage Platform can support up to 192 ports. The Fibre Channel support industry standard 2Gb/sec, 4Gb/sec and 8Gb/sec transfer rates.

But you can limit the amount of memory that zfs ARC cache can consume at the start of the system by using the limiting the variable “zfs:zfs_arc_max” in /etc/system file.

The values for the above variable must be set in bytes of memory that it can consume and can be in hexadecimal or decimal formats.

For example, if you want to limit the zfs ARC cache to 30GB, set the above variable as follows.

set zfs:zfs_arc_max = 0×780000000

or

set zfs:zfs_arc_max = 32212254720

This is more useful in assessing the memory requirements of applications. If this variable is not set when zfs file systems are configured, by looking at free memory output from sar or vmstat commands, you will not see any memory free in the system. So we will not able to report correctly the amount of memory that is consumed by the applications running on the system. However, by setting this variable, we can limit the memory usage by ZFS ARC cache which can be seen in increase of free memory value in all *stat scripts in Solaris.