In short Special circumstances can cause an overload of the SVC/Storwize upper cache. The impact of this situation can be very dramatic. In this example, the customer environment was near to a complete standstill. It is sometimes amazing how quickly small problems can develop into a complete fiasco .
The performance problem is very difficult to understand if it is an upper cache overflow, because the performance of VDisks are extremely bad while no effects can be seen in the partition caches or mdisks.
In this customer... [More]
The BVQ story ...
The storage challenge
Even a small storage environment may be complex. Questions about cost optimization or availability are hard to answer because the knowledge about actual and future performance needs is often not available or not current. Analysis and help in case of performance bottlenecks is not possible because of missing tools or the available tools do not have the technical depth or are too complex to be used.
The start of the... [More]
It is a long time ago, that I added an analysis video to the wiki.
The following is a very nice situation where we had to find the reason for a very high load on a node port.
You can find the video here:
Link to Video
Download video as mp4
This is a a new performance analysis whitepaper in the technical section of BVQWIKI
The unbalanced VDisk mirror problem
Many performance issues are caused by an unbalanced situation in the two sides of VDisk mirrors.
One finds, that a mirrored Volume runs into latency peaks which cannot be explained by the Volume behavior itself. The managed disk groups (mdg) in the storage backend show completely different load and latencies.
We have recognized that often the distribution of primaries can cause this problem.
Another new feature of the latest BVQ offline scanner allows to scan more than one systems at once.
This is especially helpful when you plan to use BVQ Copy Services Package to analyze Metro Mirror or Global Mirror connected systems.
A profile is created per Storage system, which is then used for this one system to separate the data. In this way, many different SVCs and or Storwize systems can be scanned at the same time.
Find more information here
I just finished an performance bottleneck analysis for an 8 nmode cluster with several 100 TB and found one thing which I think might be a commonly made mistake in many of the bigger SVC environments. This picture shows the MDisk performance of SSD storage which is not acceptable. We have 37 ms latency on 37000 IOPS which is a bad value for this technology.
This picture shows one common reason, for bad storage performance whilst SVC and the backend storage look bored.
BVQ can help to find the reasons for these kind of problems and show... [More]
it's so easy ! We have already carried out the first analysis, in which we have used the BVQ offline scanner.
A big advantage was :
No need to install anything!
The process is so simple:
1 The offline scanner is downloaded from our website
2 The offline scanner is started and collects performance data
3 The collected performance data will then be sent to us and analyzed
In this way, we have solved various typical error situations for our customers.
* Nonprefered Path... [More]
I'm going to be in Las Vegas from Feb 22 until Feb 27 to visit the IBM Pulse conference.
This is a pretty good time to meet and talk about BVQ business opportunities.
Or do some technical deep dives into the inner world of SVC or Storwize performance characteristics.
Who ever is interested on meeting - just send me a note.
The BVQ use cases and experiences page has been re designed
you will now find all information grouped by themes
BVQ use cases and experiences
BVQ usage concepts
Saving Costs with BVQ, ROI related documents
BVQ VMWare Integration Package
BVQ Analysis: Performance bottleneck analysis
BVQ Analysis: SVC Storwize nodes, global cache, CPU load
BVQ Analysis: SAN, node ports, buffer credits
BVQ Analysis: Managed Disk Group Cache
BVQ Analysis: Compression,... [More]
This was a very exiting and successful week!
This is a customer success story where BVQ helped to fix performance issues in a Red Hat Enterprise Virtualization (RHEV) environment with IBM Storwize V7000 storage. The Red Hat environment is set up with six storage volumes which have been thin provisioned and compressed to optimize the capacity efficiency.
Unacceptable performance of the RHEV
The situation in the beginning of the project was that the performance of the RHEV environment was reduced due... [More]
Overview: cost savings potential with BVQ
There are several areas in which BVQ can help to reduce procurement costs, operating costs and operating risks.
This document describes these areas where BVQ helps to save costs.
Reduction of purchasing costs
With measurements and appropriate presentation of results in the form of treemaps associated with IO density colorations, BVQ provides an excellent base to obtain a better assessment about the required amount of performance. In most cases this leads... [More]
I'm really excited about this next option in our mdisk group analysis section. It is the first time that we can analyse into the internal SVC queues down to the storage system.
With this new option we get a much better view of the performance problems of the storage system.
It is obvious that this backend storage system was in deep performance troubles from 5 am to 6:30 am.
Have a look to this:
The picture belongs to this whitepaper
With using min and max cache fullness values one can better analysis what is going on in the SVC or Storwize cache. This is espacially helpful when you want to find the root causes of performance bottleneck.
This document belongs to the BVQ Use Case Performance Analysis
In this example we will see that the usage of the VAAI is somehow hidden from ordinary monitoring methods because the VAAI data streams are not visible on the top SCSI layers. So it might look like a system reacts overloaded without any reason but the reason can be uncovered when you look deeper into the system when your analysis tool is tailored good enough to SVC/Storwize and enables you to see needed the performance indicators.
You find more detail here : full document
last week I had a discussion with one of our customers - we were talking about sens and nonsense of monitoring.What is the difference between monitoring and analysis?
The reason for this discussion was again an article in the web with an performance analysis which stopped at the point where the problem became visible in the graphs but there was no further step to deeper understand an solve this problem.
The main reason for this was, that there was no tool which allowed to do an deeper analysis.
This analysis tool was a... [More]