A NetApp Technical Diary

As a non-programmer, I’ve always been reticent to use anything with acronyms like API and SDK, relying instead on issuing a full command line using rsh or ssh. That works for a while until you want to start doing things like checking for errors – and until you get fed up with 90% of the [...]

api, filer, General, how-to, python, Scripts, sdk, zapi NetApp, ONTAP

Those that view this site regularly, I apologise for the lack of updates. About 2 years ago I significantly moved roles, I moved into much more of an architects role and started engaging at a higher level with my customers. This means I’m really not as hands on with NetApp technology as I used to be. I’ve noticed this more as I’ve been designing new NetApp technology and I have to lean on our skilled consultants more to help with the technical detail. I really hope that the information on this website still serves as a great reference for people and that it continues to have great success, but I’m afraid I won’t be able to commit any new content here.

As my role has changed, I’ve decided to re-invent my online presence, so this isn’t the end of my social networking involvement! You can still find me on Twitter @ckranz and I have started a new blog with more relevance to what I’m doing now. As I am now a Solutions Architect, involved in cloud designs, high level strategies and general thoughts, it seems fair that I continue share my experiences.

Please visit my new blog at: outofthis.net/work for continued stories and thoughts from me. I certainly haven’t hung up my NetApp hat just yet, but I don’t feel I can help the community with my technical expertise and knowledge any more.

If you are interested in being a guest blogger on WAFL to share your NetApp skills and experiences, please get in contact. This has become a popular website and it’d be great to see fresh new content on it again.

General NetApp

I’m very honoured and grateful to have been asked recently by Lisa Crewe from NetApp to put together a guest blog post on her NetApp blog. The first of these has gone live this afternoon, covering some of the fundamentals to “What is the Cloud”. I was quite keen to not talk specific products or solutions, but more the softer aspects of what you need to consider in order to start planning towards your own Cloud offering.

http://blogs.netapp.com/crewe_oncommand/2011/05/guest-post-what-is-the-cloud.html

General cloud, NetApp, W

As you do, I was reading up on RAID levels while in the bath. The topic of atomicity came up, and it’s something I wanted to share.

Not usually the most reliable source of technical data, but I’ll quote Wikipedia to help explain atomicity to set the stage. Taken from http://en.wikipedia.org/wiki/RAID under the section of “Problems with RAID”…

This is a little understood and rarely mentioned failure mode for redundant storage systems that do not utilize transactional features. Database researcher Jim Gray wrote “Update in Place is a Poison Apple”^[28] during the early days of relational database commercialization. However, this warning largely went unheeded and fell by the wayside upon the advent of RAID, which many software engineers mistook as solving all data storage integrity and reliability problems. Many software programs update a storage object “in-place”; that is, they write a new version of the object on to the same disk addresses as the old version of the object. While the software may also log some delta information elsewhere, it expects the storage to present “atomic write semantics,” meaning that the write of the data either occurred in its entirety or did not occur at all.

This has come back into light recently but under a different guise with SSD write failure problems. Many SSD manufacturers and enterprise storage vendors are addressing this with new firmware that writes all data sequentially, never over-writing a data block until all of the disk has been written then starting over-writing blocks from the start (that have obviously been freed up first).  Read more... (977 words, 1 image, estimated 3:54 mins reading time)

General atomicity, bit failure, byte failure, NetApp, poison apple, RAID, WAFL

Almost seamless! Sort of…

As with most of my thoughts, it started with an innocent customer query. EMC have some very cool inbuilt tools for doing seamless CIFS data migration, but NetApp don’t. It’s something that often causes a fair amount of problems and some careful planning with NetApp as we don’t have this. But I was thinking today, we kinda do, I just don’t think we leverage the tools available properly.

Enter widelinks. Here is an excerpt from a NetApp KB article on the topic (KB 3011420)…

A symbolic link is a special file created by NFS clients that points to another file or directory. Widelink entries are a way to redirect absolute symbolic links on the filer. They allow the symbolic link destination to be a share on the same filer or on another filer. The following examples illustrate how to create a symlink from volume to qtree on the same filer, and from volume to volume on different filers.

What does this mean and why will my life be easier after reading the rest of this article?

So if I have a nice shiny new NetApp filer (or an old one I haven’t got round to migrating my CIFS data onto yet), and I have my old CIFS file server that is rapidly approaching failure or out of support. I can create my new file and share structure on my NetApp, and then use widelinks to redirect the user to the CIFS file server while I worry about all the data copy out of hours without having the ball-ache of copying all my data all at once.  Read more... (1385 words, 3 images, estimated 5:32 mins reading time)

Command Line, General cifs, NetApp, symbolic links, symlinks, widelinks

I’m definitely looking forward to VMWorld this year! I haven’t been since Cannes in 2007, so I’m looking forward to seeing how much it’s grown and even more so as I have my VCDX Defense out of the way so I’m relatively stress free! The sessions seem a little overwhelming, I’ll be spending the next few weeks catching up on missed sessions online I think!

If you are there, come track me down. I’ll be hanging about with the NetApp guys a little, teasing the other storage vendors, and I’ll definitely be at the NetApp Communities event on Tuesday night – http://communities.netapp.com/thread/10582.

Here is a quick list of the sessions I think I’ll attend. I’m still undecided about a few and have narrowed it down to 2. I’ll be sure to watch a lot more sessions than this after the event as there is a lot of good stuff I’m missing out on! I look forward to seeing everyone there though!

Monday
9am : ALT2002 – vCenter VSheild
10am : SS1055 – Partner Super Session
10.30am : TA6720 – Troubleshooting with ESXTOP for Advanced Users
12pm : BC7773 – SRM Misconceptions and Misconfigurations
12pm : GD01 – vSphere Security
1.30pm : BC7729 – Intelligent HA, Application awareness with HA
3pm : TA7994 – vStorage Update for vSphere 4.1

Tuesday
11am : MA7140 – vCloud Architecture Design Strategies
12.30pm : TA8218 – VMware Storage Vision
2pm : SE8206 – Security Hardening Guidelines for vSphere
2pm : ALT3001 – SRM Extended Configuration
3.30pm : DV7706 – View Composer, Technical Deep Dive
3.30pm : SE8389 – Architectural Overview of Security for the Private Cloud
5pm : ALT3003 – Performance Tuning for vSphere  Read more... (440 words, 1 image, estimated 1:46 mins reading time)

General communities, NetApp, vmworld

I had the task of replicating an existing customers filer to their new DR system. This was quite a large system, and as much as creating SnapMirrors isn’t particularly complex, it is time consuming. So as with many things I do, I wrote a script to help my achieve this task quicker, and go have yet another coffee.

You’ll need pre-shared keys setup (as ever), but the rest is prompted for. There’s no data validation (no surprise). The script will output 4 text files, one for creating/restricting all the volumes, one for the “snapmirror.conf”, on to be used in place of “snapmirror.conf” while you are doing the baseline initialization, and finally one to actually initialize the snapmirrors. There’s no intelligence around concurrent streams, so initialization is still a bit of a juggling act and waiting game.

Please let me know if you find this useful.

#!/bin/bash

echo “Please enter the name of the PRIMARY filer: ”
read PRI_FILER

echo “Please enter the name of the DR filer: ”
read DR_FILER

ConnectString=”ssh -c 3des”
#ConnectString=”rsh”

SnapMirrorHour=22
SnapMirrorMinute=0
SnapMirrorStagger=5

echo “” > ${DR_FILER}_filer_volumes.txt
echo “” > ${DR_FILER}_snapmirror.conf
echo “” > ${DR_FILER}_sm_initialize.txt
echo “” > ${DR_FILER}_snapmirror_init.conf

for AGGR in `${ConnectString} $PRI_FILER “aggr status” | awk ‘$2!~/State/{print $1}’`
do
 SIZE=`${ConnectString} $PRI_FILER “df -Ah $AGGR” | sed ‘s/\([0-9][KMGT]\)B/\1/g’ | awk ‘$1!~/.snapshot|Aggregate/{print $2}’`
 for VOL in `${ConnectString} $PRI_FILER “aggr show_space $AGGR” | awk ‘$1!~/Space/{print $0}’ | awk ‘$4~/volume|file|none/{print $1}’`
 do
  echo “vol create ${VOL} -s none ${AGGR} ${SIZE} ” >> ${DR_FILER}_filer_volumes.txt
  echo “vol restrict ${VOL}” >> ${DR_FILER}_filer_volumes.txt
  echo “snapmirror initialize -S ${PRI_FILER}:${VOL} ${DR_FILER}:${VOL}” >> ${DR_FILER}_sm_initialize.txt
  echo “${PRI_FILER}:${VOL} ${DR_FILER}:${VOL} – ${SnapMirrorMinute} ${SnapMirrorHour} * *” >> ${DR_FILER}_snapmirror.conf
  echo “${PRI_FILER}:${VOL} ${DR_FILER}:${VOL} – - – - -” >> ${DR_FILER}_snapmirror_init.conf

  SnapMirrorMinute=`expr $SnapMirrorMinute + $SnapMirrorStagger`  Read more... (328 words, estimated 1:19 mins reading time)

Command Line initialise, initialize, NetApp, setup, SnapMirror

Excuse the absence in both presence and posts. It’s been a roller-coaster past year with personal injury and flat-out work schedules, so I have had little time or motivation to blog or show my face around the communities. My apologies, and I am determined to break this habit and get back into things once again! But enough of the chatter, get on with the writings…

This isn’t something I see very often, but when I do, it’s interesting to see the stats speak for themselves. I’m with a customer who had a scripted deployment of their NetApp estate a few years ago, and it wasn’t designed or delivered with too much care or attention (something I want to discuss another day). They have a VMware estate with SQL, Exchange and other things. It all runs across a total of over 100 15k FC spindles. It’s not a huge estate in comparison with other sites, so I’m intrigued into why they have such performance issues.

Now when you run through “sysstat –u”, you can see that the filer itself is doing very little, quite happily getting on with what it should do. But the disk is hitting 100% quite often. Immediately this shows a disk problem. They need more spindles, obviously?

Firstly there is an imbalance of spindles. They have a second aggregate on the partner controller that only has test volumes. I get permission to remove this and hot, I re-allocate these to the other controller and expand the existing aggregate. This doubles the spindle count, but I know it’s not going to do anything for existing performance (in that the data won’t automatically redistribute itself!).  Read more... (799 words, estimated 3:12 mins reading time)

General disk performance, fragmentation, hot spindles, NetApp, reallocate, stats show

We are currently going through a fairly large project internally, and part of this is a “risk register” against the business. Now this includes a lot more information than just simply data on disk, but also people, reputation and so on. For me, now that I have started this project, that is a key part of data protection.

 

It’s an interesting topic, and something that I’d like to share with you at this early stage in my own project as it makes you look at the storage aspects in a different light.

 

What affects a piece of data’s risk class?

 

1.  Who has access to it?
2. How confidential is it?
3. Does it have a tangible value?
4. How portable is it?
5. Could it potentially damage the business reputation?
6. Is it protected?
7.   … probably a lot more!

 

Some of these are all questions we already have asked about the data sets as we need to define snapshot, replication and tape policies, but data protection goes a lot further than just this. Interestingly the Zemanta plugin for my blog has linked “data protection” with “Information Privacy”, which is a key point!

 

Who has access to it?

 

Not just from a front-end authorised point of view, although you do need to know this. Payroll for instance, generally it would just be HR and Accounting that have access to this, but is there a mechanism for anyone else to gain access to it? If so, is there any audit control to check who has been granted access, or who has gained access? The audit control is almost more important than the security in the first place. Security can and will always be broken, but if you can prove it was broken, then you can fix it!  Read more... (1072 words, 1 image, estimated 4:17 mins reading time)

General audit, business value, data protection, Information privacy, NetApp, Risk Analysis, risk register

http://www.youtube.com/watch?v=VWy1Sc9dtGs

New YouTube video showing some of the new features and functionality of NetApp SMVI 2.0. Not sure on a release date just yet, but looks promising!

NetApp News NetApp, smvi, snapmanager, VMware