Testing Tools and Addons - Metalogix SSMM with Large Document Libraries continued

Overview

I have increased the ram on my test machines to 4GB each as I continue to test MetaLogix SharePoint Site Migration Manager with large files. I have a document library in SharePoint 2003 that is about 20GB in size and has files up to 650MB. This is not the typical load for SharePoint, but the engineering SharePoints that I am looking at migrating have several of these.


Testing a copy list operation:



Lots of options for list copying:

I want to get an idea on performance although many factors come into play fortunately the logging is very thorough and easy to access so I don’t have to spend time grabbing start and stop times.

I am still seeing memory exceptions when copying files although the system has plenty of memory:


I remember issues around webdav and memory and looked up the article from the SharePoint Blogs (http://www.sharepointblogs.com/smc750/archive/2009/05/19/uploading-content-into-sharepoint-let-me-count-the-ways.aspx) here is an excerpt:

“Ok this seems simple enough. As you can see it is not as complex as using the Copy web service. However, it does not support sending any metadata long with the file content. This can be a major problem if the document library has multiple content types, so the new file will be put into the document library with the default content type. Another big issue is if the default content type has required fields. The file will remain checked out until the fields are populated. This prevents other users from seeing the document or from being returned in any searches. It is a great solution if you are just bulk migrating data from an external data store to SharePoint. You more than likely will have to do extra work afterwards. Adding metadata after uploading will also cause the creation of extra versions of the document being created unnecessarily. The fact that it does not use the soap protocol but straight http makes it more scalable than the copy web service. Unfortunately, it still suffers from the fact that it uses a byte array to upload the file. So sooner or later you will run into “out of memory “ exceptions. So how can I create a folder before using WebDav? You can use the lists web service to accomplish this:”

I am sending a quick question to Jeremy at Metalogix to see if they use Webdav for the copy process. The next step is to see if I can copy the files that it fails to copy one at a time through SharePoint Site Migration Manager. With good logging I can setup a process to catch the errors and have them manually moved, ideally though I do not want the people who are executing the migration to have to leave the Migration Manager window. In a large migration, two of the most important factors are the migration execution plan and the responsiveness of the team.

A Quick Mention - SP2 Bug Trial License Issue

A Quick Mention

The SP2 License bug has been documented quite a few times already and I am not aware of any downtime that has resulted from the issue. To fix the bug all that you need to do is put the key into the software again. As long as you have good documentation of your installation that should not be a problem. (PS always document your implementation) I held of on this quick mention until the actual Microsoft KB was posted. Click here for all the details: http://support.microsoft.com/kb/971620



I have seen the license synchronizer job stall out on servers, if necessary you may have to go to each server and force it through with stsadm.

Testing Tools and Addons - Metalogix SSMM with Large Document Libraries continued

Overview

Continuing testing with some large site collections. Just to reiterate, I am working on a migration that involves engineering content that is large in both number of items (site size) and size of items so I have created a 20gb site collection full of 600mb files.

During testing I was getting errors attempting to move larger files so I increased the RAM on my test Virtual Machines to 3GB each. I emailed Jeremy at Metalogix as well and he suggested there could be a resource issue and to attempt moving just the Document Library or the item. I am looking at a 10TB farm so I can’t really move one file at a time unless it’s the occasional exception… But overall I have had fairly good results so let’s proceed:

I am beefing up my servers to 4gb of ram:


To go any higher wouldn’t help, none of the old servers have more than 4GB so if I can’t do it now I’m out of luck.

I have more than enough disk and io available for the operation:



For some reason it wanted to be activated again… I thought I had a few more days. Guess not I received the key on the 5/11 and it was good until 5/24. 13 days may seem like a lot of time for testing but with a large migration and many variables that’s a pretty small window. I am sending out for another key.


Metalogix has been very responsive so far whenever I sent a message so I am sure they will not disappoint.

Implementing a Help Desk - Stopping SharePoint Services Quickly

Overview

I am presently working on a datacenter transformation project where all of the SharePoint servers are being moved to a new location. Since there is replication going on, I need to shutdown all the SharePoint services and verify that the program files have been replicated to a new server. (We are using Database Mirroring for the SharePoint databases.) To help out this process I created a fast and dirty PowerShell script to stop processes. I examined several methods on the net but found I like using sc.exe best. This is the kind of operation the help desk should be able to initiate at need.

'Stop Services on SharePoint Servers'

$SharePointServers = Import-Csv SharePointServers.csv

$SharePointServices = Import-Csv SharePointServices.csv

foreach ($entry in $SharePointServers)

{

foreach ($entry2 in $SharePointServices)

{

sc.exe $entry.Server stop $entry2.name

*sc.exe $entry.Server config iisadmin start=disabled

}

}

A test in my lab found this to be the best method to avoid error handling issues, etc. If the service is already stopped the script will continue without issue.

*Update I had to actually disable iisadmin so the webservices would not start up again. I also had some issues with our monitoring software comming up behind me and restarting the sql services.

Testing Tools and Addons - Metalogix SSMM with Large Document Libraries continued

Overview

Continuing my testing with large document libraries. I was able to move the sites but with some issues. From where I left off last:

I was able to get just over 20GB of ISO images in the document library on my test site and I started the copy processes. I followed the logs pretty closely during and after, here is how it starts:





The end result of the process was that:


The errors related to System Out of Memory issues:


During the process I watched the Task Manager and the MLMigrationLL.exe grab memory. I believe that when this process fails to grab memory in sufficient time the file copy fails.
I did not see this activity on the SharePoint 2007 server, but I might have just missed it.

I increased my test box to 3gb of memory and set SQL server to max at 1GB. I notices that the SQL server was picking up memory (as it should) during the last process and I want to make sure that there is sufficient for MLMigrationLL.exe.

You can see the memory ramp up when it gets to those files:


Plenty available:


Still getting out of memory exceptions though. This is a bummer. I will have to send these results to Jeremy:

Testing Tools and Addons - Metalogix SSMM with Large Document Libraries

Overview

I am looking at using Metalogix SharePoint Site Migration Manager in a new 2003 to 2007 migration project. I like the interface and flexibility of the program, but this client has large files and large document libraries which is a consideration for the migration process. In a typical migration I would move entire databases, with Metalogix I can rebuild the sites in to SharePoint 2007 one element at a time. I am on a tight schedule once the migration starts so I need to factor in how much time an item by item rebuild will take and what Metalogix will do when it encounters a 20 GB Document Library. First thing I need to do is prepare the test site in my VM environment.

I am putting 127GB drives on my test VMs for the migration:


Set the defaults in SQL Server to make sure databases are going to the E:\ I created:


Added a new content database in Central Admin and set the portal content db to “offline” so that new sites would be on my e: database.



Most of the content I am migrating is WSS 2.0 sites within the Portal so that is what I am creating:



Better make sure I ease up on upload limits, the sites that I am migrating have a 650mb upload limit because of the type of data they store. I am going to use iso images of various sizes in my test some of them really large. Let’s see if it stutters with a bunch of 600mb isos.

Copying files into the Document Library may take a while:

Testing Tools and Addons - Metalogix SharePoint Site Migration Manager Continued

Metalogix (Jeremy) sent me two messages in less than an hour with three different possibilities.

• Does the server use FBA?
• Is the firewall enabled?
• And check out this Microsoft article http://support.microsoft.com/kb/896861

The Microsoft Article addresses a loopback issue. I was able to browse sites from the server, but I chose method 2: disabling loop back and restarted the server.

Added registry key and rebooted:


Everything seems to work now. Excellent! That was pretty good response time!

Testing Tools and Addons - Metalogix SharePoint Site Migration Manager

Overview

There are a lot of tools and solutions out there for SharePoint and I spend a lot of time testing them so I am starting a new category for my blog on these trials and tribulations.

Today, I have been testing the Metalogix Migration Manager and I setup two All-in-One VMs. One is a x32 SharePoint Portal Server 2003 machine with Windows 2003 SP2. The other is an x64 SharePoint 2007 Enterprise machine on Windows 2003 SP2. I will probably proceed with a x64 Windows 2008 test later if I can run the program successfully with this bunch.

The program and extension installed smoothly on both machines. Each machine is part of its own domain and uses a single administrative account. Site Migration Manager was able to connect to both local hosts, but that’s pretty much where it ended. The SPS 2003 Machine was able to browse the SharePoint sites while the 2007 gives me an error.

Here is the 2003 box, I can get to it just fine from either server using the tool:



Oddly enough every time I try to connect to the sites on the SharePoint 2007 VM I get an error:



As a first troubleshooting step I apply an policy to the web application granting my account full control (I tend to shoot from the hip this wasn't part of the documentation):

I can browse the site just fine (before and after the policy)


I read through the documentation closely and the user account must be a farm administrator and have read/write access to the database:

Checked out my Farm Administrators in Central Admin and I was all set there:

I checked SQL as well and off course those permissions are all set.
I emailed the Sales Engineer, Jeremy, with the specs and versions of the systems. I am running this on 2007 6219 but I expect to go to SP2 once I have it working. Support is a really important for any package so we will see how fast they respond.

Implementing a Help Desk - Migrating Users between Domains

Overview

In a global environment users are often moving from regional domains. Most global organizations have domains for America, Europe, Asia, etc. Often when these users are moved SharePoint is overlooked. Migrating users is also essential when migrating SharePoint sites between domains during divestitures or acquisitions where it should be scripted in batch or Powershell. In short, if users are moving around between active directory accounts, the help desk should be ready with the migrate user operation.

Stsadm.exe –o migrateuser –oldlogin Domain\name - newlogin domain\name

In the past, I have batched this out and set it up as a process for help desk, but Powershell is the way to go and overall it’s much faster. This is what my novice expertise with Powershell has come up with:

$migrateusers = Import-Csv migrateusers.csv

foreach($entry in $migrateusers)

{

stsadm.exe -o migrateuser -oldlogin $entry.oldNames -newlogin $entry.NewNames

}

Configuring SharePoint - Full Text Indexes in SharePoint 2003 and WSS 2.0

Overview

This is an oldie but I am working on a SPS 2003 migration this week. Someone brought to my attention that a full text catalog was not working for a particular database. This is something you will only have to worry about for SPS 2003 and there are plenty of links on this: http://technet.microsoft.com/en-us/library/cc287630.aspx (Scroll Down to Full Text Search Issues)

If your full text search has failed, start off by doing an inspection of your database, look under storage for the Full Text Catalog:


Look into the properties of your catalog and check out the item count:


If its 0 or doesn’t grow when you add items you may have a problem. Check out the status:

The status should be idle unless you have given it something to do.

Check out the free space on the drive where you keep your indexes, and if that’s clear and you still have problems you may need to remove the full text index and recreate it. You can do this by executing the stored procedure proc_DisableFullTextSearch.
When you execute the procedure refresh your of the catalog and make sure that its gone:





Re-enable the search by executing proc_EnableFullTextSearch and refresh your view


The index should be back and its probably in an building stage. This can last pretty long if your database are large. If you still have difficulties, check your SQL Full Text Logs. They should be in the logs folder under your sql installation folder and they will start with the letters SQLFT.

Configure SharePoint - More MOSS SP2 Testing

Overview

I am doing more testing with SP2.

Progress

Apply my Testing SharePoint SP2 Checkpoint (IE Snapshot) to my virtual machine to test out some more solutions:

Make sure that I am indeed at the right version:





Before I deploy some custom solutions, I am going to deploy the Microsoft templates and solutions on this machine. I extract alltemplates.exe and use a batch file that runs my add solution / deploy solution commands:









If you look you can see the timer jobs go through:





And from solutions management you can see they are deployed:





Tested the Help Desk template and it seems fine. I will probably test a few more.





Typically I would trust the Microsoft solutions to work with a Service Pack, but I wouldn't test a Production environment until I tested the particular template.



I am taking another snapshot from here and I am going to test a solution from a custom product









This package installs customize branding…. The features appear to install and activate but I don’t see anything happen with the site:







So far I don’t see any changes, I could be missing some support files, but I don’t even see the folder structure this solution should create.... Out of time so I will have to save a snap shot and dig in deeper later.

Capacity Planning - SharePoint Disk Requirements

Overview

Maybe the most important and least understood part of designing a SharePoint solution is making sure that you have enough drive performance. I try to gather three key factors before making a judgment.

The Read / Write Ratio

This is really tricky with SharePoint, a Collaboration file Sharing farm is going to have a different requirement than a SharePoint Publishing Portal with mostly static pages. Microsoft has some guidance from WSS 2.0. You can look at the various read and write operations from the link. A generous conclusion would be 3 to 1 which I go with when in doubt. Some practical considerations might be your file size upload limit and site collection size limits. There is also a lot of overhead in the operation:

“Note: There are roughly two supporting transactions for each end-user transaction. In other words, the end-user operations make up about a third of the total transaction load on the server.”
http://technet.microsoft.com/en-us/library/cc767371.aspx

IOPS Requirements

IOPS Requirements are really well defined by Microsoft on the SharePoint Best Practices Home page



http://technet.microsoft.com/en-us/library/cc850692.aspx

Let’s take an example 10TB farm where 3TB are dedicated to OS, Transation Logs and the Search etc at 2 IOPS/GB and 7TB comprises the content databases. Our requirements will equate to:

3000 x 2 IOPS/GB = 6000 IOPS
7000 x .75 IOPS/GB = 5250 IOPS


Raw Disk Performance

When determining IOPS how quickly the read/write head on the disk can access the data is very important, there is a really good article at the link below I use for the calculation which takes into account our variables.
“

http://users.ca.astound.net/baspence/AIXtip/sizing_disk_for_io.htm”

So with 15k disks we would require:

((.66 +(2 *.33))(6000))/150 = ~53 Disks for OS and Everything Else
((.66 +(2 *.33))(5250))/150 = ~46 Disks for Content Databases

Disk Size: 3000/53 = 56GB Disk Size
Disk Size: 7000/46 = 152GB Disk Size

Of Course you can buy larger disks but as you use more space your performance will degrade.