The premise of my need is that I have WebPages that have content for everyone, and content customized for the specific user viewing the webpage.  I lump this user specific content under the term “Personalization”.

Personalization is a powerful mechanism for making the Web more useful for people.  Most of us are familiar with the personalization on Amazon.com where they target items to you based on your viewing and buying habits.

On www.VastRank.com (a College Review Website site that I created) when a user is viewing a College Profile page, they are also shown partial college reviews for other colleges that they may be interested in.  I go deep into the personalization implementation in my Google I/O presentation Using AJAX APIs to Navigate User-Generated Content.  Here is a slide from that deck that illustrates the “personalized” suggestions that are shown to users on the right hand side of college profile pages. 

The main issue here is that Googlebot is now going to see partial user reviews for other colleges on different college profile pages. This causes two major issues:

• Duplicate content problems
• Text from other college reviews show up in the search results for Other College Profile pages in the Google.com Search

Should Google support robots-nocontent?

One implementation that would solve my problem would be if Google implemented Yahoo’s robots-nocontent tag.  I recently tweeted back and forth on this topic with Google’s Matt Cutts on Twitter:

Jon Kragh

“Google (@mattcutts) please support Robots-Nocontent – I have partial content tailored for each specific user, that should not be indexed”

Matt Cutts

mattcutts@jonkragh we looked at how many sites use robots nocontent on the web and it was miniscule, so we decided not to do it.

Jon Kragh

@mattcutts if Google blessed Robots-Nocontent more people would use it-should personalized content (i.e. suggestions) be loaded via AJAX?

So at this point, it looks like robots-nocontent is not one of Google’s top priorities.

robots-nocontent is not right either

Initially I looked at robots-nocontent because it was the closest thing I could find for a partial content indexing solution.  However, robots-nocontent is so general, that it might not be the best long-term solution for the web.  Looking back at what I have conceptually, I have content for everyone, and content for a particular user.

A better solution than robots-nocontent would be to create a new tag for this scenario that has more meaning.

robots-user-specific-content

I suggest a new tag that can denotes content that will be different for each user that visits the website: robots-user-specific-content.  For the time being Google could ignore that content, much like an implementation of robots-nocontent would work.  However, having this extra meaning would allow for possible extensions to Google in the future, where a user could search with their identity and get results back from websites with content personalized for that user. 

What can I do now?

For now I am stuck!  I am considering loading personalized content through AJAX to avoid having it indexed.  However, this would be a guess on my part on Google’s indexing algorithm.  Will Google index AJAX content?  Will it penalize me because I’m loading different content in a section of a page via AJAX each time Googlebot visits my site?  This is where I would like your feedback, and hopefully some guidance from Google!

Cheers,

Jon

At the time I was worried because I was unsure of what would happen next.  This was the first time I really cared about SEO.  When I worked at barnesandnoble.com, people would just show up and pump millions of dollars of products through my code a day.  It was magic, and I thought, dam I am good!  Well, the reality is they have a brand, and I do not!

If your site does not appear in Google Search results, it is pretty much death to your site if you depend on organic traffic.  Google matters much less if you have a website that people have to use for their school, job, bank account or whatever.  But when you depend on organic traffic, you rely on Google.

If your site drops in SERP, the Google webmaster tools do not tell you in general about any penalties on your site.  I imagine that this is so that crafty little SEO experts can not reverse engineer some of Google’s algorithms.  So during these times of doom and gloom for your site, you are pretty much left to looking at SEO discussion sites and the Google Webmaster Help discussion group.

Luckily, Vast Rank returned to Google’s search results after a huge drop from October 26th, 2008 – November 5th, 2008.  I am posting this to show other webmasters that there is hope if your new site suddenly disappears from Google search results.   So take a deep breath, the world is not ending, and wait it out! I’m serious,  take a deep breathe, and count 1000-1, 1000-2, 1000-3. Now Exhale.  Calm yet?  If not, go here, or keep reading, I have proof that there is hope.

Below is a screen screen shot from Vast Rank’s Google Analytics page.  The screen shot shows traffic from launch day until today (notice the huge drop in traffic in October-November).

I updated quite a few things on Vast Rank during the downtime period: accurate sitemap last mod dates, unique title tags, better meta descriptions, etc.  Some of these items were called out in the Google Webmaster Tools for Vast Rank, and other items were just common sense items that I needed to get to implementing (i.e. accurate last mod dates in the xml sitemap).  However, I think the biggest reason I saw the huge drop was that Vast Rank was so new. 

If you are a webmaster experiencing SERP drops, it would be great for you to post a comment so we can collectively share our experiences!

Vast Rank was steadily growing since its launch (a month ago) and last week was at its peek with a lot of organic (search) traffic from Google.

I saw this drop start on Saturday, October 25th and as a result I have started to dig into the Google Webmaster Guidelines and the world of SEO.  It looks like Google has algorithms that determine whether your site is “spammy”, and may penalize a site that it thinks is spam.  Since Vast Rank is so new and has a fairly low Page Rank, I am concerned I may have fallen prey to this algorithm.

The tough part about all of this is Google may or may not tell you if you are under one of these penalties.  I believe this is so that people can not use these warnings to reverse engineer their algorithms.

For the most part, I am hoping that my major drop in SERP is because my site is barely a month old.  However, I am taking a serious look at Vast Rank to see where I can improve things.  Here are my top tasks:

• Create accurate last mod dates in my SiteMap for all College Profiles and for the ratings & comments pages
• Make sure my title Tags are unique
• Review all meta description tags

The accurate last mod dates are going to be the most challenging but should only take a few hours to get it all setup.

Edit: I posted an update to my findings here.