The premise of my need is that I have WebPages that have content for everyone, and content customized for the specific user viewing the webpage.  I lump this user specific content under the term “Personalization”.

Personalization is a powerful mechanism for making the Web more useful for people.  Most of us are familiar with the personalization on Amazon.com where they target items to you based on your viewing and buying habits.

On www.VastRank.com (a College Review Website site that I created) when a user is viewing a College Profile page, they are also shown partial college reviews for other colleges that they may be interested in.  I go deep into the personalization implementation in my Google I/O presentation Using AJAX APIs to Navigate User-Generated Content.  Here is a slide from that deck that illustrates the “personalized” suggestions that are shown to users on the right hand side of college profile pages.

The main issue here is that Googlebot is now going to see partial user reviews for other colleges on different college profile pages. This causes two major issues:

• Duplicate content problems
• Text from other college reviews show up in the search results for Other College Profile pages in the Google.com Search

Should Google support robots-nocontent?

One implementation that would solve my problem would be if Google implemented Yahoo’s robots-nocontent tag.  I recently tweeted back and forth on this topic with Google’s Matt Cutts on Twitter:

Jon Kragh

“Google (@mattcutts) please support Robots-Nocontent – I have partial content tailored for each specific user, that should not be indexed”

Matt Cutts

mattcutts@jonkragh we looked at how many sites use robots nocontent on the web and it was miniscule, so we decided not to do it.

Jon Kragh

@mattcutts if Google blessed Robots-Nocontent more people would use it-should personalized content (i.e. suggestions) be loaded via AJAX?

So at this point, it looks like robots-nocontent is not one of Google’s top priorities.

robots-nocontent is not right either

Initially I looked at robots-nocontent because it was the closest thing I could find for a partial content indexing solution.  However, robots-nocontent is so general, that it might not be the best long-term solution for the web.  Looking back at what I have conceptually, I have content for everyone, and content for a particular user.

A better solution than robots-nocontent would be to create a new tag for this scenario that has more meaning.

robots-user-specific-content

I suggest a new tag that can denotes content that will be different for each user that visits the website: robots-user-specific-content.  For the time being Google could ignore that content, much like an implementation of robots-nocontent would work.  However, having this extra meaning would allow for possible extensions to Google in the future, where a user could search with their identity and get results back from websites with content personalized for that user.

What can I do now?

For now I am stuck!  I am considering loading personalized content through AJAX to avoid having it indexed.  However, this would be a guess on my part on Google’s indexing algorithm.  Will Google index AJAX content?  Will it penalize me because I’m loading different content in a section of a page via AJAX each time Googlebot visits my site?  This is where I would like your feedback, and hopefully some guidance from Google!

Cheers,

Jon

How did I get noticed by Google?

Recently, I saw a beer commercial that said “Rule #1 of how to get the girl: be good looking!”.  So here is my rule of how to get noticed by Google “Rule #1 of how to get noticed by Google: use the Google APIs in some kick-ass way!”.  Google is building a developer platform and they want to show off developers utilizing said platform. So if you do something cool with the Google APIs, it is easier to get noticed than you may think.

How did I get asked to present at Google I/O?

My main “in” at Google was that anytime that I did something rather cool with the Google APIs, I pinged the Google Employee Developer I met in the Google Developer Newsgroups.  I did not expect anything in return from him. I thought “hey I just implemented something really cool and they might be interested”. It turns out that they were indeed interested and extremely friendly.  Every time that I implemented something substantially new and cool, we would have an open dialog about what would make my application better and they would say things such as “hey did you know we have the Client Location API, this would be really cool for your site…”.  It turns out that I must have pinged my Google contact at the exact right time (with information about major updates to my site).  I had just completed some advanced Google Maps coding and once they saw it, they officially invited me to Google IO to be in the Developer Sandbox and to present a session.

Preparing my presentation

So now there I was, Google asked me (OMG!) to come out and speak and they wanted me to send out a rough concept of what I might want to present.  Initially I proposed a session called “Building a cool app on an hour a day, a rebel’s view of the world”.  That presentation would have been all about the mindset that I had and steps I took to successfully building www.VastRank.com in my spare time.   Google liked that presentation, but they wanted it to be more focused on the APIs since that is what a lot of visitors to Google I/O would be looking for.

So I changed my presentation to “Using AJAX APIs to Navigate User-Generated Content”.  The presentation was about how do you take all of this information that users have submitted and sort and present it to other users in an interesting and useful  way.  The title of the session was play on a Don Box title “Navigating the Programmable Web”.

Rehearsing with the Google AJAX API Team

Once I had my presentation prepared, I rehearsed it over two video conferences with Adam Feldman, the Google AJAX APIs Product Manager.  Adam was extremely smart, and had a way to hear what I was saying and then summarize some things into a much more compelling concise point.

I arrived to the conference a day early, where I was able to do my presentation in the actual conference room to a group of Googlers.  I  am so glad that I had that opportunity, because I rushed through my first live run and was 15 minutes short of the target time. I needed to slow down the next day!

The Session: Using AJAX APIs to Navigate User-Generated Content

Here is a link to the full to the official Google I/O Session page which has the video and slides.

• 00:24 – Who am I?
• 01:26 – Introduction to VastRank.com
• 02:56 – Overview of the Google AJAX APIs Used
• 07:15 – Using Google Maps on the College Profile
• 11:41 – VastRank.com Client Architecture
• 12:39 – Setting the Map Center
• 14:07 – Loading Map Markers via AJAX
• 15:07 – Algorithm for What is in View on a Map
• 28:25 – Map InfoWindows
• 30:38 – Geocoding via AJAX and HTTP
• 34:45 – Personalization & Suggestion Engine (ClientLocation)
• 39:28 – Translation / Languages API
• 45:01 – Questions

The After Party

Google I/O 2009 was a two day event.  I presented on day one and at the end of day one Google through a massive after party.  Here I took a quick video near the end of the evening.

Cheers,
Jon

Recently I was invited by Google to attend Google I/O 2009.  Google invited me to show off Vast Rank in the Developer Sandbox, and also present a session to attendees: Using AJAX APIs to Navigate User-Generated Content.

Soon, Google will be posting my session on YouTube.  I am anxiously waiting for that! In the meantime, here is a very quick impromptu interview of me speaking about Vast Rank and my use of the Google APIs.  The interviewers provided me with the questions ahead of time, but I did not have enough  time to read them since I was so focused on preparing for my session.  So here I am shooting from the hip, answering questions about the Google APIs.

At the time I was worried because I was unsure of what would happen next.  This was the first time I really cared about SEO.  When I worked at barnesandnoble.com, people would just show up and pump millions of dollars of products through my code a day.  It was magic, and I thought, dam I am good!  Well, the reality is they have a brand, and I do not!

If your site does not appear in Google Search results, it is pretty much death to your site if you depend on organic traffic. Google matters much less if you have a website that people have to use for their school, job, bank account or whatever.  But when you depend on organic traffic, you rely on Google.

If your site drops in SERP, the Google webmaster tools do not tell you in general about any penalties on your site.  I imagine that this is so that crafty little SEO experts can not reverse engineer some of Google’s algorithms.  So during these times of doom and gloom for your site, you are pretty much left to looking at SEO discussion sites and the Google Webmaster Help discussion group.

Luckily, Vast Rank returned to Google’s search results after a huge drop from October 26th, 2008 – November 5th, 2008.  I am posting this to show other webmasters that there is hope if your new site suddenly disappears from Google search results.   So take a deep breath, the world is not ending, and wait it out! I’m serious,  take a deep breathe, and count 1000-1, 1000-2, 1000-3. Now Exhale.  Calm yet?  If not, go here, or keep reading, I have proof that there is hope.

Below is a screen screen shot from Vast Rank’s Google Analytics page.  The screen shot shows traffic from launch day until today (notice the huge drop in traffic in October-November).

I updated quite a few things on Vast Rank during the downtime period: accurate sitemap last mod dates, unique title tags, better meta descriptions, etc.  Some of these items were called out in the Google Webmaster Tools for Vast Rank, and other items were just common sense items that I needed to get to implementing (i.e. accurate last mod dates in the xml sitemap).  However, I think the biggest reason I saw the huge drop was that Vast Rank was so new.

If you are a webmaster experiencing SERP drops, it would be great for you to post a comment so we can collectively share our experiences!

I have been a Microsoft oriented developer all of my life so when I built Vast Rank, I wanted to find a decent profile system and forum system  based on .NET that I could build on top of.  Community Server was by far the best platform to use when I started developing Vast Rank in 2007.

One of the challenges that I faced was that I wanted to extend the Community Server User Profiles in such a way that those profiles can be queried based on the extended properties.  Before you follow this post any further, if you are considering going down path, ask yourself “do I really need to query these extended attributes?”, because it gets complicated.  If you don’t need to query this extended user profile data, but just store and display it, then I suggest using the out of the box ExtendedAttirute storage mechanism that comes with CS. It is a heck of a lot less work.

Anyway, I spent the time extending user profiles and detailed it in an old post here. In that post I went through changing the source code in the CS SDK, but here I will post a few blurbs from the CS forums that I posted about how to extend CS User profiles without changing any CS SDK source code.  I originally did these modifications in CS 2007, then I rolled them into CS 2008.5 in one day.  (Some of the CS code changed a little but it was easy to fix up my extensions to align with these changes).  So the big payoff of using this new strategy: ease of upgrade to new versions of CS.

The basic “A Ha” moment for this new extension strategy was, hey doesn’t CS use the provider model?  Maybe I can hook in at that level to override and extend the CRUD and search operations.  Well that worked.

I am posting some CS specific source code directly from Vast Rank here:

Download

Since so many people were interested in more details around this new strategy, I slotted an hour out of the many other dev things I’m working on to put this up for my friends over in CS land. I figured it was better to get something into your hands than nothing at all…

So… the code posted does not compile because it depends on Community Server (which you can download from them) and my own Service / Data Access Layer (which I’m not posting:) ).  But, if you are pretty good dev you can look through the code and get some “A Ha” moments.

The code has a sample configuration where I load my overridden version of the the CommonDataProvider.  That file in my project is called “VRSqlCommonDataProvider” and it shows how I extended the CRUD and search code.  I also included a few of my theme files where you can see where I integrate the extended data (creating, editing, viewing, and searching profiles).  There are also some sub forms and helpers in there.  You’re just going to have to dig through…

Here are a couple of links to my “A HA” moments from the CS forums. You can test these things live at http://www.vastrank.com

• My “A HA”, I bet I can extend CRUD operations via the provider layer: here: Extended User Data – Might have a better way
• My “A HA”, I can search this crap too but it’s complicated (I chimed in on this thread):  Expanding Member Search

Please chime in on this post if this helps ya, or if you have any feedback.  As I said it’s a little sloppy but I wanted to get something into my friends hands over on the CS forums.

Vast Rank was steadily growing since its launch (a month ago) and last week was at its peek with a lot of organic (search) traffic from Google.

I saw this drop start on Saturday, October 25th and as a result I have started to dig into the Google Webmaster Guidelines and the world of SEO.  It looks like Google has algorithms that determine whether your site is “spammy”, and may penalize a site that it thinks is spam.  Since Vast Rank is so new and has a fairly low Page Rank, I am concerned I may have fallen prey to this algorithm.

The tough part about all of this is Google may or may not tell you if you are under one of these penalties.  I believe this is so that people can not use these warnings to reverse engineer their algorithms.

For the most part, I am hoping that my major drop in SERP is because my site is barely a month old.  However, I am taking a serious look at Vast Rank to see where I can improve things.  Here are my top tasks:

• Create accurate last mod dates in my SiteMap for all College Profiles and for the ratings & comments pages
• Make sure my title Tags are unique
• Review all meta description tags

The accurate last mod dates are going to be the most challenging but should only take a few hours to get it all setup.

Edit: I posted an update to my findings here.

Featured Projects are somewhat buried on the Google API pages.  The first screen shot here shows where Vast Rank is featured on the Google Maps API page.  The second screen shot below shows Vast Rank featured on the Google AJAX search API page.

Special thanks goes out to Ben Lisbakken on the Google API team for being so helpful with my questions and for hooking me up with the featured spot on Google.