Blog of YorZ

Fresh from the Oodle blog - Craigslist content is no longer being included in the Oodle index. Evidently, Craigslist requested that its content not be crawled by Oodle going forward. It’s not yet clear if Craig is opting-out in response to some especially naughty act by team Oodle or if Craigslist just decided Oodle’s approach was more of a threat than a helpful traffic source.  Without knowing the motivation, we don’t see how this is good for consumers, but we can clearly see why it is good defensive move for Craigslist and bad news for Oodle.

A lot of folks are asking “why block Oodle and not the other aggregators (e.g., Indeed and SimplyHired) or even Google / Yahoo?” Indeed (no pun intended), the other aggregators must be asking this question and hoping that Craigslist doesn’t extend the blockade. As our last post suggested, it presents a huge problem (at least in the near-term) for the aggregators if the major boards opt-out of their indices – especially Craigslist given that it’s the biggest classifieds site.

The discussion on John Battelle’s Searchblog is also interesting. One reader speculates on the reason why Craigslist may be blocking Oodle:

“If one scraper is using too much from one source it DOES become infringement and competition. I’m not familiar with Craigslist’s specific beef with Oodle, but I suspect that’s it.”

Since the job-specific aggregators aren’t indexing all of Craigslist’s classifieds – only the job listings – perhaps Oodle will be the only man thrown overboard.

And, in case you were wondering, YorZ has no intention of blocking any of the aggregators from crawling our site at this time.

Despite all the criticism of job boards, it seems that many employers favor them even above their own websites. While we haven’t done a rigorous review of the posting patterns of employers on the boards, there’s ample anecdotal evidence. For instance, in looking at the 20 most recent listings for software programmers posted this morning on Craigslist San Francisco and Monster (10 from each site) we found that 13 (65%) listings were not found on the corporate website. The remaining 7 (35%) were found on the corporate website, but the listings aren’t always identical. We found a few examples of the job board listing including critical information that was omitted from the version on the corporate website – one corporate website version neglected to mention that a position was for the graveyard shift.

We’re not sure why companies don’t keep their listings in sync with what they post to job boards, but most of them must not see a positive enough return on the time investment it takes to keep the corporate website up to date. Maybe they don’t think people visit the jobs section of their site. Maybe it’s just too difficult to make changes.

It raises a couple other questions for us, too:

1. Why haven’t the ATS companies been more successful here? Many good companies sell great ATS products.
2. How credible is it that the job aggregators can exist without the cooperation of the major job boards?  Crawling corporate websites as the primary source of content seems like a flawed back-up plan if companies don’t start keeping their corporate websites up to date. And even if rising traffic from job aggregators gives employers more incentive to post open positions to their corporate websites, what confidence should job seekers have that stale listings have been taken down?