Spidering jobs – a necessity, luxury, or neither?
Getting jobs from an employer and onto a job board has been a challenge from the earliest days of the industry. Initially, the only way to do so was manually – the employer filled out a web form and submitted the job ad to the job board. Needless to say, this method (which, of course, is still widely used) was time-consuming and provided many opportunities for typos, dropped information, and other inaccuracies.
Needless to say, job boards didn’t like this arrangement – the harder it was to post jobs, the less likely their customers were to buy more job ads than absolutely needed. Employers didn’t like it, for it was tedious, slow, and yet another task for their (under-staffed) HR departments.
As more employers created their own websites, the first section they added – after the obligatory picture of their dynamic management team, of course – was a careers area. In addition to tidbits about benefits and working conditions, the careers area had job postings. These were created in exactly the same manner as those posted to a job board – via a simple web form. Almost immediately, HR managers everywhere began to wonder: “Why should I enter this information twice? Why not figure out some way to have it automatically transferred to the job board of my choice?”
Solutions quickly emerged. With a bit of programming, jobs could be ‘fed’ to job boards vis a RSS or XML file. For many companies, however, the ‘bit of programming’ was a stumbling block – they lacked programmers, time, money, or all of the above. Occasionally job boards would jump in and create the feeds for their clients. But by and large the responsibility fell on the employer to either create a feed or enter jobs the old-fashioned way.
Then along came a spider (sorry, couldn’t resist).
Spidering jobs is the ultimate ‘hands-off’ solution as far as employers are concerned. The software visits the employer site on a periodic basis (usually every 1-3 days), gathers the job data, then posts it directly to the job board. From the job board’s perspective, spidering involves either working with a 3rd party vendor or utilizing internal tech time – each time an employer’s site is added, the spider must be set up to read the correct areas of the site, then match the data fields to corresponding fields in the job board’s job posting database. In a perfect world, the spider is set up and then it just runs. In reality, spiders often require ‘fine tuning’ before they work – and if the employer changes their careers area, the spider must be changed as well.
So – is spidering jobs (or site scraping, or whatever you prefer to call it) a ‘must’ for job boards? A luxury? Or neither?
Well…it depends. Are you losing clients because they find it difficult or tedious to post jobs on your site? Are you losing revenue by making the posting of multiple jobs more difficult? Are your competitors offering spidering? (Don’t forget – some aggregators actually spider employer sites prior to making a sales call). The answers to these questions will determine whether spidering jobs is a nice option – or a necessity.
[Want to get Job Board Doctor posts via email? Subscribe here.]. [Check out the JobBoardGeek podcast archive!]
This Post Has 8 Comments
Comments are closed.
Spidering, scraping, wrapping, or whatever you want to call it is becoming less and less important for job boards as more and more employers use applicant tracking systems and other services which are integrated with many of the leading job boards.
Our site, for example, receives postings through these intermediaries for hundreds of employers. We strongly prefer these feeds over spidering because the end results tend to be much more accurate. If we spider then how the job gets posted to our site is only as accurate as the person who programs the spider and, even more concerning, only as accurate as allowed by the employer’s web site.
One of the biggest challenges is that a spider created perfectly today can be brutally inaccurate tomorrow if the employer makes even a small change to the way they display their jobs. If the employer switches the location of the compensation field with the number of hours required per week field and fails to notify the job board then the posting will show up properly on the employer site but the job board will show the hours required in the compensation field and the compensation in the hours required field.
Steven, I agree that ATSs can produce more accurate data for your site. But ATSs are still far from universal – utilization varies widely from industry to industry. You’re also correct that when employers change their job postings, the spiders have problems. Sometimes I wish it could just happen via ‘magic’! But (oddly enough) my wish has not yet come true.
Jeff is right – ATSs are far from universal and scraping jobs directly from company websites gives a lot of jobs visibility they would otherwise never get. Scraping jobs directly from legitimate company websites also helps filter out a lot of the fake job ads that appear on many job boards. How gracefully the spidering software handles changes, like the company switching their ATS for example, provides an opportunity for companies utilizing these methods to differentiate themselves from their competitors.
Hi Jeff,
do you have a list of job spidering software options?
Thanks for the article!
Daniel, that’s a very good question – no, I don’t. But I should! I’ll work on it. – Jeff
I think, as spiders become more popular and companies pop-up that provide job spidering as a service (there have been some already), we’re going to see the technology evolve to the point where it is as accurate as an XML feed. The same thing happened with resume/CV parsers and I don’t see a reason it won’t get to that point as spidering services mature. The variety of scenarios they encounter ends up being the catalyst for those types of improvements. How soon that happens though? That’s yet to be seen. With the general availability of XML feeds and spreadsheet imports, I’m not sure how quick the industry is going to push that direction. It’s enough of a sales tool that we’ve started to do it. It’s not quite a “necessity” but it’s definitely nice to be able to offer and it certainly helps with sales as it really is the past of least resistance for an aggregator. Until the technology improves, I don’t see a lot of smaller niche job boards bothering with it yet.
We have been offering this service to Job Boards for years. Steven is correct, when sites change the spiders break, however that’s where good error handling comes in. We have developed a solution that allows us to fix any broken spiders within 24 hours. This takes away the headache from the Job Board owners and allows them to do what they do best.
Spiders are very much needed i have noticed many companies are taking advantage of this and charging to index jobs via xml feed?