Turning "Paste a URL" Into Structured Data
One of the more interesting engineering problems I've worked on at EpicHire has been something deceptively simple on the surface: a recruiter pastes a job posting URL into our platform, and a few seconds later we have a fully structured job listing (title, company, location, compensation, job type, categories, and a nicely formatted description) ready to publish.
The "few seconds later" part is doing a lot of work. Under the hood, that single text field kicks off a pipeline that fetches the page, figures out what kind of site it's even looking at, extracts the real content, and hands it to Claude to turn into clean structured data. What started as a 274-line proof of concept grew into a system that now reliably handles job postings across dozens of different Applicant Tracking Systems (ATSes). These are the platforms companies use to host their job boards, like Workday, Greenhouse, and Lever.
This post is about how that system evolved, what broke along the way, and what I learned building it. I'm not going to walk through the exact implementation because a lot of that is proprietary to EpicHire, but the architecture and the lessons are things I think are worth sharing.
V1: The naive approach
The first version was about as simple as you'd guess. Given a URL:
- Fetch the raw HTML
- Strip out obvious junk like nav bars, footers, and cookie banners
- Grab whatever looked like the main content area
- Dump the cleaned text into Claude and ask it to extract everything
Claude was responsible for the entire job: title, company, location, compensation, job type, and even a summary of the role that it wrote itself. That last part is the detail that eventually forced a rewrite.
This worked... okay. For simpler, mostly-static job pages, it was fine. But a huge chunk of the job postings coming through the platform were failing silently or coming back with garbage data. The common thread was single-page applications (SPAs). These sites have the actual job content rendered client-side by JavaScript after the page loads. A basic HTML fetch just gets you an empty shell. No amount of clever text cleanup fixes that, because there's no text there to clean.
V2: Going direct to the source
The real unlock was realizing that most major ATS platforms expose their own APIs under the hood that the frontend JavaScript calls to render the page. If I could identify those calls myself, I could skip HTML scraping entirely for the platforms with the most volume and get clean, structured data straight from the source.
This became a general pattern: try a direct integration first, and only fall back to generic HTML scraping if that fails.
ExtractedContent content = tryFetchFromKnownSource(url);
if (content == null) {
content = fetchAndCleanHtml(url);
}
Simple in concept, but each platform turned out to have its own quirks, and figuring out the pattern each one used was the most fun part of the project. I'd open dev tools, watch the network tab, and note what requests the page itself was making to load its own data.
One recurring gotcha: some platforms put the job ID in the URL fragment (the part after #), which browsers never send to the server at all. If I tried to fetch the page normally first and parse the ID out afterward, it was already gone. The fix was pulling it straight out of the raw URL string before making any request:
// The job ID lives in the URL fragment, which never reaches the server.
// Has to be extracted from the raw URL string before any network call is made.
private static final Pattern FRAGMENT_JOB_ID =
Pattern.compile("#/details/([A-Za-z0-9+/=_*@!~.-]+)");
Matcher m = FRAGMENT_JOB_ID.matcher(url);
if (m.find()) {
String jobId = m.group(1);
return fetchFromPlatformApi(jobId);
}
Another platform's API would redirect me into an infinite loop no matter what I tried. Turned out it was validating the Referer header before serving data and the HTTP client I'd started with silently dropped that header on redirect. Switching clients fixed it in about five minutes, but finding that took a while:
// This API checks for a Referer header, which our original HTTP client
// wasn't preserving through redirects. Switching clients fixed it.
HttpClient client = HttpClient.newBuilder()
.followRedirects(HttpClient.Redirect.NORMAL)
.build();
HttpRequest request = HttpRequest.newBuilder()
.uri(URI.create(apiUrl))
.header("Referer", originalJobPageUrl)
.GET()
.build();
HttpResponse<String> response =
client.send(request, HttpResponse.BodyHandlers.ofString());
A third platform didn't expose any API at all. The data turned out to be bootstrapped directly into the page as an inline JavaScript object, and getting that page to render server-side (rather than the empty SPA shell) required first hitting a session-check endpoint to pick up a cookie:
// Step 1: hit a session-check endpoint first to get a cookie the real page needs.
Map<String, String> cookies = Jsoup.connect(sessionCheckUrl).execute().cookies();
// Step 2: fetch the actual page using that cookie.
Document doc = Jsoup.connect(jobPageUrl).cookies(cookies).get();
// The job data isn't a JSON API response, it's bootstrapped inline as a JS
// object literal inside a script tag. Regex can't safely handle arbitrarily
// nested braces, so we walk them by hand to find the matching close.
int start = scriptText.indexOf('{', scriptText.indexOf("bootstrapFunctionName("));
int depth = 0, end = -1;
for (int i = start; i < scriptText.length(); i++) {
char c = scriptText.charAt(i);
if (c == '{') depth++;
else if (c == '}' && --depth == 0) { end = i; break; }
}
JsonNode data = objectMapper.readTree(scriptText.substring(start, end + 1));
None of this is documented anywhere by these platforms. It's all inference from watching real traffic.
At the same time, I fixed something that had been bugging me since v1: Claude was writing its own summary of each job instead of using the recruiter's actual description. That's obviously not what you want in a hiring platform. So v2 also introduced pulling the actual job description HTML and converting it into the rich-text format our app uses internally, preserving bold text, headers, and bullet lists instead of collapsing everything into a flat paragraph. Claude's job shifted from "write a description" to "extract structured fields from the real one."
V3: The description was too greedy
The next problem showed up once more job boards were flowing through the pipeline: on a number of sites, the logic for finding "the job description" was too generous. It would grab an entire content container that included the actual description plus surrounding page elements like related job listings, footer text, sometimes an entire "why work here" marketing block. Formatting that into rich text looked awful, because rich text formatting assumes you're formatting a description, not an entire webpage.
The fix was a more deliberate content-targeting strategy: a prioritized list of known selectors for platforms I'd already seen, and if none matched a fallback heuristic that scores candidate elements by how much they look like a description and picks the densest one:
private Element findDescriptionElement(Element container, Document doc) {
String[] knownSelectors = {
"[class*='job-description']",
"[id*='job-description']",
"[class*='posting-content']",
// ...a couple dozen more, one per platform we'd run into
};
for (String selector : knownSelectors) {
Element match = container.selectFirst(selector);
if (match != null) return match;
}
// No known selector matched, score each candidate block by how many
// paragraph/list/heading children it directly has, and pick the densest.
Element best = null;
int bestScore = 2;
for (Element candidate : container.select("div, section, article")) {
int score = countDescriptionLikeChildren(candidate);
if (score > bestScore) {
bestScore = score;
best = candidate;
}
}
return best != null ? best : container;
}
This was a bigger quality jump than it sounds. Job description formatting went from "usually pretty rough" to "looks right basically every time."
I also leaned more on a public standard that some job pages already implement: schema.org's JobPosting structured data, embedded as JSON-LD. When it's present, it's a clean, standardized source for title, company, and location with no scraping heuristics needed:
// Many modern job pages embed structured data using the public schema.org
// JobPosting format: a clean, standardized fallback when there's no direct
// API integration for that platform.
for (Element script : doc.select("script[type='application/ld+json']")) {
JsonNode ld = objectMapper.readTree(script.data());
if ("JobPosting".equals(ld.path("@type").asText(null))) {
String title = ld.path("title").asText(null);
String company = ld.path("hiringOrganization").path("name").asText(null);
// ...pull location, employment type, etc. the same way
}
}
Scaling out
From there it became a matter of volume. Every new enterprise client meant a chance their job board ran on a platform I hadn't supported yet, so I kept adding direct integrations. By now the system supports close to a dozen distinct platforms directly, with generic HTML extraction as the catch-all for everything else.
The high-level shape of the whole thing looks roughly like this:
parse(url)
└─ normalize the URL (some platforms need URL cleanup first)
└─ try known platform integrations, in order
└─ if one succeeds → structured data, done
└─ [fallback] generic HTML fetch + cleanup
└─ look for structured metadata (schema.org JobPosting, if present)
└─ isolate the real description content
└─ fall back to a content-density heuristic if no known pattern matches
└─ hand everything to Claude for final structured extraction
Claude sits at the end of that pipeline doing what it's actually good at: reading messy, unpredictable text and turning it into consistent structured output including title formatting, location parsing, compensation ranges, and job type classification. The scraping and platform-specific logic in front of it exists entirely to make sure Claude is working from clean, complete input instead of having to guess around missing or noisy data.
What I'd tell past me
A few things I took away from this:
- SPAs mean "find the real API," not "give up and scrape." Almost every SPA has a JSON API or an inline data blob feeding it. It's just a matter of finding it.
- Never let an LLM regenerate content you can extract verbatim. My first instinct was to let Claude summarize the job description, which felt efficient but was actually a correctness problem in disguise. Recruiters need their actual posting to show up, not an AI's version of it.
- "Find the description" is a real algorithm problem, not a scraping afterthought. A prioritized selector list plus a density-based fallback covered both the platforms I explicitly planned for and the ones I didn't.
- The fallback pattern (try the specific thing, then the generic thing) scales really well. It let me add new platform support incrementally without ever having to touch the generic path.
This system now runs in production for EpicHire, handling job postings from Fortune 500 companies on the platform without manual intervention. It's a good example of what I like about building with LLMs in the loop: the model isn't doing the whole job, it's doing the one part it's uniquely good at, sitting on top of a lot of deliberate, boring, correct engineering that makes sure it has good input to work with.