From raw job postings to searchable records and filtered insights.
A personal engineering project spanning a Python ingestion pipeline and a Next.js website. It combines recoverable crawling, versioned parsing, AI extraction checked against source excerpts, and a compact serving database. Visitors can search and filter jobs by structured requirements and skills, then explore statistics for the same selection.
Job listings are spread across websites with different page structures, fields, and ways of navigating results. JobLake collects those pages, keeps the raw input, and transforms it into records that a small search application can serve.
Why build it?
The project is a practical way to work through an entire data lifecycle: discovery, collection, validation, storage, and retrieval. The interesting part is connecting these steps while keeping enough state to understand failures and resume work.
Scope: a personal learning project with a live website. The architecture demonstrates implemented behavior; it is not a claim of large-scale production operation.
Actual JobLake interface, captured on 24 September 2026. Listings change over time.
02Engineering notes
A clear boundary between collection and serving.
The pipeline runs with local PostgreSQL and MinIO. PostgreSQL is authoritative for crawl state and normalized records. A separate synchronization operation publishes a compact active-listing view to Supabase PostgreSQL; the Next.js website reads that view on Vercel.
Local PostgreSQL→Manual sync→Supabase serving→Next.js on Vercel
Airflow orchestrates ingestion and an independent, optional AI enrichment phase. Validated requirements and normalized skill keys join current parsed records. A separate sync publishes active listings to power search, skill filters, and statistics with data coverage.
The data flow
Discover: source adapters traverse listing pages and register canonical URLs in PostgreSQL crawl state.
Fetch: pending detail URLs are fetched and validated before their HTML is accepted into MinIO.
Parse: parsers read retained objects, verify their integrity, extract fields, and record validation issues.
Persist: accepted or partial results are saved with parser identity and raw-data provenance in local PostgreSQL.
Publish: manual reconciliation stages a consistent local snapshot and updates the remote serving tables transactionally.
Serve: Next.js Server Components query the serving database for search, details, filters, and statistics.
The repository contains adapters, parsers, and configuration for ITviec, TopCV, TopDev, VietnamWorks, Devwork, CareerViet, Vieclam24h, CareerLink, JobsGO. These are supported sources, not a guarantee that every source is available or current at every moment.
YAML configuration controls targets, pagination, transport, delays, retries, and validation. Fetchers support HTTP requests and browser-based collection. Detail work comes from the persisted queue, so a detail run does not require discovery to finish in the same process.
Raw data is a separate layer
Detail HTML is stored in MinIO. The database records the object locator, byte length, and SHA-256. Validation checks status, content type when present, minimum content size, allowed hosts and paths, required selectors, and known block pages. A challenge page should not become an apparently successful job record.
Current source configurations do not retain discovery-page HTML. The CLI’s full phase runs discovery and detail only; parsing is an explicit, independently restartable phase.
Normalize the useful fields. Preserve the context.
Source parsers extract data from source-specific HTML and structured page data. A shared dataclass model carries titles, employers, descriptions, requirements, benefits, skills, locations, dates, and source payloads. Shared helpers normalize text and derive province/city values.
Some values intentionally remain raw, including salary and experience descriptions. The system does not claim to have a universal salary model or resolved employer identities.
Parsing verifies object size and SHA-256 before reading the HTML. Quality assessment distinguishes accepted, partial, and rejected output. Accepted and partial records can be persisted; rejected attempts retain their validation issues in crawl state.
Storage responsibilities
Layer
What it holds
MinIO
Raw detail HTML used as parser input.
crawl_state
Runs, discovery coverage, job lifecycle, fetch/parse attempts, and object references.
ref.sources
Stable source identities and display names.
core
Source postings and versioned parse results, with one current result per posting.
serving on Supabase
A compact active-job copy and search support for the website.
A uniqueness constraint on posting, raw hash, parser name, and parser version makes repeated writes idempotent. A partial unique index enforces one current parse result per posting. Alembic tracks local schema changes.
The sync checks source identities and lifecycle baselines, reads a repeatable local snapshot, and stages remote rows. It upserts current content, retains last good content for still-active listings without fresh parsed output, and removes listings whose authoritative state is no longer active. Verification runs before the transaction commits.
Raw HTML, source payloads, parser metadata, and processing history are not copied into the serving tables. A dry-run stages data and reports changes without applying them.
Optional AI enrichment with source evidence
A separate, manually triggered Airflow DAG extracts experience bounds, seniority levels, work mode, employment type, and required versus preferred skills from eligible parsed content. Ingestion and serving sync can continue without enrichment. The website never calls a model during a search request.
Structured output is checked against a fixed schema, allowed values, numeric bounds, and verbatim excerpts from the input. Missing information stays unknown. Matching excerpts establish provenance; they do not guarantee that every interpretation is correct. Successful results are tied to a content hash so changed job descriptions do not reuse an older extraction.
A persistent queue records attempts, delayed retries, provider cooldowns, and request/token reservations. One valid result is enough per content version; no second model acts as a judge. The regular enrichment run covers eligible new or changed content. A separate backfill DAG can select older active jobs by date, source, or posting ID, with a read-only preview by default and explicit job/API-attempt limits. It shares the existing queue and provider budgets; publishing results still requires a separate serving sync. This capability does not imply complete coverage. Evidence and provider metadata stay in the pipeline.
The worker also supports grouping up to three jobs in one Gemini request, validating each result against its own ID and source text. Successful members are retained when another member fails. The current configuration still uses one job per request; grouped requests are an implemented option, not a measured cost or quality improvement.
Filters that preserve missing information
Database-side filters combine minimum experience, seniority, work mode, multiple cities and sources, and a recent time window before pagination. Values within a group use OR; groups combine with AND. Unknown values remain selectable, and original job content stays available for comparison.
When a publication date is absent, filtering and sorting use the posting’s first-seen timestamp. The interface labels this as first recorded rather than claiming it is the publication date. Shared in-flight reads reduce duplicate queries within one web instance; bounded queues and query timing logs help diagnose overload.
Skill matching and filtered insights
A shared catalogue maps known skill names and aliases to 221 canonical keys. Search accepts up to ten skills: ANY matches at least one, while ALL requires every selected key. Visitors can limit matching to required skills or include preferred skills. Unknown labels remain in the original extraction; the registry does not infer skills from job titles.
The statistics dashboard uses the same filters as the job list. It shows top skills and distributions by minimum experience, seniority, work mode, location, and source. Clicking a group opens the corresponding job selection, and switching between search and statistics preserves filters in the URL.
Counts include their denominator, calculation time, and coverage for extracted and normalized data. Missing enrichment is not interpreted as a job having no requirements. These are snapshots of the collected listings, not estimates of the entire labor market; duplicate postings across sources are still separate.
Matching limit: skill lists do not preserve every alternative in the original wording. A requirement such as “Python or C#” can yield both skill keys, so an ALL match still needs to be checked against the original job description.
Search and the web read path
PostgreSQL full-text search uses normalized text and GIN indexes. Plain queries can match the final token as a prefix from three normalized characters; quoted phrases, OR, and exclusions retain their web-search semantics. Source and province/city filters narrow the result set.
The Next.js application uses server-only Postgres.js queries with parameters and a read-only role. It renders search, job details, and aggregate statistics. Pagination fetches one extra row to detect another page instead of presenting an invented total.
Bounded in-memory caches have hard expiry and are local to each Vercel instance. Expired data is not used as a fallback for database errors. The application does not fall back to demo listings in production.
Each source has an Airflow DAG for discovery → detail → parse. Source DAGs are manually triggered and start paused. A shared three-slot pool limits concurrent work across sources; serving sync and raw cleanup reserve all three slots to avoid overlapping ingestion. PostgreSQL advisory locks coordinate work for each source.
Later ingestion phases can process already-available work even when an earlier phase fails. A watcher preserves the failed DAG result. Task retries and persisted state serve different purposes: Airflow retries execution, while the application decides which records are safe to resume.
Retention has a cost
Raw cleanup defaults to dry-run, checks eligibility against lifecycle and parsing state, and protects active, unparsed, or in-progress data. It records deletion intent, validates the object, and marks intentional removal. Parsed records remain, but deleted HTML cannot be reparsed without another fetch.
Protecting the public read path
The database credential stays on the server, with a restricted reader role and TLS certificate and hostname verification. Search inputs, page sizes, pending cache loads, and database reads have explicit limits. Job lists omit long descriptions; details load separately.
The deployment record from 2 October documents Vercel Bot Protection challenges and an IP rate limit on data-reading routes. These controls reduce automated traffic; they do not prevent copying public data or prove large-scale capacity.
Deployment boundaries
Docker Compose defines local PostgreSQL and MinIO, with a separate Airflow stack using LocalExecutor. The web application is deployed on Vercel and reads Supabase through a restricted server-side connection. The pipeline does not run inside the web deployment.
Current limits: manual ingestion and sync, source markup changes, local service availability, and cache expiry all affect freshness. Listings are not deduplicated across source websites, and there is no claimed uptime or throughput target.
Source HTML changes, and a parser fix should not always require another crawl.
Constraint
Fetching a page can involve browser automation, delays, and transient failures.
Decision
Keep detail HTML in MinIO; record its locator and hash in PostgreSQL. Parse from stored objects with versioned parsers.
Trade-off
Storage retention and consistency between objects and database state need explicit handling.
Result
A new parser version can process retained raw HTML again. Purged HTML must be fetched again.
Publish a compact serving copy
Problem
The public website needs current listings, not crawler state or every parse attempt.
Constraint
The local pipeline remains authoritative, while the web application runs on Vercel.
Decision
Reconcile active listings into a separate Supabase serving schema, then read it with a restricted database role.
Trade-off
Two databases introduce synchronization work and a freshness gap. Updates are not streamed in real time.
Result
The website reads a smaller data model; raw objects and processing history stay with the pipeline.
Require complete discovery before expiring a listing
Problem
A missing URL can mean an incomplete crawl rather than a removed job.
Constraint
Pagination can fail, repeat pages, or stop early; configured search scopes can change.
Decision
Track coverage and a scope hash. Apply lifecycle changes only after a complete qualifying scan; changed scopes establish a new baseline.
Trade-off
A failed or incomplete scan leaves previous lifecycle information in place, so stale records can remain.
Result
Partial scans do not incorrectly expire listings. This tracks listing presence, not verified HTTP deletion.
Use PostgreSQL to serve search
Problem
Visitors need keyword search across normalized job data and predictable source/location filters.
Constraint
A separate search service would add another deployment and synchronization boundary.
Decision
Use PostgreSQL full-text search, GIN indexes, normalized text, and final-token prefix matching.
Trade-off
Search follows PostgreSQL tokenization rules; it is not fuzzy matching or semantic search.
Result
The Next.js application queries one serving database for search, details, filters, and aggregate statistics.
08Engineering notes
Problems encountered along the way.
These are concrete failure modes addressed in the current implementation. They matter more to this project than a list of technologies.
An interrupted upload is an ambiguous success
MinIO may accept an object before the process commits raw_ready in PostgreSQL. The pipeline records the expected key, byte length, and SHA-256 before upload, then inspects unfinished uploads on the next run. This makes recovery a state transition that can be checked, rather than a blind retry.
A successful final task can hide an earlier failure
The source DAGs allow later phases to process available work with all_done. A watcher task uses one_failed so a successful parse does not make a failed ingestion run look healthy. The DAG status and individual phase results need to tell the same story.
A listing can still be active even when a fresh parse is unavailable. Serving reconciliation retains the last good content for active listings and refreshes last_seen_at. Removal follows authoritative lifecycle state, rather than the absence of newly parsed content.
A slow database read is more than SQL execution time
The web application accounts for queueing, connection setup, and the query within one deadline. It bounds in-flight reads, permits one retry for selected transient failures, and resets the client when the deadline expires. Logs distinguish operations and sanitized error codes; they do not establish the root cause of every slow query.
09Engineering notes
What I learned.
Make progress explicit.
Fetched, uploaded, parsed, and published are different states. Recording those boundaries makes it possible to decide what is safe to retry.
Preserve the evidence behind a record.
Raw-object references, hashes, parser versions, and validation issues make a normalized row explainable. A row alone cannot show how it was produced.
Define what absence actually means.
An incomplete scan, a rejected parse, and an expired listing need different treatment. Combining them into one failure flag loses information.
Treat the user-facing read path as part of the pipeline.
Data is only useful if it can be read predictably. Database permissions, query behavior, caching, and honest error states matter alongside ingestion.
10Engineering notes
What I would improve next.
These are next steps, not features the project already provides.
Evaluate extraction quality and coverage
Use a labeled sample to assess extracted requirements and skill aliases. Expand enrichment coverage deliberately, and benchmark grouped model requests before changing the current single-job default.
Measure freshness end to end
Surface the last successful crawl, parse, and serving sync together. A healthy task alone does not prove that the website has fresh data.
Expand failure and recovery exercises
Add repeatable scenarios for interrupted uploads, stale parse claims, failed reconciliation, and restoring local state and raw storage together.
Make operations easier to repeat
Document a deliberate ingestion and sync cadence, recovery procedures, and source-change checks before depending on unattended scheduling.
Evaluate matching across sources
The same role may appear on different websites. Explore cross-source duplicate detection with a labeled sample before merging records or changing statistics.
11Engineering notes
Inspect the work.
This case study was checked against both repositories on 3 October 2026. Older architecture notes sometimes describe SQLite state or a smaller Airflow setup; this page follows the current code and configuration.
The frontend repository, joblake-web, is private. Its implementation was reviewed for this case study; there is no public source link for it.
Reviewed local pipeline revision d6cdce7 · Web revision 1cce2d1. Source links remain pinned to the earlier pipeline revision 9092792. They illustrate the original ingestion and serving design; the newer enrichment, backfill, grouping, and skill features were reviewed in the local code and are not represented by those older links.
The repository includes tests for parsers, state transitions, integrity checks, lifecycle tracking, cleanup, and serving behavior. Their presence documents tested scenarios; it does not establish production scale.
