Tech Stack
GoRustPythonJavaC++Distributed SystemsData PipelinesHTTPNetworkingParallel ProcessingLarge DatasetsDebuggingNlpMachine LearningLlm PretrainingRetrieval SystemsHeadless BrowsersChrome Devtools ProtocolPlaywrightPuppeteerProxy SystemsIp RotationCloudData Quality
Solid badges = required, outlined = preferred
Responsibilities
- Build and maintain large-scale web crawlers across diverse domains.
- Design high-throughput, fault-tolerant systems for data collection (millions to billions of URLs/day).
- Handle anti-bot systems, rate limits, and dynamic/JS-heavy sites.
- Develop pipelines for cleaning, deduplication, filtering, and normalization.
- Monitor crawl performance, coverage, and data quality; iterate quickly.
Soft Skills
Problem SolvingCross-Functional CollaborationIteration
Benefits
- Dental
- Equity
- Health Insurance
- Remote Work
- Vision
Culture
Async-FirstDocumentation-DrivenFast-PacedHigh OutputLow EgoMission-DrivenCustomer-ObsessedTransparent Leadership
Requirements
Regions: Worldwide
Get jobs like this in your inbox
Weekly Go, Rust, Python hiring trends and salary data — free.
Join 8 engineers getting weekly insights
Get market intelligence in your inbox
Free weekly insights on tech hiring trends, salaries, and in-demand stacks.
Already a subscriber? Sign in
About Wynd Labs
Industry: ai
Size: startup
Wynd Labs builds infrastructure to deliver massive amounts of high-quality public web data to companies training powerful AI models, operating a distributed crawler and pipelines for ingesting, segmenting, and annotating billions of files. The company is a lean, technical team focused on expanding possibilities for open web data and AI.
View company profile →Compensation
Equity: equity package
Similar Jobs
Data Engineer
Wynd Labs · |Remote (United States)
Remote
Full-Stack SWE, Data Acquisition (Foundations)
OpenAI · San Francisco
RL Deep Learning Engineer
midpage.ai · Comfort working with messy, real-world document data (legal filings, PDFs, long-form text)
$400k
RL Deep Learning Engineer (Remote)
midpage.ai · |Remote (Everywhere)
Remote
$400k
Senior Software Engineer
midpage.ai · Exposing core services on our data via APIs, MCPs, websockets.