Tech Stack
Responsibilities
- Build and maintain large-scale web crawlers across diverse domains.
- Design high-throughput, fault-tolerant systems for data collection (millions to billions of URLs/day).
- Handle anti-bot systems, rate limits, and dynamic/JS-heavy sites.
- Develop pipelines for cleaning, deduplication, filtering, and normalization.
- Monitor crawl performance, coverage, and data quality; iterate quickly.
Soft Skills
Problem SolvingCross-Functional CollaborationIteration
Benefits
- Remote Work
- Health Insurance
- Dental
- Vision
- Equity
Culture
Async-FirstDocumentation-DrivenFast-PacedHigh OutputLow Ego
Requirements
Regions: Worldwide
Get jobs like this in your inbox
Weekly Go, Rust, Python hiring trends and salary data — free.
Join 6 engineers getting weekly insights
Get market intelligence in your inbox
Free weekly insights on tech hiring trends, salaries, and in-demand stacks.
Already a subscriber? Sign in
About Wynd Labs
Industry: ai
Size: startup
Wynd Labs builds infrastructure to deliver massive amounts of high-quality public web data to companies training powerful AI models, operating a distributed crawler and pipelines for ingesting, segmenting, and annotating billions of files. The company is a lean, technical team focused on expanding possibilities for open web data and AI.
View company profile →Compensation
Equity: equity package