Lead collection should not be built around evading blocks. Rotating proxies can provide geographic routing, uptime, or distributed access to sources that explicitly permit automated requests, but using them to bypass rate limits, CAPTCHAs, login restrictions, or bans creates legal, contractual, privacy, and reputational risk. A defensible agent uses official APIs and licensed datasets first, collects only necessary business data, identifies itself where appropriate, and stops when access is denied.
Start with an authorised source hierarchy
Choose sources in this order:
- Your own opt-in forms, CRM, webinar registrations, and customer referrals.
- Official APIs and partner programmes.
- Licensed business-data providers such as Apollo, ZoomInfo, Cognism, People Data Labs, or Clearbit/HubSpot offerings, subject to current contracts and regional availability.
- Public company websites that permit the intended automated access.
- Manual research for ambiguous or restricted sources.
Do not scrape LinkedIn pages with bots. LinkedIn’s published rules prohibit third-party scraping and automated activity. Do not bypass paywalls, authentication, robots controls, CAPTCHAs, or technical restrictions on any site.
Define lawful purpose and minimum fields
Before collection, document the market, role, purpose, legal basis, notice obligations, retention, deletion, and opt-out process. GDPR, ePrivacy rules, CCPA/CPRA, CAN-SPAM, CASL, and sector laws have different scopes; obtain qualified advice for the actual jurisdictions.
Minimise the schema. A B2B research record may need company, domain, country, industry, company size band, relevant public role, source URL, capture date, and consent or lawful-basis status. Personal phone numbers, private emails, sensitive traits, and unrelated profile data create risk without improving qualification.
| Field | Preferred source | Validation |
|---|---|---|
| Company domain | Official site or registry | Resolve and match legal/company name |
| Industry | Company description plus taxonomy | Human review for ambiguous firms |
| Employee band | Licensed provider or filing | Store range and date, not false precision |
| Business contact | Opt-in or licensed source | Verify provenance and suppression status |
| Buying signal | Public, relevant event | Record source and avoid personal inference |
Design the agent as a bounded workflow
Use deterministic stages: receive target account, check authorisation, fetch allowed sources, extract fields, validate, deduplicate, score against explicit criteria, and queue for human review. The model can classify descriptions and draft a research note; it should not decide whether consent exists or invent missing contacts.
Define maximum pages, requests, cost, and elapsed time per company. Require a source URL for every extracted claim. If an essential field is unavailable, return null. Stop on 401, 403, 429, CAPTCHA, robots prohibition, or a terms conflict and send the record to manual review.
Suitable orchestration options include n8n, Make, Zapier, Apify for permitted crawling, or a custom queue and worker system. Verify current terms, pricing, and data-processing roles for every vendor.
Use APIs before browser extraction
APIs provide structured fields, authentication, documented quotas, and more stable contracts. Use a service account, store keys in a secret manager, and request the smallest scopes. Honour rate-limit headers and retry only according to documentation.
Cache responses within the permitted period. A company profile that changes monthly should not be fetched hundreds of times per day. Track provider record IDs and update timestamps.
Licensed data does not eliminate compliance duties. Confirm whether the contract permits marketing use, enrichment, onward transfer, retention, and the target regions. Maintain suppression lists across all ingestion paths.
Configure web extraction conservatively
For an authorised public site, identify the pages and fields in advance. Use a descriptive user agent and contact address if the site’s policy or crawler convention calls for it. Request slowly, cache pages, and avoid repeated rendering of heavy JavaScript.
Prefer one known egress IP that the site can allowlist. A rotating proxy pool should be considered only when the source explicitly permits distributed access or the service requires regional routing for authorised testing. Document the provider, locations, data handling, session behaviour, and abuse-response process.
Never rotate in response to a block. Treat a block as a stop signal. Residential proxy networks can carry additional ethical and security concerns about how endpoints were obtained; conduct vendor due diligence before any legitimate use.
Extract into a strict schema
Parse deterministic fields with selectors or API mappings. Send only ambiguous text to the model, along with an allowlist of categories and an unknown option. Validate model output against JSON Schema.
Keep raw evidence separately from clean CRM fields. Record the source URL, access time, extraction version, and confidence. For a claimed technology or office location, distinguish first-party evidence from a model inference.
Do not let a page tell the agent to change its instructions. Web content is untrusted input. Strip scripts, limit content size, and isolate the browser worker from secrets and internal networks.
Deduplicate and validate
Canonicalise domains by removing protocol, www, tracking parameters, and paths. Match companies by domain first, then carefully by legal name and country. Do not merge two franchises or subsidiaries merely because names resemble each other.
Check email syntax and domain records without sending unsolicited verification messages. Avoid catch-all certainty. Apply existing opt-outs and do-not-contact status before a lead reaches a sequence.
Human reviewers should inspect high-value records, ambiguous company matches, unusual personal data, and every source flagged for terms uncertainty. Record reviewer decisions to improve rules.
Score fit without sensitive profiling
Use transparent business criteria such as industry, employee band, supported region, documented technology, and public project need. Avoid inferring health, ethnicity, religion, politics, union membership, sexual orientation, or other sensitive characteristics.
Keep the fit score separate from outreach permission. A company can be a perfect customer profile and still be inappropriate to contact through a particular channel. Explain each score with source-backed fields.
Route to human-approved outreach
Create or update the CRM record with provenance, freshness, and suppression status. Generate a research brief and draft message, but require a person to verify relevance and send through a lawful channel. Do not auto-enrol every extracted record.
Measure qualified conversations, opt-outs, complaints, bounce rate, corrections, and deletion requests—not only record volume. Shut down sources that generate poor provenance or harm signals.
Cost, reliability, and governance
Calculate API fees, proxy traffic, browser compute, model tokens, retries, storage, CRM seats, validation, and compliance work. Cheap scraping can become expensive when websites change or legal teams must investigate provenance.
Monitor empty output, row-count changes, 4xx and 429 responses, duplicate spikes, model-schema failures, and unusual proxy regions. Keep a kill switch and source-level disable control. Delete raw pages according to policy.
Pros, cons, and verdict
AI agents can classify companies, reconcile names, summarise evidence, and reduce manual copying. Authorised APIs and licensed providers offer stable, structured scale. Browser extraction can fill narrow public-data gaps.
The downsides are legal ambiguity, stale data, false matches, privacy risk, source breakage, vendor cost, and abuse potential. Rotating proxies increase complexity and must not be a mechanism for evasion.
Build the first workflow from opt-in and licensed sources, with nulls for missing fields and human approval before outreach. Add a public website only after documenting permission and a stop-on-denial policy.
Our pick: official APIs and licensed B2B data with source provenance; no proxy rotation to bypass access controls.