Lead scraping pipeline

From a city and a trade to a rep's call queue

Search a city for the contractors in each trade, find who owns each business from the free state registry, fold duplicate listings into one, keep every do-not-call number away from the reps, and hand each rep a fair share of what is left.

Eight stages, and every stage shows the rows it was given, the rows it produced and its counts.

Try it

Press Play the run. 46 scraped listings travel through eight stages. Watch 5 repeats fall away at Merge and 3 red dots appear at Import.
Click Match. Cinder Peak Plumbing & Drain files through a registered-agent service, so its owner stays blank. Mesa Canyon Roofing is left unmatched on purpose, because its name is only place words and a trade.
Click Import. The 3 red rows are on the do-not-call file: one by place id, two because they share a phone line. They arrive as do-not-call and dead.
Click Dedupe. TUMBLEWEED ORCHARD ROOFING folds into Tumbleweed Orchard Roofing, while the three Marigold Mile sites share one line and all stay callable as a chain.
Click Assign. Each rep holds the same number of leads in every tier, give or take one. The lead rep2 already owned never moved.

Ran for real the project's own script, run as shipped on fake dataIllustration built in the scraper's real column layout

Press Play the run to send 46 listings through