facf8c9f69
Full-stack Next.js 16 app with three scraping pipelines: - AirScale CSV → Anymailfinder Bulk Decision Maker search - LinkedIn Sales Navigator → Vayne → Anymailfinder email enrichment - Apify Google SERP → domain extraction → Anymailfinder bulk enrichment Includes Docker multi-stage build + docker-compose for Coolify deployment. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
132 lines
3.1 KiB
Markdown
132 lines
3.1 KiB
Markdown
# LeadFlow — Lead Generation & Email Enrichment Platform
|
|
|
|
A unified platform for three lead-scraping pipelines with email enrichment via Anymailfinder.
|
|
|
|
---
|
|
|
|
## Tech Stack
|
|
|
|
- **Next.js 16** (App Router) + TypeScript
|
|
- **SQLite** via Prisma 7 + better-sqlite3
|
|
- **shadcn/ui** + Tailwind CSS
|
|
- **Anymailfinder API v5.1** — email enrichment (bulk JSON + individual search)
|
|
- **Vayne API** — LinkedIn Sales Navigator scraping
|
|
- **Apify** — Google SERP scraping
|
|
|
|
---
|
|
|
|
## Setup
|
|
|
|
### 1. Install dependencies
|
|
|
|
```bash
|
|
cd leadflow
|
|
npm install
|
|
```
|
|
|
|
### 2. Configure environment
|
|
|
|
Copy `.env.local.example` to `.env.local`:
|
|
|
|
```bash
|
|
cp .env.local.example .env.local
|
|
```
|
|
|
|
Edit `.env` and `.env.local`:
|
|
|
|
```env
|
|
APP_ENCRYPTION_SECRET=your-32-character-secret-here!!
|
|
DATABASE_URL=file:./leadflow.db
|
|
```
|
|
|
|
### 3. Run database migration
|
|
|
|
```bash
|
|
npx prisma migrate dev
|
|
```
|
|
|
|
### 4. Start the app
|
|
|
|
```bash
|
|
npm run dev
|
|
```
|
|
|
|
Open [http://localhost:3000](http://localhost:3000)
|
|
|
|
---
|
|
|
|
## API Keys — Where to Get Them
|
|
|
|
Go to **Settings** in the sidebar to enter and save credentials. All keys are AES-256 encrypted before storage.
|
|
|
|
### Anymailfinder
|
|
- Sign up at [anymailfinder.com](https://anymailfinder.com)
|
|
- Account → API → copy your key (format: starts with your account prefix)
|
|
- Pricing: 2 credits/valid decision maker email, 1 credit/person email
|
|
- Bulk API charges only when downloading results
|
|
|
|
### Apify
|
|
- Sign up at [apify.com](https://apify.com)
|
|
- Console → Account → Integrations → API tokens
|
|
- The app uses the `apify/google-search-scraper` actor (pay-per-event)
|
|
|
|
### Vayne
|
|
- Sign up at [vayne.io](https://vayne.io)
|
|
- Dashboard → API Settings → generate token
|
|
- **Connect LinkedIn** in the Vayne dashboard — Vayne manages the LinkedIn session on their end
|
|
- No need to manually export cookies
|
|
|
|
---
|
|
|
|
## Pipeline Workflows
|
|
|
|
### Tab 1 — AirScale → Email
|
|
1. Export companies from AirScale as CSV
|
|
2. Upload → map domain column
|
|
3. Select decision maker categories
|
|
4. Start Enrichment → bulk API runs asynchronously
|
|
5. Export CSV/Excel
|
|
|
|
### Tab 2 — LinkedIn Sales Navigator → Email
|
|
1. Build search in Sales Navigator, copy URL
|
|
2. Paste URL + set max results → Start Scrape
|
|
3. Vayne scrapes profiles (polls until done)
|
|
4. Select profiles → Enrich with Emails
|
|
5. Export results
|
|
|
|
### Tab 3 — SERP → Email
|
|
1. Enter search term (e.g. `"Solaranlage Installateur Deutschland"`)
|
|
2. Set country/language, enable social filter
|
|
3. Select decision maker categories
|
|
4. Start → Apify scrapes Google, domains extracted, then bulk enriched
|
|
5. Export results
|
|
|
|
---
|
|
|
|
## How Anymailfinder Bulk Works
|
|
|
|
1. All domains submitted as one POST to `/v5.1/bulk/json`
|
|
2. Poll status every 5s until `completed`
|
|
3. Download results once (credits charged at download, not submission)
|
|
4. Speed: ~1,000 domains per 5 minutes
|
|
|
|
---
|
|
|
|
## Database
|
|
|
|
SQLite at `./leadflow.db`. Inspect with:
|
|
|
|
```bash
|
|
npx prisma studio
|
|
```
|
|
|
|
---
|
|
|
|
## Troubleshooting
|
|
|
|
| Issue | Solution |
|
|
|-------|----------|
|
|
| "API key not configured" | Add key in Settings |
|
|
| Job stuck at "running" | Check server console (`npm run dev` terminal) |
|
|
| Prisma errors on build | Run `npx prisma generate && npm run build` |
|