← Back to Blog

## PARALLEL OSINT: MULTIPLE TARGETS SIMULTANEOUSLY

Published: 2026-05-24

## THE SCALING PROBLEM WITH SEQUENTIAL INVESTIGATION

A single OSINT investigation against one target takes 30–90 seconds: WHOIS lookup, subdomain enumeration, IP reputation checks, breach lookup, username search. That is fine for individual investigations. It becomes a bottleneck at scale. A threat intelligence analyst reviewing 50 suspicious IPs from a SIEM alert, a security team auditing 200 contractor email addresses for breach exposure, or a researcher investigating a botnet's C2 infrastructure spanning 300 domains — all of them face the same problem. Sequential execution at 60 seconds per target means 50 minutes for 50 targets and 5 hours for 300.

The solution is not faster tools. It is parallel execution: running investigations concurrently rather than one after another. The constraint is rate limits on the APIs being queried, not computation on the local machine. With proper concurrency control, the same 50-target investigation runs in under 5 minutes.

## ASYNCIO FUNDAMENTALS FOR OSINT PIPELINES

Python's `asyncio` is the right tool for I/O-bound OSINT work. All OpenOSINT tool functions are `async def` coroutines. To run multiple targets in parallel, wrap them with `asyncio.gather()`:

    import asyncio
    from openosint.tools.search_whois import run_whois_osint
    from openosint.tools.search_ip import run_ip_osint

    async def investigate_domain(domain: str) -> dict:
        whois_result = await run_whois_osint(domain)
        return {'domain': domain, 'whois': whois_result}

    async def bulk_investigate(domains: list[str]) -> list[dict]:
        tasks = [investigate_domain(d) for d in domains]
        return await asyncio.gather(*tasks, return_exceptions=True)

    results = asyncio.run(bulk_investigate(['example.com', 'test.org', 'demo.net']))

`return_exceptions=True` is critical in bulk runs: it prevents one failed target from canceling the entire gather. Each result is either a value or an exception instance, which you filter and log separately.

## RATE LIMIT MANAGEMENT

Unbounded parallelism gets API keys suspended. The correct pattern uses a semaphore to cap concurrent requests per API:

    import asyncio

    # Limit to 5 concurrent requests to respect API rate limits
    sem = asyncio.Semaphore(5)

    async def rate_limited_lookup(ip: str, lookup_fn) -> dict:
        async with sem:
            result = await lookup_fn(ip)
            await asyncio.sleep(0.2)  # 200ms between releases = max 5 req/s
            return result

    async def bulk_ip_check(ips: list[str], lookup_fn) -> list[dict]:
        tasks = [rate_limited_lookup(ip, lookup_fn) for ip in ips]
        return await asyncio.gather(*tasks, return_exceptions=True)

Tune the semaphore value and sleep delay to match each API's rate limits. AbuseIPDB free tier allows 1000 requests/day with no per-second limit documented; use a semaphore of 10 and no sleep. Shodan free tier: 1 request/second; semaphore of 1 with 1.0s sleep. VirusTotal free tier: 4 requests/minute; semaphore of 4 with 15s sleep between batches.

For multi-API pipelines, run a separate semaphore per API. A domain investigation that hits Shodan, VirusTotal, and AbuseIPDB simultaneously needs three independent concurrency controls, each tuned to its API's limits.

## THE OPENOSINT MULTI-TARGET RUNNER

OpenOSINT's built-in multi-target runner handles the concurrency and rate limiting automatically. Create a target list file, one per line:

    # targets.txt
    192.168.1.1
    10.0.0.1
    suspicious-domain.net
    attacker@example.com

Run the investigation:

    # AI-assisted, all targets in parallel, JSON output
    openosint multi targets.txt --json > results.json

    # Direct tool, no AI, IP reputation only
    openosint multi targets.txt --tool ip --json > ip_results.json

The multi-target runner type-detects each target (IP vs domain vs email vs username), dispatches the appropriate tool set, runs concurrently with built-in rate limiting, and writes structured JSON results. Each result object contains the target, the tool outputs, any errors, and a timestamp. See the tools reference for the full result schema.

## RESULT AGGREGATION AND DEDUPLICATION

Parallel investigation produces a list of result objects. The next step is aggregation: identifying cross-target patterns that indicate related infrastructure or coordinated activity.

    import json
    from collections import defaultdict

    with open('results.json') as f:
        results = json.load(f)

    # Group IPs by ASN to find infrastructure clusters
    asn_groups = defaultdict(list)
    for r in results:
        if r.get('tool') == 'search_ip' and 'asn' in r.get('data', {}):
            asn_groups[r['data']['asn']].append(r['target'])

    # Flag targets sharing infrastructure
    for asn, targets in asn_groups.items():
        if len(targets) > 1:
            print(f"Shared ASN {asn}: {', '.join(targets)}")

The same pattern works for any shared attribute: registrar, nameserver, creation date window, or abuse report category. Finding 15 domains that registered on the same day through the same privacy-protected registrar service is a signal that no single-target investigation would surface. This is the analytical value of parallel investigation over sequential. For generating structured reports from aggregated results, see the OSINT report generation guide. For the API-level details of each data source, see the OSINT API automation guide.

## SEE ALSO

- Automating OSINT with APIs — per-API rate limits, authentication, and Python patterns
- Generating structured OSINT reports — JSON, PDF, and Markdown output from bulk results
- OpenOSINT tools reference — multi-target runner documentation and result schemas

Free: 5 AI OSINT prompts that actually work

ChatGPT hallucinates when you give it a vague task. These 5 structured prompts force it to collect at every stage: scope → collect → pivot → verify → document. Free PDF — no card, instant download.

Download free PDF →

Want 30+ prompts? Full pack ($29) ↗

Home · Blog · Tools · GitHub