# TODO

- [X] Add the ability to start with a list of predefined urls
- [X] Add flag for headless/not headless chrome
- [X] Clean up on start in sink, link tracker, scheduler
- [ ] Remove nested builders
- [ ] Use factory for async object creation?
- [ ] Write tests using specflow
- [ ] Separate nuget packages for MongoDb, Cosmos Db, Redis, etc.
- [ ] Add support for xpath and regex
- [ ] Add caching of html pages
- [ ] Breadth first traversal with priority channels
- [ ] Refactor DAL classes with a repository pattern or Unit of Work or Facade
- [ ] Redo the async initialization with factory pattern (static ctr)
- [ ] Unify persistence configuration with single methods in ScraperEngineBuilder
- [ ] Web hook sink
- [ ] Add flexible conditions for ignoring or allowing certain pages
- [ ] Sitemap crawling support
- [ ] Add LogTo method with Console and File support
- [ ] Request auto throttling
- [ ] Add bloom filter for revisiting same urls
- [ ] Fix base url bug
