Development

• Setup dev environment
• Configure the instance
• Run
• Logging
• Upgrading
• Test and clean manually
• Release
• Useful commands
  • Fetch a resource or a site manually
  • SQL
• TODO
• Ideas
• Related work
  • crawlers
  • sitemap parsers
  • url handling
  • language detection
  • text extraction
  • deduplication
  • Extract more meta tags
  • Date parsing dependent on language