{
  "name": "scrape-it",
  "description": "A Node.js scraper for humans.",
  "keywords": [
    "scrape",
    "it",
    "a",
    "scraping",
    "module",
    "for",
    "humans"
  ],
  "license": "MIT",
  "version": "6.1.2",
  "main": "lib/index.js",
  "types": "lib/index.d.ts",
  "scripts": {
    "test": "node test"
  },
  "author": "Ionică Bizău <bizauionica@gmail.com> (https://ionicabizau.net)",
  "contributors": [
    "ComFreek <comfreek@outlook.com> (https://github.com/ComFreek)",
    "Jim Buck <jim@jimmyboh.com> (https://github.com/JimmyBoh)"
  ],
  "repository": {
    "type": "git",
    "url": "git+ssh://git@github.com/IonicaBizau/scrape-it.git"
  },
  "bugs": {
    "url": "https://github.com/IonicaBizau/scrape-it/issues"
  },
  "homepage": "https://github.com/IonicaBizau/scrape-it#readme",
  "blah": {
    "h_img": "https://i.imgur.com/j3Z0rbN.png",
    "cli": "scrape-it-cli",
    "description": [
      "----",
      "",
      "<p align=\"center\">",
      "Sponsored with :heart: by:",
      "<br/><br/>",
      "<a href=\"https://serpapi.com\">Serpapi.com</a> is a platform that allows you to scrape Google and other search engines from our fast, easy, and complete API.<br>",
      "<a href=\"https://serpapi.com\"><img src=\"https://i.imgur.com/0Pk6Ysp.png\" width=\"250\" /></a>",
      "<br/><br/>",
      "",
      "[Capsolver.com](https://www.capsolver.com/?utm_source=github&utm_medium=banner_github&utm_campaign=scrape-it) is an AI-powered service that specializes in solving various types of captchas automatically. It supports captchas such as [reCAPTCHA V2](https://docs.capsolver.com/guide/captcha/ReCaptchaV2.html?utm_source=github&utm_medium=banner_github&utm_campaign=scrape-it), [reCAPTCHA V3](https://docs.capsolver.com/guide/captcha/ReCaptchaV3.html?utm_source=github&utm_medium=banner_github&utm_campaign=scrape-it), [hCaptcha](https://docs.capsolver.com/guide/captcha/HCaptcha.html?utm_source=github&utm_medium=banner_github&utm_campaign=scrape-it), [FunCaptcha](https://docs.capsolver.com/guide/captcha/FunCaptcha.html?utm_source=github&utm_medium=banner_github&utm_campaign=scrape-it), [DataDome](https://docs.capsolver.com/guide/captcha/DataDome.html?utm_source=github&utm_medium=banner_github&utm_campaign=scrape-it), [AWS Captcha](https://docs.capsolver.com/guide/captcha/awsWaf.html?utm_source=github&utm_medium=banner_github&utm_campaign=scrape-it), [Geetest](https://docs.capsolver.com/guide/captcha/Geetest.html?utm_source=github&utm_medium=banner_github&utm_campaign=scrape-it), and Cloudflare [Captcha](https://docs.capsolver.com/guide/antibots/cloudflare_turnstile.html?utm_source=github&utm_medium=banner_github&utm_campaign=scrape-it) / [Challenge 5s](https://docs.capsolver.com/guide/antibots/cloudflare_challenge.html?utm_source=github&utm_medium=banner_github&utm_campaign=scrape-it), [Imperva / Incapsula](https://docs.capsolver.com/guide/antibots/imperva.html?utm_source=github&utm_medium=banner_github&utm_campaign=scrape-it), among others. For developers, Capsolver offers API integration options detailed in their [documentation](https://docs.capsolver.com/?utm_source=github&utm_medium=banner_github&utm_campaign=scrape-it), facilitating the integration of captcha solving into applications. They also provide browser extensions for [Chrome](https://chromewebstore.google.com/detail/captcha-solver-auto-captc/pgojnojmmhpofjgdmaebadhbocahppod) and [Firefox](https://addons.mozilla.org/es/firefox/addon/capsolver-captcha-solver/), making it easy to use their service directly within a browser. Different pricing packages are available to accommodate varying needs, ensuring flexibility for users.",
      "<a href=\"https://capsolver.com/?utm_source=github&utm_medium=github_banner&utm_campaign=scrape-it\"><img src=\"https://i.imgur.com/lCngxre.jpeg\"/></a>",
      "</p>",
      "",
      "----"
    ],
    "installation": [
      {
        "h2": "FAQ"
      },
      {
        "p": "Here are some frequent questions and their answers."
      },
      {
        "h3": "1. How to parse scrape pages?"
      },
      {
        "p": "`scrape-it` has only a simple request module for making requests. That means you cannot directly parse ajax pages with it, but in general you will have those scenarios:"
      },
      {
        "ol": [
          "**The ajax response is in JSON format.** In this case, you can make the request directly, without needing a scraping library.",
          "**The ajax response gives you HTML back.** Instead of calling the main website (e.g. example.com), pass to `scrape-it` the ajax url (e.g. `example.com/api/that-endpoint`) and you will you will be able to parse the response",
          "**The ajax request is so complicated that you don't want to reverse-engineer it.** In this case, use a headless browser (e.g. Google Chrome, Electron, PhantomJS) to load the content and then use the `.scrapeHTML` method from scrape it once you get the HTML loaded on the page."
        ]
      },
      {
        "h3": "2. Crawling"
      },
      {
        "p": "There is no fancy way to crawl pages with `scrape-it`. For simple scenarios, you can parse the list of urls from the initial page and then, using Promises, parse each page. Also, you can use a different crawler to download the website and then use the `.scrapeHTML` method to scrape the local files."
      },
      {
        "h3": "3. Local files"
      },
      {
        "p": "Use the `.scrapeHTML` to parse the HTML read from the local files using `fs.readFile`."
      }
    ]
  },
  "dependencies": {
    "@types/cheerio": "^0.22.31",
    "assured": "^1.0.15",
    "cheerio-req": "^2.0.0",
    "scrape-it-core": "^1.0.0",
    "typpy": "^2.3.13"
  },
  "devDependencies": {
    "lien": "^3.4.2",
    "tester": "^1.4.5"
  },
  "files": [
    "bin/",
    "app/",
    "lib/",
    "dist/",
    "src/",
    "scripts/",
    "resources/",
    "menu/",
    "cli.js",
    "index.js",
    "index.d.ts",
    "package-lock.json",
    "bloggify.js",
    "bloggify.json",
    "bloggify/"
  ]
}