get

commit c6371c5f0f9186fa9b332a1ce546ebb6d71b5790

Author: Connor Etherington <[email protected]>

Date:   Fri Sep 13 21:27:24 2024 +0200

    Update 13-September-2024___21:27:21

---

 README.md    | 211 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

 app.js       |  56 ++++++++++++++--

 package.json |  12 +++-

 3 files changed, 270 insertions(+), 9 deletions(-)

diff --git a/README.md b/README.md

new file mode 100644

index 0000000..ddd1ac3

--- /dev/null

+++ b/README.md

@@ -0,0 +1,211 @@

+# @agentics/get

+

+A versatile command-line tool and library for making HTTP requests, scraping web content, and automating web interactions using Node.js.

+

+## Table of Contents

+

+- [Features](#features)

+- [Installation](#installation)

+- [Usage](#usage)

+  - [Command-Line Usage](#command-line-usage)

+  - [Options](#options)

+  - [Examples](#examples)

+- [Programmatic Usage](#programmatic-usage)

+- [Contributing](#contributing)

+- [License](#license)

+

+## Features

+

+- Perform HTTP GET and POST requests.

+- Scrape web pages using selectors.

+- Extract text, HTML, links, images, and cookies.

+- Evaluate custom JavaScript on web pages.

+- Save responses and cookies to files.

+- Use customizable request headers.

+- Supports JavaScript evaluation with Puppeteer.

+

+## Installation

+

+```bash

+npm install -g @agentics/get

+```

+

+## Usage

+

+### Command-Line Usage

+

+```bash

+get [options]

+```

+

+### Options

+

+| Option                 | Alias | Type    | Description                                                                                           | Default |

+|------------------------|-------|---------|-------------------------------------------------------------------------------------------------------|---------|

+| `--get`                | `-g`  | Boolean | Perform a GET request (default).                                                                      | `true`  |

+| `--post`               | `-p`  | Boolean | Perform a POST request.                                                                               | `false` |

+| `--url <url>`          | `-u`  | String  | URL to request. **(Required)**                                                                        |         |

+| `--eval <script>`      | `-e`  | String  | Evaluate JavaScript on the page.                                                                      |         |

+| `--save`               | `-s`  | Boolean | Save response and cookies to files (`response.json`, `cookies.json`).                                 | `false` |

+| `--selector <css>`     | `-S`  | String  | CSS selector to scrape content from the page.                                                         |         |

+| `--links`              | `-l`  | Boolean | Return all links (`<a href="">`) from the page.                                                      | `false` |

+| `--text`               | `-t`  | Boolean | Return the text content of the page (default if no other output option is specified).                 | `false` |

+| `--html`               | `-H`  | Boolean | Return the HTML content of the page.                                                                  | `false` |

+| `--cookies`            | `-c`  | Boolean | Return cookies from the response headers.                                                             | `false` |

+| `--header <key:value>` | `-h`  | String  | Custom headers for the request (can be used multiple times).                                          |         |

+

+### Examples

+

+#### Basic GET Request

+

+```bash

+get -u https://example.com

+```

+

+#### POST Request with Custom Headers

+

+```bash

+get -p -u https://example.com/api -h "Content-Type: application/json" -h "Authorization: Bearer token"

+```

+

+#### Scrape Text Using a CSS Selector

+

+```bash

+get -u https://example.com -S ".article-title"

+```

+

+#### Evaluate JavaScript on the Page

+

+```bash

+get -u https://example.com -e "document.title"

+```

+

+#### Get All Links from a Page

+

+```bash

+get -u https://example.com -l

+```

+

+#### Save Response and Cookies to Files

+

+```bash

+get -u https://example.com -s

+```

+

+#### Get Page HTML

+

+```bash

+get -u https://example.com -H

+```

+

+#### Get Cookies from Response Headers

+

+```bash

+get -u https://example.com -c

+```

+

+### Combining Options

+

+You can combine multiple options to perform complex tasks. For example:

+

+```bash

+get -u https://example.com -S ".article-title" -l -c -s

+```

+

+This command will:

+

+- Scrape content matching `.article-title`.

+- Return all links from the page.

+- Return cookies from the response headers.

+- Save the response and cookies to files.

+

+## Programmatic Usage

+

+You can also use `@agentics/get` as a library in your Node.js projects.

+

+### Importing the Module

+

+```javascript

+const get = require('@agentics/get');

+```

+

+### Using `exportFunctions`

+

+The `exportFunctions` method allows you to perform web scraping tasks programmatically.

+

+#### Syntax

+

+```javascript

+const results = await get(url, options);

+```

+

+#### Parameters

+

+- `url` *(String)*: The URL to request.

+- `options` *(Object, optional)*: Configuration options.

+

+#### Available Options

+

+- `post` *(Boolean)*: Use POST request instead of GET.

+- `headers` *(Object)*: Custom headers for the request.

+- `cookies` *(Boolean)*: Return cookies from the response headers.

+- `links` *(Boolean)*: Return all links from the page.

+- `html` *(Boolean)*: Return the HTML content of the page.

+- `text` *(Boolean)*: Return the text content of the page.

+- `selector` *(String)*: CSS selector to scrape content.

+- `eval` *(String)*: JavaScript code to evaluate on the page.

+- `save` *(Boolean)*: Save response and cookies to files.

+

+#### Example

+

+```javascript

+const get = require('@agentics/get');

+

+(async () => {

+  try {

+    const url = 'https://example.com';

+    const options = {

+      headers: {

+        'User-Agent': 'CustomUserAgent/1.0',

+      },

+      text: true,

+      links: true,

+      cookies: true,

+    };

+

+    const results = await get(url, options);

+

+    console.log('Text Content:', results.text);

+    console.log('Links:', results.links);

+    console.log('Cookies:', results.cookies);

+  } catch (error) {

+    console.error('Error:', error);

+  }

+})();

+```

+

+#### Default Options

+

+If no options are provided, the following defaults are used:

+

+```javascript

+{

+  cookies: true,

+  links: true,

+  html: true,

+  text: true,

+}

+```

+

+## Contributing

+

+Contributions are welcome! Please open an issue or submit a pull request on [GitLab](https://gitlab.com/a4to/get).

+

+## License

+

+This project is licensed under the [MIT License](LICENSE).

+

+---

+

+**Author:** Connor Etherington

+**Email:** [[email protected]](mailto:[email protected])

diff --git a/app.js b/app.js

index e16ac9a..f09578e 100755

--- a/app.js

+++ b/app.js

@@ -123,6 +123,11 @@ class Scraper {

     await this.ensureInitialized();

     return this.response;

   }

+

+  async cookies() {

+    await this.ensureInitialized();

+    return this.response.headers['set-cookie'];

+  }

 }

 const argv = yargs

@@ -182,13 +187,12 @@ const argv = yargs

 const method = argv.p ? 'POST' : 'GET';

 const url = argv.u;

-const headers = argv.h

-  ? argv.h.reduce((acc, header) => {

-      const [key, value] = header.split(':');

-      acc[key.trim()] = value.trim();

-      return acc;

-    }, {})

-  : {};

+

+const headers = argv.h ? argv.h.reduce((acc, header) => {

+  const [key, value] = header.split(':');

+  acc[key.trim()] = value.trim();

+  return acc;

+}, {}) : {};

 const config = { method, headers, url, withCredentials: true };

@@ -215,6 +219,7 @@ const scraper = new Scraper(url, config);

     if (argv.H) console.log(await scraper.getHtml());

     if (argv.S) console.log(await scraper.scrape(argv.S));

     if (argv.e) console.log(await scraper.evaluate(argv.e));

+    if (argv.c) console.log(await scraper.cookies());

     if (argv.save) {

       const response = await scraper.getResponse();

       saveFiles(response);

@@ -224,3 +229,40 @@ const scraper = new Scraper(url, config);

   }

 })();

+const exportFunctions = async (url, options = {}) => {

+  if (Object.keys(options).length === 0) {

+    options = {

+      cookies: true,

+      links: true,

+      html: true,

+      text: true,

+    };

+  }

+

+  const scrapeOptions = {

+    headers: options.headers || {},

+    method: options.post ? 'POST' : 'GET',

+  };

+

+  const scraper = new Scraper(url, scrapeOptions);

+  await scraper.ensureInitialized();

+

+  const results = {};

+

+  if (options.text) results.text = await scraper.pageText();

+  if (options.links) results.links = await scraper.links();

+  if (options.html) results.html = await scraper.getHtml();

+  if (options.selector) results.scraped = await scraper.scrape(options.selector);

+  if (options.eval) results.evaluated = await scraper.evaluate(options.eval);

+  if (options.cookies) results.cookies = await scraper.cookies();

+

+  if (options.save) {

+    const response = await scraper.getResponse();

+    saveFiles(response);

+  }

+

+  return results;

+};

+

+module.exports = exportFunctions;

+

diff --git a/package.json b/package.json

index a7823aa..2aa571e 100644

--- a/package.json

+++ b/package.json

@@ -1,7 +1,8 @@

 {

-  "name": "get",

+  "name": "@agentics/get",

   "version": "0.1.0",

   "main": "app.js",

+  "private": false,

   "scripts": {

     "start": "node ${npm_package_main}",

     "dev": "nodemon ${npm_package_main}"

@@ -9,7 +10,14 @@

   "bin": {

     "get": "./app.js"

   },

-  "author": "Connor Etherington <[email protected]>",

+  "repository": {

+    "type": "git",

+    "url": "git+https://gitlab.com/a4to/get.git"

+  },

+  "author": {

+    "name": "Connor Etherington",

+    "email": "[email protected]"

+  },

   "license": "MIT",

   "dependencies": {

     "axios": "^1.7.7",