InsightConnect Marketplace

Get URL

Back to Marketplace

Get URL

v2.0.0

Get URL plugin downloads files by URL using HTTP, HTTPS, or FTP

Tags: url, file, download, http, ftp

Triggers
  • Poll URL

Actions
  • Get URL

Description

The Get URL plugin allows you to download files from a URL. Supported protocols are HTTP, HTTPS, and FTP.

This plugin's cache is enabled across workflows to store previously downloaded files to reduce future web requests. To reduce the number of subsequent requests the Etag and If-Modified-Since fields are also checked.

Key Features

  • Get the contents of a web page
  • Check a web page for changes

Requirements

This plugin does not contain any requirements.

Documentation

Setup

This plugin does not contain a connection.

Technical Details

Actions

Get URL

This action is used to download the contents of a URL.

Input
Name Type Default Required Description Enum Example
url string None True URL to Download None https://example.com
checksum string None False Checksum verification (MD5, SHA1, SHA256) None 0800fc577294c34e0b28ad2839435945
is_verify boolean True True Validate certificate None True
timeout integer 60 False Timeout in seconds None 60
user_agent string Mozilla/5.0 False Send requests with user agent None Mozilla/5.0

Example input:

{
  "is_verify": false,
  "timeout": 60,
  "url": "https://httpbin.org/base64/a29tYW5kIHRlc3Q%3D"
}
Output

This action returns the contents of the URL and an HTTP status code.

Name Type Required Description
bytes bytes False Bytes
status_code integer False Status code

Example output:

{
  "bytes": "UmFwaWQ3IEluc2lnaHRDb25uZWN0Cg==",
  "status_code": 200
}

Triggers

Poll URL

This trigger is used to monitor the contents of a URL for changes. The contents are returned when a change has been detected.

Input
Name Type Default Required Description Enum Example
url string None True URL to Download None https://example.com
poll integer 60 False Poll in seconds None 60
is_verify boolean True True Validate certificate None True
user_agent string Mozilla/5.0 False Send requests with user agent None Mozilla/5.0

Example input:

{
  "is_verify": true,
  "poll": 60,
  "url": "https://httpbin.org/base64/a29tYW5kIHRlc3Q%3D"
}
Output

This action returns the contents of the URL and an HTTP status code.

Name Type Required Description
bytes bytes False Bytes
status_code integer False Status code

Example output:

{
  "bytes": "UmFwaWQ3IEluc2lnaHRDb25uZWN0Cg==",
  "status_code": 200
}

Custom Output Types

This plugin does not contain any custom output types.

Troubleshooting

Some web servers do not support cache control mechanisms, or do not use them properly.

Version History

  • 2.0.0 - Use input and output constants | Add example inputs | Changed Exception to PluginException | Added "f" strings | Move test from actions to connection | Change in return file key to bytes | Add new input User Agent to Get File action and Poll URL trigger
  • 1.0.1 - New spec and help.md format for the Extension Library
  • 1.0.0 - Support web server mode
  • 0.1.1 - SSL bug fix in SDK
  • 0.1.0 - Initial plugin

Links

References

This plugin does not contain any references.

plugin_spec_version: v2
extension: plugin
products: [insightconnect]
name: get_url
title: Get URL
description: Get URL plugin downloads files by URL using HTTP, HTTPS, or FTP
version: 2.0.0
vendor: rapid7
support: community
status: []
resources:
  source_url: https://github.com/rapid7/insightconnect-plugins/tree/master/get_url
  license_url: https://github.com/rapid7/insightconnect-plugins/blob/master/LICENSE
tags:
- url
- file
- download
- http
- ftp
hub_tags:
  use_cases: [data_utility]
  keywords: [url, file, download, http, ftp]
  features: []
enable_cache: true
triggers:
  poll_file:
    title: Poll URL
    description: Download modified file by URL
    input:
      url:
        title: URL
        type: string
        description: URL to Download
        required: true
        example: 'https://example.com'
      poll:
        title: Poll
        description: Poll in seconds
        type: integer
        default: 60
        required: false
        example: 60
      is_verify:
        title: Is Verify
        type: boolean
        description: Validate certificate
        required: true
        default: true
        example: true
      user_agent:
        title: User Agent
        type: string
        description: Send requests with user agent
        required: false
        default: "Mozilla/5.0"
        example: "Mozilla/5.0"
    output:
      bytes:
        title: Base64 Encoded File
        type: bytes
        required: false
        description: Bytes
      status_code:
        title: Status Codes
        type: integer
        required: false
        description: Status code
actions:
  get_file:
    title: Get URL
    description: Download a file by URL
    input:
      url:
        title: URL
        type: string
        description: URL to Download
        required: true
        example: 'https://example.com'
      timeout:
        title: Timeout
        description: Timeout in seconds
        type: integer
        default: 60
        required: false
        example: 60
      checksum:
        title: Checksum
        type: string
        description: Checksum verification (MD5, SHA1, SHA256)
        required: false
        example: 0800fc577294c34e0b28ad2839435945
      is_verify:
        title: Is Verify
        type: boolean
        description: Validate certificate
        required: true
        default: true
        example: true
      user_agent:
        title: User Agent
        type: string
        description: Send requests with user agent
        required: false
        default: "Mozilla/5.0"
        example: "Mozilla/5.0"
    output:
      bytes:
        title: Base64 Encoded File
        type: bytes
        required: false
        description: Bytes
      status_code:
        title: Status Codes
        type: integer
        required: false
        description: Status code
Other plugins
Ivanti Security Controls
Rapid7   |   v1.5.0
Plugin
Get
McAfee ePO
Rapid7   |   v5.0.0
Plugin
Get
BlackBerry CylancePROTECT
Rapid7   |   v1.1.0
Plugin
Get
Fortinet FortiGate
Rapid7   |   v4.0.4
Plugin
Get
Jira
Rapid7   |   v6.0.1
Plugin
Get