How to Use Curl Effectively on Linux and OSX

Overview

The curl command is the prevailing utility for working with the Web from a command-line. The utility provides the same functionality as any web brower, without the UI, that enables you to mimic any type of your request done from a browser. In this tutorial, you will learn ways to use Curl effectively from a command-line to perform common requests.

Data can be downloaded from a website, forms can be posted to, typicaly RESTful commands can be tested against applications. The list of possibilities to endless with this popular tool.

You could effectively use curl in place of Postman, for example. However, there are certain niceties about Postman that we recognize, so it depends on what you need out of the tool.

What You Will

  • Simple Get requests
  • Posting data
  • Basic Authentication
  • Using Headers

Simple Examples

The following are commonly used simple curl commands. They

Simple GET Request

curl http://www.serverlab.ca

Following 301 and 302 Redirects with Curl

curl -L http://www.serverlab.ca
curl --location http://www.serverlab.ca

Curl with Basic Authentication

curl –basic -u <user> –digest

Curl HTTPS endpoints without verifying TLS

curl --insecure <host>
curl --insecure https://www.serverlab.ca

Curl requests through specific network interfaces

curl --interface <name> <host>
curl --interface eth0:1 https://www.serverlab.ca

Downloading Files or Outputting Results to File

When data needs to be downloaded from a website, such as downloading a script or binary file, the output needs to be outputted to disk. For these use cases, the -o or –output flags are used with the curl command.

curl http://www.serverlab.ca/downloads/install-script.sh -o install-script.sh

Setting TCP/IP Version for Requests

Requests will automatically use IPv4 or IPv6 depending on the system network configuration. However, there are times were you would want to explicitly set which version should be used. Curl provides the –ipv4 and –ipv6 flags for this purpose.

To send requests via IPv4, use the -4 or –ipv4 flags.

curl -4 http://www.serverlab.ca

curl –ipv4 http://www.serverlab.ca

To send requests via IPv6, use the -6 or –ipv6 flags.

curl -6 http://www.serverlab.ca

curl -ipv6 http://www.serverlab.ca

Post Requests with Curl

Not every curl command is a GET request. The following examples show you how to perform POSTS to endpoints, and how to set send data with the command. The -X flag is used to set requests to another type, which in the following examples will be POST.

curl -X POST http://www.severlab.ca/example-form.php

A post without data isn’t very useful. To add data from the command-line the -d option is used. This flag causes curl to pass the content-type value as application/x-www-form-urlencoded.

curl -X POST -d field1=example1 -d field2=example2 http://www.serverlab.ca/api

When posting form field data, the data should be url-encoded. This is done using –data-urlencode flag.

curl -X POST --data-urlencode title="my notes" content="this is a note" http://www.serverlab.com/admin/save-note.php

To interactively send data to the endpoint, the @ symbol is used with the -d flag.

curl -X POST [email protected] http://www.serverlab.ca/example-form.php

Data values can be fetched from a file instead, and similar to posting data interactively you use the @ symbol, but with a file name. When using files line breaks are removed, which allows you nicely format the file for better legibility.

curl -X [email protected] http://www.serverlab.ca/example-form.php

Sending Curl Requests through a Proxy

Many organizations use a forward proxy server to handle all external Internet requests. In environments where a proxy isn’t already set with HTTP_PROXY, for example, or where a separate proxy must be used, the -x flag is used.

Simple Proxy Request

curl -x http://my.proxy.com:8080 http://www.serverlab.ca

Some proxies require a username and password. This can be done by including the -U or –proxy-user flags to the command.

curl -U <user:password> -x [protocol:]<proxy-server>[:port] <endpoint>
curl -U student:password1 -x https://my.proxy.server.com:80 https://google.com