Skip to main content

An intro to cURL: The basics of the transfer tool

 

Introduction

cURL is a transfer tool used to transfer data from or to a server. It supports various internet protocols of transfer, including:

  • DICT
  • FILE
  • FTP, FTPS
  • GOPHER
  • HTTP, HTTPS
  • IMAP, IMAPS
  • LDAP, LDAPS
  • POP3, POP3S
  • RTMP, RTSP
  • SCP, SFTP
  • SMB, SMBS
  • SMTP, SMTPS
  • TELNET and TFTP

We can perform useful tricks with cURL like proxy support, user authentication, FTP upload, HTTP post, SSL connections, cookies, file transfer resume, Metalink, and more.

Important: cURL is powered by libcurl for all transfer-related features.

cURL is used from the command-line. It’s mostly commonly used to quickly test out APIs during software development. I personally use cURL when I want to test out all my APIs in my Node.js server. It is a really handy tool for developers.

Postman is cool, but cURL is super-cool. – Chidume Nnamdi

In this post, we will go through basic cURL functions and options. We will also learn how we can use cURL to perform GET, POST requests on an API endpoint.

cURL options

--request or -X

--request and -X specify a custom request method you can use when communicating with the HTTP server. The specified request method will be used instead of the method otherwise used (which defaults to GET).

To perform a POST request:

curl --request POST

To perform a GET request:

curl --request GET

If we don’t specify the request via --request, it will default to GET.

--url</code

This specifies the URL that we will be fetching or transferring data to.

From the cURL docs:

Specify a URL to fetch. This option is mostly handy when you want to specify URL(s) in a config file.

If the given URL is missing a scheme name (such as “http://&#8221; or “ftp://&#8221;, etc) then cURL will make a guess based on the host.

If the outermost subdomain name matches DICT, FTP, IMAP, LDAP, POP3, or SMTP, then that protocol will be used. Otherwise, HTTP will be used.

For example, if you want to perform a GET request on your local server at localhost:3000, you’ll want to set the --url to be localhost:3000:

curl --request GET \
    --url http://localhost:3000

To perform a POST at the same URL:

curl --request POST \
    --url http://localhost:3000

Note: The backslash \ is used to separate options in cURL.

For an external API, it’s still the same thing.

Let’s say you want to fetch a list of movies from https://moviesdb.com/movies/all.

curl --request GET \
    --url https://moviesdb.com/movies/all

The list of all movies in moviesdb will be fetched and printed.

--header or -H

This option is used to set headers to requests.

From the cURL docs:

Extra header to include in the request when sending HTTP to a server. You may specify any number of extra headers. Note that if you should add a custom header that has the same name as one of the internal ones curl would use, your externally set
header will be used instead of the internal one.

This mirrors what we do in normal programming, especially in JavaScript using XMLHttpRequest:

const xhttp = new XMLHttpRequest()

xhttp.setHeader("Content-Type", "application/json")

Headers are used to communicate to the web server the type of data coming in, or the type of data to expect. The data type sent should be of the same type specified in the headers.

We can use headers for CORS permission, and even for permission on certain types of request methods. There is a plethora of stuff we can do with headers.

So in cURL, we can set headers using the --header option:

curl --request POST \
  --url http://localhost:5000/api/user \
  --header 'content-type: application/json'

Here, we are making a POST request to http://localhost:5000/api/user endpoint and we are telling the server the data we are sending is a JSON data type via the --header content-type: application/json'.

--data or -d

This option is used to send data to an HTTP server. This is mostly used in POST requests because we send data to the server we want to add to the datastore. So in cURL, we have the option of specifying our data to POST by setting the --data option.

From the cURL manual:

Sends the specified data in a POST request to the HTTP server, in the same way that a browser does when a user has filled in an HTML form and presses the submit button.

Here’s an example:

curl --request POST \
  --url http://localhost:5000 \
  --header 'content-type: application/json' \
  --data '{"name":"Arrow","description":"bad movie","rating":"7.0","image":"michaeljackson.png"}'

Here, we are performing a POST request to http://localhost:5000 endpoint.

We set the data to send to the server in the --data option, which is: '{"name":"Arrow","description":"bad movie","rating":"7.0","image":"michaeljackson.png"}.

This is a movie object we want to add to the http://localhost:5000 datastore.

Using cURL with Node.js/Express.js

Let’s see how we can set up a Node.js/Express.js server and use cURL to test the endpoints:

// server.js

/** require dependencies */
const express = require("express")
const cors = require('cors')
const bodyParser = require('body-parser')
const helmet = require('helmet')

const app = express()

let port = 5000 || process.env.PORT

/** set up middlewares */
app.use(cors())
app.use(bodyParser.json({limit: '50mb'}))
app.use(helmet())

let users = []
let currId  Date.now()

app.get("api/user/:id", (req, res, next) => {
        const userId = req.params.id
        const users = users.filter(user => user.id === userId)
        if(users.length > 0)
            res.send({ user: users[0] })
        else
            res.send({mesg: "No user found."})
        next()
})

app.get("api/users", (req, res, next) => {
    res.send({ users })
    next()
})

app.post("api/user", (req, res, next) => {
        let bdy = req.body
        bdy = { id: ++currId, ...bdy }
        users.push(bdy)
        res.send({ user: bdy })
        next()    
})

/** start server */
app.listen(port, () => {
    console.log(`Server started at port: ${port}`);
});

We have small API endpoints here:

  • GEt api/users
  • GET api/user/:id
  • POST api/user

It fetches all users, a specific user by its id, and can add a new user.

Fire up the server:

node server.js

Server started at port: 5000

Now, to perform a POST request at api/user, we will set the --url to http://localhost:5000/api/user, and --request to POST, the --header to content-type: application/json.

We want to add the user:

name: "Chidume",
age: 28

So the data will be '{"name": "Chidume", "age": "28"}'.

So the cURL command will be:

curl --request POST \
  --url http://localhost:5000/api/user \
  --header 'content-type: application/json' \
  --data '{"name": "Chidume", "age": "28"}'

{ id: 5122435464, name: "Chidume", age: 28 }

We see the result from the server: the user is added to the datastore with its id.

Let’s add another user:

curl --request POST \
  --url http://localhost:5000/api/user \
  --header 'content-type: application/json' \
  --data '{"name": "Nnamdi", "age": "21"}'

{ id: 5122435465, name: "Nnamdi", age: 21 }

We added another user, “Nnamdi”, and we can see the result from the server with its id.

Now, let’s query for “Chidume” via the api/user/:id. Here, :id will be the id 5122435464. So the cURL will be:

curl --request GET \
  --url http://localhost:5000/api/user/5122435464

{ user: { id: 5122435464, name: "Chidume", age: 28 } }

The result from the server returns the user with id 5122435464 which is “Chidume”.

To query for “Nnamdi”:

curl --request GET \
  --url http://localhost:5000/api/user/5122435465

{ user: { id: 5122435465, name: "Nnamdi", age: 21 } }

Ta-da!!

Now let’s query for a non-existent user:

curl --request GET \
  --url http://localhost:5000/api/user/2000000000


{ mesg: "No user found." }

That’s the result set on the server to return when we hit a non-existent user.

Now, let’s query ALL users:

curl --request GET \
  --url http://localhost:5000/api/users


{ users: [ { id: 5122435465, name: "Nnamdi", age: 21 }, { id: 5122435464, name: "Chidume", age: 28 } ] }

As you can see, all the users in the server are returned.

Conclusion

In this tutorial, we won’t over the basic options of cURL, with examples on how to use each of them, We then we developed a Node/Express server with three endpoints to demonstrate how we can use the cURL options addressed here to perform GET/POST request with cURL to the server.

There are many options to use with cURL. What we addressed here are the basic options you can use to quickly test your API endpoints.

Comments

Post a Comment

Popular posts from this blog

4 Ways to Communicate Across Browser Tabs in Realtime

1. Local Storage Events You might have already used LocalStorage, which is accessible across Tabs within the same application origin. But do you know that it also supports events? You can use this feature to communicate across Browser Tabs, where other Tabs will receive the event once the storage is updated. For example, let’s say in one Tab, we execute the following JavaScript code. window.localStorage.setItem("loggedIn", "true"); The other Tabs which listen to the event will receive it, as shown below. window.addEventListener('storage', (event) => { if (event.storageArea != localStorage) return; if (event.key === 'loggedIn') { // Do something with event.newValue } }); 2. Broadcast Channel API The Broadcast Channel API allows communication between Tabs, Windows, Frames, Iframes, and  Web Workers . One Tab can create and post to a channel as follows. const channel = new BroadcastChannel('app-data'); channel.postMessage(data); And oth...
Post a Comment
Read more

Certbot SSL configuration in ubuntu

  Introduction Let’s Encrypt is a Certificate Authority (CA) that provides an easy way to obtain and install free  TLS/SSL certificates , thereby enabling encrypted HTTPS on web servers. It simplifies the process by providing a software client, Certbot, that attempts to automate most (if not all) of the required steps. Currently, the entire process of obtaining and installing a certificate is fully automated on both Apache and Nginx. In this tutorial, you will use Certbot to obtain a free SSL certificate for Apache on Ubuntu 18.04 and set up your certificate to renew automatically. This tutorial will use a separate Apache virtual host file instead of the default configuration file.  We recommend  creating new Apache virtual host files for each domain because it helps to avoid common mistakes and maintains the default files as a fallback configuration. Prerequisites To follow this tutorial, you will need: One Ubuntu 18.04 server set up by following this  initial ...
Post a Comment
Read more

Working with Node.js streams

  Introduction Streams are one of the major features that most Node.js applications rely on, especially when handling HTTP requests, reading/writing files, and making socket communications. Streams are very predictable since we can always expect data, error, and end events when using streams. This article will teach Node developers how to use streams to efficiently handle large amounts of data. This is a typical real-world challenge faced by Node developers when they have to deal with a large data source, and it may not be feasible to process this data all at once. This article will cover the following topics: Types of streams When to adopt Node.js streams Batching Composing streams in Node.js Transforming data with transform streams Piping streams Error handling Node.js streams Types of streams The following are four main types of streams in Node.js: Readable streams: The readable stream is responsible for reading data from a source file Writable streams: The writable stream is re...
Post a Comment
Read more