- curl: how to specify target hostname for https request
- 3 Answers 3
- How to send a header using a HTTP request through a cURL call?
- 11 Answers 11
- Example 1: Single Header
- Example 2: Multiple Headers
- 10 cURL Command Usage with Real-Time Example
- Verify if you can connect to the URL
- Save URL/URI output to file
- Show request and response header
- Using a proxy to connect
- Test URL with injecting header
- Display only response header
- Connect HTTPS/SSL URL and ignore any SSL certificate error
- Connect using a specific protocol (SSL/TLS)
- Download file from FTP Server
- Using Host Header
- How about using cURL online?
curl: how to specify target hostname for https request
I have a x.example which serves traffic for both a.example and b.example . x.example has certificates for both a.example and b.example . The DNS for a.example and b.example is not yet set up. If I add an /etc/hosts entry for a.example pointing to x.example ‘s ip and run curl -XGET https://a.example , I get a 200. However if I run curl —header ‘Host: a.example’ https://x.example , I get:
I would think it would use a.example as the host. Maybe I’m not understanding how SNI/TLS works. Because a.example is an HTTP header the TLS handshake doesn’t have access to it yet? But the URL itself it does have access to?
3 Answers 3
Indeed SNI in TLS does not work like that. SNI, as everything related to TLS, happens before any kind of HTTP traffic, hence the Host header is not taken into account at that step (but will be useful later on for the webserver to know which host you are connecting too).
So to enable SNI you need a specific switch in your HTTP client to tell it to send the appropriate TLS extension during the handshake with the hostname value you need.
In case of curl , you need at least version 7.18.1 (based on https://curl.haxx.se/changes.html) and then it seems to automatically use the value provided in the Host header. It alo depends on which OpenSSL (or equivalent library on your platform) version it is linked to.
See point 1.10 of https://curl.haxx.se/docs/knownbugs.html that speaks about a bug but explains what happens:
When given a URL with a trailing dot for the host name part: «https://example.com./», libcurl will strip off the dot and use the name without a dot internally and send it dot-less in HTTP Host: headers and in the TLS SNI field.
The —connect-to option could also be useful in your case. Or —resolve as a substitute to /etc/hosts , see https://curl.haxx.se/mail/archive-2015-01/0042.html for am example, or https://makandracards.com/makandra/1613-make-an-http-request-to-a-machine-but-fake-the-hostname You can add —verbose in all cases to see in more details what is happening. See this example: https://www.claudiokuenzler.com/blog/693/curious-case-of-curl-ssl-tls-sni-http-host-header ; you will also see there how to test directly with openssl .
If you have a.example in your /etc/hosts you should just run curl with https://a.example/ and it should take care of the Host header and hence SNI (or use —resolve instead)
So to answer your question directly, replace
curl --header 'Host: a.example' https://x.example curl --connect-to a.example:443:x.example:443 https://a.example and it should work perfectly.
How to send a header using a HTTP request through a cURL call?
There is a good way to learn how to use curl for http requests by examples. Download the newest version of Postman, make any http request configuration as you wish at user interface level (post, put, get.. for instance, with headers and json body ) and then click in «generate code» and choose «curl» option. It gives you the equivalent command line.
11 Answers 11
-H/--header (HTTP) Extra header to use when getting a web page. You may specify any number of extra headers. Note that if you should add a custom header that has the same name as one of the internal ones curl would use, your externally set header will be used instead of the internal one. This allows you to make even trickier stuff than curl would normally do. You should not replace internally set headers without knowing perfectly well what you're doing. Remove an internal header by giving a replacement without content on the right side of the colon, as in: -H "Host:". curl will make sure that each header you add/replace get sent with the proper end of line marker, you should thus not add that as a part of the header content: do not add newlines or carriage returns they will only mess things up for you. See also the -A/--user-agent and -e/--referer options. This option can be used multiple times to add/replace/remove multi- ple headers. Example 1: Single Header
curl --header "X-MyHeader: 123" www.google.com Example 2: Multiple Headers
curl --header "Accept: text/javascript" --header "X-Test: hello" -v www.google.com You can see the request that curl sent by adding the -v option.
10 cURL Command Usage with Real-Time Example
Invicti Web Application Security Scanner – the only solution that delivers automatic verification of vulnerabilities with Proof-Based Scanning™.
cURL is a command-line tool to get or send data using URL syntax.
If you are working as a developer or in the support function, you must be aware of cURL command usage to troubleshoot web applications. cURL is a cross-platform utility means you can use on Windows, MAC, and UNIX.
The following are some of the most used syntaxes with an example to help you.
Verify if you can connect to the URL
If you are working on a UNIX system and trying to connect the external URL, the first thing you wish to do is to check if you can access the URL through curl.
It will not throw any output. However, if the server can’t connect, then you will get errors such as could not resolve the host.
[root@gf-lab tmp]# curl helloitdoesntexist.com curl: (6) Could not resolve host: helloitdoesntexist.com; Unknown error [root@gf-lab tmp]#Save URL/URI output to file
If you have to save the URL or URI contents to a specific file, you can use the following syntax
curl https://yoururl.com > yoururl.html[root@gf-lab tmp]# curl https://gf.dev > /tmp/gfhtml % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 18557 0 18557 0 0 72565 0 --:--:-- --:--:-- --:--:-- 72772 [root@gf-lab tmp]#Above example will save all the contents from gf.dev to /tmp/gf.html
Show request and response header
If you are having issues and would like to validate, you are getting the expected request and response header.
[root@gf-lab tmp]# curl -v https://geekflare.com * About to connect() to geekflare.com port 443 (#0) * Trying 104.25.134.107. * Connected to geekflare.com (104.25.134.107) port 443 (#0) * Initializing NSS with certpath: sql:/etc/pki/nssdb * CAfile: /etc/pki/tls/certs/ca-bundle.crt CApath: none * SSL connection using TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 * Server certificate: * subject: CN=ssl371609.cloudflaressl.com,OU=PositiveSSL Multi-Domain,OU=Domain Control Validated * start date: Nov 07 00:00:00 2019 GMT * expire date: May 15 23:59:59 2020 GMT * common name: ssl371609.cloudflaressl.com * issuer: CN=COMODO ECC Domain Validation Secure Server CA 2,O=COMODO CA Limited,L=Salford,ST=Greater Manchester,C=GB > GET / HTTP/1.1 > User-Agent: curl/7.29.0 > Host: geekflare.com > Accept: */* > < HTTP/1.1 200 OK < Date: Sat, 09 Nov 2019 19:41:37 GMT < Content-Type: text/html; charset=UTF-8 < Transfer-Encoding: chunked < Connection: keep-alive < Set-Cookie: __cfduid=d2ce6cd359ebc0b6eb5ff3a454ed042021573328497; expires=Sun, 08-Nov-20 19:41:37 GMT; path=/; domain=.geekflare.com; HttpOnly; Secure < Vary: Accept-Encoding < Link: ; rel="https://api.w.org/" < Link: ; rel=shortlink < X-SRCache-Fetch-Status: HIT < X-SRCache-Store-Status: BYPASS < X-Frame-Options: SAMEORIGIN < X-Powered-By: EasyEngine v4.0.12 < Via: 1.1 google < CF-Cache-Status: DYNAMIC < Strict-Transport-Security: max-age=15552000; preload < X-Content-Type-Options: nosniff < Alt-Svc: h3-23=":443"; ma=86400 < Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct" < Server: cloudflare < CF-RAY: 533243e4bcd4bbf4-LHR Download at a limit rate
If you are working on optimization and would like to see how much time does it take to download at a particular speed, you can:-
curl –-limit-rate 2000B https://gf.devUsing a proxy to connect
Very handy if you are working on the DMZ server where you need to connect to the external world using a proxy.
curl --proxy yourproxy:port https://yoururl.comTest URL with injecting header
You can use curl by inserting a header with your data to test or troubleshoot the particular issue. Let’s see the following example to request with Content-Type.
curl --header 'Content-Type: application/json' http://yoururl.comBy doing above, you are asking curl to pass Content-Type as application/json in the request header.
Display only response header
If you are doing some troubleshooting and quickly want to check the response header, you can use the following syntax.
[root@gf-lab tmp]# curl --head https://chandan.io HTTP/1.1 200 OK Date: Sat, 09 Nov 2019 19:51:23 GMT Content-Type: text/html Connection: keep-alive Set-Cookie: __cfduid=d3cb2c7b8e566ad99c870b0af12b0f1eb1573329083; expires=Sun, 08-Nov-20 19:51:23 GMT; path=/; domain=.chandan.io; HttpOnly X-GUploader-UploadID: AEnB2Uo96JhvJmR2zYUL-Ndh2ta3UD_ykQAB5C7O8cjZQhCf-GxHQ0MsodSzRnl3guSN3ywAYNjtWcPXfwDXjLg3bQ-P5vQMOA Expires: Sat, 09 Nov 2019 20:51:23 GMT Cache-Control: public, max-age=3600 Last-Modified: Mon, 06 Aug 2018 10:45:47 GMT x-goog-generation: 1533552347482034 x-goog-metageneration: 1 x-goog-stored-content-encoding: identity x-goog-stored-content-length: 24620 x-goog-hash: crc32c=DpDPAQ== x-goog-hash: md5=cIP/3rusdUx12Zla1kf1yA== x-goog-storage-class: MULTI_REGIONAL Accept-Ranges: bytes CF-Cache-Status: DYNAMIC Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct" Server: cloudflare CF-RAY: 53325234dc2fbb9a-LHR [root@gf-lab tmp]#Connect HTTPS/SSL URL and ignore any SSL certificate error
When you try to access SSL/TLS cert secured URL and if that is having the wrong cert or CN doesn’t match, then you will get the following error.
curl: (51) Unable to communicate securely with peer: requested domain name does not match the server's certificate.Good news, you can instruct cURL to ignore the cert error with --insecure flag.
curl --insecure https://yoururl.comConnect using a specific protocol (SSL/TLS)
Very handy to test if a particular URL can handshake over a specific SSL/TLS protocol.
curl --sslv3 https://yoururl.comand for different TLS versions
curl --tlsv1 https://example.com curl --tlsv1.0 https://example.com curl --tlsv1.1 https://example.com curl --tlsv1.2 https://example.com curl --tlsv1.3 https://example.comDownload file from FTP Server
You can use curl to download the file as well by specifying your username and password.
curl -u user:password -O ftp://ftpurl/style.cssYou can always use “-v” with any syntax to print in verbose mode.
Using Host Header
The host header is useful to test the target URL over IP when the requested content is only available when the host header matches. Or, if you want to test the application using load balancer IP/URL.
curl --header 'Host: targetapplication.com' https://192.0.0.1:8080/How about using cURL online?
Yes, that’s possible with the following tools. You can execute cURL remotely.
Online CURL – a lightweight tool to fetch the URL online and the possibility to add the following options.
--connect-timeout --cookie --data --header --head --location --max-time --proxy --request --user --url --user-agentcURL command line builder – this one is different. It helps you to build the curl command where you can enter information in nice UI, and at the bottom, you will get the cURL command.
cURL is a useful utility to troubleshoot a real-time connectivity issue, and I hope the above helps you. If you are looking to learn more, then I would recommend Linux Command Line Basics online course.