it-swarm.com.de

Cloudflare speichert die Seite nicht über eine CURL-Anfrage

One.php erstellt zwei separate CURL-Anfragen an two.php . Ich möchte, dass Cloudflare die erste Antwort zwischenspeichert und diesen Cache in der zweiten Antwort bereitstellt. Aus irgendeinem Grund speichert Cloudflare jedoch keine der Antworten zwischen. Wenn ich dann two.php mit meinem Browser besuche, wird er immer noch nicht zwischengespeichert. Wenn ich dann in meinem Browser two.php aktualisiere, erhalte ich endlich einen Cache-HIT. Wie löse ich Caching in Cloudflare über CURL-Anforderungen aus?

Bare-Bones-Test:

./one.php

header("Cache-Control: no-store");
$url = "http://www.example.com/two.php";

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url );
curl_exec($ch); // prints current timestamp, see two.php
curl_close($ch);

sleep(1);

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url );
curl_exec($ch); // prints current timestamp, see two.php
curl_close($ch);

./two.php

header( 'Cache-Control: public, max-age=10' );
print( time()."<br>\r\n" );

Ausgiebiger Test:

Ich habe versucht, den User-Agent-Header zu setzen und das cfduid-Cookie von Cloudflare zu verwenden. Ich habe versucht, die CURL-Anforderungen auf und zwischen zwei verschiedenen Websites auf zwei verschiedenen Servern abzusetzen. Ich habe versucht, HTTP und HTTPS. Ich habe versucht, one.php von direct.example.com aus auszuführen (wodurch Cloudflare umgangen wird).

./one.php

header("Cache-Control: no-store");

$url = "http://www.example.com/target.php";
$request_headers = array(
  'Connection: keep-alive',
  'Upgrade-Insecure-Requests: 1',
  'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36',
  'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
  'Accept-Encoding: compressed',
  'Accept-Language: en-US,en;q=0.9',
);

// Fetch CFDUID Cookie
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url );
curl_setopt($ch, CURLOPT_HTTPHEADER, $request_headers);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_NOBODY, 1);
$response_headers = curl_exec($ch);
curl_close($ch);

// Extract Cookies
preg_match_all( '/^Set-Cookie:\s*(.+?);?\s*$/mi', $response_headers, $cookies);
$cookies = implode( ';', $cookies[1] );

// Setup First request
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url );
curl_setopt($ch, CURLOPT_HTTPHEADER, $request_headers);
curl_setopt($ch, CURLOPT_COOKIE, $cookies );

// Print First Response
echo( "First Response:<br>\r\n<pre>" );
curl_exec($ch);
curl_close($ch);
echo( "</pre>" );

// Pause
sleep(1);

// Setup Second request
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url );
curl_setopt($ch, CURLOPT_HTTPHEADER, $request_headers);
curl_setopt($ch, CURLOPT_COOKIE, $cookies );

// Record CURL Request Info
curl_setopt($ch, CURLOPT_VERBOSE, true);
$verbose = fopen('php://temp', 'w+');
curl_setopt($ch, CURLOPT_STDERR, $verbose);

// Print Second Response
echo( "Second Response:<br>\r\n<pre>" );
curl_exec($ch);
curl_close($ch);
echo( "</pre>" );

// Show CURL Request Info
rewind($verbose);
$verboseLog = stream_get_contents($verbose);
echo "CURL Verbose Info:<br>\r\n<pre>", htmlspecialchars($verboseLog), "</pre>\r\n";

./two.php

header("Cache-Control: public, max-age=10");
header( "set-cookie: test1=true; test2=true" );
echo( time() . "\r\n" );
var_dump( getallheaders() );

Erweiterte Testausgabe (zensiert):

First Response:
1536190748
array(25) {
  ["Content-Type"]=>
  string(0) ""
  ["Content-Length"]=>
  string(1) "0"
  ["X-Original-Url"]=>
  string(20) "/target.php"
  ["Was-Default-Hostname"]=>
  string(33) "*****.azurewebsites.net"
  ["X-Site-Deployment-Id"]=>
  string(15) "*****"
  ["Disguised-Host"]=>
  string(21) "www.*****.com"
  ["X-Arr-Log-Id"]=>
  string(36) "c157*****e003"
  ["Is-Service-Tunneled"]=>
  string(1) "0"
  ["Client-Ip"]=>
  string(21) "*.*.*.175:18968"
  ["X-Waws-Unencoded-Url"]=>
  string(20) "/target.php"
  ["Cf-Connecting-Ip"]=>
  string(12) "*.*.*.33"
  ["Upgrade-Insecure-Requests"]=>
  string(1) "1"
  ["Cf-Visitor"]=>
  string(17) "{"scheme":"http"}"
  ["X-Forwarded-Proto"]=>
  string(4) "http"
  ["Cf-Ray"]=>
  string(20) "455c*****-ATL"
  ["X-Forwarded-For"]=>
  string(35) "*.*.*.33, *.*.*.175:18968"
  ["Cf-Ipcountry"]=>
  string(2) "US"
  ["User-Agent"]=>
  string(115) "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/*.*.*.106 Safari/537.36"
  ["Max-Forwards"]=>
  string(2) "10"
  ["Host"]=>
  string(21) "www.*****.com"
  ["Cookie"]=>
  string(279) "__cfduid=d2be*****0748; expires=Thu, 05-Sep-19 23:39:08 GMT; path=/; domain=.*****.com; HttpOnly;test1=true; test2=true;ARRAffinity=91cd*****4f3f;Path=/;HttpOnly;Domain=www.*****.com"
  ["Accept-Language"]=>
  string(14) "en-US,en;q=0.9"
  ["Accept-Encoding"]=>
  string(4) "gzip"
  ["Accept"]=>
  string(85) "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8"
  ["Connection"]=>
  string(10) "Keep-Alive"
}

Second Response:
1536190749
array(25) {
  ["Content-Type"]=>
  string(0) ""
  ["Content-Length"]=>
  string(1) "0"
  ["X-Original-Url"]=>
  string(20) "/target.php"
  ["Was-Default-Hostname"]=>
  string(33) "*****.azurewebsites.net"
  ["X-Site-Deployment-Id"]=>
  string(15) "*****"
  ["Disguised-Host"]=>
  string(21) "www.*****.com"
  ["X-Arr-Log-Id"]=>
  string(36) "588f*****3a02"
  ["Is-Service-Tunneled"]=>
  string(1) "0"
  ["Client-Ip"]=>
  string(21) "*.*.*.175:19820"
  ["X-Waws-Unencoded-Url"]=>
  string(20) "/target.php"
  ["Cf-Connecting-Ip"]=>
  string(12) "*.*.*.33"
  ["Upgrade-Insecure-Requests"]=>
  string(1) "1"
  ["Cf-Visitor"]=>
  string(17) "{"scheme":"http"}"
  ["X-Forwarded-Proto"]=>
  string(4) "http"
  ["Cf-Ray"]=>
  string(20) "455c8*****-ATL"
  ["X-Forwarded-For"]=>
  string(35) "*.*.*.33, *.*.*.175:19820"
  ["Cf-Ipcountry"]=>
  string(2) "US"
  ["User-Agent"]=>
  string(115) "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/*.*.*.106 Safari/537.36"
  ["Max-Forwards"]=>
  string(2) "10"
  ["Host"]=>
  string(21) "www.*****.com"
  ["Cookie"]=>
  string(279) "__cfduid=d2be*****0748; expires=Thu, 05-Sep-19 23:39:08 GMT; path=/; domain=.*****.com; HttpOnly;test1=true; test2=true;ARRAffinity=91cd*****4f3f;Path=/;HttpOnly;Domain=www.*****.com"
  ["Accept-Language"]=>
  string(14) "en-US,en;q=0.9"
  ["Accept-Encoding"]=>
  string(4) "gzip"
  ["Accept"]=>
  string(85) "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8"
  ["Connection"]=>
  string(10) "Keep-Alive"
}

CURL Verbose Info:
* Hostname www.*****.com was found in DNS cache
*   Trying *.*.*.46...
* TCP_NODELAY set
* Connected to www.*****.com (*.*.*.46) port 80 (#0)
> GET /target.php HTTP/1.1
Host: www.*****.com
Cookie: __cfduid=d2be*****0748; expires=Thu, 05-Sep-19 23:39:08 GMT; path=/; domain=.*****.com; HttpOnly;test1=true; test2=true;ARRAffinity=91cd*****4f3f;Path=/;HttpOnly;Domain=www.*****.com
Connection: keep-alive
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/*.*.*.106 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Accept-Encoding: compressed
Accept-Language: en-US,en;q=0.9

< HTTP/1.1 200 OK
< Date: Wed, 05 Sep 2018 23:39:10 GMT
< Content-Type: text/html; charset=UTF-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< Cache-Control: public, max-age=10
< Vary: Accept-Encoding
< Set-Cookie: test1=true; test2=true
< X-Powered-By: PHP/5.6.37
< X-Powered-By: ASP.NET
< CF-Cache-Status: MISS
< Server: cloudflare
< CF-RAY: 455c*****-ATL
< 
* Connection #0 to Host www.*****.com left intact
7
skibulk

Anfragen an CloudFlare werden geografisch verteilt. Möglicherweise erreicht Ihr Server einen anderen CloudFlare Edge-Standort als Sie (derzeit sind es 152), sodass Sie nicht von dessen Anrufen profitieren.

1
ceejayoz

Schauen Sie sich dieses Github-Repo an Cloudflare-Bypass . Sein Code hat einen Parameter cache on/off. Was ich für hilfreich halte. Seht, wie er sein Code-Snippet bläst.

<?php
require __DIR__ . '/../vendor/autoload.php';
use CloudflareBypass\RequestMethod\CFCurl;
$curl_cf_wrapper = new CFCurl(array(
    'cache'         => true,   // Caching now enabled by default; stores clearance tokens in Cache folder
    'max_retries'   => 5       // Max attempts to try and get CF clearance
));
// Get Example: 1
$ch = curl_init("http://thebot.net/");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36');
echo $curl_cf_wrapper->exec($ch); // Done! NOTE: HEAD requests not supported!
curl_close($ch);
1
elegant-user